Harbor benchmark
Agents do better work with Nous AI: +14 points.
Same agent, same task, with and without Nous AI. With Nous AI, agents met 88% of each task's checks; without it, 74%. Every model tested improved.
Harbor standard v2: 38 paired runs, run by Nous AI on three small, fast models, closed 1 November 2026. Switch between With and Without to see the gain. Round v3, on the rewritten skills, is running; these figures change when it closes. Scores are the share of each task's automated checks met.
+14 pts
average gain with Nous AI (88% vs 74%)
28 of 38
runs won by Nous AI; 3 lost, 7 tied
3 of 3
models improved, +13 to +15 pts each
+9.4 to +18.5
95% interval of the gain: above zero throughout
Current round · harbor-standard-v4
Every skill, every model, as the runs come in.
The skills agents get today, each run one model doing one work task twice, with the skill and without it. Target: 15 runs per skill and model.
No runs yet in this round. Runs arrive as the worker completes them.
Previous round · harbor-standard-v2
The skills before their latest rewrite.
The same tasks, checks and harness, measuring the previous version of the skills. Kept on its own, never pooled with the current round.
38 runs · with the skill 88% vs 74% without (+14 pts; 95% interval +9.4 pts to +18.5 pts) · 28 wins, 3 losses, 7 ties · last run 1 Nov, 15:53 UTC
| Skill | Model | Runs | With | Without | Gain | Won · lost · tied |
|---|---|---|---|---|---|---|
| Teaching (10 of 15) | Claude | 4 | 100% | 94% | +6 pts | 2 · 0 · 2 |
| Gemma | 3 | 88% | 77% | +11 pts | 2 · 0 · 1 | |
| gpt-oss | 3 | 85% | 71% | +14 pts | 2 · 0 · 1 | |
| Curriculum design (6 of 15) |
Pick a task
A containerized job with a written brief and fixed checks.
Run both sides
The Nous AI side gets skills; the baseline gets none.
Score the file
Code checks the deliverable. No model judges it.
Compare the pair
The result is the difference Nous AI made on that task.
Methodology
How the Harbor benchmark works.
The Harbor benchmark is Nous AI's own. It uses Harbor, an open-source framework that runs AI agents such as Claude Code, Codex, Gemini CLI and OpenHands on tasks inside containers, and scores each result with a verifier.
A container per task
Each task ships a written instruction, a starting environment and a verifier, so every run starts from the same files.
Two variants of each task
The Nous AI variant has Nous AI's skills installed where agents look for them, with an AGENTS.md. The baseline gets the same instruction and environment, without skills.
One skill pipeline per round
Each round measures the skills one version of Nous AI's skill pipeline writes. A skill may be rewritten within a round as its data changes; a new pipeline starts a new round.
Earlier run · October 2026
An earlier, smaller run on a different setup.
Harbor in Docker, fewer tasks, four to thirteen runs per model. Kept as history; the rounds above are the measure.
| Model (repository default) | Without Nous AI | With Nous AI | Gain | Runs |
|---|---|---|---|---|
| gpt-oss 20B | 63.2% | 76.7% | +13.5 | 13 |
| Claude Haiku 4.5 | 87.3% | 91.9% | +4.6 | 4 |
| Gemma 4 | 67.3% | 74.3% | +7.0 | matched |
Matched comparison, means across the matched runs: 81.9% with Nous AI, 77.8% without (+4.1 pts). A small sample on a different harness: it does not carry over to the current skills.
Next areas
Sales, customer success and operations.
Not benchmarked yet. Results for them will be published once their tasks run.
1 of 3
Sales
Prospecting, pipeline management and outreach workflows.
Example: "Build a prospecting plan for 50 target companies."
Not yet benchmarked.