dollama · model field report · updated jul 2026
We run the best models the network can actually carry.
dollama standardises on a shared set of open models so every request gets consistent quality, wherever it lands. Picking them means weighing two things at once: how good a model is, and whether the machines people contribute can run it well. This page is the standing record of how we choose: the method, the ten candidates we graded, and why each network role runs the model it does — decisions we'll keep re-making as the network grows.
How we choose
Every model gets onto — or off — the network the same way: a broad screen, a full-tier stress test for any role that's up for a decision, and a hand-read of the transcripts. The same gate applies whether a model is new to the board or defending a slot it already holds.
Quick tier, every candidate, all three roles — 6 scenarios × 1 repeat on a reference RTX 3060. Cheap and deliberately coarse: scores land on a 1/6 grid, so a tie at 100 means "in contention", not "proven equal".
Full tier: 3 repeats, harder scenarios (including a 338-tool catalogue), real confidence intervals. The quick-tier ceilings break apart and the contenders separate.
Numbers can't see a hollow pass or a format fumble. Every shortlisted run is read by hand; where scores tie, substance and cost per request decide.
Capability, native, and the gate between them
Every graded run ends one of three ways: native (solved cleanly, first shot), recovered (the model stumbled, dollama's retry/recovery loop re-prompted it to a good answer), or failed. Capability counts success with the loop allowed; native counts first-shot passes only.
The recovery loop is real dollama value — qwen3.5:4b as a worker passes 62% first-shot yet grades 100 capability with the loop behind it. To stop that from flattering fragile models, Recommended requires the 95% (Wilson) lower bound of the native score to clear the role's bar. A model the pipeline merely rescues can't qualify.
The candidates
Ten models graded across dollama's three roles — planner, worker, helper. Each cell is the quick-tier capability score (0–100) with the first-shot native score beneath, coloured by verdict. Speed and memory are measured on a 12 GB RTX 3060: VRAM runs base (weights, at 4k) → peak (at the model's max context, mostly KV); usable context is the inflection point where throughput starts to fall (models survive further, slower).
| Model | qualityPlanner | qualityWorker | qualityHelper | toolsBasics | measuredSpeed t/s | measuredVRAM base→peak | measuredUsable ctx |
|---|
Where the scores lie — in both directions
A binary tool-call grader misleads both ways, which is why step 3 of the method exists. False negatives: the phi4 family (the † rows) reasons well but narrates <note> tags or declines the call instead of emitting Read() — those scores measure format adherence, not capability. False positives: correct tool plumbing can pass a task on hollow reasoning — string checks matching the model's own tool output rather than any answer it wrote — and we've caught that in high scorers, not just stragglers (a grader fix is on the list). No grade on this page stands alone: the "read by hand" columns in §04 carry as much weight as the numbers.
Throughput and memory as the context window grows, on the reference RTX 3060. The dot on each line marks the inflection point — where speed starts dropping off. Tap a model to isolate it.
What contributor machines can run
Quality is only half the decision — a network model has to run well on the machines people actually contribute. Today's fleet is a handful of hand-built machines, and it sets the second constraint on every pick.
The honest headline: on the three real GPUs, almost every model in the board runs — the differentiator isn't whether it fits but how much context and how fast. qwen3.5:9b and gemma4:12b both peak near 7.6–7.8 GB and hold a 64k usable window (128k survivable) on the 12 GB and 16 GB machines; on the 10 GB 3080 they run comfortably at a trimmed context.
The lone exception is phi4:14b: it climbs to ~10 GB and caps out near a 16k window, so it fits the 3060 and the M1 but crowds the 3080. That memory ceiling — not raw capability — is exactly the kind of thing that keeps a model off the standard fleet.
So we lean on models that are both strong and comfortable on mid-range contributor cards. As bigger GPUs join, the heavier tiers open up on their own.
Each machine plotted by its VRAM and how fast it runs the universal helper model (qwen3.5:4b). Filled dot = measured; hollow = estimated from GPU class — only the RTX 3060 is directly benchmarked. A role needs both enough VRAM for its model (helper ~3 GB, worker ~6 GB, planner ~8 GB) and enough speed to serve (~12 t/s) — the bands are rough. All three real GPUs clear every role; the CPU/iGPU stragglers can't hold a model at all (they help with speech-to-text instead).
The network roles — and the model each gets
Every request touches up to three roles: a planner decomposes the task, a worker executes it, and a helper handles curation, summarisation and sub-agents. Each role gets one standard model — chosen by the §01 gate, constrained by the §03 hardware.
Writes the decomposition plan the worker executes — it runs on every request, so cost weighs as much as quality. Highest-graded planner and roughly a third the tokens of the runner-up.
The main inference: reasons, calls tools, edits code, streams the answer. Recommended at full tier — identical quality to the 12B alternative, ~47% faster to serve.
Curation, summarisation, classifier calls, sub-agents. Deliberately the same model as the planner — one artifact serves both roles and fits 2 GB-class machines.
Three roles, two models — on purpose. Planner and helper share qwen3.5:4b; what separates them is configuration, not weights (the planner keeps a higher context floor, 16k vs 8k, so plans never truncate). The rest of this section is the full-tier evidence behind each slot.
| Planner candidate | Capability | Native · 95% CI | Verdict | Plan quality — read by hand | Tokens | Time |
|---|---|---|---|---|---|---|
| qwen3.5:4b | 85 | 79% · 62–89 | Recommended | Substantive — concrete plans with correct line citations | 841 | 44s |
| ornith:9b | 78 | 82% · 66–91 | Recommended | Inflated — 2 of 5 "passes" are empty answers that cleared string checks on tool output alone | 739 | 75s |
| gemma4:12b | 76 | 79% · 62–89 | Marginal | Correct but verbose — 1.8× the tokens, and weak on the anti-stall scenario | 1528 | 113s |
| qwen3.5:9b | 74 | 79% · 62–89 | Recommended | Mostly real, but misses the profile-planning task entirely (0/3) | 962 | 62s |
The planner: qwen3.5:4b
It wins on all three axes at once. Highest capability, genuinely substantive plans (we read them — precise and correctly cited), and the most efficient by a wide margin: a good plan in 841 tokens / 44s against gemma4:12b's 1528 / 113s, for a step that runs on every request. ornith's near-identical score is a mirage — two of its "passes" are empty answers that satisfied the string checks from tool output alone (the false-positive mode from §02), so its real plan quality sits below the number. gemma4:12b is thorough but Marginal here, and pays for its depth in tokens and seconds. With turn counts essentially equal, the honest separators are substance and cost — and qwen3.5:4b takes both.
| Worker candidate | Capability | Native · 95% CI | Verdict | Behaviour — read by hand | Tokens |
|---|---|---|---|---|---|
| qwen3.5:9b | 100 | 95% · 77–99 | Recommended | Fast, native-clean, and finds the task-relevant tool 3/3 first-shot in a 338-tool catalogue. The leanest of the five. Our worker pick. | 103 |
| gemma4:12b | 100 | 95% · 77–99 | Recommended | Matches qwen3.5:9b on quality, needle included 3/3. The cost is verbosity — ~2.5× the tokens — and it's the slowest of the shortlist. | 264 |
| ornith:9b | 90 | 81% · 60–92 | Marginal | Zero-recovery on everyday work and 3/3 on the needle — its weak-tool-calls reputation is a planner trait, not a worker one. Held back by the rest of the suite, not the catalogue. | 185 |
| qwen3.5:4b | 100 | 62% · 41–79 | Marginal | Fastest and nearly the leanest, but the most recovery-dependent: it emits a tool call then falls silent, and dollama's retry loop lands it. Its 62% first-shot is exactly the gap the native gate exists to catch. | 113 |
| gemma4:e4b | 76 | 62% · 41–79 | Marginal | The noisiest model we grade: 95 → 86 → 86 → 76 across four runs on byte-identical inputs, so its point score means little. Verbose. | 332 |
The worker: qwen3.5:9b
Two models clear Recommended at identical quality — 100 capability, 95% first-shot, CI 77–99 — so the slot is decided on cost-to-serve: qwen3.5:9b runs ~50 t/s to gemma4:12b's ~34 (about 47% faster) and emits ~2.5× fewer tokens per response, at a similar ~8 GB footprint. On a volunteer network, throughput is served requests. One footnote the transcripts earned: the needle scenario originally failed across the board because of a tool-presentation bug in our pipeline — the one relevant tool was buried in a recovery menu — and the numbers above are against the corrected pipeline. Reading failures before blaming models is standing policy here; more than once the failure has been ours.
The helper: qwen3.5:4b
All three finalists clear the helper bar at full tier (qwen3.5:9b 100, qwen3.5:4b 98, gemma4:12b 90), so the helper is chosen for the fleet, not the leaderboard: qwen3.5:4b is already the planner model — one pulled artifact serves two roles — it fits 2 GB-class machines and CPU boxes that could never hold the 9B, and at ~76 t/s it's the fastest of the three. Helper work (curation, summarisation, classification) also tolerates the recovery loop far better than worker-grade code editing does, so its retry-dependence costs little here.
We'll re-evaluate
These picks are a snapshot, not a constitution. The grading deliberately runs ahead of the hardware — provisioning for the network we're hoping to grow, not describing steady-state load — and three things will move the answer. Each one re-opens the §01 gate.
Contributors arrive
Bigger cards open heavier tiers the current fleet can't hold. The role bands in §03 shift with the hardware, and models graded "too heavy" today come back into play.
New models release
The open-model pace hasn't slowed. Any Ollama model can be screened with the same harness, and a new candidate that clears the gate takes the slot — incumbency earns nothing.
The pipeline improves
Presentation and recovery fixes have flipped failing scenarios to passing ones without touching a model. A score is a property of the model and the pipeline — so pipeline changes trigger a re-grade too.
Curious about a model we didn't test?
The harness that produced this page ships in the CLI. Point it at any Ollama model and any role — it'll pull, benchmark, and grade it the same way.
$ dollama eval --model <name> --role workerMeasurement notes
- Quality:
dollama eval --role {planner|worker|helper} --tier quickfor the §02 board — 6 scenarios × 1 repeat, single RTX 3060, no error bars. Shortlists re-run at--tier full(3 repeats, Wilson confidence intervals) for the §04 tables. - Speed / memory: throughput and the VRAM/context curve from
dollama benchmarkon an RTX 3060 (12 GB); VRAM in binary GiB (MB / 1024). Base = weights at 4k; peak = at the model's max context. Your numbers scale with your card. - Usable context is the benchmark's inflection point (where t/s falls off); models survive to a higher context before running out of memory.
- More context isn't always better. A bigger window relaxes the pipeline's budget pressure, so it stops trimming the tool catalog — and small models handle a lean, relevant toolset better than a bloated one. We measured a worker score higher on ordinary edits with a trimmed catalog than with a roomy one; dollama now keeps the catalog lean regardless of window size. One more reason usable, not survivable, is the number that matters.
- † phi4 family: scores reflect text-tool-format adherence, not reasoning — see the note in §02.
- phi4-reasoning:14b is excluded: at ~16 t/s its thinking traces peg the GPU long enough to hard-lock the test box.