Experiment · Coding agents

Three coding agents, 900 trials, eight at a time

24 October 2026 · By Jan Janssens · Read time: 8 min · ← All experiments

Our last test gave two models short coding problems. This one gives three models a small repository, a shell and a bug or a feature, and lets each work like an agent: read the code, edit one file, run the tests, finish. We ran 300 tasks per model with eight running at once. Claude Haiku 5.5 scored highest and was by far the fastest. Our local Qwen3.8-27B and Mistral Large 4 finished level on correctness. Qwen was the slowest of the three and the only one that cost nothing per token.

Before the numbers: we expected the local model to do better when many tasks run in parallel. It sped up a lot compared with running one task at a time, but it did not catch the hosted models. The details are below, along with what this test cannot tell you.

The result

How we set it up

The benchmark is a set of 75 task families, each in four variants, which makes 300 tasks. Each task is a miniature repository in Python or JavaScript with one editable implementation file, protected tests and a protected policy file. The agent gets four tools (list files, read a file, write a file, run a shell command) inside a sandbox with no network. It has 30 turns and 60 tool calls. A task counts as solved only when every public and hidden check passes and no protected file has been touched. A timeout or an infrastructure error counts as a failure.

The three agents were Qwen3.8-27B on our own hardware (vLLM, FP8, multi-token prediction), mistral-large-4 through the Mistral API, and claude-haiku-5-5 through the Anthropic API. Each model ran in its own phase with the same task order and eight workers. We used each provider's default sampling and reasoning settings, so the thinking effort is not identical across models. The harness kept the eight workers busy in every phase: average worker utilisation was 97 to 99 percent.

Speed: eight tasks at a time

Throughput here means tasks that passed every check, per hour of wall-clock time for the phase, including container set-up, model calls, tool use and grading.

Running eight at once helped the local model a great deal. On an earlier, partial run where it worked one task at a time, its median agent time was about 124 seconds. With eight in flight it was about 175 seconds. Each task became roughly 40 percent slower while eight went through together, so the local machine delivered several times the throughput of a single stream. That comparison uses a different subset of tasks, so treat the exact ratio as an estimate.

It did not win, though. All eight Qwen workers share one GPU, which sat at about 98 percent utilisation. The hosted models run on capacity we do not have to share with ourselves. Mistral solved about three times as many tasks per hour as Qwen, and Haiku about twelve times as many. Equal worker counts do not mean equal hardware.

Correctness

The intervals resample whole task families, because the four variants of a family are not independent. Qwen and Mistral are statistically level: on the 300 matched tasks Qwen solved 10 that Mistral missed and Mistral solved 12 that Qwen missed, a difference of 0.7 points with an interval running from minus 3.7 to plus 2.3. Haiku is ahead of both. Its lead over Qwen is 4.7 points (interval 1.3 to 8.7) and over Mistral 4.0 points (interval 0.3 to 8.0). Those are small gaps, but they hold up under the family-level resampling.

How each model failed

Mistral's seven timeouts all happened within about four minutes of each other, which points to a slow patch at the provider and not seven unlucky tasks. Qwen's five were spread across its phase. Neither is a wrong answer, but both count as failures, so they cost those models points. Mistral's two protected-file cases were both edits to policy.json, which the task rules forbid. We have not looked into why.

Where the tasks were hard

Of the 75 families, 23 had at least one failure from some model. Four tasks were failed by all three: three variants of js-dependency_graph and one of js-changelog. Two other JavaScript families, js-field_projection and js-schema_validation, failed three of four variants for both Qwen and Mistral and none for Haiku.

In Python the three models are almost indistinguishable. Nearly all of Haiku's lead comes from JavaScript.

Cost

Qwen cost nothing per token. Mistral cost $15.54 for the 300 trials, plus an amount we cannot see for the seven calls that timed out, because the billing for a timed-out call is unknown. That is about five and a half cents per solved task at the least. Haiku cost $1.05, which is about a third of a cent per solved task. The local figure leaves out electricity and the price of the hardware, so it is zero only in the sense that nobody sends us an invoice for it.

What we would not conclude

• That Haiku is the best model. It is the best agent on this suite. The tasks are small synthetic repositories, not real production code, and it is not SWE-bench.
• That the local model loses on speed in general. Eight workers on one GPU is one setup. A different batching limit or a second card would change the numbers.
• That Qwen and Mistral are equal. They are level on this suite within the noise. The failure types differ, and the suite is only 75 families.
• Anything about long context. The tasks fit comfortably in a small window. The question of what a million-token window buys is still open.
• That thinking effort is matched. We used each provider's defaults, and the models spent different amounts of output on the same task.

What we take from it

For agentic coding of this size, the local 27B model is within noise of a hosted frontier model on correctness, and it keeps the code and the prompts on our own hardware. What it gives up is throughput, and Haiku is clearly the strongest and cheapest of the hosted options. If speed on a large batch matters, the hosted models win. If privacy and zero marginal cost matter, the local one holds up. We will run the long-context and multi-file tests next, since those are where these results are least likely to carry over.

Provenance

Run on 24 October 2026, run id 20261024T132040Z-d90cce: 900 of 900 planned trials, one repeat each, eight workers per phase, a 120 second limit per request and a 32,768-token output cap per task. The first two attempts failed before any trial ran because our local configuration pointed at the wrong port and used a model name the server did not serve; we fixed both and started again. An earlier sequential run was stopped partway through and is used above only for the single-stream timing comparison. Cost figures are estimates from the run's own usage records, not invoices. The numbers on this page come from the saved run data, not from memory.

Short version: Haiku is ahead, Qwen and Mistral are level, the local GPU turns parallel work into several times the throughput but not enough to match the cloud, and four tasks beat all three.