# Can a local model drive a coding agent?

> Across 18 runs, both models completed the task that needed no tools and failed every task that needed one. Zero tool calls were executed, by either model, on any run.

Version 1.0. 18 runs. Conducted 2026-08-04 by Syntora.

This measures whether the agent loop completes, not how good the model is at writing code. Those are different questions, and only one of them decides whether the workflow works at all.

## Method

The harness resets the fixture repository before every run, judges success mechanically rather than by eye (t3 passes only if subtract is actually in the file on disk afterwards), and writes one JSON object per run. Analysis is a separate script over that file, so any number here can be recomputed or disputed without rerunning the benchmark.

| Task | Needs | Prompt | Passes when |
| --- | --- | --- | --- |
| t1_text | No tools | Reply with exactly the word READY and nothing else. | The transcript contains READY. |
| t2_read | One tool call | Read the file math.js in this directory and reply with only the name of the exported function. | The transcript names the exported function. |
| t3_edit | Read then write | Edit math.js to add an exported function named subtract that returns a - b. Keep the existing add function. | math.js on disk actually contains subtract afterwards. |

## Results

| Model | Task | n | Median ms | Min ms | Max ms | Succeeded | Tools executed |
| --- | --- | --- | --- | --- | --- | --- | --- |
| qwen2.5-coder:1.5b | t1_text | 3 | 5903 | 5563 | 9529 | 2/3 | 0/3 |
| qwen2.5-coder:1.5b | t2_read | 3 | 6762 | 5702 | 7945 | 0/3 | 0/3 |
| qwen2.5-coder:1.5b | t3_edit | 3 | 6810 | 6256 | 7132 | 0/3 | 0/3 |
| qwen2.5-coder:7b | t1_text | 3 | 9470 | 8641 | 23758 | 3/3 | 0/3 |
| qwen2.5-coder:7b | t2_read | 3 | 8830 | 6760 | 12929 | 0/3 | 0/3 |
| qwen2.5-coder:7b | t3_edit | 3 | 8041 | 8031 | 12118 | 0/3 | 0/3 |

## Is this the model, the server, or Atlas?

**Method.** Both Ollama endpoints were called directly with curl, bypassing Atlas entirely, with a correctly formed OpenAI-style tools array. If the model returns a structured tool call to curl but not to Atlas, the fault is Atlas's.

**Finding.** Both endpoints returned tool_calls: null and put the tool call in the message content as a JSON string. The OpenAI-compatible /v1/chat/completions endpoint and the native /api/chat endpoint behaved identically.

**Conclusion.** The model does not emit structured tool calls in this configuration. Atlas sends a correct request and receives text where a tool call should be. This is not an Atlas defect and not an Ollama defect, and switching endpoints would not fix it.

## Reproduce it

Both commands run against a local Ollama. Neither involves Atlas. Anyone can check the central finding in under a minute.

```sh
curl -s http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "messages": [{"role":"user","content":"Read math.js and name the exported function."}],
    "tools": [{"type":"function","function":{
      "name":"read_file",
      "parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}
    }}]
  }'
```

What comes back:

```text
finish_reason: stop
tool_calls:    null
content:       "{\"name\": \"read_file\", \"arguments\": {\"path\": \"math.js\"}}"
```

## Findings

### Local models are fine for conversation, and not yet for agent work

The 7b model answered every text-only prompt, with a median of 9.5 seconds. That is a usable local assistant. The moment a task required reading or editing a file it failed, every time, on both models. The gap is not quality; it is protocol.

### The tools capability flag does not mean what it looks like

Ollama reports tools as a capability for both models, and Atlas faithfully surfaces that as tools in its model list. In practice neither model produced a structured tool call. The flag describes template support, not demonstrated behavior, so treat it as a hint rather than a guarantee.

### Parameter count did not fix it

Going from 1.5b to 7b improved the text task from 2/3 to 3/3 and changed nothing about tools: still 0/6. This is not a case of needing a slightly bigger model.

### Cold start is worth planning for

The slowest 7b run was 23.8 seconds against a median of 9.5. First contact loads the model; after that it is steady. Judge a local setup on its warm numbers, but expect the first prompt to feel broken if you do not know this.

## Environment

| Component | Value |
| --- | --- |
| Atlas | 1.22.0, installed release binary |
| Ollama | 0.30.11, client and server |
| Models | qwen2.5-coder:1.5b and qwen2.5-coder:7b, Q4_K_M |
| CPU | AMD Ryzen 7 5800X, 8 cores, 16 threads |
| Memory | 32 GB |
| GPU | NVIDIA GeForce RTX 3050, 8 GB VRAM, driver 610.74 |
| Operating system | Windows 11, build 10.0.26200, x64 |
| Model placement | Both models fit in 8 GB VRAM, so inference was GPU-resident |

## Who should care

### If you are choosing a local model for agentic coding

Test tool calling before anything else, with the curl command above. It takes a minute and it is the capability the whole workflow rests on. Benchmarks that report code quality will not tell you this.

### If you are running Atlas against a local model today

Expect a good local assistant and not an autonomous agent. Sessions that read, edit, and run commands need a model that emits structured tool calls. Use a local model for reasoning and a tool-calling model for the agent loop.

### If you build or serve local models

The gap between advertising a tools capability and emitting a structured tool call is where this breaks. Publishing which of your models actually return tool_calls, verified rather than declared, would be genuinely useful.

## Limitations

Eighteen runs on one machine. The categorical result is solid; almost nothing else here is.

- One machine, one operating system, one GPU. Nothing here establishes what happens on Apple silicon, on CPU-only inference, or with more VRAM.
- Two models from one family at one quantization. qwen2.5-coder Q4_K_M is not a proxy for local models in general, and a different family may well behave differently.
- Three trials per cell, eighteen runs total. Enough to establish a categorical result (zero tool calls in eighteen runs) and not enough for a precise latency distribution. The medians are indicative, not authoritative.
- One agent, one prompt per task, non-interactive mode. A different prompt, a system prompt tuned for tool use, or a constrained-decoding grammar might change the outcome. None of those were tried.
- Timings include full Atlas process startup, not just inference. They describe what a user waits, which is the useful number, but they are not model throughput.
- This measures whether an agent loop completes, not code quality. A model that passes here could still write bad code, and a model that fails here might write excellent code when asked directly.

---

Canonical HTML: https://runatlas.sh/local-model-benchmark
Source of truth: src/content/local-model-benchmark.ts (this file is generated from it, never hand-edited).
Licence: Atlas is proprietary with a free core. It is not open source and there is no public source repository.