QDevHow well does each model write jq?
Every provider runs the real NL→jq pipeline over a fixed corpus of 27 questions. Each generated query is scored by running itand comparing the output to a hand-verified answer — so it's graded on the result, not on how the jq is written.
Same pipeline as the extension: schema-only grounding, a bounded self-correcting agent loop, jq in a Web Worker. See how it works.
Correct-answer rate
Share of the 27 cases whose committed query produced the exact expected result. The dashed line marks the first try— before the loop's self-correction.
How accuracy has moved
Pass rate (%) per provider across the last 9 recorded runs. Model updates, prompt tuning, and corpus changes all show up here.
Time to a query
Median generation time per question (p90 in the label). On-device models are wall-clock; QDev Pro shows server-side model compute — the network hop from the benchmark's location is deducted, since edge compute removes it at launch.
Where each model is strong
Pass rate sliced by what the query exercises — filters, aggregation, string handling, recursion. Localises a weakness to a class of question.
| TAG | QDev Pro | Ollama | WebLLM | Chrome built-in |
|---|---|---|---|---|
| aggregation | 88% | 88% | 63% | 38% |
| array | 100% | 100% | 100% | 0% |
| boolean | 100% | 100% | 67% | 0% |
| case-insensitive | 100% | 100% | 0% | 0% |
| count | 100% | 100% | 75% | 50% |
| date | 100% | 100% | 0% | 100% |
| descent | 100% | 100% | 0% | 0% |
| empty | 100% | 100% | 100% | 0% |
| existence | 100% | 100% | 100% | 0% |
| filter | 100% | 100% | 63% | 13% |
| flatten | 100% | 50% | 75% | 25% |
| group | 100% | 100% | 0% | 0% |
| interpolation | 100% | 100% | 0% | 100% |
| keys | 100% | 50% | 0% | 0% |
| large-doc | 100% | 67% | 67% | 100% |
| max | 0% | 0% | 100% | 0% |
| merge | 100% | 0% | 100% | 0% |
| min | 100% | 100% | 100% | 100% |
| negation | 100% | 100% | 100% | 0% |
| nested | 100% | 75% | 50% | 25% |
| numeric | 100% | 100% | 100% | 0% |
| object | 100% | 0% | 0% | 0% |
| project | 100% | 100% | 67% | 67% |
| range | 100% | 100% | 100% | 0% |
| recursion | 100% | 100% | 0% | 0% |
| repair | 100% | 0% | 0% | 100% |
| slice | 100% | 100% | 100% | 0% |
| sort | 100% | 100% | 0% | 0% |
| streaming | 100% | 100% | 100% | 100% |
| string | 100% | 100% | 0% | 67% |
| sum | 100% | 100% | 0% | 0% |
| top-level | 100% | 100% | 0% | 0% |
| unique | 100% | 100% | 0% | 0% |
What it takes to run each one
The trade behind the accuracy: setup, first-run download, and whether anything leaves the device.
| PROVIDER | REQUIRES | FIRST-RUN DOWNLOAD | DATA EGRESS | STATUS |
|---|---|---|---|---|
QDev Pro frontier models | Hosted proxy | none | → schema + question | 96% pass |
WebLLM auto | WebGPU | ~1–2 GB first run | ● nothing | 52% pass |
Chrome built-in (Gemini Nano) gemini-nano | Chrome Prompt API (Gemini Nano) | browser-managed | ● nothing | 26% pass |
Ollama (local) qwen2.5-coder | Ollama @ localhost:11434 | model pull (varies) | ● nothing | 81% pass |
Warm-up (model download + compile), one-time per session: WebLLM 2.0s.
What the numbers say
Bottom line: QDev Pro is the fastest, most accurate path with zero setup — the local models are a free, run-it-yourself alternative that trade some accuracy and speed for it.