grok-4.6
99.5%
Channels
The local endpoint records 64.2% of the available signal against 99.5% from the strongest paid channel, a gap of 35.3 points. Each tick is one task, full height a clean pass; the short and missing ticks are where a channel dropped out.
fable
98.4%
qwen3.8-max
98.4%
gemini-3.8-flash
95.2%
glm-5.2
94.1%
deepseek-v4-pro
93.4%
gpt-5.6-luna
93.1%
qwen3.8-27b-local
64.2%
buildfixdiagnosereviewassert
clean passpartialzerohard tier
Core against hard
Harder tiers exist to separate channels the core tier cannot. On the hard tier qwen3.8-27b-local gives up 0.15 against core, 7 channels hold within 0.02, and the field spreads over 0.47 points — the number that says whether the tier ranks anyone.
grok-4.6core1.00hard0.99-0.00
fablecore0.98hard1.00+0.02
qwen3.8-maxcore0.99hard0.97-0.01
gemini-3.8-flashcore0.95hard0.95-0.01
glm-5.2core0.94hard0.95+0.01
deepseek-v4-procore0.91hard0.98+0.07
gpt-5.6-lunacore0.91hard0.99+0.08
qwen3.8-27b-localcore0.69hard0.53-0.15
rightmost column: change from core to hard, so a negative number is a channel losing ground on the hardest tier
By task type
Mean score per channel. Build and fix tasks execute hidden tests; diagnose and review are graded blind against written ground truth.
| channel | build | fix | diagnose | review | assert |
|---|---|---|---|---|---|
| grok-4.6 | 0.99 | 1.00 | 1.00 | 0.98 | 1.00 |
| fable | 1.00 | 1.00 | 0.96 | 0.97 | 1.00 |
| qwen3.8-max | 0.96 | 1.00 | 1.00 | 1.00 | 1.00 |
| gemini-3.8-flash | 1.00 | 1.00 | 0.88 | 0.91 | 1.00 |
| glm-5.2 | 0.99 | 0.75 | 0.92 | 0.98 | 1.00 |
| deepseek-v4-pro | 0.98 | 1.00 | 0.83 | 0.97 | 1.00 |
| gpt-5.6-luna | 0.94 | 1.00 | 0.88 | 1.00 | 1.00 |
| qwen3.8-27b-local | 0.48 | 0.75 | 0.71 | 0.88 | 0.96 |
Where qwen3.8-27b-local loses
21 of 37 tasks scored below a clean pass, 7 of them at zero. A zero is rarely diffuse weakness: it is usually one small defect that costs the whole task.
| task | type | tier | local | field | what happened |
|---|---|---|---|---|---|
| B10 | build | core | 0.00 | 1.00 | 0/9 hidden tests |
| B2 | build | core | 0.00 | 1.00 | 0/12 hidden tests |
| B8 | build | core | 0.00 | 1.00 | 0/16 hidden tests |
| B9 | build | core | 0.00 | 1.00 | 0/7 hidden tests |
| H2 | build | hard | 0.00 | 0.98 | 0/14 hidden tests |
| H6 | build | hard | 0.00 | 1.00 | 0/15 hidden tests |
| F4 | fix | core | 0.00 | 1.00 | 0/6 hidden tests |
| H4 | build | hard | 0.07 | 0.93 | 1/14 hidden tests |
| B6 | build | core | 0.10 | 0.80 | 1/10 hidden tests |
| D1 | diagnose | core | 0.50 | 1.00 | right subsystem, wrong mechanism |
| D3 | diagnose | core | 0.50 | 1.00 | right subsystem, wrong mechanism |
| D4 | diagnose | core | 0.50 | 0.57 | right subsystem, wrong mechanism |
| D7 | diagnose | core | 0.50 | 0.93 | right subsystem, wrong mechanism |
| HD1 | diagnose | hard | 0.50 | 1.00 | right subsystem, wrong mechanism |
| HD2 | diagnose | hard | 0.50 | 0.86 | right subsystem, wrong mechanism |
| HD4 | diagnose | hard | 0.50 | 1.00 | right subsystem, wrong mechanism |
| H3 | build | hard | 0.57 | 1.00 | 8/14 hidden tests |
| HR1 | review | hard | 0.73 | 0.97 | found=['P1', 'P3', 'P5', 'P7'] fp=0 |
| R1 | review | core | 0.92 | 0.99 | found=['P1', 'P2', 'P3', 'P4', 'P5', 'P6'] fp=1 |
| I2 | assert | core | 0.93 | 1.00 | 13/14 hidden tests |
| B1 | build | core | 0.94 | 1.00 | 15/16 hidden tests |
Gap by type
build+0.50
fix+0.21
diagnose+0.21
review+0.09
assert+0.04
Gap by tier
core+0.27
hard+0.44
localfield mean
What the score costs
Quality against spend for the whole run. gpt-5.6-luna reaches 93.1% for $0.036; the local channel is free but sits 35.3 points under the best paid channel.
horizontal: run cost, log scale · vertical: % of available points
What a point costs
Tokens spent per point scored, so a channel that talks four times as much for the same score shows it here. qwen3.8-27b-local spends 977 completion tokens per point; qwen3.8-max spends 16,342. Verbosity is length relative to the field median on the same task (1.0 = typical); answer share is the fraction of a reply that is the deliverable rather than prose.
| channel | raw | cost $ | $ / point | points / $ | tokens / point | verbosity | answer share | reasoning share | s / point |
|---|---|---|---|---|---|---|---|---|---|
| grok-4.6 | 36.83 | 1.061 | 0.0288 | 35 | 4,642 | 1.13 | 0.85 | — | 65.8 |
| fable | 36.42 | 0.000 | 0.0000 | free | 9,154 | 1.76 | 0.66 | 0.88 | 112.3 |
| qwen3.8-max | 36.41 | 3.577 | 0.0982 | 10 | 16,342 | 4.09 | 0.88 | — | 341.9 |
| gemini-3.8-flash | 35.22 | 1.201 | 0.0341 | 29 | 9,074 | 1.65 | 0.65 | 0.93 | 55.5 |
| glm-5.2 | 34.83 | 0.513 | 0.0147 | 68 | 5,787 | 0.92 | 0.90 | — | 50.5 |
| deepseek-v4-pro | 34.55 | 0.396 | 0.0115 | 87 | 9,782 | 2.25 | 0.79 | — | 121.9 |
| gpt-5.6-luna | 34.46 | 0.036 | 0.0010 | 964 | 1,652 | 0.47 | 0.86 | — | 14.7 |
| qwen3.8-27b-local | 23.76 | 0.000 | 0.0000 | free | 977 | 0.21 | 0.71 | — | 27.4 |
Pareto front — no channel scores higher for less: fable, grok-4.6
Completion tokens per task
| task | grok-4.6 | fable | qwen3.8-max | gemini-3.8-flash | glm-5.2 | deepseek-v4-pro | gpt-5.6-luna | qwen-local |
|---|---|---|---|---|---|---|---|---|
| B1 | 4,314 | 16,538 | 23,703 | 21,075 | 4,564 | 7,476 | 2,059 | 782 |
| B10 | 1,506 | 1,617 | 5,000 | 2,448 | 350 | 1,486 | 678 | 415 |
| B2 | 4,202 | 4,074 | 24,485 | 18,317 | 15,979 | 5,962 | 1,916 | 694 |
| B3 | 2,166 | 1,866 | 4,203 | 2,041 | 845 | 1,937 | 758 | 901 |
| B4 | 3,106 | 5,435 | 27,729 | 5,060 | 3,338 | 2,128 | 2,170 | 840 |
| B5 | 567 | 2,174 | 2,178 | 1,145 | 623 | 609 | 395 | 877 |
| B6 | 10,364 | 44,188 | 46,005 | 28,436 | 14,914 | 11,095 | 3,528 | 768 |
| B7 | 21,688 | 51,500 | 50,668 | 31,237 | 17,653 | 6,711 | 3,691 | 1,041 |
| B8 | 4,128 | 6,945 | 13,359 | 24,483 | 9,069 | 1,119 | 2,707 | 1,345 |
| B9 | 2,075 | 3,758 | 11,807 | 3,870 | 1,545 | 2,435 | 1,003 | 1,049 |
| H1 | 4,323 | 7,527 | 18,560 | 5,564 | 8,094 | 36,119 | 2,605 | 954 |
| H2 | 6,657 | 6,893 | 30,678 | 6,964 | 23,480 | 21,861 | 2,694 | 1,751 |
| H3 | 11,608 | 20,066 | 36,248 | 10,161 | 6,218 | 30,304 | 2,869 | 1,342 |
| H4 | 22,842 | 22,075 | 56,103 | 46,940 | 26,770 | 51,689 | 3,922 | 3,919 |
| H5 | 3,423 | 4,821 | 22,335 | 3,719 | 10,408 | 26,895 | 1,093 | 882 |
| H6 | 1,661 | 3,337 | 4,360 | 5,033 | 2,552 | 21,999 | 642 | 259 |
| F1 | 1,154 | 772 | 2,694 | 1,578 | 366 | 1,137 | 288 | 237 |
| F2 | 9,076 | 37,576 | 36,714 | 22,897 | 32,244 | 5,409 | 2,831 | 641 |
| F3 | 314 | 211 | 324 | 659 | 91 | 297 | 132 | 175 |
| F4 | 928 | 3,199 | 12,370 | 4,730 | 886 | 2,378 | 723 | 442 |
| D1 | 2,625 | 1,509 | 12,442 | 1,783 | 1,845 | 12,329 | 1,802 | 258 |
| D2 | 3,754 | 19,618 | 8,717 | 844 | 562 | 14,423 | 1,231 | 138 |
| D3 | 1,497 | 726 | 3,660 | 948 | 438 | 923 | 691 | 133 |
| D4 | 7,011 | 2,792 | 12,616 | 14,754 | 958 | 6,469 | 2,232 | 248 |
| D5 | 4,201 | 9,426 | 9,707 | 1,774 | 964 | 1,489 | 1,733 | 226 |
| D6 | 1,365 | 932 | 3,104 | 788 | 413 | 1,700 | 516 | 189 |
| D7 | 2,961 | 4,841 | 7,613 | 1,567 | 1,346 | 2,472 | 729 | 235 |
| D8 | 3,751 | 4,356 | 5,877 | 1,694 | 3,476 | 6,425 | 1,222 | 219 |
| HD1 | 1,760 | 941 | 11,995 | 1,200 | 392 | 940 | 714 | 185 |
| HD2 | 10,670 | 22,642 | 15,634 | 33,735 | 3,524 | 209 | 1,248 | 235 |
| HD3 | 1,849 | 1,410 | 4,224 | 987 | 775 | 1,830 | 713 | 236 |
| HD4 | 1,296 | 1,277 | 10,716 | 1,085 | 1,102 | 5,487 | 1,250 | 197 |
| R1 | 2,214 | 2,841 | 10,773 | 1,469 | 884 | 10,582 | 1,024 | 412 |
| R2 | 3,005 | 5,682 | 9,700 | 3,952 | 1,570 | 12,078 | 1,917 | 344 |
| HR1 | 4,971 | 8,903 | 36,622 | 4,249 | 1,924 | 19,794 | 2,588 | 233 |
| I1 | 1,192 | 707 | 1,241 | 1,376 | 651 | 1,102 | 346 | 247 |
| I2 | 765 | 233 | 905 | 1,036 | 757 | 672 | 260 | 153 |
Every task
The full record. Task prompts and hidden tests stay unpublished: a task set that leaks stops measuring anything.
| task | type | tier | grok-4.6 | fable | qwen3.8-max | gemini-3.8-flash | glm-5.2 | deepseek-v4-pro | gpt-5.6-luna | qwen-local |
|---|---|---|---|---|---|---|---|---|---|---|
| B1 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.94 |
| B10 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 |
| B2 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 |
| B3 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| B4 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| B5 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| B6 | build | core | 0.90 | 1.00 | 0.70 | 1.00 | 0.90 | 1.00 | 0.10 | 0.10 |
| B7 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.86 | 1.00 | 1.00 |
| B8 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 |
| B9 | build | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 |
| H1 | build | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| H2 | build | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.86 | 1.00 | 0.00 |
| H3 | build | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.57 |
| H4 | build | hard | 1.00 | 1.00 | 0.71 | 1.00 | 1.00 | 0.93 | 0.86 | 0.07 |
| H5 | build | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| H6 | build | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 |
| F1 | fix | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| F2 | fix | core | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 |
| F3 | fix | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| F4 | fix | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 |
| D1 | diagnose | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.50 |
| D2 | diagnose | core | 1.00 | 0.50 | 1.00 | 1.00 | 1.00 | 0.50 | 1.00 | 1.00 |
| D3 | diagnose | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.50 |
| D4 | diagnose | core | 1.00 | 1.00 | 1.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.50 |
| D5 | diagnose | core | 1.00 | 1.00 | 1.00 | 1.00 | 0.50 | 0.50 | 1.00 | 1.00 |
| D6 | diagnose | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| D7 | diagnose | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.50 | 0.50 |
| D8 | diagnose | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| HD1 | diagnose | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.50 |
| HD2 | diagnose | hard | 1.00 | 1.00 | 1.00 | 0.50 | 0.50 | 1.00 | 1.00 | 0.50 |
| HD3 | diagnose | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| HD4 | diagnose | hard | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.50 |
| R1 | review | core | 1.00 | 0.92 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.92 |
| R2 | review | core | 1.00 | 1.00 | 1.00 | 0.80 | 1.00 | 0.91 | 1.00 | 1.00 |
| HR1 | review | hard | 0.93 | 1.00 | 1.00 | 0.92 | 0.93 | 1.00 | 1.00 | 0.73 |
| I1 | assert | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| I2 | assert | core | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.93 |