Change log — every version, and why it changed

This document's history, kept separate from the report so the report can stay about the measurement. Each entry states what changed and the numbers that forced the change; a correction is only worth anything if it records what was wrong before it.

Versions before 1.3 are summarised compactly: they predate the build register (§4.1), the cap discipline (§7.2) and the provenance manifest (§20.5), and their numbers should be read with those absences in mind.


Version 1.31 (2026-11-21)

Mellum2.1-12B comes back from retirement: re-measured at 8192, better on code, weaker on knowledge, and no long context at all.

The Q4_K_M file was retired on 2026-11-05 with a 0.9146 code row and a 0.8083 knowledge row. It was restored to disk on 2026-11-20 and run again on fork b11581-1a33095 (halo-box-strix-llama.cpp, commit 19b458b5a, kernel 7.3.4-1, run 20261120-173555-mellum21-12b-fork-cap8192-r2): 0.9451 code (155/164 at cap 8192), 0.7750 knowledge (93/120 at 8192), 12/14 riddles, 3/3 preflight, 114.0 t/s decode at 2519 t/s prefill. Against the row it replaces that is +5 code items, −4 knowledge items, +2 riddles and +2 t/s, so the model moves up four places on code. The table's cells follow the rule this report has always used — best value per axis — so code, riddles and decode come from the re-run and the knowledge cell stays at the 2026-10-18 0.8083 (97/120); both numbers are now in §9.1 rather than one of them silently winning. Both needle probes fail on the re-run — 2k and 8k truncated at 1,024 tokens, retried at the same budget, still empty — and §9.1 says so rather than leaving the axis implied.

The re-measure is the only measurement in this edition. Everything else here is a count, an ordering or a caption that the roster change and a re-read moved.

what changedbeforenow
Table 4, mellum21-fork row0.9146 / 0.8083 / 10-14 / 2723 / 111.9, 2026-10-18, fork pre-identity, ✗ 11-050.9451 / 0.8083 / 12-14 / 2723 / 114.0, 2026-11-20, fork b11581-1a33095, ✓
Table 4 ordersix rows out of sequence (0.9024, 0.8598, 0.8476, 0.8354, 0.8000, 0.7866 inside lower blocks)sorted by HumanEval, no-score rows at the foot; 36 rows changed position, no value changed
§9.1 mellum block111.9 t/s, "0.91 code", cap caveat only114.0 t/s, 0.9451 code, both knowledge runs (0.8083 carried, 0.7750 re-measured), the failed needles named
§11.3 decode leaderboard111.9 t/s, ✗ 11-05114.0 t/s, ✓
§5.1 and §5.2 quant pairsQ4_K_M 111.9 t/s (+22%), 0.9146 / 0.8083114.0 t/s (+24%), 0.9451 / 0.8083 (+6.7 / +10.8 pp)
§10.2 family spread, mellum0.8780-0.9146 Δ.037, 0.7000-0.8083 Δ.108, 10-11/14, 92-112 t/s0.8780-0.9451 Δ.067, 0.7000-0.8083 Δ.108, 11-12/14, 92-114 t/s
§19.2 knowledge against speed"the best knowledge-per-second point on this machine is mellum21, still 5.8pp under the ceiling"claim withdrawn, the two comparison sentences that hold now stated — see below
Table 8 registry✓ sha256 · retired 2026-11-05✓ sha256 · restored 2026-11-20
Table 5 footnotemellum21-fork listed among the rows retired since the snapshotit is named as retired and then restored, with its snapshot cells left as the 2026-10-18 run
header roster51 files, 898.1 GiB, 31 files with a row, 32 rows54 files, 948.2 GiB, 32 files with a row, 32 rows
§0.3 and §18.3 servable count3233
§9 alias denominator52 of 9652 of 98
Figure 15 / 16 / 17 / 21 captions58 models / 56 models / ten cap pairs / "of 44"58 / 56 / eleven / "of 45", read off the regenerated artwork

One claim was false before this edition touched it. §19.2 said "the best knowledge-per-second point on this machine is mellum21, still 5.8pp under the ceiling". Computed as MMLU ÷ decode the leader is the retired 31B (0.8667 ÷ 17.0 = 0.0510 per t/s), and among files decoding above 50 t/s it is the live driver (0.8500 ÷ 56.9 = 0.0149); mellum21's 0.8083 ÷ 111.9 = 0.0072 never led that ratio at either cap. The sentence now states the two comparisons that hold — the two fastest files read 0.7750 and 0.6917, 9.2 and 17.5 pp under the ceiling, and no file above 50 t/s beats the driver's 0.8500.

Two figure captions disagreed with their own artwork, the failure mode the v1.29 entry already recorded: regenerating the figures moved the n = footers and the prose above them did not move with it. Figure 17 now says eleven cap pairs (the artwork drew eleven) and Figure 21 says "of 45" where the denominator had moved. And one figure defect was caught by the captions moving under it: charts.py's alias table had no entry for the re-measure's mellum21-12b, so the same weight file plotted as two models — which inflated Figures 15 and 16 by one point each (58 → 59, 56 → 57) and would have shipped a figure in which mellum appeared twice, once at each of its two runs. The alias is now folded into mellum21-fork in canonical(), the figures plot one point per file again, and the captions read 58 and 56 — the same numbers they carried before, now for the right reason.

What was deliberately left alone. Table 5's 2026-10-31 snapshot keeps every cell it published, including mellum21-fork's 0.9146/0.8083 — a snapshot that silently updates is not a snapshot — and the footnote is what carries the restore. §18.3's run count and hour total are unchanged: this edition adds one completed run and the line was not re-derived from a new basis. The 16 GiB page stays at v1.37, because nothing in its class matrix moved: no matrix row was added, the 21 rows still resolve to 18 weight files on disk including three that carry two rows each, and charts16.py regenerates its eight figures with no diff against the published set. An Inkling2-mini triage arm was still measuring while this edition was built, so it is excluded from the matrix until its HumanEval axis closes.

Version 1.30 (2026-11-20)

Hy4-flash leaves the disk the same day it was measured: the row stays, the 79 GiB does not.

The IQ2 arm added in v1.29 answered its question — no slot — so its weights were deleted rather than kept as a 79 GiB exhibit. Both shards and the vision projector were hashed against AtomicChat/Hy4-flash-GGUF's published LFS oids and all three matched, so the deletion is reversible to the byte from the retirement ledger. Table 4 keeps the row and marks the disk cell retired; every score, caption and figure on the page is byte-identical to v1.29, because nothing was re-measured.

what changedbeforenow
Table 4, Hy4-flash row, disk cell✓ PR-tree✗ 11-20
§11.4 riddle table, same row✓ PR-tree✗ 11-20
header roster53 .gguf files, 977.0 GiB, 32 files / 33 rows carrying a measurement51 files, 898.1 GiB, 31 files / 32 rows — the two shards (78.88 GiB) and the projector (0.91 GiB) are the removal
§0.3 and §18.3 servable count3332
§9.1 per-model blockverdict onlyverdict plus the deletion, its hashes and where the restore recipe lives
§21 change lognine entries in brieften

What was deliberately left alone. The 16 GiB page stays at v1.26 (Hy4 was never class-legal at a 15 GiB envelope), Table 5's servable snapshot stays dated 2026-10-31, and no figure was regenerated — charts.py plots the archive, and the archive did not change, so the n = in every scatter footer still reads what v1.29 published. The launcher run-hy4-flash.sh and the hy4flash-pr build tree stay in place: PR #32517 is still open, and the tree is what would serve a re-download.

Freed: 79.79 GiB (78.88 of weights, 0.91 of projector). Disk went from 445 GiB free to 524 GiB.

Version 1.29 (2026-11-20)

A 218.9B MoE measured on an unmerged build tree: the row is real, the slot is not — and four figure captions had drifted from their artwork.

tencent/Hy4-flash (218,938,353,741 params, 42 blocks, 512 experts top-16 plus one shared, MLA attention, 1M training context) declares general.architecture = hy_v4_flash. Neither the serving fork nor the Bolt tree registers that string — both know only hy_v4, the earlier base model — so every GGUF of it dies at load with unknown model architecture. The measurement therefore came from upstream PR #32517 (open, mergeable, head Little0o0:hy4-flash f8d4cc4), built into its own tree so the unit-serving fork was untouched. Vision is dead in that tree (the projector fails load_hparams: unknown projector type and the server exits), so the arm is text-only, which is the row's build cell.

what changedbeforenow
Table 4—new row: Hy4-flash (IQ2_S/IQ2_XXS), 218.9B, IQ2_S, 0.9207 (164@512), 0.6167 (120@512), 14/14, 276 / 18.9 t/s, 2026-11-19, hy4flash-pr b1-f8d4cc4, ✓ PR-tree
header roster49 files, 896.9 GiB, 31 files / 32 rows, checked 2026-11-1753 files, 977.0 GiB, 32 files / 33 rows, checked 2026-11-20 — the two DFlash draft files (1.24 GiB) and this arm's two shards (78.88 GiB) are the additions
header measurement window2026-10-07 → 2026-11-172026-10-07 → 2026-11-19
§9 alias denominators49 of 8752 of 96, restated from the archive: 96 distinct aliases carry at least one measurement, 52 a Table 4 row
§0.3 and §18.3 servable count3033 — Table 4's ✓ column already read 32 before this edition, so the prose had drifted by two
§18.3 cost line207 runs, 76.9 h, window to 2026-11-12226 runs, 95.1 h over the 200 rows that record a wall time, window to 2026-11-19
Figure 15 / 16 captions"40 measured rows" / "38 measured rows"58 / 56 measured models — the figures' own footers read 57 and 55 at the last publish, so the captions had been wrong since the 2026-11-17 refresh
Figure 17 captionnine models measured twiceten cap pairs at n ≥ 100
Figure 21 caption"of 20""of 44" models with all six axes measured

The numbers. Caps are 512 on all three capability axes, which is what a 2.6 bpw file leaves at 78.88 GiB of weights and 32k context: 0.9207 code (151/164), 0.6167 knowledge (74/120), 0.7667 / 0.8444 IFEval (30), 14/14 riddles, 3/3 preflight, both needles, 276 t/s prefill and 18.9 t/s decode. It is sixteenth on code with a tie at 0.9207, level with the 30B Q3 build on knowledge, 0.23 under the driver — and the second-slowest decoder in Table 4, at 3.5x Ornith-35B's decode cost on the same suite (5,984 s of wall against 683 s).

Why it keeps no slot. Not the architecture: the trim. The only less-damaged builds of this file are Q3_K at 144.7 GiB and Q4_K_M at 156.8 GiB, neither of which this box can hold, so "test it at a better quantisation" is not an available experiment at any effort. That is why the row is published and the file is not promoted to a unit.

Deliberately not re-derived. Table 5's servable snapshot is dated 2026-10-31 and is left as a snapshot; §10.2's family spreads gain nothing, because Hy4 is the only member of its family in the archive; Figure 23's power trace was not re-picked. The 16 GiB page (~/docs/16gb-vram.md, v1.26) is unchanged: a 79 GiB file is not class-legal at a 15 GiB envelope, so no table or figure on that page moves.


Version 1.28 (2026-11-18)

The row now describes what the box serves, and the archive's "grafted heads never pay" rule gets its one exception.

Ornith-1.5-35B (:8086) was re-measured on 2026-11-18 with its own in-weights MTP head engaged (--spec-type draft-mtp --spec-draft-n-max 1), which is how the unit has served since that date. Table 4 and Table 5 carry the new cells; the previous ones were the spec-off baseline of 2026-11-01.

what changedbeforenow
Ornith-1.5-35B HumanEval0.8354 (137/164@4096)0.9024 (148/164@4096) — +14/-3 flips, sign test p=0.006, no cap-pinned row in either arm
Ornith-1.5-35B MMLU-Pro0.4417 (53/120@4096)0.4250 (51/120@4096) — +13/-15 flips, p=0.85, noise
Ornith-1.5-35B riddles13/1414/14
Ornith-1.5-35B prefill / decode1633 / 74.1 t/s1507 / 80.5 t/s — spec-on trades prefill for decode
Ornith-1.5-35B build and datefork b3-1a33095, 2026-11-01fork b11581-19b458b, 2026-11-18

"Every grafted draft head loses" was false and is withdrawn. §9.1 had said the baked MTP head's +28.7% was "the exact opposite of every grafted draft head tried here". An external DFlash2 drafter (0.53 GiB) on ornith-122b at n-max 1 measured +25% decode (23.34 → 29.21 t/s) with output-neutral quality: HumanEval 0.9512 → 0.9634 (156 → 158, and both losses are cap-pinned rows), MMLU-Pro 0.7917 → 0.8167 (95 → 98), riddles 12/14 → 14/14, sign test p=0.33 over 17 flips. The ornith-122b row keeps its 2026-10-20 spec-off figures — the drafter is an on-demand configuration, not how that unit always runs — and the arm's numbers are quoted in the §9.1 text instead. Two further DFlash2 heads grafted onto the 35B (one fetched specifically for this test, 747 MiB) lost 9.6% and 13.8% at 87.6% and 73.8% acceptance, which is where the withdrawn sentence came from.

Downstream edits, because a row change falsifies sentences that never mention it: §8's noise example now cites the 122B arm's 156 → 158 and the 35B's 137 → 148, in place of the 35B's old 2-item move; §9.1's 35B-trough paragraph now says three of the four land in 0.82-0.87 code and the fourth left the band; the Axion comparison quotes 80.5 t/s / 0.4250; Table 6's ornith/qwen3.5 row moves to 0.4250-0.7917 (Δ.367) for knowledge, 11-14/14 riddles and 23-80 t/s decode; §16.4's ablation bill is restated as 33.5 points against 0.9024, with an explicit caveat that the two sides are now measured under different serving configs.

Version 1.27 (2026-11-17)

Two rows, and the first controlled uncensoring test this box can run.

lfm25-24b-a2b-htc and its thinking arm join Table 4. Same base checkpoint as lfm25-24b-a2b, same QAD-Q4_0 recipe, byte-identical 12.65 GiB file, one build: only the refusal prior was ablated, so for the first time the difference between the two rows is abliteration and nothing else.

what changedbeforenow
measured rows30 on disk (header)32 rows over 31 files, recounted 2026-11-17
roster size807.6 GiB (checked 2026-11-12)896.9 GiB, 49 .gguf files, mmproj excluded
§9.1 per-model analysisone LFM2.5 entrythe abliterated twin has its own paragraph
the three axes—0.6890 / 0.2667 / 12-14 with the think block closed; 0.8902 / 0.6167 / 14-14 with it open

The result is not the folklore. Against the stock arms at matched caps the twin loses 9 items of MMLU-Pro on 120 and gains one IFEval prompt on 50 — knowledge cost, compliance neutral — while the abliteration is supposed to cost both and buy neither. What it actually buys is measured by the benign refusal probe rather than by these three axes: on 200 prompts the stock file refuses 50 outright and hedges 13, the twin refuses none, 25.0% → 0.0% with 63 per-item verdict changes. The cost sits in the analysis channel, not in what the weights know.

The 16 GiB page carries the same pair as its v1.24, where it also takes the IFEval lead from the stock arms.

Version 1.26 (2026-11-12)

Read-through of the whole document: no measurement changed, nine statements about them did.

A full pass over the published text against its own Table 4, the archive and the other page on this site. Every item below was a contradiction with something already in the document, not a new measurement.

what was wrongwhat it saidwhat is true
models tested48 (§0.0)49 rows in Table 4
measured rows on disk29 (header)30 (§0.3 and the table agree)
Figure 15 / 16 captions"30 models"40 and 38 rows carry both axes
Figure 17 captionsix models at two capsnine
retired files plotted (§19)tennineteen
switcher / units (appendix)2430
MMLU-Pro"four-option"ten-option
IFEval"30 prompts"50 since 2026-11-09

The superlatives were the substantive error. "It is the best HumanEval and the best MMLU-Pro on disk" (§9.1) and "first on knowledge among everything on disk" (§0.0, §7.2, §11) were asserted in four places and were wrong in two ways at once: lfm25-24b-a2b (thinking) is on disk and scores 0.9695 on code, and qwen38fn-abl's 0.8750 knowledge is a live row — measured on 40 items rather than 120, which the table records and the prose ignored. All four now read "the best knowledge score among the full 120-item rows still on disk", and the 40-item row is named wherever the claim is made. The same pass replaced two recommendations in §0.0 that pointed at deleted files: "a 12B model at 111 tokens/second" was mellum21-fork (retired 2026-11-05) and "a million-token document: Kimi-Linear" was Kimi-Linear-48B-A3B-abl (retired the same day). Long context is now stated as what it is — 64k, held by three live rows — rather than as a capability nothing on disk has.

Cross-references. §0.2 sent a reader to §6.2 for the MMLU-Pro protocol; §6.2 is the availability guard and the protocol is §7.2. The pointer is fixed. Every other §N reference resolves correctly, which is worth recording because the document has over a hundred of them.

Tables. The lettered Table 4b is now Table 5 and everything after it shifts by one, so the table of contents line "Tables 1-13" is true instead of counting a lettered table as a thirteenth. The snapshot table keeps its purpose — it is Table 4 restricted to what was servable on 2026-10-31, with an endpoint column a ranking cannot carry — and its closing note no longer claims a 37-file inventory that lives in §20.3, which is a file map.

Figures. None of the 24 figures was referenced from the prose: each existed only as a caption. All 24 now have a lead-in sentence naming them. Figures 1-14 also carried their captions above the artwork while 15-24 carried theirs below; every caption now sits under its figure. §19 states the chart encoding once — bubble area is parameter count, hollow markers are floor estimates — instead of leaving it under §19.1 for ten figures to share.

Coverage. §4.1's build register gains the row it was missing: fork HEAD 19b458b5a, server string b11581-1a33095, the build that produced the clover row, which §9's preamble had promised was tabulated there. §9.1 gains a block covering the eleven 16 GiB-roster rows, which had no per-model analysis at all, and its clover paragraph now covers the IQ2_M fork row rather than only the retired Q4_K_M experiment.

Appendix. Three port-map defects: 8094 named twice under two different names, 8182 and 8185 each listing a live and a retired model with no marker, and the 8090 line that the privacy pass had turned into "the hardware monitor (hardware monitor, private network only)". §20.1's example commands all targeted :8099, which the same appendix calls script-only with its unit disabled; they now use :8082 and say so. The malformed ~/.hermes/.env && set -a && . ~/.hermes/.env line is fixed.

Also: riddles were described as "five classic riddles and arithmetic traps" beside a row that says 14 probes — five riddles and nine arithmetic traps; §0.0's "2 bits lose the facts" claim now names the 3.05 bpw counterexample that keeps 0.8417; and §21 now says where the full history lives.


Version 1.25 (2026-11-12)

Clover-35B-A3B-Preview measured in AR mode: the last gap in the 16 GiB roster closes, and it takes that roster's knowledge lead.

modelHEMMLU-ProIFEval prompt/instdecodeprefillsizeverdict
Clover-35B-A3B-Preview IQ2_M0.9451 (155/164 @8192)0.8417 (101/120 @8192)0.8400 / 0.8289 (50@4096)67.5 t/s1849 t/s12.32 GiBkept

A block-diffusion MoE — 40 blocks, 256 experts, 10 full-attention and 30 linear-attention layers, 34.7B parameters — served by the fork at tree HEAD 19b458b5a ("models: add Clover (clover_moe) support, ported from upstream #31306"), whose version string reads b11581-1a3309516: build 11581 on the 1a3309516 base, which is why the build column in Table 4 names both. Flash attention is on and the KV cache is q8_0 on the ten full-attention layers, so the row is comparable with the rest of the matrix at the level that matters — it is not a second flag set measured on a second build.

Measured AR-only, and that is the caveat the row carries everywhere it appears. llama.cpp runs this architecture autoregressively: there is no parallel block decode, so this is the causal-decode arm of weights trained for something else. It says what these weights do in this harness. It does not say what the model does when decoded the way it was trained.

What the axes say. 0.9451 code (155/164) is second on the 16 GiB roster behind lfm25-24b-a2b (thinking) at 0.970, and inside the same ±3.8-point interval as the rows at 0.933 and 0.939 — a top cluster, not a win. 0.8417 knowledge (101/120) is a different matter: it leads that roster by 10 items over axion-1.0-35b-iq3 (0.758), and across this whole matrix it is the best 120-item knowledge score of any live row except the driver (gemma4-26b-uncen, 0.8500). At 12.32 GiB that inverts the page's quantisation argument again: a 3.05 bpw diffusion export keeps knowledge that 4.5 bpw dense rows do not.

The cap is worth 28 points on this model, measured twice. IFEval read 0.560 prompt-strict / 0.5526 instruction-strict at the task's own 1280-token pin and 0.8400 / 0.8289 at 4096 — the same class of artefact groot-19b-a3b (0.220 → 0.740) and axion-1.0-35b-iq3 (0.300 → 0.780) produced, and the reason those arms carry 4096-8192 caps in Table 4.

Against the Q4_K_M row this replaces, and what is not claimed. The archive already had a Clover row: 0.8598 code / 0.4000 knowledge, Q4_K_M at cap 4096, measured 2026-11-03 on the upstream build that trades flash attention for the architecture, so its KV cache was f16. The two rows differ in quantisation, build, KV cache and cap at once, so this edition does not attribute the 44-point knowledge difference to any one of them. What it will say is that cap alone moves this model by 28 points on IFEval, which is enough to make the old 0.4000 unquotable as a property of the weights.

Roster and counts. 49 measured rows, 30 servable, 207 runs, 76.9 h, 87 aliases. The 16 GiB page gains the fourteenth row, its §3 knowledge and overall verdicts move, and its §13 gap bullet for this model is replaced by the AR-only caveat above.


Version 1.24 (2026-11-12)

The 16 GiB-card roster joins the matrix — eleven rows, same box and build, different cap.

Between 2026-11-09 and 2026-11-12 the box was re-used to answer a second question: what fits and what scores on a 16 GiB graphics card, with the fit arithmetic done against a 16 GiB budget instead of this host's 100 GiB pool. That work produced the 16 GiB page; this edition puts its models into Table 4 so the main matrix is one record rather than two. Same box, same fork build (b3-1a33095), same harness and the same per-item records — but a different generation cap, because on a card the cap is what the KV budget leaves you.

modelHEMMLU-Proriddlesdecodeprefillsizeverdict
lfm25-24b-a2b (thinking)0.9695 (159/164 @8192)0.6917 (83/120 @4096)14/14111.2 t/s3360 t/s12.65 GiBkept
Groot-19B-A3B0.9390 (154/164 @4096)0.7167 (86/120 @4096)13/1464.8 t/s2176 t/s11.63 GiBkept
Axion-1.0-35B IQ3_XXS0.9329 (153/164 @8192)0.7583 (91/120 @4096)14/1493.8 t/s1543 t/s12.69 GiBkept
gemma-4-12B-QAT0.8963 (147/164 @512)0.5750 (69/120 @512)14/1427.8 t/s911 t/s6.26 GiBkept
gemma-4-E4B-QAT0.8659 (142/164 @512)0.4667 (56/120 @512)14/1457.8 t/s2243 t/s3.93 GiBkept
qwen3-14b0.8537 (140/164 @512)0.6083 (73/120 @512)14/1423.5 t/s743 t/s8.38 GiBkept
qwen3-30b-a3b0.8537 (140/164 @512)0.6167 (74/120 @512)14/1479.7 t/s1910 t/s13.70 GiBkept
mistral-small-3.1-24b0.8232 (135/164 @512)0.6083 (73/120 @512)12/1414.9 t/s460 t/s13.35 GiBkept
qwen3-8b0.7866 (129/164 @512)0.5833 (70/120 @512)13/1439.0 t/s1300 t/s4.68 GiBkept
gpt-oss-20b-mxfp40.5854 (96/164 @512)0.5583 (67/120 @512)14/1474.4 t/s2169 t/s11.28 GiBkept
gemma-4-26B-A4B-QAT + MTP drafter——14/1454.9 t/s1886 t/s15.06 GiB at 8kperf arm

The cap is the finding, not a footnote. Seven of the eleven are non-thinking arms and were measured at a 512-token generation cap on both capability axes — the budget a 16 GiB card leaves after weights and KV — so their cells describe what the card delivers, not the model's ceiling. The three that deliberate were re-run at 4096-8192 for that reason: lfm25-24b-a2b reads MMLU 0.1917 at 512 and 0.6917 at 4096 on the same weights, which is a 50-point swing produced entirely by the cap, and it is the clearest single illustration on this box of why cap discipline outranks model choice in a benchmark comparison. gemma-4-26B-A4B-QAT and base lfm25-24b-a2b already had rows here at 2048 and 4096, so those files appear once, at their better run; their 512-cap numbers stay on the 16 GiB page as the cap evidence.

What moved as a result. lfm25-24b-a2b (thinking) takes the top HumanEval cell in Table 4 at 0.9695 (159/164), ahead of gemma4-31b at 0.9634 (retired 2026-10-30) — so the best-coding row on the board is now a 12.65 GiB file that is also the second-fastest decoder measured here. §11.3 gains it at 111.2 t/s (second only to mellum21-fork at 111.9) and §11.4 grows from 8 rows with all fourteen riddles to 16. The counts are recounted from the archive rather than inherited: 206 runs, 75.2 h, 87 aliases, 48 matrix rows, 29 rows still servable on disk — §0.3, §9's preamble and §18.3 now say those numbers, and the roster header cites files on disk (49 .gguf, 807.6 GiB, checked 2026-11-12) instead of a family count that three different readings of the directory could give three different answers for.

Not in this edition: Clover-35B-A3B-Preview IQ2_M, the twelfth row of that roster, was mid-measurement when this was published — HumanEval 0.9451 (155/164 @8192) was in, MMLU-Pro and IFEval were still running. It gets its own paragraph and its own row when the axes land, and it will be measured under the same AR-only caveat as the Q4_K_M row it replaces.


Version 1.23 (2026-11-09)

One arrival at the far end of the quantisation curve — and a publication gap in the edition before it.

modelHEMMLU-ProIFEval prompt/instdecodeprefillsizeverdict
Lite-Beam M2.7 229B-A10B IQ2_XXS0.8000 (48/60 @4096)0.6750 (27/40 @4096)0.2000 / 0.155634.1 t/s261 t/s61.36 GiB, two shardskept

MiniMax-M2.7 shape, 228.69B total with ~A10B active, exported as a mixed IQ2_XXS at 2.0625 bpw across two shards totalling 61.36 GiB, 204,800-token training context served at 16,384. It is the largest model by parameter count this box has measured — Groot-65B's 79.07 GiB remains the largest file — and it does not fit all-resident beside the driver: it runs --load-mode mmap --lazy-mode auto on port 8182 (run-litebeam-m27.sh, no systemd unit), KV cache q8_0, inside the same 100 GiB GTT pool the driver and the boot unit share.

The speed is what two bits buy. 261 t/s prefill and 34.1 t/s decode, TTFT 7.0 s on a 1,826-token prompt. Capability was taken at a 4096 cap on both axes: 0.8000 code (48/60) and 0.6750 knowledge (27/40), riddles 10/14, preflight 3/3, and both needles — 2k and 8k — retrieved first time, no retry. IFEval is the weak axis and it is not close: 0.2000 prompt-strict / 0.1556 instruction-strict over 30 items, the lowest instruction-following score of any row that passes preflight. That is a property of the export, not the harness: the same prompts at 4096 on the Q4 files of the same family score 0.7-0.9.

It wins nothing except parameter count, and the table should say so. The 176.9B qwen38fn-abl at IQ3_S takes both axes (0.8476 code, 0.8750 knowledge) at 52.9 t/s in 78.78 GiB: 52B fewer parameters, 17 GiB more disk, and better on code, knowledge and speed. Lite-Beam is kept for what it uniquely is — a 228B model inside 62 GiB, with a 200k training context and both needles — not because it beats the resident abl. Wall clock for the whole suite was 6,985 s (1.94 h), HumanEval 3,196 s of it and MMLU-Pro 2,047 s.

The process failure this edition exists to correct. The run was measured and catalogued on 2026-11-08 (last axis written 02:51) and the publish pipeline ran at 04:15 the same morning. Because the report body is hand-written and the pipeline only regenerates what the documents already say, that commit (ef2b137) published the catalogue row and three charts redrawn from it — f15-quality-speed, f16-knowledge-speed, f21-profiles — with no prose, no leaderboard row and no change-log entry behind it. The site showed a point on two figures that no sentence introduced. The row is written here, the leaderboards follow from the matrix, and the counts in §9, §18.3 and §20 that had drifted by one run were recounted from the catalogue: 37 matrix rows, 155 runs, 60.6 hours, 75 aliases, 16 servable model families. The rule this points at is the one already in §4.1: the pipeline cannot publish prose it was not given, so a measured row is not a published row until the sentence exists.

Two rendering defects fixed in passing. In Table 4b the closing note sat between the sixth and seventh rows, so the last three rows (qwen38fn-abl, Ornith-1.5-35B, ornith35b) had been rendering as literal pipe text outside the table since v1.20 — visible on every page view, unnoticed in every diff. The note now follows the table.

The second was the renderer itself. md2html.py emitted one <li> per source line and closed the list at the first wrapped line, so every bullet longer than one line was published as a one-line list item followed by a loose paragraph: §4.1's five bullets, §21's version list and half the retirement notes came out that way, with stray ** in the HTML wherever bold crossed a wrap (393 <strong> spans before the fix, 403 after). The renderer now absorbs indented continuation lines into the item they continue, and it is published in tools/ so the pages can be rebuilt from the markdown this repository holds.

This edition was published in three commits. 285c0a7 carried the new row, the recounted totals and the Table 4b repair; the second added §4.3 and the renderer fix above; the third rewrote §13. Between the first and the second the page was live for about an hour with the row but without the rebuild register — the same class of half-published state this edition opens by correcting, caught the same day and in the same way: by reading the rebuilt page rather than the diff.

§13 rewritten, and one figure in it corrected. The comparison against Nvidia and Apple is now organised around the question that decides the purchase — what caps the GPU's working set — rather than around a feature matrix: VRAM (hardware, 24-32 GB), Apple's wired limit (a driver default at ~75% of RAM, 96 GB of 128, raisable with sysctl iogpu.wired_limit_mb), and GTT here (a kernel parameter, 100 GiB, with an 8 GiB carve-out unused). It adds the disk-to-GPU path for each (a PCIe copy on Nvidia, none on either UMA machine), the bandwidth arithmetic per token, the software gap on its own terms (CUDA as a platform; Metal/MLX as a narrower one; Vulkan/llama.cpp and partial ROCm here), and a failure-modes subsection, because the three machines fail on different inputs.

The correction: v1.22's §13.2 read "this box reads 13.4 GiB per token for its driver at 59.5 t/s". 13.4 GiB is the file size. Read per token — 4B active at 4.5 bpw, so ~2.25 GB, rising to 2.5-3 GiB with attention and KV reads — the implied effective bandwidth is 134-190 GB/s, inside the 120-280 GB/s band §2.2 reports, not the 797 GB/s the old sentence implied. The decode comparison it supported (3-4× a 4090 on models that fit) survives, because it rests on the bandwidth ratio, not on that figure.

Every number about the other two platforms is a vendor specification with the source linked, and none of them was benchmarked for this report — stated in the section rather than left for a reader to discover.

Build identity. Every number here is fork b3-1a33095, kernel 7.3.1-1-cachyos, Mesa 26.2.5 / vulkan-radeon 26.2.5, ggml-vulkan ABI 0.25.3. The four trees rebuilt on the evening of 2026-11-08 — upstream-llama, flint and sensenova at upstream 7d7f028 (0.9.0-dev, ABI 0.29.0), prism at 6447906bf (0.7.0-dev, ABI 0.27.0) — serve nothing in this report and appear in no row. They are now tabulated in §4.3, because a reader building one of them gets a binary four ggml ABI minor versions away from the one the numbers came from; the build register remains a single-fork register until a benchmark is run, with paired reps, on one of them.


Version 1.22 (2026-11-06)

One arrival, and it is the fastest file on the box — with a +16.7 pp cap correction behind its score.

modelHEMMLU-ProIFEval prompt/instdecodeprefillsizeverdict
Laguna-XS-2.1 Q4_K_M0.8667 (52/60 @4096)0.7000 (28/40 @8192)0.7667 / 0.844487.7 t/s1321 t/s18.88 GiBkept

poolside's Laguna XS 2.1 is a 33B-total / 3B-active MoE under arch laguna — 256 experts with 1 shared, top-8, 40 layers (10 global attention, 30 sliding-window at 512 tokens), 262k context, OpenMDW-1.1 (permissive, commercial use allowed), interleaved thinking with a per-request toggle. The served fork already registers the arch, so it needed no rebuild: 18.88 GiB at Q4_K_M loads in about ten seconds and holds GTT 48.7 GiB while the driver and the boot model keep serving — nothing has to be evicted to run it.

It is the fastest thing measured here. 87.7 t/s decode and 1321 t/s prefill beat every full-suite row in the archive end to end, including Axion-1.0-35B's 76.2 t/s for a slightly smaller file.

The scores needed the right budget, and this is the third time that has been true. At cap 2048 the suite read 0.7000 code and 0.6000 knowledge — with 35 of 308 generations pinned at the cap. Code then went to 0.8667 (52/60) at 4096 (ten items gained; six of the eight remaining misses were still cap-pinned, two actually wrong) and knowledge went 0.6000 → 0.6750 (4096) → 0.7000 (8192): +3 items then +1, with the 8192 misses splitting six pinned and six wrong, so the curve has flattened and ~0.70 is where this model's knowledge genuinely sits. Caps travel with every figure here; the arms are standalone axis runs rather than t20 and are named in full — ~/bench/runs/humaneval-20261106-192625.jsonl, mmlupro-20261106-194159.jsonl, mmlupro-20261106-200654.jsonl. At 4096 its code sits between Axion's and Swift's, its knowledge below Axion's, and its IFEval (0.77 prompt-strict / 0.84 instruction-strict) is the best of any non-driver row except Swift's. Riddles 10/14.

BF16 of the same weights is a trap on this platform. Neither serving tree compiles coopmat2 shaders, and the Vulkan backend gates fast BF16 matmul on coopmat_bf16_support / coopmat2_bf16_support, so the identical file in BF16 runs 11.6 t/s — 7.5× slower than the K-quant. It was measured, then deleted along with a third-party DFlash draft head that refuses to load (done_getting_tensors: wrong number of tensors; expected 64, got 58). Both deletions carry their sha256s and restore recipes in the ledger; the Q4 file's hash is 1ac70791…9287f6cb.

One operational note worth carrying forward. llama-model-guard.timer restarted the channel driver two minutes into the first Laguna window, because a transient bench-* unit is not a llama-* unit and the guard's "another unit is active, defer" rule never applied. The fix is a hold file: touch ~/llama-local/.guard-hold before stopping the driver for a benchmark, remove it after.


Version 1.21 (2026-11-05)

Roster pass: five files retired, 177.7 GiB freed, nothing lost that was not already on the record.

retiredsizemeasured (kept in Table 4)why
Mellum2.1-12B-A2.5B-Thinking Q4_K_M7.52 GiBHE 0.9146 (164@8192) · MMLU 0.8083 · tg 111.9orphan — no script, unit or switch case survived
LFM2.5 DSpark draft head (F16)0.62 GiBnever scoredits only consumer was a one-off A/B script
Groot-65B-A6B Q4_K_M + MTP + projector79.07 GiBHE 0.9000 (54/60) · MMLU 0.7500 (30/40) · tg 27.6 (35.7 +MTP)second unhooked ~79 GiB reasoner beside the protected qwen38fn-abl
Nemotron-3.5-Super-VL-120B-A12B UD-Q3_K_XL + projector62.86 GiBHE 0.9512 (164@4096) · MMLU 0.7667 (120@4096) · tg 27.1third 120B-class file; +2.4 pp of code at 1.8× the time per token
Kimi-Linear-48B-A3B-abliterated i1-Q4_K_M27.66 GiBHE 0.7866 (164@8192) · MMLU 0.5833 · tg 74.8no unit; loses both capability axes to kept files

Every file was re-hashed before deletion and all 11 matched their repos' published LFS oids (169.60 GiB of Tier B read back off disk, plus the Tier-A pair), so each one is restorable to the byte. Neither pass deleted anything on a hunch: the two Tier-A files were orphans (no run script, unit, switch case or Hermes provider), and the three Tier-B files each duplicated a role already held by a protected or favoured model.

What replaced the space is a cleaner roster rather than a smaller one: 15 model families at 629.11 GiB, of which 204 GiB is protected (the two Qwen3-Next files, the Flash-Next abl and the gemma QAT), 270 GiB is favoured (the Ornith set, the Distill, both gpt-oss builds), 16.75 GiB answers the channel on :8100, and the rest is one deliberate spare (Swift-1.5-Flash-Next-Heretic, 78.78 GiB) plus Axion-1.0-35B (19.71 GiB), MiniCPM-V-4.7 (27.60 GiB) and LFM2.5-24B (12.65 GiB).

Registry work done with the deletions, because a weight file with a live reference is a trap: the llama-nemotron-super-120b unit was removed and stripped from all 13 survivor Conflicts= lists (with a daemon-reload), the nemotron-super-120b and kimi switch cases were dropped and all three usage strings rebuilt (22 cases remain, syntax checked), and four run scripts were archived under retired-scripts/. Older stale peers still named in those same Conflicts= lines — llama-mellum, llama-kimi, llama-b4-27b and the rest — are inert and left for a dedicated registry audit.

Two things this page keeps on purpose. Table 4 retains all five rows, with ✗ and the retirement date, because the recommendations elsewhere in the report are argued from them. And the servable snapshot from 2026-10-31 keeps its dead endpoints with a note under it: a snapshot that silently shrinks stops being evidence. The restore recipes — repo, file, exact byte count, sha256 — are in the retirement ledger, along with the run ids behind every score above.

One acknowledged loss: the Nemotron Super was the only vision-capable 120B on the box. Its projector went with it, and re-fetching it is a 64 GB, 40-minute job with a script already written.


Version 1.20 (2026-11-05)

Two arrivals kept, one correction to an old row, and a harness finding that changes how every IFEval number on this page should be read.

modelHE @2048MMLU @2048IFEval prompt/instdecodesizeverdict
Axion-1.0-35B Q4_K_M0.8500 (51/60)0.7750 (31/40)0.2667 / 0.2000 at the 1280 pin, 0.7333 / 0.6444 at 409676.2 t/s19.71 GiBkept — best capacity per gigabyte
Swift-1.5-Flash-Next-Heretic IQ3_S0.9167 (55/60)0.8000 (32/40)0.8667 / 0.822225.6 bare, 54.63 with the MTP head78.78 GiBkept on disk, not hooked

Axion-1.0-35B is a qwen35moe MoE at 34.66B total, roughly A3B active, from Axion1-0 under Apache-2.0, with the release paper (arXiv 2611.03233) pitching open agent foundation models. At Q4_K_M it occupies 19.71 GiB and produces 1422 t/s prefill and 76.2 t/s decode — the fastest full-suite result in the archive after the two small instruction models, and no other 20 GiB file pairs 0.8500 code with 0.7750 knowledge at that rate (qwen38-35b: 74.8 t/s, 0.6417 knowledge; Ornith-1.5-35B: 74.1 t/s, 0.4417). Riddles 10/14 with all four misses pinned at the cap.

Swift-1.5-Flash-Next-Heretic is a Heretic V2 abliteration of Qwen3.8-Flash-Next at IQ3_S: 176.9B params, 78.78 GiB over eight shards, shipped as an AR-mode GGUF with no projector. Its first measurement read 25.6 t/s against the resident qwen38fn-abl's 52.9 for the same shape, which looked like a property of the file. It is not — it was a property of the bench. The two files drive the same MTP head (mtp-Qwen3.8-Flash-Next-Q8_0), and with it Swift runs at 54.63 t/s against the abl's 55.39 at 88.1% acceptance and mean draft length 3.64. The rule this restates is the one from v1.19: a paired drafter is worth 2x on this family, so a decode number quoted without its spec-decode arm is not a measurement of the model.

Matched at n=60/40/30 the two are a tie with opposite strengths — Swift takes code (0.9167 against 0.8667) and instruction-following (0.87 against 0.77 prompt-strict), the abl takes knowledge (0.8750 against 0.8000) — three items each way, inside one standard error. No retirement follows from it: the resident file keeps its unit and Swift sits on disk as a spare for code-heavy work rather than displacing anything.

The harness finding: IFEval has been measured at 1280 tokens, not at the cap quoted beside it. lm-eval's IFEval task defines its own max_gen_toks, which overrides the value passed in --model_args; § of this page already recorded that pin as the cause of one nonsensical historical row, and it generalises. On a thinking model it is not a rounding error: Axion scores 0.2667 prompt-strict / 0.2000 instruction-strict at the pin, and 0.7333 / 0.6444 with --gen_kwargs max_gen_toks=4096. Verified twice on the same server, same 30 items, same build — the only variable was the cap. Asked directly, "a sentence about a dog without using the letter e" cost the model 3,010 tokens and then answered correctly. Read every 4-decimal IFEval figure in this archive, including the driver's 0.93, as a 1280-token measurement; for models that terminate promptly that is enough, and for models that deliberate it is a floor.

Correction: qwen38fn-abl had no valid MMLU-Pro row. The matrix carried "invalid (1/120)" for it since 2026-10-13. Re-measured on the same build with the same chat 0-shot bare-letter method over 40 items it scores 0.8750 (35/40) — the highest knowledge figure in the table. The old cell recorded a harness outcome, not the model's knowledge, and the row now shows the measurement.

Operational note. The roster units carry mutual Conflicts=: starting llama-qwen38fn-abl.service silently stops the gemma driver on :8100. Any back-to-back comparison of two roster files has to restore the driver explicitly afterwards, and a benchmark that assumes the channel stayed up will be wrong.


Version 1.19 (2026-11-05)

One arrival, kept on demand — and a footprint rule that only appears at this size.

modelHE @2048MMLU @2048needlesdecodesizeverdict
Groot-65B-A6B Q4_K_M0.9000 (54/60)0.7500 (30/40)2k + 8k PASS27.6 t/s (35.7 with MTP)84.91 GB / 79.07 GiBkept, on demand — not a driver

Groot-65B-A6B-261023 is the largest file measured here, and the model name understates it: the GGUF header declares 116,514,464,640 parameters under qwen4exp — 48 blocks, 256 experts routed 10 per token, 12 full-attention and 36 Gated DeltaNet layers, plus the per-layer embedding bank this family carries. Six files (four Q4_K_M shards, a BF16 projector and a 4.89 GiB MTP draft head) were fetched sequentially at ~45 MB/s and every one was checked against the Hugging Face tree API's lfs.oid, size and sha256.

The number that changed the operating rules is not a score: it is MemAvailable. Served all-resident on upstream-master-bolt it pinned 71.0 GiB of GTT and left the machine with about 1 GiB available — the process then carried 21 GB of anonymous RSS and 21.7 GB of zram, and a first suite attempt (four slots, 32k context, vision projector loaded) was globally OOM-killed after seven minutes, with the kernel recording the victim's own anonymous RSS as 10 MB. GTT is unreclaimable system RAM on this box, not a separate pool, so "the weights fit the 100 GiB pool" is not the same claim as "the model fits the machine". Re-served on the halo-box fork with --load-mode mmap --lazy-mode auto it holds 50.4 GiB of GTT, runs at the same speed within noise (27.66 vs 27.43 t/s over three greedy samples), and co-exists with the driver and the boot model — measured together at GTT 73.9 GiB with 38 GiB still available.

What it scores. Code 0.9000 (54 of 60 at cap 2048) and knowledge 0.7500 (30 of 40): both subsets, so the row carries its item counts and is ranked against other subsample rows, not against the 164/120 columns. Decode 27.6 t/s is about 1.8x slower than the 116.8B gpt-oss file at similar active size, and the weak axis is instruction-following — IFEval 0.60 prompt-strict, 0.56 instruction-strict over 30 items, against 0.93 for Qwen3-Next-80B. Riddles read 9/14, but all five misses returned exactly the 2048-token cap with empty content; re-asked at 4096 the family-relations puzzle answers correctly after 1,953 thinking tokens, so the row is deliberation length, not capability. Per-request reasoning_effort is live on this template: the same arithmetic probe costs 943 tokens at default, 95 at low, 64 at medium, 2 at none.

The drafter result reverses a rule this page has argued for a month. Every grafted -md head measured here lost 8-11% despite 87-92% acceptance, and the standing advice was --spec-type none for MoE models. The MTP head that ships with Groot's own weights gained +28.7% (27.66 → 35.66 t/s, 76.3% acceptance, mean draft length 3.29). Provenance decides, not acceptance ratio: a head trained and exported with the trunk pays; a head borrowed from a sibling does not. Note also that this build exports no llamacpp:spec_decode_* gauges for qwen4exp — the ratio exists only in the server log.

Verdict: kept, on demand, not adopted. The driver still leads on code, knowledge, instruction-following and speed; this is a burst model for hard reasoning and code, and the first one that can sit beside the driver while it does it. Recipe: ~/llama-local/run-groot65b.sh [nomtp|mtp], port 8182, -c 16384 -np 1 -b 1024 -ub 256, q8_0 KV, GROOT_LAZY=auto.


Version 1.18 (2026-11-03)

Two arrivals measured, both retired — and one of them reverses the verdict in v1.16.

modelHE @4096MMLU @4096needlesdecodesizeverdict
Clover-35B-A3B-Preview Q4_K_M0.8598 (141/164)0.4000 (48/120)2k + 8k PASS53.7 t/s20.79 GiBdeleted — dominated on both axes by a model 6 t/s slower
Qwen3.8-FN-32B-A6B "Prune2" QMix——2k PASS, 8k FAIL20.0 t/s48.31 GiBdeleted — repetition on Vulkan, no factual knowledge on CPU

Clover is the first diffusion-trained model on this box: a Qwen3.6-35B-A3B finetune under arch clover_moe (upstream PR #31306), which needed its own upstream tree because no local fork registers the arch. Its code is one standard error above its base family (141 vs 137 of 164) and its knowledge is at base-family level (0.4000 vs 0.4500), against the driver's 0.9512 / 0.8417 at 48.9 t/s. Riddles print 14/14 with three items at exactly 4096 tokens — the substring grader matching a restatement — so the committed rate is 11/14. Arch support is AR mode only: the self-speculative diffusion decode that is the model's whole point is a separate unmerged PR, so the number above is the AR fallback, not a judgement on diffusion decoding.

The v1.16 correction. That entry recorded the 32B-A6B prune as unloadable: "header says 177 experts, tensors hold 64, the router still has 177 rows". The observation was right and the conclusion was wrong. The file has three expert banks per layer — ffn_*_exps at Q4_K/Q5_0, _tier1 at Q2_K, _tier2 at Q1_0 — and the bank dims sum to the router width exactly: block 0 is 64+60+53 = 177, block 1 is 64+60+30 = 154, block 24 is 64+41+10 = 115. Reading tier 0 alone is what made it look inconsistent, and the header patch tried in v1.16 was never the fix. Built against llama.cpp tag b10662 plus the publisher's own runtime/variable-runtime.patch, it loads in 12 s and serves at 32k context.

It is still unusable, for two separate reasons, and the split is the useful part. On Vulkan it degenerates into repetition on anything non-trivial — 17 * 23 = 17 * 23 = …, a 1024-token wall of / at the 8k needle, sometimes empty content — and it does so with q8_0 KV and flash-attn on AND with f16 KV and flash-attn off, which exonerates the cache quant and implicates the hand-rolled tier graph. On the CPU arm (-ngl 0, same tree) it is fluent and terminating with instruction-following intact (2+2, 5*6, "TEST"), but the knowledge is gone: France and Australia both answer "No", the largest planet is "Earth", 17*23 is "37". Pruning 3,659 layer/expert slots plus a 1-bit tier bank cost the facts.

Roster unchanged in size — both files were deleted the day they were measured — at 15 model files and 630 GiB. The archive grew to 147 runs, 70 model aliases, 54.6 hours of measured execution over 2026-10-07 → 2026-11-03. Both retirements carry sha256s and restore recipes in the ledger, including the build recipe for the patched runtime so the Prune2 result can be re-tested if the publisher re-exports without the Q1_0 tier.

Version 1.17 (2026-11-01)

An executive summary in plain language, and the technical one demoted to where it belongs. §0 now opens with five findings written without jargon — the small model wins, bigger is slower for no accuracy gain, coding and knowledge are separate skills, compression costs knowledge before skill, uncensoring is free when done properly — followed by what a sweep costs (0.14 kWh, about 3.5p) and an explicit "what this cannot tell you" paragraph (164 code problems and 120 knowledge questions per model, so gaps under ~5 points are noise).

The seven technical claims that used to open the report are now §0.1, the conventions are §0.2, and the roster inventory is §0.3. Nothing was discarded: the plain version answers "what did you find" and the technical version answers "what does that mean for this hardware".

Two corrections while rewriting it: an earlier draft called the driver "the best coder", which is not true — minicpm-v47-35b ties it at 0.9573 at cap 4096 — so the claim is now "joint-first on code, first on knowledge among models on disk"; and the recommendation list named a model that has since been deleted.

Version 1.16 (2026-11-01)

The week's most interesting shape does not load, and the reason is in the file. Qwen3.8-Flash-Next-32B-A6B "Prune2" (48.31 GiB, qwen4exp, 32B total / 6B active) was fetched to test the one gap in this roster — an active-parameter count between the driver's 4B and the 120B class's 12B. It fails in both trees:

check_tensor_dims: 'blk.0.ffn_down_exps.weight' expected 640, 2560, 177, got 640, 2560, 64, 1

The header claims 177 experts, the expert tensors hold 64, and the router still has 177 rows — so no single expert_count satisfies the file. Patching the header to 64 moves the error one tensor along to ffn_gate_inp, which is the proof that the export is internally inconsistent rather than mistagged. Recorded in the not-runnable reference, deleted with its hash, and noted in the ledger so nobody re-downloads 48 GiB to rediscover it. A re-export with matching counts would be worth testing on shape alone.

That makes three arrivals measured or rejected today (Dark-Craving, Holo4, this one) and none added: the roster is 17, at 630 GiB.

Version 1.15 (2026-11-01)

Two of the week's most-downloaded releases measured, both dominated; and a rebuild tested and refused.

modelHE @4096MMLU @4096riddlesdecodesizeverdict
Dark-Craving-26B-A4B v0.3 Q4_K_M0.9390 (154/164)0.8167 (98/120)14/1447.3 t/s16.45 GiBdeleted — dominated by the driver on every axis
Holo4-35B-A3B MXFP40.8598 (141/164)0.3250 (39/120)13/1460.5 t/s18.43 GiBdeleted — no knowledge retrieval

Dark-Craving is the driver's exact shape and lands 3 and 4 items behind it — inside sampling error — while decoding 17 % slower in 0.8 GiB more. Holo4 returns all 120 MMLU items as a bare letter in 0.2 s each; against a 25 % chance floor for ten options, 32.5 % is a finetune that does not retrieve knowledge when the answer is a single token, while its generative axes (0.8598 code, 13/14 riddles) stay competent. Both were deleted with byte counts, sha256s and restore recipes in the ledger.

§4.2 is new: the fork's origin/HEAD was 1,041 commits ahead with a Vulkan MoE MUL_MAT_ID fusion and a q8_0 mat-vec fix, so it was built and A/B'd on two weight classes. Decode flat to −2.8 %, quality identical, prefill noise — not adopted, and the matrix stays on one build identity rather than becoming a mixture.

Version 1.14 (2026-11-01)

A provenance manifest, and licences stated for the first time. §20.5 lists every servable artefact with its upstream repository and the licence the card declares, and records which bytes are provably upstream's: 13 of 17 artefacts verify by sha256 against the repository's LFS object id, re-computed on disk rather than copied from the fetch log. The four that do not are named rather than filled in — they predate the record, and their hashes are in data/provenance.json so anyone can close the gap in a minute.

That closes the largest hole in a document about reproducibility: it described 30 models' behaviour in detail while never saying where a single file came from or what licence governed it.

Version 1.13 (2026-11-01)

Eight stale facts corrected, notation moved to the front, sampling error tabulated. The headline count said "33 models" when the matrix has 30 rows out of 50 measured files; the roster was described as 15 units and 602 GiB in two places (it is 17 and 630); §18.3 still said "33 model files, 16 remain servable"; and §9.1's nemotron-super-120b paragraph quoted 0.9329, the number §21's own errata had already replaced with 0.9512 — the one defect of the kind this report exists to prevent. §12 still retired Kimi-Linear, which is on disk with a million-token window, and figures 15-16 said 27 models.

MMLU, GTT and SE are now defined in §0 rather than in §8 and the glossary; "the box" is gone in favour of "this machine"; §18.4 states that the 40-60 h were attended time. §8 gains a sampling-error table, so "is two items a real difference" has one answer instead of three mentions. Figures carry <title>/<desc> taken from their own captions.

Version 1.12 (2026-11-01)

A per-axis leaderboard page, generated from this matrix. leaderboards.html ranks the same 30 rows five ways — HumanEval, MMLU-Pro, reasoning traps, decode, prefill — with cap, item count, build and date on every line, and a separate table for the one sub-2048 floor so a floor is never ranked against a ceiling. It is generated by scripts/leaderboards.py from the sanitised report, not hand-written, so the two views cannot disagree; the build gate refuses to publish if either page stops linking the other.

Deliberately absent: a composite index (weighting code against knowledge requires assumptions this report will not make for the reader) and a per-watt ranking (power is measured in §15 but not paired with throughput per model, so any such number would be invented).

Version 1.11 (2026-11-01)

Two new files triaged at the same cap, one accepted and one rejected.

modelHumanEvalMMLU-Proriddlesperfverdict
Kimi-Linear-48B-A3B-abl i1-Q4_K_M, 27.66 GiB0.7866 (129/164) @81920.5833 (70/120) @819213/14pp 1774 / tg 74.8kept — 1M native ctx
Apex-35B-V1.0 Q4_K_M, 21.23 GiB0.4146 (68/164) @40960.5800 (58/100) @4096, thinking off9/14 at both 4096 and 8192pp 1992 / tg 61.5rejected and deleted

Kimi's row matches the safety-tuned kimi retired on 2026-10-29 item for item on code (0.7866), at a 10% higher decode rate, and it is the only million-token model on the roster. Its KV cache must be f16 on both sides (n_embd_head_v = 72 does not divide a q8_0 block of 32, and mixed types are refused), which caps the measured window at 65536 beside the driver.

Apex is the more instructive failure. At 4096 five of fourteen riddle items returned completion_n equal to the cap with empty content; at 8192, three did. It spends the whole budget thinking and never emits an answer, so this is not a cap artefact and no larger budget fixes it. 2203 s for 164 HumanEval items — 13.4 s/item at 61 t/s — is the same signature. Recorded in the rejected-candidates reference and deleted with its sha256, so nobody re-downloads 21 GiB to rediscover it.

Roster 16 → 17 servable; 630 GiB. Tables 4, 4b, 5, the four league tables and the §9.1 family paragraph are updated. The throughput league is now actually sorted by decode, which it had not been.

Version 1.10 (2026-11-01)

The last two cap-512 floors were re-measured at cap 4096, and they behave differently on each axis. Both models were served from a transient unit on :8086 beside the driver, so no resident model moved and GTT stayed at 53 of 100 GiB.

runmodelHumanEval @4096was @512MMLU-Pro @4096was @512riddlesperf
20261101-034435qwen38-35b0.8720 (143/164)0.8537 (140/164)0.6417 (77/120)0.5083 (61/120)14/14 (was 13/14)pp 1550 / tg 74.8
20261101-040537Ornith-1.5-35B Q4_K_M0.8354 (137/164)0.8232 (135/164)0.4417 (53/120)0.4167 (50/120)13/14pp 1633 / tg 74.1

Both scored 3/3 preflight and passed both needles; wall 1237 s and 661 s. The verdicts differ in the way that matters: qwen38-35b was a genuine floor — the 512 cap cost it 16 knowledge items (+13.3 pp, far outside SE at n=120) while costing only 3 code items — whereas Ornith-1.5-35B was already honest (+2 code, +3 knowledge items, both inside noise). §9.1 now carries that as a principle rather than a footnote: the cap's damage is axis-dependent, so one axis read at 512 cannot tell you which case you are in.

Rows updated: Tables 4 and 4b (both rows, with measured 2026-11-01 and build fork b3-1a33095), the §9.1 "35B trough" paragraph, §16.4's ablation comparison (it excused the boot file's 0.5671 against a cap-512 row that no longer exists), Table 5's two family rows, the riddles/throughput/knowledge league tables, and the open-items paragraph. §16.4 also gains the reason qwen38fn-abl stays poisoned: 78.1 GiB cannot sit beside a resident model, and the driver is not stopped for a measurement.

Publication fault, fixed: the weekly timer fired at 04:05 GMT while these runs were in flight and published a row that had only its identity and perf axes written. scripts/publish.sh now refuses to build while a bench-* unit is active or any run file is younger than 15 minutes, logging the skip instead.

Version 1.9.2 (2026-10-31)

Storage named, and bounded. §2.3 records the drives (BIWIN X570 PRO 4 TB for system and models, a Netac 2 TB spare) and the zram swap device, and states plainly what they do and do not affect: load time, not decode. Decode is LPDDR5X-bandwidth-bound and prefill is shader-bound, so no figure in this report depends on the SSD — which is worth saying, because "faster NVMe makes local LLMs faster" is a common and wrong inference. Paging being compressed RAM rather than disk is called out for the same reason.

Version 1.9.1 (2026-10-31)

The hardware is now named. The report described "this machine" throughout, because the sanitiser replaced the hostname; that over-corrected, since the chassis model is public product information and is the first thing a reader needs to place the numbers. It is now stated as a Minisforum MS-S1 MAX with the DMI values that prove it (product_family, product_name, sys_vendor, board, BIOS, EC) in the header and §2. The hostname, addresses and paths remain unpublished.

Version 1.9 (2026-10-31)

What it cost, stated as a range and labelled an estimate. §18.3 now leads with the only measured figure — 118 runs, 47.7 hours of benchmark execution — and drops the stale "33 models, 24 servable, 109 GiB" sentence, which predated this month's retirements. §18.4 adds the effort accounting: 222-318 hours total, of which 40-60 needed a person, and 40-60 to reproduce elsewhere now that the tools are published. It is the only number in the report that is not traceable to a run, so it says that about itself in the first line.

Version 1.8 (2026-10-31)

Every row now names the build that produced it. §4.1 has always recorded the commit behind each flavour, but only at flavour-and-window granularity, and the published catalogue carried nothing finer than fork/upstream. Both matrices gain a build column (fork b3-1a33095, bolt b1-a30fb49, flint pre-identity …), and the data export now carries tree plus build_info per run — the commit, without the absolute path. 28 of 29 matrix rows and all 16 servable rows resolved; Ling-3.0-flash carries fork pre-identity because its row has no HumanEval cell and its run predates stamping.

That closes the difference between a claim and a checkable claim: the document says "build identity per run", and it is now possible, from the published data, to see which commit sits behind every row.

Version 1.7 (2026-10-31)

Prior art, and the context curve that was missing. Reading two other public datasets — lhl's llama-bench sweeps and pepuscz's Flash-class results — made two gaps obvious and both are now closed or named:

Version 1.6 (2026-10-31)

Every row now says when it was measured. Tables 4 and 4b gained a measured column — the date of the best-scoring run for that row, derived by joining each row's HumanEval and MMLU values (with their caps) against ~/runs/catalogue.json and taking that run's timestamp. 28 of 28 matrix rows and 16 of 16 servable rows resolved; the one row with no HumanEval cell (Ling-3.0-flash) was dated from its MMLU value alone.

The join is scripted, not typed: two rows scoring identically at the same cap would have been a coin-flip, so the derivation prefers name overlap and then the most recent run, and it is re-runnable against the catalogue. What the column does not tell you is the model's release date, only when this box measured it — the spread runs from 2026-10-10 to 2026-10-31.

Version 1.5 (2026-10-31)

One re-measurement, on the boot model: ornith35b (Ornith-1.5-35B-Uncensored-YMQ-M, IQ3_XXS) had been measured only at cap 512, which the table's own rules call a floor. It was re-run at 4096:

Version 1.4 (2026-10-31)

One new measurement and no method change: Qwen3-Next-80B UD-Q4_K_XL was the only artefact on disk with no capability row, and it was the last one worth spending compute on when the 2026-10-30 model sweep turned up nothing fit to download (700 repos, 131 artefacts in the 3–70 GiB band, all either dense, over-pool, a re-quant, or an arch in neither build tree).

Version 1.3 (2026-10-30)

This edition changes facts, not method. Every number still comes from a file in ~/runs/ (120 run files at the time of writing; ~/runs/catalogue.json rebuilt 2026-10-29 15:53); nothing here is re-measured except where said. The only new measurement taken for this edition is the gemma4-26b-a4b Q8_0 re-run on 2026-10-30.

The roster moved. Ten artefacts were retired after the previous edition, in two passes, and the recipes for all ten are in ~/llama-local/RETIRED-models.md:

  2026-10-29   61.6 GiB   Kimi-Linear-48B-A3B (28.0), Hy-Memory-30B-A3B (17.0),
                          Mellum2.1-12B Q6_K (10.1) and its MXFP4 twin (6.5)
  2026-10-30  245.1 GiB   Ling-3.0-flash Q4_K_M (71.7), Swift Qwen3.8-Flash-Next IQ2_XS (63.5),
                          gemma4-26B-A4B Q8_0 (26.1), Nemotron-Bolt-74B IQ2_XXS (23.1),
                          Flint-40B-A2.5B i1-Q4_K_M (22.5), gemma4-31B QAT Q4_0 (18.0),
                          Huihui Coder390 IQ3_S1 (11.3), Nemotron-Bolt-E4B Q8 (8.9)

The models directory is 630 GiB and holds 17 artefacts, every one with a capability row — the last blank cell (Qwen3-Next-80B UD-Q4_K_XL) was filled on 2026-10-31. Fourteen user units remain, one of them enabled (llama-gemma4-26b-uncen, :8100), plus the system boot unit orient-ornith (:8094).

The driver changed. Since 2026-10-28 the channel is answered by the uncensored QAT Q4_K_M build of gemma4-26b-a4b on :8100, not the QAT Q4_0 build on :8099. Version 1.2 called :8099 the deployed driver; that unit is disabled and nothing listens on the port. §0, §6.1, §9.1, §11 and §12 are corrected accordingly, and the endpoint map in §20.2 now lists only live ports.

Three rows in Table 4 were wrong, and are fixed:

  1. qwen3next-q6 was published as 0.9207 (164@2048). No run supports it: its best is 0.9296 over 142 items, and 0.9207 is ornith-mtp-21g's figure, copied onto it. Symptom to watch for: two unrelated models with a score identical to four decimals.
  2. nemotron-super-120b was published as 0.9329 (164@4096). Its runs read 0.9512 (156/164) and 0.9470 (154/164) at that cap; the printed figure matched neither.
  3. gemma4-26b-a4b and gemma4-26b-qat-q4 were listed as two models. They are one file measured twice (156/164 and 154/164, MMLU 101/120 both). The run alias was shared with the Q8_0 baseline.

What was deliberately not changed. §10's family-spread analysis, §13's platform comparison, §14's hosted comparison, §15's power section, §16's abliteration analysis and §17's DeepSeek deployment analysis were left alone: none of their inputs moved, and re-cutting them would mix a roster change into an argument about method. Rows for deleted models are retained throughout, marked ✗ with the retirement date, because a table that silently drops its losers cannot be audited.

Figures rebuilt, and four pipeline bugs fixed. §19's ten figures were regenerated from the rebuilt catalogue, and doing it accurately turned up three defects that had been silently distorting them: (a) charts.py merged every run under one alias, so the Q8_0 gemma and the QAT file collapsed into each other and both ornith35b files did too — it now canonicalises one name per weight file, and figure 19's fourth pair is a real Q8_0→QAT comparison instead of a file against itself; (b) it took the latest run per axis rather than the best, contradicting §9's stated rule — it now takes the best, so figures and Table 4 agree; (c) §8's poisoned-MMLU rule was applied by deleting the whole run rather than the cell, so qwen38fn-abl's good HumanEval row (0.8476) disappeared from every figure along with its 1/120 MMLU — the filter now blanks the axis and keeps the rest; (d) runs-catalogue.py read the FIRST row of each axis, and IFEval writes an empty n=0 row before its scored n=30 row, so every IFEval score in the catalogue was being dropped — fixed, which is why the live driver now appears in figures 18 and 21. Figure 23 also used to plot the retired flint40b-abl run, whose 1-second samples had aged out of the 72 h raw window so it could no longer be rebuilt; it now plots the 2026-11-01 Kimi-Linear full suite, the longest run still inside that window.

Known open items. qwen38fn-abl's MMLU-Pro cell is a poisoned 1/120 rather than a score, and it cannot be re-measured without stopping the driver (the file is 78.1 GiB, past the pool beside a resident model), which this project does not do; and the driver's context curve (§23.2) now covers 32k, 64k and 113k prompts — but no other model on disk has been measured beyond an 8k needle since the kernel moved to 7.3.0-5.



Back to the report · deleted models and restore recipes · leaderboards by axis.


Source, data and build scripts: github.com/strix-halo-128-benchmarks/strix-halo-128-benchmarks.github.io.