AtomicChat/dsv4-eval-artifacts

DeepSeek-V4-Flash-0731 — quantization measurements

Everything needed to reproduce, audit or extend the numbers published in AtomicChat/DeepSeek-V4-Flash-0731-GGUF: the reference logits, the evaluation corpus, the raw tool output for every quant we measured, and the parsed results.

Every GGUF of this model that we could find on the Hub was measured here — ours, unsloth's, bartowski's, ggml-org's, antirez's and others — on one machine, against one reference, with one command. Publishers normally report numbers from their own harness, which makes cross-vendor comparison meaningless. These files exist so that anyone can check ours instead of trusting them.

[!IMPORTANT] Measured using 8x5090

Files

The .kld file stores the reference model's full probability distribution at every scored token position — roughly 258 KB per token at this vocabulary size. It is what makes the KL-divergence numbers comparable: every quant is compared against these exact logits.

Result schema

{"repo": "bartowski/DeepSeek-V4-Flash-0731-GGUF",
 "name": "MXFP4",
 "bytes": 145678901234,
 "ppl": "4.5446",
 "kld": "0.156403",
 "rms": "12.686",
 "top1": "87.369"}

ppl is Mean PPL(Q) from the KL-divergence block, kld is Mean KLD, rms is RMS Δp, top1 is Same top p — the share of positions where the quant picks the same next token as the reference. Note that Mean PPL(Q) and the standalone Final estimate: PPL printed by the same tool are different aggregations and do not match; the logs contain both.

Measurement setup

Hardware matters here, and it is not optional

The routed experts of this model are 96% of its weights and they are stored in MXFP4. llama.cpp has two paths for that format — unpack to BF16 and use a normal tensor-core matmul, or feed the packed 4-bit data into block-scaled instructions. The second is gated on compute capability ≥ 12.0, which covers consumer Blackwell only. H100 and H200 are 9.0, B200 is 10.0, B300 is 10.3; all take the first path despite having FP4 hardware.

Same file, same corpus, same commit, reference model:

A 4–5% difference from the GPU alone. Reproducing these numbers requires consumer Blackwell and a build that targets it — compiling for sm_90 on a 5090 gives the H100 numbers, because the native kernel never lands in the binary.

Reproducing

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/24162/head:dsv4 && git checkout dsv4
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build -j --target llama-perplexity

hf download AtomicChat/dsv4-eval-artifacts --repo-type dataset --local-dir .

./build/bin/llama-perplexity \
  -m <any-quant>-00001-of-*.gguf \
  -f wiki-alt.txt --kl-divergence-base ref5632.kld --kl-divergence \
  -ngl 99 -c 5632 -b 5632

To rebuild the reference from scratch instead of downloading it, run the same command against AD-BF16 with only --kl-divergence-base and no --kl-divergence. Takes about ten minutes and should print Final estimate: PPL = 4.5381.

Caveats

• —Absolute values are not comparable to numbers published elsewhere. Other publishers use different corpora, context lengths and hardware. Compare within one table.
• —51 chunks at ctx 5632 gives roughly ±0.003 on mean KLD. Differences smaller than that are noise.
• —Quants of derived models — expert-pruned, abliterated, distilled — are deliberately excluded. KL-divergence against this reference would measure the difference between models, not the cost of quantization.

License

MIT. Derived from deepseek-ai/DeepSeek-V4-Flash-0731. Produced by Atomic Chat.