gitcommit90/Qwen3.8-Flash-Next-NVFP4-DenseFP8-One-Spark

Qwen3.8-Flash-Next NVFP4 + Dense FP8 — One DGX Spark

A tested one-DGX-Spark deployment of `nvidia/Qwen3.8-Flash-Next-NVFP4` with targeted in-loader FP8 conversion of the remaining large BF16 dense projections, native MTP K=3, piecewise CUDA graphs, prefix caching, and C6 scheduling.

47–53 tok/s short code · 37–43 tok/s long-context/thinking · 60 tok/s structured · 969K-token BF16 KV pool · 72/72 BENCHY low-effort cases across C1–C6

Release status

This page describes a runtime recipe; it does not yet contain transformed model weights. The dense projections are converted to FP8 during model loading. A stock-vLLM safetensors export is a separate planned artifact and is not claimed here.

The reproducible runtime, overlays, benchmark evidence, and setup scripts live in the companion GitHub repository: https://github.com/gitcommit90/qwen38-flash-next-nvfp4-one-spark.

What changed

NVIDIA's checkpoint stores the main routed experts in NVFP4 but leaves attention, Gated DeltaNet, lm_head, and other dense layers in BF16. Profiling on GB10 showed approximately 67% of decode kernel time in BF16 dense GEMVs.

The runtime recipe converts only the large decode-critical projections:

• —linear_attn.in_proj_qkvz
• —linear_attn.out_proj
• —self_attn.qkv_proj
• —self_attn.o_proj
• —lm_head

Small hyper-connection, shared-expert, gate, indexer, and auxiliary matrices remain BF16 because FP8 kernel overhead made them slower on this device. The selected kernel is Marlin FP8 W8A16: FP8 weights with BF16 activations.

Results

Measured on one NVIDIA DGX Spark / GB10, TP1, 262,144-token model context, BF16 KV, MTP K=3, C6 scheduler capacity:

The observed K3 agentic loop reached 74–81 decode tok/s, but shorter generated tool calls favored that figure. The robust comparison is 3.95 emitted tokens per target-model step at K3 versus 2.96 at K2 (+33%).

BENCHY agent evaluation

A 24-run C1–C6 × effort matrix was executed without retries, edits, or overwritten passes: 288 total cases.

Recommended default for agent workloads: `reasoning_effort: low`.

Quality characterization

Teacher-forced target-logit comparison against the NVIDIA checkpoint with BF16 dense layers, 30 texts / 2,019 positions:

• —Top-1 agreement: 0.886
• —Mean KL divergence: 0.040
• —Margin >2 nats preserved: 1,013/1,014

No fine-tuning was performed. Native MTP speculation verifies candidate tokens against the transformed target.

Runtime configuration

• —Source checkpoint revision tested: fab0aecb760cec45227f6656abcaafa11abca87a
• —vLLM base: 8a728663c1c3eeace834a95f5654fa653cc1998c
• —ExLlamaV3 extension: e648f1a131365aae15920073e761a3fa5a527654
• —Dense FP8 groups: gdn,attn,lm_head
• —Dense kernel: Marlin FP8 W8A16
• —MTP: K=3, EAGLE trailing-block drop disabled
• —CUDA graphs: piecewise
• —BF16 KV allocation: 28 GiB
• —Prefix caching: enabled
• —Async scheduling: disabled
• —Maximum sequences: 6

Intended use

Local text, coding, tool-use, and agent workloads on one DGX Spark where decode latency, usable concurrency, and large BF16 KV capacity are more important than retaining byte-identical BF16 dense weights.

Limitations

• —This page currently has no standalone dense-FP8 checkpoint; use the companion runtime recipe.
• —The FP8 transformation changes target numerics relative to NVIDIA's original mixed checkpoint.
• —Full 250K cache branching/retrieval qualification remains pending.
• —Vision was not included in the performance/quality gate reported here.
• —Results are hardware-, workload-, runtime-, and sampling-specific.
• —The runtime overlays are pinned to one vLLM base and must be ported deliberately to newer versions.

License

The runtime code is Apache-2.0. Upstream model weights remain governed by the NVIDIA Open Model License and applicable Qwen terms. This repository does not redistribute those weights.