Apppublic

reyden009/speculative-decoding-paper

sourceHugging Facemitupdated 2mo agoView on Hugging Face

0likes

App README

Speculative Decoding on a 16 GB Consumer GPU

Acceptance, modern drafters, and quantization in Gemma 4 12B and Qwen3-8B

Martin Ulises Millan Guerrero · 2026-08-08

Abstract

Speculative decoding (SD) is a mature family of lossless inference acceleration techniques, yet the evidence for its modern drafters — EAGLE-3, DFlash, and DSpark — comes almost entirely from datacenter GPUs serving batched workloads. The community that runs quantized GGUF models on a single 16 GB consumer GPU has no empirical guidance on which drafter to use, at which draft length, or with which quantization.

This study presents the first systematic evaluation of modern drafters on a 16 GB consumer NVIDIA GPU (RTX 4060 Ti) with official checkpoints in llama.cpp: 26 target × drafter configurations over 1,474 greedy-decoded prompts across nine benchmarks in three cognitive domains, per-position acceptance curves for 20 configurations, and a draft-length sweep (k ∈ {5, 7, 10}, 21 runs).

Key results

Finding	Result
DSpark fastest Qwen3-8B drafter	1.64× at Q4 · 1.93× at Q8_0
EAGLE-3 robust, quantization-invariant	1.39×–1.63× (acceptance nearly constant)
Vanilla 1.7B surprisingly competitive	1.41×
DFlash fails on Qwen3-8B	0.63× (α ≈ 0.009)
DFlash / MTP on Gemma 4 12B	2.09×–2.30× / 2.39×–2.71×
Confidence gating (p-min 0.6)	acceptance 0.616 → 0.714, but throughput 87.6 → 80.7 tok/s (single-user)
Quantization	barely affects acceptance; Q4 draft saves ≈ 1 GiB VRAM

Resources

—Companion repository (code, data, analysis): reyden009/speculative-decoding-lab
—Paper (PDF): paper.pdf
—arXiv-ready source bundle: arxiv-bundle.tar.gz
—Markdown source: paper.md

Full paper

Setup: RTX 4060 Ti 16 GB · Intel i5-12400 · llama.cpp (CUDA) · Gemma 4 12B and Qwen3-8B GGUF · everything user-space, no sudo.