darkmaniac7/TokForge-AccelerationPack-Draft

TokForge

• —Website: https://tokforge.ai
• —Discord: https://discord.gg/Acv3CBtfVm
• —Google Play: https://play.google.com/store/apps/details?id=dev.tokforge
• —iOS TestFlight: https://testflight.apple.com/join/jnufjzRr

Runs on-device in the TokForge app.

TokForge Acceleration Pack — Qwen3 KL-Distilled Draft Model

Overview

KL-distilled Qwen3-0.6B draft model for speculative decoding in TokForge. Trained on 10,000 teacher samples from Qwen3-8B. On our test devices, speculative decoding with this draft and dense Qwen3 targets measured +34% to +43% faster decode in chat workloads.

What This Is

A small (0.6B) draft model that:

• —Runs on CPU alongside your main GPU model
• —Predicts tokens in parallel — main model batch-verifies and accepts correct ones
• —KL distillation matches the teacher's full logit distribution, not just top-1 tokens
• —KL distillation targets higher acceptance rates than stock or abliterated drafts

Measured Results

On our test devices, speculative decoding with this draft and dense Qwen3 targets measured +34% to +43% faster decode in chat workloads. Uplift is per device and per workload: backend routing, prompt length, and thermal state all change the outcome, and some device and target pairings do not benefit.

Why KL Distillation?

KL divergence loss teaches the draft model to match the teacher's probability distribution across ALL tokens, not just the most likely one. This is critical because MNN's greedy sampler needs the draft's top-1 to match the teacher's top-1 — and KL training optimizes exactly for this.

Training Details

• —Teacher: Qwen3-8B-HF (base)
• —Student: Qwen3-0.6B-HF (base, LoRA r=16, alpha=32)
• —Data: 10,000 teacher-generated samples (prose, code, Q&A)
• —Loss: 80% KL divergence + 20% cross-entropy, temperature=1.5
• —Training: 3 epochs, batch=4, grad_accum=4 (1,875 optimizer steps)
• —Final KL: 0.339 (21% lower than v1's 0.43 trained on 1K samples)
• —Hardware: 2x NVIDIA RTX PRO 6000 Blackwell (teacher GPU 0, student GPU 1)
• —Export: MNN Q4 quantization (quantbit=4, quantblock=128)

Optimal Draft Config

Best with: target OpenCL, draft CPU, d=3, thread_num=2, power=high.

config_cpu.json:

{
    "backend_type": "cpu",
    "thread_num": 2,
    "precision": "low",
    "memory": "low",
    "sampler_type": "greedy",
    "power": "high"
}

Why thread_num: 2? On Android's WALT CPU governor, using too many threads (4+) can cause the scheduler to spread work across efficiency cores at low frequency. 2 threads stay on performance cores at high clock speeds.

Compatible Target Models

• —Qwen3-8B: supported
• —Qwen3-14B: supported
• —Qwen3-4B (MNN): Supported, enabled by default (CPU speculative decoding, fleet-verified on our test devices)
• —Qwen3.5: Not compatible (different architecture: LinearAttention vs full MHA)

SoC Compatibility

Usage

Download via TokForge app: Settings > Spec Decode > Download Acceleration Pack

Typical TokForge recipe

• —target backend: opencl
• —draft backend: cpu
• —draft threads: 2
• —d=3
• —greedy sampling
• —low precision / low memory

Version History

License: Apache 2.0 Source: KL-distilled from Qwen/Qwen3-0.6B using Qwen/Qwen3-8B as teacher Built with: TokForge

Limitations and Intended Use

• —Best current evidence is strongest on Qwen3-8B and Qwen3-14B.
• —Device uplift depends heavily on backend routing, prompt length, and thermal state.
• —This is a TokForge-specific runtime bundle, not a standard Transformers checkpoint.

Community

• —Website: tokforge.ai
• —Discord: Join the Discord
• —Collection: TokForge Mobile Draft Models