Team Ai
Modelpublic

BobbieBieee/Osprey-Speculative-Decoding

sourceHugging Facemitupdated 24d agoView on Hugging Face
0likes
Model Card

Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Accepted at EMNLP 2026.

Paper: https://arxiv.org/abs/2609.09338 Code: https://github.com/LeanModels/Osprey

Adapted Osprey drafters for three target models. Every checkpoint starts from the same target-agnostic pretrained backbone (Qwen3-4B pruned to 2 layers, pretrained on FineWeb) and is adapted to its target with vocabulary alignment, zero-initialized QKV expansion, and on-policy EAGLE-3 distillation.

PathTargetAdaptation data
qwen3-8b/osprey-chatQwen/Qwen3-8Bchat, 65k
qwen3-8b/osprey-codeQwen/Qwen3-8Bcode, 65k
qwen3-8b/osprey-commonsenseQwen/Qwen3-8Bcommonsense, 65k
qwen3-8b/osprey-financeQwen/Qwen3-8Bfinance, 65k
qwen3-8b/osprey-mathQwen/Qwen3-8Bmath, 65k
llama33-70b/ospreymeta-llama/Llama-3.3-70B-InstructOpen-PerfectBlend
minimax-m25/ospreyMiniMaxAI/MiniMax-M2.5code, 70k

Each directory holds config.json and model.safetensors. Serving requires the SGLang patch shipped in the code repository (sglang_patches/), which adds multi-layer EAGLE-3 drafts and keeps the drafter's own embedding at load time.

Also in this repository:

PathContents
pretrained/qwen3-4b-2layer-fineweb-55kThe Stage-2 backbone every drafter above starts from: Qwen3-4B pruned to 2 layers and pretrained on FineWeb for 55k steps. Input to scripts/osprey/convert_checkpoint.py.
data/*_train_65k_qwen3_8B_4096.jsonlThe five Qwen3-8B adaptation splits (chat, code, commonsense, finance, math): 65k prompts each, responses regenerated by Qwen3-8B.
data/*_eval_512_qwen3_8B_4096.jsonlThe five 512-prompt evaluation splits behind Figure 2 and Table 1.
data/code_train_70k_minimax_m25_4096.jsonlThe MiniMax-M2.5 adaptation split: 70,897 coding prompts with responses regenerated by MiniMax-M2.5 (Tables 2–3).
data/code_eval_512_minimax_m25_4096.jsonlIts 512-prompt held-out slice.
data/perfectblend_train_100k_llama33_70b.jsonlThe Llama-3.3-70B-Instruct adaptation split: 100,000 conversations from Open-PerfectBlend with responses regenerated by the target (Table 1).
data/perfectblend_eval_512_llama33_70b.jsonlIts 512-prompt held-out slice.

The Llama-3.3-70B-Instruct splits are drawn from frankleeeee/PerfectBlend-Regenerated-Llama-3.3-70B-Instruct (1,419,775 conversations): records are numbered in file order, shuffled once with random.seed(42), and the first 100,000 / next 512 taken as train / eval.

The MiniMax-M2.5 splits take their prompts from the code subset of NVIDIA's Nemotron-Post-Training-Dataset-v2 (CC BY 4.0) and keep that dataset's metadata fields. The generator field is inherited from the source and does not describe the assistant turns here, which were regenerated by MiniMax-M2.5.

Citation

bibtex
@inproceedings{bie2026osprey,
  title     = {Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding},
  author    = {Bie, Fengxiang and Jian, Yuqing and Yu, Yifan and Zhou, Zhongzhu and Shao, Zelei and Athiwaratkun, Ben and Song, Shuaiwen Leon and Xu, Chenfeng and Wu, Xiaoxia and Zhang, Tianyi},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026},
}