Modelpublic

Pradheep1647/eagle3-speculative-decoding-policy

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes

Model Card

EAGLE3 Speculative Decoding -- Energy-Aware Policy Models

Eight models, one problem: pick (speculative_num_steps, speculative_eagle_topk, speculative_num_draft_tokens) for sglang + EAGLE3 so GPU energy utilization lands inside a 95-98% band. All trained on the `eagle3-speculative-decoding-energy-sweep` dataset. Full writeup, sweep mechanism, and live-validated results: project README.

Shared I/O contract -- input is a 4-dim state [batch_size/8.0, gpu_temp_c/100.0, gpu_mem_used_mb/8192.0, gpu_util_pct/100.0]; output is an index into the same 19-action space (RL/policy.py in the repo above), decoded to the three sglang flags.

file	algorithm	format	load
`mlp_bandit/policy.pth`	contextual bandit (QNetwork)	torch state_dict	`QNetwork(4, 19).load_state_dict(torch.load(...))`
`lookup_table/model.json`	per-bs empirical best	JSON	`json.load(open(...))` -- `{batch_size: {"config": [...], "mean_reward": ...}}`
`linucb/model.npz`	LinUCB	numpy	`np.load(...)` -- `theta` (per-action weight vectors), `A` (per-action design matrices), `actions`
`thompson_sampling/model.npz`	Bayesian linear TS	numpy	`np.load(...)` -- `mean`, `cov` (per-action posteriors), `actions`
`gbt/model.joblib`	gradient boosted trees	sklearn	`joblib.load(...)` -- `GradientBoostingRegressor`, features `[state, action_idx/19]`
`doubly_robust/model.joblib`	doubly robust (direct model)	sklearn	`joblib.load(...)` -- `LinearRegression`, IPS correction term is not persisted (recomputed from raw sweep data at pick time)
`cql/policy.pth`	conservative Q-learning	torch state_dict	`QNetwork(4, 19).load_state_dict(torch.load(...))`
`bcq/policy.pth`	discrete BCQ	torch state_dict	`torch.load(...)` -- dict with `behavior_state_dict`, `q_state_dict`, `actions`

Which one to actually use

mlp_bandit, lookup_table, and doubly_robust agree on every batch size and are the live-validated picks. cql and bcq collapsed to the non-speculative baseline past bs=1 (overly conservative default hyperparameters against this reward scale) and are not recommended -- kept here for completeness, not as a suggested pick. See the project README for the full live A/B numbers per algorithm.

Hardware this was validated on

RTX 4060 Laptop GPU (8GB), unsloth/Llama-3.2-1B-Instruct target + rescommons/SpecForge-EAGLE3-Llama-3.2-1B-Instruct draft, 80W power cap. Picks are specific to this hardware/model pair -- retrain on the linked dataset (or a fresh sweep) before trusting these on different hardware.