manifesta/adaption_verified_math_code_instruct
054
1---2license: apache-2.03library_name: peft4base_model: google/gemma-4-31B-it5base_model_relation: adapter6pipeline_tag: image-text-to-text7language:8 - en9datasets:10 - manifesta/verified-math-code-17k11tags:12 - lora13 - peft14 - adapter15 - math16 - code17 - reasoning18 - step-by-step19 - verified-data20 - gemma421 - autoscientist22 - adaption23 - negative-result24inference: false25model-index:26 - name: adaption_verified_math_code_instruct27 results:28 - task:29 type: text-generation30 name: Verified math and code instruction following31 dataset:32 name: Verified Math and Code 17k (held-out)33 type: manifesta/verified-math-code-17k34 split: test35 metrics:36 - type: win-rate37 name: Win rate vs base (adapted / base)38 value: 46.039 - type: loss40 name: Held-out eval loss41 value: 1.2242 source:43 name: Adaption AutoScientist held-out evaluation44 url: https://huggingface.co/manifesta/adaption_verified_math_code_instruct45---46 47<p align="center">48 <img src="https://huggingface.co/manifesta/adaption_verified_math_code_instruct/resolve/main/assets/banner.png" width="100%" alt="Show Your Work: a verified math and code adapter for Gemma 4 31B">49 <a href="https://github.com/A1VARA5/verified-math-code-17k"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-pipeline%20%2B%20verifier-181717?logo=github"></a>50 <a href="https://github.com/A1VARA5/verified-math-code-17k/actions/workflows/verify.yml"><img alt="verify" src="https://github.com/A1VARA5/verified-math-code-17k/actions/workflows/verify.yml/badge.svg"></a>51</p>52 53[](https://manifesta.adaptionlabs.app/)54[](https://demo-theta-one-40.vercel.app)55[](https://huggingface.co/google/gemma-4-31B-it)56[](https://huggingface.co/datasets/manifesta/verified-math-code-17k)57[](#evaluation)58[](#numerical-stability)59[](https://adaptionlabs.ai)60 61# Show Your Work: Verified Math and Code (Gemma 4 31B LoRA)62 63A PEFT LoRA adapter trained on 17,586 math and code problems where the answers were independently checked: 9,104 math rows carry a gold answer computed separately, 4,156 code rows were executed against unit tests, and 3,740 general rows were deliberately left unverified as an anti-forgetting slice.64 65**It lost. The head-to-head result is 46 wins for the adapted model against 54 for the base.** That is a regression, not a tie and not a win. Training loss fell from 3.48 to 1.24 and held-out eval loss fell from 2.37 to 1.22, and the model still came out behind.66 67This card gives the diagnosis from the weights and the training state rather than dressing the number up. There are two causes and they compound: the base was already strong on this domain, and the run was numerically unstable, with 29 of 59 optimizer steps hitting the gradient clip.68 69## Contents70 71- [TL;DR](#tldr)72- [Quickstart](#quickstart)73- [What is actually in these weights](#what-is-actually-in-these-weights)74- [Evaluation](#evaluation)75- [Why it lost](#why-it-lost)76- [Numerical stability](#numerical-stability)77- [The cross-run finding](#the-cross-run-finding)78- [Training](#training)79- [Intended use and limits](#intended-use-and-limits)80- [Dataset](#dataset)81- [Live interface](#live-interface)82- [Related models](#related-models)83- [License](#license)84- [Citation](#citation)85 86## TL;DR87 88- **What it is:** a rank-8 LoRA adapter on `google/gemma-4-31B-it`, 10,956,800 trainable parameters, 43.9 MB.89- **What it was for:** step by step derivations for math and code, with the working shown rather than just the answer.90- **The number:** 46 for the adapted model, 54 for the base. A regression of 8 points.91- **The reasons:** the base scored 8.0 out of 10 on the source corpus before any adaptation, the highest starting point of the four datasets in this project, so there was almost nothing to add. On top of that, 29 of 59 steps exceeded the clip threshold of 2, peaking at 677.92- **Good news on loading:** unlike the sibling adapters in this project, the declared base model string is correct and resolves. `google/gemma-4-31B-it` is ungated and Apache-2.0.93- **Built by:** MANIFESTA (Aivaras Navardauskas) for the Adaption AutoScientist Challenge, Math and Code.94 95## Quickstart96 97The base string in `adapter_config.json` is `google/gemma-4-31B-it`, which resolves, is ungated, and is Apache-2.0 licensed. No workaround needed here. (The two chart adapters in this project both shipped broken base strings, so this is worth stating explicitly.)98 99Gemma 4 is a conditional-generation checkpoint with a vision tower, so use the image-text-to-text auto class even when you only want text.100 101```python102import torch103from transformers import AutoModelForImageTextToText, AutoProcessor104from peft import PeftModel105 106BASE = "google/gemma-4-31B-it"107ADAPTER = "manifesta/adaption_verified_math_code_instruct"108 109processor = AutoProcessor.from_pretrained(BASE)110base = AutoModelForImageTextToText.from_pretrained(111 BASE, torch_dtype=torch.bfloat16, device_map="auto"112)113model = PeftModel.from_pretrained(base, ADAPTER)114model.eval()115 116messages = [{117 "role": "user",118 "content": [{"type": "text", "text":119 "A factory makes 1,440 pencils in 8 hours. Two machines are down, cutting output "120 "by 40%. How many pencils are made in a 6 hour shift? Show your work."}],121}]122 123inputs = processor.apply_chat_template(124 messages, add_generation_prompt=True, tokenize=True,125 return_dict=True, return_tensors="pt",126).to(model.device)127 128with torch.inference_mode():129 out = model.generate(**inputs, max_new_tokens=512, do_sample=False)130print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))131```132 133Because the base string is correct, the one-line loader works too:134 135```python136from peft import AutoPeftModelForCausalLM137model = AutoPeftModelForCausalLM.from_pretrained("manifesta/adaption_verified_math_code_instruct")138```139 140Optional merge:141 142```python143model = model.merge_and_unload()144```145 146Given the evaluation result, the most useful thing to do with this snippet is run the same prompt through the base with and without the adapter attached and compare. That is what the judge did, and the base won.147 148## What is actually in these weights149 150Read straight out of `adapter_model.safetensors` and `adapter_config.json`.151 152| Property | Value |153| --- | --- |154| PEFT type | LoRA, `task_type: CAUSAL_LM` |155| Rank `r` | 8 |156| `lora_alpha` | 8 (scaling factor alpha/r is exactly 1.0) |157| `lora_dropout` | 0.0 |158| `bias` | none |159| `use_rslora` / `use_dora` | false / false |160| Target modules | `q_proj`, `v_proj` |161| `exclude_modules` | empty |162| Tensors in the file | 220 |163| Decoder layers touched | 60 of 60 for `q_proj`, **50 of 60** for `v_proj` |164| Vision tensors | **0** |165| Trainable parameters | 10,956,800 (float32, 43.9 MB) |166| Share of the 31B base | roughly 0.035% |167 168### The v_proj gap is architecture, not a bug169 170`v_proj` is missing on exactly ten layers: **5, 11, 17, 23, 29, 35, 41, 47, 53 and 59**. Every sixth layer.171 172Those are precisely the layers listed as `full_attention` in the base model's `layer_types`. The other fifty are `sliding_attention`. Checking the base model's own weight map confirms it: `model.layers.5.self_attn` contains `q_proj`, `k_proj`, `o_proj`, `q_norm` and `k_norm`, and **no `v_proj.weight` at all**, while `model.layers.4.self_attn` does have one.173 174The reason is in the base config: `attention_k_eq_v: true`, with `num_global_key_value_heads: 4` and `global_head_dim: 512` on the global layers. On full-attention layers the key and value projections are shared, so there is no separate `v_proj` module for PEFT to wrap. It asked for `v_proj` on all 60 layers and silently got it on 50.175 176Nothing failed. But it does mean the adapter covers less of the attention stack than `target_modules: [q_proj, v_proj]` suggests, and the layers it covers least are the global-attention layers, which are the ones carrying long-range context. For multi-step derivations, long-range context is the part that matters most.177 178### No vision tensors179 180Gemma 4 31B is multimodal, with a 27-layer vision tower. This adapter touched none of it. For a text-only math and code task that is the right call, so it is recorded here for completeness rather than as a criticism. It is a different story on the [chart QA adapter](https://huggingface.co/manifesta/adaption_scientific_chart_qa_17k), where the frozen vision tower is the whole explanation for that run's null result.181 182## Evaluation183 184Judged by Adaption's AutoScientist evaluator on a held-out split, against the same base model it was trained from. Head-to-head, blind.185 186| Metric | Base | Adapted | Change |187| --- | --- | --- | --- |188| **Win rate** | **54** | **46** | **-8 points** |189| Held-out eval loss (first eval) | 2.3657 | | |190| Held-out eval loss (final) | | 1.2200 | -1.146 |191| Training loss | 3.482 (step 1) | 1.238 (step 59) | -2.244 |192 193Eval loss across the five checkpoints: 2.3657, 1.5618, 1.3206, 1.2417, 1.2200.194 195<p align="center">196 <img src="https://huggingface.co/manifesta/adaption_verified_math_code_instruct/resolve/main/win-rates.png" width="80%" alt="Win rates, adapted versus base">197</p>198 199**Read those two rows together, because they disagree.** Loss dropped by 64% and the model still lost the head-to-head by 8 points. A falling loss curve says the adapter learned to reproduce the training distribution. It says nothing about whether reproducing that distribution is an improvement over what the base already did. Here it was not.200 201This is the single most useful thing in this repo: eval loss is not the headline metric, and if you optimise against it you can ship a regression with a beautiful chart attached.202 203## Why it lost204 205Two causes, and they compound.206 207**1. The base was already good, so there was no room.** Adaptive Data scored the source corpus at 8.0 out of 10 before adaptation, grade B, the highest starting quality of the four datasets built in this project. Adaptation moved it to 8.2, a +2.5% lift, and the grade did not change. Compare the agronomy corpus at 5.0 to 8.4 (+68.0%) and the chart corpus at 6.0 to 7.1 (+18.3%).208 209Gemma 4 31B is instruction-tuned and already writes competent step by step derivations for exactly this kind of problem. A rank-8 adapter on two projections, roughly 0.035% of parameters, cannot add mathematical capability the base lacks. What it can do is change style and formatting. When the base's existing style is already good, changing it is as likely to hurt as help, and a blind judge comparing two competent derivations will punish any drift toward the training corpus's quirks.210 211**2. Two thirds of the updates were clipped.** See below.212 213**3. The global-attention layers got the least adaptation.** As covered above, the ten `full_attention` layers received `q_proj` only. Those are the layers handling long-range dependency, which is what a multi-step derivation depends on to keep an intermediate result consistent from line four to line twelve.214 215**What would need to change:** a lower learning rate or a longer warmup to stop the gradient spikes, a higher rank if the goal is genuinely new capability rather than style transfer, and a harder corpus. Problems the base already solves at 8.0 out of 10 are the wrong training signal. The value should sit in problems it gets wrong.216 217## Numerical stability218 219This run was not stable.220 221| Gradient norm statistic | Value |222| --- | --- |223| Clipping threshold (`max_grad_norm`) | 2 |224| Peak gradient norm | **677.21** (step 38) |225| Median gradient norm | 1.97 |226| Minimum | 0.389 |227| Steps above 1 | 39 of 59 (66%) |228| **Steps above the clip threshold of 2** | **29 of 59 (49%)** |229| Steps above 10 | 9 of 59 (15%) |230| Steps above 100 | 5 of 59 (8%) |231 232Largest spikes: step 38 at 677.2, step 15 at 645.0, step 40 at 389.4, step 14 at 172.5, step 26 at 138.6. The final step of the run, step 59, was still at 40.1.233 234<p align="center">235 <img src="https://huggingface.co/manifesta/adaption_verified_math_code_instruct/resolve/main/training-metrics.png" width="80%" alt="Training metrics">236</p>237 238What clipping at 2 does when the true norm is 677 is keep the direction and throw away the magnitude. The update becomes a fixed-size step along whatever direction that batch happened to point in, and the relative weighting between batches is gone. When that happens on half the steps of a 59-step run, the optimizer is not really following the loss surface any more, it is taking a sequence of equal-length steps in noisy directions.239 240The loss still came down, which is worth sitting with. A falling loss curve does not certify a healthy run.241 242For contrast, the [chart QA adapter](https://huggingface.co/manifesta/adaption_scientific_chart_qa_17k) trained on the same platform with the same clip threshold peaked at 0.72 and clipped on zero of its 34 steps. Same settings, completely different numerical behaviour, which points at the data and the base rather than the configuration.243 244(The Adaption run dashboard reports the peak as 744.9. The 677.21 above is the maximum in `trainer_state.json` in this repo, which is the number you can verify yourself. Both are two to three orders of magnitude over the threshold and the conclusion is the same.)245 246## The cross-run finding247 248Three completed AutoScientist runs, three different base models. Win rate tracked how weak the base already was on the domain, and did not track dataset size.249 250| Run | Base | Params | Rows ingested | Steps | Rows per step | Win rate (adapted vs base) |251| --- | --- | --- | --- | --- | --- | --- |252| Chart QA, first build | Qwen3.5 9B | 9B | 6,976 | 21 | 332 | 51 vs 49 |253| Chart QA 17k | Gemma 3 27B | 27B | 17,070 | 34 | 502 | 50 vs 50 |254| **Verified math and code (this model)** | Gemma 4 31B | 31B | 17,586 | 59 | 298 | **46 vs 54** |255 256**The bigger the base, the smaller the win.** 9B scored 51, 27B scored 50, 31B scored 46. This model sits at the bottom of that trend and it is the only one that went backwards.257 258**Scaling the corpus did nothing.** The chart corpus was rerun at 2.4x the rows, 6,976 to 17,070, and the win rate went 51 to 50. Because `batch_size` is set to `max`, extra rows widen the batch rather than adding updates: step count went 21 to 34 while rows per step went 332 to 502. More rows per step is a smoother gradient estimate for the same small number of updates, not more learning.259 260<p align="center">261 <img src="https://huggingface.co/manifesta/adaption_verified_math_code_instruct/resolve/main/assets/base_model_size_vs_win_rate.png" width="80%" alt="Base model size against win rate across three runs">262</p>263 264**Caveat, stated plainly:** three data points, three different base models, three different corpora, three different domains. Base size, base family and task difficulty all move together, so nothing here isolates a cause. It is not a controlled experiment and should not be read as one. Testing it properly means holding the corpus fixed and varying only the base.265 266## Training267 268Trained with [Adaption](https://adaptionlabs.ai) AutoScientist, SFT with LoRA, on Adaptive Data output.269 270| Hyperparameter | Value |271| --- | --- |272| Base model | `google/gemma-4-31B-it` (resolves correctly) |273| Method | SFT, LoRA (PEFT), chat format |274| LoRA rank / alpha / dropout | 8 / 8 / 0.0 |275| Trainable modules | `q_proj`, `v_proj` (requested on 60 layers, attached on 60 and 50) |276| Learning rate | 1e-4 |277| Scheduler | cosine, 0.5 cycles, warmup ratio 0.1, min LR ratio 0.1 |278| Weight decay | 0 |279| Max gradient norm | 2 |280| Epochs | 1 |281| Batch size | `max` (per-device train batch size 1 with accumulation) |282| Optimizer steps | 59 |283| Evaluations | 5, every 11 steps |284| Train on inputs | false |285| Precision | bfloat16 base, float32 adapter weights |286| Total FLOPs | 4.073e18 |287| Adaption job ID | `9f46e2b9-ac73-4b97-ad13-cc6ccd1501cf` |288| Training experiment ID | `3ce14a29-6d5d-48ea-849f-372a6102c360` |289| Adaption dataset ID | `6ad1a83f-b806-4a20-ba6d-239029d30711` |290 291## Intended use and limits292 293**Use it for**294 295- Reproducing and inspecting this regression. That is the honest primary use.296- Studying the gap between a falling loss curve and a falling win rate, which this run demonstrates about as cleanly as it can be demonstrated.297- Studying gradient instability in short LoRA runs, using the 59 logged steps in `trainer_state.json`.298- A baseline for a rerun with a lower learning rate, a longer warmup, or a harder corpus.299 300**Do not use it for**301 302- Production math or code assistance. Stock `google/gemma-4-31B-it` beat it 54 to 46 on the held-out split. Use the base.303- Anything where a wrong derivation carries cost: engineering, finance, dosing, safety calculations. Verified training data does not make outputs verified.304- Executing generated code without a sandbox. The training data was executed against unit tests; the model's output at inference time was not.305 306**Technical limits**307 308- The adapter is behind its own base model on the only head-to-head evaluation that was run. Treat the weights as a research artefact.309- Rank 8 on two projections, roughly 0.035% of base parameters, one epoch, 59 optimizer steps.310- `v_proj` is absent on the ten global-attention layers. Not a defect, but the coverage is thinner than the config implies.311- Half the optimizer steps were clipped. The weights reflect a run that did not converge cleanly.312- English only.313- Adapter weights are float32 while the base is bfloat16. PEFT casts on load, but the adapter file is double the size a bf16 export would be.314- One judge, one held-out split. Reported as-is, no reruns, no best-of selection.315 316## Dataset317 318Trained on Adaption dataset `6ad1a83f-b806-4a20-ba6d-239029d30711`, built from [`manifesta/verified-math-code-17k`](https://huggingface.co/datasets/manifesta/verified-math-code-17k) (also on [Kaggle](https://www.kaggle.com/datasets/aivarasnavardauskas/verified-math-code-17k)).319 320- 17,586 rows ingested, expanded to 21,063 rows in the training split after adaptation321- 9,104 math rows with an independently computed gold answer322- 4,156 code rows executed against unit tests, 100% of them run323- 3,740 general instruction rows deliberately left unverified, as an anti-forgetting slice324- Domain mix after adaptation: math 56%, code 25%, science 3%, with a long tail across roughly 40 further domains at 1% or below325 326<p align="center">327 <img src="https://huggingface.co/manifesta/adaption_verified_math_code_instruct/resolve/main/assets/verification.png" width="90%" alt="What was verified inside the 17,000 rows">328</p>329 330Adaptive Data lifted the corpus quality before training:331 332| Adaptive Data metric | Before | After |333| --- | --- | --- |334| Quality score | 8.0 | 8.2 (+2.5%) |335| Grade | B | B |336| Percentile | 15.3 | 17.8 |337 338That +2.5% is the smallest lift of the four datasets built in this project, and it is the clue. The corpus started at 8.0, so there was very little for adaptation to fix, and by extension very little for the model to learn that it did not already know. **+2.5% is an improvement in the data, not in the model.** The model's own number went the other way.339 340## Live interface341 342The full written case for this entry, every claim paired with the command that checks it, is at343[The Verdict](https://demo-theta-one-40.vercel.app).344 345**Show Your Work** runs at [manifesta.adaptionlabs.app](https://manifesta.adaptionlabs.app/). Give it a problem, get a step by step derivation with a verification panel, and use the "Change a number" button to check that the derivation actually recomputes rather than pattern matching to a memorised answer.346 347<p align="center">348 <img src="https://huggingface.co/manifesta/adaption_verified_math_code_instruct/resolve/main/assets/ui_02_show_your_work_solved.png" width="90%" alt="Show Your Work solving a multi-step word problem with a verification panel">349</p>350 351## Related models352 353- [`manifesta/adaption_scientific_chart_qa_17k`](https://huggingface.co/manifesta/adaption_scientific_chart_qa_17k), scientific chart QA on Gemma 3 27B. 50 vs 50, a null result, because the LoRA covered zero vision tensors while the task is entirely visual. Its declared base string 404s.354- [`manifesta/scientific-chart-qa-lora-qwen3.5-9b`](https://huggingface.co/manifesta/scientific-chart-qa-lora-qwen3.5-9b), the first chart build on a 9B base. 51 vs 49, 21 optimizer steps, LoRA on only 8 of 32 decoder layers and again zero vision tensors. Its declared base `togethercomputer/Qwen3.5-9B` also 404s.355- [`manifesta/brandvoice-marketing-model`](https://huggingface.co/manifesta/brandvoice-marketing-model), the Part 1 marketing adapter that did work, 56% win rate on Llama 3.3 70B. Style transfer on a base with a weak default style, which is the case where a small adapter has room to move.356 357## Everything behind these weights is public358 359The dataset this adapter was trained on, the scripts that built it, and a verifier that rechecks360every number claimed here against the live artifacts:361 362**<https://github.com/A1VARA5/verified-math-code-17k>**363 364```bash365git clone https://github.com/A1VARA5/verified-math-code-17k366cd verified-math-code-17k367python verify.py368```369 370Standard library only, no install step and no account. 23 checks, and the same 23 run on a daily371schedule in GitHub Actions, so the badge above goes red if any claim on this card stops being true.372The tensor facts in this card are among the checks: layer coverage is re-derived from the published373`adapter_model.safetensors` by name, not copied from the config.374 375Dataset: <https://huggingface.co/datasets/manifesta/verified-math-code-17k>376 377 378## License379 380- **Adapter weights:** Apache-2.0.381- **Base model:** [`google/gemma-4-31B-it`](https://huggingface.co/google/gemma-4-31B-it), Apache-2.0 and ungated. No access request needed.382- **Training dataset:** [`manifesta/verified-math-code-17k`](https://huggingface.co/datasets/manifesta/verified-math-code-17k), CC0-1.0.383 384## Citation385 386```bibtex387@misc{showyourwork_mathcode_2026,388 title = {Show Your Work: a verified math and code adapter for Gemma 4 31B, and why it regressed},389 author = {Navardauskas, Aivaras},390 year = {2026},391 note = {MANIFESTA. Adapted with Adaptive Data by Adaption, AutoScientist Challenge, Math and Code. Win rate 46 vs 54. Half the optimizer steps clipped.},392 howpublished = {\url{https://huggingface.co/manifesta/adaption_verified_math_code_instruct}}393}394```395 396Built with [Adaptive Data by Adaption](https://adaptionlabs.ai). Platform documentation at [docs.adaptionlabs.ai](https://docs.adaptionlabs.ai). Dataset, adapter and analysis by MANIFESTA (Aivaras Navardauskas) for the AutoScientist Challenge.397