Modelpublic

JohnP1/d1a-e4b-mlx-q8

sourceHugging Faceapache-2.0updated 16h agoView on Hugging Face

0likes90downloads

Model Card

D1A-E4B v0.6 · MLX 8-bit

JohnP1/d1a-e4b v0.6 for Apple Silicon. The build is the same as v0.5's:

—linear layers and token embeddings 8-bit (group 64), per-layer embeddings 4-bit;
—pointer head fp32;
—calibration temperature 1.95 in d1a_config.json;
—about 6 GB on disk, plus 1 GB of photo, voice and video encoders.

Only the first weight shard and the pointer head differ from v0.5: the per-layer embeddings (the second shard) and the encoders are byte-identical. The scores below were measured on this build.

D1A is a small open decision model. One document and a set of typed questions go in (yes/no, choice, score), and a calibrated probability for every option comes out, in one forward pass, with no generated text. It speaks the TypeSafe System One API (POST /v1/systemone), so the TypeSafe SDK and existing clients work against it.

v0.6 labels pull requests the way they are labelled today. On 487 pull requests created after all of its training, severity goes from 74.7% to 79.1% (+4.3 points) and change type from 86.7% to 90.6% (+3.9) against v0.5, and P1 recall returns to v0.4's 0.35 (v0.5: 0.22). It also gains on long documents (+2.0), agent-factory routing (+3.4) and held-out transfer (+1.4), with lower calibration error on most suites. It costs hard decisions (−2.7) and change type on the older frozen PR test set (−4.1), so it ships as a release exception (below). v0.5 stays available. See Results.

Release exception. v0.6 does not pass D1A's release rule against v0.5. scripts/decide.py vetoes it (V1) on hard decisions: hard-v1 −2.7 points [−4.9, −0.5], 61 fixes against 90 breaks, past the −2.0 limit for a card suite. It also loses change type on the older frozen PR test set, −4.1 [−6.0, −2.3] (19 fixes, 58 breaks), and pooled over all 21 lines it is level with v0.5, −0.20 [−0.98, +0.52]. It ships anyway, by the owner's decision of 2026-10-10: the model's live job, the one model on the owner's Mac mini, is labelling current pull requests, and on PRs newer than all of its training it is clearly better. The likeliest reading of the disagreement is that the project's labelling conventions have changed over time and v0.6 follows the current ones. Stay on v0.5 (JohnP1/d1a-e4b-mlx-q8@v0.5) if you rely on hard-v1-style reasoning or on the older PR-label conventions.

What's inside

A LoRA adapter (rank 16 on the attention and MLP projections: q, k, v, o, gate, up, down) and a small pointer head (256-dimensional) on google/gemma-4-E4B at revision 411aa17b. Each version continues training the previous one, so v0.6 contains every stage below:

Version	Stage	Training data	Settings
v0.1	General decisions	decision-v7 suite: ~12,500 records from public classification datasets plus generated policy and rule data	2 epochs, lr 5e-5
(v0.2)	Kev's later skills	dates and missing evidence (1,425), hard decisions, developer-tool decisions and long documents (hard-v1, devtools-v1, documents-v1 training partitions, states up to 4,096 tokens), decision-v7 replay 4,000	1 epoch, lr 2e-5
v0.2	Japanese and agent routing	Japanese decisions from JGLUE (JNLI, JCommonsenseQA, JSTS; 6,000), agent-factory and model-routing questions (2,745), decision-v7 replay 3,000	1 epoch, lr 5e-5
v0.3	Pull-request labeling	3,500 English and 768 Japanese pull requests labeled with change type, blast radius and severity (closed PRs of NousResearch/hermes-agent, MIT; Japanese copies machine-translated; severity follows the labeling job's rules for docs-only and dependency PRs), replay: decision-v7 1,500, JGLUE 500, routing 300	1 epoch, lr 5e-5, documents up to 1,536 tokens
v0.4	Pull-request labeling, round 2	the 4,063 English PRs v0.3 did not see, its 2,396 rare-label PRs (blast radius, P0, P1, P4) again, 2,185 more PRs with a blast-radius label, the 768 Japanese PRs again; replay: decision-v7 1,500, JGLUE 500, routing 300 (11,540 records)	from v0.3, 1 epoch, lr 5e-5, documents up to 1,536 tokens, 1,443 steps on one L4
v0.5	Kev's skills, all of them	the whole hard-v1 (6,000) and devtools-v1 (5,320) training partitions, documents-v1 1,000, JGLUE 650, PR labels 1,350 (so the earlier skills are replayed, not forgotten), decision-v7 replay 1,000: 15,320 records	from v0.4 through a 300-step pilot on 2,400 records of the same mix; 1 epoch, lr 2e-5, states up to 6,400 tokens, 1,915 steps on one A100
v0.6	Current pull-request labels	479 hermes-agent PRs created 2026-10-02 to 10-05, shown twice (once in each half of training); documents-v1 2,520 never trained before; routing: agent factory 600, generic 600; decision-v7 1,000; low replay of every earlier skill: hard-v1, devtools-v1, JGLUE and PR labels 300 each, Japanese and blast-radius PRs 100 each: 7,078 records	from v0.5, 1 epoch, lr 2e-5, states up to 5,120 tokens, 885 steps on one Apple M1 Max (MPS)

Before training, the v0.6 mix was screened against 47 evaluation partitions, the 487 newer PRs below included: none of their documents or PRs is in it.

Calibration: a single temperature, T = 1.95, fitted on pooled held-out rows (decision-v7 calibration split and PR-labeling development set; calibration error 0.093 before, 0.015 after, out of fold), stored in d1a_config.json. v0.6 carries no per-use-case temperatures yet, so a request with "use_case": "routing" is read at 1.95 too; a refitted routing temperature is planned as v0.6.1.

Results

Pull requests newer than all training

487 hermes-agent pull requests created after every PR in v0.6's training, held back before v0.6 was trained (the test was registered in advance). All three models are MLX 8-bit at their served temperatures, with the serving context. Differences are paired over PRs, with 95% bootstrap intervals:

Question	Majority answer	v0.4	v0.5	v0.6	v0.6 − v0.5
Severity (P0–P4)	67.8%	77.4%	74.7%	79.1%	+4.3 [+0.8, +7.8] (52 fixes, 31 breaks)
Change type	51.5%	86.4%	86.7%	90.6%	+3.9 [+1.2, +6.6] (31 fixes, 12 breaks)
P1 recall (23 PRs)		0.35	0.22	0.35
P0 recall (7 PRs)		0.29	0.29	0.71

Against v0.4, v0.6 is +1.6 [−0.8, +4.1] on severity and +4.1 [+1.4, +7.0] on change type. A cheaper alternative, v0.5 with outcome memory fitted on the same recent labels, reached 89.3% on change type but only 76.4% on severity (+1.6 [−1.0, +4.3]), with P1 recall 0.17, and failed its own bar.

Every suite, against v0.5

Held-out data only. Both builds are MLX 8-bit (this repository) at their served temperatures (v0.5 1.78, v0.6 1.95), scored with d1a.eval.benchmark through D1A's quality gate and scripts/decide.py. Differences are paired over questions, with 95% bootstrap intervals:

Set	v0.5	v0.6	Δ [95% CI]	Calibration error, v0.5 → v0.6
Hard decisions (hard-v1, 1,083 questions)	71.5%	68.8%	−2.7 [−4.9, −0.5]	0.045 → 0.043
Developer-tool decisions (devtools-v1, 1,074)	69.8%	70.1%	+0.3 [−1.4, +1.9]	0.048 → 0.037
Long documents (documents-v1, 920)	87.5%	89.5%	+2.0 [+0.1, +3.7]	0.042 → 0.028
Held-out transfer-v4 (764, sources never trained on)	71.6%	73.0%	+1.4 [−1.2, +4.0]	0.106 → 0.055
General decisions (decision-v7, 1,468)	85.3%	83.9%	−1.4 [−2.9, +0.0]	0.030 → 0.015
JGLUE development / test (1,500 each, Japanese)	82.1% / 80.2%	81.7% / 80.5%	−0.3 / +0.3	0.026 → 0.042 / 0.024 → 0.039
Agent factory development (640)	90.5%	93.9%	+3.4 [+1.4, +5.6]	0.130 → 0.091
Model routing: generic (270) / hand-labelled (45)	98.9% / 100%	98.5% / 97.8%	−0.4 / −2.2	0.029 → 0.015 / 0.032 → 0.014
External: semif-v1 (252) / typesafe-v1 (102)	84.7% / 74.5%	83.3% / 74.5%	−1.4 / +0.0	0.110 → 0.095 / 0.104 → 0.129
External: wanli-v1 (256) / wanli-v2 (1,002)	72.7% / 67.1%	70.3% / 67.1%	−2.3 [−5.5, +0.8] / +0.0	0.097 → 0.101 / 0.123 → 0.072
Dates (900) / unknowable facts (525) / assertions (1,020)	92.7% / 68.2% / 94.4%	92.1% / 67.8% / 94.9%	−0.6 / −0.4 / +0.5	0.028 → 0.053 / 0.062 → 0.058 / 0.051 → 0.056
PR labels, English test (953 PRs)¹: change type / severity / blast radius (139)	89.4% / 76.7% / 57.6%	85.3% / 77.9% / 61.9%	−4.1 [−6.0, −2.3] / +1.2 [−1.3, +3.6] / +4.3 [−2.2, +10.8]	0.038 → 0.030
PR labels, English development (946 PRs)¹: change type / severity / blast radius (138)	90.8% / 74.3% / 58.7%	90.7% / 72.5% / 52.9%	−0.1 / −1.8 [−3.9, +0.3] / −5.8 [−11.6, +0.0]	0.020 → 0.017
PR labels, Japanese test (92 PRs, all questions)¹	76.2%	72.5%	−3.6 [−8.1, +0.5]	0.071 → 0.043
Owner's labelling job, replayed (193 questions, review-bot labels): type / blast radius	86.3% / 63.3%	90.5% / 66.3%	+3.6 [−1.0, +8.3] overall
39 recent PRs from the owner's repositories, labels checked by hand²: type / blast radius / severity	84.2% / 74.4% / 65.6%	86.8% / 66.7% / 65.6%	−1.8 [−7.4, +3.7] overall
Report only: transfer-v2 (560) / transfer-v9 (1,046)	74.1% / 65.3%	73.8% / 65.9%	−0.4 / +0.6	0.074 → 0.041 / 0.083 → 0.054

Pooled over all 21 lines, v0.6 − v0.5 = −0.20 [−0.98, +0.52]. Latency is unchanged: v0.6 over v0.5 on short requests 1.0009 [0.9991, 1.0035], within run-to-run noise (measured interleaved).

¹ These PR sets are older than the 487 above. v0.5's numbers here are its 8-bit build at T 1.78 over the full sets, so they differ a little from v0.5's own card, which scored PR labels on the bf16 PyTorch checkpoint. ² The PRs the owner's labeling job saw on 2026-10-06 and 07, labelled by the owner on the playground's review page.

Against Kev-4B's model card (fp32, development splits; an approximate comparison): hard decisions 78.6, developer tools 73.9, long documents 89.1, transfer 81.7. v0.6 is level with Kev-4B on long documents.

Calibration: v0.6's single temperature fits its calibration sets (0.015 out of fold), and its calibration error is lower than v0.5's on 15 of the 22 suites the gate scores (most on transfer-v4, agent factory and wanli-v2). It is higher on the other 7, most on dates (0.028 → 0.053), typesafe-v1 (0.104 → 0.129) and JGLUE (0.026 → 0.042).

Photos, voice and video

media/ holds Gemma 4 E4B's own vision and audio encoders (bf16, ~1 GB, unchanged from the base: the LoRA adapter only touches the text model; byte-identical to v0.5's). With them d1a.serving.serve answers questions about a photo, a voice clip or a video with this same model.

—The endpoint is POST /v1/systemone/media: a /v1/systemone request plus {"media": {"type": "image" | "audio" | "video", "data": <base64>}}.
—The encoders turn the media into soft tokens, which the 8-bit language model reads right after <state>.
—A video is read as 16 timestamped frames; its sound is not used.
—The encoders are fetched on the first such request; a plain download skips them.

No media in training (zero-shot). On the playground's photo, voice and video examples, v0.6 changes 1 of 14 requests from v0.5 (one video example).

The questions it was trained for

PR labeling works best with these questions, worded exactly so, over a document of the form title / author / stats / body / files (one - status path +added/-deleted line per file):

—type (choice): "Primary change type from files and body, not the title prefix." Options: type/bug, type/docs, type/feature, type/perf, type/refactor, type/security, type/test.
—blast (choice): "How far a mistake in this PR spreads in production." Options: review:blast-contained (one module), -moderate (one subsystem), -broad (shared helper or config), -massive (auth, permissions, or all paths).
—sev (choice): "How serious the problem this PR addresses is — not the risk of merging the diff as-is." Options P0 to P4.

The PR-labeler recipe has the exact wording, the document builder and the training and scoring scripts. The mix builder builds and screens mixes like v0.6's. Other questions about other documents work as in v0.2.

Run it

On Apple Silicon (the PyTorch checkpoint, for NVIDIA GPUs and CPUs, is JohnP1/d1a-e4b v0.6):

bash

pip install "d1a[serve] @ git+https://github.com/jonpol01/d1a"
python -m d1a.serving.serve --run JohnP1/d1a-e4b-mlx-q8@v0.6 --port 8009

or in-process: from d1a import D1A; D1A.load("JohnP1/d1a-e4b-mlx-q8@v0.6").decide(state, questions). For hard-v1-style reasoning or the older PR-label conventions, load @v0.5.

Limits

—Hard decisions are 2.7 points below v0.5 (68.8% against 71.5%; the release exception above), and 10 below Kev-4B (78.6).
—Change type on older PRs is worse: 85.3% against 89.4% on the frozen English test set, and 72.5% against 76.2% on the Japanese one (all questions, 92 PRs). It calls more PRs type/bug than v0.5 did. On current PRs it is better (above).
—Blast radius on the owner's own repositories is worse: 67% against 74% on 39 hand-checked PRs (1 fix, 4 breaks). v0.6 was not trained for blast radius; on the larger frozen test set it is 62% against 58%.
—Severity: on the 487 newer PRs, P2 recall falls (0.55 against v0.5's 0.79) while P3 and P0 rise; P0 rests on 7 PRs. Severity on the older development set is 1.8 points lower.
—One temperature for every question, and no use-case temperatures (routing's is planned for v0.6.1). Calibration error is higher than v0.5's on JGLUE, dates and typesafe-v1.
—Smaller losses against v0.5: decision-v7 −1.4, wanli-v1 −2.3, one of 45 hand-labelled routing questions.
—The recent PR labels come from one large open-source project and a few days of it; if its conventions move again, so will these numbers.
—Questions are in English; the documents can be English or Japanese.

Versions

Tag	What it adds
`v0.1`	general decisions (decision-v7, 2 epochs, calibrated)
`v0.2`	Kev's later skills, Japanese, agent routing
`v0.3`	pull-request labeling, English and Japanese
`v0.4`	pull-request labeling round 2: severity, Japanese, less biased blast radius
`v0.5`	Kev's harder skills back (hard, developer-tool and long-document decisions), PR labels held
`v0.6`	current pull-request labels (severity +4.3, type +3.9 on newer PRs); a release exception: hard decisions −2.7, so pin `v0.5` for those

Older tag names (v0.1-2epoch, v0.1.1-2epoch-calibrated, v0.2-hybrid) still work.

License and data

Apache-2.0. Base model: Gemma 4 by Google (Apache-2.0). Code: github.com/jonpol01/d1a, built on Kev by Jared Palmer (Apache-2.0). Japanese decision data derived from JGLUE by Yahoo Japan Corporation and Waseda University (CC BY-SA 4.0). Pull-request data from NousResearch/hermes-agent (MIT); the labeled PR dataset itself, including the recent PRs, is private.