Modelpublic

whadupapp/goff-lite

sourceHugging Faceotherupdated 17d agoView on Hugging Face

0likes238downloads

Model Card

goff Lite

A standalone, single-label scoring head over microsoft/harrier-oss-v1-0.6b, fine-tuned on typed decisions and merged so vLLM can serve it without a PEFT adapter at runtime.

The model scores one candidate at a time. A question's candidates are each rendered into their own prompt, scored independently, and softmaxed together within the question to give a distribution over candidates. It is not a chat model and does not generate text.

—Architecture: Qwen3ForSequenceClassification, 28 layers, hidden size 1024, 1 output label (problem_type: regression)
—Precision: bfloat16 (~1.19 GB)
—Context window: 32,768 tokens (inherited from the base model). Training prompts were truncated at 1024.

https://goff-ai.com/

Training

Fine-tuned from microsoft/harrier-oss-v1-0.6b with FuRA (Full-Rank Adaptation with spectral preconditioning) on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj, full per-block rank, output_one_block factorization, SVD conversion, with the score head trained alongside via modules_to_save.

Objective is a listwise RLCD recipe: soft-target cross entropy (weight 1.0), noisy-logit policy gradient with 4 samples per question, log and spherical proper-scoring rewards (spherical weight 0.75), and ranked probability score for ordinal score questions (weight 1.0). Exploration sigma annealed 0.4 → 0.1.

Data

23,526 typed questions from three sources, split into 22,350 train / 1,176 validation by case, so questions that share one input state never straddle the split:

Source	Questions	Notes
`LocalLLaMA/typed-decisions`	6,000	`all` config, `train` split; 1,200 cases × 5 questions, all soft-labelled
`jaredpalmer/kev` eval suites	7,222	the `train` splits checked into the repo at commit `90990a5`, converted to one question per record. Not the `jaredpalmer/kev-suites` Hub mirror, which holds the larger training pool. 24 sources: banking77, AG News, Yelp, BoolQ, MNLI, SST-5, TREC, DBpedia, IMDB, plus kev's generated policy, contrastive and composition sets
goff synthetic decisions	10,304	161 hand-written situation seeds × 64 generated examples, German and English, 23% carrying prompt-injection or authority-pressure content labelled with the honest answer. Not public.

Question types across the mix: 11,232 choice, 7,220 noul, 5,074 score.


Epochs	6 (2,100 steps; trained for 4, then resumed for 2 more with the cosine schedule rebuilt over the full length)
LR / schedule	2e-4, cosine, 70 warmup steps
Batch	64, no gradient accumulation, bf16, gradient checkpointing
Max length	1024 tokens; questions whose longest candidate prompt exceeded it were dropped, not truncated (0 of 23,526)
Seed	42
Device	1 × NVIDIA A100-SXM4-80GB
Checkpoint selection	best held-out accuracy; the released checkpoint is the final one (step 2,100), which ties the selected checkpoint on accuracy and is better calibrated

Held-out validation (the mixed split above, not comparable to the benchmark below): accuracy 0.740, soft accuracy 0.661, soft cross entropy 0.855, Brier 0.309, ECE 0.083, score within-one 0.915.

Compute

Trained on GPU VMs from Lyceum. The full recipe — 6 epochs over 23,526 questions, plus the evaluation passes — runs in about five hours on a single A100 80GB there.

Evaluation

Scored on the `LocalLLaMA/typed-decisions` test split (400 cases, 2,000 typed decisions, revision ea930645), ECE at 15 bins, confidence taken as max candidate probability for every backend. Numbers below are from the merged weights published here.

Metric	goff Lite	`convaiinnovations/laya` (typed-decisions run)
Accuracy	0.749	0.766
Soft accuracy	0.517	0.471
Soft cross entropy	0.906	0.884
Brier	0.070	0.062
ECE (15 bin)	0.096	0.213
KL	0.139	0.117
Total variation	0.168	0.174
Score MAE	0.262	0.242
Score within-one	0.990	0.995

Read this honestly: the Laya comparison checkpoint remains the stronger point predictor on this split, while this model is markedly better calibrated (ECE 0.096 vs 0.213) and carries more probability mass on the right candidate (soft accuracy 0.517 vs 0.471). Per question type, accuracy is 0.732 choice, 0.840 noul, 0.694 score.

Two comparisons worth recording. The previous release, trained on typed-decisions alone, scored 0.720 accuracy / 0.092 Brier / 0.099 ECE on this split; adding the kev and synthetic data and the training changes above moved that to 0.749 / 0.070 / 0.096. And the merged weights reproduce the adapter they came from to within one decision out of 2,000 (adapter: accuracy 0.7495, Brier 0.0700, soft CE 0.9064).

Latency on the benchmark host (A100, batch 128, merged weights): 70.6 ms per case, 14.1 ms per decision.

Serving

bash

vllm serve whadupapp/goff-lite --max-model-len 32768

Score each candidate prompt separately, apply the fitted calibration temperature, then softmax the scores within each question. The temperature must be fitted on a held-out calibration split — not on the training or checkpoint-selection split.

Limitations

—Trained and evaluated only on typed decisions (choice, noul, score); behavior outside that prompt format is unmeasured.
—Evaluated on typed-decisions only. Roughly 44% of the training data is synthetic material generated for goff's own domains, and no public benchmark covers it.
—The window is 32,768 tokens, but no training prompt exceeded 1024; behavior on longer inputs is unmeasured.
—Calibration is fitted downstream; raw scores out of this model are uncalibrated. ECE on the benchmark is 0.096, worse than the 0.071 of an earlier attention-only run — if calibration matters more than accuracy for your use, that trade is worth measuring.
—German coverage is partial: the synthetic portion is mixed German and English, and the benchmark is English-only, so German behavior is unmeasured.
—Latency figures were measured on an A100 and do not transfer to CPU serving.

Provenance

export_manifest.json in this repo records the base revision, adapter SHA-256, exporter commit, library versions, the fold precision, and the base-vs-merged logit parity check.

The adapter is folded in float32 and the result cast to bfloat16. Parity is measured against the bf16 adapter path, so it includes the cast: max absolute logit difference 0.125 over 8 prompts. That is two bfloat16 representable steps at this model's logit magnitudes (spacing is 0.0625 near 14), and the fold itself is exact — in float32 the merged model matches the adapter to 1.8e-5. The tolerance was set to 0.2 for this export for that reason; a broken fold shows up far above it.