cowWhySo/toolcall-verifier-classifier-production

Tool-call Verifier Classifier

This document tracks the current tool-call verifier training state for Forge. It is a recovery playbook, not a promotion record. The current published tool-call artifact is telemetry-only and must stay in shadow mode until a replacement passes notebook gates, ONNX parity, release shadow replay, and advisory replay.

The classifier is a DeBERTa sequence-classification sidecar over serialized tool-call contexts. Current published artifacts use serialize_state_v1; new replacement runs should use toolcall-verifier-input/v2 with serialize_state_v2. It runs after deterministic validation: syntax, JSON schema, unknown tools, required steps, prerequisites, unsafe batches, and terminal-tool rules remain Rust-owned and authoritative.

Current Status

The default published tool-call pin was updated to the June 11 high-coverage run (548fb906e65a7061504e5100702834d55e31a02f). This resolves the training distribution failure/regression of b8e292b4de5725250bd1698eb5c795ffcb1a4cde (which had F1 0.681 and valid recall 0.41). The new candidate pin achieves a test macro F1 of 0.9014, valid recall of 0.9824, and wrong_tool_semantic precision of 0.9890. However, it still fails the strict 0.005 false objection promotion gate (obtaining 0.0068), so it should remain in shadow mode.

Labels

Production mode uses six labels:

Raw deterministic labels collapse into deterministic_invalid: invalid_args_schema, missing_required_args, unknown_tool, premature_terminal, missing_prerequisite, unsafe_parallel_batch, and malformed_tool_call.

Current Notebook Settings

These are the current recovery defaults that should be preserved unless a new run gives a concrete reason to change them.

Dataset Mix

Use group-preserving sampling by example_group_id. If a hard negative is included, keep the paired valid/corrected row in the same group so splitting and sampling do not separate the contrastive pair.

Private Generated Dataset

The private generated dataset used for agent_training_hf in the latest run is addenda/forge-eval-3k-v2/agent_training.notebook.jsonl from the Hugging Face repo cowWhySo/forge-toolcall-verifier-openrouter-2650-v1 (revision 01eedcb861324df5fe5b6584ed4f12995b103d0f), containing 724 agent-derived rows:

This legacy dataset is useful as Forge-style valid-call coverage, but it is not strong wrong-tool training evidence. In this run, 246/247 private wrong-tool rows used a literal synthetic_unrelated_tool distractor, so the negative boundary is mostly a name-level shortcut. The latest pasted evaluation showed agent_training_hf accuracy around 0.975, while the large wrong-tool confusions still came from public datasets. Do not infer from that private score that the classifier has learned real wrong-tool semantics.

For the next private addendum, use forge-dataset reviewed rows rather than the legacy distractor dataset. The generator now creates targeted alternatives only from verified-valid captures and reviewer/verifier-accepts them before training:

• —prefer real competing tools from the same observed task group when available;
• —include paired valid rows in the same example_group_id;
• —keep schema-valid arguments for the distractor so the label remains semantic wrong-tool, not deterministic invalid or wrong-argument noise;
• —include bounded repeated-tool (tool_not_needed) and underspecified-request (needs_clarification) alternatives;
• —mine high-confidence reviewed quarantines, such as uv lock requested but make build executed, into paired wrong-argument or wrong-tool examples only after verification accepts them as training rows.

Recommended private capture-review mix for the next OpenRouter addendum:

--review-max-alternatives-per-group 4 \
--review-max-alternative-ratio 0.50

After generation, require forge-dataset validate and split_manifest.json to show nonzero counts for valid, wrong_tool_semantic, wrong_arguments_semantic, tool_not_needed, and needs_clarification before using the addendum in a production notebook run.

Uploaded Eval Files

Use this hard-negative glob:

FORGE_HARD_NEGATIVE_GLOB = "/content/*hard_negatives.jsonl"

The previous glob, /content/*.hard_negatives.jsonl, did not match files named rust_smoke.tool_call_hard_negatives.jsonl or rust_smoke.final_response_hard_negatives.jsonl. A corrected T4 audit showed the hard-negative loader working: forge_hard_negative rows were present, with 7 corrected positives and 6 corrected error-recovery positives.

Telemetry files such as proxy_classifier_budget_8192.jsonl and rust_smoke.jsonl are diagnostics only. Mine them for top-k failures, but do not feed raw top-k telemetry into training or use it as promotion evidence.

Train Rebalance

High-coverage and T4 profiles intentionally use different rebalance behavior. The T4 profile is for cheap diagnosis; it is not promotion evidence.

Non-valid caps remain:

Valid-Protection Slices

Track these slices on validation and test. Apply valid recall and false-objection gates when a slice has at least 25 valid rows.

• —terminal-like tools: respond, summarize, report, submit_*, present, recommend, and diagnose,
• —corrected error-recovery positives,
• —fixed-width numeric string arguments, especially zero-padded values such as 0010,
• —no-op valid calls with empty argument objects.

Promotion Gates

The immediate notebook gates are:

Passing the notebook gates is necessary but not sufficient. Promotion also requires FP32 ONNX parity, shadow release replay, false-objection mining, and a later clean advisory replay.

Since the v5e notebook patch, promotion_gate_report.json is the single source of truth for notebook-side promotability. It carries promotion_status (blocked or promotable_pending_replay, never plain promotable), blocked_reasons[], and artifact_promotable, which are mirrored into artifact_manifest.json, thresholds.json, candidate_thresholds.json, test_metrics.json, and training_run_summary.json; the notebook raises if any exported file claims promotability while the gate report is blocked. The per-eval eval_checkpoint_constrained_promotable flag is checkpoint-selection telemetry only and uses the strict core gates, replacing the old ambiguous eval_constrained_promotable key. The report also embeds diagnostic-only threshold_sweep, confidence_margin_diagnostics, and per_source_diagnostics keys that never block promotion, and the run exports high_confidence_mistakes.jsonl for manual audit of confident wrong predictions.

Lessons Learned

Do Not Threshold Around A Bad Boundary

The current published pin learned a bad boundary: valid calls were pushed into wrong_tool_semantic. Lowering or raising thresholds cannot fix that. Treat that artifact as telemetry-only.

Public Coverage Is The Backbone

The bad high-VRAM setup over-corrected toward private data: private fraction 0.60, private weight 4x, and public-only caps around 6000 rows. That shrunk broad valid/wrong-tool/wrong-argument coverage and collapsed valid-call generalization. Current defaults restore public coverage and keep private rows as a tuning slice.

Hard Negatives Must Stay Paired

Hard negatives without their valid/corrected counterparts teach the classifier to object broadly. Keep pairs together with example_group_id, and evaluate their slices separately.

Numeric Formatting Is Semantic

For the error_recovery smoke tool, {"count":"0010"} is valid and {"count":"10"} is wrong for that schema. This must be trained and evaluated as a semantic argument distinction, not treated as a harmless formatting issue.

T4 Runs Are Diagnostics

T4 runs exposed data-path and balance issues but are not promotion candidates:

The current T4-only rebalance backs off semantic-negative upsampling entirely and keeps extra protected-valid duplication enabled. This is a diagnostic attempt to separate the effects of global valid balance and protected valid support from semantic-negative pressure. Use T4 to iterate on data flow, not to decide promotion. If T4 continues to fail after data-quality fixes, prefer a high-coverage GPU run with a longer context window over more t4_fast ratio chasing.

The openrouter-train-3k result changes the immediate diagnosis. It no longer looks like the model primarily pushes valid calls into wrong_tool_semantic. Instead, it is too permissive on public wrong-tool rows: 755/1139 test wrong_tool_semantic rows were predicted valid, while private agent_training_hf rows were already mostly correct. Fix generated and public wrong-tool evidence before changing gates or thresholds.

The latest auto/t4_proven sidecars also exposed a reporting issue: split balancing produced 25 corrected error-recovery valid rows in both validation and test, but the evaluation slice mask reported zero rows. Slice diagnostics must use the precomputed valid_protection_* columns when present, not only metadata reparsing after JSON dataset reload.

High-Coverage Recovery Is Closer

The best recovery signal so far came from a high-coverage run after public downsampling was disabled:

That candidate still failed the 0.005 false-objection gate and was not promoted. The latest run on 2026-06-11 (detailed in Latest Run Results below) further improved key metrics: valid recall reached 0.9824, wrong_tool_semantic precision reached 0.9890, and valid false objection at 0.90 was reduced to 0.0068. However, it still fails the strict 0.005 false objection promotion gate on the test set.

Quantized ONNX Is A Separate Candidate

A prior quantized parity result had FP32/quantized top-label agreement around 0.342. Quantized output cannot be trusted just because PyTorch or FP32 ONNX looks good. Calibrate thresholds against the artifact that will actually run.

Required parity gates:

If quantized parity fails, write the parity report, stop packaging/upload, and use FP32 ONNX for replay. Publish quantized only as shadow telemetry until parity is fixed.

Final-Response Verifier Is Separate

The final-response verifier is a separate artifact family and is not mature enough for active behavior. A recent runtime replay labeled 302/302 final responses as failed_to_acknowledge_data_gap at low confidence. Keep it shadow-only and document/evaluate it separately.

Latest Run Results (2026-06-11)

The latest high-coverage run on June 11, 2026, was executed with enable_forge_augmentation=True and enable_final_response_verifier=True.

Dataset Statistics

During preprocessing, 33,056 deterministic invalid rows were removed. In addition, 62 rows were quarantined due to source-quality flags (forge_argument_semantic, forge_contrastive_wts, forge_hard_negative, and forge_synthetic), leaving 290,019 rows after quarantine.

After applying group-preserving label caps (max 50,000 per label), the dataset size was reduced to 226,599 rows, preserving all preferred private HF rows.

Capped training rows by source and label:

• —Salesforce/xlam-function-calling-60k: 130,870 rows (valid: 47,237, wrongarguments: 45,568, wrongtool: 37,221, needsclarification: 538, toolnot_needed: 12,844)
• —glaiveai/glaive-function-calling-v2: 48,763 rows (valid: 19,398, wrongarguments: 18,350, wrongtool: 5,314, needsclarification: 237, toolnot_needed: 5,414)
• —Team-ACE/ToolACE: 27,713 rows (valid: 9,486, wrongarguments: 8,926, wrongtool: 7,184, needsclarification: 120, toolnot_needed: 2,017)
• —agent_training_hf: 724 rows (valid: 413, wrongarguments: 38, wrongtool: 32, toolnotneeded: 241)
• —forge_error_recovery_protected: 2,559 rows (valid: 525, wrongarguments: 1,509, wrongtool: 525)
• —forge_fixed_width_numeric: 1,874 rows (valid: 570, wrong_arguments: 1,304)
• —forge_trace: 1,069 rows (valid: 1,051, wrong_arguments: 18)
• —forge_error_recovery_numeric: 419 rows (valid: 60, wrong_arguments: 359)
• —forge_augmented: 100 rows (needs_clarification: 100)

Final split sizes:

• —Train: 190,692 rows (after valid rebalancing duplication factor of 2)
• —Validation: 11,293 rows
• —Test: 22,370 rows

Training Profile

• —Device: NVIDIA RTX PRO 6000 Blackwell Server Edition (95 GB VRAM)
• —Batch Size: 64 (gradient accumulation: 1)
• —Max Sequence Length: 1,280
• —Optimizer: adamw_torch_fused
• —Gradient Checkpointing: Disabled
• —Epochs: 5

Training Progress & Best Checkpoint

The best model checkpoint was saved at step 13384 (end of Epoch 4) based on the gate_deficit_score metric.

Test Evaluation Results

Evaluated on the held-out test split of 22,370 rows:

Test Classification Report

                          precision    recall  f1-score   support

                   valid       0.96      0.98      0.97      7836
     wrong_tool_semantic       0.99      0.97      0.98      4921
wrong_arguments_semantic       0.99      0.98      0.98      7543
         tool_not_needed       1.00      1.00      1.00      1969
     needs_clarification       0.85      0.44      0.58       101
   deterministic_invalid       0.00      0.00      0.00         0

                accuracy                           0.98     22370
               macro avg       0.80      0.73      0.75     22370
            weighted avg       0.98      0.98      0.98     22370

Test Confusion Matrix

Per-Source and Per-Label Accuracies

Per-Source Accuracy:

• —Salesforce/xlam-function-calling-60k: 14,071 rows, Accuracy: 97.73% (Avg Conf: 0.9882)
• —glaiveai/glaive-function-calling-v2: 4,775 rows, Accuracy: 99.52% (Avg Conf: 0.9971)
• —Team-ACE/ToolACE: 2,822 rows, Accuracy: 95.11% (Avg Conf: 0.9642)
• —forge_error_recovery_protected: 257 rows, Accuracy: 100.00% (Avg Conf: 0.9993)
• —forge_fixed_width_numeric: 197 rows, Accuracy: 99.49% (Avg Conf: 0.9977)
• —forge_trace: 152 rows, Accuracy: 99.34% (Avg Conf: 0.9987)
• —agent_training_hf: 48 rows, Accuracy: 85.42% (Avg Conf: 0.9578)
• —forge_error_recovery_numeric: 35 rows, Accuracy: 97.14% (Avg Conf: 0.9783)
• —forge_augmented: 13 rows, Accuracy: 100.00% (Avg Conf: 0.9612)

Per-Label Accuracy:

• —valid: 7,836 rows, Accuracy: 98.24% (Avg Conf: 0.9805)
• —wrong_arguments_semantic: 7,543 rows, Accuracy: 97.95% (Avg Conf: 0.9910)
• —wrong_tool_semantic: 4,921 rows, Accuracy: 97.20% (Avg Conf: 0.9902)
• —tool_not_needed: 1,969 rows, Accuracy: 99.85% (Avg Conf: 0.9996)
• —needs_clarification: 101 rows, Accuracy: 43.56% (Avg Conf: 0.8513)

Guarded-Objection Sweep Details

Valid-call false block rate at different logit thresholds on the test set:

• —@ 0.80: 73 / 7836 = 0.0093
• —@ 0.90: 52 / 7836 = 0.0066
• —@ 0.95: 39 / 7836 = 0.0050
• —@ 0.98: 30 / 7836 = 0.0038
• —@ 0.99: 22 / 7836 = 0.0028

Threshold Policy

The exported default mode is shadow, with default action allow. Thresholds are policy metadata, not proof that enforcement is safe.

Recommended local policy:

{
  "schema_version": "toolcall-verifier-thresholds/v1",
  "mode": "shadow",
  "default_action": "allow",
  "labels": {
    "valid": {
      "action": "allow",
      "advisory_min_confidence": 0.0,
      "enforce_min_confidence": 1.01
    },
    "wrong_tool_semantic": {
      "action": "shadow_only_until_eval_proven",
      "advisory_min_confidence": 1.01,
      "enforce_min_confidence": 1.01
    },
    "wrong_arguments_semantic": {
      "action": "shadow_only_until_eval_proven",
      "advisory_min_confidence": 1.01,
      "enforce_min_confidence": 1.01
    },
    "tool_not_needed": {
      "action": "shadow_only_until_eval_proven",
      "advisory_min_confidence": 1.01,
      "enforce_min_confidence": 1.01
    },
    "needs_clarification": {
      "action": "shadow_only_until_eval_proven",
      "advisory_min_confidence": 1.01,
      "enforce_min_confidence": 1.01
    },
    "deterministic_invalid": {
      "action": "deterministic_only",
      "advisory_min_confidence": 1.01,
      "enforce_min_confidence": 1.01
    }
  }
}

Candidate calibrated thresholds may be recorded for diagnostics, but non-valid active thresholds should remain above 1.0 until shadow replay and advisory replay both pass.

Input Format

The current published classifier expects the canonical serialized format produced by serialize_state_v1. New replacement artifacts should use serialize_state_v2, which keeps the v1 body and appends SCORING_METADATA.

SCHEMA_VERSION:
toolcall-verifier-input/v1

USER_REQUEST:
Generate a sales report from the Q4 2024 dataset.

WORKFLOW_STATE:
required_steps=['fetch_sales_data', 'analyze_sales']
completed_steps=[]
pending_steps=['fetch_sales_data', 'analyze_sales']
terminal_tools=['report']
recent_errors=[]

AVAILABLE_TOOLS:
report: Produce the final report from findings.
PARAMETERS: {"properties": {"summary": {"type": "string"}}, "required": ["summary"], "type": "object"}

fetch_sales_data: Fetch sales data for a given quarter and year.
PARAMETERS: {"properties": {"quarter": {"type": "integer"}, "year": {"type": "integer"}}, "required": ["quarter", "year"], "type": "object"}

analyze_sales: Analyze the loaded sales data and produce findings.
PARAMETERS: {"properties": {}, "type": "object"}

CANDIDATE_CALL:
{"arguments": {"summary": "Done."}, "name": "report"}

Runtime integrations should byte-compare serializer output against serializer_fixture.json before trusting model scores.

Runtime Files

Required artifact files:

model.onnx
labels.json
thresholds.json
candidate_thresholds.json
artifact_manifest.json
input_schema.json
serializer_fixture.json
tokenizer_config.json
special_tokens_map.json
added_tokens.json
spm.model
config.json
training_run_summary.json
test_metrics.json
promotion_gate_report.json
valid_protection_slice_metrics.json
onnx_parity_report.json

model_quantized.onnx may be published only when quantized parity passes. If it does not pass, treat it as telemetry-only and prefer FP32 ONNX for replay.

Rust Deployment Guidance

Recommended integration order:

1. Parse provider response.
2. Validate format, known tool names, and JSON-schema arguments.
3. Enforce required steps, prerequisites, terminal rules, and unsafe batches.
4. If the call is still valid-looking, run the classifier.
5. Shadow mode: log classifier verdict only.
6. Advisory mode: use classifier verdict to choose better nudges.
7. Enforce mode: block only high-confidence semantic labels after eval proof.

Loading failures should fail closed for strict deployment modes. Scoring failures should fail open in shadow and advisory modes, with telemetry.

Promotion Ladder

1. 1.Train replacement.
2. 2.Require good PyTorch validation/test metrics.
3. 3.Require good FP32 ONNX parity.
4. 4.Require good quantized parity, or skip quantized active use.
5. 5.Run release eval in shadow.
6. 6.Mine false objections and top-k disagreement rows.
7. 7.Run advisory replay.
8. 8.Consider enforcement only after advisory replay is clean.

Minimum replay matrix:

no_classifier
classifier_fp32_onnx_shadow
classifier_quantized_onnx_shadow
classifier_fp32_onnx_advisory
classifier_quantized_onnx_advisory

Promotion must show:

• —valid recall at least 0.94,
• —valid false objection at confidence 0.90 at most 0.005,
• —wrong_tool_semantic precision at least 0.90,
• —valid-protection slice gates for any slice with at least 25 valid rows,
• —no regression in terminal-tool workflows,
• —no regression in summarize/report workflows,
• —no regression in fixed-width numeric strings or corrected error-recovery calls,
• —acceptable p95/p99 latency and proxy RSS,
• —stable behavior across real Forge tool schemas, not only public datasets.