benchmarkheaven/weiche-395m

Weiche-395M: a small local decision model for LLM auto-routing

Weiche (German for a railway switch) reads one user turn and decides which kind of model should handle it. It is ModernBERT-large (395M) with typed decision heads, trained for the open-source auto-model-router. One forward pass returns:

It runs on CPU through ONNX Runtime (no PyTorch needed) and in the browser through onnxruntime-web (WebGPU).

Results

All numbers come from held-out data that was never used for training. The eval scripts and raw result JSON files are in eval/ and in the GitHub repo. The baselines are the router's existing decision paths: its heuristic, the local Laya 421M classifier (backend: local) and the hosted Jev API (backend: hosted). Jev was called only for this evaluation. Its outputs were never used for training, calibration or data selection.

The "+ fitted calibration" rows give the trait-only classifiers a per-tier logistic calibration from their traits to P(success). We fitted it on a separate calibration split, which is more help than the router's fixed success curve gives them. Weiche's success heads need no calibration. Routing picks, for each request, argmax P(success_t) − λ·cost_t and sweeps λ, as the router's expected-cost policy does. APGR is the average share of the quality gap between always-small and always-strong that is recovered across 20 cost budgets.

RouterBench held-out (n=1500; tiers Mistral-7B / Mixtral-8x7B / GPT-4-1106)

Always-small quality 28.4 %, always-mid 47.2 %, always-strong 68.6 %. An oracle that knows every outcome matches always-strong quality at 17.1 % of its cost and tops out at 77.2 %.

Measured router-style tasks finished after the data freeze (n=1103; tiers Mistral-Nemo-12B / Gemma-4-31B / DeepSeek-V3.2)

Always-small quality 42.5 %, always-mid 95.0 %, always-strong 94.0 %. An oracle that knows every outcome matches always-strong quality at 38.3 % of its cost and tops out at 98.6 %.

The difficulty rank correlation compares against the averaged rubric labels of two model families. On that measure Weiche and hosted Jev are close (0.739 vs 0.753).

Use it in auto-model-router

policy:
  classifier: {backend: local-route-head, model: benchmarkheaven/weiche-395m, variant: fp16, threads: 2}

variant is fp16 (default, matches fp32 within 1e-3), int4 (423 MB, 96 % category agreement with fp32) or int8 (lossy for this architecture, 92 % agreement; not recommended). The first request downloads the chosen ONNX file once.

Use it directly (ONNX Runtime, Python)

import numpy as np, onnxruntime as ort
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

d = snapshot_download("benchmarkheaven/weiche-395m", allow_patterns=["model_fp16.onnx*", "tokenizer.json", "routerhead.py"])
tok = Tokenizer.from_file(f"{d}/tokenizer.json"); tok.enable_truncation(512)
sess = ort.InferenceSession(f"{d}/model_fp16.onnx", providers=["CPUExecutionProvider"])

def render(request, context=""):   # the training-time input format (see routerhead.py)
    if len(request) > 2400:
        request = request[:1400] + "\n[...]\n" + request[-1000:]
    return f"Context: {context.strip()[:600] or '(new conversation)'}\nRequest: {request}"

ids = np.array([tok.encode(render("Fix the flaky test in tests/test_api.py and run the suite",
                                  "3 messages, 2 from the user. Tools: Bash, Read, Edit.")).ids])
category, scalars, noul, succ, route = sess.run(None, {"input_ids": ids, "attention_mask": np.ones_like(ids),
                                                       "state": np.zeros((1, 14), np.float32)})

PyTorch: encoder/ holds the fine-tuned ModernBERT (HF format) and heads.safetensors holds the heads. routerhead.py defines RouterHead and render:

from transformers import AutoModel, AutoTokenizer
from safetensors.torch import load_file
from routerhead import RouterHead, probs, render
enc = AutoModel.from_pretrained("benchmarkheaven/weiche-395m", subfolder="encoder")
model = RouterHead(enc, 1024).eval()
model.load_state_dict(load_file("heads.safetensors"), strict=False)

Browser: model_int4.onnx ran under onnxruntime-web 1.30 with the WebGPU execution provider (eval/webgpu_smoke_int4.txt). Its outputs matched CPU ONNX Runtime within 1e-4. That check used Chrome's software WebGPU adapter, so it proves operator coverage, not speed. model_fp16.onnx needs a GPU with shader-f16; the software adapter has none, so the fp16 browser path is not yet verified on real hardware.

Training data

We did not use any TypeSafe/Jev output, any sealed benchmark item, or any model response text. The data combines:

• —RouterBench (MMLU, HellaSwag, GSM8K, WinoGrande and MBPP subsets only): measured correctness and cost of 11 LLMs per prompt.
• —EmbedLLM (Apache-2.0): measured correctness of 112 open models per question.
• —RouteLLM gpt4_dataset (Apache-2.0; only UltraChat, Anthropic-HH, FLAN and TruthfulQA prompts): GPT-4-judged adequacy of Mixtral's answer.
• —Chatbot Arena 55k (Apache-2.0): human preference between a weaker and a stronger model.
• —Synthetic router turns (ours, CC-BY-4.0): coding agents, IDE chat, support copilots, batch jobs and more, each with the router's own context summary. DeepSeek-V3.2 and Qwen3.8-27B wrote them. Two model families labelled them blind (author, plus Gemma-4-31B or GLM-5.1 as critic). We keep a row only when both agree on the category; any other field where they disagree is masked.
• —Measured router-style tasks (ours, CC-BY-4.0): procedurally generated requests (maths, dates, code output, logic, tables, strings, money) whose gold answer code computes. We ran each one on Mistral-Nemo-12B, Gemma-4-31B and DeepSeek-V3.2 and recorded correctness, tokens, cost and latency.

We split by a hash of the normalised question text into train 86 %, calib 4 % and test 10 %. The same question can never appear in two splits. The pool owner checked the frozen files against a sealed router benchmark pool (exact match, 8-gram containment and MiniLM cosine) and found zero overlaps. Every source and its licence is listed in DATA-LICENCES.md.

Training: 2 epochs, AdamW (encoder 3e-5, heads 1e-3), batch 32, max 512 tokens, bf16, on one RTX PRO 6000 (33 minutes). Both arms together (this model and a Qwen3-0.6B arm) cost about USD 3.40 of GPU time.

Limits (read before relying on it)

• —In-distribution advantage. The held-out test shares its source distributions with the training data, while Laya and Jev are zero-shot. The router's own 70 held-out tasks are the cleanest out-of-distribution check. There Weiche gets category right 94.9 % of the time (Jev 79.7 %). Its difficulty ranking on those tasks is weak (rank correlation 0.19; Jev 0.30).
• —The success heads predict abstract tiers. Small / mid / strong mean the panels above (7B–13B, 30B–70B-class and GPT-4-class models). With a very different model line-up, use the traits, or recalibrate succ on your own outcomes.
• —The route head is experimental. On the held-out test, exact cost arithmetic over the success heads beat the learned route head (mean regret USD 0.035 vs 0.050 per decision). On post-freeze tasks the route head was better (0.067 vs 0.083). The router computes cache and latency economics exactly, so feeding it succ is the recommended use.
• —English-centric (ModernBERT). Other languages are untested.
• —int8 is lossy for this architecture. Use fp16 or int4.
• —Not a JevBench result and not evaluated on any sealed benchmark.

Licence

Apache-2.0, inherited from ModernBERT-large. The synthetic and measured data we created are CC-BY-4.0. Public datasets keep their own licences (see DATA-LICENCES.md).