FluidInference/intern-decision-0.8b-coreml

Intern-Decision-0.8B for Core ML

Core ML export of internlm/Intern-Decision-0.8B (Shanghai AI Laboratory, Apache-2.0, snapshot 85a0cc5a): a typed-decision model fine-tuned from Qwen3.5-0.8B that answers a set of choice / score / noul (yes/no) questions about a JSON state in one prefill pass. The text path is exported here; the vision tower is not (text-only requests).

Files

Inputs: hidden [1, L, 1024] (embedding rows of the right-padded prompt), cos / sin [L, 64] (RoPE tables for positions 0…L−1), field_onehot [F, L] (row i selects the token before field i's <decision>). Output: logits [F, 62] over the answer symbols; take the first n entries for a field with n options, softmax, divide log-probs by the temperature. fp16, GPU (cpuAndGPU), iOS 17 / macOS 14. Swift runtime: InternDecisionManager in FluidUse; conversion scripts in mobius (models/computer-use/intern-decision-0.8b/coreml).

What the package computes

Intern-Decision renders the request as one chat prompt: a fixed system prompt, a user turn with the state as JSON and the decision schema (one line per field, its options mapped to the answer symbols A–Z, a–z, 0–9), and an assistant turn that is a JSON skeleton with one <decision> token per field. The logits at the position immediately before each marker, restricted to that field's first n symbols, are its answer; the published temperature (2.7478 for 0.8B) rescales the restricted softmax without changing the argmax. Nothing is generated.

DecisionRow (decision_export.py) is one fixed-length request: the Qwen3.5 decoder from qwen35_export.py (the Kev-0.8B / Cua-S1-4B export), the final norm at the pre-marker positions selected with a one-hot map per field, and the 62 tied-embedding rows of the answer symbols. Token embeddings are gathered on the host (embeddings.f16); the host applies the restricted softmax and temperature. Prompt compilation and the chat template come from the checkpoint's own inference.py and tokenizer, so the token stream is the reference's.

The fixed prompt (system prompt, headings, skeleton) is about 250 tokens, so the smallest three-field request is 319 tokens; 40% of Jevbench-Hard exceeds 1,024 tokens (max 4,074).

Fidelity

Reference: the checkpoint's DecisionEngine in fp32 on the Apple GPU (MPS), probabilities after temperature scaling, on the bundled accuracy suites from the Intern-Decision repo (benchmarks/accuracy-v1, shuffled with seed 0). The fp32 PyTorch wrapper matches the reference to 3.6e-6 (50 Typed Decision fields).

Reference and Core ML accuracy against the suite labels are identical on every subset (reports in the mobius directory). int4 per-block compression needs an iOS 18 deployment target and was not built.

Latency

One request, 319 input tokens, three fields (choice, yes/no, score), the model card's RTX 4090 shape. M5 Pro (24 GB), warmed, bench.py; Core ML on CPU_AND_GPU, times include the host embedding gather and RoPE tables.

The pass is compute-bound and scales with the bucket, not the request, so ship the smallest bucket that fits. int8 weights leave GPU time unchanged (88 ms at L512_F8) and halve the package (955 → 480 MB). ComputeUnit.ALL matches CPU_AND_GPU: the Gated DeltaNet backbone does not run on the Neural Engine (see the Kev-0.8B notes). The model card reports 34 ms for this request on an RTX 4090.