mlboydaisuke/LFM2.5-350M-ExecuTorch

LFM2.5-350M — ExecuTorch XNNPACK 8da4w

lfm2_5_350m_xnnpack_8da4w.pte (252.8 MB)

• —Source: LiquidAI/LFM2.5-350M (hybrid conv/attention, 16 layers)
• —License: LFM Open License v1.0
• —Quantization: 8da4w (8-bit dynamic activation / 4-bit weight) + 8-bit embedding (embedding_quantize: "8,0")
• —Export: executorch 1.4.0 export_llm, dynamic shape, maxseqlength 2048, XNNPACK extended_ops
• —Config: llm_params/lfm2_5_350m_xnnpack_8da4w.yaml

Verification (Mac arm64, 2026-08-13)

Generation gate 3/3 via examples/models/llama/runner/native.py (greedy):

Known cosmetic warning: params vocab_size (65536) != tokenizer vocab size.

iPhone 17 Pro (ETBench, XNNPACK CPU, default threads, 2026-08-13)

Outputs correct (Paris; coherent 128-token story).

Usage note — chat template is required. This is an instruct model: raw untemplated text makes it emit <|im_end|> immediately (looks like broken generation but is not). Always wrap prompts as <|startoftext|><|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n, eos ids [7].

Core ML build (Neural Engine, iOS 18+)

lfm2_5_350m_coreml.pte (715 MB, no quantization)

The same model lowered to ExecuTorch's Core ML delegate instead of XNNPACK. This is the first Core ML LLM build on this shelf, because until now the path did not run: a model holding a mutable buffer was handed to the delegate as a buffer, Core ML compiled it into a state, and the runtime — told by take_over_mutable_buffer=False that it has no state — failed at execute with The input feature for layers_7_conv_conv_state must be an MLState. The cause is in tag_constant_data, which decided a buffer was mutated by looking at its direct users; LFM2.5's short-convolution state is written a step after it is read, so it was taken for a constant. Fix in pytorch/executorch#21979.

The `.pte` needs no patched runtime. The fix is export-side; the file below was verified on stock executorch 1.4.0 from pip.

Verification (Mac arm64, macOS 15, 2026-08-21)

Greedy next-token argmax against LiquidAI/LFM2.5-350M in fp32 eager, five prompts, every position after the second counted — 27 of 27 agree:

Decode 88-93 tok/s on the Mac, median of 16 steps after a warm-up of four.

Reset the cache between sequences. The KV cache persists across execute calls, so a second prompt started at position 0 reads the first one's keys. Load a fresh method per sequence. Measuring without this is what made an earlier version of these numbers look like the model was wrong.

Why no quantization

coreml_quantize: c4w builds and runs but loses accuracy: same gate, 21/27 = 0.778, and it misses Jupiter and four. On Qwen3.5-0.8B the same setting scores 3/21. Four-bit weights through this path are not currently shippable, so the Core ML build here is unquantised and correspondingly large. The XNNPACK 8da4w file above remains the small one.