mlboydaisuke/Gemma-4-12B-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) --> This model has no row on DeviceMark, the on-device LLM leaderboard. <!-- gen-cards:devicemark end -->

Gemma 4 12B (dense) — Core AI

Apple Core AI (.aimodel) conversion of Google's Gemma 4 12B dense text decoder, ported directly from the QAT release `google/gemma-4-12B-it-qat-q4_0-unquantized`. Decode-only (one token per call), for Apple Silicon (M-series Macs); the runners checked are under Usage.

First Core AI runtime for a ≥16-head × head_dim-512 full-attention model. Gemma 4 12B's full (global) attention layers have a 16-head × 512 Q tensor (16 KB fp16) that overflows MPSGraph's GPU decode scratch heap — the stock SDPA crashes at the first token (apple/coreai-models#27). These bundles ship a custom Metal flash-decode kernel on the full layers that removes the offending op, so the model runs. (The plain non-kernel bundles still crash — these _msdpa bundles are the runnable ones.)

<!-- gen-cards:use-it begin id=gemma-4-12b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->

Use it

New to Core AI? [Start with CoreAIKit 0.7.3](https://github.com/john-rocky/coreai-kit#readme). Follow its requirements and first-run steps for qwen3-0.6b, then open the same release's ChatDemo. The README records the tested OS/SDK and download size; model and device coverage is stated per example.

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let tldr = try await CoreAI.summarize(text, options: .model("gemma-4-12b"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone --branch 0.7.3 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Gemma 4 12B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run -c release chat-cli --model gemma-4-12b --prompt "What can you do, offline?"

Use Xcode build 27A266a from the release's .xcode-pin; adjust the app path if your installation is named differently.

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let id = "gemma-4-12b"
let chat: ChatSession
if id == "qwen3-0.6b" {
    // Freeze the release starter; other selections retain the live catalog's
    // model-specific dispatch (including paired Gemma bundles).
    guard let model = ModelCatalog.builtin.entry(id: id)?.modelID else {
        throw CoreAIKitError.modelNotAvailableOnPlatform(id: id)
    }
    chat = try await ChatSession(model: model)
} else {
    chat = try await ChatSession(catalog: "gemma-4-12b")
}
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

When Apple's FoundationModels built-in model isn't enough, keep your session code and swap the model — one line. CoreAIKit's `KitLanguageModel` plugs supported chat bundles into the system LanguageModelSession. Tool calling depends on the model's dialect; guided generation also requires a compatible sequential engine. Check the linked support matrix before using either capability.

The take-home is `Examples/ChatDemo/Sources/QuickStart.swift` — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• —SPM: https://github.com/john-rocky/coreai-kit (exact 0.7.3) → product CoreAIKit
• —Info.plist: none needed
• —Entitlements: none needed (macOS)
• —First run downloads the model — ~14,698 MB (Mac) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• —Measure in Release — Debug is ~3× slower on per-token host work <!-- gen-cards:use-it end -->

Bundles (gpu-pipelined/)

The _g8 suffix is the higher-occupancy flash-decode kernel (8 SIMD-groups per head split the global layers' KV scan) — it holds throughput at long context (int8 decode 17.5 → 20.3 tok/s at 1024 generated tokens vs the simple kernel) with identical numerics.

int8 is the verified-clean default (its teacher-forced greedy reproduces the fp32 oracle's "The capital of France is Paris." exactly). int4 is the faster / smaller option (16 GB-Mac accessible) at a small quality cost — the same precision class as MLX 4-bit, not a conversion bug (the int8 graph is exact).

Architecture

Clean dense gemma4_unified text decoder — no PLE / AltUp / Laurel / MoE / KV-sharing (unlike the on-device E2B/E4B siblings). 48 layers, hidden 3840, 16 heads, vocab 262144, final logit softcap 30, tied embeddings. 5:1 sliding:full interleave; dual headdim (sliding 256 / full `globalheaddim` 512); full layers use a single global KV head with `attentionkeqv` (value = raw k_proj). Both attention shapes ride one growing KV pair (2 states); the full layers' SDPA runs as a custom Metal flash-decode kernel.

Usage

These bundles are decode-only: each call takes one token, so the prompt has to go in one token at a time. Apple's llm-runner does that with its sequential engine and one-token prefill chunks; with its default engine it stops at "Preparing AI asset" with Shape at dimension 1 of 256 is not a valid substitution for source shape 1. Checked on 2026-10-02 with apple/coreai-models 5de95c6, macOS 27.0 and an M4 Max (the int4 bundle answers "The capital of France is Paris."):

hf download mlboydaisuke/Gemma-4-12B-CoreAI \
    --include "gpu-pipelined/gemma4_12b_qat_decode_int4linsym_msdpa_g8/*" \
    --local-dir ./gemma4-12b-coreai

llm-runner --model ./gemma4-12b-coreai/gpu-pipelined/gemma4_12b_qat_decode_int4linsym_msdpa_g8 \
    --inference-engine-variant coreai-sequential --chunk-size 1 --chunk-threshold 1 \
    --sampling-strategy greedy --prompt "What is the capital of France?" --max-tokens 64

CoreAIKit feeds the prompt this way on its own: ChatSession(bundleAt:) loads a downloaded bundle directory (checked with the int4 bundle and CoreAIKit 0.7.3, same answer).

Each bundle is self-contained: the .aimodel, metadata.json, and the Gemma tokenizer. Both bundles were re-saved on 2026-09-14 (weights unchanged) because the June files no longer load on macOS 27, so download again if your copy is older.

Conversion

Community zoo (recipe, overlays, model card): github.com/john-rocky/coreai-model-zoo → `zoo/gemma4-12b.md`.

License

Gemma 4 is released under the Apache License 2.0 — see the base card google/gemma-4-12B-it-qat-q4_0-unquantized and Google's Gemma 4 license page. The conversion (Core AI bundles, custom Metal kernel) adds no additional restrictions.

<!-- funnel:v1 -->

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->