Team Ai
Modelpublic

mlboydaisuke/Nemotron-3-Nano-4B-CoreAI

sourceHugging Faceotherupdated 16d agoView on Hugging Face
0likes8.9kdownloads
Model Card

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) --> ![DeviceMark](https://devicemark.github.io/) Measured decode — iPhone 17 Pro: 14.7 tok/s · Mac (M4 Max): 100 tok/s (DeviceMark row nemotron-4b, int8hu bundle · data) <!-- gen-cards:devicemark end -->

Nemotron-3-Nano-4B — Core AI

Nemotron-3 Nano 4B (hybrid Mamba-2 / Transformer, NVIDIA) running on Apple Core AI — measured 16.0 tok/s decode on iPhone 17 Pro and 85.2 tok/s on an M4 Max GPU, greedy output token-identical to fp32. Same-model llama.cpp / MLX numbers on the M4 Max (88.4 / 176.8 tok/s, 4-bit): apple-silicon-llm-bench, results/hybrid.

Decode-only (S=1) Core AI bundles for NVIDIA's Mamba2 + attention + MLP hybrid (42 blocks: 21 Mamba2 / 17 MLP / 4 GQA NoPE attention), int8 weights with an absmax int8 head. No custom Metal kernel — at S=1 the selective scan is a single recurrence step, so the graph is loop-free.

variantassetfor
gpu-pipelined/nemotron_3_nano_4b_decode_int8hu.aimodelMac (JIT specialization)
ios-h18p/nemotron_3_nano_4b_decode_int8hu.h18p.aimodelciPhone (AOT — a 4B graph cannot specialize on-device)

<!-- gen-cards:use-it begin id=nemotron-3-nano-4b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->

Use it

New to Core AI? [Start with CoreAIKit 0.7.3](https://github.com/john-rocky/coreai-kit#readme). Follow its requirements and first-run steps for qwen3-0.6b, then open the same release's ChatDemo. The README records the tested OS/SDK and download size; model and device coverage is stated per example.

Measured decode — iPhone 17 Pro: 14.7 tok/s · Mac (M4 Max): 100 tok/s (DeviceMark row nemotron-4b, int8hu bundle · data)

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

swift
let tldr = try await CoreAI.summarize(text, options: .model("nemotron-3-nano-4b"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

bash
git clone --branch 0.7.3 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Nemotron-3-Nano 4B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run -c release chat-cli --model nemotron-3-nano-4b --prompt "What can you do, offline?"

Use Xcode build 27A266a from the release's .xcode-pin; adjust the app path if your installation is named differently.

💻 Build with it — complete; the glue is kit API, copy-paste runs:

swift
import CoreAIKit

let id = "nemotron-3-nano-4b"
let chat: ChatSession
if id == "qwen3-0.6b" {
    // Freeze the release starter; other selections retain the live catalog's
    // model-specific dispatch (including paired Gemma bundles).
    guard let model = ModelCatalog.builtin.entry(id: id)?.modelID else {
        throw CoreAIKitError.modelNotAvailableOnPlatform(id: id)
    }
    chat = try await ChatSession(model: model)
} else {
    chat = try await ChatSession(catalog: "nemotron-3-nano-4b")
}
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is `Examples/ChatDemo/Sources/QuickStart.swift` — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

  • —SPM: https://github.com/john-rocky/coreai-kit (exact 0.7.3) → product CoreAIKit
  • —Info.plist: none needed
  • —Entitlements: iOS needs com.apple.developer.kernel.increased-memory-limit — a 4.3 GB bundle is past the default jetsam limit
  • —First run downloads the model — ~4,626 MB (Mac) / ~4,629 MB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
  • —Measure in Release — Debug is ~3× slower on per-token host work <!-- gen-cards:use-it end -->

Measured: 16.0 tok/s decode on an iPhone 17 Pro (cooled, AOT h18p, bandwidth-saturated) and 85.2 tok/s on an M4 Max GPU. Greedy output is token-identical to the fp32 transformers rollout on the probe prompts.

Requires COREAI_CHUNK_THRESHOLD=1 (S=1 bundle) and an engine that carries two extra fixed-shape states (the Mamba conv columns + SSM state) alongside the KV cache.

Port + recipe: coreai-model-zoo / nemotron-3-nano

<!-- funnel:v1 -->

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->