Modelpublic

altic-dev/nemotron-3-diarization-coreml

sourceHugging Faceotherupdated 13d agoView on Hugging Face

0likes908downloads

Model Card

Nemotron 3 Diarization — Core ML

Core ML export of the final `nvidia/Nemotron-3-Diarization` checkpoint (revision 98fcee8e866bb534eee85d1eb3817170700ae717), for on-device speaker diarization on macOS 14+ / iOS 17+.

The license terms of the source checkpoint apply to this derivative.

Files

File	Purpose
`nemotron_3_diarization.mlpackage`	Streaming Sortformer step: pre-encoder, Fast-Conformer, transformer, speaker heads
`learnable_sil_emb.f32`	512 raw little-endian float32 values: the trained silence embedding for the speaker cache
`config.json`	Streaming geometry, cache and post-processing settings, I/O shapes, SHA-256 of every file
`conversion/`	Scripts that produced the package

Precision: fp16 storage, fp32 compute

Weights are stored as fp16 and widened to fp32 at load time (constexpr_cast), so every op computes in fp32. The package stays at fp16 size (~199 MB).

—The checkpoint is bf16, which fp16 represents almost exactly (0.005% of values differ, by less than 3e-8).
—A plain fp16-compute export drifts on long audio: small per-chunk rounding feeds back through the speaker cache and compounds. On a 36-minute meeting it matched the reference on only 98.4–98.9% of frames and added a speaker. This happened on GPU, CPU and Neural Engine alike.

Validation

Frame-level agreement with NVIDIA's NeMo reference, same streaming loop and post-processing, on real meeting recordings:

Audio	NeMo speakers	Core ML speakers	Frame agreement
70 s, one microphone	2	2	100.00%
36 min, microphone track	4	4	99.94%
36 min, application-audio track	5	5	100.00%

Speed: ~21 ms per 27.2 s chunk on Apple silicon (about 2 s for a 36-minute track).

Interface

One call processes one chunk; the caller owns the speaker cache and FIFO update between calls (NeMo streaming_update).

Input	Shape	Type
`chunk`	1 × 3040 × 128	float32 log-mel, 10 ms hop
`chunk_lengths`	1	int32
`spkcache`	1 × 264 × 512	float32
`spkcache_lengths`	1	int32
`fifo`	1 × 40 × 512	float32
`fifo_lengths`	1	int32

Output	Shape
`spkcache_fifo_chunk_preds`	1 × 684 × 8
`chunk_pre_encode_embs`	1 × 380 × 512
`chunk_pre_encode_lengths`	1

Required runtime settings

These must match training, or the model splits one voice across several speaker slots:

—spkcache_sil_frames_per_spk = 1
—Fill silence slots in the speaker cache with learnable_sil_emb.f32, not a running mean of silent frames.
—Offline profile: chunk_len 340, chunk_right_context 40, fifo_len 40, spkcache_len 264, spkcache_update_period 300.