LS-ML/Ornith-1.0-35B-ModelOpt-NVFP4-Expert

Ornith 1.0 35B ModelOpt NVFP4 Expert

This repository contains a community ModelOpt NVFP4 experts-only quantization of `deepreinforce-ai/Ornith-1.0-35B`.

This is not an official DeepReinforce release. The source BF16 checkpoint is unchanged and remains available from the upstream repository.

Model Details

The repository name uses Expert, but the quantization target is experts-only: the MoE expert linear weights are quantized while embeddings, lm head, visual modules, attention, linear-attention modules, and shared experts are excluded. See hf_quant_config.json and config.json for the exact ModelOpt quantization metadata.

Benchmark Summary

The checkpoint was benchmarked against the upstream BF16 model on the same DGX Spark / GB10, same vLLM nightly container, same 262K serving profile, same FP8 KV cache allocation, and same benchmark harness.

Performance summary:

Accuracy summary:

Detailed performance and accuracy benchmark tables are included below.

Benchmark Graphics

Performance benchmark:

[image]

Accuracy and quality benchmark:

[image]

Quantization

The checkpoint was produced with NVIDIA ModelOpt from a fused-expert staging copy of the upstream BF16 model. The upstream checkpoint stores split expert tensors; the staging copy fused expert gate/up/down tensors into the layout expected by current Transformers and ModelOpt. The upstream BF16 checkpoint itself was not modified.

Quantization metadata:

The quantization notes used during export are included in QUANTIZATION_NOTES.md.

Serving With vLLM

Validated serving stack:

• —vllm/vllm-openai:nightly
• —ModelOpt quantization loader: --quantization modelopt
• —FlashInfer attention backend
• —Blackwell / GB10 tested with CUTE_DSL_ARCH=sm_121a
• —OpenAI-compatible chat completions
• —Text, image, and Qwen3 XML tool-call smoke tests

Example full-context DGX Spark / GB10 launch:

docker run --rm \
  --name ornith35-nvfp4-vllm \
  --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=32g \
  -p 8000:8000 \
  -v /path/to/models:/models \
  -v ~/.cache:/root/.cache \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e CUTE_DSL_ARCH=sm_121a \
  vllm/vllm-openai:nightly \
  --model /models/Ornith-1.0-35B-ModelOpt-NVFP4-Expert \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name ornith35-nvfp4 \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --kv-cache-memory-bytes 28G \
  --attention-backend flashinfer \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-chunked-prefill

Lower-memory smoke profile:

docker run --rm \
  --name ornith35-nvfp4-vllm-smoke \
  --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=32g \
  -p 8000:8000 \
  -v /path/to/models:/models \
  -v ~/.cache:/root/.cache \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e CUTE_DSL_ARCH=sm_121a \
  vllm/vllm-openai:nightly \
  --model /models/Ornith-1.0-35B-ModelOpt-NVFP4-Expert \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name ornith35-nvfp4 \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --kv-cache-memory-bytes 4G \
  --attention-backend flashinfer \
  --max-model-len 8192 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-chunked-prefill

For non-thinking deterministic chat/eval requests, use:

{
  "temperature": 0,
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}

Validation

The quantized checkpoint was validated on a local DGX Spark / GB10 against the BF16 upstream checkpoint using the same vLLM runtime profile, same maximum context, same FP8 KV cache allocation, and the same benchmark harness.

Performance Benchmark Results

All text rows below used unique prompts, prefix caching disabled, max_tokens=128, and submitted concurrency 1, 2, or 4.

Throughput units are tokens per second. TTFT is max time to first token for the concurrent batch.

The serial 4096 x 4096 image test:

Startup and memory observations:

Accuracy Benchmark Results

Quality runs used deterministic decoding with temperature=0, chat_template_kwargs={"enable_thinking": false}, and paired per-item scoring. The key quantization regression count is "BF16 correct / NVFP4 wrong".

ToolEvalBench version 2.0.7 was run sequentially over 69 standard scenarios:

Interpretation: this NVFP4 export is close to BF16 on broad multiple-choice, multimodal, OCR, and MBPP-style code tasks, but it showed clear regressions on GPQA Diamond and HumanEval+ in this run.

Caveats

• —This is a community quantization, not an official upstream release.
• —The checkpoint has been validated with vLLM ModelOpt loading. Other loaders may not support this ModelOpt NVFP4 format.
• —vLLM marks ModelOpt NVFP4 support as experimental, so revalidate after major vLLM, ModelOpt, CUDA, or FlashInfer changes.
• —The export does not include FP8 KV q/prob scaling factors. When serving with --kv-cache-dtype fp8, vLLM reports that it uses scale 1.0; treat this as a quality caveat for accuracy-sensitive workloads.
• —GPQA used the ungated fingertap/GPQA-Diamond mirror because the official dataset was gated at the time of evaluation.
• —HumanEval+ and MBPP+ used EvalPlus prompts and expanded inputs, but scoring used a local subprocess checker because the official EvalPlus sandbox failed on the local macOS host with resource-limit errors.
• —BFCL v3 10-each should be treated as raw paired signal only; ToolEvalBench is the stronger tool-use benchmark in this report.

Responsible Use

Use this model consistently with the upstream model license and any applicable laws or platform policies. Because this is a quantized derivative, evaluate it for your own target domain before relying on it in production or accuracy-sensitive workflows.

License And Attribution

The upstream model is MIT licensed. This quantized release preserves the MIT license and attribution to deepreinforce-ai/Ornith-1.0-35B.

Upstream model: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B

Quantized by LS-ML.