apolo13x/Qwen3.5-9B-quantized.w4a16

Qwen3.5-9B-quantized.w4a16

This is a quantized version of Qwen/Qwen3.5-9B. This model accepts text and images as inputs and generates text as outputs. The weights were quantized to 4-bit integers (W4A16) using GPTQ via llm-compressor with 512 calibration samples from nvidia/Nemotron-Post-Training-Dataset-v2, reducing the model size from 18.0 GB to 10.7 GB (~1.7x reduction) with effectively lossless accuracy (>100% average recovery).

Quantization Details

• —Scheme: W4A16
• —Calibration: 512 samples (256 reasoning-on + 256 reasoning-off) from Nemotron-Post-Training-Dataset-v2
• —Max sequence length: 4096
• —dampening_frac: 0.01

Inference

This model is supported in vLLM 0.17.0. To serve the model:

vllm serve Kbenkhaled/Qwen3.5-9B-quantized.w4a16 \
    --reasoning-parser qwen3 \
    --enable-prefix-caching

Evaluation

Evaluated with lm-evaluation-harness, 0-shot, thinking mode ON.