Modelpublic

RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8

sourceHugging Faceapache-2.0updated 4d agoView on Hugging Face

0likes450downloads

Model Card

<h1 align: center; style="display: flex; align-items: center; gap: 10px; margin: 0;"> Qwen3.5-35B-A3B-quantized.w8a8 </h1>

Model Overview

—Model Architecture: Qwen/Qwen3.5-35B-A3B
—Input: Text, Image
—Output: Text
—Model Optimizations:
—Weight quantization: Int8
—Activation quantization: Int8

This model is a quantized version of Qwen/Qwen3.5-35B-A3B. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.

Model Optimizations

This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-35B-A3B to Int8 data type, ready for inference with vLLM.

This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.

Only the weights and activations of the linear operators are quantized using LLM Compressor. Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, token embeddings and MTP layers are kept in original precision.

Deployment

Use with vLLM

This model can be deployed efficiently using vLLM.

1.Text-Only: Skip the vision encoder to free up memory for additional KV cache:

vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --language-model-only

1.Multimodal (Text + Image): Serve with full vision support:

vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3

Image input is supported by the upstream architecture but was not evaluated for this release.

1.Tool Call: Enable tool use support:

vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

Tool calling was not evaluated for this release.

1.Multi-Token Prediction (MTP): For speculative decoding:

vllm serve RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'

Send requests to the server:

python

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8"

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

The checkpoint retains native MTP weights in BF16. This speculative-decoding configuration was not tested with the quantized checkpoint.

Creation

This model was created by applying LLM Compressor with w8a8 integer quantization using the GPTQModifier calibrating on the ultrachat dataset, as presented in the code snippet below.

python

from pathlib import Path

from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.utils import load_context

MODEL_ID = "Qwen/Qwen3.5-35B-A3B"
DATASET = "ultrachat"
SPLIT = "train_sft[:512]"
save_dir = Path("gptq-ultrachat")
save_dir.parent.mkdir(parents=True, exist_ok=True)

with load_context(Qwen3_5MoeForConditionalGeneration):
    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
        MODEL_ID,
        device_map="auto_offload",
    )
processor = AutoProcessor.from_pretrained(MODEL_ID)

recipe = GPTQModifier(
    targets="Linear",
    scheme="W8A8",
    ignore=[
        "lm_head",
        "re:.*mlp.gate$",
        "re:.*mlp.shared_expert_gate.*",
        "re:.*norm.*",
        "re:.*embed_tokens.*",
        "re:.*visual.*",
        "re:.*conv1d.*",
    ],
)

oneshot(
    model=model,
    processor=processor,
    dataset=DATASET,
    splits=SPLIT,
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
    moe_calibrate_all_experts=True,
)

model.save_pretrained(str(save_dir), save_compressed=True)
processor.save_pretrained(str(save_dir))
print(f"Saved GPTQ UltraChat checkpoint to {save_dir}")

</details>

Evaluation

The BF16 reference and this GPTQ W8A8 checkpoint were served with vLLM in --language-model-only mode. Accuracy scores are means over seeds 1234, 2345, and 3456. Recovery is the quantized score divided by the BF16 score.

Accuracy

<table> <thead> <tr> <th>Category</th> <th>Benchmark</th> <th>Qwen/Qwen3.5-35B-A3B (BF16)</th> <th>RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8</th> <th>Recovery</th> </tr> </thead> <tbody> <tr> <td rowspan="4"><b>Reasoning</b></td> <td>GSM8K Platinum, strict match</td> <td>95.0648%</td> <td>94.8167%</td> <td>99.74%</td> </tr> <tr> <td>Math500</td> <td>84.8667%</td> <td>84.9333%</td> <td>100.08%</td> </tr> <tr> <td>GPQA Diamond</td> <td>84.1751%</td> <td>83.6700%</td> <td>99.40%</td> </tr> <tr> <td>MMLU COT Llama</td> <td>89.1112%</td> <td>88.7552%</td> <td>99.60%</td> </tr> <tr> <td rowspan="2"><b>Instruction Following</b></td> <td>IFEval prompt-level strict</td> <td>90.5730%</td> <td>90.3882%</td> <td>99.80%</td> </tr> <tr> <td>IFEval instruction-level strict</td> <td>93.1655%</td> <td>92.8457%</td> <td>99.66%</td> </tr> </tbody> </table>

WikiText-2 perplexity

Perplexity was measured on the full 62-document test set from the registered wikitext task. Lower is better.

Metric	Qwen/Qwen3.5-35B-A3B (BF16)	RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8	Change
Word perplexity	7.46346	7.50737	+0.5883%
Byte perplexity	1.45628	1.45788	+0.1100%
Bits per byte	0.542285	0.543868	+0.2921%

Reproduction

The accuracy evaluations used the evaluation frameworks lm-evaluation-harness and Lighteval. GSM8K Platinum and IFEval used their registered 0-shot prompts; Math500 and GPQA Diamond used the registered Lighteval tasks. MMLU used mmlu_cot_llama with 5-shot examples formatted as multi-turn chat and a maximum of 8,192 generated tokens. All accuracy tasks used temperature 1.0, top-p 0.95, top-k 20, and presence penalty 1.5.

The checkpoint was served with vLLM 0.30.1rc1.dev674+g87954c5b0, tensor parallel size 2, the Qwen 3 reasoning parser, and text-only mode.

<details> <summary>Evaluation commands and settings</summary>

vLLM server

bash

vllm serve "$MODEL_ID" \
  --served-model-name "$SERVED_MODEL" \
  --port "$PORT" \
  --tensor-parallel-size 2 \
  --max-model-len 69632 \
  --reasoning-parser qwen3 \
  --language-model-only

The same serving and evaluation settings were used for the BF16 reference by replacing the model ID with Qwen/Qwen3.5-35B-A3B.

IFEval strict (`lm_eval`, 0-shot)

bash

lm_eval --model local-chat-completions \
  --tasks ifeval \
  --model_args "model=${SERVED_MODEL},max_length=69632,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path "./eval/ifeval_seed${SEED}.json" \
  --seed "$SEED" \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=32000"

MMLU COT Llama (`lm_eval`, 5-shot)

Restart vLLM with a 20,480-token context for this task, then run:

bash

for SEED in 1234 2345 3456; do
  lm_eval --model local-chat-completions \
    --tasks mmlu_cot_llama \
    --model_args "model=${SERVED_MODEL},max_length=20480,base_url=${API_BASE}/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \
    --num_fewshot 5 \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --output_path "./eval/mmlu_cot_llama_seed${SEED}.json" \
    --seed "$SEED" \
    --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=20,presence_penalty=1.5,seed=${SEED},max_gen_toks=8192"
done

Math500 and GPQA Diamond (`lighteval`, 0-shot)

bash

mkdir -p ./eval
for SEED in 1234 2345 3456; do
  CONFIG="./eval/lighteval_seed${SEED}.yaml"
  cat > "$CONFIG" <<EOF
model_parameters:
  provider: hosted_vllm
  model_name: hosted_vllm/${SERVED_MODEL}
  base_url: ${API_BASE}/v1
  api_key: ""
  timeout: 3600
  concurrent_requests: 64
  generation_parameters:
    max_new_tokens: 65536
    presence_penalty: 1.5
    seed: ${SEED}
    temperature: 1.0
    top_k: 20
    top_p: 0.95
EOF

  lighteval endpoint litellm "$CONFIG" 'math_500|0' \
    --output-dir "./eval/math500_seed${SEED}" --save-details
  lighteval endpoint litellm "$CONFIG" 'gpqa:diamond|0' \
    --output-dir "./eval/gpqa_diamond_seed${SEED}" --save-details
done

WikiText-2 perplexity

The PPL task used lm-eval's local-completions model against the vLLM /v1/completions endpoint, with the Qwen3.5 tokenizer, max_length=20480, num_concurrent=64, max_retries=3, and timeout=3600. It used a separate server configured for a 20,480-token context:

bash

lm_eval --model local-completions \
  --tasks wikitext \
  --model_args 'model=RedHatAI/Qwen3.5-35B-A3B-quantized.w8a8,max_length=20480,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=Qwen/Qwen3.5-35B-A3B,tokenizer_backend=huggingface,timeout=3600' \
  --num_fewshot 0 \
  --seed 1234 \
  --output_path ./eval/wikitext2_ppl/results

</details>

Model Overview

Model Optimizations

Deployment

Use with vLLM

Creation

Evaluation

Accuracy

WikiText-2 perplexity

Reproduction

vLLM server

IFEval strict (lm_eval, 0-shot)

MMLU COT Llama (lm_eval, 5-shot)

Math500 and GPQA Diamond (lighteval, 0-shot)

WikiText-2 perplexity

IFEval strict (`lm_eval`, 0-shot)

MMLU COT Llama (`lm_eval`, 5-shot)

Math500 and GPQA Diamond (`lighteval`, 0-shot)