inference-optimization/Llama-4-Scout-1.7B-0.4B-Instruct

Llama-4-Scout-1.7B-0.4B-Instruct

This is a tiny version of meta-llama/Llama-4-Scout-17B-16E-Instruct created for testing and development.

Model Details

• —Base Model: meta-llama/Llama-4-Scout-17B-16E-Instruct
• —Architecture: llama4 (multimodal vision-language with MoE)
• —Total Parameters: 1.72B
• —Activated Parameters: ~0.43B (1 expert activated per token out of 4)

Configuration Changes

The following parameters were reduced from the original model:

Architecture Preservation

The tiny model maintains the original Llama-4-Scout architecture patterns:

• —MoE Structure: Retained mixture-of-experts with shared expert
• —Attention Pattern: Maintains the chunkedattention/fullattention pattern (every 4th layer is full_attention)
• —No-RoPE Layers: Preserved the pattern where 3 out of every 4 layers use alternative position encoding

Checkpoint Structure

The model is saved as a single safetensors file following the original checkpoint structure:

• —language_model.model.layers.{X}.feed_forward.experts.*
• —language_model.model.layers.{X}.feed_forward.shared_expert.*
• —vision_model.model.layers.{X}.*

This structure is compatible with transformers' Llama4ForConditionalGeneration.

Usage

from transformers import Llama4ForConditionalGeneration, AutoProcessor

model = Llama4ForConditionalGeneration.from_pretrained(
    "inference-optimization/Llama-4-Scout-1.7B-0.4B-Instruct",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("inference-optimization/Llama-4-Scout-1.7B-0.4B-Instruct")

# Text-only input
text = "Hello, world!"
inputs = processor.tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=20)
print(processor.tokenizer.decode(outputs[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. 1.Config Modification: Reduced layers, experts, and hidden dimensions while preserving architectural patterns
2. 2.Weight Initialization: Randomly initialized weights using the model's init_weights() method
3. 3.Fine-tuning Attempt: Attempted text-only fine-tuning on a small corpus (note: the multimodal architecture made standard text-only fine-tuning ineffective, but the model structure is valid)
4. 4.Validation: Verified model loads correctly and can perform inference

Notes

Important: This is a tiny model with randomly initialized weights intended for testing and development purposes only. It is not trained and will not produce meaningful outputs. The vision tower is completely untrained.

Use Cases

• —Testing model loading and inference pipelines
• —Validating quantization and compression workflows
• —Debugging multimodal model handling
• —CI/CD pipeline testing with realistic model sizes
• —Memory profiling and optimization experiments

Limitations

• —Randomly initialized weights (not trained)
• —Will generate nonsensical outputs
• —Vision capabilities are non-functional
• —Not suitable for any production use or evaluation benchmarks

Technical Warnings

When loading this model, you may see the warning:

[transformers] `rope_parameters`'s high_freq_factor field must be greater than low_freq_factor

This is a known issue with the Llama-4 config and can be safely ignored.