motionlabs/random-llama-small

Random-Llama-Small

Model Overview

Random-Llama-Small is a randomly initialized transformer-based language model with approximately 2 billion parameters, built using the LLaMA architecture. It is designed for research purposes, providing a starting point for pretraining or fine-tuning on custom datasets. The model uses the tokenizer from HuggingFaceTB/SmolLM2-1.7B-Instruct and is configured for causal language modeling. As a randomly initialized model, it produces incoherent outputs until trained, making it ideal for researchers studying transformer training dynamics or developing custom language models.

Key Details

• —Architecture: LLaMA (Causal Language Model)
• —Parameters: ~2B
• —Hidden Size: 2304
• —Layers: 22
• —Attention Heads: 36 (with 9 key-value heads for grouped-query attention)
• —Intermediate Size: 9216
• —Vocabulary Size: 128256
• —Tokenizer: Imported from HuggingFaceTB/SmolLM2-1.7B-Instruct
• —Precision: bfloat16
• —Max Context Length: 131,072 tokens (with RoPE scaling)
• —License: MIT

LLaMA Architecture

The LLaMA architecture, developed by Meta AI, is a family of efficient transformer-based models optimized for research. Random-Llama-Small follows this design, incorporating several key features:

Core Components

• —Decoder-Only Transformer: Predicts the next token in a sequence based on prior tokens, suitable for autoregressive tasks like text generation.
• —Grouped-Query Attention (GQA): 36 attention heads with only 9 key-value heads, improving efficiency and reducing memory/compute cost.
• —Rotary Position Embeddings (RoPE): Embeds positional information with scaling, enabling a context length of up to 131,072 tokens.
• —Swiglu Activation: Uses SiLU (Swish) activation in the FFN for improved expressiveness.
• —RMSNorm: Root Mean Square Layer Normalization replaces LayerNorm for stability and faster convergence.
• —Tied Embeddings: Input and output embeddings share weights (tie_word_embeddings=True), reducing parameter count by ~295M.

Benefits of LLaMA Architecture

• —Efficiency: High throughput, low memory use.
• —Scalability: Works well across model sizes.
• —Flexibility: Long-context support and task adaptability.
• —Research-Friendly: Great for exploring attention, positional encoding, and training dynamics.

Random-Llama-Small Specifics

This model uses random weights and:

• —Has ~2B parameters across 22 layers.
• —Uses a 2304 hidden size and 9216 FFN size.
• —Supports 128K+ vocab tokens and bfloat16 precision.
• —Supports extended context lengths of 131,072 tokens.

Intended Use

• —Research on transformer dynamics, optimization, or architectural changes.
• —Baseline for pretraining or task-specific fine-tuning.
• —Experimentation with scaling laws or custom architectures.

Out-of-Scope Use

• —Not for direct production deployment.
• —Not suitable for tasks needing coherence or accuracy without training.

Usage

Requirements

• —transformers >= 4.45.0
• —torch >= 2.0
• —GPU with ≥ 6GB VRAM (24GB+ for training)

Inference Example

# Use a pipeline as a high-level helper
from transformers import pipeline

messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe = pipeline("text-generation", model="reflex-ai/random-llama-small")
print(pipe(messages))

Note: Outputs will be random and incoherent due to the model’s untrained state.

Training Example

from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling, LlamaForCausalLM, AutoTokenizer

model = LlamaForCausalLM.from_pretrained("your_username/random-llama-small")
tokenizer = AutoTokenizer.from_pretrained("your_username/random-llama-small")

training_args = TrainingArguments(
    output_dir="./random_llama_small_finetuned",
    per_device_train_batch_size=4,
    num_train_epochs=3,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=your_dataset,
    data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
)

trainer.train()

Limitations

• —Random Initialization: Needs significant training to be useful.
• —Resource Intensive: High computational cost.
• —No Pretraining Data: Users must provide their own.
• —Tokenizer Constraint: May not suit all domains.

Benefits and Potential

• —Customizability: A blank slate for full control of objectives and data.
• —Research Insights: Ideal for understanding early-stage LLM behavior.
• —Scalable Baseline: Balances size and research feasibility.
• —Extended Context: Useful for long-form tasks post-training.

Model Configuration

{
  "architectures": ["LlamaForCausalLM"],
  "hidden_size": 2304,
  "num_hidden_layers": 22,
  "num_attention_heads": 36,
  "num_key_value_heads": 9,
  "intermediate_size": 9216,
  "vocab_size": 128256,
  "max_position_embeddings": 131072,
  "rope_scaling": {
    "factor": 32.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "torch_dtype": "bfloat16",
  "tie_word_embeddings": true
}

Ethical Considerations

• —Untrained Safety: No immediate harmful outputs, but ethics matter during training.
• —Environmental Impact: Large-scale training consumes energy; optimize and use green compute.
• —Accessibility: Resource requirements may limit use by smaller research teams.

Contact

For questions or issues, please open an issue on the Hugging Face repository.

Model card created on April 20, 2025.