Modelpublic

nikitastheo/phonelm-text-eus-eng

sourceHugging Faceupdated 10d agoView on Hugging Face

0likes385downloads

Model Card

nikitastheo/phonelm-text-eus-eng

Trained with src/train_clm/train_causal_lm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

—Seed: 45
—Base config: model_configs/gpt_base_config.json
—Tokenizer: included in this repository
—Max steps: 30000
—Learning rate: 0.0001
—LR scheduler: linear
—Warmup steps: 3000
—Batch size (per device): 32
—Gradient accumulation steps: 1
—Total train batch size: 32
—Language switch step: 10000