jhu-clsp/mmBERT-decay-data

MMBERT Decay Phase Data

   

Phase 3 of 3: Annealed language learning decay phase (100B tokens) with massive multilingual expansion to 1833 languages.

📊 Data Composition

NOTE: there are multiple decay data mixtures: this mixture described below is the Decay-Cont mixture. However, the data in this repository is the Decay-Eng. If you are interested in the others, please let me know so I can prioritize it.

🌍 Massive Language Coverage

This phase dramatically expands language coverage to 1833 languages, implementing the novel Cascading Annealed Language Learning (ALL) approach:

• —Temperature Schedule: τ=0.3 (most uniform sampling)
• —Low-resource Focus: Includes 1723 new languages with minimal data
• —Rapid Learning: Demonstrates 68% performance improvement on Tigrinya and 26% on Faroese
• —Script Diversity: Covers virtually all writing systems in FineWeb2

Key Innovation: Annealed Language Learning

Rather than training on all languages simultaneously, MMBERT uses a cascading approach:

1. 1.Phase 1: 60 high-resource languages (τ=0.7)
2. 2.Phase 2: 110 languages including mid-resource (τ=0.5)
3. 3.Phase 3: 1833 languages with focus on low-resource (τ=0.3)

This enables rapid learning of new languages while maintaining performance on high-resource ones.

⚙️ Key Features

• —Ultra-low Masking: 5% mask rate for optimal learning efficiency
• —Model Merging: Three decay variants (English-focused, 110-lang, 1833-lang) merged using TIES. This is the English focused version.
• —Quality Focus: Emphasizes highest-quality data sources

🚀 Usage

For decay phase training, see the ModernBERT repo: https://github.com/AnswerDotAI/ModernBERT

Direct Access

Use the script at this link to load any section of the dataset on the fly. This will fail if you try to access too many samples though, due to HF rate-limiting. To download the full dataset, use HF Hub's Snapshot Download.

🎯 Performance Impact

The decay phase demonstrates remarkable efficiency in low-resource language learning:

• —Tigrinya (TiQuAD): 68% improvement (12.1 F1 points) from including the language
• —Faroese (FoQA): 26% improvement (15.4 F1 points)
• —SOTA Performance: Can even outperforms GPT-4o, Gemini 2.5 Pro
• —Rapid Acquisition: Significant gains with only 100B tokens of exposure

🔗 Related Resources

• —Models: mmBERT Model Suite
• —Phase 1: Pre-training Data (2.3T tokens)
• —Phase 2: Mid-training Data (600B tokens)
• —Checkpoints: Training Checkpoints
• —Paper: Arxiv link
• —Code: GitHub Repository

Citation

@misc{marone2025mmbertmodernmultilingualencoder,
      title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning}, 
      author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2509.06888},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.06888}, 
}