Team Ai
Datasetpublic

JustACluelessKidAtSchool/tiny-slm-pretraining-corpus

🚀 Ultra High-Quality Tiny SLM Pre-Training Corpus (<100GB) A state-of-the-art, balanced 7-domain pre-training dataset engineered specifically for Small Language Models (Tiny SLMs: 50M – 2B parameters) such as SmolLM2, SmolLM3, MobileLLM, Llama 3.2 1B, and custom architectures. 100% compatible with Unsloth Studio, Unsloth AI, Hugging Face datasets, and PyTorch DataLoaders. 📊 Dataset Statistics Total Documents: 20,066,075 Train: 19,663,898 Validation: 402,177… See the full description on the dataset page: https://huggingface.co/datasets/JustACluelessKidAtSchool/tiny-slm-pretraining-corpus.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
0likes22downloads
Dataset Card

🚀 Ultra High-Quality Tiny SLM Pre-Training Corpus (<100GB)

A state-of-the-art, balanced 7-domain pre-training dataset engineered specifically for Small Language Models (Tiny SLMs: 50M – 2B parameters) such as SmolLM2, SmolLM3, MobileLLM, Llama 3.2 1B, and custom architectures.

100% compatible with Unsloth Studio, Unsloth AI, Hugging Face `datasets`, and PyTorch DataLoaders.

📊 Dataset Statistics

  • —Total Documents: 20,066,075
  • —Train: 19,663,898
  • —Validation: 402,177
  • —Total Tokens: ~23.87 Billion tokens
  • —Train Tokens: ~23.40B
  • —Validation Tokens: ~0.47B
  • —Format: Zstandard Compressed Parquet

🔬 7-Domain Pre-Training Mixture

DomainSource RepositoryDescription & Quality FilterToken Share
Synthetic TextbooksHuggingFaceTB/smollm-corpuscosmopedia-v2 textbooks and topic deep-dives28.0%
Educational WebHuggingFaceFW/fineweb-eduSample 10BT (score >= 2.8 filter)23.5%
Step-by-Step MathHuggingFaceTB/finemathfinemath-4plus & infiwebmath-4plus (score 4+)19.5%
Clean Polyglot Codecodeparrot/github-code-cleanPython, Rust, C++, C, Go, JS, TS, Java, SQL, Shell12.5%
General Knowledgewikimedia/wikipedia + mlfoundations/dclm-baseline-1.0-parquetEnglish Wikipedia (dense factual) & DCLM web corpus7.8%
Reasoning & Tool TracesHuggingFaceTB/smoltalk2OpenThoughts3, Nemotron R1, and agentic tool traces6.5%
Stories & Narrativeroneneldan/TinyStoriesSynthetic coherent narrative, dialogue, & vocabulary2.1%

🦥 Quickstart: Unsloth Studio & Hugging Face

1. Load with Hugging Face Datasets

python
from datasets import load_dataset

# Load Train Split
train_dataset = load_dataset("JustACluelessKidAtSchool/tiny-slm-pretraining-corpus", split="train")

# Load Validation Split
val_dataset = load_dataset("JustACluelessKidAtSchool/tiny-slm-pretraining-corpus", split="validation")

2. Pre-Train with Unsloth in 1 Command

python
from unsloth import FastLanguageModel
from datasets import load_dataset
from transformers import TrainingArguments
from trl import SFTTrainer

# Load Tiny Model (e.g. SmolLM2-135M or Llama-3.2-1B)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="HuggingFaceTB/SmolLM2-135M",
    max_seq_length=2048,
    load_in_4bit=False,
)

# Load Dataset from Hub
dataset = load_dataset("JustACluelessKidAtSchool/tiny-slm-pretraining-corpus", split="train")

# Train with Unsloth
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        output_dir="./tiny_slm_output",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=8,
        max_steps=5000,
        learning_rate=5e-4,
        fp16=True,
        optim="adamw_8bit",
    ),
)
trainer.train()

3. In Unsloth Studio GUI

  1. 1.Set Dataset to: JustACluelessKidAtSchool/tiny-slm-pretraining-corpus
  2. 2.Text Column: text