leibnitz-lab/cosmopedia-distributed-training

Cosmopedia distributed training checkpoints

Two 98.4 M parameter language models trained on Cosmopedia for 10 epochs, differing only in the psi hyperparameter.

The pair isolates the effect of psi: parameter count, data, schedule and learning rate are identical across the two, so a difference between them is attributable to that setting alone.

Loading

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    "leibnitz-lab/cosmopedia-distributed-training",
    filename="cosmopedia_m98405376_psi25_lr0-0003_distributed_epoch10.pt",
)
state = torch.load(path, map_location="cpu")

These are raw PyTorch state dicts from a custom distributed trainer, not transformers models.