Modelpublic

jacquelinehe/tinycomma-1.8b-llama3-tokenizer

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face

2likes607downloads

Model Card

TinyComma 1.8B

TinyComma 1.8B is a 1.8B parameter, decoder-only base LM trained entirely on permissively licensed data from the Common Pile. Different from the official Comma model series, TinyComma 1.8B uses the 128K-vocabulary Llama3 tokenizer to ensure compatibility with two-model decoding setups. We trained TinyComma 1.8B to support our research on inference-time copyright mitigation.

—Paper: Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
—Repository: jacqueline-he/anchored-decoding
—Project Page: Interactive Demo

Benchmarking TinyComma 1.8B

We benchmarked TinyComma 1.8B and several other permissively trained base models on several common natural language understanding tasks from the OLMES evaluation suite.

<p align="center"> <img src="https://huggingface.co/datasets/jacquelinehe/tinycomma-assets/resolve/main/pretraining_benchmark.png" width="800"><br> <em>Benchmarking results using OLMES. TinyComma 1.8B outperforms other models of its size range.</em> </p>

Training details

We trained TinyComma 1.8B using the lingua training framework. Pre-training consists of two stages: (1) a 156B-token generation training stage over the entire Common Pile, following original domain weights specified by Kandpal et al., 2025, and (2) a 13.5B-token cooldown stage on a weighted mixture of three high-quality domains (70% Wikimedia, 15% DOAB, and 15% Data Provenance Initiative data). Our hardware is a single node of 8 140 GiB H200 GPUs. Model configuration and pre-training hyperparameter details are below:

<div style="text-align: center;"> <table style="margin: 0 auto;"> <caption>TinyComma 1.8B model configuration.</caption> <thead> <tr> <th>Params</th> <th>Head Dim.</th> <th>Hidden Size</th> <th>Attn. Heads</th> <th>Hidden Layers</th> <th>KV Heads</th> </tr> </thead> <tbody> <tr> <td>1,758,562,304</td> <td>64</td> <td>2048</td> <td>32</td> <td>24</td> <td>32</td> </tr> </tbody> </table> </div> <br><br> <div style="text-align: center;"> <table style="margin: 0 auto;"> <caption>TinyComma 1.8B pretraining configuration.</caption> <thead> <tr> <th>Hyperparameters</th> <th>Values</th> </tr> </thead> <tbody> <tr> <td>Optimizer</td> <td>AdamW (<i>β</i><sub>1</sub>=0.9, <i>β</i><sub>2</sub>=0.95)</td> </tr> <tr> <td>Learning rate</td> <td>3e<sup>−3</sup> for Stage 1, 1e<sup>−3</sup> for Stage 2</td> </tr> <tr> <td>Weight decay</td> <td>0.033 for Stage 1</td> </tr> <tr> <td>Batch size</td> <td>4M tokens</td> </tr> <tr> <td>Warmup</td> <td>1000 steps for Stage 1, none for Stage 2</td> </tr> <tr> <td>Schedule</td> <td>Cosine schedule for Stage 1, linear schedule for Stage 2</td> </tr> <tr> <td>Sequence length</td> <td>Pack to 2048 tokens</td> </tr> </tbody> </table> </div>

Citation

bibtex

@article{he2026anchored,
  title={{Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model}},
  author={Jacqueline He and Jonathan Hayase and Wen-tau Yih and Sewoong Oh and Luke Zettlemoyer and Pang Wei Koh},
  journal={arXiv preprint},
  year={2026}
}