gpt2
Datasets
All datasets matching “gpt2”fineweb10B-gpt2
fineweb10B-gpt2
This repo contains the GPT-2 tokens for fineweb10B, just as would be generated by https://github.com/KellerJordan/modded-nanogpt/tree/master (or llm.c).
You can download from this repo instead of re-tokenizing to save a couple hours of setup on a new machine.
fineweb100B-gpt2fineweb-edu-gpt2fineweb-edu-100B-gpt2-token-shardsFineWeb Edu 100B dataset tokenized with GPT-2 tokenizer using the code in llm.c repo.
Skylion007-openwebtext-tokenizer-gpt2fineweb-edu-sample-10BT-gpt2tokenized