Alignment-Lab-AI/Open-Web-Math

<img src="imgs/OpenWebMath-left.png" width="300">

Keiran Paster\, [Marco Dos Santos](https://marco-dossantos.github.io/)\, Zhangir Azerbayev, Jimmy Ba

GitHub | ArXiv | PDF

OpenWebMath is a dataset containing the majority of the high-quality, mathematical text from the internet. It is filtered and extracted from over 200B HTML files on Common Crawl down to a set of 6.3 million documents containing a total of 14.7B tokens. OpenWebMath is intended for use in pretraining and finetuning large language models.

You can download the dataset using Hugging Face:

from datasets import load_dataset
ds = load_dataset("open-web-math/open-web-math")

OpenWebMath Contents

The dataset is structured as follows:

{
  "text": ...,  # document text.
  "url": ...,  # document url.
  "date": ...,  # date the page was crawled.
  "metadata": ...,  # JSON containing information from the extraction process.
}

OpenWebMath contains documents from over 130k different domains, including data from forums, educational pages, and blogs. The dataset contains documents covering mathematics, physics, statistics, computer science, and more. The following table shows the most common domains in OpenWebMath by character count.

OpenWebMath Pipeline

<img src="imgs/pipeline.png" alt="Overview of the OpenWebMath Pipeline">

OpenWebMath builds on the massive Common Crawl dataset, which contains over 200B HTML documents. We filtered the data to only include documents that are: (1) in English, (2) contain mathematical content, and (3) are of high quality. We also put a strong emphasis on extracting LaTeX content from the HTML documents as well as reducing boilerplate in comparison to other web datasets.

The OpenWebMath pipeline consists of five steps:

1. 1.Prefiltering HTML Documents:
2. 2.We apply a simple prefilter to all HTML documents in Common Crawl in order to skip documents without mathematical content to unnecessary processing time.
3. 3.Text Extraction:
4. 4.Extract text, including LaTeX content, from the HTML documents while removing boilerplate.
5. 5.Content Classification and Filtering:
6. 6.Apply a FastText language identification model to keep only English documents.
7. 7.Filter high perplexity documents using a KenLM model trained on Proof-Pile.
8. 8.Filter non-mathematical documents using our own MathScore model.
9. 9.Deduplication:
10. 10.Deduplicate the dataset using SimHash in text-dedup.
11. 11.Manual Inspection:
12. 12.Inspect the documents gathered from previous steps and remove low quality pages.

For a detailed discussion on the processing pipeline, please refer to our paper.

License

OpenWebMath is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: https://commoncrawl.org/terms-of-use/. We do not alter the license of any of the underlying data.

Citation Information

@misc{paster2023openwebmath,
      title={OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text},
      author={Keiran Paster and Marco Dos Santos and Zhangir Azerbayev and Jimmy Ba},
      year={2023},
      eprint={2310.06786},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}