commoncrawl/gneissweb-annotation-host-testing-v1

GneissWeb Annotations

GneissWeb Annotations, powered by IBM Research's GneissWeb methodology, is a dataset of quality and category annotations applied to the Common Crawl corpus.

This dataset enables precise filtering of web content across medical, educational, technology, and scientific domains, making it easier to build high-quality corpora for research projects, language models, and specialized applications.

Learn more about the annotation process and methodology in our official blog post.

What's Inside

GneissWeb Annotations uses the GneissWeb bloom filter made publicly available by IBM, along with IBM’s Data Prep Kit (now a Linux Foundation AI & Data project) and the GneissWeb groups’ category classifiers.

• —Medical - Health information, medical research, and clinical content
• —Education - Learning materials, academic resources, and educational platforms
• —Technology - Software documentation, technical guides, and tech industry content
• —Science - Research publications, scientific articles, and academic work

You can access annotations at two levels of granularity:

• —Host-level - Aggregate statistics for entire domains, perfect for broad filtering (This dataset)
• —URL-level - Individual URL classifications for precise content selection

Getting the Data

Access the dataset through:

• —Hugging Face: https://huggingface.co/datasets/commoncrawl/gneissweb-annotation-host-testing-v1 (Host)
• —Hugging Face: https://huggingface.co/datasets/commoncrawl/gneissweb-annotation-url-testing-v1 (URL)
• —AWS S3: s3://commoncrawl/projects/gneissweb-annotation-testing-v1

Example usage

Check out the gneissweb examples in the cc-index-annotations github repository.

Schema

Host Index Files

Aggregated domain-level annotations for efficient filtering by source.

URL Index Files

Granular annotations for individual URLs.

Applications

This dataset opens up numerous possibilities:

• —Train domain-specific language models with curated web data
• —Conduct research on content quality distribution across the web
• —Create filtered datasets for specific industries or use cases
• —Combine with other Common Crawl signals (language, TLD, etc.) for multi-dimensional filtering

Attribution

When using our data in your work, please cite the https://commoncrawl.org and let us know, we'd love to hear from you!

Licensing

Common Crawl Foundation's standard terms and condition apply, see https://commoncrawl.org/terms-of-use for more details.