Datasetpublic

commoncrawl/host-index-testing-v2

Common Crawl Host Index v2 GitHub: https://github.com/commoncrawl/cc-host-index Each crawl, we generate a Host Index, which aggregates information about each web hosted visited during the crawl. The information is aggregated from the Common Crawl columnar index, web graph, and raw crawler logs. Quickstart The dataset is Hive-partitioned on crawl (data/crawl=CC-MAIN-2025-18/*.parquet). Open the whole dataset once, then filter with WHERE crawl = '...': because… See the full description on the dataset page: https://huggingface.co/datasets/commoncrawl/host-index-testing-v2.

sourceHugging Faceupdated 25d agoView on Hugging Face

0likes6.4kdownloads

settings

This repository belongs to commoncrawl on Hugging Face.

Team Ai never edits a repository it does not host. Visibility, licence, collaborators and gating are all managed at the source.

namehost-index-testing-v2

visibilitypublic

licencenot set

gatedno

ownercommoncrawl

Account settings