datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
HPLT2.0_cleanedNB: HPLT2.0 is now superseded by a newer release:
HPLT3.0
We recommed switching to v3.0, unless you have a compelling reason to stay on 2.0.
This is a large-scale collection of web-crawled documents in 191 world languages, produced by the HPLT project.
The source of the data is mostly Internet Archive with some additions from Common Crawl.
For a detailed description of the dataset, please refer to our website and our pre-print.
The Cleaned variant of HPLT Datasets v2.0
This is… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/HPLT2.0_cleaned.hplt2_edu_scores
HPLT2-Edu-scores
Dataset summary
HPLT2-JQL-Education is a model-annotated language subset of HPLT2, spanning 35 languages.
Our model-annotations allow for a filtering that achieves higher-quality training outcomes without excessively aggressive data reduction.
The original FW2 heuristic filtering method serves as our baseline, providing reference points for both the volume of retained tokens and downstream model performance.
For example, in the Spanish language case… See the full description on the dataset page: https://huggingface.co/datasets/JQL-AI/hplt2_edu_scores.hpltv2-llama33-edu-annotation
HPLT version 2.0 educational annotations
This dataset contains annotations derived from HPLT v2 cleaned samples.
There are 500,000 annotations for each language if the source contains at least 500,000 samples.
We prompt Llama-3.3-70B-Instruct to score web pages based on their educational value following FineWeb-Edu classifier.
Note 1: The dataset contains the prompt (using the first 1500 characters of the text sample), the scores, and the full Llama 3 generation. The column "idx"… See the full description on the dataset page: https://huggingface.co/datasets/LumiOpen/hpltv2-llama33-edu-annotation.hplt3_edu_scores
HPLT3-Edu-scores
Dataset summary
HPLT3-JQL-Education is a model-annotated language subset of HPLT3, spanning 36 languages.
Our model-annotations allow for a filtering that achieves higher-quality training outcomes without excessively aggressive data reduction.
HPLT3-Edu-scores was created based on scores assigned by a deep learning classifier trained to identify educational samples using Snowflake's Arctic-embed-m-v2.0 embeddings.
For all training ablations, we used… See the full description on the dataset page: https://huggingface.co/datasets/Eurolingua/hplt3_edu_scores.indic-hplt-v2
Indic HPLT v2
A multilingual pretraining corpus of 34,605,630 documents (~25.5B estimated tokens, ~218 GB raw JSONL) across 13 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data.
This is the larger successor to Indic HPLT v1 (9.8M docs, 11 languages). Compared to v1, this release adds 3 new Indic languages (Nepali, Odia, Assamese) and ~3.5× more documents overall.
Quick Start
from datasets import load_dataset
# Full training… See the full description on the dataset page: https://huggingface.co/datasets/ashtok897/indic-hplt-v2.indic-hplt-v1
Indic HPLT v1
A multilingual pretraining corpus of 9,836,075 documents (~8.4B estimated tokens) across 10 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data.
Quick Start
from datasets import load_dataset
# Full training split
ds = load_dataset("ashtok897/indic-hplt-v1", split="train")
# Filter by language
hi_ds = ds.filter(lambda x: x["lang"] == "hi")
# Streaming (recommended for large-scale use)
ds =… See the full description on the dataset page: https://huggingface.co/datasets/ashtok897/indic-hplt-v1.european-hplt-v1
European HPLT v1
A multilingual pretraining corpus of 45,031,396 documents (~50.9B estimated tokens, ~190 GB raw JSONL) across 41 European languages, built from HPLT Monolingual v3 high-quality web crawl data.
The corpus spans Germanic, Romance, Slavic, Celtic, Baltic, Finno-Ugric, Greek, and other European language families. Every document has an HPLT WDS quality score of 10 or higher (the top of the quality distribution).
Quick Start
from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/ashtok897/european-hplt-v1.turkish-hplt2-filteredHPLT-zhHPLT3_DE_0.9_Quantile_Adult_FilteredHPLT_Finnish_fineweb_edu_predictedlumiopen-hpltv2-llama33-edu-annotation-etHPLT_1.2_fi_cleaned2508-datasets-evals
HPLT 3.0: Details on Corpus Comparison Results
Dataset Description
This dataset contains fine-grained results from our HPLT 3.0 release evaluations comparing the new HPLT 3.0 corpora with the previous HPLT 2.0 version, FineWeb2, and MADLAD-400. We pretrain 2.2B Llama-style decoder models on 100B tokens for each selected language and evaluate them using HPLT-E, a multilingual evaluation framework for comprehensive multi-prompt k-shot evaluation across 124 tasks and 500+… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2508-datasets-evals.fra-hplt
French HPLT
A French pretraining corpus of 720 405 documents built from HPLT Monolingual v3 high-quality web crawl data.
The corpus is streaming-ready and includes per-document metadata: language confidence, URL, quality score, character/word counts, and per-register scores. Every document has an HPLT WDS quality score of 10 or higher (the top of the quality distribution).
Quick Start
from datasets import load_dataset
# Streaming
ds =… See the full description on the dataset page: https://huggingface.co/datasets/LeMoussel/fra-hplt.HPLT3_DE_0.9_Quantile_Adult_Filtered_PropelaHPLT3_DE_0.8_QuantileHPLT-short-12505-deduplication-evals
HPLT 3.0: Deduplication Strategy Comparison Results
Dataset Description
This dataset contains fine-grained results from our HPLT 3.0 pre-release evaluations comparing different data deduplication stategies for the pre-HPLT 3.0 corpora with the previous HPLT 2.0 version. We compare the following data deduplication strategies to guide our design choices, and guard against data quality regression compared to HPLT 2.0: pre-HPLT 3.0 CD (per-crawl deduplication), pre-HPLT 3.0… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2505-deduplication-evals.HPLT3_DE_0.9_Quantilehplt-uk-100kHPLT3_pol_LatnHPLT3 dataset, Polish split. Should be a lot of tokens, probably 200B+ depending on tokenizer used.
not sure what license it has, I am not a lawyer. Find more dataset and license info here: https://hplt-project.org/datasets/v3.0
mos-latn-hpltHPLT2-filtered-edu-tr2508-wds-evals
HPLT 3.0: Details on WDS-based Sampling Evaluation Results
Dataset Description
This dataset contains fine-grained results from our HPLT 3.0 release evaluations comparing the new HPLT 3.0 corpora sampled using different Web Document Scorer (WDS) thresholds, focusing on Spanish and French. We compare three configurations: Top, Random, and Bottom. Random sampling represents the default approach, drawing uniformly on the full corpus, while Top and Bottom take advantage of the… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2508-wds-evals.