datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
dclm-baseline-1.0-parquet
DCLM-baseline
Note: this is an identical copy of https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0, where all the files have been mapped to a parquet format.
DCLM-baseline is a 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks.
Below are comparisions of model trained on DCLM-baseline with other models in the 7B regime.
Model
Params
Tokens
Open dataset?
CORE
MMLU
EXTENDED
Open weights, closed datasets… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet.ettin-parquetraw_v0.1_parquet
Common Pile v0.1 — Parquet Consolidated
Description
This dataset bundles all “raw” corpora from the Common Pile v0.1 Raw Data collection, converted to Apache Parquet and consolidated in a single repository.
Nothing has been filtered or modified; the only changes are:
Format: original JSON → Parquet
Layout: many repositories → one consolidated dataset
Extra column: a len_category bucket for quick length-based filtering
Only the three original columns (id, text… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/raw_v0.1_parquet.Youtube-Common-First-600-Parquettest_librispeech_parquetmodular-s2orc-parquetdoc-formats-parquet-1euler-source-parquets-realTCGA-12K-parquet
TCGA-12K Parquet
Attribution
This dataset contains 224 x 224 JPEG patches from whole-slide images originally downloaded from The Cancer Genome Atlas (TCGA) that are available in the NCI Genomic Data Commons (GDC) Open Access tier. We mirror and repackage a commonly used ~12k WSI subset in parquet format for ease of training. We exclude patches that did not pass HSV thresholding, following the procedure in Kaiko.AI's Midnight paper. Patches were randomly sampled across… See the full description on the dataset page: https://huggingface.co/datasets/medarc/TCGA-12K-parquet.icmr-parquetAmericanStories-parquet
AmericanStories (parquet)
A parquet-native reformat of dell-research-harvard/AmericanStories — article-level full text of ~20 million U.S. newspaper scans (1774–1963) from the Library of Congress's Chronicling America collection, originally extracted by Dell et al. (arXiv:2308.12477).
This repo exists so the dataset loads in one line with the standard datasets / polars / pyarrow / dask stack, with no custom loading script and full Dataset Viewer support on the Hub.… See the full description on the dataset page: https://huggingface.co/datasets/biglam/AmericanStories-parquet.m2d2-wiki-decon-parquetnfpc-parquet-dataset
AML Mule Account Detection Challenge
Data Format: All files are in Apache Parquet format (Snappy compression). Use pandas.read_parquet(), pyarrow.parquet, or any Parquet-compatible reader. Transaction files are split across batch-N/ subdirectories.
Primary Objective/Problem Statement
Identify mule accounts used for money laundering from banking transaction and account data. Given labelled training data and unlabelled test accounts, predict which test accounts are mules.… See the full description on the dataset page: https://huggingface.co/datasets/preetisheoran/nfpc-parquet-dataset.livecodebench_code_generation_lite_parquet
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
🏠 Home Page •
💻 GitHub Repository •
🏆 Leaderboard •
📄 Paper
Change Log
Since LiveCodeBench is a continuously updated benchmark, we provide different versions of the dataset. Particularly, we provide the following versions of the dataset:
release_v1: The initial release of the dataset with problems released between May 2023 and Mar 2024 containing 400… See the full description on the dataset page: https://huggingface.co/datasets/marianna13/livecodebench_code_generation_lite_parquet.nsynth-parquetparquet_dataultra_v3_sft_parquetmasakhaner-x-parquetfsdkaggle2019-parquet
FSDKaggle2019
FSDKaggle2019[1] is an audio dataset containing 29,266 audio files annotated with 80 labels of the AudioSet Ontology.
FSDKaggle2019 has been used for the DCASE Challenge 2019 Task 2, which was run as a Kaggle competition titled Freesound Audio Tagging 2019.
All audio clips are provided as uncompressed PCM 16 bit, 44.1 kHz, mono audio files.
This version of database could be found and downloaded from here.
Data Split Statistics
Curated
Noisy
Test… See the full description on the dataset page: https://huggingface.co/datasets/mteb/fsdkaggle2019-parquet.nemotron_extra_sft_parquetdeepplanning-parquet
DeepPlanning Parquet Standardized
This dataset is a Parquet-standardized release of DeepPlanning benchmark assets, built for analytics, reproducibility, and easier programmatic usage.
Source And Provenance
Raw dataset: Qwen/DeepPlanning
Benchmark code: QwenLM/Qwen-Agent/benchmark/deepplanning
Paper: DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints
Maintainer: tuandunghcmut
Important note:
The raw HF dataset contains environment… See the full description on the dataset page: https://huggingface.co/datasets/tuandunghcmut/deepplanning-parquet.euler-source-parquetsCESNET-TLS-YEAR22-PARQUET
CESNET-TLS-Year22 — canonical flow parquet
CESNET-TLS-Year22 (507,739,073 TLS
flows over the full year 2022 from the CESNET2 backbone, 180 service labels)
converted from the cesnet-datazoo ORIG HDF5 database into a canonical
flow-record parquet schema: 357 daily parquet files, exactly 507,739,073
rows, 39.5 GB zstd.
label_service carries the authoritative APP label decoded from the
PyTables enum embedded in the source database; servicemap.csv (included)
documents the services.… See the full description on the dataset page: https://huggingface.co/datasets/Lystea/CESNET-TLS-YEAR22-PARQUET.esc50-parquetpile-uncopyrighted-parquetRadGenome-Brain_MRI_parquetblbooks-parquet
Dataset Card for British Library Books
This dataset is the same as https://huggingface.co/datasets/TheBritishLibrary/blbooks, however, this version is stored as parquet to avoid needing to run a datasets script. This also makes loading this dataset much quicker.
Dataset Summary
This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were… See the full description on the dataset page: https://huggingface.co/datasets/biglam/blbooks-parquet.sample-parquetSample Parquet dataset for testing purposes
DocMMIR_parquetfinqa-parquet
FinQA - Parquet
Modernized version of ibm-research/finqa in Parquet format.
Original dataset uses a deprecated loading script. This version is pre-processed and ready to use with modern datasets library.
Structure
Each row contains a single text field: pre_text + "\n" + question + "\n" + post_text
Rows: ~8,000 QA pairs (train + dev + test)
Source: ~2,800 financial reports
Split: all data combined (use your own split logic)
Usage
from datasets… See the full description on the dataset page: https://huggingface.co/datasets/whpthomas/finqa-parquet.