datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
dclm-baseline-1.0-parquet
DCLM-baseline
Note: this is an identical copy of https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0, where all the files have been mapped to a parquet format.
DCLM-baseline is a 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks.
Below are comparisions of model trained on DCLM-baseline with other models in the 7B regime.
Model
Params
Tokens
Open dataset?
CORE
MMLU
EXTENDED
Open weights, closed datasets… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet.ettin-parquettest_librispeech_parquetraw_v0.1_parquet
Common Pile v0.1 — Parquet Consolidated
Description
This dataset bundles all “raw” corpora from the Common Pile v0.1 Raw Data collection, converted to Apache Parquet and consolidated in a single repository.
Nothing has been filtered or modified; the only changes are:
Format: original JSON → Parquet
Layout: many repositories → one consolidated dataset
Extra column: a len_category bucket for quick length-based filtering
Only the three original columns (id, text… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/raw_v0.1_parquet.Youtube-Common-First-600-Parquetmodular-s2orc-parqueteuler-source-parquets-realdoc-formats-parquet-1icmr-parquetTCGA-12K-parquet
TCGA-12K Parquet
Attribution
This dataset contains 224 x 224 JPEG patches from whole-slide images originally downloaded from The Cancer Genome Atlas (TCGA) that are available in the NCI Genomic Data Commons (GDC) Open Access tier. We mirror and repackage a commonly used ~12k WSI subset in parquet format for ease of training. We exclude patches that did not pass HSV thresholding, following the procedure in Kaiko.AI's Midnight paper. Patches were randomly sampled across… See the full description on the dataset page: https://huggingface.co/datasets/medarc/TCGA-12K-parquet.m2d2-wiki-decon-parquetultra_v3_sft_parquetnfpc-parquet-dataset
AML Mule Account Detection Challenge
Data Format: All files are in Apache Parquet format (Snappy compression). Use pandas.read_parquet(), pyarrow.parquet, or any Parquet-compatible reader. Transaction files are split across batch-N/ subdirectories.
Primary Objective/Problem Statement
Identify mule accounts used for money laundering from banking transaction and account data. Given labelled training data and unlabelled test accounts, predict which test accounts are mules.… See the full description on the dataset page: https://huggingface.co/datasets/preetisheoran/nfpc-parquet-dataset.AmericanStories-parquet
AmericanStories (parquet)
A parquet-native reformat of dell-research-harvard/AmericanStories — article-level full text of ~20 million U.S. newspaper scans (1774–1963) from the Library of Congress's Chronicling America collection, originally extracted by Dell et al. (arXiv:2308.12477).
This repo exists so the dataset loads in one line with the standard datasets / polars / pyarrow / dask stack, with no custom loading script and full Dataset Viewer support on the Hub.… See the full description on the dataset page: https://huggingface.co/datasets/biglam/AmericanStories-parquet.livecodebench_code_generation_lite_parquet
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
🏠 Home Page •
💻 GitHub Repository •
🏆 Leaderboard •
📄 Paper
Change Log
Since LiveCodeBench is a continuously updated benchmark, we provide different versions of the dataset. Particularly, we provide the following versions of the dataset:
release_v1: The initial release of the dataset with problems released between May 2023 and Mar 2024 containing 400… See the full description on the dataset page: https://huggingface.co/datasets/marianna13/livecodebench_code_generation_lite_parquet.Solaris_parquetnsynth-parquetNQ-F_1min_OHLCV_Parqueteuler-source-parquetsparquet_datanemotron_extra_sft_parquetfsdkaggle2019-parquet
FSDKaggle2019
FSDKaggle2019[1] is an audio dataset containing 29,266 audio files annotated with 80 labels of the AudioSet Ontology.
FSDKaggle2019 has been used for the DCASE Challenge 2019 Task 2, which was run as a Kaggle competition titled Freesound Audio Tagging 2019.
All audio clips are provided as uncompressed PCM 16 bit, 44.1 kHz, mono audio files.
This version of database could be found and downloaded from here.
Data Split Statistics
Curated
Noisy
Test… See the full description on the dataset page: https://huggingface.co/datasets/mteb/fsdkaggle2019-parquet.masakhaner-x-parquetdeepplanning-parquet
DeepPlanning Parquet Standardized
This dataset is a Parquet-standardized release of DeepPlanning benchmark assets, built for analytics, reproducibility, and easier programmatic usage.
Source And Provenance
Raw dataset: Qwen/DeepPlanning
Benchmark code: QwenLM/Qwen-Agent/benchmark/deepplanning
Paper: DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints
Maintainer: tuandunghcmut
Important note:
The raw HF dataset contains environment… See the full description on the dataset page: https://huggingface.co/datasets/tuandunghcmut/deepplanning-parquet.pile-uncopyrighted-parquetesc50-parquetDocMMIR_parquetRadGenome-Brain_MRI_parquetchoice_msgs_parquetsample-parquetSample Parquet dataset for testing purposes