datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
MINT-1T-HTML
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-HTML.ml-intern-sessions
ML Intern session traces
This dataset contains ML Intern coding agent session traces uploaded from local
ML Intern runs. The traces are stored as JSON Lines files under sessions/,
with one file per session.
Links
ML Intern demo: https://smolagents-ml-intern.hf.space
ML Intern CLI: https://github.com/huggingface/ml-intern
Data description
Each *.jsonl file contains a single ML Intern session converted to a
Claude-Code-style event stream for the… See the full description on the dataset page: https://huggingface.co/datasets/lewtun/ml-intern-sessions.MINT-1T-PDF-CC-2023-14
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2023-14.chempile-mlift
ChemPile-MLIFT
A comprehensive multimodal dataset for chemistry property prediction using vision large language models
📋 Dataset Summary
ChemPile-MLIFT is a dataset designed for multimodal chemistry property prediction tasks, specifically focusing on the prediction of chemical properties using vision large language models (VLLMs). It is part of the ChemPile project, which aims to create a comprehensive collection of chemistry-related data for training LLMs. The… See the full description on the dataset page: https://huggingface.co/datasets/jablonkagroup/chempile-mlift.MINT-1T-PDF-CC-2024-10
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2024-10.MINT-1T-PDF-CC-2024-18
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2024-18.MINT-1T-PDF-CC-2023-40
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2023-40.MINT-1T-PDF-CC-2023-23
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2023-23.MINT-1T-ArXiv
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-ArXiv.MINT-1T-PDF-CC-2023-06
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2023-06.lingshu_training_data_medical_domain
Website
🤖 7B Model
🤖 8B Model based on InternVL3
🤖 32B Model
MedEvalKit
Technical Report
Lingshu MCP
Lingshu Medical MLLM Training Data (Medical Domain)
This dataset contains the medical-domain training data used in the multi-stage training of the Lingshu Medical Multimodal Large Language Model (MLLM). General-domain data has been removed; only medical data is included.
The training… See the full description on the dataset page: https://huggingface.co/datasets/lingshu-medical-mllm/lingshu_training_data_medical_domain.ai-ml-foundations-book-collection
Introduction
I put this collection together after spending a lot of time reading what I think are some of the best books on AI, machine learning, deep learning, probabilistic modeling, optimization, reinforcement learning, transformers, LLMs, validation, and fairness. I want to share this with the community for one simple reason: I want to give people a structured path through the books that actually help them understand things deeply, instead of sending them through random… See the full description on the dataset page: https://huggingface.co/datasets/Ujjwal-Tyagi/ai-ml-foundations-book-collection.MINT-1T-PDF-CC-2023-50
🍃 MINT-1T:Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
🍃 MINT-1T is an open-source Multimodal INTerleaved dataset with 1 trillion text tokens and 3.4 billion images, a 10x scale-up from existing open-source datasets. Additionally, we include previously untapped sources such as PDFs and ArXiv papers. 🍃 MINT-1T is designed to facilitate research in multimodal pretraining. 🍃 MINT-1T is created by a team from the University of Washington in… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/MINT-1T-PDF-CC-2023-50.optiq-lab-traces
OptiQ Lab Traces
Research and tool-calling sessions produced by OptiQ Lab, the local web UI that ships with mlx-optiq. Each session is a complete run: a deep-research report built from live web sources, or a multi-turn agent loop driving the Lab's own sandboxed tools.
The dataset is 866 sessions in HuggingFace Session-Traces format (the agent-traces viewer). Each .jsonl file is one session: a header line carrying the run's metadata, then one message per turn.
The two… See the full description on the dataset page: https://huggingface.co/datasets/mlx-community/optiq-lab-traces.ComPile
Dataset Card for ComPile: A Large IR Dataset from Production Sources
Changelog
Release
Programming Languages
Description
v1.0
C/C++, Rust, Swift, Julia
Fine Tuning-scale dataset of 602GB of deduplicated LLVM (bitcode) IR
Dataset Summary
ComPile contains over 2.7TB of permissively-licensed source code compiled to (textual) LLVM
intermediate representation (IR) covering C/C++, Rust, Swift, and Julia.
The dataset was created by hooking into LLVM… See the full description on the dataset page: https://huggingface.co/datasets/llvm-ml/ComPile.MLS-Bench-Tasks
MLS-Bench Tasks
MLS-Bench is a benchmark for machine learning science. Where most agent benchmarks reward engineering one fixed instance — clean the data, tune the pipeline, climb a leaderboard — MLS-Bench asks the harder question: can an AI agent propose a new component, loss, optimizer, or training procedure whose gain transfers across settings, seeds, datasets, and scales?The benchmark contains 140 tasks across 12 ML research domains. Each task fixes a research scaffold… See the full description on the dataset page: https://huggingface.co/datasets/Bohan22/MLS-Bench-Tasks.fineweb-edu-gemma4-1024
FineWeb-Edu — pre-tokenized for fast LM pretraining (Gemma tokenizer, ArrayRecord/Grain)
Pre-tokenized FineWeb-Edu
(sample/100BT), packed into fixed-length sequences and stored as
ArrayRecord shards for zero-overhead
streaming with Grain. No on-the-fly tokenization
at train time — you read int32 tokens straight off disk.
Format
Tokenizer: google/gemma-4-12B-it (vocab size 262144). Documents are
separated by the EOS token id 1.
Packing: the token stream is… See the full description on the dataset page: https://huggingface.co/datasets/mlnomad/fineweb-edu-gemma4-1024.orpo-dpo-mix-40k
ORPO-DPO-mix-40k v1.2
This dataset is designed for ORPO or DPO training.
See Fine-tune Llama 3 with ORPO for more information about how to use it.
It is a combination of the following high-quality DPO datasets:
argilla/Capybara-Preferences: highly scored chosen answers >=5 (7,424 samples)argilla/distilabel-intel-orca-dpo-pairs: highly scored chosen answers >=9, not in GSM8K (2,299 samples)
argilla/ultrafeedback-binarized-preferences-cleaned: highly scored chosen answers >=5 (22… See the full description on the dataset page: https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k.optiq-code-traces
OptiQ Code Traces
Gold-verified agentic software-engineering trajectories, produced by OptiQ Code, the terminal coding agent for local models on a Mac. Each trajectory is a full tool-calling run against a real repository bug, and every resolved label is set by executing the gold tests (FAIL_TO_PASS + PASS_TO_PASS) after applying the model's patch, never by the agent's own self-report.
The dataset is 1,789 agent sessions in HuggingFace Session-Traces format (the agent-traces… See the full description on the dataset page: https://huggingface.co/datasets/mlx-community/optiq-code-traces.DroidCall
DroidCall: A Dataset for LLM-powered Android Intent Invocation
paper|github
DroidCall is the first open-sourced, high-quality dataset designed for fine-tuning LLMs for accurate intent invocation on Android devices.
This repo contains data generated by DroidCall. The process of data generation is shown in the figure below
Details can be found in our paper and github repository.
What is Android Intent Invocation?
Android Intent is a key machanism in Android that allows… See the full description on the dataset page: https://huggingface.co/datasets/mllmTeam/DroidCall.csharp-ml-complation
C# full-line completion dataset (Roslyn)
Caret-based full-line completion samples extracted from permissively licensed C# repositories with
csharp-dataset-prepare. Each sample is an exact editor position:
left_context ends at the caret, target_text is the rest of the physical line (no newline), right_context follows it.
Original source is reconstructable from offsets; every row carries repository_id, revision, relative_path and license
for attribution.
Config
Content… See the full description on the dataset page: https://huggingface.co/datasets/dvislobokov/csharp-ml-complation.edu-fineweb-10BLogics-SWE-Env-2.5K
Logics-SWE-Env-2.5K
2,553 software engineering task instances · 1,771 repositories · 4 programming languages
🤗 Related model: Logics-SWE-Qwen3.6-27B
📄 Paper: One to More, More to One
💻 GitHub: AgenticBigBang
Overview
What is this dataset?
Logics-SWE-Env-2.5K is a collection of repository-level software engineering tasks for research on coding agents and environment-based reinforcement learning. It contains 2,553 unique task instances from 1,771… See the full description on the dataset page: https://huggingface.co/datasets/Logics-MLLM/Logics-SWE-Env-2.5K.ml-intern-sessions
ML Intern session traces
This dataset contains ML Intern coding agent session traces uploaded from local
ML Intern runs. The traces are stored as JSON Lines files under sessions/,
with one file per session.
Links
ML Intern demo: https://smolagents-ml-intern.hf.space
ML Intern CLI: https://github.com/huggingface/ml-intern
Data description
Each *.jsonl file contains a single ML Intern session converted to a
Claude-Code-style event stream for the… See the full description on the dataset page: https://huggingface.co/datasets/clem/ml-intern-sessions.MLLM-as-a-Judgeliquidrandom-data
liquidrandom-data
Diverse seed data for ML/LLM training data generation pipelines.
Used by the liquidrandom Python package.
Dataset Summary
This dataset contains 520,080 seed data samples across 24 categories,
generated using a hierarchical taxonomy tree approach with LLM-based quality validation
and fuzzy deduplication. Data is stored as Parquet with zstd compression.
Categories
Category
Samples
File
Coding Tasks
30,069… See the full description on the dataset page: https://huggingface.co/datasets/mlech26l/liquidrandom-data.go-ml-complation
Go full-line completion dataset (go/types)
Caret-based full-line completion samples extracted from permissively licensed Go repositories with the goflc builder
(Go go/parser + go/types). Each sample is an exact editor position: left_context ends at the caret, target_text
is the rest of the physical line (no newline, trailing whitespace excluded), right_context follows it. Original source
is reconstructable from offsets (byte offsets as used by Go tooling, plus UTF-16 offsets for… See the full description on the dataset page: https://huggingface.co/datasets/dvislobokov/go-ml-complation.ai-ml-foundations-book-collection
Introduction
I put this collection together after spending a lot of time reading what I think are some of the best books on AI, machine learning, deep learning, probabilistic modeling, optimization, reinforcement learning, transformers, LLMs, validation, and fairness. I want to share this with the community for one simple reason: I want to give people a structured path through the books that actually help them understand things deeply, instead of sending them through random courses… See the full description on the dataset page: https://huggingface.co/datasets/amine-khelif/ai-ml-foundations-book-collection.ReasonMed
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning
📄 Paper |
💻 Code |
📊 Dataset
ReasonMed is the largest open-source medical reasoning dataset to date, containing 370 K high-quality question–answer examples with multi-step chain-of-thought (CoT) rationales and concise summaries. We distilled these from 1.75 M initial reasoning paths generated by three competitive large-language models (Qwen-2.5-72B, DeepSeek-R1-Distill-Llama-70B, and… See the full description on the dataset page: https://huggingface.co/datasets/lingshu-medical-mllm/ReasonMed.ReFusion
ReFusion
Dataset Summary
This dataset is the training corpus used for ReFusion, as described in our paper. It comprises approximately 3.7 million high-quality instruction tuning samples consolidated from several state-of-the-art open-source datasets. The data covers diverse domains including mathematics, coding, and general instruction following.
Composition & Sources
The dataset is constructed from the following sources:
MAmmoTH
OpenMathInstruct-2 (1M… See the full description on the dataset page: https://huggingface.co/datasets/GSAI-ML/ReFusion.