datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Apertus-v1.5-Preference-Data
Apertus 1.5 Preference Dataset
This is the preference dataset used for the offline DPO stage of Apertus v1.5 alignment training, applied to the 70B model.
The prompts come from Ai2's Olmo 3 Dolci-Instruct-DPO dataset. We only reuse the prompts from Dolci-Instruct-DPO; all chosen / rejected responses in this dataset were generated by us.
How this dataset was built
Prompts. Taken from Dolci-Instruct-DPO (ODC-BY).
Response generation and annotation. Every prompt was… See the full description on the dataset page: https://huggingface.co/datasets/swiss-ai/Apertus-v1.5-Preference-Data.preference_dataset_mixture2_and_safe_pku
Copy from https://huggingface.co/datasets/weqweasdas/preference_dataset_mixture2_and_safe_pku
Reward Model Overview
This is the data mixture used for the reward model weqweasdas/RM-Mistral-7B, trained with the script https://github.com/WeiXiongUST/RLHF-Reward-Modeling .
Also see a short blog for the training details (data mixture, parameters...): https://www.notion.so/Reward-Modeling-for-RLHF-abe03f9afdac42b9a5bee746844518d0
Model Details
If you have any question… See the full description on the dataset page: https://huggingface.co/datasets/OpenRLHF/preference_dataset_mixture2_and_safe_pku.pair-preference-dataset-700K_standardproactivity_preference_dataset
ProVoice study 1 — driver state, vehicle context and preferred Level of Autonomy
Driving-simulator data from the population data collection of the ProVoice /
ProActivity project (CARLA 0.10): 12 drivers × 2 sessions, ~20 Hz
multimodal driver-state and vehicle frames, and 1,446 driver-assigned
Level-of-Autonomy (LoA) labels stating how autonomously an in-vehicle
assistant should act on a given task. Drivers were prompted every 20 s about
two randomly drawn in-vehicle tasks and… See the full description on the dataset page: https://huggingface.co/datasets/ProVoice-proactivity/proactivity_preference_dataset.PRO-STEP-Preference-Data
PRO-STEP: DPO Preference Pairs
Step-level preference pairs used to train the PRO-STEP policy model via Direct Preference Optimization.
Paper: PRO-STEP: Step-level Process Reward Optimization for Retrieval-Augmented GenerationCode: GitHub Repository
Pairs: 15,877 (after outcome filter)
Source questions: 5,000 from HotpotQA + MuSiQue + 2WikiMultiHopQA training splits
Generation: PRM-guided MCTS (K=3 branching, depth 7, 64 rollouts/question, V(s) = Q̄(s) + α · r̂(s) with α=0.3)… See the full description on the dataset page: https://huggingface.co/datasets/MinKeonKim/PRO-STEP-Preference-Data.preference_dataset_mixture2_and_safe_pku
Reward Model Overview
This is the data mixture used for the reward model weqweasdas/RM-Mistral-7B, trained with the script https://github.com/WeiXiongUST/RLHF-Reward-Modeling .
Also see a short blog for the training details (data mixture, parameters...): https://www.notion.so/Reward-Modeling-for-RLHF-abe03f9afdac42b9a5bee746844518d0
Model Details
If you have any question with this reward model and also any question about reward modeling, feel free to drop me an… See the full description on the dataset page: https://huggingface.co/datasets/weqweasdas/preference_dataset_mixture2_and_safe_pku.pair-preference-dataset-700K_subset-15-out-of-16_standardpreference_dataset_mix2
Dataset Card for "preference_dataset_mix2"
More Information needed
stackexchange-preference-datamath-preference-dataset
Dataset Card for math-preference-dataset
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/sdiazlor/math-preference-dataset/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/sdiazlor/math-preference-dataset.RLCD-generated-preference-data-split
Dataset Card for "RLCD-generated-preference-data-split"
More Information needed
pair-preference-dataset-700K_subset-7-out-of-8_standardpreference_data_sh_wo_checklistpair-preference-dataset-700K_subset-9-out-of-10_standardpreference_dataset_mixture2_and_safe_pku150k
Dataset Card for "preference_dataset_mixture2_and_safe_pku150k"
More Information needed
1004_ti2t_preference_dataset_supple_30kSYNTHETIC-1-Preference-Data
SYNTHETIC-1: Two Million Crowdsourced Reasoning Traces from Deepseek-R1
SYNTHETIC-1 is a reasoning dataset obtained from Deepseek-R1, generated with crowdsourced compute and annotated with diverse verifiers such as LLM judges or symbolic mathematics verifiers. This is the SFT version of the dataset - the raw data and SFT dataset can be found in our 🤗 SYNTHETIC-1 Collection.
The dataset consists of the following tasks and verifiers that were implemented in our library genesys:… See the full description on the dataset page: https://huggingface.co/datasets/PrimeIntellect/SYNTHETIC-1-Preference-Data.pair-preference-dataset-700K_subset-2-of-3_standardexample-preference-dataset
Dataset Card for example-preference-dataset
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/Yuriy81/example-preference-dataset/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/Yuriy81/example-preference-dataset.pair-preference-dataset-700K_subset-2-of-4_gemma-2b_1of4_iter3_conf-0.8_bs128_lr1e-5_conf-0.8example-generate-preference-dataset
Dataset Card for example-preference-dataset
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/sdiazlor/example-preference-dataset/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/distilabel-internal-testing/example-generate-preference-dataset.gst-india-preference-dataset-prep-smalldataset-for-annotation-v2-annotated
合成された質問と2つの応答文のペアに対して、日本語を母語とするチームメンバーが、好ましい応答文を人手でアノテーションしました
HelpSteer2-preferenceに習い、選好だけでなく選好の強度も[-3, 3]の範囲で付与しました
アノテーション時のメモ
ジャンルは以下
簡単な一般知識(wikipediaを読まずに回答できる系)
難しめの一般知識(wikipediaを読んだら回答できる系)
歴史上の出来事の論述
医療知識(応急処置系)
機械学習の課題と解決方法
化学式の解説
架空の物語生成
ロールプレイ
詩の創作
素因数分解や偶数奇数判定などの簡単な数学タスク
コーディングタスク
アルゴリズムやシステムのメリデメの解説
美術や思想についての論述
日本語の文法の解説
その他
LLMの定型文として登場するフレーズは、
「もちろんです」
「～も見逃せません」
「～も見過ごせません」
「総じて、」
「まず始めに、～さらに、～次に、～まとめると、」
データセットを目視で読み込んだ印象… See the full description on the dataset page: https://huggingface.co/datasets/preference-team/dataset-for-annotation-v2-annotated.pair-preference-dataset-700K_subset-4-of-4_gemma-2b_1of4_iter1_conf-0.8_bs128_lr1e-5_conf-0.8preference_dataset_mixture2_and_safe_pku
Reward Model Overview
This is the data mixture used for the reward model weqweasdas/RM-Mistral-7B, trained with the script https://github.com/WeiXiongUST/RLHF-Reward-Modeling .
Also see a short blog for the training details (data mixture, parameters...): https://www.notion.so/Reward-Modeling-for-RLHF-abe03f9afdac42b9a5bee746844518d0
Model Details
If you have any question with this reward model and also any question about reward modeling, feel free to drop me an… See the full description on the dataset page: https://huggingface.co/datasets/living-box/preference_dataset_mixture2_and_safe_pku.pair-preference-dataset-700K_subset-4-of-4_gemma-2b_1of4_iter1_bs128_lr1e-5_conf-0.90930_ta2t_preference_dataset_20Kpair-preference-dataset-700K_subset-3-of-4_gemma-2b_1of4_iter1_conf-0.8_bs128_lr1e-5_conf-0.8pair-preference-dataset-700K_subset-2-of-2_standardMNLP_M1_Preference_dpo_dataset
M1 Preference Data for DPO
Dataset Description
This dataset contains processed M1 preference data for DPO training.
Created by: CS-552 Stochastic Parrots Team
Date: May 24, 2025
Version: 1.0
Number of examples: 17615
Dataset Source
This dataset is derived from the M1 preference data collected through interactions with large language models (like ChatGPT) for CS-552 (Modern Natural Language Processing) at EPFL. The preference data consists of… See the full description on the dataset page: https://huggingface.co/datasets/stochastic-parrots/MNLP_M1_Preference_dpo_dataset.