datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
proofwriter
Dataset Card for "proofwriter"
More Information needed
esci
Dataset Card for "esci"
ESCI product search dataset
https://github.com/amazon-science/esci-data/
Preprocessings:
-joined the two relevant files
-product_text aggregate all product text
-mapped esci_label to full name
@article{reddy2022shopping,
title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
author={Chandan K. Reddy and Lluís Màrquez and Fran Valero and Nikhil Rao and Hugo Zaragoza and Sambaran Bandyopadhyay and Arnab Biswas and Anlu… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/esci.foliohttps://github.com/Yale-LILY/FOLIO
@article{han2022folio,
title={FOLIO: Natural Language Reasoning with First-Order Logic},
author = {Han, Simeng and Schoelkopf, Hailey and Zhao, Yilun and Qi, Zhenting and Riddell, Martin and Benson, Luke and Sun, Lucy and Zubova, Ekaterina and Qiao, Yujie and Burtell, Matthew and Peng, David and Fan, Jonathan and Liu, Yixin and Wong, Brian and Sailor, Malcolm and Ni, Ansong and Nan, Linyong and Kasai, Jungo and Yu, Tao and Zhang, Rui and Joty, Shafiq and… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/folio.procedural-typed-decisions
procedural-typed-decisions
Procedurally generated decision problems. Each row is one structured state
(JSON, or a table, CSV, key=value lines, or prose for the arithmetic,
retrieval, and aggregation configs) with several typed questions over that same state, following the
Jev / System One request shape: choice (pick one criterion), noul (a
number in [0, 1]; a probability or a yes/no), and score (an ordered rubric).
Every answer is computed exactly from the state by rules that… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/procedural-typed-decisions.chaos-mnli-ambiguity
chaos-mnli-ambiguity
ChaosNLI, MNLI portion: 1,599 MNLI pairs relabeled by 100 annotators each (Nie et al., 2020).
label_dist and label_count follow the entailment/neutral/contradiction order, and gini is the Gini
coefficient of label_dist (0 = annotators evenly split, 1 = unanimous). Built from the jsonl first uploaded
here, which flattens the ChaosNLI release (https://github.com/easonnie/ChaosNLI) and adds gini; the
variable-key label_counter (a duplicate of label_count) is… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/chaos-mnli-ambiguity.jigsaw_toxicityplanbench
Dataset Card for "planbench"
https://arxiv.org/abs/2206.10498
@article{valmeekam2024planbench,
title={Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change},
author={Valmeekam, Karthik and Marquez, Matthew and Olmo, Alberto and Sreedharan, Sarath and Kambhampati, Subbarao},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}
QuALITY
Dataset Card for "QuALITY"
@article{bowman2022quality,
title={QuALITY: Question Answering with Long Input Texts, Yes!},
author={Bowman, Samuel R and Chen, Angelica and He, He and Joshi, Nitish and Ma, Johnny and Nangia, Nikita and Padmakumar, Vishakh and Pang, Richard Yuanzhe and Parrish, Alicia and Phang, Jason and others},
journal={NAACL 2022},
year={2022}
}
blog_authorship_corpussocial-chemestry-101help-desk-tickets
Help Desk Tickets
Processed tables derived from version 3 of Mohammad Abdellatif's Mendeley Data
dataset. The source contains real helpdesk tickets and associated workflow
records from an international software company, covering January 2007 through
March 2023. Identifiers and message content were masked by the source authors
to protect privacy while retaining context.
The default config is reporter_messages, with one row per issue that has
reporter-authored text. Utterances are… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/help-desk-tickets.goal-step-wikihowhttps://github.com/zharry29/wikihow-goal-step
@inproceedings{zhang-etal-2020-reasoning,
title = "Reasoning about Goals, Steps, and Temporal Ordering with {W}iki{H}ow",
author = "Zhang, Li and
Lyu, Qing and
Callison-Burch, Chris",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url =… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/goal-step-wikihow.simlexlogical-entailmenthttps://github.com/google-deepmind/logical-entailment-dataset
@inproceedings{
evans2018can,
title={Can Neural Networks Understand Logical Entailment?},
author={Richard Evans and David Saxton and David Amos and Pushmeet Kohli and Edward Grefenstette},
booktitle={International Conference on Learning Representations},
year={2018},
url={https://openreview.net/forum?id=SkZxCk-0Z},
}
acceptability-prediction@inproceedings{lau-etal-2015-unsupervised,
title = "Unsupervised Prediction of Acceptability Judgements",
author = "Lau, Jey Han and
Clark, Alexander and
Lappin, Shalom",
booktitle = "Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
month = jul,
year = "2015",
address = "Beijing, China",
publisher = "Association for… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/acceptability-prediction.cycic_multiplechoicehttps://colab.research.google.com/drive/16nyxZPS7-ZDFwp7tn_q72Jxyv0dzK1MP?usp=sharing
@article{Kejriwal2020DoFC,
title={Do Fine-tuned Commonsense Language Models Really Generalize?},
author={Mayank Kejriwal and Ke Shen},
journal={ArXiv},
year={2020},
volume={abs/2011.09159}
}
added for
@article{sileo2023tasksource,
title={tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation},
author={Sileo, Damien},
url=… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/cycic_multiplechoice.humicroedit
humicroedit
Humicroedit (SemEval-2020 task 7): news headlines edited to be funny.
subtask-1 grades one edited headline (meanGrade averages five 0-3 funniness grades); headline and
edited render the original headline and its edit from the <word/> markup. subtask-2 compares two
edits of the same headline (label 1 or 2 is the funnier one, 0 a tie).
Original data: SemEvalWorkshop/humicroedit. Repackaged as parquet for tasksource by scripts/upload_repackaged.py.
puzztehttps://bitbucket.org/RoxanaSz/puzzte/src/master/
@article{szomiu2021puzzle,
title={A Puzzle-Based Dataset for Natural Language Inference},
author={Szomiu, Roxana and Groza, Adrian},
journal={arXiv preprint arXiv:2112.05742},
year={2021}
}
boolq-natural-perturbationsBoolQ questions with semantic alteration and human verifications
@article{khashabi2020naturalperturbations,
title={Natural Perturbation for Robust Question Answering},
author={D. Khashabi and T. Khot and A. Sabhwaral},
journal={arXiv preprint},
year={2020}
}
measuring-hate-speech-votes
measuring-hate-speech-votes
Measuring Hate Speech (Kennedy et al., 2020; Sachdeva et al., 2022), one row per comment with vote counts.
The source has one row per (comment, annotator). Each survey item becomes a list of vote counts over its
ordinal codes, in code order: 0-4 for the nine Likert items, where a higher code is more hateful
(sentiment: strongly positive to strongly negative; respect: strongly respectful to strongly disrespectful;
insult, humiliate, dehumanize… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/measuring-hate-speech-votes.fool-me-twicehttps://github.com/google-research/fool-me-twice
@inproceedings{eisenschlos-etal-2021-fool,
title = "Fool Me Twice: Entailment from {W}ikipedia Gamification",
author = {Eisenschlos, Julian Martin and
Dhingra, Bhuwan and
Bulian, Jannis and
B{\"o}rschinger, Benjamin and
Boyd-Graber, Jordan},
booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/fool-me-twice.lewidi
lewidi
Learning with Disagreements (LeWiDi, SemEval-2023 Task 11 and its 2025 edition): soft labels from every annotator.
One config per dataset: md_agreement (offensiveness, 5 annotators), hs_brexit (hate speech, 6), armis
(Arabic misogyny and sexism, 3), conv_abuse (abuse in user turns of chatbot dialogues, 3 or more), csc
(sarcasm rated 1-6), mp (MultiPICo irony, multilingual) and varierrnli (NLI where each annotator may accept
several labels). soft_label lists the share of… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/lewidi.winowhyhttps://github.com/HKUST-KnowComp/WinoWhy
@inproceedings{zhang2020WinoWhy,
author = {Hongming Zhang and Xinran Zhao and Yangqiu Song},
title = {WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge},
booktitle = {Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) 2020},
year = {2020}
}
nli-veridicality-transitivity@inproceedings{yanaka-etal-2021-exploring,
title = "Exploring Transitivity in Neural {NLI} Models through Veridicality",
author = "Yanaka, Hitomi and
Mineshima, Koji and
Inui, Kentaro",
booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume",
year = "2021",
pages = "920--934",
}
paradehttps://github.com/heyunh2015/PARADE_dataset
@inproceedings{he-etal-2020-parade,
title = "{PARADE}: {A} {N}ew {D}ataset for {P}araphrase {I}dentification {R}equiring {C}omputer {S}cience {D}omain {K}nowledge",
author = "He, Yun and
Wang, Zhuoer and
Zhang, Yin and
Huang, Ruihong and
Caverlee, James",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/parade.english-gradinghttps://www.kaggle.com/competitions/feedback-prize-english-language-learning
scifact_entailmentSciFact entailment pairs (data-only; train/validation).
sts-companionhttps://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark
The companion datasets to the STS Benchmark comprise the rest of the English datasets used in the STS tasks organized by us in the context of SemEval between 2012 and 2017.
Authors collated two datasets, one with pairs of sentences related to machine translation evaluation. Another one with the rest of datasets, which can be used for domain adaptation studies.
@inproceedings{cer-etal-2017-semeval,
title = "{S}em{E}val-2017 Task 1:… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/sts-companion.monotonicity-entailment@inproceedings{yanaka-etal-2019-neural,
title = "Can Neural Networks Understand Monotonicity Reasoning?",
author = "Yanaka, Hitomi and
Mineshima, Koji and
Bekki, Daisuke and
Inui, Kentaro and
Sekine, Satoshi and
Abzianidze, Lasha and
Bos, Johan",
booktitle = "Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP",
year = "2019",
pages = "31--40",
}
offensive-humor@article{tang2022naughtyformer,
title={The Naughtyformer: A Transformer Understands Offensive Humor},
author={Tang, Leonard and Cai, Alexander and Li, Steve and Wang, Jason},
journal={arXiv preprint arXiv:2211.14369},
year={2022}
}