mteb/cqadupstack-programmers

1---2annotations_creators:3- derived4language:5- eng6license: apache-2.07multilinguality: monolingual8task_categories:9- text-retrieval10task_ids: []11config_names:12- corpus13tags:14- mteb15- text16dataset_info:17- config_name: default18  features:19  - name: query-id20    dtype: string21  - name: corpus-id22    dtype: string23  - name: score24    dtype: float6425  splits:26  - name: test27    num_bytes: 4545228    num_examples: 167529- config_name: corpus30  features:31  - name: _id32    dtype: string33  - name: title34    dtype: string35  - name: text36    dtype: string37  splits:38  - name: corpus39    num_bytes: 3454641240    num_examples: 3217641- config_name: queries42  features:43  - name: _id44    dtype: string45  - name: text46    dtype: string47  splits:48  - name: queries49    num_bytes: 6028150    num_examples: 87651configs:52- config_name: default53  data_files:54  - split: test55    path: qrels/test.jsonl56- config_name: corpus57  data_files:58  - split: corpus59    path: corpus.jsonl60- config_name: queries61  data_files:62  - split: queries63    path: queries.jsonl64---65<!-- adapted from https://github.com/huggingface/huggingface_hub/blob/v0.30.2/src/huggingface_hub/templates/datasetcard_template.md -->66 67<div align="center" style="padding: 40px 20px; background-color: white; border-radius: 12px; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.05); max-width: 600px; margin: 0 auto;">68  <h1 style="font-size: 3.5rem; color: #1a1a1a; margin: 0 0 20px 0; letter-spacing: 2px; font-weight: 700;">CQADupstackProgrammersRetrieval</h1>69  <div style="font-size: 1.5rem; color: #4a4a4a; margin-bottom: 5px; font-weight: 300;">An <a href="https://github.com/embeddings-benchmark/mteb" style="color: #2c5282; font-weight: 600; text-decoration: none;" onmouseover="this.style.textDecoration='underline'" onmouseout="this.style.textDecoration='none'">MTEB</a> dataset</div>70  <div style="font-size: 0.9rem; color: #2c5282; margin-top: 10px;">Massive Text Embedding Benchmark</div>71</div>72 73CQADupStack: A Benchmark Data Set for Community Question-Answering Research74 75|               |                                             |76|---------------|---------------------------------------------|77| Task category | t2t                              |78| Domains       | Programming, Written, Non-fiction                               |79| Reference     | http://nlp.cis.unimelb.edu.au/resources/cqadupstack/ |80 81 82## How to evaluate on this task83 84You can evaluate an embedding model on this dataset using the following code:85 86```python87import mteb88 89task = mteb.get_tasks(["CQADupstackProgrammersRetrieval"])90evaluator = mteb.MTEB(task)91 92model = mteb.get_model(YOUR_MODEL)93evaluator.run(model)94```95 96<!-- Datasets want link to arxiv in readme to autolink dataset with paper -->97To learn more about how to run models on `mteb` task check out the [GitHub repitory](https://github.com/embeddings-benchmark/mteb). 98 99## Citation100 101If you use this dataset, please cite the dataset as well as [mteb](https://github.com/embeddings-benchmark/mteb), as this dataset likely includes additional processing as a part of the [MMTEB Contribution](https://github.com/embeddings-benchmark/mteb/tree/main/docs/mmteb).102 103```bibtex104 105@inproceedings{hoogeveen2015,106  acmid = {2838934},107  address = {New York, NY, USA},108  articleno = {3},109  author = {Hoogeveen, Doris and Verspoor, Karin M. and Baldwin, Timothy},110  booktitle = {Proceedings of the 20th Australasian Document Computing Symposium (ADCS)},111  doi = {10.1145/2838931.2838934},112  isbn = {978-1-4503-4040-3},113  location = {Parramatta, NSW, Australia},114  numpages = {8},115  pages = {3:1--3:8},116  publisher = {ACM},117  series = {ADCS '15},118  title = {CQADupStack: A Benchmark Data Set for Community Question-Answering Research},119  url = {http://doi.acm.org/10.1145/2838931.2838934},120  year = {2015},121}122 123 124@article{enevoldsen2025mmtebmassivemultilingualtext,125  title={MMTEB: Massive Multilingual Text Embedding Benchmark},126  author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff},127  publisher = {arXiv},128  journal={arXiv preprint arXiv:2502.13595},129  year={2025},130  url={https://arxiv.org/abs/2502.13595},131  doi = {10.48550/arXiv.2502.13595},132}133 134@article{muennighoff2022mteb,135  author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo{\"\i}c and Reimers, Nils},136  title = {MTEB: Massive Text Embedding Benchmark},137  publisher = {arXiv},138  journal={arXiv preprint arXiv:2210.07316},139  year = {2022}140  url = {https://arxiv.org/abs/2210.07316},141  doi = {10.48550/ARXIV.2210.07316},142}143```144 145# Dataset Statistics146<details>147  <summary> Dataset Statistics</summary>148 149The following code contains the descriptive statistics from the task. These can also be obtained using:150 151```python152import mteb153 154task = mteb.get_task("CQADupstackProgrammersRetrieval")155 156desc_stats = task.metadata.descriptive_stats157```158 159```json160{161    "test": {162        "num_samples": 33052,163        "number_of_characters": 34048829,164        "num_documents": 32176,165        "min_document_length": 61,166        "average_document_length": 1056.7033814022875,167        "max_document_length": 21955,168        "unique_documents": 32176,169        "num_queries": 876,170        "min_query_length": 15,171        "average_query_length": 55.1837899543379,172        "max_query_length": 149,173        "unique_queries": 876,174        "none_queries": 0,175        "num_relevant_docs": 1675,176        "min_relevant_docs_per_query": 1,177        "average_relevant_docs_per_query": 1.9121004566210045,178        "max_relevant_docs_per_query": 149,179        "unique_relevant_docs": 1675,180        "num_instructions": null,181        "min_instruction_length": null,182        "average_instruction_length": null,183        "max_instruction_length": null,184        "unique_instructions": null,185        "num_top_ranked": null,186        "min_top_ranked_per_query": null,187        "average_top_ranked_per_query": null,188        "max_top_ranked_per_query": null189    }190}191```192 193</details>194 195---196*This dataset card was automatically generated using [MTEB](https://github.com/embeddings-benchmark/mteb)*