datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
code_search_net
Dataset Card for CodeSearchNet corpus
Dataset Summary
CodeSearchNet corpus is a dataset of 2 milllion (comment, code) pairs from opensource libraries hosted on GitHub. It contains code and documentation for several programming languages.
CodeSearchNet corpus was gathered to support the CodeSearchNet challenge, to explore the problem of code retrieval using natural language.
Supported Tasks and Leaderboards
language-modeling: The dataset can be used to… See the full description on the dataset page: https://huggingface.co/datasets/code-search-net/code_search_net.code-search-net-python
Dataset Card for "code-search-net-python"
Dataset Description
Homepage: None
Repository: https://huggingface.co/datasets/Nan-Do/code-search-net-python
Paper: None
Leaderboard: None
Point of Contact: @Nan-Do
Dataset Summary
This dataset is the Python portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-python.CodeSearchNetEmploying the MTEB evaluation framework's dataset version, utilize the code below for assessment:
import mteb
import logging
from sentence_transformers import SentenceTransformer
from mteb import MTEB
logger = logging.getLogger(__name__)
model_name = 'intfloat/e5-base-v2'
model = SentenceTransformer(model_name)
tasks = mteb.get_tasks(
tasks=[
"AppsRetrieval",
"CodeFeedbackMT",
"CodeFeedbackST",
"CodeTransOceanContest",
"CodeTransOceanDL"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet.CodeSearchNet-ccrEmploying the MTEB evaluation framework's dataset version, utilize the code below for assessment:
import mteb
import logging
from sentence_transformers import SentenceTransformer
from mteb import MTEB
logger = logging.getLogger(__name__)
model_name = 'intfloat/e5-base-v2'
model = SentenceTransformer(model_name)
tasks = mteb.get_tasks(
tasks=[
"AppsRetrieval",
"CodeFeedbackMT",
"CodeFeedbackST",
"CodeTransOceanContest",
"CodeTransOceanDL"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ccr.CodeSearchNetRetrieval
CodeSearchNetRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
The dataset is a collection of code snippets and their corresponding natural language queries. The task is to retrieve the most relevant code snippet for a given query.
Task category
t2t
Domains
Programming, Written
Reference
https://huggingface.co/datasets/code_search_net/
Source datasets:
code-search-net/code_search_net
How to evaluate on this task
You can evaluate an embedding… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CodeSearchNetRetrieval.CodeSearchNet-ccrSWE-bench_Verified-code-searchinstructional_code-search-net-python
Dataset Card for "instructional_code-search-net-python"
Dataset Summary
This is an instructional dataset for Python.
The dataset contains two different kind of tasks:
Given a piece of code generate a description of what it does.
Given a description generate a piece of code that fulfils the description.
Languages
The dataset is in English.
Data Splits
There are no splits.
Dataset Creation
May of 2023
Curation Rationale
This… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/instructional_code-search-net-python.SWE-rebench-code-searchcode_search_net_clean
Dataset Card for "code_search_net_clean"
More Information needed
codesearchnet
Dataset Card for CodeSearchNet
This dataset is a collection of comment-code pairs of various programming languages. See code_search_net for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
Dataset Subsets
pair subset
Columns: "comment", "code"
Column types: str, str
Examples:{
'comment': 'Computes the new parent id for the node being moved.\n\n@return int',
'code': "protected function… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/codesearchnet.code_x_glue_tc_nl_code_search_adv
Dataset Card for "code_x_glue_tc_nl_code_search_adv"
Dataset Summary
CodeXGLUE NL-code-search-Adv dataset, available at https://github.com/microsoft/CodeXGLUE/tree/main/Text-Code/NL-code-search-Adv
The dataset we use comes from CodeSearchNet and we filter the dataset as the following:
Remove examples that codes cannot be parsed into an abstract syntax tree.
Remove examples that #tokens of documents is < 3 or >256
Remove examples that documents contain special tokens… See the full description on the dataset page: https://huggingface.co/datasets/google/code_x_glue_tc_nl_code_search_adv.code_search_net
CodeSearchNet
This is an unofficial reupload of the code_search_net dataset in the parquet format. I have also removed the columns func_code_tokens, func_documentation_tokens, and split_name as they are not relevant. The original repository relies on a Python module that is downloaded and executed to unpack the dataset, which is a potential security risk but importantly raises an annoying warning. As a plus, parquets load faster.
Original model card:
Dataset Card for… See the full description on the dataset page: https://huggingface.co/datasets/claudios/code_search_net.neural_code_searchNeural-Code-Search-Evaluation-Dataset presents an evaluation dataset consisting of natural language query and code snippet pairs and a search corpus consisting of code snippets collected from the most popular Android repositories on GitHub.code_search_le512owl_code_search_hard_negative_datasets-Pre_kd
Owl Code Search Hard Negative Datasets
Knowledge Distillation (KD) ベースのハードネガティブ付きコード検索データセットです。コード検索モデルShuu12121/CodeSearch-ModernBERT-Crow-v3-large-len1024-Plusを教師モデルとして、各コメントと説明コメントのペアのデータセットから各クエリに対する関数の類似度スコアを計算し、ハードネガティブ（正解に類似しているが不正解の文書）を付与しています。
概要
目的: コード検索モデルの Contrastive Learning / Knowledge Distillation ファインチューニング
言語: Go, Java, JavaScript, PHP, Python, Ruby, Rust, TypeScript（8言語）
総サンプル数: 4,787,740
データサイズ: 8.73 GB（展開後） / 3.37 GB（ダウンロード時）
フォーマット:… See the full description on the dataset page: https://huggingface.co/datasets/Shuu12121/owl_code_search_hard_negative_datasets-Pre_kd.SWE-bench_Lite-code-searchcode-search-net-java
Dataset Card for "code-search-net-java"
Dataset Summary
This dataset is the Java portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the function does.
Languages
The dataset's comments are in English and the functions are coded in Java
Data Splits
Train, test, validation labels are included in the… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-java.ann-codesearch-4m
Cleaning
Unlike the original dataset, the func_code_string column was updated to remove any comments and keep just the code.
The original version can still be found in the whole_func_string.
import re
def remove_comments_docstrings(code, language):
if language == 'python':
# Remove docstrings
code = re.sub(r'"""(.*?)"""', '', code, flags=re.DOTALL)
code = re.sub(r"'''(.*?)'''", '', code, flags=re.DOTALL)
# Remove comments
code =… See the full description on the dataset page: https://huggingface.co/datasets/unum-cloud/ann-codesearch-4m.codesearchnet-codegen
Dataset Card for CodeSearchNet for CodeGen
This is a processed version of the CodeSearchNet dataset. Namely, I separated the doc (documentation/docstring), sign (function signature), and output (function body) into separate fields; doc and sign are concatenated (according to the correct order of the programming language) into the problem field, making it suitable for the code generation task.
Dataset Details
Dataset Description
Curated by: [More… See the full description on the dataset page: https://huggingface.co/datasets/pengyunie/codesearchnet-codegen.CodeSearchNetCCRetrieval
CodeSearchNetCCRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
The dataset is a collection of code snippets. The task is to retrieve the most relevant code snippet for a given code snippet.
Task category
t2t
Domains
Programming, Written
Reference
https://arxiv.org/abs/2407.02883
Source datasets:
CoIR-Retrieval/CodeSearchNet-ccr
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CodeSearchNetCCRetrieval.code_search_net_filesCodeSearchNet corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation.owl_code_search_hard_negative_datasets_V2_kdcpp-code-code_search_net-style
C++ Dataset
documentation source: https://huggingface.co/docs/datasets/main/en/repository_structure
Supported Tasks and Leaderboards
language-modeling: The dataset can be used to train a model for modelling programming languages, which consists in building language models for programming languages.
Language
C++ programming language
Dataset Structure
Data Instances
A data point consists of a function code along with its documentation.… See the full description on the dataset page: https://huggingface.co/datasets/malteklaes/cpp-code-code_search_net-style.code-search-net-tokenizer-books-chunked-tokenizedcode_search_net_python_10000_examplescode_search_net_ru_enThe CodeSearhNet Dataset translated into Russian. Translation was done with Qwen3-8B model.
SWE-smith-py-code-searchCodeSearchNet-ruby-qrelsEmploying the CoIR evaluation framework's dataset version, utilize the code below for assessment:
import coir
from coir.data_loader import get_tasks
from coir.evaluation import COIR
from coir.models import YourCustomDEModel
model_name = "intfloat/e5-base-v2"
# Load the model
model = YourCustomDEModel(model_name=model_name)
# Get tasks
#all task ["codetrans-dl","stackoverflow-qa","apps","codefeedback-mt","codefeedback-st","codetrans-contest","synthetic-
# text2sql","cosqa","codesearchnet"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ruby-qrels.CodeSearchNet-ruby-queries-corpus
Dataset Card for "CodeSearchNet-ruby-queries-corpus"
More Information needed