datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
code_search_net
Dataset Card for CodeSearchNet corpus
Dataset Summary
CodeSearchNet corpus is a dataset of 2 milllion (comment, code) pairs from opensource libraries hosted on GitHub. It contains code and documentation for several programming languages.
CodeSearchNet corpus was gathered to support the CodeSearchNet challenge, to explore the problem of code retrieval using natural language.
Supported Tasks and Leaderboards
language-modeling: The dataset can be used to… See the full description on the dataset page: https://huggingface.co/datasets/code-search-net/code_search_net.code-search-net-python
Dataset Card for "code-search-net-python"
Dataset Description
Homepage: None
Repository: https://huggingface.co/datasets/Nan-Do/code-search-net-python
Paper: None
Leaderboard: None
Point of Contact: @Nan-Do
Dataset Summary
This dataset is the Python portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-python.CodeSearchNetEmploying the MTEB evaluation framework's dataset version, utilize the code below for assessment:
import mteb
import logging
from sentence_transformers import SentenceTransformer
from mteb import MTEB
logger = logging.getLogger(__name__)
model_name = 'intfloat/e5-base-v2'
model = SentenceTransformer(model_name)
tasks = mteb.get_tasks(
tasks=[
"AppsRetrieval",
"CodeFeedbackMT",
"CodeFeedbackST",
"CodeTransOceanContest",
"CodeTransOceanDL"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet.CodeSearchNet-ccrEmploying the MTEB evaluation framework's dataset version, utilize the code below for assessment:
import mteb
import logging
from sentence_transformers import SentenceTransformer
from mteb import MTEB
logger = logging.getLogger(__name__)
model_name = 'intfloat/e5-base-v2'
model = SentenceTransformer(model_name)
tasks = mteb.get_tasks(
tasks=[
"AppsRetrieval",
"CodeFeedbackMT",
"CodeFeedbackST",
"CodeTransOceanContest",
"CodeTransOceanDL"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ccr.CodeSearchNetRetrieval
CodeSearchNetRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
The dataset is a collection of code snippets and their corresponding natural language queries. The task is to retrieve the most relevant code snippet for a given query.
Task category
t2t
Domains
Programming, Written
Reference
https://huggingface.co/datasets/code_search_net/
Source datasets:
code-search-net/code_search_net
How to evaluate on this task
You can evaluate an embedding… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CodeSearchNetRetrieval.CodeSearchNet-ccrcode_search_net_clean
Dataset Card for "code_search_net_clean"
More Information needed
codesearchnet
Dataset Card for CodeSearchNet
This dataset is a collection of comment-code pairs of various programming languages. See code_search_net for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
Dataset Subsets
pair subset
Columns: "comment", "code"
Column types: str, str
Examples:{
'comment': 'Computes the new parent id for the node being moved.\n\n@return int',
'code': "protected function… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/codesearchnet.code_search_net
CodeSearchNet
This is an unofficial reupload of the code_search_net dataset in the parquet format. I have also removed the columns func_code_tokens, func_documentation_tokens, and split_name as they are not relevant. The original repository relies on a Python module that is downloaded and executed to unpack the dataset, which is a potential security risk but importantly raises an annoying warning. As a plus, parquets load faster.
Original model card:
Dataset Card for… See the full description on the dataset page: https://huggingface.co/datasets/claudios/code_search_net.code-search-net-java
Dataset Card for "code-search-net-java"
Dataset Summary
This dataset is the Java portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the function does.
Languages
The dataset's comments are in English and the functions are coded in Java
Data Splits
Train, test, validation labels are included in the… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-java.codesearchnet-codegen
Dataset Card for CodeSearchNet for CodeGen
This is a processed version of the CodeSearchNet dataset. Namely, I separated the doc (documentation/docstring), sign (function signature), and output (function body) into separate fields; doc and sign are concatenated (according to the correct order of the programming language) into the problem field, making it suitable for the code generation task.
Dataset Details
Dataset Description
Curated by: [More… See the full description on the dataset page: https://huggingface.co/datasets/pengyunie/codesearchnet-codegen.CodeSearchNetCCRetrieval
CodeSearchNetCCRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
The dataset is a collection of code snippets. The task is to retrieve the most relevant code snippet for a given code snippet.
Task category
t2t
Domains
Programming, Written
Reference
https://arxiv.org/abs/2407.02883
Source datasets:
CoIR-Retrieval/CodeSearchNet-ccr
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CodeSearchNetCCRetrieval.code_search_net_filesCodeSearchNet corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation.code-search-net-tokenizer-books-chunked-tokenizedcode_search_net_python_10000_examplescode_search_net_ru_enThe CodeSearhNet Dataset translated into Russian. Translation was done with Qwen3-8B model.
CodeSearchNet-ruby-qrelsEmploying the CoIR evaluation framework's dataset version, utilize the code below for assessment:
import coir
from coir.data_loader import get_tasks
from coir.evaluation import COIR
from coir.models import YourCustomDEModel
model_name = "intfloat/e5-base-v2"
# Load the model
model = YourCustomDEModel(model_name=model_name)
# Get tasks
#all task ["codetrans-dl","stackoverflow-qa","apps","codefeedback-mt","codefeedback-st","codetrans-contest","synthetic-
# text2sql","cosqa","codesearchnet"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ruby-qrels.CodeSearchNet-ruby-queries-corpus
Dataset Card for "CodeSearchNet-ruby-queries-corpus"
More Information needed
CodeSearchNetcode_search_netcode-search-net/code_search_net , already loaded and converted to paraquet so you dont have to enable remote execution to use it .
codesearchnet-python-raw-457kcode-search-net-python
Dataset Card for "code-search-net-python"
Dataset Description
Homepage: None
Repository: https://huggingface.co/datasets/Nan-Do/code-search-net-python
Paper: None
Leaderboard: None
Point of Contact: @Nan-Do
Dataset Summary
This dataset is the Python portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the… See the full description on the dataset page: https://huggingface.co/datasets/khemprogrammer/code-search-net-python.CodeSearchNet-Pythoncode-search-net-javascript
Dataset Card for "code-search-net-javascript"
Dataset Summary
This dataset is the JavaScript portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the function does.
Languages
The dataset's comments are in English and the functions are coded in JavaScript
Data Splits
Train, test, validation labels are… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-javascript.codesearchnet-python-rawcode-search-net-php
Dataset Card for "code-search-net-php"
Dataset Summary
This dataset is the Php portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the function does.
Languages
The dataset's comments are in English and the functions are coded in Php
Data Splits
Train, test, validation labels are included in the dataset… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-php.codesearchnet-mirrorCodeSearchNet-ccr-javascript-queries-corpus
Dataset Card for "CodeSearchNet-ccr-javascript-queries-corpus"
More Information needed
CodeSearchNet-python-queries-corpus
Dataset Card for "CodeSearchNet-python-queries-corpus"
More Information needed
codesearchnet-python-pep8-v1