Datasetpublic

minishlab/tokenlearn-cornstack-docs-coderankembed-v2

minishlab/tokenlearn-cornstack-docs-coderankembed-v2 Dataset Card This dataset was created with Tokenlearn for training Model2Vec models on code retrieval. It contains mean token embeddings produced by nomic-ai/CodeRankEmbed, used as training targets for static embedding distillation. The dataset contains code documents from CornStack across 6 programming languages (100,000 rows per language, 600,000 total). Dataset Details Field Value Source CornStack… See the full description on the dataset page: https://huggingface.co/datasets/minishlab/tokenlearn-cornstack-docs-coderankembed-v2.

sourceHugging Faceupdated 5mo agoView on Hugging Face

1likes34downloads

Dataset Card

minishlab/tokenlearn-cornstack-docs-coderankembed-v2 Dataset Card

This dataset was created with Tokenlearn for training Model2Vec models on code retrieval. It contains mean token embeddings produced by nomic-ai/CodeRankEmbed, used as training targets for static embedding distillation.

The dataset contains code documents from CornStack across 6 programming languages (100,000 rows per language, 600,000 total).

Dataset Details

Field	Value
Source	CornStack (nomic-ai)
Embedding model	nomic-ai/CodeRankEmbed
Embedding dimension	768
Languages	Python, Java, PHP, Go, JavaScript, Ruby
Rows per language	100,000
Total rows	600,000
Field	`document`

Source Datasets

Language	Source
`python`	nomic-ai/cornstack-python-v1
`java`	nomic-ai/cornstack-java-v1
`php`	nomic-ai/cornstack-php-v1
`go`	nomic-ai/cornstack-go-v1
`javascript`	nomic-ai/cornstack-javascript-v1
`ruby`	nomic-ai/cornstack-ruby-v1

Dataset Structure

Column	Type	Description
`text`	`string`	Truncated input text (tokenizer max length 512)
`embedding`	`list[float32]`	Mean token embedding from `nomic-ai/CodeRankEmbed`, excluding BOS/EOS tokens

Usage

Load a single language config:

python

from datasets import load_dataset

# Load Python code documents
dataset = load_dataset("minishlab/tokenlearn-cornstack-docs-coderankembed", name="python")

# Load all languages and concatenate
from datasets import concatenate_datasets
all_langs = concatenate_datasets([
    load_dataset("minishlab/tokenlearn-cornstack-docs-coderankembed", name=lang)["train"]
    for lang in ["python", "java", "php", "go", "javascript", "ruby"]
])

Creation

Featurized from CornStack using nomic-ai/CodeRankEmbed with mean token pooling (BOS/EOS excluded). Two sampling seeds (42 and 100) were used with a 10k streaming shuffle buffer to maximise diversity. Texts are truncated to 512 tokens.

Library Authors

Tokenlearn was developed by the Minish team consisting of Stephan Tulkens and Thomas van Dongen.

Citation

@software{minishlab2024model2vec,
  author       = {Stephan Tulkens and {van Dongen}, Thomas},
  title        = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year         = {2024},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17270888},
  url          = {https://github.com/MinishLab/model2vec},
  license      = {MIT}
}