data-processing
gpt2-dp-mod-datasets-txt-processingTerminal-data_processinggpt2-dp-mod-datasets-txt-processing-rarity-allgpt2-cocnat-mod-datasets-txt-processinggpt2-concat-mod-datasets-txt-processing-rarity-allnemotron-terminal-data_processing__Qwen3-8Binstruction-dataset-processing-filesvideo-dataset-processing-scripts
peacock-data-public-datasets-idc-enfm-dataprocessing
BharatGPT
Data Curation for English Foundational Model
RedPajamaV2
RedPajamaV2 directory contains scripts for processing and filtering of RedPajamaV2 dataset.
Overall pipeline is as follows:
URL Filtering scripts/utils/url_filtering.py: The first step of filtering is URL based. We use a blocklist of URLs known to contain inappropriate content. This list is taken from blocklistproject. It contains categories for advertisements, gambling, adult content, etc. All… See the full description on the dataset page: https://huggingface.co/datasets/applied-ai-018/peacock-data-public-datasets-idc-enfm-dataprocessing.easy_5000_data_processingmedium_5000_data_processingnemotron-terminal-data_processing
nemotron-terminal-data_processing
Per-source partition of nvidia/Nemotron-Terminal-Corpus,
filtered to source == "data_processing". The difficulty column preserves the original
easy / medium / mixed split (na for the dataset_adapters/* files, which
did not carry a difficulty label).
Partitioning scheme:
adapters_{code,math,swe} — rows from dataset_adapters/{code,math,swe}.parquet
{skill} (e.g. debugging, security, …) — rows from
synthetic_tasks/skill_based/{easy,medium… See the full description on the dataset page: https://huggingface.co/datasets/laion/nemotron-terminal-data_processing.medium_5000-data_processing_n100k1medium_5000_data_processing_fixed