Datasetpublic

EB-Sky/python-code-10gb

EB-Sky Python Code (10GB) ~10GB of filtered Python source code for domain-adaptive pretraining. Source Filtered subset of codeparrot/codeparrot-clean (originally collected from public GitHub repositories). Processing Size filter: 200–200,000 characters per file Removed auto-generated files (header heuristics) Dropped files matching common hardcoded-secret patterns Exact deduplication (SHA-1) Email addresses redacted Format Single… See the full description on the dataset page: https://huggingface.co/datasets/EB-Sky/python-code-10gb.

sourceHugging Faceotherupdated 12d agoView on Hugging Face

1likes244downloads

Dataset Card

EB-Sky Python Code (10GB)

~10GB of filtered Python source code for domain-adaptive pretraining.

Source

Filtered subset of codeparrot/codeparrot-clean (originally collected from public GitHub repositories).

Processing

—Size filter: 200–200,000 characters per file
—Removed auto-generated files (header heuristics)
—Dropped files matching common hardcoded-secret patterns
—Exact deduplication (SHA-1)
—Email addresses redacted

Format

Single text column. Parquet shards under data/.

Load it

python

from datasets import load_dataset
ds = load_dataset("EB-Sky/python-code-10gb", split="train")

License

Code retains the original licenses of its source repos. Review before commercial use.