Datasetpublic

EB-Sky/python-code-10gb

EB-Sky Python Code (10GB) ~10GB of filtered Python source code for domain-adaptive pretraining. Source Filtered subset of codeparrot/codeparrot-clean (originally collected from public GitHub repositories). Processing Size filter: 200–200,000 characters per file Removed auto-generated files (header heuristics) Dropped files matching common hardcoded-secret patterns Exact deduplication (SHA-1) Email addresses redacted Format Single… See the full description on the dataset page: https://huggingface.co/datasets/EB-Sky/python-code-10gb.

sourceHugging Faceotherupdated 15d agoView on Hugging Face

1likes246downloads

rootREADME.md

README.md39 linesDownload Raw Back to root

1---2license: other3task_categories:4- text-generation5language:6- code7size_categories:8- 1M<n<10M9---10 11# EB-Sky Python Code (10GB)12 13~10GB of filtered Python source code for domain-adaptive pretraining.14 15## Source16Filtered subset of17[codeparrot/codeparrot-clean](https://huggingface.co/datasets/codeparrot/codeparrot-clean)18(originally collected from public GitHub repositories).19 20## Processing21- Size filter: 200–200,000 characters per file22- Removed auto-generated files (header heuristics)23- Dropped files matching common hardcoded-secret patterns24- Exact deduplication (SHA-1)25- Email addresses redacted26 27## Format28Single `text` column. Parquet shards under `data/`.29 30## Load it31```python32from datasets import load_dataset33ds = load_dataset("EB-Sky/python-code-10gb", split="train")34```35 36## License37Code retains the original licenses of its source repos.38Review before commercial use.39