codeparrot/codeparrot-valid-near-deduplication

CodeParrot 🦜 Dataset after near deduplication (validation) Dataset Description A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-valid-near-deduplication.