uv-scripts/deduplication
Semantic Deduplication UV Script Part of uv-scripts — self-contained UV scripts you run locally or on Hugging Face Jobs in one command. Remove duplicate / near-duplicate text samples from a Hugging Face dataset by semantic similarity — clean training data and prevent train/test leakage. Uses SemHash with Model2Vec embeddings: CPU-optimized, no GPU required. Quick start # CPU is enough — run on Hugging Face Jobs hf jobs uv run --flavor cpu-upgrade --secrets… See the full description on the dataset page: https://huggingface.co/datasets/uv-scripts/deduplication.
This repository belongs to uv-scripts on Hugging Face.
Team Ai never edits a repository it does not host. Visibility, licence, collaborators and gating are all managed at the source.