deduplication
codeparrot-valid-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (validation)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-valid-near-deduplication.codeparrot-train-near-deduplication
CodeParrot 🦜 Dataset after near deduplication (train)
Dataset Description
A dataset of Python files from Github. We performed near deduplication of this dataset split codeparrot-clean-train from codeparrot-clean. Exact deduplication can miss a fair amount of nearly identical files. We used MinHash with a Jaccard threshold (default=0.85) to create duplicate clusters. Then these clusters are reduced to unique files based on the exact Jaccard similarity. Fore more details… See the full description on the dataset page: https://huggingface.co/datasets/codeparrot/codeparrot-train-near-deduplication.core-2020-05-10-deduplication
Dataset Card for CORE Deduplication
Dataset Summary
CORE 2020 Deduplication dataset (https://core.ac.uk/documentation/dataset) contains 100K scholarly documents labeled as duplicates/non-duplicates.
Languages
The dataset language is English (BCP-47 en)
Citation Information
@inproceedings{dedup2020,
title={Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings},
author={Gyawali, Bikash and Anastasiou, Lucas and… See the full description on the dataset page: https://huggingface.co/datasets/pinecone/core-2020-05-10-deduplication.KingDesign-Deduplication
Kingdesign
Made with ❤️ using 🦥 Unsloth Studio
all was generated with Unsloth Recipe Studio. It contains 5,351 generated records.
🚀 Quick Start
from datasets import load_dataset
# Load the main dataset
dataset = load_dataset("JaveyZou/KingDesign", "data", split="train")
df = dataset.to_pandas()
📊 Dataset Summary
📈 Records: 5,351
📋 Columns: 3
✅ Completion: 89.2% (6,000 requested)
📋 Schema & Statistics
Column
Type
Column Type
Unique (%)… See the full description on the dataset page: https://huggingface.co/datasets/JaveyZou/KingDesign-Deduplication.2505-deduplication-evals
HPLT 3.0: Deduplication Strategy Comparison Results
Dataset Description
This dataset contains fine-grained results from our HPLT 3.0 pre-release evaluations comparing different data deduplication stategies for the pre-HPLT 3.0 corpora with the previous HPLT 2.0 version. We compare the following data deduplication strategies to guide our design choices, and guard against data quality regression compared to HPLT 2.0: pre-HPLT 3.0 CD (per-crawl deduplication), pre-HPLT 3.0… See the full description on the dataset page: https://huggingface.co/datasets/HPLT/2505-deduplication-evals.distributed_agent_message_deduplication_guard_teaser
🏆 Distributed Agent Workflows – Message De-duplication & Idempotency Guard (FAANG v2.5 Gold Teaser)
Enterprise Autonomous Sentinel • 100% AST Static Analysis Engine
This free teaser contains the 1-Click Ollama Modelfile, the official EU_AI_ACT_ANNEX_IV_AUDIT.md compliance audit, and an AST-verified evaluation dataset slice.
🚀 1-Click Local Run with Ollama
ollama create distributed_agent_message_deduplication_guard -f Modelfile
ollama run… See the full description on the dataset page: https://huggingface.co/datasets/emgena/distributed_agent_message_deduplication_guard_teaser.