juzharii/text-mining-ce-dataset
Vietnamese Legal Cross-Encoder Dataset Training data for a cross-encoder reranker on Vietnamese legal documents. Source Built from YuITC/Vietnamese-Legal-Documents. Schema Column Type Description qid int64 Query ID cid int64 Document (context) ID query string Legal question document string Candidate document label int64 1 = positive, 0 = negative split string train or test negative_type string random, same_topic_wrong_article… See the full description on the dataset page: https://huggingface.co/datasets/juzharii/text-mining-ce-dataset.
Vietnamese Legal Cross-Encoder Dataset
Training data for a cross-encoder reranker on Vietnamese legal documents.
Source
Built from YuITC/Vietnamese-Legal-Documents.
Schema
Splits
Train
- Positives: gold (query, document) pairs from the source dataset
- Random negatives: 1 per query, sampled via rejection sampling
- Hard negatives: up to 3 per query, mined via hybrid BM25 + dense retrieval with 4-gate filter
Approximate ratio: ~3.6 negatives per positive.
Test
- Positives only — kept clean for retrieval eval / reranker eval
- No negatives mined
Hard Negative Mining
Pipeline: hybrid BM25 + dense retrieval (bkai-foundation-models/vietnamese-bi-encoder) with 4-gate filter:
- Gate 1: CID exact match
- Gate 2: MD5 exact text duplicate
- Gate 3: MinHash LSH near-duplicate (≥85% Jaccard)
- Gate 4: Cosine semantic similarity (threshold 0.92)
Candidates retrieved: top-50 from each retriever, rank window skips top-5 (too close to gold). Mining was distributed across 4 workers on Google Colab.
Usage
from datasets import load_dataset
ds = load_dataset("juzharii/text-mining-ce-dataset")
# Cross-encoder training
train = ds['train'] # query, document, label
# Eval
test = ds['test'] # positives only