Datasetpublic

vllm-sr/halueval-spans

HaluEval Span-Level Dataset (LLM-Detected) 🔍 High-quality span-level hallucination detection dataset converted from HaluEval using Qwen2.5-72B-Instruct for precise span detection and RAGTruth-compatible labeling. Quick Start from datasets import load_dataset dataset = load_dataset("llm-semantic-router/halueval-spans") Why This Dataset? Problem Previous Solution This Dataset HaluEval has binary labels only NLI-based conversion ✅… See the full description on the dataset page: https://huggingface.co/datasets/vllm-sr/halueval-spans.

sourceHugging Facemitupdated 9mo agoView on Hugging Face

1likes125downloads

Dataset Card

HaluEval Span-Level Dataset (LLM-Detected)

🔍 High-quality span-level hallucination detection dataset converted from HaluEval using Qwen2.5-72B-Instruct for precise span detection and RAGTruth-compatible labeling.

Quick Start

python

from datasets import load_dataset

dataset = load_dataset("llm-semantic-router/halueval-spans")

Why This Dataset?

Problem	Previous Solution	This Dataset
HaluEval has binary labels only	NLI-based conversion	✅ LLM-based span detection
NLI marks entire sentences	Sentence-level spans	✅ Partial spans (avg 15% coverage)
Generic "hallucinated" label	Single label	✅ 4 RAGTruth-compatible types
Full-response marking	50%+ of response marked	✅ 99.4% partial spans (<50% coverage)

What's New (v2.1 - LLM-Detected + Normalized)

This version uses Qwen2.5-72B-Instruct for span detection AND normalizes prompts to match RAGTruth format:

Metric	v1 (NLI)	v2.0 (LLM)	v2.1 (Normalized)
Span granularity	Sentence-level	Token-level	Token-level
Partial spans (<50%)	~60%	99.4%	99.4%
Label types	1	4 (RAGTruth)	4 (RAGTruth)
Prompt format	`Document:...Summary:`	`Document:...Summary:`	`Summarize the following text within N words:`
task_type	`summarization`	`summarization`	`Summary` (matches RAGTruth)
Fields	8	8	7 (matches RAGTruth exactly)

Statistics

—10,000 summarization samples
—8,905 samples with detected hallucinations (89%)
—16,359 total hallucinated spans
—1.8 average spans per hallucinated sample
—99.4% partial spans (not marking entire responses)

Label Distribution (RAGTruth-Compatible)

Label	Count	Percentage	Description
Evident Baseless Info	6,754	41.3%	Information clearly not in context
Evident Conflict	4,718	28.8%	Clearly contradicts context
Subtle Baseless Info	2,594	15.9%	Plausible but unverifiable claims
Subtle Conflict	2,293	14.0%	Subtly contradicts context

Format

json

{
  "prompt": "Summarize the following text within 150 words:\nFrance is a country located in Western Europe...",
  "answer": "France is located in Asia and is known for...",
  "labels": [
    {"start": 0, "end": 25, "label": "Evident Conflict"}
  ],
  "task_type": "Summary",
  "split": "train",
  "dataset": "halueval_summarization",
  "language": "en"
}

Fields

Field	Type	Description
prompt	string	Task instruction + document (RAGTruth format)
answer	string	LLM-generated summary to evaluate
labels	list	Span annotations with character offsets and RAGTruth labels
task_type	string	Task type (`Summary` - matches RAGTruth)
split	string	Data split (train)
dataset	string	Source dataset identifier
language	string	Language code (en)

Label Types

Label	Description	Example
Evident Conflict	Directly contradicts the source	"Anne Frank survived" when document says she died
Evident Baseless Info	Information not present in source	Adding dates/facts not mentioned
Subtle Conflict	Indirectly contradicts via implication	Misattributing quotes or actions
Subtle Baseless Info	Plausible but unverifiable additions	"likely" claims without evidence

Detection Method

Used Qwen2.5-72B-Instruct via vLLM for semantic span detection:

python

prompt = """Analyze the summary for hallucinations.

DOCUMENT:
{document}

SUMMARY:
{summary}

For each hallucinated span, provide:
- exact_text: the hallucinated phrase
- label: Evident Conflict | Evident Baseless Info | Subtle Conflict | Subtle Baseless Info
- reason: brief explanation
"""

Why LLM over NLI?

Aspect	NLI Approach	LLM Approach
Granularity	Sentence-level only	Sub-sentence spans
Context	Limited window	Full document context
Labels	Binary (contradiction/entailment)	Multi-class semantic
Understanding	Pattern matching	Semantic reasoning

Validation

All spans were validated for:

—✅ Valid indices - within answer bounds
—✅ Correct extraction - extracted text matches span
—✅ Partial coverage - not marking entire responses
—✅ Word boundaries - no mid-word cuts

Use Cases

—🎯 Train token-level hallucination detectors with RAGTruth compatibility
—📊 Augment RAGTruth for more diverse training data
—🔬 Research on fine-grained hallucination types
—⚖️ Multi-label classification (4 hallucination types)
—📝 Summarization-focused hallucination detection

Compatibility

This dataset uses the same label schema as RAGTruth, making it suitable for:

—Combined training with RAGTruth
—Transfer learning from RAGTruth models
—Evaluation using RAGTruth benchmarks

—llm-semantic-router/modernbert-base-32k-haldetect - 32K hallucination detector
—RAGTruth - Primary span-level dataset
—HaluEval - Original binary dataset
—LettuceDetect - Hallucination detection framework

Citation

bibtex

@misc{halueval_llm_spans_2026,
  title={HaluEval LLM-Detected Span-Level Dataset},
  author={llm-semantic-router},
  year={2026},
  howpublished={Hugging Face Hub},
  note={Converted from HaluEval using Qwen2.5-72B-Instruct for RAGTruth-compatible span detection}
}

License

MIT License (same as original HaluEval)

Changelog

v2.1 (2026-01)

—Normalized prompts to match RAGTruth format: Summarize the following text within 150 words:
—Fixed task_type: summarization → Summary (matches RAGTruth)
—Removed original_prompt field (now 7 fields, matching RAGTruth exactly)
—Ready for combined training with RAGTruth

v2.0 (2026-01)

—Replaced NLI-based conversion with LLM-based span detection (Qwen2.5-72B-Instruct)
—Added 4 RAGTruth-compatible label types
—Improved span granularity (99.4% partial spans)
—Validated all span indices

v1.0 (2024)

—Initial NLI-based conversion using DeBERTa-FEVER-ANLI