Datasetpublic

konsman/quantum-physics-0.5-corpus

quantum-physics-0.5-corpus Dataset Description This is a domain-specific corpus created using ontology-guided filtering from FineWeb-Edu. Dataset Creation Source: HuggingFaceFW/fineweb-edu Filtering Method: Semantic similarity to subdomain centroids (embedding-based) Pipeline: Ontology-Guided Domain Corpus Builder Dataset Structure Each chunk contains: text: The text content (256-512 tokens) subdomain_id: Assigned subdomain… See the full description on the dataset page: https://huggingface.co/datasets/konsman/quantum-physics-0.5-corpus.

sourceHugging Facemitupdated 10mo agoView on Hugging Face

0likes30downloads

Dataset Card

quantum-physics-0.5-corpus

Dataset Description

This is a domain-specific corpus created using ontology-guided filtering from FineWeb-Edu.

Dataset Creation

—Source: HuggingFaceFW/fineweb-edu
—Filtering Method: Semantic similarity to subdomain centroids (embedding-based)
—Pipeline: Ontology-Guided Domain Corpus Builder

Dataset Structure

Each chunk contains:

—text: The text content (256-512 tokens)
—subdomain_id: Assigned subdomain
—similarity_score: Cosine similarity to subdomain centroid
—token_count: Number of tokens
—source_dataset: Original dataset name
—source_id: Original document ID
—chunk_index: Position within source document

Usage

python

from datasets import load_dataset

dataset = load_dataset("konsman/quantum-physics-0.5-corpus")

# Access filtered chunks
for chunk in dataset['train']:
    print(chunk['text'])
    print(chunk['subdomain_id'])
    print(chunk['similarity_score'])

License

MIT License

Citation

Generated using the Ontology-Guided Domain Corpus Builder pipeline.