Datasetpublic

uv-scripts/dataset-stats

Dataset Statistics UV scripts for analyzing HuggingFace datasets using streaming mode. Scripts finepdfs-stats.py - Temporal Educational Quality Analysis Analyze educational quality trends across CommonCrawl dumps using Polars streaming. Answers: "Is the web getting more educational over time?" Features: Polars streaming (no download of 300GB+ dataset) Temporal analysis across 106 CommonCrawl dumps (2013-2025) ASCII chart visualizations Uploads… See the full description on the dataset page: https://huggingface.co/datasets/uv-scripts/dataset-stats.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

4likes65downloads

Dataset Card

Dataset Statistics

UV scripts for analyzing HuggingFace datasets using streaming mode.

Scripts

`finepdfs-stats.py` - Temporal Educational Quality Analysis

Analyze educational quality trends across CommonCrawl dumps using Polars streaming. Answers: "Is the web getting more educational over time?"

Features:

—Polars streaming (no download of 300GB+ dataset)
—Temporal analysis across 106 CommonCrawl dumps (2013-2025)
—ASCII chart visualizations
—Uploads results to HF Hub with auto-generated dataset card
—Supports single language or all 70+ languages

Quick Examples:

bash

# Quick test (10K samples)
uv run https://huggingface.co/datasets/uv-scripts/dataset-stats/raw/main/finepdfs-stats.py \
    --limit 10000 --show-plan

# Analyze English PDFs
uv run https://huggingface.co/datasets/uv-scripts/dataset-stats/raw/main/finepdfs-stats.py \
    --output-repo username/my-stats

# Analyze ALL 70+ languages
uv run https://huggingface.co/datasets/uv-scripts/dataset-stats/raw/main/finepdfs-stats.py \
    --all-languages --output-repo username/my-stats

Run on HF Jobs (recommended for full dataset):

bash

hf jobs uv run \
    -s HF_TOKEN \
    -e HF_XET_HIGH_PERFORMANCE=1 \
    https://huggingface.co/datasets/uv-scripts/dataset-stats/raw/main/finepdfs-stats.py \
    -- --all-languages --output-repo username/finepdfs-temporal-stats

Example output: davanstrien/finepdfs-temporal-stats-all

Performance:

—50M docs in ~14 minutes (~60K docs/sec)
—Single scan using Polars HF Hub integration
—Works on HF Jobs CPU instances

Related Scripts

Check out other scripts in the uv-scripts organization:

—dataset-creation: Create datasets from PDFs and other formats
—vllm: GPU-accelerated classification and inference
—ocr: Document OCR using vision-language models

Why UV Scripts?

UV scripts are self-contained Python scripts that:

—Run with a single uv run command (no setup required)
—Include all dependencies in PEP 723 inline metadata
—Work seamlessly on both local machines and HF Jobs

Learn more about UV: https://docs.astral.sh/uv/