uv-scripts/dataset-stats
Dataset Statistics UV scripts for analyzing HuggingFace datasets using streaming mode. Scripts finepdfs-stats.py - Temporal Educational Quality Analysis Analyze educational quality trends across CommonCrawl dumps using Polars streaming. Answers: "Is the web getting more educational over time?" Features: Polars streaming (no download of 300GB+ dataset) Temporal analysis across 106 CommonCrawl dumps (2013-2025) ASCII chart visualizations Uploads… See the full description on the dataset page: https://huggingface.co/datasets/uv-scripts/dataset-stats.
Sync from GitHub via hub-sync
Add hf-jobs tag to README frontmatter
fix: Use sink_parquet for true streaming (minimal memory)
feat: Use IterableDataset for true streaming (no disk writes)
fix: Add memory controls to prevent OOM on HF Jobs
Add long-context PDF extraction script
Add link to Polars HF Hub docs
Update README.md
Simplify README to only document existing script
Add finepdfs-stats.py documentation to README
Make card more generic for different language scopes
Add .gitignore for local output files
Remove local test files
Add uv-script tag, fix Key Finding to use yearly averages
Fix: define configs properly in README for viewer
Fix: use exist_ok=True for create_repo
Add ASCII trend charts to temporal analysis
Add finepdfs-stats.py - Polars streaming aggregation demo
initial commit