nutrientdocs/doc-split-benchmark

1---2license: cc-by-sa-4.03pretty_name: Doc-Split Benchmark (page-stream segmentation)4task_categories: [image-classification, text-classification]5language: [multilingual]6tags: [page-stream-segmentation, document-boundary-detection, document-ai, benchmark]7size_categories: [1K<n<10K]8configs:9- config_name: our20010  data_files: [{split: test, path: our200/test-*.parquet}]11---12 13# Doc-Split Benchmark14 15The **evaluation slice** for page-stream segmentation — the exact set behind the16[leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-split-leaderboard) and the cloud-VLM17comparison. Self-contained (page images embedded), with a reference scorer so results are reproducible.18This is the *benchmark*, not the training corpus (which stays private).19 20- 🏆 **Leaderboard:** [doc-split-leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-split-leaderboard)21- 🎯 **Demo:** [doc-split-demo](https://huggingface.co/spaces/nutrientdocs/doc-split-demo)22- 🟢 **Model:** [doc-split-mini-e5](https://huggingface.co/nutrientdocs/doc-split-mini-e5) (open weights)23- 🌍 **OpenPSS cuts:** [openpss-mirror](https://huggingface.co/datasets/nutrientdocs/openpss-mirror) (SHORT/LONG, self-contained)24 25## Cuts26 27| Cut | Where | Notes |28|---|---|---|29| **our-200** | this repo (`our200` config) | 200 license-clean synthetic-concat streams, ~78% boundary rate |30| **OpenPSS short / long** | [openpss-mirror](https://huggingface.co/datasets/nutrientdocs/openpss-mirror) | Dutch; sparse- & dense-boundary regimes |31| TABME++ / Tobacco800 | upstream (not redistributed) | reported for reference; licenses/gating prevent re-hosting |32 33## Load34 35```python36from datasets import load_dataset37ds = load_dataset("nutrientdocs/doc-split-benchmark", "our200", split="test")38# one row = one page; group by stream_id, order by position.39```40 41## Schema42 43| Field | Type | Meaning |44|---|---|---|45| `stream_id` | string | groups pages into one ordered stream |46| `position` | int | page index within the stream (0-based) |47| `boundary` | int | **1 = starts a new document** (position 0 always 1) |48| `page_text` | string | page OCR text (may be empty) |49| `image` | image | page image (512px) |50| `source` | string | provenance |51 52## How to score53 54Metric: **boundary page-F1** (internal positions, page 0 forced) + Cohen's **κ**, WindowDiff, Pk.55The reference scorer ships in this repo:56 57```bash58python score.py --pred your_preds.json      # preds: {stream_id: [0/1 per page]}59```60 61`score.py` uses `docsplit.eval.metrics.aggregate` (bundled). κ is reported because a degenerate62predict-none/all can look fine on F1 but scores κ ≈ 0 on the sparse cuts. See `SUBMISSION.md` to add a row to63the leaderboard.64 65`manifest.json` records provenance (per-source stream/page counts, boundary rates, seed).66 67## License & attribution68 69CC-BY-SA-4.0 for the our-200 packaging (Nutrient-generated synthetic concatenations). OpenPSS cuts are in70`openpss-mirror` under their upstream terms. TABME++/Tobacco800 are referenced, not redistributed.71 72## About the author73 74<a href="https://nutrient.io/">75  <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" />76</a>77 78This project is maintained and funded by [Nutrient](https://nutrient.io/) - The #1 PDF SDK library for viewing, editing, eSigning, and more.79