shash42/forecast-news
Forecast News Deduplicated daily news corpus used by forecast-sim and future-sim. Snapshot 31,859,020 articles 3,463 daily partitions Coverage: 2016-08-26 through 2026-08-31 Snapshot published: 2026-09-18 Stored data size: approximately 158.5 GiB The Parquet files are the canonical complete representation. The repository also contains daily JSONL files where available and compact headline JSON files used by article-browsing workflows. Layout Files… See the full description on the dataset page: https://huggingface.co/datasets/shash42/forecast-news.
Forecast News
Deduplicated daily news corpus used by forecast-sim and future-sim.
Snapshot
- 31,859,020 articles
- 3,463 daily partitions
- Coverage: 2016-08-26 through 2026-08-31
- Snapshot published: 2026-09-18
- Stored data size: approximately 158.5 GiB
The Parquet files are the canonical complete representation. The repository also contains daily JSONL files where available and compact headline JSON files used by article-browsing workflows.
Layout
Files are partitioned by date:
~~~text YYYY/MM/DD/articlesb0000.parquet YYYY/MM/DD/articles.jsonl YYYY/MM/DD/headlinesb0000.json ~~~
Parquet columns:
~~~text id, title, source, date, datepublish, datemodify, url, content, authors, description ~~~
A precomputed LanceDB search table for this corpus is published in shash42/forecast-news-embeddings.