AgenticFinLab/PortBench-RawData

1---2language:3- en4license: mit5size_categories:6- 1GB < n < 10GB7tags:8- finance9- portfolio-management10- multi-asset11- benchmark12- LLM13- correlation14task_categories:15- other16---17 18[![Paper](https://img.shields.io/badge/arXiv-2605.27887-b31b1b)](https://arxiv.org/abs/2605.27887)19[![Code](https://img.shields.io/badge/GitHub-PortBench-blue)](https://github.com/AgenticFinLab/portbench)20[![Homepage](https://img.shields.io/badge/Homepage-portbench.github.io-orange)](https://portbench.github.io/)21 22# PortBench-RawData23 24This repository contains the **raw collected data and preprocessed asset files** for [PortBench](https://github.com/AgenticFinLab/portbench). 25 26The data spans **2015–2025** across **six heterogeneous asset classes**: Equities, Bonds, Commodities, Real Estate, Cryptocurrency, and Cash.27 28---29 30## Repository Structure31 32```33PortBench-RawData/34├── raw_data/                          # Raw collected data (~4.6 GB)35│   ├── fred/                          # FRED macroeconomic indicators (60 series)36│   │   ├── bonds/                     # Yield curve, credit spreads, TIPS37│   │   ├── cash/                      # Fed funds rate, CPI, GDP, employment38│   │   ├── commodities/               # Oil, gold, agriculture spot indices39│   │   └── real_estate/               # Case-Shiller, HPI, REIT indices40│   ├── kaggle/                        # Kaggle supplementary data (~4 GB)41│   │   ├── commodities/               # Commodity futures and spot prices42│   │   ├── cryptocurrency/            # Crypto OHLCV data43│   │   ├── equities/                  # Stock data with news text44│   │   └── real_estate/               # REIT and property data45│   ├── sec/                           # SEC EDGAR filings46│   │   └── equities/                  # 10-K, 10-Q filings for US equities47│   ├── yahoo/                         # Yahoo Finance price data48│   │   ├── bonds/                     # Bond ETF prices and yields49│   │   ├── cash/                      # Money market and treasury ETF data50│   │   ├── commodities/               # Commodity ETF data51│   │   ├── cryptocurrency/            # Crypto ETF and trust data52│   │   ├── equities/                  # 72 tickers (broad-market, sector, factor ETFs)53│   │   └── real_estate/               # REIT ETF data54│   └── metadata.json                  # Dataset metadata summary55│56└── processed/                         # Preprocessed asset data57    ├── equities.csv                   # 126 equity tickers, aligned daily58    ├── bonds.csv                      # 15 bond series, aligned daily59    ├── commodities.csv                # 16 commodity series, aligned daily60    ├── real_estate.csv                # 10 real estate series, aligned daily61    ├── cryptocurrency.csv             # 12 cryptocurrency series, aligned daily62    ├── cash.csv                       # 4 cash equivalent series, aligned daily63    ├── correlation_matrix.csv         # 183×183 Pearson correlation matrix64    ├── asset_class_map.json           # Ticker-to-asset-class mapping65    └── time_ranges.json               # Per-ticker date coverage ranges66```67 68---69 70## Data Sources71 72| Source | Coverage | Content | Tickers/Series |73|--------|----------|---------|----------------|74| [Yahoo Finance](https://finance.yahoo.com/) | 2015–2025 | Daily OHLCV, adjusted close, volume for ETFs and stocks across all 6 asset classes | 72 tickers |75| [FRED](https://fred.stlouisfed.org/) | 2015–2025 | Macroeconomic indicators: yield curve (DGS1–DGS30), TIPS real yields (DFII5/10/30), breakeven inflation (T5YIE/T10YIE), Fed funds rate (DFF/FEDFUNDS), CPI (CPIAUCSL/CPILFESL), GDP, employment (PAYEMS), housing (CSUSHPINSA), VIX | 60 series |76| [Kaggle](https://www.kaggle.com/) | 2015–2025 | Supplementary cryptocurrency OHLCV, commodity futures, equities with news sentiment, real estate data | ~45 datasets |77| [SEC EDGAR](https://www.sec.gov/edgar/) | 2015–2025 | 10-K and 10-Q filings for US equities, parsed text | Equity filings |78 79---80 81## Coverage by Asset Class82 83| Asset Class | Raw Tickers | Processed | Role in Portfolio |84|-------------|------------|-----------|-------------------|85| **Equities** | 127 | 126 | Return engine; diversified via sector/factor ETFs (SPY, QQQ, XLE, XLF, VXUS, etc.) |86| **Bonds** | 15 | 15 | Fixed-income hedging; Treasury, corporate, high-yield (TLT, IEF, HYG, LQD, etc.) |87| **Commodities** | 16 | 16 | Inflation hedge; gold, oil, natural gas, agriculture (GLD, USO, UNG, DBC, etc.) |88| **Real Estate** | 10 | 10 | Diversification; REIT sector ETFs (VNQ, IYR, SCHH, etc.) |89| **Cryptocurrency** | 12 | 12 | High-risk allocation; major + mid-cap (BTC, ETH, SOL, DOGE, etc.) |90| **Cash** | 4 | 4 | Capital preservation; money market, short-term Treasuries (BIL, SGOV, SHV) |91 92Cross-asset correlations exhibit the key structural property exploited by PortBench's dual-layer scoring: **intra-class correlations are strongly positive** (0.4–0.6+), while **inter-class correlations are near-zero or negative**, meaning true diversification requires cross-class allocation, not just many tickers within one class.93 94---95 96## Preprocessing Details97 98- **Calendar alignment**: All series aligned to a common business-day calendar; gaps ≤5 days forward-filled; longer gaps retained as NaN for pairwise-complete correlation estimation.99- **Market regime labels**: Each asset class labeled as bull/bear/sideways/crisis using MA crossover (50/200-day) + 15% max-drawdown crisis threshold.100- **Data splits**: Train (2015–2022), Validation (2023–2024), Test (2025), with year-end boundaries.101- **Correlation matrix**: 183×183 Pearson correlation matrix computed from daily simple returns over the full training period using pairwise-complete observations; frozen and not re-estimated.102 103---104 105## Related Datasets106 107This repository contains the raw data foundation. See also:108 109- [PortBench-Market](https://huggingface.co/datasets/AgenticFinLab/PortBench-Market) — The processed market base dataset (`portbench.csv`) with all six asset classes merged at daily frequency, plus visualization figures.110- [PortBench-QA](https://huggingface.co/datasets/AgenticFinLab/PortBench-QA) — 6,269 question-answer pairs across 7 templates (T1–T7) probing correlation-based financial reasoning, with train/val/test splits.111 112Both are part of the [PortBench collection](https://huggingface.co/collections/AgenticFinLab/portbench).113 114---115 116## Citation117 118```bibtex119@article{zhao2026portbench,120  title={PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management},121  author={Zhao, Yuxuan and Chen, Sijia and Su, Ningxin},122  journal={arXiv preprint arXiv:2605.27887},123  year={2026}124}125```126 127---128 129## License130 131This dataset is released under the MIT License. Data sourced from Yahoo Finance, FRED, Kaggle, and SEC EDGAR is subject to their respective terms of service.