ParallelLLC/algorithmic_trading

1---2title: Backtest Reality Check3emoji: 🎲4colorFrom: blue5colorTo: gray6sdk: gradio7sdk_version: 5.49.18app_file: app.py9pinned: true10license: apache-2.011short_description: Your backtest is probably lying to you. This proves it.12tags:13  - finance14  - quantitative-finance15  - algorithmic-trading16  - backtesting17  - statistics18  - time-series19---20 21# Backtest Reality Check22 23**Your backtest is probably lying to you.**24 25Pick a market and a trading rule. This Space runs the backtest — and then spends26the rest of its effort trying to prove the result was luck.27 28Most backtesting tools answer *"how much would this have made?"*. That is the easy29question, and the answer is almost always flattering. This one answers the question30you need before risking money: **how much of that was luck?**31 32## The four ways a backtest lies, and the test for each33 34| The lie | The test |35|---|---|36| The market had no structure to find | **Permutation test** — re-run your rule on hundreds of shuffled markets |37| You tried 200 things and reported the best | **Deflated Sharpe Ratio** — charge for every variant you tried |38| The parameters were fitted to the past | **PBO + walk-forward** — does the in-sample winner keep winning? |39| The edge is smaller than the costs | **Cost stress test** — triple the friction and see what survives |40 41Each contributes to a single **Reality Score** out of 100, with a grade from A to F.42The scale is deliberately harsh. Most strategies people post online score below 40.43 44## Two labs45 46**The Lab** validates a timing rule on one asset. **The Portfolio Lab** validates a47book that ranks many names — and it gets a harder null: we keep every date's gross48exposure, net exposure and position count exactly as they were and randomise only49**which name got which weight**. A book that beats that is picking names. One that50doesn't was being paid for style exposure you can buy in an ETF, which the factor51regression measures directly.52 53It also measures survivorship rather than assuming it away. A universe where every54name is still trading after ten years was chosen after the fact, and every number55computed on it is an upper bound.56 57## Try this first58 59Run the **Arena** tab on `SPY`. On most markets and most date ranges, plain60**buy & hold** tops the leaderboard, and the **coin flip** control out-ranks61several respectable-looking strategies. That is not a bug in the app — it is the62finding.63 64## How the permutation test works65 66We take the real price series and shuffle it. Each bar's gap, high, low, body and67volume are kept intact, but their **order** is destroyed. The result is a market68with the same volatility and the same fat tails, and no exploitable structure at69all. Then we re-run *your exact rule* on hundreds of these shuffled markets.70 71If your Sharpe ratio sits comfortably inside that cloud, your rule found nothing72a coin-flip market would not also have handed it.73 74## No look-ahead, by construction75 76A strategy emits a target exposure at each bar's close using only data up to that77bar. The engine holds `position[t] = target[t - lag]` with `lag >= 1`, so a signal78computed on Tuesday's close cannot earn Tuesday's move. That is the single line79where look-ahead could enter, and the test suite asserts it directly.80 81## Use it from Python82 83```python84from algotrader import LabConfig, run_lab85 86report = run_lab(LabConfig(symbol="SPY", strategy="sma_cross"))87print(report.verdict["grade"], report.verdict["score"])88print(report.permutation.p_value, report.dsr["dsr"], report.pbo["pbo"])89```90 91Or from the command line:92 93```bash94python -m algotrader.cli lab --symbol SPY --strategy donchian_breakout --permutations 50095python -m algotrader.cli arena --symbol BTC-USD96```97 98## Data99 100Live prices come from Yahoo Finance. When the network is unavailable or rate-limited,101the app falls back to a deterministic market simulator with regime switching, fat102tails and volatility clustering — and says so, clearly, on every result. The103statistics remain valid; they are just measured on a simulated market.104 105## References106 107- Bailey & López de Prado (2014), *The Deflated Sharpe Ratio*108- Bailey, Borwein, López de Prado & Zhu (2016), *The Probability of Backtest Overfitting*109- Masters (2018), *Permutation and Randomization Tests for Trading System Development*110 111---112 113Apache-2.0. Research tooling, not investment advice. Nothing here is a114recommendation to trade.115