zhaoli/tac-exp-dev
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add regime gate walk-forward test (EVIDENCE#050)

Browse Source 
- 3 detector types (dispersion/vol/HMM) × 14 configs across 5 years
- Dispersion gates: 0% trip rate everywhere (dead)
- Vol gates: trip differential +32-47pp but destroy returns in good years
- HMM gates: +6pp differential, hmm_0.7 improves 2023/2025 but kills 2026
- Guard candidate regime gate REFUTED (ch 11)
- Script: book/scripts/regime_gate_bt.py
- Results: book/data/regime_gate/regime_gate_trip_rates.csv
This commit is contained in:
zhaoli
2026-08-20 22:14:31 +00:00
parent 4831a2770c
commit 718b048df9
6 changed files with 1636 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
book/EVIDENCE.md
+1
View File
@@ -78,6 +78,7 @@ Experiments 8–18 record metrics under a legacy schema (`ls_sharpe`, `maxdd_wit
|----|-------|--------|-----------|
| EVIDENCE#048 | Streaming IC circuit-breaker (`ic_min_rankic`, `ICGateTopkDropoutStrategy` in `tac_qlib/contrib/strategy/ic_gate.py`) trip-rate study: with thresholds 0.02–0.06, the gate trips on 25–50% of days in every year (2021–2026), freezing TopkDropout's rotation out of losers. A gate that trips every year cannot separate good years from bad. Do not deploy live. | ad-hoc scripted study on exp 52/53 pred/label artifacts, `tac_qlib/tac_qlib/contrib/strategy/ic_gate.py`, `tac_qlib/tac_qlib/risk_limits.py` | yes — guard candidate 3 REFUTED |
| EVIDENCE#049 | Perturbation stress test on Config A 2026 (exp 52, pred from run `9f98ea5c`): same signal, varying topk (5/10/15), n_drop (1/2/3), costs (base/high/5×base). **topk**: 10 optimal (32.8% raw, Sharpe 1.98); 5 loses ~0.5pp, 15 loses ~6.5pp. **n_drop**: 1 optimal; 2 loses ~6pp, 3 loses ~4pp. **costs**: immaterial — 5× cost increase (25bp/35bp/$15) drops return only 0.17pp (32.84%→32.67%). maxDD stable −5.8% to −7.0% across all perturbations. **Within the 2026 window the edge is robust to parameter perturbation.** The problem remains that it does not exist in other windows (ch 11). | ad-hoc rd_backtest grid on exp 52 pred.pkl, `book/data/perturbation/config_a_2026_sensitivity.json` | yes — within-window robustness confirmed |
| EVIDENCE#050 | Regime gate walk-forward test across 5 years (2021–2026): three detector types (dispersion, vol, HMM) × 14 configs. **Dispersion gates**: 0% trip rate everywhere — CS std of 22d returns never crosses any threshold. **Vol gates** (best: `vol_low_max20`): opens 92% in 2026 vs 60% in bad years (+32pp differential), but 2026 gated return collapses from +25.5% to +4.3% — the gate closes on profitable days. **HMM gates** (best: `hmm_0.7`): opens 37% in 2026 vs 31% in bad years (+6pp differential), 2026 return drops from +25.5% to +10.8%. No detector type achieves the goal of selective protection: tripping more in bad years while preserving good-year returns. The gate measures current market state, not whether yesterday's signals will predict today's returns. | scripted simulation: `book/scripts/regime_gate_bt.py`, results `book/data/regime_gate/regime_gate_trip_rates.csv`, pred.pkl from exp 52 (2024–2026) and exp 56 (2021, 2023) | yes — guard candidate regime gate REFUTED |
## External references (book/references/)