zhaoli/tac-exp-dev
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ch11: add EVIDENCE#051 — comprehensive model search confirms regime gate robustness to model selection

Browse Source
This commit is contained in:
zhaoli
2026-08-20 22:38:52 +00:00
parent 718b048df9
commit 8f7ce7bd8e
2 changed files with 33 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
book/chapters/11-walk-forward-and-guards.md
+27 -1
View File
@@ -80,6 +80,31 @@ The 2026 edge is fragile *across* windows but robust *within* the 2026 window. A
Key takeaways: topk=10 is the sweet spot (topk=15 dilutes the signal by ~6.5pp). n_drop=1 is best; more rotation hurts. Costs are almost immaterial — even 5× base costs drop return by only 0.17pp, because the strategy is low-turnover and the gross edge is large. maxDD is stable (−5.8% to −7.0%) across all perturbations. **Within the one good window, the edge is not a parameter-tuning artifact.** The fragility is entirely across windows (regime dependence), not within them.
## Model search & robustness of the regime gate finding
The regime gate study (Guard 6, EVIDENCE#050) used pred.pkl files from exp 52 (Configs A/C, 2024–2026) and exp 56 (2021, 2023). A comprehensive query of all MLflow experiments confirms the regime gate finding is robust to model selection:
| Rank | Exp | Test Window | RankICIR | Net Return | Notes |
|------|-----|-------------|----------|------------|-------|
| 1 | 36 | 2026 only | **0.507** | −4.6% | 22d label, single-window |
| 2 | 44 | 2026 only | **0.507** | −4.9% | Same pred as #1 |
| 3 | 35 | 2026 only | **0.352** | −9.9% | 10d label |
| 4 | 51 | 2026 only | **0.352** | +1.2% | Same pred as #3 |
| 5 | 58 | 2025 only | **0.289** | −3.4% | Adaptive 2y |
| 6 | 11 | 2026 only | **0.276** | +3.1% | Single seed |
| 7 | 33 | 2026 only | **0.259** | −0.9% | 10 seeds |
| 8 | **52-C** | **2024–2026** | **0.244** | **+12.5%** | Walk-forward, weekly |
| 9 | **52-A** | **2024–2026** | **0.244** | **−1.4%** | Walk-forward, TopkDrop |
`PROVEN — EVIDENCE#051` (comprehensive `rd_exp_list` query, run metadata).
Key observations:
1. **The 22-day label models (exp 36/44) have the highest RankICIR (0.507) but negative returns** — high IC does not guarantee profitable trading. The 22d label predicts longer-horizon moves that don't translate to short-term alpha after costs.
2. **Most high-RankICIR models are single-window (2026 only)** — they lack the multi-year coverage needed for the regime gate study. Walk-forward coverage (2021–2026) is limited to exp 52 (2024–2026) and exp 56 (2021, 2023).
3. **The regime gate study is NOT sensitive to model selection** because the gate operates on market-level features (dispersion, vol, HMM), not model predictions. Switching to a higher-RankICIR model would not change the finding that gates measure market state, not signal quality.
4. **Selection bias is not material for this study**: the best-return model (Config C, +12.5%) also has the best RankICIR (0.244) among walk-forward configs. The RankICIR and returns rankings are concordant.
## Desk rules distilled from this chapter
1. Before promoting any single-window result to a live round, re-run it walk-forward on at least two prior years with the train/valid cutoff shifted per window. If the edge does not survive, it is a regime artifact, not a strategy.
@@ -142,4 +167,5 @@ The vol gates show the largest trip differential — they open on more days in 2
| `EVIDENCE#047` | exp 56, staleness analysis on the exp 53/54 pred/label artifacts, branch `exp/56-window-staleness-isolation-the-m2-sharpe` |
| Guard 3 (`ic_min_rankic`) | `tac_qlib/tac_qlib/contrib/strategy/ic_gate.py` (ICGateTopkDropoutStrategy), `tac_qlib/tac_qlib/risk_limits.py`; trip-rate study on exp 52/53 preds |
| `EVIDENCE#049` | Perturbation stress test on Config A 2026 (exp 52, pred `9f98ea5c`): topk/n_drop/cost grid, `book/data/perturbation/config_a_2026_sensitivity.json` |
| `EVIDENCE#050` | Regime gate walk-forward test (2021–2026): 3 detector types × 14 configs; scripted simulation `book/scripts/regime_gate_bt.py`, results `book/data/regime_gate/regime_gate_trip_rates.csv` |
| `EVIDENCE#050` | Regime gate walk-forward test (2021–2026): 3 detector types × 14 configs; scripted simulation `book/scripts/regime_gate_bt.py`, results `book/data/regime_gate/regime_gate_trip_rates.csv` |
| `EVIDENCE#051` | Comprehensive model search: all experiments ranked by RankICIR; regime gate study robust to model selection; `rd_exp_list` + `rd_exp_get_run` queries |