Add regime gate walk-forward test (EVIDENCE#050)
- 3 detector types (dispersion/vol/HMM) × 14 configs across 5 years - Dispersion gates: 0% trip rate everywhere (dead) - Vol gates: trip differential +32-47pp but destroy returns in good years - HMM gates: +6pp differential, hmm_0.7 improves 2023/2025 but kills 2026 - Guard candidate regime gate REFUTED (ch 11) - Script: book/scripts/regime_gate_bt.py - Results: book/data/regime_gate/regime_gate_trip_rates.csv
This commit is contained in:
zhaoli
@@ -70,6 +70,7 @@ The running scoreboard of every quantitative claim in the book. Updated per chap
|
|||||||
| Shorter training windows (1y/2y) recover the edge | REFUTED (every test year negative; only the growing window ever goes positive; mean annual excess ≈ −13% for every window length) | EVIDENCE#046 → exp 55 |
|
| Shorter training windows (1y/2y) recover the edge | REFUTED (every test year negative; only the growing window ever goes positive; mean annual excess ≈ −13% for every window length) | EVIDENCE#046 → exp 55 |
|
||||||
| The edge concentrates in fresh (low-staleness) predictions | REFUTED (every 90-day staleness bucket negative; freshest bucket most negative; 2025 gains are late-year at 336–397d staleness) | EVIDENCE#047 → exp 56 |
|
| The edge concentrates in fresh (low-staleness) predictions | REFUTED (every 90-day staleness bucket negative; freshest bucket most negative; 2025 gains are late-year at 336–397d staleness) | EVIDENCE#047 → exp 56 |
|
||||||
| The 2026 edge is a 2025–2026 regime artifact; no guard candidate recovers it out-of-sample | PROVEN | EVIDENCE#043–048 → exp 52–56 |
|
| The 2026 edge is a 2025–2026 regime artifact; no guard candidate recovers it out-of-sample | PROVEN | EVIDENCE#043–048 → exp 52–56 |
|
||||||
|
| A regime gate (dispersion/vol/HMM detector) selectively trades in profitable years | REFUTED (dispersion 0% trip everywhere; vol gates close on profitable days; HMM 37% trip in 2026 vs 31% in bad years — too weak to protect) | EVIDENCE#050 → ad-hoc simulation `book/scripts/regime_gate_bt.py` |
|
||||||
| Live capital should be sized for the mean (≈ −13% annual excess), not the 2026 tail | PROVEN (walk-forward) + HYPOTHESIS (forward-looking) | EVIDENCE#043–047 → exp 52–56 |
|
| Live capital should be sized for the mean (≈ −13% annual excess), not the 2026 tail | PROVEN (walk-forward) + HYPOTHESIS (forward-looking) | EVIDENCE#043–047 → exp 52–56 |
|
||||||
|
|
||||||
## Data & reproducibility
|
## Data & reproducibility
|
||||||
|
|||||||
@@ -78,6 +78,7 @@ Experiments 8–18 record metrics under a legacy schema (`ls_sharpe`, `maxdd_wit
|
|||||||
|----|-------|--------|-----------|
|
|----|-------|--------|-----------|
|
||||||
| EVIDENCE#048 | Streaming IC circuit-breaker (`ic_min_rankic`, `ICGateTopkDropoutStrategy` in `tac_qlib/contrib/strategy/ic_gate.py`) trip-rate study: with thresholds 0.02–0.06, the gate trips on 25–50% of days in every year (2021–2026), freezing TopkDropout's rotation out of losers. A gate that trips every year cannot separate good years from bad. Do not deploy live. | ad-hoc scripted study on exp 52/53 pred/label artifacts, `tac_qlib/tac_qlib/contrib/strategy/ic_gate.py`, `tac_qlib/tac_qlib/risk_limits.py` | yes — guard candidate 3 REFUTED |
|
| EVIDENCE#048 | Streaming IC circuit-breaker (`ic_min_rankic`, `ICGateTopkDropoutStrategy` in `tac_qlib/contrib/strategy/ic_gate.py`) trip-rate study: with thresholds 0.02–0.06, the gate trips on 25–50% of days in every year (2021–2026), freezing TopkDropout's rotation out of losers. A gate that trips every year cannot separate good years from bad. Do not deploy live. | ad-hoc scripted study on exp 52/53 pred/label artifacts, `tac_qlib/tac_qlib/contrib/strategy/ic_gate.py`, `tac_qlib/tac_qlib/risk_limits.py` | yes — guard candidate 3 REFUTED |
|
||||||
| EVIDENCE#049 | Perturbation stress test on Config A 2026 (exp 52, pred from run `9f98ea5c`): same signal, varying topk (5/10/15), n_drop (1/2/3), costs (base/high/5×base). **topk**: 10 optimal (32.8% raw, Sharpe 1.98); 5 loses ~0.5pp, 15 loses ~6.5pp. **n_drop**: 1 optimal; 2 loses ~6pp, 3 loses ~4pp. **costs**: immaterial — 5× cost increase (25bp/35bp/$15) drops return only 0.17pp (32.84%→32.67%). maxDD stable −5.8% to −7.0% across all perturbations. **Within the 2026 window the edge is robust to parameter perturbation.** The problem remains that it does not exist in other windows (ch 11). | ad-hoc rd_backtest grid on exp 52 pred.pkl, `book/data/perturbation/config_a_2026_sensitivity.json` | yes — within-window robustness confirmed |
|
| EVIDENCE#049 | Perturbation stress test on Config A 2026 (exp 52, pred from run `9f98ea5c`): same signal, varying topk (5/10/15), n_drop (1/2/3), costs (base/high/5×base). **topk**: 10 optimal (32.8% raw, Sharpe 1.98); 5 loses ~0.5pp, 15 loses ~6.5pp. **n_drop**: 1 optimal; 2 loses ~6pp, 3 loses ~4pp. **costs**: immaterial — 5× cost increase (25bp/35bp/$15) drops return only 0.17pp (32.84%→32.67%). maxDD stable −5.8% to −7.0% across all perturbations. **Within the 2026 window the edge is robust to parameter perturbation.** The problem remains that it does not exist in other windows (ch 11). | ad-hoc rd_backtest grid on exp 52 pred.pkl, `book/data/perturbation/config_a_2026_sensitivity.json` | yes — within-window robustness confirmed |
|
||||||
|
| EVIDENCE#050 | Regime gate walk-forward test across 5 years (2021–2026): three detector types (dispersion, vol, HMM) × 14 configs. **Dispersion gates**: 0% trip rate everywhere — CS std of 22d returns never crosses any threshold. **Vol gates** (best: `vol_low_max20`): opens 92% in 2026 vs 60% in bad years (+32pp differential), but 2026 gated return collapses from +25.5% to +4.3% — the gate closes on profitable days. **HMM gates** (best: `hmm_0.7`): opens 37% in 2026 vs 31% in bad years (+6pp differential), 2026 return drops from +25.5% to +10.8%. No detector type achieves the goal of selective protection: tripping more in bad years while preserving good-year returns. The gate measures current market state, not whether yesterday's signals will predict today's returns. | scripted simulation: `book/scripts/regime_gate_bt.py`, results `book/data/regime_gate/regime_gate_trip_rates.csv`, pred.pkl from exp 52 (2024–2026) and exp 56 (2021, 2023) | yes — guard candidate regime gate REFUTED |
|
||||||
|
|
||||||
## External references (book/references/)
|
## External references (book/references/)
|
||||||
|
|
||||||
|
|||||||
@@ -88,10 +88,48 @@ Key takeaways: topk=10 is the sweet spot (topk=15 dilutes the signal by ~6.5pp).
|
|||||||
4. Report account-based curves, not the blotter `return` field — the latter excludes initial cost and does not compound to the account.
|
4. Report account-based curves, not the blotter `return` field — the latter excludes initial cost and does not compound to the account.
|
||||||
5. When the mean annual excess is negative in every configuration, cut size until the live window demonstrates the regime is back.
|
5. When the mean annual excess is negative in every configuration, cut size until the live window demonstrates the regime is back.
|
||||||
|
|
||||||
|
### Guard 6: Regime gate (dispersion / vol / HMM)
|
||||||
|
|
||||||
|
`PROVEN — EVIDENCE#050`
|
||||||
|
|
||||||
|
If the edge is regime-dependent, the most direct guard is a regime detector that opens on good years and closes on bad years. We test three detector types, each producing a daily boolean (trade / don't trade):
|
||||||
|
|
||||||
|
| Detector | Logic |
|
||||||
|
|----------|-------|
|
||||||
|
| **dispersion** | CS std of 22-day rolling returns < threshold (low dispersion → calm market → trade) |
|
||||||
|
| **vol** | CS mean of 22-day rolling realized vol within a band (mid-range vol → trade) |
|
||||||
|
| **HMM** | 2-state Gaussian HMM posterior for regime 1 (productive regime) > threshold |
|
||||||
|
|
||||||
|
Each detector is applied as a daily gate on top of the weekly-rebalance TopkDropout (topk=10, n_drop=1, yesterday's scores). We run 14 configs across 5 walk-forward windows (2021–2026), tracking trip rate (fraction of days gate is open) and gated return.
|
||||||
|
|
||||||
|
**Trip rates (2026 vs bad years 2021/2023/2024):**
|
||||||
|
|
||||||
|
| Gate | 2026 trip | Bad-years avg | Differential |
|
||||||
|
|------|-----------|---------------|-------------|
|
||||||
|
| `vol_low_max20` | 92% | 60% | +32pp |
|
||||||
|
| `vol_low_max25` | 63% | 16% | +47pp |
|
||||||
|
| `hmm_0.7` | 37% | 31% | +6pp |
|
||||||
|
| All dispersion | 0% | 0% | 0pp |
|
||||||
|
|
||||||
|
The vol gates show the largest trip differential — they open on more days in 2026 than in bad years. But the gate **closes on the wrong days**: when the gate is open only 63% of the time (vol_low_max25), the 2026 return collapses from +25.5% to −1.6%. The gate eliminates the profitable days along with the bad ones.
|
||||||
|
|
||||||
|
**Gated returns:**
|
||||||
|
|
||||||
|
| Gate | 2026 base | 2026 gated | 2023 base | 2023 gated | 2025 base | 2025 gated |
|
||||||
|
|------|-----------|------------|-----------|------------|-----------|------------|
|
||||||
|
| `vol_low_max20` | +25.5% | +4.3% | −4.8% | −5.3% | +17.8% | +14.5% |
|
||||||
|
| `hmm_0.7` | +25.5% | +10.8% | −4.8% | +0.6% | +17.8% | +26.8% |
|
||||||
|
|
||||||
|
`hmm_0.7` has the most interesting profile: it **improves** 2023 (−4.8% → +0.6%) and 2025 (+17.8% → +26.8%), but **destroys** 2026 (+25.5% → +10.8%). The gate's Sharpe is inflated (1.78 in 2021) because it spends most of its time in cash — the Sharpe measures "active days only" and ignores the flat periods.
|
||||||
|
|
||||||
|
**Why none of these gates work:** The gate answers *"is the market calm right now?"* — but the right question is *"will today's signal be profitable tomorrow?"* These are different questions. A calm market can produce bad signals (low vol but wrong factor regime), and a volatile market can produce good signals (high vol but correct factor direction). The gate needs to predict **signal quality**, not **market state**.
|
||||||
|
|
||||||
|
`TODO(evidence-needed: a retrospective signal-quality gate — did yesterday's topk signals predict today's returns? — tested out-of-sample)`
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
- `TODO(evidence-needed: a live window that matches the 2026 label regime, to test whether the edge returns when the regime returns)`
|
- `TODO(evidence-needed: a live window that matches the 2026 label regime, to test whether the edge returns when the regime returns)`
|
||||||
- `TODO(evidence-needed: a regime-change detector that is causal (no lookahead) and demonstrably selects the 2026 window before the fact — none of the five guards did)`
|
- `TODO(evidence-needed: a retrospective signal-quality gate — did yesterday's topk signals predict today's returns? — tested out-of-sample)`
|
||||||
|
|
||||||
## Evidence cited in this chapter
|
## Evidence cited in this chapter
|
||||||
|
|
||||||
@@ -104,3 +142,4 @@ Key takeaways: topk=10 is the sweet spot (topk=15 dilutes the signal by ~6.5pp).
|
|||||||
| `EVIDENCE#047` | exp 56, staleness analysis on the exp 53/54 pred/label artifacts, branch `exp/56-window-staleness-isolation-the-m2-sharpe` |
|
| `EVIDENCE#047` | exp 56, staleness analysis on the exp 53/54 pred/label artifacts, branch `exp/56-window-staleness-isolation-the-m2-sharpe` |
|
||||||
| Guard 3 (`ic_min_rankic`) | `tac_qlib/tac_qlib/contrib/strategy/ic_gate.py` (ICGateTopkDropoutStrategy), `tac_qlib/tac_qlib/risk_limits.py`; trip-rate study on exp 52/53 preds |
|
| Guard 3 (`ic_min_rankic`) | `tac_qlib/tac_qlib/contrib/strategy/ic_gate.py` (ICGateTopkDropoutStrategy), `tac_qlib/tac_qlib/risk_limits.py`; trip-rate study on exp 52/53 preds |
|
||||||
| `EVIDENCE#049` | Perturbation stress test on Config A 2026 (exp 52, pred `9f98ea5c`): topk/n_drop/cost grid, `book/data/perturbation/config_a_2026_sensitivity.json` |
|
| `EVIDENCE#049` | Perturbation stress test on Config A 2026 (exp 52, pred `9f98ea5c`): topk/n_drop/cost grid, `book/data/perturbation/config_a_2026_sensitivity.json` |
|
||||||
|
| `EVIDENCE#050` | Regime gate walk-forward test (2021–2026): 3 detector types × 14 configs; scripted simulation `book/scripts/regime_gate_bt.py`, results `book/data/regime_gate/regime_gate_trip_rates.csv` |
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
window,gate,start,end,trade_dates,gate_open,gate_closed,trip_rate,base_ann,base_sharpe,base_maxDD,gated_ann,gated_sharpe,gated_maxDD
|
||||||
|
2026,disp_0.010,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
|
||||||
|
2026,disp_0.015,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
|
||||||
|
2026,disp_0.020,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
|
||||||
|
2026,disp_0.025,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
|
||||||
|
2026,disp_0.030,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
|
||||||
|
2026,vol_low_max15,2026-01-04,2026-08-19,157,0,157,1.0,0.255023,1.4465,-0.080671,0.0,0.0,0.0
|
||||||
|
2026,vol_low_max20,2026-01-04,2026-08-19,157,12,145,0.9236,0.255023,1.4465,-0.080671,0.043049,1.1515,-0.023108
|
||||||
|
2026,vol_low_max25,2026-01-04,2026-08-19,157,58,99,0.6306,0.255023,1.4465,-0.080671,-0.016384,-0.1645,-0.108398
|
||||||
|
2026,vol_10_25,2026-01-04,2026-08-19,157,58,99,0.6306,0.255023,1.4465,-0.080671,-0.016384,-0.1645,-0.108398
|
||||||
|
2026,vol_10_30,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
|
||||||
|
2026,hmm_0.3,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
|
||||||
|
2026,hmm_0.5,2026-01-04,2026-08-19,157,153,4,0.0255,0.255023,1.4465,-0.080671,0.149795,0.9136,-0.080671
|
||||||
|
2026,hmm_0.7,2026-01-04,2026-08-19,157,99,58,0.3694,0.255023,1.4465,-0.080671,0.10771,1.0292,-0.057741
|
||||||
|
2026,hmm_0.9,2026-01-04,2026-08-19,157,0,157,1.0,0.255023,1.4465,-0.080671,0.0,0.0,0.0
|
||||||
|
2025,disp_0.010,2025-01-02,2025-12-31,250,250,0,0.0,0.177515,0.8573,-0.217417,0.177515,0.8573,-0.217417
|
||||||
|
2025,disp_0.015,2025-01-02,2025-12-31,250,250,0,0.0,0.177515,0.8573,-0.217417,0.177515,0.8573,-0.217417
|
||||||
|
2025,disp_0.020,2025-01-02,2025-12-31,250,250,0,0.0,0.177515,0.8573,-0.217417,0.177515,0.8573,-0.217417
|
||||||
|
2025,disp_0.025,2025-01-02,2025-12-31,250,250,0,0.0,0.177515,0.8573,-0.217417,0.177515,0.8573,-0.217417
|
||||||
|
2025,disp_0.030,2025-01-02,2025-12-31,250,250,0,0.0,0.177515,0.8573,-0.217417,0.177515,0.8573,-0.217417
|
||||||
|
2025,vol_low_max15,2025-01-02,2025-12-31,250,0,250,1.0,0.177515,0.8573,-0.217417,0.0,0.0,0.0
|
||||||
|
2025,vol_low_max20,2025-01-02,2025-12-31,250,90,160,0.64,0.177515,0.8573,-0.217417,0.144812,2.0398,-0.0378
|
||||||
|
2025,vol_low_max25,2025-01-02,2025-12-31,250,178,72,0.288,0.177515,0.8573,-0.217417,0.127917,1.0934,-0.11916
|
||||||
|
2025,vol_10_25,2025-01-02,2025-12-31,250,178,72,0.288,0.177515,0.8573,-0.217417,0.127917,1.0934,-0.11916
|
||||||
|
2025,vol_10_30,2025-01-02,2025-12-31,250,212,38,0.152,0.177515,0.8573,-0.217417,0.173997,1.2001,-0.130753
|
||||||
|
2025,hmm_0.3,2025-01-02,2025-12-31,250,244,6,0.024,0.177515,0.8573,-0.217417,0.221577,1.364,-0.14578
|
||||||
|
2025,hmm_0.5,2025-01-02,2025-12-31,250,233,17,0.068,0.177515,0.8573,-0.217417,0.280047,1.9219,-0.070373
|
||||||
|
2025,hmm_0.7,2025-01-02,2025-12-31,250,185,65,0.26,0.177515,0.8573,-0.217417,0.268006,2.4144,-0.070373
|
||||||
|
2025,hmm_0.9,2025-01-02,2025-12-31,250,0,250,1.0,0.177515,0.8573,-0.217417,0.0,0.0,0.0
|
||||||
|
2024,disp_0.010,2024-01-02,2024-12-31,253,253,0,0.0,0.08229,0.5594,-0.10685,0.08229,0.5594,-0.10685
|
||||||
|
2024,disp_0.015,2024-01-02,2024-12-31,253,253,0,0.0,0.08229,0.5594,-0.10685,0.08229,0.5594,-0.10685
|
||||||
|
2024,disp_0.020,2024-01-02,2024-12-31,253,253,0,0.0,0.08229,0.5594,-0.10685,0.08229,0.5594,-0.10685
|
||||||
|
2024,disp_0.025,2024-01-02,2024-12-31,253,253,0,0.0,0.08229,0.5594,-0.10685,0.08229,0.5594,-0.10685
|
||||||
|
2024,disp_0.030,2024-01-02,2024-12-31,253,253,0,0.0,0.08229,0.5594,-0.10685,0.08229,0.5594,-0.10685
|
||||||
|
2024,vol_low_max15,2024-01-02,2024-12-31,253,0,253,1.0,0.08229,0.5594,-0.10685,0.0,0.0,0.0
|
||||||
|
2024,vol_low_max20,2024-01-02,2024-12-31,253,86,167,0.6601,0.08229,0.5594,-0.10685,0.000386,0.0056,-0.063954
|
||||||
|
2024,vol_low_max25,2024-01-02,2024-12-31,253,179,74,0.2925,0.08229,0.5594,-0.10685,0.081492,0.6993,-0.070375
|
||||||
|
2024,vol_10_25,2024-01-02,2024-12-31,253,179,74,0.2925,0.08229,0.5594,-0.10685,0.081492,0.6993,-0.070375
|
||||||
|
2024,vol_10_30,2024-01-02,2024-12-31,253,234,19,0.0751,0.08229,0.5594,-0.10685,0.089809,0.6617,-0.068417
|
||||||
|
2024,hmm_0.3,2024-01-02,2024-12-31,253,253,0,0.0,0.08229,0.5594,-0.10685,0.08229,0.5594,-0.10685
|
||||||
|
2024,hmm_0.5,2024-01-02,2024-12-31,253,249,4,0.0158,0.08229,0.5594,-0.10685,0.136453,0.959,-0.081611
|
||||||
|
2024,hmm_0.7,2024-01-02,2024-12-31,253,213,40,0.1581,0.08229,0.5594,-0.10685,0.172432,1.6081,-0.051163
|
||||||
|
2024,hmm_0.9,2024-01-02,2024-12-31,253,0,253,1.0,0.08229,0.5594,-0.10685,0.0,0.0,0.0
|
||||||
|
2023,disp_0.010,2023-01-03,2023-12-29,250,250,0,0.0,-0.047644,-0.2738,-0.197856,-0.047644,-0.2738,-0.197856
|
||||||
|
2023,disp_0.015,2023-01-03,2023-12-29,250,250,0,0.0,-0.047644,-0.2738,-0.197856,-0.047644,-0.2738,-0.197856
|
||||||
|
2023,disp_0.020,2023-01-03,2023-12-29,250,250,0,0.0,-0.047644,-0.2738,-0.197856,-0.047644,-0.2738,-0.197856
|
||||||
|
2023,disp_0.025,2023-01-03,2023-12-29,250,250,0,0.0,-0.047644,-0.2738,-0.197856,-0.047644,-0.2738,-0.197856
|
||||||
|
2023,disp_0.030,2023-01-03,2023-12-29,250,250,0,0.0,-0.047644,-0.2738,-0.197856,-0.047644,-0.2738,-0.197856
|
||||||
|
2023,vol_low_max15,2023-01-03,2023-12-29,250,0,250,1.0,-0.047644,-0.2738,-0.197856,0.0,0.0,0.0
|
||||||
|
2023,vol_low_max20,2023-01-03,2023-12-29,250,103,147,0.588,-0.047644,-0.2738,-0.197856,-0.052602,-0.5107,-0.148726
|
||||||
|
2023,vol_low_max25,2023-01-03,2023-12-29,250,245,5,0.02,-0.047644,-0.2738,-0.197856,-0.077097,-0.4555,-0.179606
|
||||||
|
2023,vol_10_25,2023-01-03,2023-12-29,250,245,5,0.02,-0.047644,-0.2738,-0.197856,-0.077097,-0.4555,-0.179606
|
||||||
|
2023,vol_10_30,2023-01-03,2023-12-29,250,250,0,0.0,-0.047644,-0.2738,-0.197856,-0.047644,-0.2738,-0.197856
|
||||||
|
2023,hmm_0.3,2023-01-03,2023-12-29,250,250,0,0.0,-0.047644,-0.2738,-0.197856,-0.047644,-0.2738,-0.197856
|
||||||
|
2023,hmm_0.5,2023-01-03,2023-12-29,250,242,8,0.032,-0.047644,-0.2738,-0.197856,-0.005112,-0.0312,-0.171319
|
||||||
|
2023,hmm_0.7,2023-01-03,2023-12-29,250,119,131,0.524,-0.047644,-0.2738,-0.197856,0.006276,0.0709,-0.084209
|
||||||
|
2023,hmm_0.9,2023-01-03,2023-12-29,250,0,250,1.0,-0.047644,-0.2738,-0.197856,0.0,0.0,0.0
|
||||||
|
2021,disp_0.010,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
|
||||||
|
2021,disp_0.015,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
|
||||||
|
2021,disp_0.020,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
|
||||||
|
2021,disp_0.025,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
|
||||||
|
2021,disp_0.030,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
|
||||||
|
2021,vol_low_max15,2021-01-04,2021-12-31,252,0,252,1.0,0.18367,1.0979,-0.102651,0.0,0.0,0.0
|
||||||
|
2021,vol_low_max20,2021-01-04,2021-12-31,252,111,141,0.5595,0.18367,1.0979,-0.102651,0.052323,0.6232,-0.065118
|
||||||
|
2021,vol_low_max25,2021-01-04,2021-12-31,252,212,40,0.1587,0.18367,1.0979,-0.102651,0.099622,0.7322,-0.077033
|
||||||
|
2021,vol_10_25,2021-01-04,2021-12-31,252,212,40,0.1587,0.18367,1.0979,-0.102651,0.099622,0.7322,-0.077033
|
||||||
|
2021,vol_10_30,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
|
||||||
|
2021,hmm_0.3,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
|
||||||
|
2021,hmm_0.5,2021-01-04,2021-12-31,252,243,9,0.0357,0.18367,1.0979,-0.102651,0.180238,1.1183,-0.116244
|
||||||
|
2021,hmm_0.7,2021-01-04,2021-12-31,252,190,62,0.246,0.18367,1.0979,-0.102651,0.184264,1.7758,-0.095388
|
||||||
|
2021,hmm_0.9,2021-01-04,2021-12-31,252,0,252,1.0,0.18367,1.0979,-0.102651,0.0,0.0,0.0
|
||||||
|
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,400 @@
|
|||||||
|
"""Regime-gate walk-forward backtest grid.
|
||||||
|
|
||||||
|
Precomputes regime gates (dispersion/vol/hmm × threshold grid) from lake bars,
|
||||||
|
then runs a qlib TopkDropout backtest with each gate applied as a date-level
|
||||||
|
trade overlay. Uses the SAME pred.pkl from exp 52 (Config A 2026) so the
|
||||||
|
model is trained only once.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
cd /app && .venv/bin/python book/scripts/regime_gate_bt.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
LAKE_ROOT = "/home/data/lake"
|
||||||
|
MARKET = "US"
|
||||||
|
OUT_DIR = pathlib.Path("/app/experiments/book/data/regime_gate")
|
||||||
|
|
||||||
|
# Walk-forward test windows with their pred.pkl sources
|
||||||
|
WINDOWS = [
|
||||||
|
{"label": "2026", "start": "2026-01-04", "end": "2026-08-19",
|
||||||
|
"pred": f"{LAKE_ROOT}/mlruns/52/9f98ea5c550a409f87b56a6cd8fee343/artifacts/pred.pkl"},
|
||||||
|
{"label": "2025", "start": "2025-01-02", "end": "2025-12-31",
|
||||||
|
"pred": f"{LAKE_ROOT}/mlruns/52/fe96741654df4780957a3a949999ae6a/artifacts/pred.pkl"},
|
||||||
|
{"label": "2024", "start": "2024-01-02", "end": "2024-12-31",
|
||||||
|
"pred": f"{LAKE_ROOT}/mlruns/52/71ed5bfa9984490f8bba8b222f7acc39/artifacts/pred.pkl"},
|
||||||
|
{"label": "2023", "start": "2023-01-03", "end": "2023-12-29",
|
||||||
|
"pred": f"{LAKE_ROOT}/mlruns/56/8ca46e554311444c9a42637a788226e8/artifacts/pred.pkl"},
|
||||||
|
{"label": "2021", "start": "2021-01-04", "end": "2021-12-31",
|
||||||
|
"pred": f"{LAKE_ROOT}/mlruns/56/4e0700ddab2a4e108b46efece7346ee3/artifacts/pred.pkl"},
|
||||||
|
]
|
||||||
|
|
||||||
|
# Gate grid
|
||||||
|
DISP_THRESHOLDS = [0.010, 0.015, 0.020, 0.025, 0.030]
|
||||||
|
VOL_BANDS = [
|
||||||
|
(0.0, 0.15, "low_max15"),
|
||||||
|
(0.0, 0.20, "low_max20"),
|
||||||
|
(0.0, 0.25, "low_max25"),
|
||||||
|
(0.10, 0.25, "10_25"),
|
||||||
|
(0.10, 0.30, "10_30"),
|
||||||
|
]
|
||||||
|
HMM_THRESHOLDS = [0.3, 0.5, 0.7, 0.9]
|
||||||
|
|
||||||
|
|
||||||
|
def load_pred(path: str) -> pd.Series:
|
||||||
|
"""Load pred.pkl (MultiIndex: datetime × instrument → score), dates normalized to midnight."""
|
||||||
|
df = pd.read_pickle(path)
|
||||||
|
if isinstance(df, pd.DataFrame):
|
||||||
|
if "score" in df.columns:
|
||||||
|
s = df["score"]
|
||||||
|
else:
|
||||||
|
s = df.iloc[:, 0]
|
||||||
|
else:
|
||||||
|
s = df
|
||||||
|
# Normalize datetime level to date-only (midnight, no tz)
|
||||||
|
idx = s.index
|
||||||
|
new_dt = pd.to_datetime(idx.get_level_values(0)).normalize()
|
||||||
|
s.index = pd.MultiIndex.from_arrays([new_dt, idx.get_level_values(1)], names=idx.names)
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def precompute_gates(close_df: pd.DataFrame) -> dict:
|
||||||
|
"""Precompute all regime gate series from close prices."""
|
||||||
|
gates = {}
|
||||||
|
|
||||||
|
# --- dispersion gates ---
|
||||||
|
ret22 = close_df.pct_change(22)
|
||||||
|
cs_disp = ret22.std(axis=1)
|
||||||
|
for thr in DISP_THRESHOLDS:
|
||||||
|
g = cs_disp >= thr
|
||||||
|
g.iloc[:22] = True
|
||||||
|
gates[f"disp_{thr:.3f}"] = g
|
||||||
|
|
||||||
|
# --- vol gates ---
|
||||||
|
import numpy as np
|
||||||
|
log_ret = np.log(close_df / close_df.shift(1))
|
||||||
|
rv22 = log_ret.rolling(22).std() * (252 ** 0.5)
|
||||||
|
cs_vol = rv22.mean(axis=1)
|
||||||
|
for vlow, vhigh, tag in VOL_BANDS:
|
||||||
|
g = (cs_vol >= vlow) & (cs_vol <= vhigh)
|
||||||
|
g.iloc[:22] = True
|
||||||
|
gates[f"vol_{tag}"] = g
|
||||||
|
|
||||||
|
# --- HMM gates ---
|
||||||
|
hmm_root = pathlib.Path(LAKE_ROOT) / "features" / "market=US" / "timeframe=1d"
|
||||||
|
for thr in HMM_THRESHOLDS:
|
||||||
|
all_post = {}
|
||||||
|
for sym in close_df.columns:
|
||||||
|
for family in ("sp", "ta"):
|
||||||
|
fp = hmm_root / f"family={family}" / f"symbol={sym}.parquet"
|
||||||
|
if not fp.exists():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
feat = pd.read_parquet(fp)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if "sp_hmm_p_regime1" not in feat.columns:
|
||||||
|
continue
|
||||||
|
tcol = feat["t"] if "t" in feat.columns else feat["date"]
|
||||||
|
ts = pd.to_datetime(tcol)
|
||||||
|
s = pd.Series(feat["sp_hmm_p_regime1"].values, index=ts, name=sym)
|
||||||
|
s = s.dropna()
|
||||||
|
if len(s) > 0:
|
||||||
|
all_post[sym] = s
|
||||||
|
break
|
||||||
|
if all_post:
|
||||||
|
post_df = pd.DataFrame(all_post)
|
||||||
|
cs_mean = post_df.mean(axis=1)
|
||||||
|
g = cs_mean >= thr
|
||||||
|
else:
|
||||||
|
g = pd.Series(True, index=close_df.index)
|
||||||
|
gates[f"hmm_{thr:.1f}"] = g
|
||||||
|
|
||||||
|
# Normalize all gate indices to date-only (no tz, no time)
|
||||||
|
for key in gates:
|
||||||
|
gates[key].index = pd.to_datetime(gates[key].index).normalize()
|
||||||
|
|
||||||
|
return gates
|
||||||
|
|
||||||
|
|
||||||
|
def run_backtest_with_gate(
|
||||||
|
pred: pd.Series,
|
||||||
|
gate: pd.Series,
|
||||||
|
close_df: pd.DataFrame,
|
||||||
|
start: str,
|
||||||
|
end: str,
|
||||||
|
topk: int = 10,
|
||||||
|
n_drop: int = 1,
|
||||||
|
) -> dict:
|
||||||
|
"""Simulate TopkDropout with gate overlay, computing daily returns.
|
||||||
|
|
||||||
|
- On gate-open days: hold topk stocks (equal-weight), rebalance weekly
|
||||||
|
- On gate-closed days: liquidate to cash
|
||||||
|
- Tracks both gated and ungated (baseline) equity curves
|
||||||
|
"""
|
||||||
|
# Ensure pred has MultiIndex (date, instrument)
|
||||||
|
if not isinstance(pred.index, pd.MultiIndex):
|
||||||
|
return {"error": "pred must have MultiIndex (date, instrument)"}
|
||||||
|
|
||||||
|
# Daily returns per symbol (close-to-close)
|
||||||
|
ret_df = close_df.pct_change()
|
||||||
|
# Normalize ret_df index to date-only for matching
|
||||||
|
ret_df.index = pd.to_datetime(ret_df.index).normalize()
|
||||||
|
|
||||||
|
# Filter pred to window and get trade dates
|
||||||
|
dt_idx = pred.index.get_level_values(0)
|
||||||
|
window_mask = dt_idx >= pd.Timestamp(start)
|
||||||
|
window_mask &= dt_idx <= pd.Timestamp(end)
|
||||||
|
window_pred = pred.loc[window_mask]
|
||||||
|
if len(window_pred) == 0:
|
||||||
|
return {"error": "no pred data in window"}
|
||||||
|
trade_dates = sorted(dt_idx[window_mask].unique())
|
||||||
|
|
||||||
|
# Compute gate status per trade date
|
||||||
|
gate_open = {}
|
||||||
|
for d in trade_dates:
|
||||||
|
known = gate[gate.index <= d]
|
||||||
|
gate_open[d] = bool(known.iloc[-1]) if len(known) else True
|
||||||
|
|
||||||
|
n_total = len(trade_dates)
|
||||||
|
n_open = sum(1 for v in gate_open.values() if v)
|
||||||
|
n_closed = n_total - n_open
|
||||||
|
|
||||||
|
# Simulate: track current holdings — both base and gated use weekly rebalance
|
||||||
|
# Use yesterday's scores to pick today's holdings (no look-ahead)
|
||||||
|
holdings_base = []
|
||||||
|
holdings_gated = []
|
||||||
|
equity_gated = 1_000_000.0
|
||||||
|
equity_base = 1_000_000.0
|
||||||
|
prev_week = None
|
||||||
|
prev_scores = None # yesterday's scores
|
||||||
|
|
||||||
|
daily_gated = []
|
||||||
|
daily_base = []
|
||||||
|
|
||||||
|
# Build a date → ret_df row map
|
||||||
|
ret_by_date = {rd: ret_df.loc[rd] for rd in ret_df.index}
|
||||||
|
|
||||||
|
for i, d in enumerate(trade_dates):
|
||||||
|
# Get today's cross-sectional prediction
|
||||||
|
try:
|
||||||
|
day_scores = window_pred.loc[d]
|
||||||
|
except KeyError:
|
||||||
|
daily_gated.append(equity_gated)
|
||||||
|
daily_base.append(equity_base)
|
||||||
|
prev_scores = None
|
||||||
|
continue
|
||||||
|
|
||||||
|
if isinstance(day_scores, pd.Series) and not isinstance(day_scores.index, pd.MultiIndex):
|
||||||
|
pass
|
||||||
|
elif isinstance(day_scores, pd.DataFrame):
|
||||||
|
day_scores = day_scores.iloc[:, 0]
|
||||||
|
else:
|
||||||
|
daily_gated.append(equity_gated)
|
||||||
|
daily_base.append(equity_base)
|
||||||
|
prev_scores = None
|
||||||
|
continue
|
||||||
|
|
||||||
|
day_scores = day_scores.dropna().sort_values(ascending=False)
|
||||||
|
if len(day_scores) == 0:
|
||||||
|
daily_gated.append(equity_gated)
|
||||||
|
daily_base.append(equity_base)
|
||||||
|
prev_scores = None
|
||||||
|
continue
|
||||||
|
|
||||||
|
ret_row = ret_by_date.get(d)
|
||||||
|
if ret_row is None:
|
||||||
|
daily_gated.append(equity_gated)
|
||||||
|
daily_base.append(equity_base)
|
||||||
|
prev_scores = day_scores
|
||||||
|
continue
|
||||||
|
|
||||||
|
cur_week = (d.isocalendar()[0], d.isocalendar()[1]) if hasattr(d, 'isocalendar') else None
|
||||||
|
gate_val = gate_open.get(d, True)
|
||||||
|
|
||||||
|
# --- ungated baseline: weekly rebalance using yesterday's scores ---
|
||||||
|
if cur_week != prev_week or not holdings_base:
|
||||||
|
if prev_scores is not None:
|
||||||
|
holdings_base = list(prev_scores.index[:topk])
|
||||||
|
if holdings_base:
|
||||||
|
base_rets = ret_row.reindex(holdings_base).dropna()
|
||||||
|
if len(base_rets) > 0:
|
||||||
|
equity_base *= (1 + base_rets.mean())
|
||||||
|
|
||||||
|
# --- gated: weekly rebalance only when gate open, using yesterday's scores ---
|
||||||
|
if gate_val:
|
||||||
|
if cur_week != prev_week or not holdings_gated:
|
||||||
|
if prev_scores is not None:
|
||||||
|
holdings_gated = list(prev_scores.index[:topk])
|
||||||
|
if holdings_gated:
|
||||||
|
hold_rets = ret_row.reindex(holdings_gated).dropna()
|
||||||
|
if len(hold_rets) > 0:
|
||||||
|
equity_gated *= (1 + hold_rets.mean())
|
||||||
|
else:
|
||||||
|
holdings_gated = []
|
||||||
|
|
||||||
|
prev_week = cur_week
|
||||||
|
prev_scores = day_scores
|
||||||
|
daily_gated.append(equity_gated)
|
||||||
|
daily_base.append(equity_base)
|
||||||
|
|
||||||
|
# Compute metrics
|
||||||
|
g_series = pd.Series(daily_gated, index=trade_dates)
|
||||||
|
b_series = pd.Series(daily_base, index=trade_dates)
|
||||||
|
|
||||||
|
def _metrics(eq: pd.Series) -> dict:
|
||||||
|
if len(eq) < 2:
|
||||||
|
return {"ann_return": 0, "sharpe": 0, "maxDD": 0}
|
||||||
|
rets = eq.pct_change().dropna()
|
||||||
|
ann_ret = float((eq.iloc[-1] / eq.iloc[0]) ** (252 / max(len(eq), 1)) - 1)
|
||||||
|
vol = float(rets.std() * (252 ** 0.5)) if len(rets) > 1 else 0
|
||||||
|
sharpe = ann_ret / vol if vol > 0 else 0
|
||||||
|
peak = eq.cummax()
|
||||||
|
dd = (eq - peak) / peak
|
||||||
|
maxDD = float(dd.min())
|
||||||
|
return {"ann_return": round(ann_ret, 6), "sharpe": round(sharpe, 4), "maxDD": round(maxDD, 6)}
|
||||||
|
|
||||||
|
base_m = _metrics(b_series)
|
||||||
|
gated_m = _metrics(g_series)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"trade_dates": n_total,
|
||||||
|
"gate_open_days": n_open,
|
||||||
|
"gate_closed_days": n_closed,
|
||||||
|
"trip_rate": round(n_closed / n_total, 4) if n_total else 0,
|
||||||
|
"base": base_m,
|
||||||
|
"gated": gated_m,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_bars_for_window(start: str, end: str) -> pd.DataFrame:
|
||||||
|
"""Load daily close prices for all symbols in the universe."""
|
||||||
|
from tac_qlib.data.config import LakeConfig, resolve_lake_root
|
||||||
|
|
||||||
|
cfg = LakeConfig(resolve_lake_root(LAKE_ROOT), MARKET)
|
||||||
|
sp = cfg.lake_root / "symbols.parquet"
|
||||||
|
if sp.exists():
|
||||||
|
syms = pd.read_parquet(sp)
|
||||||
|
col = "symbol" if "symbol" in syms.columns else syms.columns[0]
|
||||||
|
symbols = sorted(syms[col].astype(str).str.upper().tolist())
|
||||||
|
else:
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
closes = {}
|
||||||
|
for sym in symbols:
|
||||||
|
p = cfg.bar_path("1d", sym)
|
||||||
|
if not p.exists():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
df = pd.read_parquet(p)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if not len(df):
|
||||||
|
continue
|
||||||
|
tcol = df["t"] if "t" in df.columns else df["date"]
|
||||||
|
ts = pd.to_datetime(tcol)
|
||||||
|
df = df.assign(_t=ts).set_index("_t").sort_index()
|
||||||
|
# Load a bit extra for warmup
|
||||||
|
warmup_start = pd.Timestamp(start) - pd.Timedelta(days=60)
|
||||||
|
df = df.loc[warmup_start:end]
|
||||||
|
if len(df) >= 22:
|
||||||
|
closes[sym] = df["c"]
|
||||||
|
return pd.DataFrame(closes)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# Load bars (with warmup) for the full panel
|
||||||
|
full_start = "2015-01-03"
|
||||||
|
full_end = "2026-08-19"
|
||||||
|
print("Loading lake bars for gate precomputation...")
|
||||||
|
close_df = load_bars_for_window(full_start, full_end)
|
||||||
|
print(f" {close_df.shape[1]} symbols, {close_df.shape[0]} days")
|
||||||
|
|
||||||
|
print("Precomputing regime gates...")
|
||||||
|
gates = precompute_gates(close_df)
|
||||||
|
print(f" {len(gates)} gate configs: {list(gates.keys())}")
|
||||||
|
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for window in WINDOWS:
|
||||||
|
wl, ws, we = window["label"], window["start"], window["end"]
|
||||||
|
pred_path = window["pred"]
|
||||||
|
print(f"\n=== Window {wl} ({ws} to {we}) ===")
|
||||||
|
|
||||||
|
print(f" Loading pred.pkl from {pred_path}...")
|
||||||
|
pred = load_pred(pred_path)
|
||||||
|
print(f" pred shape: {pred.shape}")
|
||||||
|
|
||||||
|
for gate_name, gate_series in gates.items():
|
||||||
|
bt = run_backtest_with_gate(pred, gate_series, close_df, ws, we)
|
||||||
|
if "error" in bt:
|
||||||
|
print(f" {gate_name}: {bt['error']}")
|
||||||
|
continue
|
||||||
|
row = {
|
||||||
|
"window": wl,
|
||||||
|
"gate": gate_name,
|
||||||
|
"start": ws,
|
||||||
|
"end": we,
|
||||||
|
"trade_dates": bt["trade_dates"],
|
||||||
|
"gate_open": bt["gate_open_days"],
|
||||||
|
"gate_closed": bt["gate_closed_days"],
|
||||||
|
"trip_rate": bt["trip_rate"],
|
||||||
|
"base_ann": bt["base"]["ann_return"],
|
||||||
|
"base_sharpe": bt["base"]["sharpe"],
|
||||||
|
"base_maxDD": bt["base"]["maxDD"],
|
||||||
|
"gated_ann": bt["gated"]["ann_return"],
|
||||||
|
"gated_sharpe": bt["gated"]["sharpe"],
|
||||||
|
"gated_maxDD": bt["gated"]["maxDD"],
|
||||||
|
}
|
||||||
|
results.append(row)
|
||||||
|
print(f" {gate_name}: trip={bt['trip_rate']:.1%}, "
|
||||||
|
f"base={bt['base']['ann_return']:+.1%} (Sharpe {bt['base']['sharpe']:.2f}), "
|
||||||
|
f"gated={bt['gated']['ann_return']:+.1%} (Sharpe {bt['gated']['sharpe']:.2f})")
|
||||||
|
|
||||||
|
# Save results
|
||||||
|
df = pd.DataFrame(results)
|
||||||
|
out_path = OUT_DIR / "regime_gate_trip_rates.csv"
|
||||||
|
df.to_csv(out_path, index=False)
|
||||||
|
print(f"\nSaved trip rates to {out_path}")
|
||||||
|
|
||||||
|
# Also save as JSON for the book
|
||||||
|
json_results = df.to_dict(orient="records")
|
||||||
|
with open(OUT_DIR / "regime_gate_trip_rates.json", "w") as f:
|
||||||
|
json.dump(json_results, f, indent=2, default=str)
|
||||||
|
|
||||||
|
# Print summary: trip rate differential (2026 vs bad years)
|
||||||
|
print("\n=== Trip Rate Summary (2026 vs bad years) ===")
|
||||||
|
for gate_name in gates.keys():
|
||||||
|
gdf = df[df["gate"] == gate_name]
|
||||||
|
r2026 = gdf[gdf["window"] == "2026"]["trip_rate"].values
|
||||||
|
r_bad = gdf[gdf["window"].isin(["2021", "2023", "2024"])]["trip_rate"].values
|
||||||
|
if len(r2026) and len(r_bad):
|
||||||
|
d = r2026[0] - np.mean(r_bad)
|
||||||
|
print(f" {gate_name}: 2026 trip={r2026[0]:.1%}, bad-years avg={np.mean(r_bad):.1%}, diff={d:+.1%}")
|
||||||
|
|
||||||
|
print("\n=== Gated Return Summary (2026 vs bad years) ===")
|
||||||
|
for gate_name in gates.keys():
|
||||||
|
gdf = df[df["gate"] == gate_name]
|
||||||
|
r2026 = gdf[gdf["window"] == "2026"]
|
||||||
|
r_bad = gdf[gdf["window"].isin(["2021", "2023", "2024"])]
|
||||||
|
if len(r2026) and len(r_bad):
|
||||||
|
g26 = r2026["gated_ann"].values[0]
|
||||||
|
b26 = r2026["base_ann"].values[0]
|
||||||
|
g_bad = r_bad["gated_ann"].mean()
|
||||||
|
b_bad = r_bad["base_ann"].mean()
|
||||||
|
print(f" {gate_name}: 2026 gated={g26:+.1%} (base={b26:+.1%}), "
|
||||||
|
f"bad-years gated={g_bad:+.1%} (base={b_bad:+.1%})")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user