book: add EVIDENCE#036-041, update claims for Q12-Q18 findings

- EVIDENCE#036: Q12 22d label + weekly rebalance still fails (−4.88%) - EVIDENCE#037: Q13 weekly rebalance edge is window-dependent (−4.21%) - EVIDENCE#038: Q15 single-seed worse (clean-lake conf of #016) - EVIDENCE#039: Q16 HMM features degrade on clean data (−6.06%) - EVIDENCE#040: Q17 realized-moments improve portfolio (+9.90% IR 0.99) - EVIDENCE#041: Q18 OptimalStopControl loses to TopkDropout (−6.21%) Claims updated: H2 OptStop PROVEN, moments claim REFUTED on clean data, weekly edge window-dependent PROVEN, HMM refuted, seed count updated. Open questions marked done.
2026-08-20 05:31:13 +00:00
parent 6145cfeb62
commit 63763db38c
2 changed files with 17 additions and 6 deletions
@@ -52,6 +52,12 @@ Experiments 8–18 record metrics under a legacy schema (`ls_sharpe`, `maxdd_wit
 | EVIDENCE#033 | Q14 out-of-universe validation: compact stochastic set on 30 liquid single-stock names (AAPL,MSFT,NVDA,…). RankIC −0.0198 (needed >0.03), ICIR −0.073 (needed >0.15) — signal is noise on this universe. Net P&L positive (+10.02% ann, IR 0.668, maxDD −6.67%) but that is top-10 concentration luck, not predictive signal. Train RankIC 0.316 shows the model overfits to the 50-ETF panel. | exp 50, run `809ff460…` (mlflow exp 50 `tac-rd-q14-out-of-universe`), branch `exp/50-q14-compact-stochastic-set-generalizes-t` | yes — Q14 FAIL (signal does not generalize cross-universe) |
 | EVIDENCE#034 | Q19 variance-ratio study (Lo-MacKinlay robust VR): 71-ETF panel, 2015–2026. Median VR < 1 at all horizons — 5d: 0.925, 10d: 0.900, 20d: 0.884. 37–47% of ETFs have VR < 1 with |z| > 2 (significant mean-reversion). Only 1–3% show significant momentum. Assets are mean-reverting at short horizons on the clean lake. Note: pooled trend_slope_5 beta is strongly positive (+3.80, t=237) — the cross-sectional signal does NOT capture time-series mean-reversion. | scripted study, `book/data/evidence/q19-vr/vr_study.py`, VR_stats.csv, VR_summary.json | yes — Q19 PROVEN (market-structure claim) |
 | EVIDENCE#035 | Q20 effective independent names: eigenvalue analysis on 71-ETF correlation matrix (test window 2026-01-04 to 2026-08-10). Participation ratio = 4.46. Top-4 eigenvalues explain 66.8% of variance. 4 eigenvalues above Marchenko-Pastur bound (2.86). The 50-ETF book has ≈4.5 effective independent names — confirming the chat-derived claim. This explains why topk 10→20 adds no breadth (EVIDENCE#024). | scripted study, `book/data/evidence/q20-effective-names/eigenanalysis.py`, eigenanalysis_50etf.csv, eigen_summary_50etf.json | yes — Q20 PROVEN (diversification claim) |
+| EVIDENCE#036 | Q12 22d label + weekly rebalance: same IC/RankIC as Q05 (IC 0.097, RankIC 0.117 — identical training), but weekly recompute cannot rescue the stale signal. Net −4.88% (IR −0.566), gross +1.46%, maxDD −10.49%. The 22d label's problem is not daily turnover alone — the signal itself is stale. | exp 44, run `aed45c54…` (mlflow exp 44), branch `exp/44-q12-label22d-weekly` | yes — Q12 FAIL (redundant with Q05, confirms signal-stale hypothesis) |
+| EVIDENCE#037 | Q13 weekly rebalance on 2025 OOS window (train→2024-08-30, test 2025-01-02..2025-12-31): edge is window-dependent. IC 0.031 (vs Q07's 0.050), RankIC 0.073 (vs 0.066), L/S Sharpe 1.19 (vs 4.54). Net −4.21% (IR −0.523), maxDD −10.66%. Q07's +12.51% (IR 1.24) was specific to the 2026-01-04..2026-08-10 window. Weekly rebalance is not a robust edge. | exp 45, run `e5ac7a5d…` (mlflow exp 45), branch `exp/45-q13-weekly-oos` | yes — Q13 FAIL (limits Q07's generalizability) |
+| EVIDENCE#038 | Q15 single-seed vs 5-seed: 1 seed loses to 5 seeds on every metric. RankIC 0.044 vs 0.066, RankICIR 0.160 vs 0.255, net −2.89% (IR −0.278) vs +12.51% (IR 1.24). Clean-lake confirmation of EVIDENCE#016 (2-seed < 5-seed). Seed count is load-bearing. | exp 46, run `8d49e0be…` (mlflow exp 46), branch `exp/46-q15-single-seed` | yes — Q15 FAIL (confirms EVIDENCE#016) |
+| EVIDENCE#039 | Q16 HMM features (sp_hmm_p_regime1, sp_hmm_state) on clean data: degrades both signal and portfolio. IC 0.030 (vs 0.050 baseline), RankIC 0.048 (vs 0.066), net −6.06% (IR −0.623), L/S Sharpe 1.23 (vs 4.54). HMM regime detection adds noise, not signal. | exp 47, run `ff092e1c…` (mlflow exp 47), branch `exp/47-q16-hmm` | yes — Q16 FAIL (HMM refuted on clean data) |
+| EVIDENCE#040 | Q17 realized-moments features (sp_rskew_5/22, sp_rkurt_5/22, sp_dsv_5/22) on clean data: improves portfolio over baseline. Net +9.90% (IR 0.990), gross +14.61%, maxDD −6.49% vs baseline net +2.13% (IR 0.21). IC 0.039 (vs 0.050), RankIC 0.060 (vs 0.066) — signal metrics slightly lower but portfolio construction benefits from moment conditioning. Contradicts pre-reset EVIDENCE#004 (which was inflated by dirty data). Single run, unreproduced. | exp 48, run `e62ce326…` (mlflow exp 48), branch `exp/48-q17-moments` | yes — Q17 HYPOTHESIS (needs reproduction) |
+| EVIDENCE#041 | Q18 OptimalStopControl (entry 0.85/exit 0.7/hold 10/sl −0.08) vs TopkDropout on clean data: same signal (IC 0.050, RankIC 0.066 — identical model), worse portfolio. Net −6.21% (IR −0.640) vs baseline +2.13% (IR 0.21). Cost drag ~8.3pp. Clean-lake confirmation of pre-reset EVIDENCE#006/#007. | exp 49, run `f140dcb8…` (mlflow exp 49), branch `exp/49-q18-optstop` | yes — Q18 FAIL (confirms EVIDENCE#006/#007 on clean data) |

 ## Live execution trail