zhaoli/book-tac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

book: evidence boundary (clean-lake watermark) + ch02 cost reality + ch05 clean-lake reset — exp 21-31, chat mining

Browse Source
This commit is contained in:
TradeAC Book Agent
2026-08-18 22:45:11 +00:00
parent c93424e76c
commit 3562b7f776
16 changed files with 6955 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
book/EVIDENCE.md
+14 -10
View File
@@ -2,23 +2,27 @@
Every quantitative claim in the book lands here: id → claim → source (experiment/run/branch, round_id, script, citation) → verified?.
## Evidence boundary
**The clean-lake boundary (2026-08-18, exp 21) is the watermark.** `PROVEN` status in this book is reserved for the **Post-reset period** table (exp 21–31) and post-reset live rounds. The **Pre-clean-lake period** table below is **historical context and idea material only**: it was demonstrably inflated by lake data-quality problems (`EVIDENCE#010 → exp 21`). Pre-reset numbers may inform hypotheses but may never be cited as fact in the book.
## Key metric-schema note
Experiments 8–18 record metrics under a legacy schema (`ls_sharpe`, `maxdd_with_cost`, `excess_ann_with_cost`, `excess_ir_with_cost`, `ls_ann_return`). Experiments 21+ use the canonical `IC / ICIR / Rank IC / Rank ICIR / net_IR / net_ann_return / gross_* / Long-Short_Ann_Sharpe / net_max_drawdown`. Do not compare schemas directly; chapter text states which schema a number comes from. Additionally, exp 20's R0 note states the exp-18 baseline is not comparable to post-reset runs due to environment non-determinism, and exp 21 invalidated all pre-clean-lake positive results.
## Pre-clean-lake period (exp 8–18) — historical, superseded
## Pre-clean-lake period (exp 8–18) — historical context / idea material ONLY, superseded
| ID | Claim | Source | Verified? |
|----|-------|--------|-----------|
| EVIDENCE#001 | Baseline 1-day LGB signal weak on 2026 OOS: IC 0.017, ICIR 0.062, RankIC 0.040, RankICIR 0.161 (below 0.2 noise threshold). L/S ann +4.9%. | exp 8, run `e65cf1ec…` (mlflow exp 10), branch `exp/8-baseline-lightgbm-on-the-full-60etf-univ` | yes |
| EVIDENCE#002 | Costs erase most of the raw edge on baseline: excess +6.2% ann w/o cost (IR 0.31, MaxDD −20.4%) vs +1.6% ann after costs (IR 0.08). | exp 8 (same run) | yes |
| EVIDENCE#003 | Feature-family ablation: generic-only (jump,har,trend,hurst,signature,ret,max_move) beats all-24: RankIC 0.030→0.064, RankICIR 0.146→0.276, L/S Sharpe −0.83→+2.55, net excess −9.4%→+3.1%. | exp 9, run `7b1e7972…` (mlflow exp 11), branch `exp/9-sp5d-feature-family-ablation` | yes |
| EVIDENCE#004 | Adding 16 moment/volatility fields regresses every metric (RankIC 0.064→0.047, net excess −16.2% IR −1.57) — same failure mode as ou/hmm. | exp 11, run `a3f7d1d4…` (mlflow exp 12), branch `exp/11-sp5d-momentfeature-extension-after-exten` | yes |
| EVIDENCE#005 | 5-seed RankIC ensemble on ablated generic features: RankIC 0.0586, RankICIR 0.224, net excess +7.8% (IR 0.79), L/S Sharpe 3.71, MDD −7.9%. Best pre-clean-lake net result. | exp 12, run `0cea66d9…` (mlflow exp 16), branch `exp/12-isolate-the-multiseed-rankic-ensemble-ef` | yes (superseded by EVIDENCE#011 on clean data) |
| EVIDENCE#006 | OptimalStopControl (entry 0.85/exit 0.7/hold 10/sl −0.08) worse than TopkDropout: net excess −2.7% (IR −0.31) vs +7.8%; cost drag −11.3pp. | exp 13, run `4e1f77b4…` (mlflow exp 17), branch `exp/13-portfolioconstruction-variant-of-the-iso` | yes |
| EVIDENCE#007 | OptimalStopControlV2 (turnover band/cooldown/cap) also refuted: net −6.9% (IR −0.72) vs TopkDropout +7.8% (IR 0.79). | exp 14, run `83d7e27e…` (mlflow exp 18), branch `exp/14-enhanced-stochasticcontrol-allocation-fo` | yes |
| EVIDENCE#008 | Risk-limit A/B: $5M liquidity floor → net IR 0.81→0.98, cumDD 7.93%→5.44%; size cap 15% + conc 60% hurts (IR 0.816, ann 6.11%). | exp 18, run `28c7fa08…` (mlflow exp 21), branch `exp/18-risk-limit-control-on-the-reference-ense` | yes (pre-clean-lake, see note) |
| EVIDENCE#009 | Improvement sweep (R1-R5): 4/5 refuted; R2 momentum gate and R3 HMM gate are byte-identical no-ops; R5 MA3/EWMA marginal (IR 0.049). Conclusion: signal quality is the bottleneck, not the execution/risk layer. | exp 20, run `958198a8…` (mlflow exp 21), branch `exp/20-improve-the-risk-limit-reference-signal` | yes |
| EVIDENCE#001 | Baseline 1-day LGB signal weak on 2026 OOS: IC 0.017, ICIR 0.062, RankIC 0.040, RankICIR 0.161 (below 0.2 noise threshold). L/S ann +4.9%. | exp 8, run `e65cf1ec…` (mlflow exp 10), branch `exp/8-baseline-lightgbm-on-the-full-60etf-univ` | NOT usable as PROVEN — pre-clean-lake |
| EVIDENCE#002 | Costs erase most of the raw edge on baseline: excess +6.2% ann w/o cost (IR 0.31, MaxDD −20.4%) vs +1.6% ann after costs (IR 0.08). | exp 8 (same run) | NOT usable as PROVEN — pre-clean-lake |
| EVIDENCE#003 | Feature-family ablation: generic-only (jump,har,trend,hurst,signature,ret,max_move) beats all-24: RankIC 0.030→0.064, RankICIR 0.146→0.276, L/S Sharpe −0.83→+2.55, net excess −9.4%→+3.1%. | exp 9, run `7b1e7972…` (mlflow exp 11), branch `exp/9-sp5d-feature-family-ablation` | NOT usable as PROVEN — pre-clean-lake (idea: pruning generic beats model-specific) |
| EVIDENCE#004 | Adding 16 moment/volatility fields regresses every metric (RankIC 0.064→0.047, net excess −16.2% IR −1.57) — same failure mode as ou/hmm. | exp 11, run `a3f7d1d4…` (mlflow exp 12), branch `exp/11-sp5d-momentfeature-extension-after-exten` | NOT usable as PROVEN — pre-clean-lake (idea: panel width vs feature count) |
| EVIDENCE#005 | 5-seed RankIC ensemble on ablated generic features: RankIC 0.0586, RankICIR 0.224, net excess +7.8% (IR 0.79), L/S Sharpe 3.71, MDD −7.9%. Best pre-clean-lake net result. | exp 12, run `0cea66d9…` (mlflow exp 16), branch `exp/12-isolate-the-multiseed-rankic-ensemble-ef` | NOT usable as PROVEN — inflated by dirty lake (see EVIDENCE#010) |
| EVIDENCE#006 | OptimalStopControl (entry 0.85/exit 0.7/hold 10/sl −0.08) worse than TopkDropout: net excess −2.7% (IR −0.31) vs +7.8%; cost drag −11.3pp. | exp 13, run `4e1f77b4…` (mlflow exp 17), branch `exp/13-portfolioconstruction-variant-of-the-iso` | NOT usable as PROVEN — pre-clean-lake (idea: turnover-sensitive construction bleeds costs) |
| EVIDENCE#007 | OptimalStopControlV2 (turnover band/cooldown/cap) also refuted: net −6.9% (IR −0.72) vs TopkDropout +7.8% (IR 0.79). | exp 14, run `83d7e27e…` (mlflow exp 18), branch `exp/14-enhanced-stochasticcontrol-allocation-fo` | NOT usable as PROVEN — pre-clean-lake (idea only) |
| EVIDENCE#008 | Risk-limit A/B: $5M liquidity floor → net IR 0.81→0.98, cumDD 7.93%→5.44%; size cap 15% + conc 60% hurts (IR 0.816, ann 6.11%). | exp 18, run `28c7fa08…` (mlflow exp 21), branch `exp/18-risk-limit-control-on-the-reference-ense` | NOT usable as PROVEN — pre-clean-lake (idea: liquidity floor > concentration caps) |
| EVIDENCE#009 | Improvement sweep (R1-R5): 4/5 refuted; R2 momentum gate and R3 HMM gate are byte-identical no-ops; R5 MA3/EWMA marginal (IR 0.049). Conclusion: signal quality is the bottleneck, not the execution/risk layer. | exp 20, run `958198a8…` (mlflow exp 21), branch `exp/20-improve-the-risk-limit-reference-signal` | NOT usable as PROVEN — pre-clean-lake (idea: gates are no-ops when signal is weak) |
## Post-reset period (exp 21–31) — canonical, current