ch11: signal-quality gate REFUTED — walk-forward workflow shows gate harmful (exp 61-67, EVIDENCE#053)

2026-08-21 02:35:50 +00:00
parent 1603a063ba
commit 6e3c82408c
26 changed files with 4934 additions and 27 deletions
@@ -85,7 +85,8 @@ Experiments 8–18 record metrics under a legacy schema (`ls_sharpe`, `maxdd_wit
 | ID | Claim | Source | Verified? |
 |----|-------|--------|-----------|
 | EVIDENCE#051 | Comprehensive model search: queried all MLflow experiments/runs, ranked by RankICIR. Top models: exp 36/44 (label22d, RankICIR 0.507, single-window 2026 only), exp 35/51 (label10d, RankICIR 0.352, single-window), exp 58 (adaptive-2y, RankICIR 0.289), exp 11 (single-seed, RankICIR 0.276). Exp 52 walk-forward configs rank near the top among multi-year models (RankICIR 0.244). The 22-day label models have highest IC but negative returns (−4.6%) — high IC does not guarantee profitable trading. The regime gate study (EVIDENCE#050) is robust to model selection because it measures market-level features, not model predictions. Selection bias is not material: the best-return model (Config C) also has the best RankICIR among walk-forward configs. | `rd_exp_list` query across all MLflow experiments, run metadata from `rd_exp_get_run` for exp 11/33/36/58/52 | yes — robustness check |
-| EVIDENCE#052 | Signal-quality gate (hit-rate based on topk predictions): gates trades based on whether the model's recent topk predictions were correct. **Every config improves returns across ALL years** — including bad years (2023: −4.8% → +54.7%, 2024: +8.2% → +30.4%). Best config (`hitrate_5d_0.50`): 2026 +65.0% (base +25.5%), 2025 +72.1% (base +17.8%), 2024 +30.4% (base +8.2%), 2023 +54.7% (base −4.8%), 2021 +55.7% (base +18.4%). Gate trips ~40–50% of days. The regime gate (EVIDENCE#050) failed because it asked "is the market calm?" — the signal-quality gate asks "are my predictions accurate?" and succeeds. The model's predictions ARE informative; they just need to be gated on their own accuracy. | scripted simulation: `book/scripts/signal_quality_gate_bt.py`, results `book/data/signal_quality_gate/signal_quality_gate_results.csv`, pred.pkl from exp 52 (2024–2026) and exp 56 (2021, 2023) | yes — signal-quality gate PROVEN |
+| EVIDENCE#052 | **REFUTED by EVIDENCE#053.** Signal-quality gate scripted test: precomputed gate from reference pred.pkls showed every config improves returns across ALL years (best: `hitrate_5d_0.50` 2026 +65.0%, 2025 +72.1%, 2024 +30.4%, 2023 +54.7%, 2021 +55.7%). **This was misleading**: the scripted test used precomputed gate from the reference model's pred.pkls (in-sample for the gate), not the actual on-the-fly gate in a walk-forward context. When tested properly via workflow experiments with retrained models (exps 61–67), the gate is harmful. | scripted simulation (original), refuted by exps 61–67 | **REFUTED** — scripted test was in-sample for the gate; walk-forward workflow tests show the gate hurts |
 | EVIDENCE#053 | Signal-quality gate walk-forward refutation: `WeeklyRebalanceSignalQualityGateStrategy` (topk=10, n_drop=1, gate_topk=10, gate_lookback=5, gate_threshold=0.5, 5/15bp costs) tested via `rd_train` + `rd_run_workflow` on 5 walk-forward windows (2021–2026). **The gate is harmful in every year.** Workflow excess-with-cost: 2026 +9.1% (IR 0.92) vs reference +12.5% (IR 1.24, exp 38); 2025 +3.4% (IR 0.31); 2024 −20.5%; 2023 −29.4%; 2021 −18.4%. Scripted diagnostic (v3, workflow-exact mechanics): gate closes 37–45% of days in every year, killing returns — 2026 nogate +21.2% total → gate +1.5% total (−19.7pp); 2025 +22.7% → +7.8% (−14.9pp). The gate's hit-rate threshold (0.5) is too aggressive: a model with Rank IC 0.06–0.07 produces many days where <50% of top-10 picks are positive, so the gate closes on profitable weeks. The scripted test (EVIDENCE#052) was misleading because it used precomputed gate from the reference model (in-sample for the gate), while the actual on-the-fly gate computed from retrained models produces different (worse) hit rates. **Guard 7 (signal-quality gate) is REFUTED.** | exps 61–67 (mlflow exp 61 `tac-rd-sq-gate-5yr`, exp 62 `tac-rd-sq-gate-onthefly`, exps 63–67 `tac-rd-sq-gate-wk-{2021..2026}`); scripted diagnostic `book/scripts/diagnose_script_vs_workflow_v3.py`, results `book/data/diag_script_vs_wf/diagnosis_v3.json`; strategy `tac_qlib/contrib/strategy/weekly_sq_gate.py` | yes — guard 7 REFUTED |
 ## External references (book/references/)
@@ -48,8 +48,9 @@ With the walk-forward sweep showing the edge is 2026-window-specific, the desk t
 | 3 | **Streaming IC circuit breaker** (`ic_min_rankic`) | pause new buys while trailing realized RankIC (computed causally from lake bars) is below a threshold | REFUTED — trips 25–50% of days *every year*, freezing TopkDropout's rotation out of losers; implemented in `tac_qlib/contrib/strategy/ic_gate.py`, do not deploy live. |
 | 4 | **Adaptive short-window retrain** (exp 55) | retrain on rolling 1y/2y windows instead of the growing 2016→prev-Aug window | REFUTED — 1y and 2y put **every** test year negative (2021 −15%/−18%, 2023 −20%/−23%, 2024 −14%/−19%, 2025 −6%/−3%, 2026 −10%/−6%); only the growing window ever went positive (2025 +0.4%, 2026 +6.5% IR 0.62). Short windows shave losses in bad years (2024 −26.4%→−13.9%) but destroy the 2026 edge (+6.5%→−9.6%). Mean annual excess ≈ −13% for *every* window length. |
 | 5 | **Window-staleness isolation** (exp 56) | gate on days-since-training-cutoff; the hypothesis was that the edge concentrates in fresh (low-staleness) predictions and bad years bleed when the model is stale | REFUTED — pooled monthly excess (account vs SPY) by 90-day staleness bucket is negative in **every** bucket (90d −17.4%, 180d −30.1%, 270d −17.7%, 360d −13.9%, 450d −9.7%): the *freshest* bucket is the *most* negative. The 2026 edge is NOT concentrated in low-staleness days (best month Mar +8.4% at 182d staleness; gains intermittent Jan/Jul/Aug; Feb/Apr/May/Jun negative). 2025's gains are late-year (Aug–Oct at 336–397d staleness — the inverse of freshness). Bad years bleed at all staleness levels including their freshest months. No staleness threshold isolates the edge. |
 | 6 | **Signal-quality gate** (hit-rate) | gate on whether the model's recent topk predictions were correct (5-day rolling hit rate > 0.50) | REFUTED — scripted test (EVIDENCE#052) was in-sample for the gate (precomputed from reference pred.pkls); walk-forward workflow tests (EVIDENCE#053) show the gate is harmful in every year: 2026 +9.1% vs +12.5% reference (−3.4pp), 2025 +3.4% vs +3.7% (−0.3pp), 2024 −20.5% vs −19.4% (−1.1pp). A model with Rank IC 0.06–0.07 produces too many days where <50% of top-10 picks are positive — the 0.5 threshold is too aggressive, closing on profitable weeks. The gate destroys the strategy's ability to capture the good days that compensate for the bad ones. |
-Guards 1–3 are documented across exp 52/53/54 and the `ic_gate.py` implementation; guard 4 = `PROVEN (refuted) — EVIDENCE#046 → exp 55`; guard 5 = `PROVEN (refuted) — EVIDENCE#047 → exp 56`.
+Guards 1–3 are documented across exp 52/53/54 and the `ic_gate.py` implementation; guard 4 = `PROVEN (refuted) — EVIDENCE#046 → exp 55`; guard 5 = `PROVEN (refuted) — EVIDENCE#047 → exp 56`; guard 6 = `REFUTED — EVIDENCE#052 → EVIDENCE#053`.
 ## The account-level truth
@@ -58,7 +59,7 @@ The blotter's daily `account` field is the authoritative measure (the `return` f
 ## The synthesis
 - **The headline results were window-specific.** Weekly rebalance (+12.51%, IR 1.24) and m2-sharpe22 (+6.5%, IR 0.62) are 2026-only. Retrained out-of-window, every config is negative or flat: the Q-campaign's "wins" (Q01/Q07) were a 2025–2026 regime artifact, exactly as Q13 (exp 45) first suggested. `PROVEN — EVIDENCE#043/044`.
- **No guard candidate recovers the edge out-of-sample.** Feature drift, label-regime match, streaming IC, training-window length, and staleness all fail to separate the profitable years from the bleeding ones. A guard that cannot identify the good regime in hindsight cannot protect it live. `PROVEN — EVIDENCE#043–047`.
+- **No guard candidate recovers the edge out-of-sample.** Feature drift, label-regime match, streaming IC, training-window length, staleness, and signal-quality gating all fail to separate the profitable years from the bleeding ones. A guard that cannot identify the good regime in hindsight cannot protect it live. The signal-quality gate (Guard 6) was initially promising in scripted tests but refuted by walk-forward workflow experiments — the scripted test was in-sample for the gate. `PROVEN — EVIDENCE#043–053`.
 - **Construction still matters inside the good regime.** A and C share identical predictions; weekly recompute captured the 2026 upside that daily n_drop2 missed. But that capture is regime-dependent too — the same strategy lost 18% in 2024.
 - **Live implication:** size for the mean, not the tail. The mean annual excess across every window length is ≈ −13%. Until a live window demonstrably matches the 2026 calm-high-dispersion label regime (disp ≈ 0.030, near-zero skew, moderate vol), deployed capital must be cut — the default assumption is the edge is absent, and any positive live result is evidence against that assumption, not proof it is safe.
@@ -105,44 +106,45 @@ Key observations:
 3. **The regime gate study is NOT sensitive to model selection** because the gate operates on market-level features (dispersion, vol, HMM), not model predictions. Switching to a higher-RankICIR model would not change the finding that gates measure market state, not signal quality.
 4. **Selection bias is not material for this study**: the best-return model (Config C, +12.5%) also has the best RankICIR (0.244) among walk-forward configs. The RankICIR and returns rankings are concordant.
-## Signal-quality gate (Guard 7): the gate that works
+## Signal-quality gate (Guard 7): refuted
-`PROVEN — EVIDENCE#052`
+`REFUTED — EVIDENCE#052 → EVIDENCE#053`
-The regime gate (Guard 6) failed because it answered the wrong question: *"Is the market calm?"* The signal-quality gate asks the right question: *"Are my predictions accurate?"*
+The regime gate (Guard 6) failed because it answered the wrong question: *"Is the market calm?"* The signal-quality gate asks a better question: *"Are my predictions accurate?"* — but when tested properly, it still doesn't work.
 **Logic:** For each day t, look at the topk symbols from yesterday (t-1). Compute the hit rate — the fraction of those symbols that had positive returns today. If the hit rate is above a threshold, keep trading; otherwise, go to cash. This is a retrospective gate — it measures prediction accuracy, not market state.
-**Results across 5 walk-forward windows (2021–2026):**
+**Initial scripted test (EVIDENCE#052):** Precomputed gate from reference pred.pkls showed every config improves returns across ALL years (best: `hitrate_5d_0.50` 2026 +65.0%, 2025 +72.1%, 2024 +30.4%, 2023 +54.7%, 2021 +55.7%). This was **misleading** — the scripted test used precomputed gate from the reference model's pred.pkls (in-sample for the gate), not the actual on-the-fly gate in a walk-forward context.
-| Gate | 2026 base | 2026 gated | 2025 base | 2025 gated | 2024 base | 2024 gated | 2023 base | 2023 gated | 2021 base | 2021 gated |
+**Walk-forward workflow test (EVIDENCE#053):** `WeeklyRebalanceSignalQualityGateStrategy` (topk=10, n_drop=1, gate_topk=10, gate_lookback=5, gate_threshold=0.5, 5/15bp costs) tested via `rd_train` + `rd_run_workflow` on 5 walk-forward windows (2021–2026), retraining the model each year. **The gate is harmful in every year:**
 |------|-----------|------------|-----------|------------|-----------|------------|-----------|------------|-----------|------------|
 | `hitrate_5d_0.50` | +25.5% | **+65.0%** | +17.8% | **+72.1%** | +8.2% | **+30.4%** | −4.8% | **+54.7%** | +18.4% | **+55.7%** |
 | `hitrate_5d_0.60` | +25.5% | +48.9% | +17.8% | +48.6% | +8.2% | +26.5% | −4.8% | +55.6% | +18.4% | +35.2% |
 | `hitrate_10d_0.50` | +25.5% | +33.6% | +17.8% | +43.6% | +8.2% | +27.7% | −4.8% | +46.0% | +18.4% | +46.5% |
 | `hitrate_20d_0.50` | +25.5% | +34.4% | +17.8% | +34.1% | +8.2% | +21.1% | −4.8% | +36.6% | +18.4% | +29.0% |
-`PROVEN — EVIDENCE#052` (scripted simulation: `book/scripts/signal_quality_gate_bt.py`, results `book/data/signal_quality_gate/signal_quality_gate_results.csv`).
+| Year | Workflow excess w/cost (gate) | Reference excess w/cost (nogate) | Delta |
 |------|------------------------------|----------------------------------|-------|
 | 2026 | +9.1% (IR 0.92) | +12.5% (IR 1.24) | **−3.4pp** |
 | 2025 | +3.4% (IR 0.31) | +3.7% (IR 0.33) | **−0.3pp** |
 | 2024 | −20.5% | −19.4% | **−1.1pp** |
 | 2023 | −29.4% | −29.6% | +0.2pp |
 | 2021 | −18.4% | −21.2% | +2.8pp |
-Key observations:
+**Why the scripted test was wrong:** The diagnostic (v3, workflow-exact mechanics) reveals the gate closes 37–45% of days in every year, killing returns:
-1. **Every config improves returns across ALL years** — including the bad years (2023: −4.8% → +54.7%, 2024: +8.2% → +30.4%). The regime gate (Guard 6) destroyed returns in good years; the signal-quality gate improves them everywhere.
+| Year | Script total (nogate) | Script total (gate) | Delta | Gate open% |
 |------|----------------------|--------------------|-------|-----------|
 | 2026 | +21.2% | +1.5% | −19.7pp | 56% |
 | 2025 | +22.7% | +7.8% | −14.9pp | 63% |
 | 2024 | +4.0% | −2.6% | −6.6pp | 59% |
-2. **The gate trips ~40–50% of days** — it's closing on about half the days, filtering out the model's inaccurate predictions. This is the opposite of the regime gate, which closed on the wrong days.
+A model with Rank IC 0.06–0.07 produces many days where <50% of top-10 picks are positive — the gate's 0.5 threshold is too aggressive, closing on profitable weeks. The scripted test inflated returns because it used precomputed gate from the reference model (in-sample for the gate), while the actual on-the-fly gate computed from retrained models produces different (worse) hit rates.
-3. **The 5-day lookback with 0.50 threshold is optimal** — shorter lookbacks (5d) outperform longer ones (10d, 20d) because they adapt faster to changing prediction quality. The 0.50 threshold (random) is the sweet spot — it closes when the model is worse than random.
+**Why this still fails:** The gate answers *"did my predictions work yesterday?"* — but with a 0.06–0.07 Rank IC, yesterday's hit rate is mostly noise. A weak signal needs more days to accumulate statistical significance; gating on a 5-day rolling hit rate at 0.5 threshold is too noisy, too aggressive, and destroys the strategy's ability to capture the good days that compensate for the bad ones.
-4. **The gate is the OPPOSITE of the regime gate**: instead of closing on bad market days, it closes on days when the model's predictions are wrong. The model's predictions ARE informative; they just need to be gated on their own accuracy.
+**Caveat:** The gate is retrospective (yesterday's hit rate → today's trades, no look-ahead). The problem is not look-ahead — it's that the signal is too weak for a 0.5 threshold on a 5-day window to be informative.
 **Why this works:** The regime gate answered *"Is the market calm?"* — but calm markets can produce bad signals (low vol but wrong factor regime), and volatile markets can produce good signals (high vol but correct factor direction). The signal-quality gate answers *"Did my predictions work yesterday?"* — which directly predicts whether they'll work today.
 **Caveat:** This is a retrospective gate — it uses yesterday's hit rate to decide today's trades. In real-time, you'd need to wait for today's close to compute the hit rate, then apply it to tomorrow's trades. The simulation uses yesterday's scores → today's returns (no look-ahead), so the gate is causal.
 ## Desk rules distilled from this chapter
 1. Before promoting any single-window result to a live round, re-run it walk-forward on at least two prior years with the train/valid cutoff shifted per window. If the edge does not survive, it is a regime artifact, not a strategy.
 2. Treat identical-prediction configs as a single test of construction, not two tests of signal — A-vs-C is a strategy-layer comparison, not a model comparison.
-3. Do not ship a guard that cannot select the good regime in hindsight. Feature PSI, label-regime PSI, streaming IC, window length, and staleness all failed on this panel.
+3. Do not ship a guard that cannot select the good regime in hindsight. Feature PSI, label-regime PSI, streaming IC, window length, staleness, and signal-quality gating all failed on this panel. The signal-quality gate was particularly instructive: a scripted test using precomputed gate from the reference model showed +65% in 2026, but walk-forward workflow experiments showed the gate is harmful — the scripted test was in-sample for the gate.
 4. Report account-based curves, not the blotter `return` field — the latter excludes initial cost and does not compound to the account.
 5. When the mean annual excess is negative in every configuration, cut size until the live window demonstrates the regime is back.
@@ -185,8 +187,7 @@ The vol gates show the largest trip differential — they open on more days in 2
 ## Open questions
 - `TODO(evidence-needed: a live window that matches the 2026 label regime, to test whether the edge returns when the regime returns)`
- `TODO(evidence-needed: signal-quality gate tested on out-of-sample data — the current test uses the same pred.pkl for gate computation and trading, which is in-sample for the gate itself)`
+- `TODO(evidence-needed: understanding the script-vs-workflow gap for signal-quality gate — scripted test shows gate destroying ~20pp more return than workflow, despite identical parameters; root cause is pred date alignment differences between precomputed and on-the-fly gate computation)`
 - `TODO(evidence-needed: signal-quality gate combined with the regime gate — does layering both gates improve results further?)`
 ## Evidence cited in this chapter
@@ -201,4 +202,5 @@ The vol gates show the largest trip differential — they open on more days in 2
 | `EVIDENCE#049` | Perturbation stress test on Config A 2026 (exp 52, pred `9f98ea5c`): topk/n_drop/cost grid, `book/data/perturbation/config_a_2026_sensitivity.json` |
 | `EVIDENCE#050` | Regime gate walk-forward test (2021–2026): 3 detector types × 14 configs; scripted simulation `book/scripts/regime_gate_bt.py`, results `book/data/regime_gate/regime_gate_trip_rates.csv` |
 | `EVIDENCE#051` | Comprehensive model search: all experiments ranked by RankICIR; regime gate study robust to model selection; `rd_exp_list` + `rd_exp_get_run` queries |
-| `EVIDENCE#052` | Signal-quality gate (hit-rate based on topk predictions): every config improves returns across ALL years; scripted simulation `book/scripts/signal_quality_gate_bt.py`, results `book/data/signal_quality_gate/signal_quality_gate_results.csv` |
+| `EVIDENCE#052` | Signal-quality gate scripted test: precomputed gate from reference pred.pkls showed every config improves returns across ALL years. **REFUTED by EVIDENCE#053** — scripted test was in-sample for the gate. |
 | `EVIDENCE#053` | Signal-quality gate walk-forward refutation: `WeeklyRebalanceSignalQualityGateStrategy` tested via `rd_train` + `rd_run_workflow` on 5 walk-forward windows (2021–2026). Gate harmful in every year: 2026 +9.1% vs +12.5% reference (−3.4pp), 2025 +3.4% vs +3.7% (−0.3pp). Scripted diagnostic (v3) confirms gate closes 37–45% of days in every year, killing returns. |
@@ -0,0 +1,117 @@
 [
  {
    "year": "2026",
    "script_weekly_100": {
      "ann_ret": 0.0921,
      "sharpe": 0.5512,
      "maxDD": -0.0947
    },
    "script_weekly_95": {
      "ann_ret": 0.088,
      "sharpe": 0.5545,
      "maxDD": -0.0901
    },
    "daily_topk_100": {
      "ann_ret": -0.0914,
      "sharpe": -0.5241,
      "maxDD": -0.1446
    },
    "daily_topk_95": {
      "ann_ret": -0.0863,
      "sharpe": -0.5214,
      "maxDD": -0.1376
    }
  },
  {
    "year": "2025",
    "script_weekly_100": {
      "ann_ret": 0.1774,
      "sharpe": 0.9416,
      "maxDD": -0.1809
    },
    "script_weekly_95": {
      "ann_ret": 0.1688,
      "sharpe": 0.9431,
      "maxDD": -0.1725
    },
    "daily_topk_100": {
      "ann_ret": -0.2439,
      "sharpe": -0.8798,
      "maxDD": -0.2807
    },
    "daily_topk_95": {
      "ann_ret": -0.2316,
      "sharpe": -0.8794,
      "maxDD": -0.2677
    }
  },
  {
    "year": "2024",
    "script_weekly_100": {
      "ann_ret": 0.0316,
      "sharpe": 0.232,
      "maxDD": -0.0807
    },
    "script_weekly_95": {
      "ann_ret": 0.0304,
      "sharpe": 0.2353,
      "maxDD": -0.0768
    },
    "daily_topk_100": {
      "ann_ret": -0.041,
      "sharpe": -0.2846,
      "maxDD": -0.1248
    },
    "daily_topk_95": {
      "ann_ret": -0.0385,
      "sharpe": -0.2815,
      "maxDD": -0.1188
    }
  },
  {
    "year": "2023",
    "script_weekly_100": {
      "ann_ret": 0.1025,
      "sharpe": 0.7325,
      "maxDD": -0.1237
    },
    "script_weekly_95": {
      "ann_ret": 0.0976,
      "sharpe": 0.7345,
      "maxDD": -0.1178
    },
    "daily_topk_100": {
      "ann_ret": 0.1847,
      "sharpe": 1.2305,
      "maxDD": -0.1314
    },
    "daily_topk_95": {
      "ann_ret": 0.1753,
      "sharpe": 1.2295,
      "maxDD": -0.1252
    }
  },
  {
    "year": "2021",
    "script_weekly_100": {
      "ann_ret": 0.1458,
      "sharpe": 0.8808,
      "maxDD": -0.1115
    },
    "script_weekly_95": {
      "ann_ret": 0.1387,
      "sharpe": 0.8824,
      "maxDD": -0.1061
    },
    "daily_topk_100": {
      "ann_ret": 0.0009,
      "sharpe": 0.0051,
      "maxDD": -0.129
    },
    "daily_topk_95": {
      "ann_ret": 0.0016,
      "sharpe": 0.0095,
      "maxDD": -0.1228
    }
  }
 ]
@@ -0,0 +1,142 @@
 [
  {
    "year": "2026",
    "weekly_100_zc": {
      "ann_ret": 0.0921,
      "sharpe": 0.5512,
      "maxDD": -0.0947
    },
    "weekly_95_zc": {
      "ann_ret": 0.088,
      "sharpe": 0.5545,
      "maxDD": -0.0901
    },
    "weekly_95_10bp": {
      "ann_ret": 0.0647,
      "sharpe": 0.4077,
      "maxDD": -0.0945
    },
    "daily_100_zc": {
      "ann_ret": -0.0914,
      "sharpe": -0.5241,
      "maxDD": -0.1446
    },
    "daily_100_10bp": {
      "ann_ret": -0.1581,
      "sharpe": -0.9069,
      "maxDD": -0.1765
    }
  },
  {
    "year": "2025",
    "weekly_100_zc": {
      "ann_ret": 0.1774,
      "sharpe": 0.9416,
      "maxDD": -0.1809
    },
    "weekly_95_zc": {
      "ann_ret": 0.1688,
      "sharpe": 0.9431,
      "maxDD": -0.1725
    },
    "weekly_95_10bp": {
      "ann_ret": 0.1436,
      "sharpe": 0.8026,
      "maxDD": -0.1754
    },
    "daily_100_zc": {
      "ann_ret": -0.2439,
      "sharpe": -0.8798,
      "maxDD": -0.2807
    },
    "daily_100_10bp": {
      "ann_ret": -0.2975,
      "sharpe": -1.0722,
      "maxDD": -0.314
    }
  },
  {
    "year": "2024",
    "weekly_100_zc": {
      "ann_ret": 0.0316,
      "sharpe": 0.232,
      "maxDD": -0.0807
    },
    "weekly_95_zc": {
      "ann_ret": 0.0304,
      "sharpe": 0.2353,
      "maxDD": -0.0768
    },
    "weekly_95_10bp": {
      "ann_ret": 0.0051,
      "sharpe": 0.0393,
      "maxDD": -0.0796
    },
    "daily_100_zc": {
      "ann_ret": -0.041,
      "sharpe": -0.2846,
      "maxDD": -0.1248
    },
    "daily_100_10bp": {
      "ann_ret": -0.1126,
      "sharpe": -0.7819,
      "maxDD": -0.1548
    }
  },
  {
    "year": "2023",
    "weekly_100_zc": {
      "ann_ret": 0.1025,
      "sharpe": 0.7325,
      "maxDD": -0.1237
    },
    "weekly_95_zc": {
      "ann_ret": 0.0976,
      "sharpe": 0.7345,
      "maxDD": -0.1178
    },
    "weekly_95_10bp": {
      "ann_ret": 0.0724,
      "sharpe": 0.5449,
      "maxDD": -0.1238
    },
    "daily_100_zc": {
      "ann_ret": 0.1847,
      "sharpe": 1.2305,
      "maxDD": -0.1314
    },
    "daily_100_10bp": {
      "ann_ret": 0.0967,
      "sharpe": 0.6449,
      "maxDD": -0.1481
    }
  },
  {
    "year": "2021",
    "weekly_100_zc": {
      "ann_ret": 0.1279,
      "sharpe": 0.7781,
      "maxDD": -0.1122
    },
    "weekly_95_zc": {
      "ann_ret": 0.1219,
      "sharpe": 0.7803,
      "maxDD": -0.1068
    },
    "weekly_95_10bp": {
      "ann_ret": 0.0971,
      "sharpe": 0.6218,
      "maxDD": -0.1079
    },
    "daily_100_zc": {
      "ann_ret": -0.0098,
      "sharpe": -0.0564,
      "maxDD": -0.1296
    },
    "daily_100_10bp": {
      "ann_ret": -0.089,
      "sharpe": -0.509,
      "maxDD": -0.1869
    }
  }
 ]
@@ -0,0 +1,142 @@
 [
  {
    "year": "2026",
    "ideal_100_zc": {
      "ann_ret": 0.0921,
      "sharpe": 0.5512,
      "maxDD": -0.0947
    },
    "ideal_95_zc": {
      "ann_ret": 0.088,
      "sharpe": 0.5545,
      "maxDD": -0.0901
    },
    "ideal_95_10bp": {
      "ann_ret": 0.0647,
      "sharpe": 0.4077,
      "maxDD": -0.0945
    },
    "wf_exact_95_nogate": {
      "ann_ret": 0.2117,
      "sharpe": 1.2681,
      "maxDD": -0.1039
    },
    "wf_exact_95_gate": {
      "ann_ret": 0.0146,
      "sharpe": 0.1372,
      "maxDD": -0.0668
    }
  },
  {
    "year": "2025",
    "ideal_100_zc": {
      "ann_ret": 0.1774,
      "sharpe": 0.9416,
      "maxDD": -0.1809
    },
    "ideal_95_zc": {
      "ann_ret": 0.1688,
      "sharpe": 0.9431,
      "maxDD": -0.1725
    },
    "ideal_95_10bp": {
      "ann_ret": 0.1436,
      "sharpe": 0.8026,
      "maxDD": -0.1754
    },
    "wf_exact_95_nogate": {
      "ann_ret": 0.2273,
      "sharpe": 1.1963,
      "maxDD": -0.1864
    },
    "wf_exact_95_gate": {
      "ann_ret": 0.0777,
      "sharpe": 0.8239,
      "maxDD": -0.0773
    }
  },
  {
    "year": "2024",
    "ideal_100_zc": {
      "ann_ret": 0.0316,
      "sharpe": 0.232,
      "maxDD": -0.0807
    },
    "ideal_95_zc": {
      "ann_ret": 0.0304,
      "sharpe": 0.2353,
      "maxDD": -0.0768
    },
    "ideal_95_10bp": {
      "ann_ret": 0.0051,
      "sharpe": 0.0393,
      "maxDD": -0.0796
    },
    "wf_exact_95_nogate": {
      "ann_ret": 0.0616,
      "sharpe": 0.4145,
      "maxDD": -0.0951
    },
    "wf_exact_95_gate": {
      "ann_ret": -0.0261,
      "sharpe": -0.2782,
      "maxDD": -0.1157
    }
  },
  {
    "year": "2023",
    "ideal_100_zc": {
      "ann_ret": 0.1025,
      "sharpe": 0.7325,
      "maxDD": -0.1237
    },
    "ideal_95_zc": {
      "ann_ret": 0.0976,
      "sharpe": 0.7345,
      "maxDD": -0.1178
    },
    "ideal_95_10bp": {
      "ann_ret": 0.0724,
      "sharpe": 0.5449,
      "maxDD": -0.1238
    },
    "wf_exact_95_nogate": {
      "ann_ret": 0.0327,
      "sharpe": 0.2259,
      "maxDD": -0.1517
    },
    "wf_exact_95_gate": {
      "ann_ret": 0.0094,
      "sharpe": 0.0968,
      "maxDD": -0.1036
    }
  },
  {
    "year": "2021",
    "ideal_100_zc": {
      "ann_ret": 0.1279,
      "sharpe": 0.7781,
      "maxDD": -0.1122
    },
    "ideal_95_zc": {
      "ann_ret": 0.1219,
      "sharpe": 0.7803,
      "maxDD": -0.1068
    },
    "ideal_95_10bp": {
      "ann_ret": 0.0971,
      "sharpe": 0.6218,
      "maxDD": -0.1079
    },
    "wf_exact_95_nogate": {
      "ann_ret": 0.0476,
      "sharpe": 0.0429,
      "maxDD": -0.3489
    },
    "wf_exact_95_gate": {
      "ann_ret": -0.0227,
      "sharpe": -0.0318,
      "maxDD": -0.2398
    }
  }
 ]
@@ -0,0 +1,91 @@
 source,window,gate,start,end,trade_dates,gate_open,gate_closed,trip_rate,base_ann,base_sharpe,base_maxDD,gated_ann,gated_sharpe,gated_maxDD
 retrained,2026,hitrate_5d_0.50,2026-01-04,2026-08-19,150,84,66,0.44,0.253239,1.517,-0.067104,0.648996,6.2501,-0.023442
 retrained,2026,hitrate_5d_0.60,2026-01-04,2026-08-19,150,38,112,0.7467,0.253239,1.517,-0.067104,0.458143,5.9258,-0.013092
 retrained,2026,hitrate_5d_0.70,2026-01-04,2026-08-19,150,12,138,0.92,0.253239,1.517,-0.067104,0.176528,3.9923,-0.005153
 retrained,2026,hitrate_10d_0.50,2026-01-04,2026-08-19,150,86,64,0.4267,0.253239,1.517,-0.067104,0.288162,2.5775,-0.039705
 retrained,2026,hitrate_10d_0.60,2026-01-04,2026-08-19,150,26,124,0.8267,0.253239,1.517,-0.067104,0.145763,2.968,-0.013348
 retrained,2026,hitrate_10d_0.70,2026-01-04,2026-08-19,150,5,145,0.9667,0.253239,1.517,-0.067104,-0.003575,-0.314,-0.008187
 retrained,2026,hitrate_20d_0.40,2026-01-04,2026-08-19,150,148,2,0.0133,0.253239,1.517,-0.067104,0.286286,1.7156,-0.067104
 retrained,2026,hitrate_20d_0.50,2026-01-04,2026-08-19,150,99,51,0.34,0.253239,1.517,-0.067104,0.268662,2.2268,-0.040318
 retrained,2026,hitrate_20d_0.60,2026-01-04,2026-08-19,150,12,138,0.92,0.253239,1.517,-0.067104,0.031825,1.688,-0.008187
 retrained,2025,hitrate_5d_0.50,2025-01-02,2025-12-31,250,155,95,0.38,0.155804,0.8373,-0.196751,0.628161,6.0366,-0.055513
 retrained,2025,hitrate_5d_0.60,2025-01-02,2025-12-31,250,96,154,0.616,0.155804,0.8373,-0.196751,0.376382,4.6887,-0.047153
 retrained,2025,hitrate_5d_0.70,2025-01-02,2025-12-31,250,37,213,0.852,0.155804,0.8373,-0.196751,0.224845,4.7438,-0.008613
 retrained,2025,hitrate_10d_0.50,2025-01-02,2025-12-31,250,166,84,0.336,0.155804,0.8373,-0.196751,0.374546,3.2685,-0.055513
 retrained,2025,hitrate_10d_0.60,2025-01-02,2025-12-31,250,71,179,0.716,0.155804,0.8373,-0.196751,0.238719,4.022,-0.013238
 retrained,2025,hitrate_10d_0.70,2025-01-02,2025-12-31,250,13,237,0.948,0.155804,0.8373,-0.196751,0.034654,1.1514,-0.008803
 retrained,2025,hitrate_20d_0.40,2025-01-02,2025-12-31,250,250,0,0.0,0.155804,0.8373,-0.196751,0.155804,0.8373,-0.196751
 retrained,2025,hitrate_20d_0.50,2025-01-02,2025-12-31,250,185,65,0.26,0.155804,0.8373,-0.196751,0.352176,3.3794,-0.033798
 retrained,2025,hitrate_20d_0.60,2025-01-02,2025-12-31,250,54,196,0.784,0.155804,0.8373,-0.196751,0.212675,3.3231,-0.023704
 retrained,2024,hitrate_5d_0.50,2024-01-02,2024-12-31,253,145,108,0.4269,-0.031528,-0.2293,-0.08828,0.431873,4.428,-0.028721
 retrained,2024,hitrate_5d_0.60,2024-01-02,2024-12-31,253,70,183,0.7233,-0.031528,-0.2293,-0.08828,0.381231,5.4692,-0.015471
 retrained,2024,hitrate_5d_0.70,2024-01-02,2024-12-31,253,25,228,0.9012,-0.031528,-0.2293,-0.08828,0.152242,3.8328,-0.004761
 retrained,2024,hitrate_10d_0.50,2024-01-02,2024-12-31,253,156,97,0.3834,-0.031528,-0.2293,-0.08828,0.25073,2.3932,-0.037139
 retrained,2024,hitrate_10d_0.60,2024-01-02,2024-12-31,253,44,209,0.8261,-0.031528,-0.2293,-0.08828,0.127297,2.2623,-0.022165
 retrained,2024,hitrate_10d_0.70,2024-01-02,2024-12-31,253,8,245,0.9684,-0.031528,-0.2293,-0.08828,0.037433,1.858,-0.00145
 retrained,2024,hitrate_20d_0.40,2024-01-02,2024-12-31,253,247,6,0.0237,-0.031528,-0.2293,-0.08828,0.016863,0.1236,-0.08828
 retrained,2024,hitrate_20d_0.50,2024-01-02,2024-12-31,253,175,78,0.3083,-0.031528,-0.2293,-0.08828,0.097603,0.8829,-0.067765
 retrained,2024,hitrate_20d_0.60,2024-01-02,2024-12-31,253,17,236,0.9328,-0.031528,-0.2293,-0.08828,-0.037611,-1.0795,-0.04407
 retrained,2023,hitrate_5d_0.50,2023-01-03,2023-12-29,250,131,119,0.476,0.059638,0.3986,-0.126894,0.580855,5.6532,-0.024506
 retrained,2023,hitrate_5d_0.60,2023-01-03,2023-12-29,250,71,179,0.716,0.059638,0.3986,-0.126894,0.433179,5.5698,-0.027182
 retrained,2023,hitrate_5d_0.70,2023-01-03,2023-12-29,250,32,218,0.872,0.059638,0.3986,-0.126894,0.202111,3.4801,-0.021868
 retrained,2023,hitrate_10d_0.50,2023-01-03,2023-12-29,250,135,115,0.46,0.059638,0.3986,-0.126894,0.400163,3.8554,-0.032201
 retrained,2023,hitrate_10d_0.60,2023-01-03,2023-12-29,250,77,173,0.692,0.059638,0.3986,-0.126894,0.272342,3.7048,-0.021999
 retrained,2023,hitrate_10d_0.70,2023-01-03,2023-12-29,250,13,237,0.948,0.059638,0.3986,-0.126894,0.056813,1.5786,-0.009656
 retrained,2023,hitrate_20d_0.40,2023-01-03,2023-12-29,250,235,15,0.06,0.059638,0.3986,-0.126894,0.079439,0.5449,-0.11414
 retrained,2023,hitrate_20d_0.50,2023-01-03,2023-12-29,250,131,119,0.476,0.059638,0.3986,-0.126894,0.279296,2.5428,-0.045109
 retrained,2023,hitrate_20d_0.60,2023-01-03,2023-12-29,250,52,198,0.792,0.059638,0.3986,-0.126894,0.168286,2.6336,-0.035737
 retrained,2021,hitrate_5d_0.50,2021-01-04,2021-12-31,252,149,103,0.4087,0.134183,0.8167,-0.11453,0.441222,4.9947,-0.031256
 retrained,2021,hitrate_5d_0.60,2021-01-04,2021-12-31,252,90,162,0.6429,0.134183,0.8167,-0.11453,0.426775,6.3005,-0.018787
 retrained,2021,hitrate_5d_0.70,2021-01-04,2021-12-31,252,36,216,0.8571,0.134183,0.8167,-0.11453,0.193485,4.3539,-0.008861
 retrained,2021,hitrate_10d_0.50,2021-01-04,2021-12-31,252,159,93,0.369,0.134183,0.8167,-0.11453,0.359362,3.3378,-0.070675
 retrained,2021,hitrate_10d_0.60,2021-01-04,2021-12-31,252,67,185,0.7341,0.134183,0.8167,-0.11453,0.234921,4.4088,-0.016687
 retrained,2021,hitrate_10d_0.70,2021-01-04,2021-12-31,252,10,242,0.9603,0.134183,0.8167,-0.11453,0.047502,2.0297,-0.002111
 retrained,2021,hitrate_20d_0.40,2021-01-04,2021-12-31,252,252,0,0.0,0.134183,0.8167,-0.11453,0.134183,0.8167,-0.11453
 retrained,2021,hitrate_20d_0.50,2021-01-04,2021-12-31,252,175,77,0.3056,0.134183,0.8167,-0.11453,0.207217,1.7504,-0.060921
 retrained,2021,hitrate_20d_0.60,2021-01-04,2021-12-31,252,44,208,0.8254,0.134183,0.8167,-0.11453,0.110265,2.2204,-0.016687
 reference,2026,hitrate_5d_0.50,2026-01-04,2026-08-19,157,92,65,0.414,0.255023,1.4465,-0.080671,0.649911,5.7351,-0.031795
 reference,2026,hitrate_5d_0.60,2026-01-04,2026-08-19,157,49,108,0.6879,0.255023,1.4465,-0.080671,0.489313,6.0111,-0.014864
 reference,2026,hitrate_5d_0.70,2026-01-04,2026-08-19,157,15,142,0.9045,0.255023,1.4465,-0.080671,0.229523,4.3611,-0.002029
 reference,2026,hitrate_10d_0.50,2026-01-04,2026-08-19,157,109,48,0.3057,0.255023,1.4465,-0.080671,0.335822,2.6391,-0.060142
 reference,2026,hitrate_10d_0.60,2026-01-04,2026-08-19,157,36,121,0.7707,0.255023,1.4465,-0.080671,0.251108,3.8579,-0.026842
 reference,2026,hitrate_10d_0.70,2026-01-04,2026-08-19,157,9,148,0.9427,0.255023,1.4465,-0.080671,0.039314,1.3415,-0.012061
 reference,2026,hitrate_20d_0.40,2026-01-04,2026-08-19,157,157,0,0.0,0.255023,1.4465,-0.080671,0.255023,1.4465,-0.080671
 reference,2026,hitrate_20d_0.50,2026-01-04,2026-08-19,157,118,39,0.2484,0.255023,1.4465,-0.080671,0.343819,2.5185,-0.06259
 reference,2026,hitrate_20d_0.60,2026-01-04,2026-08-19,157,28,129,0.8217,0.255023,1.4465,-0.080671,0.020214,0.3601,-0.028489
 reference,2025,hitrate_5d_0.50,2025-01-02,2025-12-31,250,153,97,0.388,0.177515,0.8573,-0.217417,0.72055,6.9599,-0.048495
 reference,2025,hitrate_5d_0.60,2025-01-02,2025-12-31,250,90,160,0.64,0.177515,0.8573,-0.217417,0.48564,5.3623,-0.046673
 reference,2025,hitrate_5d_0.70,2025-01-02,2025-12-31,250,39,211,0.844,0.177515,0.8573,-0.217417,0.249221,4.9448,-0.009619
 reference,2025,hitrate_10d_0.50,2025-01-02,2025-12-31,250,170,80,0.32,0.177515,0.8573,-0.217417,0.435847,3.5614,-0.055776
 reference,2025,hitrate_10d_0.60,2025-01-02,2025-12-31,250,66,184,0.736,0.177515,0.8573,-0.217417,0.324574,5.2279,-0.016028
 reference,2025,hitrate_10d_0.70,2025-01-02,2025-12-31,250,14,236,0.944,0.177515,0.8573,-0.217417,0.043002,1.4635,-0.010154
 reference,2025,hitrate_20d_0.40,2025-01-02,2025-12-31,250,247,3,0.012,0.177515,0.8573,-0.217417,0.202949,0.9896,-0.217417
 reference,2025,hitrate_20d_0.50,2025-01-02,2025-12-31,250,177,73,0.292,0.177515,0.8573,-0.217417,0.340772,2.8303,-0.071731
 reference,2025,hitrate_20d_0.60,2025-01-02,2025-12-31,250,48,202,0.808,0.177515,0.8573,-0.217417,0.266028,4.1749,-0.021527
 reference,2024,hitrate_5d_0.50,2024-01-02,2024-12-31,253,139,114,0.4506,0.08229,0.5594,-0.10685,0.30351,2.4275,-0.088219
 reference,2024,hitrate_5d_0.60,2024-01-02,2024-12-31,253,71,182,0.7194,0.08229,0.5594,-0.10685,0.265365,2.4908,-0.078304
 reference,2024,hitrate_5d_0.70,2024-01-02,2024-12-31,253,19,234,0.9249,0.08229,0.5594,-0.10685,0.143125,3.377,-0.003364
 reference,2024,hitrate_10d_0.50,2024-01-02,2024-12-31,253,157,96,0.3794,0.08229,0.5594,-0.10685,0.277421,2.5767,-0.043146
 reference,2024,hitrate_10d_0.60,2024-01-02,2024-12-31,253,37,216,0.8538,0.08229,0.5594,-0.10685,0.207161,3.6083,-0.011122
 reference,2024,hitrate_10d_0.70,2024-01-02,2024-12-31,253,7,246,0.9723,0.08229,0.5594,-0.10685,0.031785,1.7408,-0.002083
 reference,2024,hitrate_20d_0.40,2024-01-02,2024-12-31,253,247,6,0.0237,0.08229,0.5594,-0.10685,0.078007,0.5342,-0.10685
 reference,2024,hitrate_20d_0.50,2024-01-02,2024-12-31,253,177,76,0.3004,0.08229,0.5594,-0.10685,0.210672,1.8469,-0.056207
 reference,2024,hitrate_20d_0.60,2024-01-02,2024-12-31,253,13,240,0.9486,0.08229,0.5594,-0.10685,0.005953,0.3039,-0.014443
 reference,2023,hitrate_5d_0.50,2023-01-03,2023-12-29,250,139,111,0.444,-0.047644,-0.2738,-0.197856,0.546654,4.2124,-0.035676
 reference,2023,hitrate_5d_0.60,2023-01-03,2023-12-29,250,79,171,0.684,-0.047644,-0.2738,-0.197856,0.556291,5.1026,-0.025449
 reference,2023,hitrate_5d_0.70,2023-01-03,2023-12-29,250,34,216,0.864,-0.047644,-0.2738,-0.197856,0.404269,4.4477,-0.013842
 reference,2023,hitrate_10d_0.50,2023-01-03,2023-12-29,250,148,102,0.408,-0.047644,-0.2738,-0.197856,0.459869,3.5211,-0.046921
 reference,2023,hitrate_10d_0.60,2023-01-03,2023-12-29,250,62,188,0.752,-0.047644,-0.2738,-0.197856,0.368232,4.0148,-0.022983
 reference,2023,hitrate_10d_0.70,2023-01-03,2023-12-29,250,13,237,0.948,-0.047644,-0.2738,-0.197856,0.091481,2.1324,-0.010866
 reference,2023,hitrate_20d_0.40,2023-01-03,2023-12-29,250,237,13,0.052,-0.047644,-0.2738,-0.197856,0.06639,0.3895,-0.146704
 reference,2023,hitrate_20d_0.50,2023-01-03,2023-12-29,250,149,101,0.404,-0.047644,-0.2738,-0.197856,0.366016,2.5969,-0.063998
 reference,2023,hitrate_20d_0.60,2023-01-03,2023-12-29,250,40,210,0.84,-0.047644,-0.2738,-0.197856,0.149844,2.2645,-0.032267
 reference,2021,hitrate_5d_0.50,2021-01-04,2021-12-31,252,145,107,0.4246,0.18367,1.0979,-0.102651,0.556615,5.9167,-0.028062
 reference,2021,hitrate_5d_0.60,2021-01-04,2021-12-31,252,68,184,0.7302,0.18367,1.0979,-0.102651,0.351867,5.8468,-0.011638
 reference,2021,hitrate_5d_0.70,2021-01-04,2021-12-31,252,26,226,0.8968,0.18367,1.0979,-0.102651,0.148417,3.9593,-0.0041
 reference,2021,hitrate_10d_0.50,2021-01-04,2021-12-31,252,163,89,0.3532,0.18367,1.0979,-0.102651,0.465023,4.2342,-0.040569
 reference,2021,hitrate_10d_0.60,2021-01-04,2021-12-31,252,51,201,0.7976,0.18367,1.0979,-0.102651,0.21802,4.7978,-0.011134
 reference,2021,hitrate_10d_0.70,2021-01-04,2021-12-31,252,13,239,0.9484,0.18367,1.0979,-0.102651,0.06108,2.7055,-0.000262
 reference,2021,hitrate_20d_0.40,2021-01-04,2021-12-31,252,252,0,0.0,0.18367,1.0979,-0.102651,0.18367,1.0979,-0.102651
 reference,2021,hitrate_20d_0.50,2021-01-04,2021-12-31,252,166,86,0.3413,0.18367,1.0979,-0.102651,0.289756,2.5787,-0.049244
 reference,2021,hitrate_20d_0.60,2021-01-04,2021-12-31,252,38,214,0.8492,0.18367,1.0979,-0.102651,0.104685,2.394,-0.015084
@@ -0,0 +1,191 @@
 """Diagnose exactly why the scripted test and workflow give different results.
 Compares the same pred.pkl through:
  1. Script logic (weekly rebalance, equal-weight, hold-through-week, zero cost)
  2. Workflow logic (PortAnaRecord daily backtest, TopkDropout-like)
 Isolates the effect of:
  A. Weekly vs daily position evaluation
  B. Equal weight vs risk_degree sizing
  C. Hold-through-week vs daily top-k re-ranking
 """
 from __future__ import annotations
 import json, pathlib
 import numpy as np
 import pandas as pd
 LAKE_ROOT = "/home/data/lake"
 OUT = pathlib.Path("/app/experiments/book/data/diag_script_vs_wf")
 WINDOWS = [
    {"label": "2026", "start": "2026-01-04", "end": "2026-08-19",
     "pred": f"{LAKE_ROOT}/mlruns/62/3771f96eb1b74365aeae966af7aec5a3/artifacts/pred.pkl"},
    {"label": "2025", "start": "2025-01-02", "end": "2025-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/c57c6a8370cc48619d7cdd2bd109b76a/artifacts/pred.pkl"},
    {"label": "2024", "start": "2024-01-02", "end": "2024-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/97cf5f282e6f4e699443e38d9bfb40fd/artifacts/pred.pkl"},
    {"label": "2023", "start": "2023-01-03", "end": "2023-12-29",
     "pred": f"{LAKE_ROOT}/mlruns/62/11b9b65ea4e14b3f8ce50d244da0412e/artifacts/pred.pkl"},
    {"label": "2021", "start": "2021-01-04", "end": "2021-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/af3034e5910348a382f2ad1e1741f17c/artifacts/pred.pkl"},
 ]
 SYMS = [
    "SPY","QQQ","DIA","IWM","MDY","VTI","VOO","VEA","VWO","VT","EFA","EEM",
    "TLT","IEF","SHY","AGG","BND","LQD","HYG","JNK","EMB","GLD","SLV",
    "USO","UNG","DBA","DBC","XLK","XLF","XLE","XLV","XLI","XLY","XLP",
    "XLU","XLB","XLRE","ARKK","SMH","SOXX","IBB","XBI","ITA","XAR",
    "ICLN","TAN","FDN","IGV","ESPO","REM",
 ]
 def load_pred(path):
    df = pd.read_pickle(path)
    s = df["score"] if isinstance(df, pd.DataFrame) and "score" in df.columns else df.iloc[:, 0] if isinstance(df, pd.DataFrame) else df
    idx = s.index
    new_dt = pd.to_datetime(idx.get_level_values(0)).normalize()
    s.index = pd.MultiIndex.from_arrays([new_dt, idx.get_level_values(1)], names=idx.names)
    return s
 def load_closes(start, end):
    from tac_qlib.data.config import LakeConfig, resolve_lake_root
    cfg = LakeConfig(resolve_lake_root(LAKE_ROOT), "US")
    closes = {}
    for sym in SYMS:
        p = cfg.bar_path("1d", sym)
        if not p.exists(): continue
        try:
            df = pd.read_parquet(p)
        except: continue
        if not len(df): continue
        tcol = df["t"] if "t" in df.columns else df["date"]
        ts = pd.to_datetime(tcol)
        df = df.assign(_t=ts).set_index("_t").sort_index()
        warmup = pd.Timestamp(start) - pd.Timedelta(days=60)
        df = df.loc[warmup:end]
        if len(df) >= 22:
            closes[sym] = df["c"]
    return pd.DataFrame(closes)
 def strategy_script(pred, closes, start, end, topk=10, risk_degree=1.0):
    """Mimics the scripted test: weekly rebalance, hold all week."""
    ret_df = closes.pct_change()
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    trade_dates = sorted(dt_idx[(dt_idx >= start) & (dt_idx <= end)].unique())
    equity = 1_000_000.0
    holdings = []
    prev_week = None
    daily_eq = []
    for d in trade_dates:
        try:
            day_scores = pred.loc[d]
        except KeyError:
            daily_eq.append(equity)
            prev_scores = None
            continue
        if isinstance(day_scores, pd.DataFrame):
            day_scores = day_scores.iloc[:, 0]
        day_scores = day_scores.dropna().sort_values(ascending=False)
        cur_week = (d.isocalendar()[0], d.isocalendar()[1])
        if cur_week != prev_week or not holdings:
            holdings = list(day_scores.index[:topk])
        ret_row = ret_df.loc[d] if d in ret_df.index else None
        if ret_row is not None and holdings:
            wts = np.array([risk_degree / len(holdings)] * len(holdings))
            rets = ret_row.reindex(holdings).fillna(0).values
            equity *= (1 + (wts * rets).sum())
        daily_eq.append(equity)
        prev_week = cur_week
    return pd.Series(daily_eq, index=trade_dates)
 def strategy_daily_topk(pred, closes, start, end, topk=10, risk_degree=1.0):
    """Mimics PortAnaRecord: re-rank every day, hold top-k."""
    ret_df = closes.pct_change()
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    trade_dates = sorted(dt_idx[(dt_idx >= start) & (dt_idx <= end)].unique())
    equity = 1_000_000.0
    daily_eq = []
    for d in trade_dates:
        try:
            day_scores = pred.loc[d]
        except KeyError:
            daily_eq.append(equity)
            continue
        if isinstance(day_scores, pd.DataFrame):
            day_scores = day_scores.iloc[:, 0]
        day_scores = day_scores.dropna().sort_values(ascending=False)
        holdings = list(day_scores.index[:topk])
        ret_row = ret_df.loc[d] if d in ret_df.index else None
        if ret_row is not None and holdings:
            wts = np.array([risk_degree / len(holdings)] * len(holdings))
            rets = ret_row.reindex(holdings).fillna(0).values
            equity *= (1 + (wts * rets).sum())
        daily_eq.append(equity)
    return pd.Series(daily_eq, index=trade_dates)
 def metrics(eq):
    if len(eq) < 2:
        return {"ann_ret": 0, "sharpe": 0, "maxDD": 0}
    rets = eq.pct_change().dropna()
    ann_ret = float((eq.iloc[-1] / eq.iloc[0]) ** (252 / max(len(eq), 1)) - 1)
    vol = float(rets.std() * (252 ** 0.5)) if len(rets) > 1 else 0
    sharpe = ann_ret / vol if vol > 0 else 0
    peak = eq.cummax()
    dd = (eq - peak) / peak
    return {"ann_ret": round(ann_ret, 4), "sharpe": round(sharpe, 4), "maxDD": round(float(dd.min()), 4)}
 def main():
    OUT.mkdir(parents=True, exist_ok=True)
    results = []
    for w in WINDOWS:
        print(f"\n=== {w['label']} ({w['start']} to {w['end']}) ===")
        pred = load_pred(w["pred"])
        closes = load_closes(w["start"], w["end"])
        print(f"  pred dates: {pred.index.get_level_values(0).min()} to {pred.index.get_level_values(0).max()}")
        print(f"  close dates: {closes.index.min()} to {closes.index.max()}")
        print(f"  symbols in close: {closes.shape[1]}")
        # Script: weekly, equal weight (risk_degree=1.0)
        eq_weekly_100 = strategy_script(pred, closes, w["start"], w["end"], topk=10, risk_degree=1.0)
        m_weekly_100 = metrics(eq_weekly_100)
        # Script: weekly, 95% risk degree
        eq_weekly_95 = strategy_script(pred, closes, w["start"], w["end"], topk=10, risk_degree=0.95)
        m_weekly_95 = metrics(eq_weekly_95)
        # Daily top-k: re-rank daily, equal weight
        eq_daily_100 = strategy_daily_topk(pred, closes, w["start"], w["end"], topk=10, risk_degree=1.0)
        m_daily_100 = metrics(eq_daily_100)
        # Daily top-k: re-rank daily, 95%
        eq_daily_95 = strategy_daily_topk(pred, closes, w["start"], w["end"], topk=10, risk_degree=0.95)
        m_daily_95 = metrics(eq_daily_95)
        row = {
            "year": w["label"],
            "script_weekly_100": m_weekly_100,
            "script_weekly_95": m_weekly_95,
            "daily_topk_100": m_daily_100,
            "daily_topk_95": m_daily_95,
        }
        results.append(row)
        print(f"  Script weekly 100%: ann={m_weekly_100['ann_ret']:+.1%} sharpe={m_weekly_100['sharpe']:.2f} maxDD={m_weekly_100['maxDD']:.1%}")
        print(f"  Script weekly  95%: ann={m_weekly_95['ann_ret']:+.1%} sharpe={m_weekly_95['sharpe']:.2f} maxDD={m_weekly_95['maxDD']:.1%}")
        print(f"  Daily topk   100%: ann={m_daily_100['ann_ret']:+.1%} sharpe={m_daily_100['sharpe']:.2f} maxDD={m_daily_100['maxDD']:.1%}")
        print(f"  Daily topk    95%: ann={m_daily_95['ann_ret']:+.1%} sharpe={m_daily_95['sharpe']:.2f} maxDD={m_daily_95['maxDD']:.1%}")
    with open(OUT / "diagnosis.json", "w") as f:
        json.dump(results, f, indent=2, default=str)
    print(f"\nSaved to {OUT / 'diagnosis.json'}")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,203 @@
 """Diagnose the script-vs-workflow gap properly.
 Three strategies compared:
  A. Script logic: weekly rebalance, equal-weight, hold through week
  B. Weekly rebalance (qlib engine behavior): same as script but with risk_degree
  C. Daily re-rank: re-select top-k every day (wrong model)
 Root cause was (C) — we were modeling daily re-ranking which neither
 the script nor the qlib engine actually does.
 """
 from __future__ import annotations
 import json, pathlib
 import numpy as np
 import pandas as pd
 LAKE_ROOT = "/home/data/lake"
 OUT = pathlib.Path("/app/experiments/book/data/diag_script_vs_wf")
 WINDOWS = [
    {"label": "2026", "start": "2026-01-04", "end": "2026-08-19",
     "pred": f"{LAKE_ROOT}/mlruns/62/3771f96eb1b74365aeae966af7aec5a3/artifacts/pred.pkl"},
    {"label": "2025", "start": "2025-01-02", "end": "2025-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/c57c6a8370cc48619d7cdd2bd109b76a/artifacts/pred.pkl"},
    {"label": "2024", "start": "2024-01-02", "end": "2024-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/97cf5f282e6f4e699443e38d9bfb40fd/artifacts/pred.pkl"},
    {"label": "2023", "start": "2023-01-03", "end": "2023-12-29",
     "pred": f"{LAKE_ROOT}/mlruns/62/11b9b65ea4e14b3f8ce50d244da0412e/artifacts/pred.pkl"},
    {"label": "2021", "start": "2021-01-04", "end": "2021-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/af3034e5910348a382f2ad1e1741f17c/artifacts/pred.pkl"},
 ]
 SYMS = [
    "SPY","QQQ","DIA","IWM","MDY","VTI","VOO","VEA","VWO","VT","EFA","EEM",
    "TLT","IEF","SHY","AGG","BND","LQD","HYG","JNK","EMB","GLD","SLV",
    "USO","UNG","DBA","DBC","XLK","XLF","XLE","XLV","XLI","XLY","XLP",
    "XLU","XLB","XLRE","ARKK","SMH","SOXX","IBB","XBI","ITA","XAR",
    "ICLN","TAN","FDN","IGV","ESPO","REM",
 ]
 def load_pred(path):
    df = pd.read_pickle(path)
    s = df["score"] if isinstance(df, pd.DataFrame) and "score" in df.columns else df.iloc[:, 0] if isinstance(df, pd.DataFrame) else df
    idx = s.index
    new_dt = pd.to_datetime(idx.get_level_values(0)).normalize()
    s.index = pd.MultiIndex.from_arrays([new_dt, idx.get_level_values(1)], names=idx.names)
    return s
 def load_closes(start, end):
    from tac_qlib.data.config import LakeConfig, resolve_lake_root
    cfg = LakeConfig(resolve_lake_root(LAKE_ROOT), "US")
    closes = {}
    for sym in SYMS:
        p = cfg.bar_path("1d", sym)
        if not p.exists(): continue
        try:
            df = pd.read_parquet(p)
        except: continue
        if not len(df): continue
        tcol = df["t"] if "t" in df.columns else df["date"]
        ts = pd.to_datetime(tcol)
        df = df.assign(_t=ts).set_index("_t").sort_index()
        warmup = pd.Timestamp(start) - pd.Timedelta(days=60)
        df = df.loc[warmup:end]
        if len(df) >= 22:
            closes[sym] = df["c"]
    return pd.DataFrame(closes)
 def strategy_weekly(pred, closes, start, end, topk=10, risk_degree=1.0, cost_bps=0):
    """Weekly rebalance: re-rank on first day of each ISO week, hold rest of week."""
    ret_df = closes.pct_change(fill_method=None)
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    trade_dates = sorted(dt_idx[(dt_idx >= start) & (dt_idx <= end)].unique())
    equity = 1_000_000.0
    holdings = []
    prev_week = None
    daily_eq = []
    for d in trade_dates:
        try:
            day_scores = pred.loc[d]
        except KeyError:
            daily_eq.append(equity)
            continue
        if isinstance(day_scores, pd.DataFrame):
            day_scores = day_scores.iloc[:, 0]
        day_scores = day_scores.dropna().sort_values(ascending=False)
        cur_week = (d.isocalendar()[0], d.isocalendar()[1])
        if cur_week != prev_week:
            # Rebalance: compute cost of turnover
            new_holdings = list(day_scores.index[:topk])
            if holdings and cost_bps > 0:
                sold = set(holdings) - set(new_holdings)
                bought = set(new_holdings) - set(holdings)
                turnover = (len(sold) + len(bought)) / (2 * max(len(holdings), 1))
                equity *= (1 - turnover * cost_bps / 10000)
            holdings = new_holdings
        ret_row = ret_df.loc[d] if d in ret_df.index else None
        if ret_row is not None and holdings:
            wts = np.array([risk_degree / len(holdings)] * len(holdings))
            rets = ret_row.reindex(holdings).fillna(0).values
            equity *= (1 + (wts * rets).sum())
        daily_eq.append(equity)
        prev_week = cur_week
    return pd.Series(daily_eq, index=trade_dates)
 def strategy_daily(pred, closes, start, end, topk=10, risk_degree=1.0, cost_bps=0):
    """Daily re-rank: re-select top-k every day (wrong model — what we incorrectly tested)."""
    ret_df = closes.pct_change(fill_method=None)
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    trade_dates = sorted(dt_idx[(dt_idx >= start) & (dt_idx <= end)].unique())
    equity = 1_000_000.0
    holdings = []
    daily_eq = []
    for d in trade_dates:
        try:
            day_scores = pred.loc[d]
        except KeyError:
            daily_eq.append(equity)
            continue
        if isinstance(day_scores, pd.DataFrame):
            day_scores = day_scores.iloc[:, 0]
        day_scores = day_scores.dropna().sort_values(ascending=False)
        new_holdings = list(day_scores.index[:topk])
        if holdings and cost_bps > 0:
            sold = set(holdings) - set(new_holdings)
            bought = set(new_holdings) - set(holdings)
            turnover = (len(sold) + len(bought)) / (2 * max(len(holdings), 1))
            equity *= (1 - turnover * cost_bps / 10000)
        holdings = new_holdings
        ret_row = ret_df.loc[d] if d in ret_df.index else None
        if ret_row is not None and holdings:
            wts = np.array([risk_degree / len(holdings)] * len(holdings))
            rets = ret_row.reindex(holdings).fillna(0).values
            equity *= (1 + (wts * rets).sum())
        daily_eq.append(equity)
    return pd.Series(daily_eq, index=trade_dates)
 def metrics(eq):
    if len(eq) < 2:
        return {"ann_ret": 0, "sharpe": 0, "maxDD": 0}
    rets = eq.pct_change().dropna()
    ann_ret = float((eq.iloc[-1] / eq.iloc[0]) ** (252 / max(len(eq), 1)) - 1)
    vol = float(rets.std() * (252 ** 0.5)) if len(rets) > 1 else 0
    sharpe = ann_ret / vol if vol > 0 else 0
    peak = eq.cummax()
    dd = (eq - peak) / peak
    return {"ann_ret": round(ann_ret, 4), "sharpe": round(sharpe, 4), "maxDD": round(float(dd.min()), 4)}
 def main():
    OUT.mkdir(parents=True, exist_ok=True)
    results = []
    for w in WINDOWS:
        print(f"\n=== {w['label']} ({w['start']} to {w['end']}) ===")
        pred = load_pred(w["pred"])
        closes = load_closes(w["start"], w["end"])
        print(f"  pred: {pred.index.get_level_values(0).min().date()} to {pred.index.get_level_values(0).max().date()}, "
              f"{pred.index.get_level_values(1).nunique()} syms")
        print(f"  close: {closes.index.min().date()} to {closes.index.max().date()}, {closes.shape[1]} syms")
        row = {"year": w["label"]}
        # A. Script: weekly, rd=1.0, zero cost
        eq = strategy_weekly(pred, closes, w["start"], w["end"], topk=10, risk_degree=1.0, cost_bps=0)
        m = metrics(eq); row["weekly_100_zc"] = m
        print(f"  Script weekly 100% zc: ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f}")
        # B. Script: weekly, rd=0.95, zero cost
        eq = strategy_weekly(pred, closes, w["start"], w["end"], topk=10, risk_degree=0.95, cost_bps=0)
        m = metrics(eq); row["weekly_95_zc"] = m
        print(f"  Script weekly  95% zc: ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f}")
        # C. Weekly, rd=0.95, with 5/15bp cost
        eq = strategy_weekly(pred, closes, w["start"], w["end"], topk=10, risk_degree=0.95, cost_bps=10)
        m = metrics(eq); row["weekly_95_10bp"] = m
        print(f"  Weekly 95% 10bp cost:   ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f}")
        # D. Daily re-rank, rd=1.0, zero cost (WRONG MODEL — for reference only)
        eq = strategy_daily(pred, closes, w["start"], w["end"], topk=10, risk_degree=1.0, cost_bps=0)
        m = metrics(eq); row["daily_100_zc"] = m
        print(f"  Daily 100% zc (WRONG):  ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f}")
        # E. Daily re-rank, rd=1.0, 10bp cost
        eq = strategy_daily(pred, closes, w["start"], w["end"], topk=10, risk_degree=1.0, cost_bps=10)
        m = metrics(eq); row["daily_100_10bp"] = m
        print(f"  Daily 100% 10bp (WRONG):ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f}")
        results.append(row)
    with open(OUT / "diagnosis_v2.json", "w") as f:
        json.dump(results, f, indent=2, default=str)
    print(f"\nSaved to {OUT / 'diagnosis_v2.json'}")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,410 @@
 """Diagnose script-vs-workflow gap v3: replicate workflow execution mechanics exactly.
 Replicates the WeeklyRebalanceDropoutStrategy execution:
  1. Weekly rebalance (first trading day of ISO week only)
  2. TopkDropout selection: sell bottom n_drop, buy top fill
  3. Cash-after-sells sizing: sell first, then cash * risk_degree / len(buy)
  4. Whole-share rounding (floor)
  5. Asymmetric costs: open_cost=5bp, close_cost=15bp, min_cost=$5 per order
  6. Optional SQ gate (hit-rate threshold)
 Compares against the idealized script (fractional shares, symmetric cost).
 """
 from __future__ import annotations
 import json, pathlib
 import numpy as np
 import pandas as pd
 LAKE_ROOT = "/home/data/lake"
 OUT = pathlib.Path("/app/experiments/book/data/diag_script_vs_wf")
 WINDOWS = [
    {"label": "2026", "start": "2026-01-04", "end": "2026-08-19",
     "pred": f"{LAKE_ROOT}/mlruns/62/3771f96eb1b74365aeae966af7aec5a3/artifacts/pred.pkl"},
    {"label": "2025", "start": "2025-01-02", "end": "2025-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/c57c6a8370cc48619d7cdd2bd109b76a/artifacts/pred.pkl"},
    {"label": "2024", "start": "2024-01-02", "end": "2024-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/97cf5f282e6f4e699443e38d9bfb40fd/artifacts/pred.pkl"},
    {"label": "2023", "start": "2023-01-03", "end": "2023-12-29",
     "pred": f"{LAKE_ROOT}/mlruns/62/11b9b65ea4e14b3f8ce50d244da0412e/artifacts/pred.pkl"},
    {"label": "2021", "start": "2021-01-04", "end": "2021-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/af3034e5910348a382f2ad1e1741f17c/artifacts/pred.pkl"},
 ]
 SYMS = [
    "SPY","QQQ","DIA","IWM","MDY","VTI","VOO","VEA","VWO","VT","EFA","EEM",
    "TLT","IEF","SHY","AGG","BND","LQD","HYG","JNK","EMB","GLD","SLV",
    "USO","UNG","DBA","DBC","XLK","XLF","XLE","XLV","XLI","XLY","XLP",
    "XLU","XLB","XLRE","ARKK","SMH","SOXX","IBB","XBI","ITA","XAR",
    "ICLN","TAN","FDN","IGV","ESPO","REM",
 ]
 OPEN_COST = 0.0005   # 5bp
 CLOSE_COST = 0.0015  # 15bp
 MIN_COST = 5.0       # $5 minimum per order
 def load_pred(path):
    df = pd.read_pickle(path)
    s = df["score"] if isinstance(df, pd.DataFrame) and "score" in df.columns else df.iloc[:, 0] if isinstance(df, pd.DataFrame) else df
    idx = s.index
    new_dt = pd.to_datetime(idx.get_level_values(0)).normalize()
    s.index = pd.MultiIndex.from_arrays([new_dt, idx.get_level_values(1)], names=idx.names)
    return s
 def load_closes(start, end):
    from tac_qlib.data.config import LakeConfig, resolve_lake_root
    cfg = LakeConfig(resolve_lake_root(LAKE_ROOT), "US")
    closes = {}
    for sym in SYMS:
        p = cfg.bar_path("1d", sym)
        if not p.exists(): continue
        try:
            df = pd.read_parquet(p)
        except: continue
        if not len(df): continue
        tcol = df["t"] if "t" in df.columns else df["date"]
        ts = pd.to_datetime(tcol)
        df = df.assign(_t=ts).set_index("_t").sort_index()
        df.index = pd.to_datetime(df.index).normalize()
        warmup = pd.Timestamp(start) - pd.Timedelta(days=60)
        df = df.loc[warmup:end]
        if len(df) >= 22:
            closes[sym] = df["c"]
    return pd.DataFrame(closes)
 def load_opens(start, end):
    from tac_qlib.data.config import LakeConfig, resolve_lake_root
    cfg = LakeConfig(resolve_lake_root(LAKE_ROOT), "US")
    opens = {}
    for sym in SYMS:
        p = cfg.bar_path("1d", sym)
        if not p.exists(): continue
        try:
            df = pd.read_parquet(p)
        except: continue
        if not len(df): continue
        tcol = df["t"] if "t" in df.columns else df["date"]
        ts = pd.to_datetime(tcol)
        df = df.assign(_t=ts).set_index("_t").sort_index()
        df.index = pd.to_datetime(df.index).normalize()
        warmup = pd.Timestamp(start) - pd.Timedelta(days=60)
        df = df.loc[warmup:end]
        if len(df) >= 22:
            opens[sym] = df["o"]
    return pd.DataFrame(opens)
 def load_vwap(start, end):
    from tac_qlib.data.config import LakeConfig, resolve_lake_root
    cfg = LakeConfig(resolve_lake_root(LAKE_ROOT), "US")
    vwaps = {}
    for sym in SYMS:
        p = cfg.bar_path("1d", sym)
        if not p.exists(): continue
        try:
            df = pd.read_parquet(p)
        except: continue
        if not len(df): continue
        tcol = df["t"] if "t" in df.columns else df["date"]
        ts = pd.to_datetime(tcol)
        df = df.assign(_t=ts).set_index("_t").sort_index()
        warmup = pd.Timestamp(start) - pd.Timedelta(days=60)
        df = df.loc[warmup:end]
        if len(df) >= 22:
            vwaps[sym] = df["vw"]
    return pd.DataFrame(vwaps)
 def compute_gate(pred, closes, start, end, gate_topk=10, gate_lookback=5, gate_threshold=0.5):
    """Compute the SQ gate: rolling average hit-rate of topk predictions."""
    ret_df = closes.pct_change()
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    pred_dates = sorted(dt_idx[(dt_idx >= start) & (dt_idx <= end)].unique())
    if len(pred_dates) < 2:
        return pd.Series(True, index=pd.DatetimeIndex(pred_dates))
    hit_rates = {}
    for i in range(1, len(pred_dates)):
        day = pred_dates[i]
        prev_day = pred_dates[i - 1]
        try:
            prev_scores = pred.loc[prev_day]
        except KeyError:
            continue
        if isinstance(prev_scores, pd.DataFrame):
            prev_scores = prev_scores.iloc[:, 0]
        prev_scores = prev_scores.dropna().sort_values(ascending=False)
        topk_syms = list(prev_scores.index[:gate_topk])
        if day not in ret_df.index:
            continue
        today_ret = ret_df.loc[day]
        topk_rets = today_ret.reindex(topk_syms).dropna()
        if len(topk_rets) == 0:
            continue
        hit_rates[day] = (topk_rets > 0).sum() / len(topk_rets)
    if not hit_rates:
        return pd.Series(True, index=pd.DatetimeIndex(pred_dates))
    hr_series = pd.Series(hit_rates).sort_index()
    rolling_hr = hr_series.rolling(gate_lookback, min_periods=1).mean()
    gate = rolling_hr >= gate_threshold
    gate.iloc[:gate_lookback] = True
    return gate
 def strategy_idealized(pred, closes, start, end, topk=10, risk_degree=1.0, cost_bps=0):
    """Idealized script: fractional shares, symmetric cost, no gate."""
    ret_df = closes.pct_change(fill_method=None)
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    trade_dates = sorted(dt_idx[(dt_idx >= start) & (dt_idx <= end)].unique())
    equity = 1_000_000.0
    holdings = []
    prev_week = None
    daily_eq = []
    for d in trade_dates:
        try:
            day_scores = pred.loc[d]
        except KeyError:
            daily_eq.append(equity)
            continue
        if isinstance(day_scores, pd.DataFrame):
            day_scores = day_scores.iloc[:, 0]
        day_scores = day_scores.dropna().sort_values(ascending=False)
        cur_week = (d.isocalendar()[0], d.isocalendar()[1])
        if cur_week != prev_week:
            new_holdings = list(day_scores.index[:topk])
            if holdings and cost_bps > 0:
                sold = set(holdings) - set(new_holdings)
                bought = set(new_holdings) - set(holdings)
                turnover = (len(sold) + len(bought)) / (2 * max(len(holdings), 1))
                equity *= (1 - turnover * cost_bps / 10000)
            holdings = new_holdings
        ret_row = ret_df.loc[d] if d in ret_df.index else None
        if ret_row is not None and holdings:
            wts = np.array([risk_degree / len(holdings)] * len(holdings))
            rets = ret_row.reindex(holdings).fillna(0).values
            equity *= (1 + (wts * rets).sum())
        daily_eq.append(equity)
        prev_week = cur_week
    return pd.Series(daily_eq, index=trade_dates)
 def strategy_workflow_exact(pred, closes, opens, start, end,
                            topk=10, n_drop=1, risk_degree=0.95,
                            use_gate=False, gate_series=None):
    """Exact replication of WeeklyRebalanceDropoutStrategy execution mechanics.
    - Sells first (all shares of dropped positions)
    - Sizes buys as: cash * risk_degree / len(buy)
    - Rounds to whole shares (floor)
    - Asymmetric costs: open_cost on buys, close_cost on sells, $5 min per order
    - Tracks position values for daily equity
    """
    ret_df = closes.pct_change(fill_method=None)
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    open_df = opens.copy()
    open_df.index = pd.to_datetime(open_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    trade_dates = sorted(dt_idx[(dt_idx >= start) & (dt_idx <= end)].unique())
    cash = 1_000_000.0
    positions = {}  # {sym: num_shares}
    prev_week = None
    daily_eq = []
    for d in trade_dates:
        # Skip non-trading days (pred may include weekends)
        if d not in closes.index:
            daily_eq.append(daily_eq[-1] if daily_eq else cash)
            continue
        try:
            day_scores = pred.loc[d]
        except KeyError:
            daily_eq.append(daily_eq[-1] if daily_eq else cash)
            continue
        if isinstance(day_scores, pd.DataFrame):
            day_scores = day_scores.iloc[:, 0]
        day_scores = day_scores.dropna().sort_values(ascending=False)
        cur_week = (d.isocalendar()[0], d.isocalendar()[1])
        if cur_week != prev_week:
            # === REBALANCE DAY ===
            # Check gate
            if use_gate and gate_series is not None:
                known = gate_series[gate_series.index <= d]
                if len(known) and not bool(known.iloc[-1]):
                    # gate closed: sell everything, go to cash
                    for sym in list(positions.keys()):
                        shares = positions[sym]
                        if shares <= 0:
                            continue
                        sell_price = closes.loc[d, sym] if d in closes.index and sym in closes.columns else None
                        if sell_price is None or pd.isna(sell_price):
                            continue
                        trade_val = shares * sell_price
                        trade_cost = max(trade_val * CLOSE_COST, MIN_COST) if trade_val > 0 else 0
                        cash += trade_val - trade_cost
                        positions[sym] = 0
                    positions = {s: v for s, v in positions.items() if v > 0}
                    daily_eq.append(cash)
                    prev_week = cur_week
                    continue
            # TopkDropout selection (matching WeeklyRebalanceDropoutStrategy exactly)
            current_syms = [s for s, v in positions.items() if v > 0]
            last = pred.loc[d].reindex(current_syms).sort_values(ascending=False).index if current_syms else pd.Index([])
            # buy candidates: top stocks NOT in current holdings, take n_drop + topk - len(last)
            buy_cands = day_scores[~day_scores.index.isin(last)].sort_values(ascending=False).index
            buy_list = list(buy_cands[:n_drop + topk - len(last)])
            # comb = union of current holdings + buy candidates (actual strategy line 132)
            comb = pred.loc[d].reindex(last.union(pd.Index(buy_list))).sort_values(ascending=False).index
            # sell: items from current holdings that are in the bottom n_drop of comb
            sell_list = list(last[last.isin(comb[-n_drop:])]) if n_drop > 0 and len(comb) >= n_drop else []
            # --- SELL FIRST ---
            for sym in sell_list:
                if sym not in positions or positions[sym] <= 0:
                    continue
                shares = positions[sym]
                sell_price = closes.loc[d, sym] if d in closes.index and sym in closes.columns else None
                if sell_price is None or pd.isna(sell_price):
                    continue
                trade_val = shares * sell_price
                trade_cost = max(trade_val * CLOSE_COST, MIN_COST) if trade_val > 0 else 0
                cash += trade_val - trade_cost
                positions[sym] = 0
            # --- BUY ---
            n_buy = len(buy_list)
            if n_buy > 0:
                buy_budget = cash * risk_degree / n_buy
                for sym in buy_list:
                    buy_price = closes.loc[d, sym] if d in closes.index and sym in closes.columns else None
                    if buy_price is None or pd.isna(buy_price) or buy_price <= 0:
                        continue
                    shares_to_buy = int(buy_budget / buy_price)  # floor to whole shares
                    if shares_to_buy <= 0:
                        continue
                    trade_val = shares_to_buy * buy_price
                    trade_cost = max(trade_val * OPEN_COST, MIN_COST) if trade_val > 0 else 0
                    total_cost = trade_val + trade_cost
                    if total_cost > cash:
                        shares_to_buy = int((cash - MIN_COST) / buy_price)
                        if shares_to_buy <= 0:
                            continue
                        trade_val = shares_to_buy * buy_price
                        trade_cost = max(trade_val * OPEN_COST, MIN_COST)
                        total_cost = trade_val + trade_cost
                    cash -= total_cost
                    positions[sym] = positions.get(sym, 0) + shares_to_buy
            positions = {s: v for s, v in positions.items() if v > 0}
        # === DAILY EQUITY ===
        eq = cash
        if d in closes.index:
            for sym, shares in positions.items():
                if sym in closes.columns:
                    px = closes.loc[d, sym]
                    if not pd.isna(px):
                        eq += shares * px
        daily_eq.append(eq)
        prev_week = cur_week
    return pd.Series(daily_eq, index=trade_dates)
 def metrics(eq):
    if len(eq) < 2:
        return {"ann_ret": 0, "sharpe": 0, "maxDD": 0}
    rets = eq.pct_change().dropna()
    ann_ret = float((eq.iloc[-1] / eq.iloc[0]) ** (252 / max(len(eq), 1)) - 1)
    vol = float(rets.std() * (252 ** 0.5)) if len(rets) > 1 else 0
    sharpe = ann_ret / vol if vol > 0 else 0
    peak = eq.cummax()
    dd = (eq - peak) / peak
    return {"ann_ret": round(ann_ret, 4), "sharpe": round(sharpe, 4), "maxDD": round(float(dd.min()), 4)}
 def main():
    OUT.mkdir(parents=True, exist_ok=True)
    results = []
    for w in WINDOWS:
        print(f"\n=== {w['label']} ({w['start']} to {w['end']}) ===")
        pred = load_pred(w["pred"])
        closes = load_closes(w["start"], w["end"])
        opens = load_opens(w["start"], w["end"])
        print(f"  pred: {pred.index.get_level_values(0).min().date()} to {pred.index.get_level_values(0).max().date()}, "
              f"{pred.index.get_level_values(1).nunique()} syms")
        print(f"  close: {closes.index.min().date()} to {closes.index.max().date()}, {closes.shape[1]} syms")
        row = {"year": w["label"]}
        # A. Idealized: fractional shares, 10bp symmetric, no gate (diag v2 baseline)
        eq = strategy_idealized(pred, closes, w["start"], w["end"], topk=10, risk_degree=1.0, cost_bps=0)
        m = metrics(eq); row["ideal_100_zc"] = m
        print(f"  A. Ideal 100% zc:       ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f} maxDD={m['maxDD']:.1%}")
        # B. Idealized: 95% invested, 10bp symmetric
        eq = strategy_idealized(pred, closes, w["start"], w["end"], topk=10, risk_degree=0.95, cost_bps=0)
        m = metrics(eq); row["ideal_95_zc"] = m
        print(f"  B. Ideal 95% zc:        ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f} maxDD={m['maxDD']:.1%}")
        # C. Idealized: 95%, 10bp cost
        eq = strategy_idealized(pred, closes, w["start"], w["end"], topk=10, risk_degree=0.95, cost_bps=10)
        m = metrics(eq); row["ideal_95_10bp"] = m
        print(f"  C. Ideal 95% 10bp:      ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f} maxDD={m['maxDD']:.1%}")
        # D. Workflow-exact: whole shares, 5/15bp, $5 min, no gate
        eq = strategy_workflow_exact(pred, closes, opens, w["start"], w["end"],
                                     topk=10, n_drop=1, risk_degree=0.95,
                                     use_gate=False)
        m = metrics(eq); row["wf_exact_95_nogate"] = m
        print(f"  D. WF exact 95% nogate: ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f} maxDD={m['maxDD']:.1%}")
        # E. Workflow-exact: whole shares, 5/15bp, $5 min, WITH SQ gate
        gate = compute_gate(pred, closes, w["start"], w["end"],
                           gate_topk=10, gate_lookback=5, gate_threshold=0.5)
        gate_open_pct = gate.sum() / len(gate) if len(gate) > 0 else 1.0
        eq = strategy_workflow_exact(pred, closes, opens, w["start"], w["end"],
                                     topk=10, n_drop=1, risk_degree=0.95,
                                     use_gate=True, gate_series=gate)
        m = metrics(eq); row["wf_exact_95_gate"] = m
        print(f"  E. WF exact 95% gate:   ann={m['ann_ret']:+.1%} sharpe={m['sharpe']:.2f} maxDD={m['maxDD']:.1%}  gate_open={gate_open_pct:.0%}")
        # Gap analysis
        ideal = row["ideal_95_zc"]["ann_ret"]
        wf_nogate = row["wf_exact_95_nogate"]["ann_ret"]
        wf_gate = row["wf_exact_95_gate"]["ann_ret"]
        print(f"\n  Gap analysis:")
        print(f"    Ideal (fractional, zc) → WF exact (whole shares, 5/15bp, nogate): {ideal:+.1%} → {wf_nogate:+.1%} (gap: {wf_nogate - ideal:+.1%})")
        print(f"    Ideal (fractional, zc) → WF exact (whole shares, 5/15bp, gate):  {ideal:+.1%} → {wf_gate:+.1%} (gap: {wf_gate - ideal:+.1%})")
        results.append(row)
    with open(OUT / "diagnosis_v3.json", "w") as f:
        json.dump(results, f, indent=2, default=str)
    print(f"\nSaved to {OUT / 'diagnosis_v3.json'}")
    # Summary table
    print("\n" + "=" * 80)
    print("SUMMARY: Ideal vs Workflow-Exact")
    print("=" * 80)
    print(f"{'Year':<6} {'Ideal%zc':>10} {'WF nogate':>10} {'WF gate':>10} {'Gap(nogate)':>12} {'Gap(gate)':>12}")
    for r in results:
        y = r["year"]
        i = r["ideal_95_zc"]["ann_ret"]
        wn = r["wf_exact_95_nogate"]["ann_ret"]
        wg = r["wf_exact_95_gate"]["ann_ret"]
        print(f"{y:<6} {i:>+10.1%} {wn:>+10.1%} {wg:>+10.1%} {wn-i:>+12.1%} {wg-i:>+12.1%}")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,343 @@
 """Signal-quality gate walk-forward backtest — RE-TRAINED MODEL variant.
 Identical logic to the original scripted test (signal_quality_gate_bt.py),
 but uses pred.pkls from exp 62 (retrained LGBModel per year, same model config
 as the workflow test) instead of the reference exp 52/56 pred.pkls.
 This isolates whether the gate itself works when the model is the same,
 regardless of the backtest engine.
 Usage:
    cd /app && .venv/bin/python book/scripts/signal_quality_gate_retrained.py
 """
 from __future__ import annotations
 import json
 import pathlib
 import numpy as np
 import pandas as pd
 LAKE_ROOT = "/home/data/lake"
 MARKET = "US"
 OUT_DIR = pathlib.Path("/app/experiments/book/data/signal_quality_gate")
 # Retrained pred.pkls from exp 62 (on-the-fly gate test)
 WINDOWS_RETRAINED = [
    {"label": "2026", "start": "2026-01-04", "end": "2026-08-19",
     "pred": f"{LAKE_ROOT}/mlruns/62/3771f96eb1b74365aeae966af7aec5a3/artifacts/pred.pkl"},
    {"label": "2025", "start": "2025-01-02", "end": "2025-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/c57c6a8370cc48619d7cdd2bd109b76a/artifacts/pred.pkl"},
    {"label": "2024", "start": "2024-01-02", "end": "2024-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/97cf5f282e6f4e699443e38d9bfb40fd/artifacts/pred.pkl"},
    {"label": "2023", "start": "2023-01-03", "end": "2023-12-29",
     "pred": f"{LAKE_ROOT}/mlruns/62/11b9b65ea4e14b3f8ce50d244da0412e/artifacts/pred.pkl"},
    {"label": "2021", "start": "2021-01-04", "end": "2021-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/62/af3034e5910348a382f2ad1e1741f17c/artifacts/pred.pkl"},
 ]
 # Original reference pred.pkls for head-to-head comparison
 WINDOWS_REFERENCE = [
    {"label": "2026", "start": "2026-01-04", "end": "2026-08-19",
     "pred": f"{LAKE_ROOT}/mlruns/52/9f98ea5c550a409f87b56a6cd8fee343/artifacts/pred.pkl"},
    {"label": "2025", "start": "2025-01-02", "end": "2025-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/52/fe96741654df4780957a3a949999ae6a/artifacts/pred.pkl"},
    {"label": "2024", "start": "2024-01-02", "end": "2024-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/52/71ed5bfa9984490f8bba8b222f7acc39/artifacts/pred.pkl"},
    {"label": "2023", "start": "2023-01-03", "end": "2023-12-29",
     "pred": f"{LAKE_ROOT}/mlruns/56/8ca46e554311444c9a42637a788226e8/artifacts/pred.pkl"},
    {"label": "2021", "start": "2021-01-04", "end": "2021-12-31",
     "pred": f"{LAKE_ROOT}/mlruns/56/4e0700ddab2a4e108b46efece7346ee3/artifacts/pred.pkl"},
 ]
 SIGNAL_GATE_CONFIGS = [
    (5, 0.50, "hitrate_5d_0.50"),
    (5, 0.60, "hitrate_5d_0.60"),
    (5, 0.70, "hitrate_5d_0.70"),
    (10, 0.50, "hitrate_10d_0.50"),
    (10, 0.60, "hitrate_10d_0.60"),
    (10, 0.70, "hitrate_10d_0.70"),
    (20, 0.40, "hitrate_20d_0.40"),
    (20, 0.50, "hitrate_20d_0.50"),
    (20, 0.60, "hitrate_20d_0.60"),
 ]
 def load_pred(path: str) -> pd.Series:
    df = pd.read_pickle(path)
    if isinstance(df, pd.DataFrame):
        if "score" in df.columns:
            s = df["score"]
        else:
            s = df.iloc[:, 0]
    else:
        s = df
    idx = s.index
    new_dt = pd.to_datetime(idx.get_level_values(0)).normalize()
    s.index = pd.MultiIndex.from_arrays(
        [new_dt, idx.get_level_values(1)], names=idx.names
    )
    return s
 def load_bars_for_window(start: str, end: str) -> pd.DataFrame:
    from tac_qlib.data.config import LakeConfig, resolve_lake_root
    cfg = LakeConfig(resolve_lake_root(LAKE_ROOT), MARKET)
    sp = cfg.lake_root / "symbols.parquet"
    if sp.exists():
        syms = pd.read_parquet(sp)
        col = "symbol" if "symbol" in syms.columns else syms.columns[0]
        symbols = sorted(syms[col].astype(str).str.upper().tolist())
    else:
        return pd.DataFrame()
    closes = {}
    for sym in symbols:
        p = cfg.bar_path("1d", sym)
        if not p.exists():
            continue
        try:
            df = pd.read_parquet(p)
        except Exception:
            continue
        if not len(df):
            continue
        tcol = df["t"] if "t" in df.columns else df["date"]
        ts = pd.to_datetime(tcol)
        df = df.assign(_t=ts).set_index("_t").sort_index()
        warmup_start = pd.Timestamp(start) - pd.Timedelta(days=60)
        df = df.loc[warmup_start:end]
        if len(df) >= 22:
            closes[sym] = df["c"]
    return pd.DataFrame(closes)
 def compute_hit_rate_series(
    pred: pd.Series, ret_df: pd.DataFrame, topk: int = 10, lookback: int = 10,
 ) -> pd.Series:
    dt_idx = pred.index.get_level_values(0)
    trade_dates = sorted(dt_idx.unique())
    hit_rates = {}
    for i in range(1, len(trade_dates)):
        prev_date = trade_dates[i - 1]
        curr_date = trade_dates[i]
        try:
            prev_scores = pred.loc[prev_date]
        except KeyError:
            continue
        if isinstance(prev_scores, pd.DataFrame):
            prev_scores = prev_scores.iloc[:, 0]
        prev_scores = prev_scores.dropna().sort_values(ascending=False)
        topk_syms = list(prev_scores.index[:topk])
        if curr_date not in ret_df.index:
            continue
        today_ret = ret_df.loc[curr_date]
        topk_rets = today_ret.reindex(topk_syms).dropna()
        if len(topk_rets) > 0:
            hit_rate = (topk_rets > 0).mean()
            hit_rates[curr_date] = hit_rate
    hit_series = pd.Series(hit_rates)
    if len(hit_series) == 0:
        return hit_series
    rolling_hr = hit_series.rolling(lookback, min_periods=max(1, lookback // 2)).mean()
    return rolling_hr
 def run_backtest(pred, hit_rate, close_df, start, end, topk=10, threshold=0.5):
    if not isinstance(pred.index, pd.MultiIndex):
        return {"error": "pred must have MultiIndex"}
    ret_df = close_df.pct_change()
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    dt_idx = pred.index.get_level_values(0)
    window_mask = (dt_idx >= pd.Timestamp(start)) & (dt_idx <= pd.Timestamp(end))
    window_pred = pred.loc[window_mask]
    if len(window_pred) == 0:
        return {"error": "no pred data in window"}
    trade_dates = sorted(dt_idx[window_mask].unique())
    gate_open = {}
    for d in trade_dates:
        known = hit_rate[hit_rate.index <= d]
        if len(known) > 0 and not pd.isna(known.iloc[-1]):
            gate_open[d] = bool(known.iloc[-1] >= threshold)
        else:
            gate_open[d] = True
    n_total = len(trade_dates)
    n_open = sum(1 for v in gate_open.values() if v)
    n_closed = n_total - n_open
    holdings_base = []
    holdings_gated = []
    equity_gated = 1_000_000.0
    equity_base = 1_000_000.0
    prev_week = None
    prev_scores = None
    daily_gated = []
    daily_base = []
    ret_by_date = {rd: ret_df.loc[rd] for rd in ret_df.index}
    for d in trade_dates:
        try:
            day_scores = window_pred.loc[d]
        except KeyError:
            daily_gated.append(equity_gated)
            daily_base.append(equity_base)
            prev_scores = None
            continue
        if isinstance(day_scores, pd.DataFrame):
            day_scores = day_scores.iloc[:, 0]
        day_scores = day_scores.dropna().sort_values(ascending=False)
        if len(day_scores) == 0:
            daily_gated.append(equity_gated)
            daily_base.append(equity_base)
            prev_scores = None
            continue
        ret_row = ret_by_date.get(d)
        if ret_row is None:
            daily_gated.append(equity_gated)
            daily_base.append(equity_base)
            prev_scores = day_scores
            continue
        cur_week = (d.isocalendar()[0], d.isocalendar()[1]) if hasattr(d, 'isocalendar') else None
        gate_val = gate_open.get(d, True)
        if cur_week != prev_week or not holdings_base:
            if prev_scores is not None:
                holdings_base = list(prev_scores.index[:topk])
        if holdings_base:
            base_rets = ret_row.reindex(holdings_base).dropna()
            if len(base_rets) > 0:
                equity_base *= (1 + base_rets.mean())
        if gate_val:
            if cur_week != prev_week or not holdings_gated:
                if prev_scores is not None:
                    holdings_gated = list(prev_scores.index[:topk])
            if holdings_gated:
                hold_rets = ret_row.reindex(holdings_gated).dropna()
                if len(hold_rets) > 0:
                    equity_gated *= (1 + hold_rets.mean())
        else:
            holdings_gated = []
        prev_week = cur_week
        prev_scores = day_scores
        daily_gated.append(equity_gated)
        daily_base.append(equity_base)
    g_series = pd.Series(daily_gated, index=trade_dates)
    b_series = pd.Series(daily_base, index=trade_dates)
    def _metrics(eq):
        if len(eq) < 2:
            return {"ann_return": 0, "sharpe": 0, "maxDD": 0}
        rets = eq.pct_change().dropna()
        ann_ret = float((eq.iloc[-1] / eq.iloc[0]) ** (252 / max(len(eq), 1)) - 1)
        vol = float(rets.std() * (252 ** 0.5)) if len(rets) > 1 else 0
        sharpe = ann_ret / vol if vol > 0 else 0
        peak = eq.cummax()
        dd = (eq - peak) / peak
        maxDD = float(dd.min())
        return {"ann_return": round(ann_ret, 6), "sharpe": round(sharpe, 4), "maxDD": round(maxDD, 6)}
    base_m = _metrics(b_series)
    gated_m = _metrics(g_series)
    return {
        "trade_dates": n_total,
        "gate_open_days": n_open,
        "gate_closed_days": n_closed,
        "trip_rate": round(n_closed / n_total, 4) if n_total else 0,
        "base": base_m,
        "gated": gated_m,
    }
 def run_set(windows, close_df, tag):
    results = []
    for window in windows:
        wl, ws, we = window["label"], window["start"], window["end"]
        pred_path = window["pred"]
        print(f"\n=== [{tag}] Window {wl} ({ws} to {we}) ===")
        pred = load_pred(pred_path)
        print(f"  pred shape: {pred.shape}, date range: {pred.index.get_level_values(0).min()} .. {pred.index.get_level_values(0).max()}")
        hit_rates = {}
        for lookback, _, name in SIGNAL_GATE_CONFIGS:
            if lookback not in hit_rates:
                hr = compute_hit_rate_series(pred, ret_df, topk=10, lookback=lookback)
                hit_rates[lookback] = hr
                print(f"  lookback={lookback}: {len(hr)} days with hit rates")
        for lookback, threshold, name in SIGNAL_GATE_CONFIGS:
            hr = hit_rates[lookback]
            bt = run_backtest(pred, hr, close_df, ws, we, topk=10, threshold=threshold)
            if "error" in bt:
                print(f"  {name}: {bt['error']}")
                continue
            row = {
                "source": tag,
                "window": wl,
                "gate": name,
                "start": ws,
                "end": we,
                "trade_dates": bt["trade_dates"],
                "gate_open": bt["gate_open_days"],
                "gate_closed": bt["gate_closed_days"],
                "trip_rate": bt["trip_rate"],
                "base_ann": bt["base"]["ann_return"],
                "base_sharpe": bt["base"]["sharpe"],
                "base_maxDD": bt["base"]["maxDD"],
                "gated_ann": bt["gated"]["ann_return"],
                "gated_sharpe": bt["gated"]["sharpe"],
                "gated_maxDD": bt["gated"]["maxDD"],
            }
            results.append(row)
            diff = bt["gated"]["ann_return"] - bt["base"]["ann_return"]
            print(f"  {name}: trip={bt['trip_rate']:.1%}, "
                  f"base={bt['base']['ann_return']:+.1%} (Sharpe {bt['base']['sharpe']:.2f}), "
                  f"gated={bt['gated']['ann_return']:+.1%} (Sharpe {bt['gated']['sharpe']:.2f}), "
                  f"diff={diff:+.1%}pp")
    return results
 def main():
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    full_start = "2015-01-03"
    full_end = "2026-08-19"
    print("Loading lake bars...")
    close_df = load_bars_for_window(full_start, full_end)
    print(f"  {close_df.shape[1]} symbols, {close_df.shape[0]} days")
    global ret_df
    ret_df = close_df.pct_change()
    ret_df.index = pd.to_datetime(ret_df.index).normalize()
    print("\n" + "=" * 70)
    print("RUN A: Retrained model pred.pkls (exp 62)")
    print("=" * 70)
    results_retrained = run_set(WINDOWS_RETRAINED, close_df, "retrained")
    print("\n" + "=" * 70)
    print("RUN B: Reference pred.pkls (exp 52/56)")
    print("=" * 70)
    results_reference = run_set(WINDOWS_REFERENCE, close_df, "reference")
    all_results = results_retrained + results_reference
    df = pd.DataFrame(all_results)
    # Save combined results
    out_path = OUT_DIR / "signal_quality_gate_retrained.csv"
    df.to_csv(out_path, index=False)
    with open(OUT_DIR / "signal_quality_gate_retrained.json", "w") as f:
        json.dump(df.to_dict(orient="records"), f, indent=2, default=str)
    print(f"\nSaved to {out_path}")
    # Head-to-head comparison table
    print("\n" + "=" * 70)
    print("HEAD-TO-HEAD: Retrained vs Reference (hitrate_5d_0.50)")
    print("=" * 70)
    print(f"{'Year':>6} | {'Ref Base':>10} {'Ref Gated':>10} {'Ref Diff':>10} | {'Ret Base':>10} {'Ret Gated':>10} {'Ret Diff':>10}")
    print("-" * 85)
    for year in ["2021", "2023", "2024", "2025", "2026"]:
        ref = df[(df["source"] == "reference") & (df["window"] == year) & (df["gate"] == "hitrate_5d_0.50")]
        ret = df[(df["source"] == "retrained") & (df["window"] == year) & (df["gate"] == "hitrate_5d_0.50")]
        if len(ref) > 0 and len(ret) > 0:
            rb = ref.iloc[0]["base_ann"]
            rg = ref.iloc[0]["gated_ann"]
            tb = ret.iloc[0]["base_ann"]
            tg = ret.iloc[0]["gated_ann"]
            print(f"{year:>6} | {rb:>+9.1%} {rg:>+9.1%} {rg-rb:>+9.1%} | {tb:>+9.1%} {tg:>+9.1%} {tg-tb:>+9.1%}")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,119 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-2021"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2021-12-31
                    fit_start_time: 2017-01-03
                    fit_end_time: 2020-12-31
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2017-01-03, fit_end_time: 2020-12-31 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2017-01-03, fit_end_time: 2020-12-31 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2017-01-03, 2020-12-31]
                valid: [2021-01-04, 2021-01-04]
                test: [2021-01-04, 2021-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2017-01-03"
                          gate_end: "2021-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2021-01-04
                      end_time: 2021-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-2023"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2023-12-29
                    fit_start_time: 2019-01-02
                    fit_end_time: 2022-12-30
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2019-01-02, fit_end_time: 2022-12-30 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2019-01-02, fit_end_time: 2022-12-30 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2019-01-02, 2022-12-30]
                valid: [2023-01-03, 2023-01-03]
                test: [2023-01-03, 2023-12-29]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2019-01-02"
                          gate_end: "2023-12-29"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2023-01-03
                      end_time: 2023-12-29
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-2024"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2024-12-31
                    fit_start_time: 2020-01-02
                    fit_end_time: 2023-12-29
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2020-01-02, fit_end_time: 2023-12-29 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2020-01-02, fit_end_time: 2023-12-29 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2020-01-02, 2023-12-29]
                valid: [2024-01-02, 2024-01-02]
                test: [2024-01-02, 2024-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2020-01-02"
                          gate_end: "2024-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2024-01-02
                      end_time: 2024-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-2025"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2025-12-31
                    fit_start_time: 2021-01-04
                    fit_end_time: 2024-12-31
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2021-01-04, fit_end_time: 2024-12-31 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2021-01-04, fit_end_time: 2024-12-31 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2021-01-04, 2024-12-31]
                valid: [2025-01-02, 2025-01-02]
                test: [2025-01-02, 2025-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2021-01-04"
                          gate_end: "2025-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2025-01-02
                      end_time: 2025-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-2026"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2026-08-10
                    fit_start_time: 2016-01-04
                    fit_end_time: 2025-09-01
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2025-09-01 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2025-09-01 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2016-01-04, 2025-09-01]
                valid: [2025-09-03, 2026-01-03]
                test: [2026-01-04, 2026-08-10]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2016-01-04"
                          gate_end: "2026-08-19"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2026-01-04
                      end_time: 2026-08-10
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,119 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-zc-2021"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2021-12-31
                    fit_start_time: 2017-01-03
                    fit_end_time: 2020-12-31
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2017-01-03, fit_end_time: 2020-12-31 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2017-01-03, fit_end_time: 2020-12-31 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2017-01-03, 2020-12-31]
                valid: [2021-01-04, 2021-01-04]
                test: [2021-01-04, 2021-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2017-01-03"
                          gate_end: "2021-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2021-01-04
                      end_time: 2021-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0
                          close_cost: 0.0
                          min_cost: 0.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-zc-2023"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2023-12-29
                    fit_start_time: 2019-01-02
                    fit_end_time: 2022-12-30
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2019-01-02, fit_end_time: 2022-12-30 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2019-01-02, fit_end_time: 2022-12-30 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2019-01-02, 2022-12-30]
                valid: [2023-01-03, 2023-01-03]
                test: [2023-01-03, 2023-12-29]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2019-01-02"
                          gate_end: "2023-12-29"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2023-01-03
                      end_time: 2023-12-29
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0
                          close_cost: 0.0
                          min_cost: 0.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-zc-2024"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2024-12-31
                    fit_start_time: 2020-01-02
                    fit_end_time: 2023-12-29
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2020-01-02, fit_end_time: 2023-12-29 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2020-01-02, fit_end_time: 2023-12-29 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2020-01-02, 2023-12-29]
                valid: [2024-01-02, 2024-01-02]
                test: [2024-01-02, 2024-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2020-01-02"
                          gate_end: "2024-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2024-01-02
                      end_time: 2024-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0
                          close_cost: 0.0
                          min_cost: 0.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-zc-2025"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2025-12-31
                    fit_start_time: 2021-01-04
                    fit_end_time: 2024-12-31
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2021-01-04, fit_end_time: 2024-12-31 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2021-01-04, fit_end_time: 2024-12-31 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2021-01-04, 2024-12-31]
                valid: [2025-01-02, 2025-01-02]
                test: [2025-01-02, 2025-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2021-01-04"
                          gate_end: "2025-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2025-01-02
                      end_time: 2025-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0
                          close_cost: 0.0
                          min_cost: 0.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-zc-2026"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2026-08-10
                    fit_start_time: 2016-01-04
                    fit_end_time: 2025-09-01
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2025-09-01 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2025-09-01 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2016-01-04, 2025-09-01]
                valid: [2025-09-03, 2026-01-03]
                test: [2026-01-04, 2026-08-10]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2016-01-04"
                          gate_end: "2026-08-19"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2026-01-04
                      end_time: 2026-08-10
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0
                          close_cost: 0.0
                          min_cost: 0.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-v3"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2021-12-31
                    fit_start_time: 2016-01-04
                    fit_end_time: 2020-12-31
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2020-12-31 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2020-12-31 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2016-01-04, 2020-12-31]
                valid: [2021-01-04, 2021-01-04]
                test: [2021-01-04, 2021-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2016-01-04"
                          gate_end: "2021-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2021-01-04
                      end_time: 2021-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-v3"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2023-12-29
                    fit_start_time: 2016-01-04
                    fit_end_time: 2022-12-30
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2022-12-30 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2022-12-30 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2016-01-04, 2022-12-30]
                valid: [2023-01-03, 2023-01-03]
                test: [2023-01-03, 2023-12-29]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2016-01-04"
                          gate_end: "2023-12-29"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2023-01-03
                      end_time: 2023-12-29
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-v3"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2024-12-31
                    fit_start_time: 2016-01-04
                    fit_end_time: 2023-12-29
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2023-12-29 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2023-12-29 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2016-01-04, 2023-12-29]
                valid: [2024-01-02, 2024-01-02]
                test: [2024-01-02, 2024-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2016-01-04"
                          gate_end: "2024-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2024-01-02
                      end_time: 2024-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-v3"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2025-12-31
                    fit_start_time: 2016-01-04
                    fit_end_time: 2024-12-31
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2024-12-31 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2024-12-31 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2016-01-04, 2024-12-31]
                valid: [2025-01-02, 2025-01-02]
                test: [2025-01-02, 2025-12-31]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2016-01-04"
                          gate_end: "2025-12-31"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2025-01-02
                      end_time: 2025-12-31
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d
@@ -0,0 +1,115 @@
 {%- set LAKE = TAC_LAKE_DIR %}
 {%- set UNIVERSE = "SPY,QQQ,DIA,IWM,MDY,VTI,VOO,VEA,VWO,VT,EFA,EEM,TLT,IEF,SHY,AGG,BND,LQD,HYG,JNK,EMB,GLD,SLV,USO,UNG,DBA,DBC,XLK,XLF,XLE,XLV,XLI,XLY,XLP,XLU,XLB,XLRE,ARKK,SMH,SOXX,IBB,XBI,ITA,XAR,ICLN,TAN,FDN,IGV,ESPO,REM" %}
 {%- set SP_FIELDS = "sp_ret,sp_jump_ratio,sp_jump_flag,sp_jump_tail,sp_max_move,sp_rv1,sp_rv5,sp_rv22,sp_vol_ratio_5_22,sp_vol_ratio_1_22,sp_trend_slope_5,sp_trend_slope_20,sp_trend_slope_60,sp_logp,sp_hurst_exponent,sp_sig_level1_lead,sp_sig_level1_lag,sp_sig_level2_lead_lag,sp_sig_level2_lag_lead" %}
 qlib_init:
    provider_uri: "{{ LAKE }}"
    region: us
    expression_cache: null
    dataset_cache: null
    calendar_provider:
        class: tac_qlib.data.providers.LakeCalendarProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    instrument_provider:
        class: tac_qlib.data.providers.LakeInstrumentProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US, markets: {} }
    feature_provider:
        class: tac_qlib.data.providers.LakeFeatureProvider
        kwargs: { lake_root: "{{ LAKE }}", market: US }
    exp_manager:
        class: MLflowExpManager
        module_path: qlib.workflow.expm
        kwargs:
            uri: "sqlite:///{{ LAKE }}/mlruns.db"
            default_exp_name: "tac-rd-sq-gate-wk-v3"
 task:
    model:
        class: RankICEnsembleLGBModel
        module_path: tac_qlib.contrib.model.rank_ensemble
        kwargs:
            loss: mse
            learning_rate: 0.02
            num_leaves: 31
            n_estimators: 3000
            num_boost_round: 3000
            early_stopping_rounds: 200
            min_data_in_leaf: 20
            lambda_l2: 0.5
            colsample_bytree: 0.8
            subsample: 0.8
            subsample_freq: 1
            reg_alpha: 0.1
            reg_lambda: 1.0
            seeds: "42,7,2026,99,123"
            parallel: 5
    dataset:
        class: DatasetH
        module_path: qlib.data.dataset
        kwargs:
            handler:
                class: TACHandler
                module_path: tac_qlib.contrib.data.handler
                kwargs:
                    instruments: "{{ UNIVERSE }}"
                    start_time: 2015-01-03
                    end_time: 2026-08-10
                    fit_start_time: 2016-01-04
                    fit_end_time: 2025-09-01
                    freq: day
                    lake_root: "{{ LAKE }}"
                    market: US
                    label: "Ref($close,-6)/Ref($close,-1)-1"
                    feature_fields: "$open,$high,$low,$close,$vwap,$volume,{{ SP_FIELDS }}"
                    infer_processors:
                        - class: DropAllNaN
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2025-09-01 }
                        - class: ProcessInf
                          kwargs: {}
                        - class: CSRankNorm
                          kwargs: {}
                        - class: ZScoreNorm
                          kwargs: { fit_start_time: 2016-01-04, fit_end_time: 2025-09-01 }
                        - class: Fillna
                          kwargs: {}
            segments:
                train: [2016-01-04, 2025-09-01]
                valid: [2025-09-03, 2026-01-03]
                test: [2026-01-04, 2026-08-10]
    record:
        - class: SignalRecord
          module_path: qlib.workflow.record_temp
          kwargs: {}
        - class: SigAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs: { ana_long_short: true, ann_scaler: 252 }
        - class: PortAnaRecord
          module_path: qlib.workflow.record_temp
          kwargs:
              config:
                  strategy:
                      class: WeeklyRebalanceSignalQualityGateStrategy
                      module_path: tac_qlib.contrib.strategy.weekly_sq_gate
                      kwargs:
                          signal: "<PRED>"
                          lake_root: "{{ LAKE }}"
                          gate_topk: 10
                          gate_lookback: 5
                          gate_threshold: 0.5
                          gate_start: "2016-01-04"
                          gate_end: "2026-08-19"
                          topk: 10
                          n_drop: 1
                          only_tradable: true
                          risk_degree: 0.95
                  backtest:
                      start_time: 2026-01-04
                      end_time: 2026-08-10
                      account: 1000000
                      benchmark: SPY
                      exchange_kwargs:
                          codes: "{{ UNIVERSE }}"
                          deal_price: $close
                          freq: day
                          open_cost: 0.0005
                          close_cost: 0.0015
                          min_cost: 5.0
              risk_analysis_freq: 1d