# tac-qlib

Run stock [qlib](https://github.com/microsoft/qlib) ML workflows (LightGBM → signal → backtest) directly on the
TradeAC parquet lake. No CSV/bin dump, no data conversion: the lake's calendar, instrument master, OHLCV bars and
pre-computed ta-lib features plug into qlib as first-class data providers, and a custom `DataHandlerLP`
(`TACHandler`) exposes them through the normal qlib dataset/processor pipeline.

```
$ qrun workflows/workflow_lgb_taclake.yaml --experiment_name tac-lake-lgb
```

trains a LightGBM, records predictions/labels, evaluates the signal (IC/RankIC), runs a daily
`TopkDropoutStrategy` backtest with cost model and risk analysis, and logs everything to mlflow (sqlite).

## Layout

```
tac-qlib/
├── tac_qlib/
│   ├── qlib_init.py                  # qlib_init() drop-in wired to the lake providers
│   ├── data/
│   │   ├── config.py                 # LakeConfig: paths + metadata readers, freq↔timeframe map
│   │   └── providers.py              # LakeCalendarProvider / LakeInstrumentProvider / LakeFeatureProvider
│   └── contrib/data/
│       └── handler.py                # TACHandler (DataHandlerLP) + DropAllNaN processor
├── workflows/
│   └── workflow_lgb_taclake.yaml     # qrun workflow: train -> signal -> backtest
├── examples/
│   └── run_backtest.py               # same loop as the workflow, plain Python (no yaml)
└── tests/
    └── test_lake_providers.py        # plain-assert smoke tests
```

## Requirements / install

- Python 3.12, `qlib` nightly (`0.1.dev2066` in the repo venv), pandas/pyarrow, lightgbm.
- The TradeAC lake (see below). `TAC_LAKE_DIR` is **mandatory** (no default) — set it to the
  lake root, or pass the `lake_root` kwargs.

## The lake (data layout)

```
$TAC_LAKE_DIR/
├── market=US/
│   └── timeframe=1d/
│       └── symbol=AAPL.parquet        # OHLCV bars: t, date, o, h, l, c, v, n, vw
├── features/
│   └── market=US/timeframe=1d/
│       └── symbol=AAPL.parquet        # ta-lib indicators, wide format: t, sma_5, rsi_14, ...
├── calendar.parquet                   # trading days per market
├── coverage.parquet                   # per (market,timeframe,symbol) loaded windows
└── symbols.parquet                    # asset master
```

Field routing (`tac_qlib/data/config.py`):

- `$open $high $low $close $volume $vwap` → bar parquet columns; `$amount` = `v * vw`, `$avg_amount` = `vw`.
- `$factor $change $trade_unit $suspend_flag` → all-NaN (not stored; the backtest Exchange only needs `$close`).
- anything else (e.g. `$rsi_14`, `$sma_20`) → a ta-lib column in the features parquet.

## Step 1 — Prepare data

The lake is populated and backfilled with the tac-engine MCP lake tools (see
`tac-engine/skills/tradeac-lake`). Typical sequence:

1. Seed the trading calendar from historical auctions (so the 1d completeness check has an expected day set):
   `backfill_lake_calendar(symbols="AAPL,MSFT,...")`.
2. Backfill bars: `get_lake_bars(symbols="AAPL,MSFT,...", timeframe="1d", start="2026-02-09")` (lazy: missing
   windows are fetched from Alpaca and persisted; `sip`/`iex` auto-fallback on 403).
3. Persist features: `get_lake_ta(symbol="AAPL", timeframe="1d", indicators="sma_5,sma_20,rsi_14,macd,bb,atr_14", persist=true)`.
   Only indicators that exist in *every* features file are auto-loaded by the handler; add columns per symbol by
   re-running `get_lake_ta`.
4. `get_lake_symbols` / `get_lake_coverage` to verify the universe and loaded windows.

`TACHandler` discovers the feature columns itself (`get_common_feature_fields` = the intersection of columns
across all features files), so no config change is needed as the lake grows.

## Step 2 — Preprocess

Preprocessing happens in `TACHandler` (a `DataHandlerLP`), composed from standard qlib processors:

- **infer** (`DEFAULT_INFER_PROCESSORS`), applied to the input features:
  1. `DropAllNaN` — drops columns that are all-NaN over the fit window (fixes the lake's fully-empty ta-lib
     columns, e.g. a `stoch_*` output that is NaN from the start). The drop set is fixed in `fit()` and applied
     identically to train/valid/test so feature columns never diverge.
  2. `ProcessInf`, `ZScoreNorm` (fit on the fit window), `Fillna`.
- **learn** (`DEFAULT_LEARN_PROCESSORS`), applied to the label: `DropnaLabel`, `CSZScoreNorm`.

Handler kwargs (used by both the workflow yaml and the Python API):

| kwarg | default | meaning |
|---|---|---|
| `instruments` | `all` | universe; list, `all`, or a named pool from `markets:` |
| `start_time` / `end_time` | – | queried window (must be within the lake calendar) |
| `fit_start_time` / `fit_end_time` | start/end | window the fit-able processors (ZScoreNorm, DropAllNaN) fit on |
| `freq` | `day` | maps to the lake timeframe (`day`→`1d`, `1min`→`1m`, …) |
| `feature_fields` | auto | raw OHLCV + common ta-lib columns; or an explicit list |
| `label` | `Ref($close,-2)/Ref($close,-1)-1` | qlib expression for the target |
| `lake_root` / `market` | `$TAC_LAKE_DIR` / `US` | lake location (required) / market partition |

Only daily (`1d`) is currently supported by the calendar provider; intraday freq raises `NotImplementedError`.

## Step 3 — Train

Either write the model task in yaml and run qrun (see *Glue with qrun*), or train in Python:

```python
from qlib.data.dataset import DatasetH
from tac_qlib.qlib_init import qlib_init
from tac_qlib.contrib.data.handler import TACHandler
from qlib.contrib.model.gbdt import LGBModel
from qlib.workflow import R

qlib_init(provider_uri=os.environ["TAC_LAKE_DIR"], market="US", freq="day")

handler = TACHandler(
    instruments="all",
    start_time="2026-03-01", end_time="2026-08-06",
    fit_start_time="2026-03-01", fit_end_time="2026-05-31",
    freq="day", lake_root=os.environ["TAC_LAKE_DIR"], market="US",
)
dataset = DatasetH(handler=handler, segments={
    "train": ("2026-03-01", "2026-05-31"),
    "valid": ("2026-06-01", "2026-06-30"),
    "test":  ("2026-07-01", "2026-08-06"),
})

model = LGBModel(n_estimators=200, learning_rate=0.05, num_leaves=15, ...)
with R.start(experiment_name="tac-lake-demo"):
    model.fit(dataset)                 # trains on the train segment
```

## Step 4 — Test / evaluate the signal

`model.predict(dataset)` returns the prediction on the **test** segment (a `(datetime, instrument)` Series).
Evaluate it with qlib's `SigAnaRecord` / `sig_analysis`:

```python
from qlib.workflow.record_temp import SigAnaRecord
from qlib.contrib.evaluate import signal_analysis

pred = model.predict(dataset)                     # "score" column
label = dataset.prepare("test", col_set="label", data_key=DataHandlerLP.DK_I)["LABEL0"]
# per-day + overall IC / ICIR / RankIC / RankICIR
report = signal_analysis(pred, label)
```

In the workflow this is automatic (`SigAnaRecord`): the run logs IC 0.0072 / ICIR 0.016 /
RankIC 0.0138 / RankICIR 0.034 for the default split — weak but the plumbing is verified.

## Step 5 — Backtesting

```python
from qlib.contrib.evaluate import backtest_daily, risk_analysis
from qlib.contrib.strategy.signal_strategy import TopkDropoutStrategy

strategy = TopkDropoutStrategy(signal=pred, topk=2, n_drop=1, only_tradable=True, risk_degree=0.95)
report_normal, positions_normal = backtest_daily(
    start_time="2026-07-01", end_time="2026-08-06",
    strategy=strategy, account=1_000_000, benchmark=None,   # lake has no index quotes
    exchange_kwargs={"codes": universe, "deal_price": "$close", "freq": "day",
                     "open_cost": 0.0005, "close_cost": 0.0015, "min_cost": 5.0},
)
risk = risk_analysis(report_normal["return"], freq="day")
```

- `TopkDropoutStrategy` is the default mapping *prediction → positions* (hold top-k, drop `n_drop` per day).
  For other sizing frameworks — equal/score-weighting, softmax, z-score, fractional Kelly, mean-variance —
  subclass `qlib.contrib.strategy.SignalStrategy` and implement `generate_trade_decision` (see
  `../.tmp/signalTrade.md` for the recipe catalogue).
- The Exchange needs `$close`; other fields the backtest probes (`$factor`, `$trade_unit`) are all-NaN and fine.
- Benchmark: pick any symbol the lake holds (e.g. `benchmark: AAPL`); null benchmark triggers benign
  "Mean of empty slice" warnings from the risk analysis.

## Step 6 — Predict

`SignalRecord` already saved `pred.pkl` (test segment) during the qrun run. For predictions on arbitrary data:

```python
pred = model.predict(dataset)                       # predict on the "test" segment
pred.to_frame("score").to_pickle("pred.pkl")        # (datetime, instrument) x ["score"]
```

To predict a live/rolling window instead of the configured test segment, point a handler's `segments["test"]`
at the window of interest, or call `model.predict(dataset, segment="test")` after overriding the segment.

## Glue everything with qrun

`workflows/workflow_lgb_taclake.yaml` wires the whole chain (init → train → signal record → signal analysis →
backtest + risk analysis) into one qrun invocation:

```bash
cd tac-qlib
qrun workflows/workflow_lgb_taclake.yaml --experiment_name tac-lake-lgb
# custom lake root:
TAC_LAKE_DIR=/path/to/lake qrun workflows/workflow_lgb_taclake.yaml --experiment_name tac-lake-lgb
```

YAML anatomy:

- `qlib_init` — points `provider_uri` at the lake and installs the lake providers by their full class paths
  (`tac_qlib.data.providers.Lake*Provider`), plus an `exp_manager` backed by `sqlite:///<lake>/mlruns.db`
  (avoids mlflow's filesystem-backend maintenance-mode opt-in). The unified R&D store lives under the lake
  root: `mlruns.db` + `mlruns/<exp>/<run>/`. Override the tracking URI with `MLRUNS_URI`.
- `task.model` — `LGBModel` hyperparameters.
- `task.dataset` — `DatasetH` over `TACHandler`; `segments.train/valid/test` split the window;
  `fit_start_time`/`fit_end_time` pin the processor fit window to train.
- `task.record` — ordered records:
  1. `SignalRecord` → writes `pred.pkl` (and `label.pkl`).
  2. `SigAnaRecord` → `sig_analysis/{ic,ric}.pkl` (IC/ICIR/RankIC/RankICIR).
  3. `PortAnaRecord` → daily `TopkDropoutStrategy` backtest + `risk_analysis_freq: 1d` →
     `portfolio_analysis/*.pkl` (report, positions, indicators, risk metrics, benchmark & cost-adjusted excess returns).

Run artifacts land under the mlflow run: `<lake>/mlruns/<exp>/<run>/artifacts/*.pkl` (metadata in `<lake>/mlruns.db`).

Template notes:

- The header uses jinja2 (`{%- set LAKE = TAC_LAKE_DIR %}`) — `TAC_LAKE_DIR` is **required** and names the
  lake root. Do **not** use `-%}` on the closing tag — it strips the newline and glues
  `qlib_init:` onto the comment line (YAML parse error).
- `qrun` is `qlib.cli.run:run` (fire): positional CONFIG_PATH + `--experiment_name` / `--uri_folder`. No
  `--config` flag.

## Manual (no-yaml) path

`examples/run_backtest.py` runs the identical loop in plain Python (good for parametrizing universe, features,
label, topk, costs):

```bash
.venv/bin/python tac-qlib/examples/run_backtest.py
.venv/bin/python tac-qlib/examples/run_backtest.py --features '$close,$rsi_14,$sma_5,$macd' \
    --universe AAPL,MSFT,TSLA,USO,SLV,TLT --topk 2 --n-drop 1 --output ./backtest_out
```

Writes `pred.pkl`, `report_normal.csv`, `positions_normal.csv`, `risk.csv` to the output dir.

## Reference

- `tac_qlib/data/providers.py` — the three lake providers; they match qlib's provider interface
  (`feature()` keyed by calendar position, `list_instruments()` with listing spans, `load_calendar()`), so the
  expression engine, `DatasetH` and the backtest `Exchange` work unchanged.
- `tac_qlib/contrib/data/handler.py` — `TACHandler` (DataHandlerLP over `QlibDataLoader`),
  `DropAllNaN`, `get_common_feature_fields`, `discover_feature_fields`.
- `tac_qlib/data/config.py` — `LakeConfig` path/reader helpers, `FREQ_TO_TIMEFRAME`, `BAR_FIELD_MAP`,
  `resolve_lake_root` (`$TAC_LAKE_DIR`, required — fails fast if unset).
- Tests (no pytest; plain asserts):

```bash
.venv/bin/python tac-qlib/tests/test_lake_providers.py
```