refactor: class-based alpha factory + month-partitioned data pipeline

Replace the old signal/strategy/backtest modules with a decoupled data → alpha → combo pipeline (parquet between phases, .pq extension). Alphas: - BaseAlpha + @register_alpha factory/plugin registry; one file per built-in (reversal, reversal_vol, momentum); external alphas via --alpha-module. Alphas are z-scored position weights, not predictors. Data: - baostock primary / akshare fallback, treated consistently. - New --universe all (~5000 A-shares via query_all_stock, filtered). - login-once batch downloader; empty-string OHLCV coerced to NaN. - Month-partitioned dataset {output_dir}/{universe}/month=YYYY-MM/*.pq with chunked durability flushes; --data-path is the dataset dir. CLI logs at INFO by default (--log-level) so progress is visible. Docs (README, CLAUDE.md) updated incl. pipeline diagram and roadmap TODOs for portfolio construction / backtest / paper trading. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-09 14:07:07 +08:00
parent 769cf25daa
commit 1caa63faeb
54 changed files with 1640 additions and 1120 deletions
@@ -1,8 +1,41 @@
 # Chinese Equity Quant Research Framework

-A modular Chinese A-share quant research framework built on
-[backtrader](https://www.backtrader.com/) for backtesting, with
-akshare (primary) and baostock (fallback) for daily bar data.
+A modular Chinese A-share quant research framework. Daily frequency only (Phase 1).
+
+It is a **decoupled, file-based pipeline**: each phase reads parquet and writes
+parquet, so phases run, cache, and inspect independently.
+
+```
+  baostock (primary)                                              one weight series
+  akshare  (fallback)                                             interpreted as a
+        │                                                            portfolio
+        ▼                                                              ▲
+ ┌──────────────┐      ┌───────────────┐      ┌───────────────┐       │
+ │     DATA     │      │     ALPHA     │      │     COMBO     │   ┌────┴─────┐
+ │   download   │─────▶│    compute    │─────▶│    combine    │   │   EVAL   │
+ │ daily bars   │      │ signal→weights│      │ merge alphas  │   │ score it │
+ └──────┬───────┘      └───────┬───────┘      └───────┬───────┘   └────┬─────┘
+        │                      │                      │                │
+        ▼                      ▼                      ▼                │
+ data/daily_bars/        alphas/*.pq            combos/*.pq            │
+  {universe}/            (ALPHA_COLUMNS)        (COMBO_COLUMNS)         │
+  month=YYYY-MM/*.pq            │                                      │
+ (DATA_COLUMNS)                 │                                      │
+        └──────── price ───────┴───────────────────────────────────────┘
+                                       │
+                                       ▼  (planned — not yet implemented)
+              ┌ ─ ─ ─ ─ ─ ─ ┐   ┌ ─ ─ ─ ─ ─ ─ ┐   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
+                 PORTFOLIO         BACKTEST           PAPER TRADING
+              │  construct  │   │   simulate  │   │  forward / live  │  TODO
+                 positions       fills + costs      execution
+              └ ─ ─ ─ ─ ─ ─ ┘   └ ─ ─ ─ ─ ─ ─ ┘   └ ─ ─ ─ ─ ─ ─ ─ ─ ┘
+
+ Each phase reads parquet and writes parquet — run, cache, and inspect
+ independently. The only interface between phases is the parquet schema.
+ Solid boxes are implemented; dashed boxes are on the roadmap (see TODO below).
+```
+
+Data comes from **baostock (primary)** with **akshare (fallback)**.

 ## Install

@@ -13,13 +46,218 @@ pip install -r requirements.txt
 ## Quick start

 ```bash
-python3 run_example.py        # end-to-end smoke test (SMA crossover on 浦发银行)
-python3 -m pytest tests/ -v   # run tests
+# 1. Download daily bars for a few symbols (writes a month-partitioned dataset).
+python3 cli.py data download \
+    --universe sh600000,sz000001,sh600519 \
+    --start-date 2024-01-01 --end-date 2024-03-31 \
+    --output-dir data/daily_bars
+
+# 2. Compute an alpha (position weights) from that data.
+#    --data-path is the dataset DIRECTORY ({output-dir}/{universe}).
+python3 cli.py alpha reversal \
+    --data-path "data/daily_bars/sh600000,sz000001,sh600519"
+
+# 3. Evaluate it (return / Sharpe / turnover / drawdown).
+python3 cli.py alpha eval \
+    --alpha-path alphas/reversal_5d.pq \
+    --data-path  "data/daily_bars/sh600000,sz000001,sh600519"
+
+# Tests
+python3 -m pytest tests/ -v        # tests/test_alpha.py is network-free; test_downloader.py hits the network
 ```

+## CLI reference
+
+All commands are subcommands of `python3 cli.py`. Add `--help` to any of them.
+
+### `data download` — fetch daily bars → partitioned parquet dataset
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `--universe` | `csi500` | `hs300`, `csi500`, `all` (~5000 A-shares), a file path (one symbol per line), or comma-separated symbols (`sh600000,sz000001`) |
+| `--start-date` | `2017-01-01` | `YYYY-MM-DD` |
+| `--end-date` | today | `YYYY-MM-DD` |
+| `--output-dir` | `data/daily_bars` | Root for the dataset directory |
+| `--symbols` | `0` | Max symbols to download (`0` = all) |
+| `--chunk-size` | `300` | Symbols per durability flush (each flush appends `.pq` files) |
+| `--adjust` | `qfq` | Price adjustment: `qfq` (forward), `hfq` (backward), `none` |
+
+Writes a **Hive-partitioned dataset** at `{output_dir}/{universe}/month=YYYY-MM/*.pq`
+(one partition per calendar month). The `{universe}` directory is rebuilt from
+scratch on each run. Downloads stream under a single baostock session and flush
+every `--chunk-size` symbols, so memory stays bounded and a crash keeps the
+partitions already written. Pass the **dataset directory** (`{output_dir}/{universe}`)
+as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
+set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
+
+### `alpha list` — show registered alpha types
+
+```bash
+python3 cli.py alpha list
+python3 cli.py alpha list --alpha-module path/to/my_alpha.py   # include an external alpha
+```
+
+### `alpha compute` — alpha class → weights parquet
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `--data-path` | (required) | Data parquet from `data download` |
+| `--alpha-name` | (required) | Label stored in the `alpha_name` column / output filename |
+| `--alpha-type` | (required) | Registry key of the alpha class (see `alpha list`) |
+| `--output-dir` | `alphas` | Output directory |
+| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
+| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
+| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
+| `--param` | — | Extra constructor param as `name=value`; repeatable |
+
+Only the params an alpha's `__init__` accepts are forwarded, so passing extras
+(e.g. `--vol-window` to a reversal alpha) is harmless.
+
+```bash
+python3 cli.py alpha compute \
+    --data-path <data>.pq \
+    --alpha-type reversal_vol --alpha-name rv_5_20 \
+    --lookback 5 --vol-window 20
+```
+
+Shortcuts for the two most common built-ins:
+
+```bash
+python3 cli.py alpha reversal     --data-path <data>.pq --lookback 5
+python3 cli.py alpha reversal-vol --data-path <data>.pq --lookback 5 --vol-window 20
+```
+
+### `alpha eval` — score an alpha as a portfolio
+
+```bash
+python3 cli.py alpha eval --alpha-path alphas/<name>.pq --data-path <data>.pq
+```
+
+Interprets the weights as a portfolio and reports cumulative return, annual
+Sharpe, annual turnover, max drawdown, and hit rate; also dumps
+`reports/<alpha_name>_eval.json`. There is deliberately **no IC/IR** — alphas are
+position weights, not return predictors.
+
+### `combo combine` — merge several alphas into one weight
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `--alpha-paths` | (required) | Comma-separated alpha parquet paths (≥ 2) |
+| `--combo-name` | (required) | Label stored in the `combo_name` column / output filename |
+| `--method` | `equal_weight` | Combination method (see `COMBO_METHODS`) |
+| `--output-dir` | `combos` | Output directory |
+
+```bash
+python3 cli.py combo combine \
+    --alpha-paths alphas/reversal_5d.pq,alphas/reversal_vol_5d_20d.pq \
+    --combo-name eq --method equal_weight
+```
+
+## Alphas: the factory / plugin interface
+
+An **alpha** is a class that maps a wide close matrix (date index × `symbol_id`
+columns) to **signed position weights** (positive = long, negative = short).
+Every alpha subclasses `BaseAlpha` (`pipeline/alpha/base.py`) and is resolved by
+name through the registry (`pipeline/alpha/registry.py`).
+
+### Minimal alpha
+
+```python
+import pandas as pd
+
+from pipeline.alpha.base import BaseAlpha
+from pipeline.alpha.registry import register_alpha
+
+
+@register_alpha
+class MyAlpha(BaseAlpha):
+    name = "my_alpha"                 # unique registry key (required)
+
+    def __init__(self, lookback: int = 5):
+        self.lookback = lookback      # declare whatever params you need
+
+    def signal(self, close: pd.DataFrame) -> pd.DataFrame:
+        # Raw score: wide (date × symbol_id), higher = stronger long, NaN where undefined.
+        return -close.pct_change(self.lookback)
+```
+
+That is the whole contract:
+
+- `name` — the `--alpha-type` key; must be unique.
+- `signal(close)` — the only required method; return a wide DataFrame.
+- `to_weights(signal)` — provided by the base class: cross-sectionally z-scores
+  each date into weights (NaN → 0). **Override** it for a different scheme (rank,
+  dollar-neutral caps, etc.).
+
+### Built-in alphas
+
+One file per alpha under `pipeline/alpha/library/`:
+
+| `--alpha-type` | Params | Description |
+| --- | --- | --- |
+| `reversal` | `lookback` | Negative trailing return (oversold scores high) |
+| `reversal_vol` | `lookback`, `vol_window` | Reversal scaled by trailing volatility |
+| `momentum` | `lookback` | Positive trailing return |
+
+Add a built-in by dropping a module in `pipeline/alpha/library/` and importing it
+from that package's `__init__.py`.
+
+### Using an alpha written outside this repo
+
+Write your `@register_alpha` class in any `.py` file, then register it at runtime
+with `--alpha-module` (a `.py` path or an importable dotted module). See the
+worked example in `examples/alphas/mean_reversion.py`:
+
+```bash
+python3 cli.py alpha compute \
+    --alpha-module examples/alphas/mean_reversion.py \
+    --alpha-type mean_reversion --alpha-name mr20 \
+    --param window=20 \
+    --data-path <data>.pq
+```
+
+`mean_reversion` declares a `window` param (not `lookback`); `--param window=20`
+supplies it and the unrelated `--lookback`/`--vol-window` defaults are ignored.
+
+## Parquet schemas
+
+The column contracts in `pipeline/common/schema.py` are the only interface
+between phases (data is stored long/tidy):
+
+- **data** (`DATA_COLUMNS`): `symbol_id, symbol_name, date, open, high, low, close, volume, amount`
+- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
+- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
+
+The data phase writes a month-partitioned dataset, so reading the dataset
+directory yields an extra `month` (`YYYY-MM`) partition column on top of
+`DATA_COLUMNS`; the alpha phase pivots by name and ignores it.
+
 ## Layout

- `data/` — unified downloader (akshare -> baostock fallback) and data schema
- `backtest/` — config, pandas->backtrader feed adapter, and `BacktestRunner`
- `strategies/` — example `SmaCross` strategy
- `analysis/` — performance reporting (sharpe, drawdown, returns, trades)
+- `cli.py` — entry point wiring the three phases together
+- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
+- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
+  `library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
+- `pipeline/combo/` — alpha combination → `combos/*.pq`
+- `pipeline/common/schema.py` — parquet column contracts
+- `data/downloader.py`, `data/universe.py` — baostock/akshare download + constituents
+- `examples/alphas/` — example external alpha(s)
+
+## Roadmap (not yet implemented)
+
+The pipeline currently ends at `combo`, and `alpha eval` only interprets a weight
+series as a portfolio for quick scoring (return / Sharpe / turnover / drawdown).
+It is **not** a true backtest — there is no transaction-cost, slippage, or
+execution modeling. The following phases are planned but not built yet:
+
+- [ ] **Portfolio construction** — turn combo weights into target positions
+  (gross/net exposure caps, per-name and sector limits, capital allocation,
+  rebalance schedule).
+- [ ] **Backtesting** — event-driven simulation over the constructed positions
+  with realistic fills, transaction costs, slippage, and borrow constraints;
+  richer P&L / risk attribution than `alpha eval`.
+- [ ] **Forward / paper trading** — run the same construction logic on live
+  daily data, track simulated fills and a running P&L without real capital.
+
+Until these land, treat `alpha eval` as a fast sanity check on a weight series,
+not a performance estimate.