Yuxuan Yan
1caa63faeb
Replace the old signal/strategy/backtest modules with a decoupled
data → alpha → combo pipeline (parquet between phases, .pq extension).
Alphas:
- BaseAlpha + @register_alpha factory/plugin registry; one file per
built-in (reversal, reversal_vol, momentum); external alphas via
--alpha-module. Alphas are z-scored position weights, not predictors.
Data:
- baostock primary / akshare fallback, treated consistently.
- New --universe all (~5000 A-shares via query_all_stock, filtered).
- login-once batch downloader; empty-string OHLCV coerced to NaN.
- Month-partitioned dataset {output_dir}/{universe}/month=YYYY-MM/*.pq
with chunked durability flushes; --data-path is the dataset dir.
CLI logs at INFO by default (--log-level) so progress is visible.
Docs (README, CLAUDE.md) updated incl. pipeline diagram and roadmap
TODOs for portfolio construction / backtest / paper trading.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
80 lines
6.7 KiB
Markdown
80 lines
6.7 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
A modular Chinese A-share quant research framework. Daily frequency only (Phase 1).
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
pip install -r requirements.txt # install deps
|
||
python3 -m pytest tests/ -v # all tests
|
||
python3 -m pytest tests/test_alpha.py -v # single file (test_alpha is network-free)
|
||
python3 -m pytest tests/test_alpha.py::test_evaluate_alpha_keys -v # single test
|
||
|
||
# Pipeline — each phase is independent: reads parquet, writes parquet.
|
||
python3 cli.py data download --universe csi500 --start-date 2017-01-01 # → data/daily_bars/csi500/ (month-partitioned)
|
||
python3 cli.py alpha reversal --data-path data/daily_bars/<universe> # --data-path is the dataset DIR
|
||
python3 cli.py alpha eval --alpha-path alphas/<file>.pq --data-path data/daily_bars/<universe>
|
||
python3 cli.py combo combine --alpha-paths a.pq,b.pq --combo-name eq --method equal_weight
|
||
```
|
||
|
||
Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` is pure and fast.
|
||
|
||
## Architecture: one decoupled pipeline
|
||
|
||
The system is a phase-based CLI (`cli.py` + `pipeline/`). Each phase communicates **only** through parquet files on disk, so phases can be run, cached, and inspected independently:
|
||
|
||
```
|
||
data → alpha → combo
|
||
```
|
||
|
||
- `pipeline/data/` — download daily bars for a universe → `data/daily_bars/{universe}/month=YYYY-MM/*.pq` (Hive-partitioned dataset; pass the `{universe}` dir as `--data-path`)
|
||
- `pipeline/alpha/` — compute one alpha's position weights from a data parquet → `alphas/*.pq`, and `alpha eval` to score it
|
||
- `pipeline/combo/` — combine several alpha parquets into one → `combos/*.pq`
|
||
|
||
The pipeline reuses two top-level helper modules via a `sys.path.insert` at the top of `pipeline/data/downloader.py`: `data/downloader.py` (network download) and `data/universe.py` (constituent lists). This path hack is load-bearing — keep it.
|
||
|
||
## Alphas are weights, not predictors
|
||
|
||
An **alpha** is a signed cross-sectional **position weight**: positive = long, negative = short. It is produced by applying a formula to the wide close matrix, then **cross-sectional z-scoring** per date (`compute_alpha` in `pipeline/alpha/compute.py`). Alphas are evaluated by **return / Sharpe / turnover / max-drawdown** in `evaluate_alpha` — interpreting the weight series as a portfolio. There is deliberately **no IC/IR** anywhere: those frame a signal as a return *predictor*, which this codebase does not do. Do not reintroduce IC-style evaluation.
|
||
|
||
## Parquet schema contracts
|
||
|
||
`pipeline/common/schema.py` defines the column contracts that are the *only* interface between phases. Any new phase or alpha must conform:
|
||
- `DATA_COLUMNS` (data output): `symbol_id, symbol_name, date, open, high, low, close, volume, amount`
|
||
- `ALPHA_COLUMNS` (alpha output): `symbol_id, date, alpha_name, weight`
|
||
- `COMBO_COLUMNS` (combo output): `symbol_id, date, combo_name, weight`
|
||
|
||
Data is stored **long/tidy**, not wide, as a Hive-partitioned dataset keyed by `month=YYYY-MM` (so reads of the dataset directory carry an extra `month` partition column, which `_pivot_close` ignores). Compute code pivots to wide (date index × symbol_id columns) internally via `_pivot_close`, where all formulas are vectorized column-wise.
|
||
|
||
## Alphas: factory + plugin pattern
|
||
|
||
Each alpha is a class subclassing `BaseAlpha` (`pipeline/alpha/base.py`), living in its own module. It implements `signal(close) -> wide DataFrame` (the raw score); the base class's `to_weights` cross-sectionally z-scores that into position weights (override for custom normalization). Subclasses declare their own typed `__init__` params (e.g. `lookback`, `vol_window`, or anything custom).
|
||
|
||
- Built-in alphas: one file each under `pipeline/alpha/library/` (`reversal.py`, `reversal_vol.py`, `momentum.py`). Each uses the `@register_alpha` decorator and is imported by `library/__init__.py` so it self-registers. **Add a built-in** by dropping a module there + adding it to that `__init__`.
|
||
- Factory: `pipeline/alpha/registry.py` — `register_alpha`, `get_alpha(name, **params)` (forwards only the params the alpha's `__init__` accepts, via signature inspection), `available_alphas()`, and `load_alpha_module(spec)`.
|
||
- **External alphas** (authored outside this repo) are the point of the design: write a `@register_alpha class MyAlpha(BaseAlpha)` in any `.py` file, then register it at runtime with `--alpha-module path/to/file.py` (or a dotted module path). See `examples/alphas/mean_reversion.py` for a working example, and `tests/test_alpha.py::test_load_external_alpha_module`.
|
||
|
||
```bash
|
||
python3 cli.py alpha list # registered alpha types
|
||
python3 cli.py alpha list --alpha-module my_alpha.py # incl. an external one
|
||
python3 cli.py alpha compute --alpha-module my_alpha.py \
|
||
--alpha-type my_alpha --alpha-name run1 --param decay=0.9 --data-path <data>.pq
|
||
```
|
||
|
||
`compute_alpha(data, alpha_name, alpha_type, **params)` in `pipeline/alpha/compute.py` resolves the class via `get_alpha`, applies `.weights()`, and melts to `ALPHA_COLUMNS`. `--lookback`/`--vol-window` are passed as params for convenience; arbitrary params go through repeatable `--param name=value`. `evaluate_alpha` (return/Sharpe/turnover, no IC) is unchanged.
|
||
|
||
Combo methods are still a plain dict registry: `COMBO_METHODS` in `pipeline/combo/combine.py`.
|
||
|
||
## Data sources & symbol conventions
|
||
|
||
- **baostock is the primary source, akshare the fallback** (`data/downloader.py`, `download_daily(source="auto")` tries baostock first). This ordering is intentional — akshare is less reliable on the deployment network.
|
||
- Internal symbol format is `sh600000` / `sz000001` (exchange prefix + code). baostock uses the dotted form `sh.600000`; akshare's `stock_zh_a_hist` wants the bare code (prefix stripped). Both download paths return the identical 8-column schema and map the `adjust` argument consistently (`qfq`/`hfq`/none → baostock `adjustflag` via `_BAOSTOCK_ADJUST`).
|
||
- baostock constituent queries (`get_hs300_stocks`, `get_zz500_stocks` in `data/universe.py`) return columns in an unreliable order, so `pipeline/data/downloader.py:_fix_baostock_columns` detects them by value pattern, not position.
|
||
- Universes accepted by `data download --universe`: `hs300`, `csi500`, `all`/`full` (every listed A-share, ~5000, via `get_all_stocks` → baostock `query_all_stock` filtered to SH 6xxxxx/68xxxx + SZ 0xxxxx/3xxxxx, excluding indices & B-shares), a file path (one symbol per line), or a comma-separated symbol list. Bulk downloads use `download_daily_batch` (one baostock login for the whole run) rather than per-symbol `download_daily`.
|
||
|
||
## Code standards
|
||
|
||
- Type hints on public functions; Google-style docstrings; 4-space indentation.
|