Yuxuan Yan
0c524c6987
Joins the bar dataset with one or more alpha parquet files on (symbol, date) for a given symbol and date range. Standalone tools/alphaview.py wired as `cli.py alphaview`, plus README docs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
325 lines
15 KiB
Markdown
325 lines
15 KiB
Markdown
# Chinese Equity Quant Research Framework
|
||
|
||
A modular Chinese A-share quant research framework. Daily frequency only (Phase 1).
|
||
|
||
It is a **decoupled, file-based pipeline**: each phase reads parquet and writes
|
||
parquet, so phases run, cache, and inspect independently.
|
||
|
||
```
|
||
baostock (primary) one weight series
|
||
akshare (fallback) interpreted as a
|
||
│ portfolio
|
||
▼ ▲
|
||
┌──────────────┐ ┌───────────────┐ ┌───────────────┐ │
|
||
│ DATA │ │ ALPHA │ │ COMBO │ ┌────┴─────┐
|
||
│ download │─────▶│ compute │─────▶│ combine │ │ EVAL │
|
||
│ daily bars │ │ signal→weights│ │ merge alphas │ │ score it │
|
||
└──────┬───────┘ └───────┬───────┘ └───────┬───────┘ └────┬─────┘
|
||
│ │ │ │
|
||
▼ ▼ ▼ │
|
||
data/daily_bars/ alphas/*.pq combos/*.pq │
|
||
{universe}/ (ALPHA_COLUMNS) (COMBO_COLUMNS) │
|
||
month=YYYY-MM/*.pq │ │
|
||
(DATA_COLUMNS) │ │
|
||
└──────── price ───────┴───────────────────────────────────────┘
|
||
│
|
||
▼ (planned — not yet implemented)
|
||
┌ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
|
||
PORTFOLIO BACKTEST PAPER TRADING
|
||
│ construct │ │ simulate │ │ forward / live │ TODO
|
||
positions fills + costs execution
|
||
└ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘
|
||
|
||
Each phase reads parquet and writes parquet — run, cache, and inspect
|
||
independently. The only interface between phases is the parquet schema.
|
||
Solid boxes are implemented; dashed boxes are on the roadmap (see TODO below).
|
||
```
|
||
|
||
Data comes from **baostock (primary)** with **akshare (fallback)**.
|
||
|
||
## Install
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
# 1. Download daily bars for a few symbols (writes a month-partitioned dataset).
|
||
python3 cli.py data download \
|
||
--universe sh600000,sz000001,sh600519 \
|
||
--start-date 2024-01-01 --end-date 2024-03-31 \
|
||
--output-dir data/daily_bars
|
||
|
||
# 2. Compute an alpha (position weights) from that data.
|
||
# --data-path is the dataset DIRECTORY ({output-dir}/{universe}).
|
||
python3 cli.py alpha reversal \
|
||
--data-path "data/daily_bars/sh600000,sz000001,sh600519"
|
||
|
||
# 3. Evaluate it (return / Sharpe / turnover / drawdown).
|
||
python3 cli.py alpha eval \
|
||
--alpha-path alphas/reversal_5d.pq \
|
||
--data-path "data/daily_bars/sh600000,sz000001,sh600519"
|
||
|
||
# Tests
|
||
python3 -m pytest tests/ -v # tests/test_alpha.py is network-free; test_downloader.py hits the network
|
||
```
|
||
|
||
## CLI reference
|
||
|
||
All commands are subcommands of `python3 cli.py`. Add `--help` to any of them.
|
||
|
||
### `data download` — fetch daily bars → partitioned parquet dataset
|
||
|
||
| Option | Default | Description |
|
||
| --- | --- | --- |
|
||
| `--universe` | `csi500` | `hs300`, `csi500`, `all` (~5000 A-shares), a file path (one symbol per line), or comma-separated symbols (`sh600000,sz000001`) |
|
||
| `--start-date` | `2017-01-01` | `YYYY-MM-DD` |
|
||
| `--end-date` | today | `YYYY-MM-DD` |
|
||
| `--output-dir` | `data/daily_bars` | Root for the dataset directory |
|
||
| `--symbols` | `0` | Max symbols to download (`0` = all) |
|
||
| `--chunk-size` | `300` | Symbols per durability flush (each flush appends `.pq` files) |
|
||
| `--adjust` | `qfq` | Price adjustment: `qfq` (forward), `hfq` (backward), `none` |
|
||
|
||
Writes a **Hive-partitioned dataset** at `{output_dir}/{universe}/month=YYYY-MM/*.pq`
|
||
(one partition per calendar month). The `{universe}` directory is rebuilt from
|
||
scratch on each run. Downloads stream under a single baostock session and flush
|
||
every `--chunk-size` symbols, so memory stays bounded and a crash keeps the
|
||
partitions already written. Pass the **dataset directory** (`{output_dir}/{universe}`)
|
||
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
|
||
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
|
||
|
||
### `alpha list` — show registered alpha types
|
||
|
||
```bash
|
||
python3 cli.py alpha list
|
||
python3 cli.py alpha list --alpha-module path/to/my_alpha.py # include an external alpha
|
||
```
|
||
|
||
### `alpha compute` — alpha class → weights parquet
|
||
|
||
| Option | Default | Description |
|
||
| --- | --- | --- |
|
||
| `--data-path` | (required) | Data parquet from `data download` |
|
||
| `--alpha-name` | (required) | Label stored in the `alpha_name` column / output filename |
|
||
| `--alpha-type` | (required) | Registry key of the alpha class (see `alpha list`) |
|
||
| `--output-dir` | `alphas` | Output directory |
|
||
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
|
||
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
|
||
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
|
||
| `--param` | — | Extra constructor param as `name=value`; repeatable |
|
||
|
||
Only the params an alpha's `__init__` accepts are forwarded, so passing extras
|
||
(e.g. `--vol-window` to a reversal alpha) is harmless.
|
||
|
||
```bash
|
||
python3 cli.py alpha compute \
|
||
--data-path <data>.pq \
|
||
--alpha-type reversal_vol --alpha-name rv_5_20 \
|
||
--lookback 5 --vol-window 20
|
||
```
|
||
|
||
Shortcuts for the two most common built-ins:
|
||
|
||
```bash
|
||
python3 cli.py alpha reversal --data-path <data>.pq --lookback 5
|
||
python3 cli.py alpha reversal-vol --data-path <data>.pq --lookback 5 --vol-window 20
|
||
```
|
||
|
||
### `alpha eval` — score an alpha as a portfolio
|
||
|
||
```bash
|
||
python3 cli.py alpha eval --alpha-path alphas/<name>.pq --data-path <data>.pq
|
||
```
|
||
|
||
Interprets the weights as a portfolio and reports cumulative return, annual
|
||
Sharpe, annual turnover, max drawdown, and hit rate; also dumps
|
||
`reports/<alpha_name>_eval.json`. There is deliberately **no IC/IR** — alphas are
|
||
position weights, not return predictors.
|
||
|
||
### `combo combine` — merge several alphas into one weight
|
||
|
||
| Option | Default | Description |
|
||
| --- | --- | --- |
|
||
| `--alpha-paths` | (required) | Comma-separated alpha parquet paths (≥ 2) |
|
||
| `--combo-name` | (required) | Label stored in the `combo_name` column / output filename |
|
||
| `--method` | `equal_weight` | Combination method (see `COMBO_METHODS`) |
|
||
| `--output-dir` | `combos` | Output directory |
|
||
|
||
```bash
|
||
python3 cli.py combo combine \
|
||
--alpha-paths alphas/reversal_5d.pq,alphas/reversal_vol_5d_20d.pq \
|
||
--combo-name eq --method equal_weight
|
||
```
|
||
|
||
### `pqcat` — inspect a parquet file, like `cat`
|
||
|
||
Quickly dump any pipeline parquet (a single `.pq` file or a partitioned dataset
|
||
directory) to stdout, without writing a throwaway script.
|
||
|
||
| Option | Default | Description |
|
||
| --- | --- | --- |
|
||
| `-n, --head N` | — | Show only the first `N` rows |
|
||
| `-t, --tail N` | — | Show only the last `N` rows |
|
||
| `-c, --columns` | — | Comma-separated subset of columns to show |
|
||
| `--info` | off | Show shape + dtypes instead of the rows |
|
||
|
||
```bash
|
||
python3 cli.py pqcat alphas/reversal_5d.pq # dump all rows
|
||
python3 cli.py pqcat data/daily_bars/csi500 --info # shape + dtypes
|
||
python3 cli.py pqcat data/daily_bars/csi500 -n 10 -c symbol_id,date,close,vwap
|
||
```
|
||
|
||
**Standalone command.** `tools/pqcat.py` has no repo dependencies, so it can be
|
||
run directly. Symlink it onto your `PATH` once and call `pqcat` from anywhere:
|
||
|
||
```bash
|
||
ln -sf "$(pwd)/tools/pqcat.py" ~/.local/bin/pqcat # ~/.local/bin must be on PATH
|
||
pqcat alphas/reversal_5d.pq --info
|
||
```
|
||
|
||
### `alphaview` — alpha weight(s) alongside bar data for one symbol
|
||
|
||
Join the bar dataset and one or more alpha parquet files on `(symbol, date)` and
|
||
print them side by side, so you can eyeball how a weight moves against price /
|
||
volume over a time range.
|
||
|
||
| Option | Default | Description |
|
||
| --- | --- | --- |
|
||
| `--data-path` | (required) | Bar dataset dir or parquet file |
|
||
| `--alpha-path` | (required) | Comma-separated alpha parquet path(s) — each `alpha_name` becomes a column |
|
||
| `--symbol` | (required) | Symbol id, e.g. `sh600000` |
|
||
| `--start-date` | — | `YYYY-MM-DD` (inclusive) |
|
||
| `--end-date` | — | `YYYY-MM-DD` (inclusive) |
|
||
| `-c, --columns` | `close,volume` | Comma-separated bar columns to show |
|
||
|
||
```bash
|
||
python3 cli.py alphaview \
|
||
--data-path data/daily_bars/csi500 \
|
||
--alpha-path alphas/reversal_5d.pq,alphas/momentum_5d.pq \
|
||
--symbol sh600000 --start-date 2024-01-01 --end-date 2024-03-31 \
|
||
-c close,volume,vwap
|
||
```
|
||
|
||
Also standalone like `pqcat` — `ln -sf "$(pwd)/tools/alphaview.py" ~/.local/bin/alphaview`.
|
||
|
||
## Alphas: the factory / plugin interface
|
||
|
||
An **alpha** is a class that maps a wide close matrix (date index × `symbol_id`
|
||
columns) to **signed position weights** (positive = long, negative = short).
|
||
Every alpha subclasses `BaseAlpha` (`pipeline/alpha/base.py`) and is resolved by
|
||
name through the registry (`pipeline/alpha/registry.py`).
|
||
|
||
### Minimal alpha
|
||
|
||
```python
|
||
import pandas as pd
|
||
|
||
from pipeline.alpha.base import BaseAlpha
|
||
from pipeline.alpha.registry import register_alpha
|
||
|
||
|
||
@register_alpha
|
||
class MyAlpha(BaseAlpha):
|
||
name = "my_alpha" # unique registry key (required)
|
||
|
||
def __init__(self, lookback: int = 5):
|
||
self.lookback = lookback # declare whatever params you need
|
||
|
||
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||
# Raw score: wide (date × symbol_id), higher = stronger long, NaN where undefined.
|
||
return -close.pct_change(self.lookback)
|
||
```
|
||
|
||
That is the whole contract:
|
||
|
||
- `name` — the `--alpha-type` key; must be unique.
|
||
- `signal(close)` — the only required method; return a wide DataFrame.
|
||
- `to_weights(signal)` — provided by the base class: cross-sectionally z-scores
|
||
each date into weights (NaN → 0). **Override** it for a different scheme (rank,
|
||
dollar-neutral caps, etc.).
|
||
|
||
### Built-in alphas
|
||
|
||
One file per alpha under `pipeline/alpha/library/`:
|
||
|
||
| `--alpha-type` | Params | Description |
|
||
| --- | --- | --- |
|
||
| `reversal` | `lookback` | Negative trailing return (oversold scores high) |
|
||
| `reversal_vol` | `lookback`, `vol_window` | Reversal scaled by trailing volatility |
|
||
| `momentum` | `lookback` | Positive trailing return |
|
||
|
||
Add a built-in by dropping a module in `pipeline/alpha/library/` and importing it
|
||
from that package's `__init__.py`.
|
||
|
||
### Using an alpha written outside this repo
|
||
|
||
Write your `@register_alpha` class in any `.py` file, then register it at runtime
|
||
with `--alpha-module` (a `.py` path or an importable dotted module). See the
|
||
worked example in `examples/alphas/mean_reversion.py`:
|
||
|
||
```bash
|
||
python3 cli.py alpha compute \
|
||
--alpha-module examples/alphas/mean_reversion.py \
|
||
--alpha-type mean_reversion --alpha-name mr20 \
|
||
--param window=20 \
|
||
--data-path <data>.pq
|
||
```
|
||
|
||
`mean_reversion` declares a `window` param (not `lookback`); `--param window=20`
|
||
supplies it and the unrelated `--lookback`/`--vol-window` defaults are ignored.
|
||
|
||
## Parquet schemas
|
||
|
||
The column contracts in `pipeline/common/schema.py` are the only interface
|
||
between phases (data is stored long/tidy):
|
||
|
||
- **data** (`DATA_COLUMNS`): `symbol_id, symbol_name, date, open, high, low, close, preclose, volume, amount, vwap, turn, pctChg, tradestatus, isST, peTTM, pbMRQ, psTTM, pcfNcfTTM`
|
||
(`vwap` = `amount / volume` — a **raw**-price daily VWAP, *not* on the adjusted
|
||
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
|
||
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
|
||
baostock valuation ratios.)
|
||
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
|
||
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
|
||
|
||
The data phase writes a month-partitioned dataset, so reading the dataset
|
||
directory yields an extra `month` (`YYYY-MM`) partition column on top of
|
||
`DATA_COLUMNS`; the alpha phase pivots by name and ignores it.
|
||
|
||
## Layout
|
||
|
||
- `cli.py` — entry point wiring the three phases together
|
||
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
|
||
- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
|
||
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
|
||
- `pipeline/combo/` — alpha combination → `combos/*.pq`
|
||
- `pipeline/common/schema.py` — parquet column contracts
|
||
- `data/downloader.py`, `data/universe.py` — baostock/akshare download + constituents
|
||
- `tools/pqcat.py` — standalone parquet inspector (`pqcat`), also wired as `cli.py pqcat`
|
||
- `tools/alphaview.py` — standalone alpha-vs-bar viewer (`alphaview`), also wired as `cli.py alphaview`
|
||
- `examples/alphas/` — example external alpha(s)
|
||
|
||
## Roadmap (not yet implemented)
|
||
|
||
The pipeline currently ends at `combo`, and `alpha eval` only interprets a weight
|
||
series as a portfolio for quick scoring (return / Sharpe / turnover / drawdown).
|
||
It is **not** a true backtest — there is no transaction-cost, slippage, or
|
||
execution modeling. The following phases are planned but not built yet:
|
||
|
||
- [ ] **Portfolio construction** — turn combo weights into target positions
|
||
(gross/net exposure caps, per-name and sector limits, capital allocation,
|
||
rebalance schedule).
|
||
- [ ] **Backtesting** — event-driven simulation over the constructed positions
|
||
with realistic fills, transaction costs, slippage, and borrow constraints;
|
||
richer P&L / risk attribution than `alpha eval`.
|
||
- [ ] **Forward / paper trading** — run the same construction logic on live
|
||
daily data, track simulated fills and a running P&L without real capital.
|
||
- [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price,
|
||
and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
|
||
brokerage data tier); the free daily sources here only expose daily bars, so
|
||
this is a separate data phase rather than extra columns on the daily schema.
|
||
|
||
Until these land, treat `alpha eval` as a fast sanity check on a weight series,
|
||
not a performance estimate.
|