chinese-equity-quant/README.md

# Chinese Equity Quant Research Framework

A modular Chinese A-share quant research framework. Daily frequency only (Phase 1).

It is a **decoupled, file-based pipeline**: each phase reads parquet and writes
parquet, so phases run, cache, and inspect independently.

```
  baostock (primary)                                              one weight series
  akshare  (fallback)                                             interpreted as a
        │                                                            portfolio
        ▼                                                              ▲
 ┌──────────────┐      ┌───────────────┐      ┌───────────────┐       │
 │     DATA     │      │     ALPHA     │      │     COMBO     │   ┌────┴─────┐
 │   download   │─────▶│    compute    │─────▶│    combine    │   │   EVAL   │
 │ daily bars   │      │ signal→weights│      │ merge alphas  │   │ score it │
 └──────┬───────┘      └───────┬───────┘      └───────┬───────┘   └────┬─────┘
        │                      │                      │                │
        ▼                      ▼                      ▼                │
 data/daily_bars/        alphas/*.pq            combos/*.pq            │
  {universe}/            (ALPHA_COLUMNS)        (COMBO_COLUMNS)         │
  month=YYYY-MM/*.pq            │                                      │
 (DATA_COLUMNS)                 │                                      │
        └──────── price ───────┴───────────────────────────────────────┘
                                       │
                                       ▼
              ┌──────────────┐   ┌──────────────┐   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
              │  PORTFOLIO   │   │   SIMULATE   │      PAPER TRADING
              │  construct   │──▶│ fills + costs│   │  forward / live  │  TODO
              │  positions   │   │   + P&L      │      execution
              └──────┬───────┘   └──────────────┘   └ ─ ─ ─ ─ ─ ─ ─ ─ ┘
                     ▼
              portfolio/*.pq
              (POSITION_COLUMNS)

 Each phase reads parquet and writes parquet — run, cache, and inspect
 independently. The only interface between phases is the parquet schema.
 Solid boxes are implemented; dashed boxes are on the roadmap (see TODO below).
```

Data comes from **baostock (primary)** with **akshare (fallback)**.

## Install

The env is managed with [uv](https://docs.astral.sh/uv/). `uv sync` builds `.venv` from `pyproject.toml` + `uv.lock`; prefix commands with `uv run` (no manual activation needed).

```bash
uv sync
```

Backtrader is an optional dependency and is **not used by the current pipeline**.
Install it only for future experiments or adapter work:

```bash
uv sync --extra backtrader
```

## Quick start

```bash
# 1. Download daily bars for a few symbols (writes a month-partitioned dataset).
uv run python cli.py data download \
    --universe sh600000,sz000001,sh600519 \
    --start-date 2024-01-01 --end-date 2024-03-31 \
    --output-dir data/daily_bars

# 2. Compute an alpha (position weights) from that data.
#    --data-path is the dataset DIRECTORY ({output-dir}/{universe}).
uv run python cli.py alpha reversal \
    --data-path "data/daily_bars/sh600000,sz000001,sh600519"

# 3. Evaluate it (return / Sharpe / turnover / drawdown).
uv run python cli.py alpha eval \
    --alpha-path alphas/reversal_5d.pq \
    --data-path  "data/daily_bars/sh600000,sz000001,sh600519"

# 4. Build tradable integer positions from alpha or combo weights.
uv run python cli.py portfolio build \
    --weights-path alphas/reversal_5d.pq \
    --data-path "data/daily_bars/sh600000,sz000001,sh600519" \
    --booksize 1000000 --portfolio-name reversal_port

# 5. Simulate next-open execution with A-share constraints, costs, and slippage.
uv run python cli.py portfolio simulate \
    --positions-path portfolio/reversal_port.pq \
    --data-path "data/daily_bars/sh600000,sz000001,sh600519" \
    --constraint suspension --constraint price_limit --constraint volume_cap \
    --cost-bps 5 --slippage-bps 5

# 6. Evaluate the constructed target weights as a continuous research portfolio.
uv run python cli.py portfolio eval \
    --positions-path portfolio/reversal_port.pq \
    --data-path "data/daily_bars/sh600000,sz000001,sh600519"

# Tests
uv run python -m pytest tests/ -v        # alpha/portfolio tests are network-free; downloader tests hit the network
```

## CLI reference

All commands are subcommands of `uv run python cli.py`. Add `--help` to any of them.

### `data download` — fetch daily bars → partitioned parquet dataset

| Option | Default | Description |
| --- | --- | --- |
| `--universe` | `csi500` | `hs300`, `csi500`, `all` (~5000 A-shares), a file path (one symbol per line), or comma-separated symbols (`sh600000,sz000001`) |
| `--start-date` | `2017-01-01` | `YYYY-MM-DD` |
| `--end-date` | today | `YYYY-MM-DD` |
| `--output-dir` | `data/daily_bars` | Root for the dataset directory |
| `--symbols` | `0` | Max symbols to download (`0` = all) |
| `--chunk-size` | `300` | Symbols per durability flush (each flush appends `.pq` files) |
| `--adjust` | `qfq` | Price adjustment: `qfq` (forward), `hfq` (backward), `none` |

Writes a **Hive-partitioned dataset** at `{output_dir}/{universe}/month=YYYY-MM/*.pq`
(one partition per calendar month). The `{universe}` directory is rebuilt from
scratch on each run. Downloads stream under a single baostock session and flush
every `--chunk-size` symbols, so memory stays bounded and a crash keeps the
partitions already written. Pass the **dataset directory** (`{output_dir}/{universe}`)
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).

### `derived` — daily custom/derived data

Derived data is daily-only v1 research data keyed by `symbol_id,date`, with one
or more numeric value columns. It can come from user CSV/parquet files or Python
plugins, and is written as a single parquet file at `derived/{name}.pq`.

The validator normalizes `date` to the trading day, requires unique
`symbol_id,date` keys, rejects duplicate columns, and rejects non-numeric value
columns. Alpha computation consumes derived data through the existing
`--feature-path` flag.

```bash
# Validate a user file without writing output.
uv run python cli.py derived validate --input-path vendor_factor.csv

# Ingest CSV/parquet into the canonical derived/ layout.
uv run python cli.py derived ingest \
    --input-path vendor_factor.csv \
    --derived-name vendor_factor

# List built-in and external derived-data plugin types.
uv run python cli.py derived list
uv run python cli.py derived list --derived-module path/to/my_derived.py

# Compute a derived file from daily and/or minute inputs.
uv run python cli.py derived compute \
    --minute-path data/minute_bars/sh600000 \
    --daily-path data/daily_bars/sh600000 \
    --derived-type minute_daily_summary \
    --derived-name minute_summary

# Join derived columns into a feature-aware alpha.
uv run python cli.py alpha compute \
    --data-path data/daily_bars/sh600000 \
    --feature-path derived/minute_summary.pq \
    --alpha-type my_feature_aware_alpha \
    --alpha-name my_run
```

For compatibility, `feature list` and `feature compute` remain available and
delegate to the same derived-data registry. Existing `features/*.pq` files are
still valid `--feature-path` inputs when they satisfy the daily numeric contract.

### `alpha list` — show registered alpha types

```bash
uv run python cli.py alpha list
uv run python cli.py alpha list --alpha-module path/to/my_alpha.py   # include an external alpha
```

### `alpha compute` — alpha class → weights parquet

| Option | Default | Description |
| --- | --- | --- |
| `--data-path` | (required) | Data parquet from `data download` |
| `--alpha-name` | (required) | Label stored in the `alpha_name` column / output filename |
| `--alpha-type` | (required) | Registry key of the alpha class (see `alpha list`) |
| `--output-dir` | `alphas` | Output directory |
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
| `--feature-path` | — | Daily derived/feature parquet file or dataset to left-join on `symbol_id,date`; repeatable |
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
| `--param` | — | Extra constructor param as `name=value`; repeatable |

Only the params an alpha's `__init__` accepts are forwarded, so passing extras
(e.g. `--vol-window` to a reversal alpha) is harmless.

```bash
uv run python cli.py alpha compute \
    --data-path <data>.pq \
    --alpha-type reversal_vol --alpha-name rv_5_20 \
    --lookback 5 --vol-window 20
```

Shortcuts for the two most common built-ins:

```bash
uv run python cli.py alpha reversal     --data-path <data>.pq --lookback 5
uv run python cli.py alpha reversal-vol --data-path <data>.pq --lookback 5 --vol-window 20
```

### `alpha eval` — score an alpha as a portfolio

```bash
uv run python cli.py alpha eval --alpha-path alphas/<name>.pq --data-path <data>.pq
```

Interprets the weights as a portfolio and reports cumulative return, annual
Sharpe, annual turnover, max drawdown, and hit rate; also dumps
`reports/<alpha_name>_eval.json`. There is deliberately **no IC/IR** — alphas are
position weights, not return predictors.

### `combo combine` — merge several alphas into one weight

| Option | Default | Description |
| --- | --- | --- |
| `--alpha-paths` | (required) | Comma-separated alpha parquet paths (≥ 2) |
| `--combo-name` | (required) | Label stored in the `combo_name` column / output filename |
| `--method` | `equal_weight` | Combination method (see `COMBO_METHODS`) |
| `--output-dir` | `combos` | Output directory |

```bash
uv run python cli.py combo combine \
    --alpha-paths alphas/reversal_5d.pq,alphas/reversal_vol_5d_20d.pq \
    --combo-name eq --method equal_weight
```

### `portfolio build` — weights → tradable positions

Turns alpha/combo weights into target weights, target yuan exposure, continuous
shares, and a lot-valid integer position book under A-share board rules.
Non-finite / non-positive construction prices are excluded before target
normalization. If a date has zero gross target after filtering, the previous
book is carried in `position_shares` and a warning is logged.

| Option | Default | Description |
| --- | --- | --- |
| `--weights-path` | (required) | Alpha or combo parquet with `symbol_id, date, weight` |
| `--data-path` | (required) | Data parquet file or partitioned dataset directory |
| `--booksize` | (required) | Target gross yuan exposure |
| `--portfolio-name` | (required) | Label stored in `portfolio_name` and output filename |
| `--price-field` | `close` | Data column used as construction price |
| `--output-dir` | `portfolio` | Output directory |

```bash
uv run python cli.py portfolio build \
    --weights-path combos/eq.pq --data-path data/daily_bars/csi500 \
    --booksize 10000000 --portfolio-name eq_10m
```

### `portfolio simulate` — constructed positions → fills + P&L

Executes the constructed `position_shares` book at the next available open,
clipping trades through repeatable constraints. It writes `fills/<name>.pq` and
`pnl/<name>.pq`. Trading costs use the simplified open-execution proportional
cash-cost model documented in
[`docs/portfolio_trading_cost_model.md`](docs/portfolio_trading_cost_model.md).

| Option | Default | Description |
| --- | --- | --- |
| `--positions-path` | (required) | Positions parquet from `portfolio build` |
| `--data-path` | (required) | Data parquet file or partitioned dataset directory |
| `--constraint` | — | Repeatable: `suspension`, `price_limit`, `volume_cap` |
| `--cost-bps` | `0.0` | Commission in basis points |
| `--slippage-bps` | `0.0` | Slippage in basis points |
| `--volume-frac` | `0.10` | Max traded value fraction for `volume_cap` |
| `--output-dir` | `.` | Base directory for `fills/` and `pnl/` |

```bash
uv run python cli.py portfolio simulate \
    --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500 \
    --constraint suspension --constraint price_limit --constraint volume_cap \
    --cost-bps 5 --slippage-bps 5
```

### `portfolio eval` — score constructed target weights

```bash
uv run python cli.py portfolio eval \
    --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500
```

Uses `target_weight` for a continuous research view: cumulative return,
annual Sharpe, annual turnover, max drawdown, Fitness, hit rate, and date count.
There is deliberately **no IC/IR**. Zero-gross carry dates remain flat in this
research view even though execution carries `position_shares`.

### `pqcat` — inspect a parquet file, like `cat`

Quickly dump any pipeline parquet (a single `.pq` file or a partitioned dataset
directory) to stdout, without writing a throwaway script.

| Option | Default | Description |
| --- | --- | --- |
| `-n, --head N` | — | Show only the first `N` rows |
| `-t, --tail N` | — | Show only the last `N` rows |
| `-c, --columns` | — | Comma-separated subset of columns to show |
| `--info` | off | Show shape + dtypes instead of the rows |

```bash
uv run python cli.py pqcat alphas/reversal_5d.pq                       # dump all rows
uv run python cli.py pqcat data/daily_bars/csi500 --info              # shape + dtypes
uv run python cli.py pqcat data/daily_bars/csi500 -n 10 -c symbol_id,date,close,vwap
```

**Standalone command.** `tools/pqcat.py` has no repo dependencies, so it can be
run directly. Symlink it onto your `PATH` once and call `pqcat` from anywhere:

```bash
ln -sf "$(pwd)/tools/pqcat.py" ~/.local/bin/pqcat   # ~/.local/bin must be on PATH
pqcat alphas/reversal_5d.pq --info
```

### `alphaview` — alpha weight(s) alongside bar data for one symbol

Join the bar dataset and one or more alpha parquet files on `(symbol, date)` and
print them side by side, so you can eyeball how a weight moves against price /
volume over a time range.

| Option | Default | Description |
| --- | --- | --- |
| `--data-path` | (required) | Bar dataset dir or parquet file |
| `--alpha-path` | (required) | Comma-separated alpha parquet path(s) — each `alpha_name` becomes a column |
| `--symbol` | (required) | Symbol id, e.g. `sh600000` |
| `--start-date` | — | `YYYY-MM-DD` (inclusive) |
| `--end-date` | — | `YYYY-MM-DD` (inclusive) |
| `-c, --columns` | `close,volume` | Comma-separated bar columns to show |

```bash
uv run python cli.py alphaview \
    --data-path data/daily_bars/csi500 \
    --alpha-path alphas/reversal_5d.pq,alphas/momentum_5d.pq \
    --symbol sh600000 --start-date 2024-01-01 --end-date 2024-03-31 \
    -c close,volume,vwap
```

Also standalone like `pqcat` — `ln -sf "$(pwd)/tools/alphaview.py" ~/.local/bin/alphaview`.

## Alphas: the factory / plugin interface

An **alpha** is a class that maps a wide close matrix (date index × `symbol_id`
columns) to **signed position weights** (positive = long, negative = short).
Every alpha subclasses `BaseAlpha` (`pipeline/alpha/base.py`) and is resolved by
name through the registry (`pipeline/alpha/registry.py`).

### Minimal alpha

```python
import pandas as pd

from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha


@register_alpha
class MyAlpha(BaseAlpha):
    name = "my_alpha"                 # unique registry key (required)

    def __init__(self, lookback: int = 5):
        self.lookback = lookback      # declare whatever params you need

    def signal(self, close: pd.DataFrame) -> pd.DataFrame:
        # Raw score: wide (date × symbol_id), higher = stronger long, NaN where undefined.
        return -close.pct_change(self.lookback)
```

That is the whole contract:

- `name` — the `--alpha-type` key; must be unique.
- `signal(close)` — the only required method; return a wide DataFrame.
- `to_weights(signal)` — provided by the base class: cross-sectionally z-scores
  each date into weights (NaN → 0). **Override** it for a different scheme (rank,
  dollar-neutral caps, etc.).

### Built-in alphas

One file per alpha under `pipeline/alpha/library/`:

| `--alpha-type` | Params | Description |
| --- | --- | --- |
| `reversal` | `lookback` | Negative trailing return (oversold scores high) |
| `reversal_vol` | `lookback`, `vol_window` | Reversal scaled by trailing volatility |
| `momentum` | `lookback` | Positive trailing return |

Add a built-in by dropping a module in `pipeline/alpha/library/` and importing it
from that package's `__init__.py`.

### Using an alpha written outside this repo

Write your `@register_alpha` class in any `.py` file, then register it at runtime
with `--alpha-module` (a `.py` path or an importable dotted module). See the
worked example in `examples/alphas/mean_reversion.py`:

```bash
uv run python cli.py alpha compute \
    --alpha-module examples/alphas/mean_reversion.py \
    --alpha-type mean_reversion --alpha-name mr20 \
    --param window=20 \
    --data-path <data>.pq
```

`mean_reversion` declares a `window` param (not `lookback`); `--param window=20`
supplies it and the unrelated `--lookback`/`--vol-window` defaults are ignored.

## Parquet schemas

The column contracts in `pipeline/common/schema.py` are the only interface
between phases (data is stored long/tidy):

- **data** (`DATA_COLUMNS`): `symbol_id, symbol_name, date, open, high, low, close, preclose, volume, amount, vwap, turn, pctChg, tradestatus, isST, peTTM, pbMRQ, psTTM, pcfNcfTTM`
  (`vwap` = `amount / volume` — a **raw**-price daily VWAP, *not* on the adjusted
  OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
  `tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
  baostock valuation ratios.)
- **derived** (`DERIVED_KEY_COLUMNS` + values): required keys `symbol_id, date`;
  value columns are user/plugin-defined and must be numeric in v1.
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
  (`target_*` are continuous construction targets; `position_shares` is the
  discretized + repaired integer book used by execution.)
- **fills** (`FILL_COLUMNS`): `symbol_id, date, portfolio_name, prev_shares, target_shares, traded_shares, realized_shares, blocked, trade_cost`
  (`date` is the execution date, i.e. the next open after the target date.)
- **pnl** (`PNL_COLUMNS`): `date, portfolio_name, gross_exposure, net_exposure, pnl, cost, turnover, n_positions`

The data phase writes a month-partitioned dataset, so reading the dataset
directory yields an extra `month` (`YYYY-MM`) partition column on top of
`DATA_COLUMNS`; the alpha phase pivots by name and ignores it.

## Layout

- `cli.py` — entry point wiring the file-based phases together
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
- `pipeline/derived/` — daily derived-data ingestion, validation, plugin registry,
  and built-in derived computations → `derived/*.pq`
- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
  `library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
- `pipeline/features/` — compatibility wrappers for the derived-data registry
- `pipeline/combo/` — alpha combination → `combos/*.pq`
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
  reference next-open simulator, and research metrics
- `pipeline/common/schema.py` — parquet column contracts
- `data/downloader.py`, `data/universe.py` — baostock/akshare download + constituents
- `tools/pqcat.py` — standalone parquet inspector (`pqcat`), also wired as `cli.py pqcat`
- `tools/alphaview.py` — standalone alpha-vs-bar viewer (`alphaview`), also wired as `cli.py alphaview`
- `examples/alphas/` — example external alpha(s)

## Roadmap / current limits

The pipeline is implemented through portfolio construction and a reference
daily execution simulator. `alpha eval` remains a fast sanity check on raw
weights; use `portfolio build`, `portfolio simulate`, and `portfolio eval` for
constructed positions, fills/costs, P&L, and target-weight research metrics.

- [x] **Portfolio construction** — turn alpha/combo weights into continuous
  targets and lot-valid integer positions under A-share board rules.
- [x] **Reference execution simulation** — next-open fills over constructed
  `position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
  and slippage controls.
- [x] **Derived/custom daily data ("Level 2")** — ingest user CSV/parquet files
  or compute plugin outputs as validated numeric daily datasets under
  `derived/{name}.pq`; alpha joins continue through `--feature-path`.
- [ ] **Optional Backtrader adapter** — Backtrader is available as the
  `backtrader` extra for possible future event-driven/broker-style experiments,
  but it is not part of the current canonical portfolio workflow.
- [ ] **Forward / paper trading** — run the same construction logic on live
  daily data, track simulated fills and a running P&L without real capital.
- [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price,
  and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
  brokerage data tier); the free daily sources here only expose daily bars, so
  this is a separate data phase rather than extra columns on the daily schema.

### Additional TODOs

The following items are intended extensions beyond the current daily
alpha-to-portfolio pipeline:

- **Long-only portfolio mode** — add a construction option that converts
  alpha/combo weights into a long-only book while preserving existing lot,
  price, suspension, and volume-cap handling.
- **Index-short hedging mode** — support portfolios that hold long A-share
  names while shorting an index or index proxy for market exposure control.
- **Expanded universe presets** — add explicit universe aliases for CSI 300,
  CSI 500, CSI 1000, and CSI 1800, while keeping file-based and comma-separated
  custom universes available.
- **Categorical derived data** — extend the numeric-only derived-data v1 contract
  to support categorical inputs such as industry classifications. In this
  project, "Level 2" means customized second-level research data produced by
  users or plugins; it does not necessarily mean exchange order-book/L2 quote
  feeds.
- **Minute bar data** — continue extending the raw minute-bar and feature
  workflow. The initial Baostock 5-minute download and daily feature plugin path
  exist; intraday execution and replacing canonical daily bars remain out of
  scope unless explicitly added later.
- **Industry data** — add industry classification inputs for filtering,
  grouping, exposure reporting, neutralization, or industry-aware portfolio
  construction.