Document implemented portfolio workflow
This commit is contained in:
Yuxuan Yan
@@ -19,23 +19,27 @@ uv run python cli.py data download --universe csi500 --start-date 2017-01-01 #
|
||||
uv run python cli.py alpha reversal --data-path data/daily_bars/<universe> # --data-path is the dataset DIR
|
||||
uv run python cli.py alpha eval --alpha-path alphas/<file>.pq --data-path data/daily_bars/<universe>
|
||||
uv run python cli.py combo combine --alpha-paths a.pq,b.pq --combo-name eq --method equal_weight
|
||||
uv run python cli.py portfolio build --weights-path combos/eq.pq --data-path data/daily_bars/<universe> --booksize 10000000 --portfolio-name eq_10m
|
||||
uv run python cli.py portfolio simulate --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/<universe> --constraint suspension --constraint price_limit --constraint volume_cap --cost-bps 5 --slippage-bps 5
|
||||
uv run python cli.py portfolio eval --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/<universe>
|
||||
```
|
||||
|
||||
Add a runtime dep with `uv add <pkg>`, a dev/test dep with `uv add --dev <pkg>` (both update `pyproject.toml` + `uv.lock`).
|
||||
|
||||
Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` is pure and fast.
|
||||
Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` and `tests/test_portfolio.py` are pure and fast.
|
||||
|
||||
## Architecture: one decoupled pipeline
|
||||
|
||||
The system is a phase-based CLI (`cli.py` + `pipeline/`). Each phase communicates **only** through parquet files on disk, so phases can be run, cached, and inspected independently:
|
||||
|
||||
```
|
||||
data → alpha → combo
|
||||
data → alpha → combo → portfolio build → portfolio simulate/eval
|
||||
```
|
||||
|
||||
- `pipeline/data/` — download daily bars for a universe → `data/daily_bars/{universe}/month=YYYY-MM/*.pq` (Hive-partitioned dataset; pass the `{universe}` dir as `--data-path`)
|
||||
- `pipeline/alpha/` — compute one alpha's position weights from a data parquet → `alphas/*.pq`, and `alpha eval` to score it
|
||||
- `pipeline/combo/` — combine several alpha parquets into one → `combos/*.pq`
|
||||
- `pipeline/portfolio/` — construct tradable positions from alpha/combo weights, simulate next-open fills under A-share constraints, and evaluate target-weight research metrics
|
||||
|
||||
The pipeline reuses two top-level helper modules via a `sys.path.insert` at the top of `pipeline/data/downloader.py`: `data/downloader.py` (network download) and `data/universe.py` (constituent lists). This path hack is load-bearing — keep it.
|
||||
|
||||
@@ -49,9 +53,18 @@ An **alpha** is a signed cross-sectional **position weight**: positive = long, n
|
||||
- `DATA_COLUMNS` (data output): `symbol_id, symbol_name, date, open, high, low, close, preclose, volume, amount, vwap, turn, pctChg, tradestatus, isST, peTTM, pbMRQ, psTTM, pcfNcfTTM` (`vwap` = `amount/volume` is a raw-price daily VWAP, *not* on the adjusted OHLC scale under qfq/hfq). The richer fields are fetched only by the **batch** path (`download_daily_batch` → `download_universe`); single-symbol `download_daily` keeps the legacy 8-column schema that `tests/test_downloader.py` pins.
|
||||
- `ALPHA_COLUMNS` (alpha output): `symbol_id, date, alpha_name, weight`
|
||||
- `COMBO_COLUMNS` (combo output): `symbol_id, date, combo_name, weight`
|
||||
- `POSITION_COLUMNS` (portfolio build output): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
|
||||
- `FILL_COLUMNS` (portfolio simulate fills): `symbol_id, date, portfolio_name, prev_shares, target_shares, traded_shares, realized_shares, blocked, trade_cost`
|
||||
- `PNL_COLUMNS` (portfolio simulate P&L): `date, portfolio_name, gross_exposure, net_exposure, pnl, cost, turnover, n_positions`
|
||||
|
||||
Data is stored **long/tidy**, not wide, as a Hive-partitioned dataset keyed by `month=YYYY-MM` (so reads of the dataset directory carry an extra `month` partition column, which `_pivot_close` ignores). Compute code pivots to wide (date index × symbol_id columns) internally via `_pivot_close`, where all formulas are vectorized column-wise.
|
||||
|
||||
## Portfolio construction and execution
|
||||
|
||||
`portfolio build` accepts either alpha or combo weights (`symbol_id, date, weight`) and normalizes only finite-weight names with finite positive construction prices. `target_*` columns are continuous research targets; `position_shares` is the discretized + repaired integer book. If a date has zero gross target after filtering, construction logs a warning and carries the previous `position_shares`, while target fields remain 0.
|
||||
|
||||
`portfolio simulate` must execute `position_shares`, not continuous `target_shares`. It fills at the next available open and clips desired deltas through repeatable constraints (`suspension`, `price_limit`, `volume_cap`). `portfolio eval` uses `target_weight` for a continuous research view, so zero-gross carry dates remain flat there. Keep IC/IR out of portfolio metrics too.
|
||||
|
||||
## Alphas: factory + plugin pattern
|
||||
|
||||
Each alpha is a class subclassing `BaseAlpha` (`pipeline/alpha/base.py`), living in its own module. It implements `signal(close) -> wide DataFrame` (the raw score); the base class's `to_weights` cross-sectionally z-scores that into position weights (override for custom normalization). Subclasses declare their own typed `__init__` params (e.g. `lookback`, `vol_window`, or anything custom).
|
||||
|
||||
@@ -23,12 +23,15 @@ parquet, so phases run, cache, and inspect independently.
|
||||
(DATA_COLUMNS) │ │
|
||||
└──────── price ───────┴───────────────────────────────────────┘
|
||||
│
|
||||
▼ (planned — not yet implemented)
|
||||
┌ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
|
||||
PORTFOLIO BACKTEST PAPER TRADING
|
||||
│ construct │ │ simulate │ │ forward / live │ TODO
|
||||
positions fills + costs execution
|
||||
└ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘
|
||||
▼
|
||||
┌──────────────┐ ┌──────────────┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
|
||||
│ PORTFOLIO │ │ SIMULATE │ PAPER TRADING
|
||||
│ construct │──▶│ fills + costs│ │ forward / live │ TODO
|
||||
│ positions │ │ + P&L │ execution
|
||||
└──────┬───────┘ └──────────────┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘
|
||||
▼
|
||||
portfolio/*.pq
|
||||
(POSITION_COLUMNS)
|
||||
|
||||
Each phase reads parquet and writes parquet — run, cache, and inspect
|
||||
independently. The only interface between phases is the parquet schema.
|
||||
@@ -64,8 +67,26 @@ uv run python cli.py alpha eval \
|
||||
--alpha-path alphas/reversal_5d.pq \
|
||||
--data-path "data/daily_bars/sh600000,sz000001,sh600519"
|
||||
|
||||
# 4. Build tradable integer positions from alpha or combo weights.
|
||||
uv run python cli.py portfolio build \
|
||||
--weights-path alphas/reversal_5d.pq \
|
||||
--data-path "data/daily_bars/sh600000,sz000001,sh600519" \
|
||||
--booksize 1000000 --portfolio-name reversal_port
|
||||
|
||||
# 5. Simulate next-open execution with A-share constraints, costs, and slippage.
|
||||
uv run python cli.py portfolio simulate \
|
||||
--positions-path portfolio/reversal_port.pq \
|
||||
--data-path "data/daily_bars/sh600000,sz000001,sh600519" \
|
||||
--constraint suspension --constraint price_limit --constraint volume_cap \
|
||||
--cost-bps 5 --slippage-bps 5
|
||||
|
||||
# 6. Evaluate the constructed target weights as a continuous research portfolio.
|
||||
uv run python cli.py portfolio eval \
|
||||
--positions-path portfolio/reversal_port.pq \
|
||||
--data-path "data/daily_bars/sh600000,sz000001,sh600519"
|
||||
|
||||
# Tests
|
||||
uv run python -m pytest tests/ -v # tests/test_alpha.py is network-free; test_downloader.py hits the network
|
||||
uv run python -m pytest tests/ -v # alpha/portfolio tests are network-free; downloader tests hit the network
|
||||
```
|
||||
|
||||
## CLI reference
|
||||
@@ -155,6 +176,64 @@ uv run python cli.py combo combine \
|
||||
--combo-name eq --method equal_weight
|
||||
```
|
||||
|
||||
### `portfolio build` — weights → tradable positions
|
||||
|
||||
Turns alpha/combo weights into target weights, target yuan exposure, continuous
|
||||
shares, and a lot-valid integer position book under A-share board rules.
|
||||
Non-finite / non-positive construction prices are excluded before target
|
||||
normalization. If a date has zero gross target after filtering, the previous
|
||||
book is carried in `position_shares` and a warning is logged.
|
||||
|
||||
| Option | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `--weights-path` | (required) | Alpha or combo parquet with `symbol_id, date, weight` |
|
||||
| `--data-path` | (required) | Data parquet file or partitioned dataset directory |
|
||||
| `--booksize` | (required) | Target gross yuan exposure |
|
||||
| `--portfolio-name` | (required) | Label stored in `portfolio_name` and output filename |
|
||||
| `--price-field` | `close` | Data column used as construction price |
|
||||
| `--output-dir` | `portfolio` | Output directory |
|
||||
|
||||
```bash
|
||||
uv run python cli.py portfolio build \
|
||||
--weights-path combos/eq.pq --data-path data/daily_bars/csi500 \
|
||||
--booksize 10000000 --portfolio-name eq_10m
|
||||
```
|
||||
|
||||
### `portfolio simulate` — constructed positions → fills + P&L
|
||||
|
||||
Executes the constructed `position_shares` book at the next available open,
|
||||
clipping trades through repeatable constraints. It writes `fills/<name>.pq` and
|
||||
`pnl/<name>.pq`.
|
||||
|
||||
| Option | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `--positions-path` | (required) | Positions parquet from `portfolio build` |
|
||||
| `--data-path` | (required) | Data parquet file or partitioned dataset directory |
|
||||
| `--constraint` | — | Repeatable: `suspension`, `price_limit`, `volume_cap` |
|
||||
| `--cost-bps` | `0.0` | Commission in basis points |
|
||||
| `--slippage-bps` | `0.0` | Slippage in basis points |
|
||||
| `--volume-frac` | `0.10` | Max traded value fraction for `volume_cap` |
|
||||
| `--output-dir` | `.` | Base directory for `fills/` and `pnl/` |
|
||||
|
||||
```bash
|
||||
uv run python cli.py portfolio simulate \
|
||||
--positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500 \
|
||||
--constraint suspension --constraint price_limit --constraint volume_cap \
|
||||
--cost-bps 5 --slippage-bps 5
|
||||
```
|
||||
|
||||
### `portfolio eval` — score constructed target weights
|
||||
|
||||
```bash
|
||||
uv run python cli.py portfolio eval \
|
||||
--positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500
|
||||
```
|
||||
|
||||
Uses `target_weight` for a continuous research view: cumulative return,
|
||||
annual Sharpe, annual turnover, max drawdown, Fitness, hit rate, and date count.
|
||||
There is deliberately **no IC/IR**. Zero-gross carry dates remain flat in this
|
||||
research view even though execution carries `position_shares`.
|
||||
|
||||
### `pqcat` — inspect a parquet file, like `cat`
|
||||
|
||||
Quickly dump any pipeline parquet (a single `.pq` file or a partitioned dataset
|
||||
@@ -284,6 +363,12 @@ between phases (data is stored long/tidy):
|
||||
baostock valuation ratios.)
|
||||
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
|
||||
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
|
||||
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
|
||||
(`target_*` are continuous construction targets; `position_shares` is the
|
||||
discretized + repaired integer book used by execution.)
|
||||
- **fills** (`FILL_COLUMNS`): `symbol_id, date, portfolio_name, prev_shares, target_shares, traded_shares, realized_shares, blocked, trade_cost`
|
||||
(`date` is the execution date, i.e. the next open after the target date.)
|
||||
- **pnl** (`PNL_COLUMNS`): `date, portfolio_name, gross_exposure, net_exposure, pnl, cost, turnover, n_positions`
|
||||
|
||||
The data phase writes a month-partitioned dataset, so reading the dataset
|
||||
directory yields an extra `month` (`YYYY-MM`) partition column on top of
|
||||
@@ -291,36 +376,34 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
|
||||
|
||||
## Layout
|
||||
|
||||
- `cli.py` — entry point wiring the three phases together
|
||||
- `cli.py` — entry point wiring the file-based phases together
|
||||
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
|
||||
- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
|
||||
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
|
||||
- `pipeline/combo/` — alpha combination → `combos/*.pq`
|
||||
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
|
||||
reference next-open simulator, and research metrics
|
||||
- `pipeline/common/schema.py` — parquet column contracts
|
||||
- `data/downloader.py`, `data/universe.py` — baostock/akshare download + constituents
|
||||
- `tools/pqcat.py` — standalone parquet inspector (`pqcat`), also wired as `cli.py pqcat`
|
||||
- `tools/alphaview.py` — standalone alpha-vs-bar viewer (`alphaview`), also wired as `cli.py alphaview`
|
||||
- `examples/alphas/` — example external alpha(s)
|
||||
|
||||
## Roadmap (not yet implemented)
|
||||
## Roadmap / current limits
|
||||
|
||||
The pipeline currently ends at `combo`, and `alpha eval` only interprets a weight
|
||||
series as a portfolio for quick scoring (return / Sharpe / turnover / drawdown).
|
||||
It is **not** a true backtest — there is no transaction-cost, slippage, or
|
||||
execution modeling. The following phases are planned but not built yet:
|
||||
The pipeline is implemented through portfolio construction and a reference
|
||||
daily execution simulator. `alpha eval` remains a fast sanity check on raw
|
||||
weights; use `portfolio build`, `portfolio simulate`, and `portfolio eval` for
|
||||
constructed positions, fills/costs, P&L, and target-weight research metrics.
|
||||
|
||||
- [ ] **Portfolio construction** — turn combo weights into target positions
|
||||
(gross/net exposure caps, per-name and sector limits, capital allocation,
|
||||
rebalance schedule).
|
||||
- [ ] **Backtesting** — event-driven simulation over the constructed positions
|
||||
with realistic fills, transaction costs, slippage, and borrow constraints;
|
||||
richer P&L / risk attribution than `alpha eval`.
|
||||
- [x] **Portfolio construction** — turn alpha/combo weights into continuous
|
||||
targets and lot-valid integer positions under A-share board rules.
|
||||
- [x] **Reference execution simulation** — next-open fills over constructed
|
||||
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
|
||||
and slippage controls.
|
||||
- [ ] **Forward / paper trading** — run the same construction logic on live
|
||||
daily data, track simulated fills and a running P&L without real capital.
|
||||
- [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price,
|
||||
and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
|
||||
brokerage data tier); the free daily sources here only expose daily bars, so
|
||||
this is a separate data phase rather than extra columns on the daily schema.
|
||||
|
||||
Until these land, treat `alpha eval` as a fast sanity check on a weight series,
|
||||
not a performance estimate.
|
||||
|
||||
Reference in New Issue
Block a user