From 459336b6cc3576008cfbb9ccf87e4b9d444e691c Mon Sep 17 00:00:00 2001 From: Yuxuan Yan Date: Wed, 10 Jun 2026 15:04:34 +0800 Subject: [PATCH] Document implemented portfolio workflow --- CLAUDE.md | 17 +++++++- README.md | 127 ++++++++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 120 insertions(+), 24 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 4000555..03d91dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,23 +19,27 @@ uv run python cli.py data download --universe csi500 --start-date 2017-01-01 # uv run python cli.py alpha reversal --data-path data/daily_bars/ # --data-path is the dataset DIR uv run python cli.py alpha eval --alpha-path alphas/.pq --data-path data/daily_bars/ uv run python cli.py combo combine --alpha-paths a.pq,b.pq --combo-name eq --method equal_weight +uv run python cli.py portfolio build --weights-path combos/eq.pq --data-path data/daily_bars/ --booksize 10000000 --portfolio-name eq_10m +uv run python cli.py portfolio simulate --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/ --constraint suspension --constraint price_limit --constraint volume_cap --cost-bps 5 --slippage-bps 5 +uv run python cli.py portfolio eval --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/ ``` Add a runtime dep with `uv add `, a dev/test dep with `uv add --dev ` (both update `pyproject.toml` + `uv.lock`). -Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` is pure and fast. +Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` and `tests/test_portfolio.py` are pure and fast. ## Architecture: one decoupled pipeline The system is a phase-based CLI (`cli.py` + `pipeline/`). Each phase communicates **only** through parquet files on disk, so phases can be run, cached, and inspected independently: ``` -data → alpha → combo +data → alpha → combo → portfolio build → portfolio simulate/eval ``` - `pipeline/data/` — download daily bars for a universe → `data/daily_bars/{universe}/month=YYYY-MM/*.pq` (Hive-partitioned dataset; pass the `{universe}` dir as `--data-path`) - `pipeline/alpha/` — compute one alpha's position weights from a data parquet → `alphas/*.pq`, and `alpha eval` to score it - `pipeline/combo/` — combine several alpha parquets into one → `combos/*.pq` +- `pipeline/portfolio/` — construct tradable positions from alpha/combo weights, simulate next-open fills under A-share constraints, and evaluate target-weight research metrics The pipeline reuses two top-level helper modules via a `sys.path.insert` at the top of `pipeline/data/downloader.py`: `data/downloader.py` (network download) and `data/universe.py` (constituent lists). This path hack is load-bearing — keep it. @@ -49,9 +53,18 @@ An **alpha** is a signed cross-sectional **position weight**: positive = long, n - `DATA_COLUMNS` (data output): `symbol_id, symbol_name, date, open, high, low, close, preclose, volume, amount, vwap, turn, pctChg, tradestatus, isST, peTTM, pbMRQ, psTTM, pcfNcfTTM` (`vwap` = `amount/volume` is a raw-price daily VWAP, *not* on the adjusted OHLC scale under qfq/hfq). The richer fields are fetched only by the **batch** path (`download_daily_batch` → `download_universe`); single-symbol `download_daily` keeps the legacy 8-column schema that `tests/test_downloader.py` pins. - `ALPHA_COLUMNS` (alpha output): `symbol_id, date, alpha_name, weight` - `COMBO_COLUMNS` (combo output): `symbol_id, date, combo_name, weight` +- `POSITION_COLUMNS` (portfolio build output): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price` +- `FILL_COLUMNS` (portfolio simulate fills): `symbol_id, date, portfolio_name, prev_shares, target_shares, traded_shares, realized_shares, blocked, trade_cost` +- `PNL_COLUMNS` (portfolio simulate P&L): `date, portfolio_name, gross_exposure, net_exposure, pnl, cost, turnover, n_positions` Data is stored **long/tidy**, not wide, as a Hive-partitioned dataset keyed by `month=YYYY-MM` (so reads of the dataset directory carry an extra `month` partition column, which `_pivot_close` ignores). Compute code pivots to wide (date index × symbol_id columns) internally via `_pivot_close`, where all formulas are vectorized column-wise. +## Portfolio construction and execution + +`portfolio build` accepts either alpha or combo weights (`symbol_id, date, weight`) and normalizes only finite-weight names with finite positive construction prices. `target_*` columns are continuous research targets; `position_shares` is the discretized + repaired integer book. If a date has zero gross target after filtering, construction logs a warning and carries the previous `position_shares`, while target fields remain 0. + +`portfolio simulate` must execute `position_shares`, not continuous `target_shares`. It fills at the next available open and clips desired deltas through repeatable constraints (`suspension`, `price_limit`, `volume_cap`). `portfolio eval` uses `target_weight` for a continuous research view, so zero-gross carry dates remain flat there. Keep IC/IR out of portfolio metrics too. + ## Alphas: factory + plugin pattern Each alpha is a class subclassing `BaseAlpha` (`pipeline/alpha/base.py`), living in its own module. It implements `signal(close) -> wide DataFrame` (the raw score); the base class's `to_weights` cross-sectionally z-scores that into position weights (override for custom normalization). Subclasses declare their own typed `__init__` params (e.g. `lookback`, `vol_window`, or anything custom). diff --git a/README.md b/README.md index 9b3e716..892314b 100644 --- a/README.md +++ b/README.md @@ -23,12 +23,15 @@ parquet, so phases run, cache, and inspect independently. (DATA_COLUMNS) │ │ └──────── price ───────┴───────────────────────────────────────┘ │ - ▼ (planned — not yet implemented) - ┌ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐ - PORTFOLIO BACKTEST PAPER TRADING - │ construct │ │ simulate │ │ forward / live │ TODO - positions fills + costs execution - └ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘ + ▼ + ┌──────────────┐ ┌──────────────┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + │ PORTFOLIO │ │ SIMULATE │ PAPER TRADING + │ construct │──▶│ fills + costs│ │ forward / live │ TODO + │ positions │ │ + P&L │ execution + └──────┬───────┘ └──────────────┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘ + ▼ + portfolio/*.pq + (POSITION_COLUMNS) Each phase reads parquet and writes parquet — run, cache, and inspect independently. The only interface between phases is the parquet schema. @@ -64,8 +67,26 @@ uv run python cli.py alpha eval \ --alpha-path alphas/reversal_5d.pq \ --data-path "data/daily_bars/sh600000,sz000001,sh600519" +# 4. Build tradable integer positions from alpha or combo weights. +uv run python cli.py portfolio build \ + --weights-path alphas/reversal_5d.pq \ + --data-path "data/daily_bars/sh600000,sz000001,sh600519" \ + --booksize 1000000 --portfolio-name reversal_port + +# 5. Simulate next-open execution with A-share constraints, costs, and slippage. +uv run python cli.py portfolio simulate \ + --positions-path portfolio/reversal_port.pq \ + --data-path "data/daily_bars/sh600000,sz000001,sh600519" \ + --constraint suspension --constraint price_limit --constraint volume_cap \ + --cost-bps 5 --slippage-bps 5 + +# 6. Evaluate the constructed target weights as a continuous research portfolio. +uv run python cli.py portfolio eval \ + --positions-path portfolio/reversal_port.pq \ + --data-path "data/daily_bars/sh600000,sz000001,sh600519" + # Tests -uv run python -m pytest tests/ -v # tests/test_alpha.py is network-free; test_downloader.py hits the network +uv run python -m pytest tests/ -v # alpha/portfolio tests are network-free; downloader tests hit the network ``` ## CLI reference @@ -155,6 +176,64 @@ uv run python cli.py combo combine \ --combo-name eq --method equal_weight ``` +### `portfolio build` — weights → tradable positions + +Turns alpha/combo weights into target weights, target yuan exposure, continuous +shares, and a lot-valid integer position book under A-share board rules. +Non-finite / non-positive construction prices are excluded before target +normalization. If a date has zero gross target after filtering, the previous +book is carried in `position_shares` and a warning is logged. + +| Option | Default | Description | +| --- | --- | --- | +| `--weights-path` | (required) | Alpha or combo parquet with `symbol_id, date, weight` | +| `--data-path` | (required) | Data parquet file or partitioned dataset directory | +| `--booksize` | (required) | Target gross yuan exposure | +| `--portfolio-name` | (required) | Label stored in `portfolio_name` and output filename | +| `--price-field` | `close` | Data column used as construction price | +| `--output-dir` | `portfolio` | Output directory | + +```bash +uv run python cli.py portfolio build \ + --weights-path combos/eq.pq --data-path data/daily_bars/csi500 \ + --booksize 10000000 --portfolio-name eq_10m +``` + +### `portfolio simulate` — constructed positions → fills + P&L + +Executes the constructed `position_shares` book at the next available open, +clipping trades through repeatable constraints. It writes `fills/.pq` and +`pnl/.pq`. + +| Option | Default | Description | +| --- | --- | --- | +| `--positions-path` | (required) | Positions parquet from `portfolio build` | +| `--data-path` | (required) | Data parquet file or partitioned dataset directory | +| `--constraint` | — | Repeatable: `suspension`, `price_limit`, `volume_cap` | +| `--cost-bps` | `0.0` | Commission in basis points | +| `--slippage-bps` | `0.0` | Slippage in basis points | +| `--volume-frac` | `0.10` | Max traded value fraction for `volume_cap` | +| `--output-dir` | `.` | Base directory for `fills/` and `pnl/` | + +```bash +uv run python cli.py portfolio simulate \ + --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500 \ + --constraint suspension --constraint price_limit --constraint volume_cap \ + --cost-bps 5 --slippage-bps 5 +``` + +### `portfolio eval` — score constructed target weights + +```bash +uv run python cli.py portfolio eval \ + --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500 +``` + +Uses `target_weight` for a continuous research view: cumulative return, +annual Sharpe, annual turnover, max drawdown, Fitness, hit rate, and date count. +There is deliberately **no IC/IR**. Zero-gross carry dates remain flat in this +research view even though execution carries `position_shares`. + ### `pqcat` — inspect a parquet file, like `cat` Quickly dump any pipeline parquet (a single `.pq` file or a partitioned dataset @@ -284,6 +363,12 @@ between phases (data is stored long/tidy): baostock valuation ratios.) - **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight` - **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight` +- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price` + (`target_*` are continuous construction targets; `position_shares` is the + discretized + repaired integer book used by execution.) +- **fills** (`FILL_COLUMNS`): `symbol_id, date, portfolio_name, prev_shares, target_shares, traded_shares, realized_shares, blocked, trade_cost` + (`date` is the execution date, i.e. the next open after the target date.) +- **pnl** (`PNL_COLUMNS`): `date, portfolio_name, gross_exposure, net_exposure, pnl, cost, turnover, n_positions` The data phase writes a month-partitioned dataset, so reading the dataset directory yields an extra `month` (`YYYY-MM`) partition column on top of @@ -291,36 +376,34 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of ## Layout -- `cli.py` — entry point wiring the three phases together +- `cli.py` — entry point wiring the file-based phases together - `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq` - `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader), `library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`) - `pipeline/combo/` — alpha combination → `combos/*.pq` +- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints, + reference next-open simulator, and research metrics - `pipeline/common/schema.py` — parquet column contracts - `data/downloader.py`, `data/universe.py` — baostock/akshare download + constituents - `tools/pqcat.py` — standalone parquet inspector (`pqcat`), also wired as `cli.py pqcat` - `tools/alphaview.py` — standalone alpha-vs-bar viewer (`alphaview`), also wired as `cli.py alphaview` - `examples/alphas/` — example external alpha(s) -## Roadmap (not yet implemented) +## Roadmap / current limits -The pipeline currently ends at `combo`, and `alpha eval` only interprets a weight -series as a portfolio for quick scoring (return / Sharpe / turnover / drawdown). -It is **not** a true backtest — there is no transaction-cost, slippage, or -execution modeling. The following phases are planned but not built yet: +The pipeline is implemented through portfolio construction and a reference +daily execution simulator. `alpha eval` remains a fast sanity check on raw +weights; use `portfolio build`, `portfolio simulate`, and `portfolio eval` for +constructed positions, fills/costs, P&L, and target-weight research metrics. -- [ ] **Portfolio construction** — turn combo weights into target positions - (gross/net exposure caps, per-name and sector limits, capital allocation, - rebalance schedule). -- [ ] **Backtesting** — event-driven simulation over the constructed positions - with realistic fills, transaction costs, slippage, and borrow constraints; - richer P&L / risk attribution than `alpha eval`. +- [x] **Portfolio construction** — turn alpha/combo weights into continuous + targets and lot-valid integer positions under A-share board rules. +- [x] **Reference execution simulation** — next-open fills over constructed + `position_shares`, with suspension, price-limit, volume-cap, transaction-cost, + and slippage controls. - [ ] **Forward / paper trading** — run the same construction logic on live daily data, track simulated fills and a running P&L without real capital. - [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price, and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or brokerage data tier); the free daily sources here only expose daily bars, so this is a separate data phase rather than extra columns on the daily schema. - -Until these land, treat `alpha eval` as a fast sanity check on a weight series, -not a performance estimate.