Document implemented portfolio workflow

CLAUDE.md

@@ -19,23 +19,27 @@ uv run python cli.py data download --universe csi500 --start-date 2017-01-01   #
 uv run python cli.py alpha reversal --data-path data/daily_bars/<universe>     # --data-path is the dataset DIR
 uv run python cli.py alpha eval --alpha-path alphas/<file>.pq --data-path data/daily_bars/<universe>
 uv run python cli.py combo combine --alpha-paths a.pq,b.pq --combo-name eq --method equal_weight
+uv run python cli.py portfolio build --weights-path combos/eq.pq --data-path data/daily_bars/<universe> --booksize 10000000 --portfolio-name eq_10m
+uv run python cli.py portfolio simulate --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/<universe> --constraint suspension --constraint price_limit --constraint volume_cap --cost-bps 5 --slippage-bps 5
+uv run python cli.py portfolio eval --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/<universe>
 ```
 
 Add a runtime dep with `uv add <pkg>`, a dev/test dep with `uv add --dev <pkg>` (both update `pyproject.toml` + `uv.lock`).
 
-Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` is pure and fast.
+Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` and `tests/test_portfolio.py` are pure and fast.
 
 ## Architecture: one decoupled pipeline
 
 The system is a phase-based CLI (`cli.py` + `pipeline/`). Each phase communicates **only** through parquet files on disk, so phases can be run, cached, and inspected independently:
 
 ```
-data  →  alpha  →  combo
+data  →  alpha  →  combo  →  portfolio build  →  portfolio simulate/eval
 ```
 
 - `pipeline/data/` — download daily bars for a universe → `data/daily_bars/{universe}/month=YYYY-MM/*.pq` (Hive-partitioned dataset; pass the `{universe}` dir as `--data-path`)
 - `pipeline/alpha/` — compute one alpha's position weights from a data parquet → `alphas/*.pq`, and `alpha eval` to score it
 - `pipeline/combo/` — combine several alpha parquets into one → `combos/*.pq`
+- `pipeline/portfolio/` — construct tradable positions from alpha/combo weights, simulate next-open fills under A-share constraints, and evaluate target-weight research metrics
 
 The pipeline reuses two top-level helper modules via a `sys.path.insert` at the top of `pipeline/data/downloader.py`: `data/downloader.py` (network download) and `data/universe.py` (constituent lists). This path hack is load-bearing — keep it.
 
@@ -49,9 +53,18 @@ An **alpha** is a signed cross-sectional **position weight**: positive = long, n
 - `DATA_COLUMNS` (data output): `symbol_id, symbol_name, date, open, high, low, close, preclose, volume, amount, vwap, turn, pctChg, tradestatus, isST, peTTM, pbMRQ, psTTM, pcfNcfTTM` (`vwap` = `amount/volume` is a raw-price daily VWAP, *not* on the adjusted OHLC scale under qfq/hfq). The richer fields are fetched only by the **batch** path (`download_daily_batch` → `download_universe`); single-symbol `download_daily` keeps the legacy 8-column schema that `tests/test_downloader.py` pins.
 - `ALPHA_COLUMNS` (alpha output): `symbol_id, date, alpha_name, weight`
 - `COMBO_COLUMNS` (combo output): `symbol_id, date, combo_name, weight`
+- `POSITION_COLUMNS` (portfolio build output): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
+- `FILL_COLUMNS` (portfolio simulate fills): `symbol_id, date, portfolio_name, prev_shares, target_shares, traded_shares, realized_shares, blocked, trade_cost`
+- `PNL_COLUMNS` (portfolio simulate P&L): `date, portfolio_name, gross_exposure, net_exposure, pnl, cost, turnover, n_positions`
 
 Data is stored **long/tidy**, not wide, as a Hive-partitioned dataset keyed by `month=YYYY-MM` (so reads of the dataset directory carry an extra `month` partition column, which `_pivot_close` ignores). Compute code pivots to wide (date index × symbol_id columns) internally via `_pivot_close`, where all formulas are vectorized column-wise.
 
+## Portfolio construction and execution
+
+`portfolio build` accepts either alpha or combo weights (`symbol_id, date, weight`) and normalizes only finite-weight names with finite positive construction prices. `target_*` columns are continuous research targets; `position_shares` is the discretized + repaired integer book. If a date has zero gross target after filtering, construction logs a warning and carries the previous `position_shares`, while target fields remain 0.
+
+`portfolio simulate` must execute `position_shares`, not continuous `target_shares`. It fills at the next available open and clips desired deltas through repeatable constraints (`suspension`, `price_limit`, `volume_cap`). `portfolio eval` uses `target_weight` for a continuous research view, so zero-gross carry dates remain flat there. Keep IC/IR out of portfolio metrics too.
+
 ## Alphas: factory + plugin pattern
 
 Each alpha is a class subclassing `BaseAlpha` (`pipeline/alpha/base.py`), living in its own module. It implements `signal(close) -> wide DataFrame` (the raw score); the base class's `to_weights` cross-sectionally z-scores that into position weights (override for custom normalization). Subclasses declare their own typed `__init__` params (e.g. `lookback`, `vol_window`, or anything custom).

README.md

@@ -23,12 +23,15 @@ parquet, so phases run, cache, and inspect independently.
  (DATA_COLUMNS)                 │                                      │
         └──────── price ───────┴───────────────────────────────────────┘
                                        │
-                                       ▼  (planned — not yet implemented)
+                                       ▼
-              ┌ ─ ─ ─ ─ ─ ─ ┐   ┌ ─ ─ ─ ─ ─ ─ ┐   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
+              ┌──────────────┐   ┌──────────────┐   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐
-                 PORTFOLIO         BACKTEST           PAPER TRADING
+              │  PORTFOLIO   │   │   SIMULATE   │      PAPER TRADING
-              │  construct  │   │   simulate  │   │  forward / live  │  TODO
+              │  construct   │──▶│ fills + costs│   │  forward / live  │  TODO
-                 positions       fills + costs      execution
+              │  positions   │   │   + P&L      │      execution
-              └ ─ ─ ─ ─ ─ ─ ┘   └ ─ ─ ─ ─ ─ ─ ┘   └ ─ ─ ─ ─ ─ ─ ─ ─ ┘
+              └──────┬───────┘   └──────────────┘   └ ─ ─ ─ ─ ─ ─ ─ ─ ┘
+                     ▼
+              portfolio/*.pq
+              (POSITION_COLUMNS)
 
  Each phase reads parquet and writes parquet — run, cache, and inspect
  independently. The only interface between phases is the parquet schema.
@@ -64,8 +67,26 @@ uv run python cli.py alpha eval \
     --alpha-path alphas/reversal_5d.pq \
     --data-path  "data/daily_bars/sh600000,sz000001,sh600519"
 
+# 4. Build tradable integer positions from alpha or combo weights.
+uv run python cli.py portfolio build \
+    --weights-path alphas/reversal_5d.pq \
+    --data-path "data/daily_bars/sh600000,sz000001,sh600519" \
+    --booksize 1000000 --portfolio-name reversal_port
+
+# 5. Simulate next-open execution with A-share constraints, costs, and slippage.
+uv run python cli.py portfolio simulate \
+    --positions-path portfolio/reversal_port.pq \
+    --data-path "data/daily_bars/sh600000,sz000001,sh600519" \
+    --constraint suspension --constraint price_limit --constraint volume_cap \
+    --cost-bps 5 --slippage-bps 5
+
+# 6. Evaluate the constructed target weights as a continuous research portfolio.
+uv run python cli.py portfolio eval \
+    --positions-path portfolio/reversal_port.pq \
+    --data-path "data/daily_bars/sh600000,sz000001,sh600519"
+
 # Tests
-uv run python -m pytest tests/ -v        # tests/test_alpha.py is network-free; test_downloader.py hits the network
+uv run python -m pytest tests/ -v        # alpha/portfolio tests are network-free; downloader tests hit the network
 ```
 
 ## CLI reference
@@ -155,6 +176,64 @@ uv run python cli.py combo combine \
     --combo-name eq --method equal_weight
 ```
 
+### `portfolio build` — weights → tradable positions
+
+Turns alpha/combo weights into target weights, target yuan exposure, continuous
+shares, and a lot-valid integer position book under A-share board rules.
+Non-finite / non-positive construction prices are excluded before target
+normalization. If a date has zero gross target after filtering, the previous
+book is carried in `position_shares` and a warning is logged.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `--weights-path` | (required) | Alpha or combo parquet with `symbol_id, date, weight` |
+| `--data-path` | (required) | Data parquet file or partitioned dataset directory |
+| `--booksize` | (required) | Target gross yuan exposure |
+| `--portfolio-name` | (required) | Label stored in `portfolio_name` and output filename |
+| `--price-field` | `close` | Data column used as construction price |
+| `--output-dir` | `portfolio` | Output directory |
+
+```bash
+uv run python cli.py portfolio build \
+    --weights-path combos/eq.pq --data-path data/daily_bars/csi500 \
+    --booksize 10000000 --portfolio-name eq_10m
+```
+
+### `portfolio simulate` — constructed positions → fills + P&L
+
+Executes the constructed `position_shares` book at the next available open,
+clipping trades through repeatable constraints. It writes `fills/<name>.pq` and
+`pnl/<name>.pq`.
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `--positions-path` | (required) | Positions parquet from `portfolio build` |
+| `--data-path` | (required) | Data parquet file or partitioned dataset directory |
+| `--constraint` | — | Repeatable: `suspension`, `price_limit`, `volume_cap` |
+| `--cost-bps` | `0.0` | Commission in basis points |
+| `--slippage-bps` | `0.0` | Slippage in basis points |
+| `--volume-frac` | `0.10` | Max traded value fraction for `volume_cap` |
+| `--output-dir` | `.` | Base directory for `fills/` and `pnl/` |
+
+```bash
+uv run python cli.py portfolio simulate \
+    --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500 \
+    --constraint suspension --constraint price_limit --constraint volume_cap \
+    --cost-bps 5 --slippage-bps 5
+```
+
+### `portfolio eval` — score constructed target weights
+
+```bash
+uv run python cli.py portfolio eval \
+    --positions-path portfolio/eq_10m.pq --data-path data/daily_bars/csi500
+```
+
+Uses `target_weight` for a continuous research view: cumulative return,
+annual Sharpe, annual turnover, max drawdown, Fitness, hit rate, and date count.
+There is deliberately **no IC/IR**. Zero-gross carry dates remain flat in this
+research view even though execution carries `position_shares`.
+
 ### `pqcat` — inspect a parquet file, like `cat`
 
 Quickly dump any pipeline parquet (a single `.pq` file or a partitioned dataset
@@ -284,6 +363,12 @@ between phases (data is stored long/tidy):
   baostock valuation ratios.)
 - **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
 - **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
+- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
+  (`target_*` are continuous construction targets; `position_shares` is the
+  discretized + repaired integer book used by execution.)
+- **fills** (`FILL_COLUMNS`): `symbol_id, date, portfolio_name, prev_shares, target_shares, traded_shares, realized_shares, blocked, trade_cost`
+  (`date` is the execution date, i.e. the next open after the target date.)
+- **pnl** (`PNL_COLUMNS`): `date, portfolio_name, gross_exposure, net_exposure, pnl, cost, turnover, n_positions`
 
 The data phase writes a month-partitioned dataset, so reading the dataset
 directory yields an extra `month` (`YYYY-MM`) partition column on top of
@@ -291,36 +376,34 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
 
 ## Layout
 
-- `cli.py` — entry point wiring the three phases together
+- `cli.py` — entry point wiring the file-based phases together
 - `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
 - `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
   `library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
 - `pipeline/combo/` — alpha combination → `combos/*.pq`
+- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
+  reference next-open simulator, and research metrics
 - `pipeline/common/schema.py` — parquet column contracts
 - `data/downloader.py`, `data/universe.py` — baostock/akshare download + constituents
 - `tools/pqcat.py` — standalone parquet inspector (`pqcat`), also wired as `cli.py pqcat`
 - `tools/alphaview.py` — standalone alpha-vs-bar viewer (`alphaview`), also wired as `cli.py alphaview`
 - `examples/alphas/` — example external alpha(s)
 
-## Roadmap (not yet implemented)
+## Roadmap / current limits
 
-The pipeline currently ends at `combo`, and `alpha eval` only interprets a weight
+The pipeline is implemented through portfolio construction and a reference
-series as a portfolio for quick scoring (return / Sharpe / turnover / drawdown).
+daily execution simulator. `alpha eval` remains a fast sanity check on raw
-It is **not** a true backtest — there is no transaction-cost, slippage, or
+weights; use `portfolio build`, `portfolio simulate`, and `portfolio eval` for
-execution modeling. The following phases are planned but not built yet:
+constructed positions, fills/costs, P&L, and target-weight research metrics.
 
-- [ ] **Portfolio construction** — turn combo weights into target positions
+- [x] **Portfolio construction** — turn alpha/combo weights into continuous
-  (gross/net exposure caps, per-name and sector limits, capital allocation,
+  targets and lot-valid integer positions under A-share board rules.
-  rebalance schedule).
+- [x] **Reference execution simulation** — next-open fills over constructed
-- [ ] **Backtesting** — event-driven simulation over the constructed positions
+  `position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
-  with realistic fills, transaction costs, slippage, and borrow constraints;
+  and slippage controls.
-  richer P&L / risk attribution than `alpha eval`.
 - [ ] **Forward / paper trading** — run the same construction logic on live
   daily data, track simulated fills and a running P&L without real capital.
 - [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price,
   and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
   brokerage data tier); the free daily sources here only expose daily bars, so
   this is a separate data phase rather than extra columns on the daily schema.
-
-Until these land, treat `alpha eval` as a fast sanity check on a weight series,
-not a performance estimate.