Add daily derived data pipeline

README.md

@@ -120,6 +120,49 @@ partitions already written. Pass the **dataset directory** (`{output_dir}/{unive
 as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
 set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
 
+### `derived` — daily custom/derived data
+
+Derived data is daily-only v1 research data keyed by `symbol_id,date`, with one
+or more numeric value columns. It can come from user CSV/parquet files or Python
+plugins, and is written as a single parquet file at `derived/{name}.pq`.
+
+The validator normalizes `date` to the trading day, requires unique
+`symbol_id,date` keys, rejects duplicate columns, and rejects non-numeric value
+columns. Alpha computation consumes derived data through the existing
+`--feature-path` flag.
+
+```bash
+# Validate a user file without writing output.
+uv run python cli.py derived validate --input-path vendor_factor.csv
+
+# Ingest CSV/parquet into the canonical derived/ layout.
+uv run python cli.py derived ingest \
+    --input-path vendor_factor.csv \
+    --derived-name vendor_factor
+
+# List built-in and external derived-data plugin types.
+uv run python cli.py derived list
+uv run python cli.py derived list --derived-module path/to/my_derived.py
+
+# Compute a derived file from daily and/or minute inputs.
+uv run python cli.py derived compute \
+    --minute-path data/minute_bars/sh600000 \
+    --daily-path data/daily_bars/sh600000 \
+    --derived-type minute_daily_summary \
+    --derived-name minute_summary
+
+# Join derived columns into a feature-aware alpha.
+uv run python cli.py alpha compute \
+    --data-path data/daily_bars/sh600000 \
+    --feature-path derived/minute_summary.pq \
+    --alpha-type my_feature_aware_alpha \
+    --alpha-name my_run
+```
+
+For compatibility, `feature list` and `feature compute` remain available and
+delegate to the same derived-data registry. Existing `features/*.pq` files are
+still valid `--feature-path` inputs when they satisfy the daily numeric contract.
+
 ### `alpha list` — show registered alpha types
 
 ```bash
@@ -137,6 +180,7 @@ uv run python cli.py alpha list --alpha-module path/to/my_alpha.py   # include a
 | `--output-dir` | `alphas` | Output directory |
 | `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
 | `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
+| `--feature-path` | — | Daily derived/feature parquet file or dataset to left-join on `symbol_id,date`; repeatable |
 | `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
 | `--param` | — | Extra constructor param as `name=value`; repeatable |
 
@@ -370,6 +414,8 @@ between phases (data is stored long/tidy):
   OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
   `tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
   baostock valuation ratios.)
+- **derived** (`DERIVED_KEY_COLUMNS` + values): required keys `symbol_id, date`;
+  value columns are user/plugin-defined and must be numeric in v1.
 - **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
 - **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
 - **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
@@ -387,8 +433,11 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
 
 - `cli.py` — entry point wiring the file-based phases together
 - `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
+- `pipeline/derived/` — daily derived-data ingestion, validation, plugin registry,
+  and built-in derived computations → `derived/*.pq`
 - `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
   `library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
+- `pipeline/features/` — compatibility wrappers for the derived-data registry
 - `pipeline/combo/` — alpha combination → `combos/*.pq`
 - `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
   reference next-open simulator, and research metrics
@@ -410,6 +459,9 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
 - [x] **Reference execution simulation** — next-open fills over constructed
   `position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
   and slippage controls.
+- [x] **Derived/custom daily data ("Level 2")** — ingest user CSV/parquet files
+  or compute plugin outputs as validated numeric daily datasets under
+  `derived/{name}.pq`; alpha joins continue through `--feature-path`.
 - [ ] **Optional Backtrader adapter** — Backtrader is available as the
   `backtrader` extra for possible future event-driven/broker-style experiments,
   but it is not part of the current canonical portfolio workflow.
@@ -419,3 +471,29 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
   and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
   brokerage data tier); the free daily sources here only expose daily bars, so
   this is a separate data phase rather than extra columns on the daily schema.
+
+### Additional TODOs
+
+The following items are intended extensions beyond the current daily
+alpha-to-portfolio pipeline:
+
+- **Long-only portfolio mode** — add a construction option that converts
+  alpha/combo weights into a long-only book while preserving existing lot,
+  price, suspension, and volume-cap handling.
+- **Index-short hedging mode** — support portfolios that hold long A-share
+  names while shorting an index or index proxy for market exposure control.
+- **Expanded universe presets** — add explicit universe aliases for CSI 300,
+  CSI 500, CSI 1000, and CSI 1800, while keeping file-based and comma-separated
+  custom universes available.
+- **Categorical derived data** — extend the numeric-only derived-data v1 contract
+  to support categorical inputs such as industry classifications. In this
+  project, "Level 2" means customized second-level research data produced by
+  users or plugins; it does not necessarily mean exchange order-book/L2 quote
+  feeds.
+- **Minute bar data** — continue extending the raw minute-bar and feature
+  workflow. The initial Baostock 5-minute download and daily feature plugin path
+  exist; intraday execution and replacing canonical daily bars remain out of
+  scope unless explicitly added later.
+- **Industry data** — add industry classification inputs for filtering,
+  grouping, exposure reporting, neutralization, or industry-aware portfolio
+  construction.