Compare commits
23 Commits
459336b6cc
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c91415aff8 | |||
| efa9d24c73 | |||
| 5fada75d23 | |||
| d05931f41a | |||
| 39a93259e1 | |||
| 5388359dc8 | |||
| c200508f9e | |||
| f25db279bf | |||
| 528620b271 | |||
| b5c8c0b8da | |||
| 31baa18ce5 | |||
| 8d908477e2 | |||
| 83a006bbe4 | |||
| 17fa75495d | |||
| 3c58a1372e | |||
| 16b4988f16 | |||
| 2c0ca53bd6 | |||
| 2ceac82325 | |||
| b7dd94b032 | |||
| 07ed6ad917 | |||
| 0a6f367fbf | |||
| 534b91aaa4 | |||
| 4a477b8f75 |
@@ -1,6 +1,7 @@
|
|||||||
__pycache__/
|
__pycache__/
|
||||||
*.py[cod]
|
*.py[cod]
|
||||||
.pytest_cache/
|
.pytest_cache/
|
||||||
|
.coverage
|
||||||
*.egg-info/
|
*.egg-info/
|
||||||
.venv/
|
.venv/
|
||||||
venv/
|
venv/
|
||||||
|
|||||||
@@ -26,6 +26,10 @@ uv run python cli.py portfolio eval --positions-path portfolio/eq_10m.pq --data-
|
|||||||
|
|
||||||
Add a runtime dep with `uv add <pkg>`, a dev/test dep with `uv add --dev <pkg>` (both update `pyproject.toml` + `uv.lock`).
|
Add a runtime dep with `uv add <pkg>`, a dev/test dep with `uv add --dev <pkg>` (both update `pyproject.toml` + `uv.lock`).
|
||||||
|
|
||||||
|
Backtrader is optional (`uv sync --extra backtrader`) and is not used by the
|
||||||
|
current pipeline. Keep `portfolio simulate` as the canonical backtest/execution
|
||||||
|
path unless an explicit future adapter is requested.
|
||||||
|
|
||||||
Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` and `tests/test_portfolio.py` are pure and fast.
|
Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` and `tests/test_portfolio.py` are pure and fast.
|
||||||
|
|
||||||
## Architecture: one decoupled pipeline
|
## Architecture: one decoupled pipeline
|
||||||
@@ -65,6 +69,8 @@ Data is stored **long/tidy**, not wide, as a Hive-partitioned dataset keyed by `
|
|||||||
|
|
||||||
`portfolio simulate` must execute `position_shares`, not continuous `target_shares`. It fills at the next available open and clips desired deltas through repeatable constraints (`suspension`, `price_limit`, `volume_cap`). `portfolio eval` uses `target_weight` for a continuous research view, so zero-gross carry dates remain flat there. Keep IC/IR out of portfolio metrics too.
|
`portfolio simulate` must execute `position_shares`, not continuous `target_shares`. It fills at the next available open and clips desired deltas through repeatable constraints (`suspension`, `price_limit`, `volume_cap`). `portfolio eval` uses `target_weight` for a continuous research view, so zero-gross carry dates remain flat there. Keep IC/IR out of portfolio metrics too.
|
||||||
|
|
||||||
|
Trading cost uses the simplified open-execution proportional cash-cost model in `docs/portfolio_trading_cost_model.md`: `abs(traded_shares * open) * (cost_bps + slippage_bps) / 10000`. Slippage is cash cost only; do not also adjust execution prices for slippage.
|
||||||
|
|
||||||
## Alphas: factory + plugin pattern
|
## Alphas: factory + plugin pattern
|
||||||
|
|
||||||
Each alpha is a class subclassing `BaseAlpha` (`pipeline/alpha/base.py`), living in its own module. It implements `signal(close) -> wide DataFrame` (the raw score); the base class's `to_weights` cross-sectionally z-scores that into position weights (override for custom normalization). Subclasses declare their own typed `__init__` params (e.g. `lookback`, `vol_window`, or anything custom).
|
Each alpha is a class subclassing `BaseAlpha` (`pipeline/alpha/base.py`), living in its own module. It implements `signal(close) -> wide DataFrame` (the raw score); the base class's `to_weights` cross-sectionally z-scores that into position weights (override for custom normalization). Subclasses declare their own typed `__init__` params (e.g. `lookback`, `vol_window`, or anything custom).
|
||||||
|
|||||||
@@ -48,6 +48,13 @@ The env is managed with [uv](https://docs.astral.sh/uv/). `uv sync` builds `.ven
|
|||||||
uv sync
|
uv sync
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Backtrader is an optional dependency and is **not used by the current pipeline**.
|
||||||
|
Install it only for future experiments or adapter work:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync --extra backtrader
|
||||||
|
```
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -113,6 +120,49 @@ partitions already written. Pass the **dataset directory** (`{output_dir}/{unive
|
|||||||
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
|
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
|
||||||
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
|
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
|
||||||
|
|
||||||
|
### `derived` — daily custom/derived data
|
||||||
|
|
||||||
|
Derived data is daily-only v1 research data keyed by `symbol_id,date`, with one
|
||||||
|
or more numeric value columns. It can come from user CSV/parquet files or Python
|
||||||
|
plugins, and is written as a single parquet file at `derived/{name}.pq`.
|
||||||
|
|
||||||
|
The validator normalizes `date` to the trading day, requires unique
|
||||||
|
`symbol_id,date` keys, rejects duplicate columns, and rejects non-numeric value
|
||||||
|
columns. Alpha computation consumes derived data through the existing
|
||||||
|
`--feature-path` flag.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Validate a user file without writing output.
|
||||||
|
uv run python cli.py derived validate --input-path vendor_factor.csv
|
||||||
|
|
||||||
|
# Ingest CSV/parquet into the canonical derived/ layout.
|
||||||
|
uv run python cli.py derived ingest \
|
||||||
|
--input-path vendor_factor.csv \
|
||||||
|
--derived-name vendor_factor
|
||||||
|
|
||||||
|
# List built-in and external derived-data plugin types.
|
||||||
|
uv run python cli.py derived list
|
||||||
|
uv run python cli.py derived list --derived-module path/to/my_derived.py
|
||||||
|
|
||||||
|
# Compute a derived file from daily and/or minute inputs.
|
||||||
|
uv run python cli.py derived compute \
|
||||||
|
--minute-path data/minute_bars/sh600000 \
|
||||||
|
--daily-path data/daily_bars/sh600000 \
|
||||||
|
--derived-type minute_daily_summary \
|
||||||
|
--derived-name minute_summary
|
||||||
|
|
||||||
|
# Join derived columns into a feature-aware alpha.
|
||||||
|
uv run python cli.py alpha compute \
|
||||||
|
--data-path data/daily_bars/sh600000 \
|
||||||
|
--feature-path derived/minute_summary.pq \
|
||||||
|
--alpha-type my_feature_aware_alpha \
|
||||||
|
--alpha-name my_run
|
||||||
|
```
|
||||||
|
|
||||||
|
For compatibility, `feature list` and `feature compute` remain available and
|
||||||
|
delegate to the same derived-data registry. Existing `features/*.pq` files are
|
||||||
|
still valid `--feature-path` inputs when they satisfy the daily numeric contract.
|
||||||
|
|
||||||
### `alpha list` — show registered alpha types
|
### `alpha list` — show registered alpha types
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -130,6 +180,7 @@ uv run python cli.py alpha list --alpha-module path/to/my_alpha.py # include a
|
|||||||
| `--output-dir` | `alphas` | Output directory |
|
| `--output-dir` | `alphas` | Output directory |
|
||||||
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
|
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
|
||||||
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
|
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
|
||||||
|
| `--feature-path` | — | Daily derived/feature parquet file or dataset to left-join on `symbol_id,date`; repeatable |
|
||||||
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
|
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
|
||||||
| `--param` | — | Extra constructor param as `name=value`; repeatable |
|
| `--param` | — | Extra constructor param as `name=value`; repeatable |
|
||||||
|
|
||||||
@@ -203,7 +254,9 @@ uv run python cli.py portfolio build \
|
|||||||
|
|
||||||
Executes the constructed `position_shares` book at the next available open,
|
Executes the constructed `position_shares` book at the next available open,
|
||||||
clipping trades through repeatable constraints. It writes `fills/<name>.pq` and
|
clipping trades through repeatable constraints. It writes `fills/<name>.pq` and
|
||||||
`pnl/<name>.pq`.
|
`pnl/<name>.pq`. Trading costs use the simplified open-execution proportional
|
||||||
|
cash-cost model documented in
|
||||||
|
[`docs/portfolio_trading_cost_model.md`](docs/portfolio_trading_cost_model.md).
|
||||||
|
|
||||||
| Option | Default | Description |
|
| Option | Default | Description |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
@@ -361,6 +414,8 @@ between phases (data is stored long/tidy):
|
|||||||
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
|
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
|
||||||
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
|
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
|
||||||
baostock valuation ratios.)
|
baostock valuation ratios.)
|
||||||
|
- **derived** (`DERIVED_KEY_COLUMNS` + values): required keys `symbol_id, date`;
|
||||||
|
value columns are user/plugin-defined and must be numeric in v1.
|
||||||
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
|
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
|
||||||
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
|
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
|
||||||
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
|
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
|
||||||
@@ -378,8 +433,11 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
|
|||||||
|
|
||||||
- `cli.py` — entry point wiring the file-based phases together
|
- `cli.py` — entry point wiring the file-based phases together
|
||||||
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
|
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
|
||||||
|
- `pipeline/derived/` — daily derived-data ingestion, validation, plugin registry,
|
||||||
|
and built-in derived computations → `derived/*.pq`
|
||||||
- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
|
- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
|
||||||
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
|
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
|
||||||
|
- `pipeline/features/` — compatibility wrappers for the derived-data registry
|
||||||
- `pipeline/combo/` — alpha combination → `combos/*.pq`
|
- `pipeline/combo/` — alpha combination → `combos/*.pq`
|
||||||
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
|
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
|
||||||
reference next-open simulator, and research metrics
|
reference next-open simulator, and research metrics
|
||||||
@@ -401,9 +459,41 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
|
|||||||
- [x] **Reference execution simulation** — next-open fills over constructed
|
- [x] **Reference execution simulation** — next-open fills over constructed
|
||||||
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
|
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
|
||||||
and slippage controls.
|
and slippage controls.
|
||||||
|
- [x] **Derived/custom daily data ("Level 2")** — ingest user CSV/parquet files
|
||||||
|
or compute plugin outputs as validated numeric daily datasets under
|
||||||
|
`derived/{name}.pq`; alpha joins continue through `--feature-path`.
|
||||||
|
- [ ] **Optional Backtrader adapter** — Backtrader is available as the
|
||||||
|
`backtrader` extra for possible future event-driven/broker-style experiments,
|
||||||
|
but it is not part of the current canonical portfolio workflow.
|
||||||
- [ ] **Forward / paper trading** — run the same construction logic on live
|
- [ ] **Forward / paper trading** — run the same construction logic on live
|
||||||
daily data, track simulated fills and a running P&L without real capital.
|
daily data, track simulated fills and a running P&L without real capital.
|
||||||
- [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price,
|
- [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price,
|
||||||
and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
|
and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
|
||||||
brokerage data tier); the free daily sources here only expose daily bars, so
|
brokerage data tier); the free daily sources here only expose daily bars, so
|
||||||
this is a separate data phase rather than extra columns on the daily schema.
|
this is a separate data phase rather than extra columns on the daily schema.
|
||||||
|
|
||||||
|
### Additional TODOs
|
||||||
|
|
||||||
|
The following items are intended extensions beyond the current daily
|
||||||
|
alpha-to-portfolio pipeline:
|
||||||
|
|
||||||
|
- **Long-only portfolio mode** — add a construction option that converts
|
||||||
|
alpha/combo weights into a long-only book while preserving existing lot,
|
||||||
|
price, suspension, and volume-cap handling.
|
||||||
|
- **Index-short hedging mode** — support portfolios that hold long A-share
|
||||||
|
names while shorting an index or index proxy for market exposure control.
|
||||||
|
- **Expanded universe presets** — add explicit universe aliases for CSI 300,
|
||||||
|
CSI 500, CSI 1000, and CSI 1800, while keeping file-based and comma-separated
|
||||||
|
custom universes available.
|
||||||
|
- **Categorical derived data** — extend the numeric-only derived-data v1 contract
|
||||||
|
to support categorical inputs such as industry classifications. In this
|
||||||
|
project, "Level 2" means customized second-level research data produced by
|
||||||
|
users or plugins; it does not necessarily mean exchange order-book/L2 quote
|
||||||
|
feeds.
|
||||||
|
- **Minute bar data** — continue extending the raw minute-bar and feature
|
||||||
|
workflow. The initial Baostock 5-minute download and daily feature plugin path
|
||||||
|
exist; intraday execution and replacing canonical daily bars remain out of
|
||||||
|
scope unless explicitly added later.
|
||||||
|
- **Industry data** — add industry classification inputs for filtering,
|
||||||
|
grouping, exposure reporting, neutralization, or industry-aware portfolio
|
||||||
|
construction.
|
||||||
|
|||||||
@@ -3,7 +3,9 @@
|
|||||||
|
|
||||||
Phases:
|
Phases:
|
||||||
data — Download daily bars to parquet
|
data — Download daily bars to parquet
|
||||||
|
derived — Ingest or compute daily derived data
|
||||||
alpha — Compute alpha weights from data
|
alpha — Compute alpha weights from data
|
||||||
|
feature — Compute daily features from minute bars
|
||||||
combo — Combine alphas into a single weight
|
combo — Combine alphas into a single weight
|
||||||
portfolio — Build tradable positions and simulate execution
|
portfolio — Build tradable positions and simulate execution
|
||||||
"""
|
"""
|
||||||
@@ -13,9 +15,12 @@ import logging
|
|||||||
import click
|
import click
|
||||||
|
|
||||||
from pipeline.data.cli import data
|
from pipeline.data.cli import data
|
||||||
|
from pipeline.derived.cli import derived
|
||||||
from pipeline.alpha.cli import alpha
|
from pipeline.alpha.cli import alpha
|
||||||
|
from pipeline.features.cli import feature
|
||||||
from pipeline.combo.cli import combo
|
from pipeline.combo.cli import combo
|
||||||
from pipeline.portfolio.cli import portfolio
|
from pipeline.portfolio.cli import portfolio
|
||||||
|
from plugins.joinquant.cli import joinquant
|
||||||
from tools.pqcat import pqcat
|
from tools.pqcat import pqcat
|
||||||
from tools.alphaview import alphaview
|
from tools.alphaview import alphaview
|
||||||
|
|
||||||
@@ -39,9 +44,12 @@ def cli(log_level):
|
|||||||
|
|
||||||
|
|
||||||
cli.add_command(data)
|
cli.add_command(data)
|
||||||
|
cli.add_command(derived)
|
||||||
cli.add_command(alpha)
|
cli.add_command(alpha)
|
||||||
|
cli.add_command(feature)
|
||||||
cli.add_command(combo)
|
cli.add_command(combo)
|
||||||
cli.add_command(portfolio)
|
cli.add_command(portfolio)
|
||||||
|
cli.add_command(joinquant)
|
||||||
cli.add_command(pqcat)
|
cli.add_command(pqcat)
|
||||||
cli.add_command(alphaview)
|
cli.add_command(alphaview)
|
||||||
|
|
||||||
|
|||||||
@@ -31,11 +31,68 @@ _BATCH_COLUMNS = [
|
|||||||
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
|
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
# Raw Baostock minute bars. The ``time`` field is usually compact
|
||||||
|
# YYYYMMDDHHMMSSmmm; parsing below also tolerates HH:MM:SS strings in tests.
|
||||||
|
_MINUTE_FIELDS = "date,time,code,open,high,low,close,volume,amount,adjustflag"
|
||||||
|
_MINUTE_NUMERIC = ["open", "high", "low", "close", "volume", "amount"]
|
||||||
|
_MINUTE_COLUMNS = [
|
||||||
|
"symbol", "datetime", "date", "time", "frequency",
|
||||||
|
"open", "high", "low", "close", "volume", "amount", "vwap", "adjustflag",
|
||||||
|
]
|
||||||
|
_MINUTE_FREQUENCIES = {"5", "15", "30", "60"}
|
||||||
|
|
||||||
|
|
||||||
class _SessionLost(Exception):
|
class _SessionLost(Exception):
|
||||||
"""baostock reported the session was dropped (``用户未登录``)."""
|
"""baostock reported the session was dropped (``用户未登录``)."""
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_minute_frequency(frequency: str | int) -> tuple[str, str]:
|
||||||
|
"""Return Baostock frequency and partition label for a minute interval."""
|
||||||
|
raw = str(frequency).strip().lower()
|
||||||
|
if raw.endswith("m"):
|
||||||
|
raw = raw[:-1]
|
||||||
|
if raw not in _MINUTE_FREQUENCIES:
|
||||||
|
raise ValueError(
|
||||||
|
f"Unsupported minute frequency '{frequency}'. "
|
||||||
|
f"Expected one of {sorted(_MINUTE_FREQUENCIES)} minutes."
|
||||||
|
)
|
||||||
|
return raw, f"{raw}m"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_minute_datetime(date: pd.Series, time: pd.Series) -> pd.Series:
|
||||||
|
"""Parse Baostock minute timestamps into pandas datetimes."""
|
||||||
|
date_dt = pd.to_datetime(date, errors="coerce")
|
||||||
|
date_compact = date_dt.dt.strftime("%Y%m%d")
|
||||||
|
time_text = time.astype(str).str.strip()
|
||||||
|
time_digits = time_text.str.replace(r"\D", "", regex=True)
|
||||||
|
|
||||||
|
full_digits = time_digits.str.slice(0, 14)
|
||||||
|
from_full = pd.to_datetime(full_digits, format="%Y%m%d%H%M%S", errors="coerce")
|
||||||
|
|
||||||
|
from_short = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
|
||||||
|
short_time = time_digits.str.len().between(1, 6)
|
||||||
|
if short_time.any():
|
||||||
|
short_digits = (
|
||||||
|
time_digits.loc[short_time]
|
||||||
|
.str.pad(6, side="right", fillchar="0")
|
||||||
|
.str.slice(0, 6)
|
||||||
|
)
|
||||||
|
from_short.loc[short_time] = pd.to_datetime(
|
||||||
|
date_compact.loc[short_time] + short_digits,
|
||||||
|
format="%Y%m%d%H%M%S",
|
||||||
|
errors="coerce",
|
||||||
|
)
|
||||||
|
|
||||||
|
from_text = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
|
||||||
|
text_time = time_text.str.contains(":", regex=False)
|
||||||
|
if text_time.any():
|
||||||
|
from_text.loc[text_time] = pd.to_datetime(
|
||||||
|
date.astype(str).loc[text_time] + " " + time_text.loc[text_time],
|
||||||
|
errors="coerce",
|
||||||
|
)
|
||||||
|
return from_full.fillna(from_short).fillna(from_text)
|
||||||
|
|
||||||
|
|
||||||
def _download_akshare(symbol: str, start: str, end: str, adjust: str = "qfq") -> Optional[pd.DataFrame]:
|
def _download_akshare(symbol: str, start: str, end: str, adjust: str = "qfq") -> Optional[pd.DataFrame]:
|
||||||
"""Download daily bars from akshare. Returns DataFrame with OHLCV columns."""
|
"""Download daily bars from akshare. Returns DataFrame with OHLCV columns."""
|
||||||
try:
|
try:
|
||||||
@@ -239,3 +296,104 @@ def download_daily_batch(
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def download_minute_batch(
|
||||||
|
symbols: Iterable[str],
|
||||||
|
start: str,
|
||||||
|
end: str,
|
||||||
|
frequency: str | int = 5,
|
||||||
|
relogin_every: int = 200,
|
||||||
|
) -> Iterator[Tuple[str, Optional[pd.DataFrame]]]:
|
||||||
|
"""Download raw Baostock minute bars for many symbols.
|
||||||
|
|
||||||
|
Minute bars are intentionally unadjusted (`adjustflag='3'`) because the
|
||||||
|
output is raw intraday market data for downstream feature aggregation, not a
|
||||||
|
tradable daily price series.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
symbols: Internal-form symbols (``sh600000`` / ``sz000001``).
|
||||||
|
start, end: ``YYYY-MM-DD`` bounds.
|
||||||
|
frequency: Baostock minute frequency. ``5``/``"5"``/``"5m"`` all mean
|
||||||
|
5-minute bars.
|
||||||
|
relogin_every: Proactively refresh the baostock session every N symbols.
|
||||||
|
|
||||||
|
Yields:
|
||||||
|
``(symbol, df)`` where ``df`` has raw minute bars or ``None`` when no
|
||||||
|
data is available.
|
||||||
|
"""
|
||||||
|
query_frequency, frequency_label = _normalize_minute_frequency(frequency)
|
||||||
|
adjustflag = _BAOSTOCK_ADJUST["none"]
|
||||||
|
|
||||||
|
def _relogin() -> None:
|
||||||
|
try:
|
||||||
|
bs.logout()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
bs.login()
|
||||||
|
|
||||||
|
def _fetch(symbol: str) -> Optional[pd.DataFrame]:
|
||||||
|
"""One Baostock minute query; returns df, None, or raises _SessionLost."""
|
||||||
|
code = f"{symbol[:2]}.{symbol[2:]}"
|
||||||
|
rs = bs.query_history_k_data_plus(
|
||||||
|
code=code,
|
||||||
|
fields=_MINUTE_FIELDS,
|
||||||
|
start_date=start,
|
||||||
|
end_date=end,
|
||||||
|
frequency=query_frequency,
|
||||||
|
adjustflag=adjustflag,
|
||||||
|
)
|
||||||
|
if rs.error_code != "0":
|
||||||
|
if "未登录" in (rs.error_msg or ""):
|
||||||
|
raise _SessionLost(rs.error_msg)
|
||||||
|
logger.warning("baostock minute error for %s: %s", symbol, rs.error_msg)
|
||||||
|
return None
|
||||||
|
rows = []
|
||||||
|
while rs.next():
|
||||||
|
rows.append(rs.get_row_data())
|
||||||
|
if not rows:
|
||||||
|
return None
|
||||||
|
|
||||||
|
df = pd.DataFrame(rows, columns=_MINUTE_FIELDS.split(","))
|
||||||
|
df[_MINUTE_NUMERIC] = df[_MINUTE_NUMERIC].apply(pd.to_numeric, errors="coerce")
|
||||||
|
df["datetime"] = _parse_minute_datetime(df["date"], df["time"])
|
||||||
|
bad_timestamps = df["datetime"].isna()
|
||||||
|
if bad_timestamps.any():
|
||||||
|
raise ValueError(
|
||||||
|
f"Could not parse {int(bad_timestamps.sum())} minute timestamp(s)"
|
||||||
|
)
|
||||||
|
df["date"] = pd.to_datetime(df["date"], errors="coerce")
|
||||||
|
df["time"] = df["datetime"].dt.strftime("%H:%M:%S")
|
||||||
|
df["frequency"] = frequency_label
|
||||||
|
df["vwap"] = (df["amount"] / df["volume"]).where(df["volume"] > 0)
|
||||||
|
df["symbol"] = symbol
|
||||||
|
return df[_MINUTE_COLUMNS].sort_values("datetime").reset_index(drop=True)
|
||||||
|
|
||||||
|
bs.login()
|
||||||
|
try:
|
||||||
|
for i, symbol in enumerate(symbols):
|
||||||
|
if i and relogin_every and i % relogin_every == 0:
|
||||||
|
_relogin()
|
||||||
|
|
||||||
|
df: Optional[pd.DataFrame] = None
|
||||||
|
for attempt in (1, 2):
|
||||||
|
try:
|
||||||
|
df = _fetch(symbol)
|
||||||
|
break
|
||||||
|
except _SessionLost:
|
||||||
|
if attempt == 1:
|
||||||
|
_relogin()
|
||||||
|
continue
|
||||||
|
logger.warning("baostock minute session lost for %s after relogin", symbol)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("baostock minute download failed for %s: %s", symbol, e)
|
||||||
|
break
|
||||||
|
|
||||||
|
if df is not None and not df.empty:
|
||||||
|
yield symbol, df
|
||||||
|
else:
|
||||||
|
yield symbol, None
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
bs.logout()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|||||||
Binary file not shown.
|
After Width: | Height: | Size: 156 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 107 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 140 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 84 KiB |
@@ -0,0 +1,352 @@
|
|||||||
|
# JoinQuant Comparison Plugin
|
||||||
|
|
||||||
|
## Why a Plugin
|
||||||
|
|
||||||
|
JoinQuant is an external execution and simulation reference. Keeping this code
|
||||||
|
under `plugins/joinquant/` prevents vendor-specific assumptions from entering
|
||||||
|
`pipeline/portfolio/`, where the internal reference simulator remains the
|
||||||
|
canonical implementation.
|
||||||
|
|
||||||
|
## What It Validates
|
||||||
|
|
||||||
|
The comparison is for system correctness:
|
||||||
|
|
||||||
|
- date alignment
|
||||||
|
- internal to JoinQuant symbol mapping
|
||||||
|
- target position generation
|
||||||
|
- once-per-day open execution timing
|
||||||
|
- lot rounding and filled shares
|
||||||
|
- position carry
|
||||||
|
- trading cost
|
||||||
|
- PnL accounting
|
||||||
|
- blocked trades from suspension, limit-up, and limit-down conditions
|
||||||
|
|
||||||
|
## What It Does Not Validate
|
||||||
|
|
||||||
|
It does not validate alpha quality, IC, IR, forecast skill, or whether the
|
||||||
|
strategy is economically useful. Differences can be expected when JoinQuant
|
||||||
|
uses different fee, slippage, cash, corporate-action, or internal rounding
|
||||||
|
rules.
|
||||||
|
|
||||||
|
## Historical Backtest Workflow
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Build internal portfolio targets.
|
||||||
|
uv run python cli.py portfolio build ...
|
||||||
|
|
||||||
|
# 2. Export JoinQuant-compatible frozen targets.
|
||||||
|
uv run python cli.py joinquant export-targets \
|
||||||
|
--positions-path portfolio/run1.pq \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--mode target_shares \
|
||||||
|
--execution-calendar-path data/daily_bars/<universe> \
|
||||||
|
--out-dir plugins_output/joinquant/targets
|
||||||
|
|
||||||
|
# 3. Generate and copy the wrapper strategy and target files into JoinQuant.
|
||||||
|
uv run python cli.py joinquant write-wrapper \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--mode target_shares \
|
||||||
|
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
|
||||||
|
|
||||||
|
# 4. Run the JoinQuant backtest or simulated trading job.
|
||||||
|
# 5. Export JoinQuant fills, positions, and daily PnL to CSV.
|
||||||
|
|
||||||
|
# 6. Ingest JoinQuant output.
|
||||||
|
uv run python cli.py joinquant ingest \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--fills-csv path/to/jq_fills.csv \
|
||||||
|
--positions-csv path/to/jq_positions.csv \
|
||||||
|
--pnl-csv path/to/jq_pnl.csv
|
||||||
|
|
||||||
|
# 7. Reconcile.
|
||||||
|
uv run python cli.py joinquant reconcile \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--targets-dir plugins_output/joinquant/targets/run1 \
|
||||||
|
--our-fills-path fills/run1.pq \
|
||||||
|
--our-positions-path portfolio/run1.pq \
|
||||||
|
--our-pnl-path pnl/run1.pq \
|
||||||
|
--jq-fills-path plugins_output/joinquant/ingested/run1/fills.pq \
|
||||||
|
--jq-positions-path plugins_output/joinquant/ingested/run1/positions.pq \
|
||||||
|
--jq-pnl-path plugins_output/joinquant/ingested/run1/pnl.pq
|
||||||
|
```
|
||||||
|
|
||||||
|
## Forward-Testing Workflow
|
||||||
|
|
||||||
|
After the T-1 close and after the data update:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py portfolio build ...
|
||||||
|
uv run python cli.py joinquant export-targets \
|
||||||
|
--positions-path portfolio/run1.pq \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--mode target_shares \
|
||||||
|
--execution-calendar-path data/daily_bars/<universe> \
|
||||||
|
--start-date T \
|
||||||
|
--end-date T
|
||||||
|
```
|
||||||
|
|
||||||
|
Before the T open, upload or expose the frozen target file to JoinQuant. During
|
||||||
|
the T open, the JoinQuant wrapper reads that file and submits orders, while the
|
||||||
|
internal simulator should run against the same frozen target. After T close or
|
||||||
|
after JoinQuant results are available, ingest the JoinQuant CSV files and run
|
||||||
|
`joinquant reconcile`.
|
||||||
|
|
||||||
|
Forward target files must be frozen before execution. Do not regenerate a
|
||||||
|
target file after observing open or close data for the same trading date. The
|
||||||
|
exporter writes a snapshot JSON with a SHA-256 hash for this reason and refuses
|
||||||
|
to overwrite existing target/snapshot files unless `--force` is passed.
|
||||||
|
|
||||||
|
For comparisons against the internal simulator, pass
|
||||||
|
`--execution-calendar-path` to `joinquant export-targets`. The positions file is
|
||||||
|
dated by construction/signal date, while the simulator executes at the next
|
||||||
|
available open. The calendar option shifts exported target files to that next
|
||||||
|
trading date, so JoinQuant reads the same target on the same execution session.
|
||||||
|
|
||||||
|
For JoinQuant 模拟盘, the browser automation has two operational phases:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Before T open: upload frozen target(s), save strategy, and start/restart 模拟盘.
|
||||||
|
uv run python cli.py joinquant write-browser-config \
|
||||||
|
--out-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
|
||||||
|
--strategy-url "https://www.joinquant.com/<your 模拟盘 page>" \
|
||||||
|
--flow sim-trade
|
||||||
|
|
||||||
|
uv run python cli.py joinquant run-browser-sim \
|
||||||
|
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
|
||||||
|
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
|
||||||
|
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
|
||||||
|
--headed
|
||||||
|
|
||||||
|
# After T close: download/export JoinQuant fills, positions, and pnl, then
|
||||||
|
# ingest/reconcile. The same run-browser-sim command can do this if the config
|
||||||
|
# includes download actions, otherwise use the ingest/reconcile commands.
|
||||||
|
```
|
||||||
|
|
||||||
|
The default simulated-trading template includes selectors for saving the
|
||||||
|
strategy and clicking simulated-trading controls such as `模拟盘`, `模拟交易`,
|
||||||
|
`启动`, and `重启`. These selectors are intentionally configurable because the
|
||||||
|
JoinQuant web UI can differ by account and page version.
|
||||||
|
|
||||||
|
## Target-Shares Mode
|
||||||
|
|
||||||
|
`target_shares` is the default and preferred correctness mode. The exported
|
||||||
|
`target_shares` field comes from the internal `position_shares` column produced
|
||||||
|
by `portfolio build`, because the internal simulator executes that discretized
|
||||||
|
integer book. The generated wrapper calls:
|
||||||
|
|
||||||
|
```python
|
||||||
|
order_target(jq_symbol, target_shares)
|
||||||
|
```
|
||||||
|
|
||||||
|
This mode makes filled shares, position carry, and blocked trades easiest to
|
||||||
|
compare.
|
||||||
|
|
||||||
|
## Target-Value Mode
|
||||||
|
|
||||||
|
`target_value` mode exports `target_value` and `target_weight` from the
|
||||||
|
portfolio file. The generated wrapper calls:
|
||||||
|
|
||||||
|
```python
|
||||||
|
order_target_value(jq_symbol, target_value)
|
||||||
|
```
|
||||||
|
|
||||||
|
This can be useful for portfolio-level comparisons, but JoinQuant may apply its
|
||||||
|
own rounding, cash, and lot rules. Differences are often classified as
|
||||||
|
`JOINQUANT_INTERNAL_ROUNDING`, `LOT_ROUNDING`, or `CASH_CONSTRAINT` depending
|
||||||
|
on the observed output.
|
||||||
|
|
||||||
|
## Symbol Mapping
|
||||||
|
|
||||||
|
Internal symbols are converted as follows:
|
||||||
|
|
||||||
|
```text
|
||||||
|
sh600000 -> 600000.XSHG
|
||||||
|
sh688001 -> 688001.XSHG
|
||||||
|
sz000001 -> 000001.XSHE
|
||||||
|
sz300001 -> 300001.XSHE
|
||||||
|
```
|
||||||
|
|
||||||
|
Reverse mapping is also supported. Invalid exchanges or unsupported A-share
|
||||||
|
prefixes raise `ValueError` instead of silently guessing.
|
||||||
|
|
||||||
|
## Wrapper Strategy Usage
|
||||||
|
|
||||||
|
Generate a configured wrapper:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant write-wrapper \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--mode target_shares \
|
||||||
|
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Copy the generated file and daily CSV target files into JoinQuant. The default
|
||||||
|
loader uses JoinQuant `read_file`, which works for uploaded files. If your
|
||||||
|
JoinQuant runtime allows HTTP or another storage backend, replace only
|
||||||
|
`_read_target_file()` in the generated strategy.
|
||||||
|
|
||||||
|
The wrapper is long-only by default:
|
||||||
|
|
||||||
|
```python
|
||||||
|
ALLOW_SHORT = False
|
||||||
|
```
|
||||||
|
|
||||||
|
Negative targets are clipped to zero and logged. Use `--allow-short` only if
|
||||||
|
the target JoinQuant account supports the required shorting mechanics.
|
||||||
|
|
||||||
|
## Ingesting JoinQuant Outputs
|
||||||
|
|
||||||
|
The ingest command accepts permissive CSV column names and writes strict plugin
|
||||||
|
schemas:
|
||||||
|
|
||||||
|
```text
|
||||||
|
plugins_output/joinquant/ingested/{portfolio_name}/fills.pq
|
||||||
|
plugins_output/joinquant/ingested/{portfolio_name}/positions.pq
|
||||||
|
plugins_output/joinquant/ingested/{portfolio_name}/pnl.pq
|
||||||
|
```
|
||||||
|
|
||||||
|
Missing cost fields default to zero. Missing blocked status defaults to zero.
|
||||||
|
Symbols and dates are normalized.
|
||||||
|
|
||||||
|
## Reading Reconciliation Reports
|
||||||
|
|
||||||
|
The reconcile command writes:
|
||||||
|
|
||||||
|
```text
|
||||||
|
plugins_output/joinquant/reconcile/{portfolio_name}/daily_reconcile.pq
|
||||||
|
plugins_output/joinquant/reconcile/{portfolio_name}/summary.csv
|
||||||
|
plugins_output/joinquant/reconcile/{portfolio_name}/summary.md
|
||||||
|
```
|
||||||
|
|
||||||
|
`daily_reconcile.pq` is per-symbol and includes target shares, internal filled
|
||||||
|
shares, JoinQuant filled shares, realized positions, trade prices, costs, PnL,
|
||||||
|
and a `diff_reason`. `summary.csv` is the daily portfolio-level view for gross
|
||||||
|
exposure, net exposure, cash, total value, PnL, cumulative PnL, turnover, and
|
||||||
|
cost.
|
||||||
|
|
||||||
|
Difference reasons include:
|
||||||
|
|
||||||
|
```text
|
||||||
|
MATCH SYMBOL_MAPPING PRICE_MISMATCH LOT_ROUNDING SUSPENSION LIMIT_UP_BLOCK
|
||||||
|
LIMIT_DOWN_BLOCK VOLUME_OR_LIQUIDITY COST_MODEL CASH_CONSTRAINT
|
||||||
|
SHORT_NOT_SUPPORTED CORPORATE_ACTION JOINQUANT_INTERNAL_ROUNDING
|
||||||
|
MISSING_IN_OUR_SYSTEM MISSING_IN_JOINQUANT UNKNOWN
|
||||||
|
```
|
||||||
|
|
||||||
|
Default tolerances are exact share matching, `1e-4` relative trade-price
|
||||||
|
tolerance, and value tolerance `max(1 yuan, 1e-6 * booksize)`. PnL tolerance is
|
||||||
|
configurable with `--pnl-tolerance`.
|
||||||
|
|
||||||
|
## Minimal Example
|
||||||
|
|
||||||
|
Create a 5-stock equal-weight or fixed-share test portfolio:
|
||||||
|
|
||||||
|
```text
|
||||||
|
sh600000, sz000001, sh600519, sz002594, sz300750
|
||||||
|
```
|
||||||
|
|
||||||
|
Build positions for a small date range, export `target_shares`, upload the CSV
|
||||||
|
files and wrapper to JoinQuant, run the JoinQuant backtest, export fills,
|
||||||
|
positions, and PnL, then run ingest and reconcile. Start with one or two days
|
||||||
|
before expanding the sample.
|
||||||
|
|
||||||
|
For the first one-stock long-only smoke test, the local side can be prepared in
|
||||||
|
one command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant prepare-smoke \
|
||||||
|
--out-dir /tmp/chinese-equity-quant-realdata
|
||||||
|
```
|
||||||
|
|
||||||
|
The command downloads a tiny public daily-bar sample, builds a fixed-share
|
||||||
|
`sh600000` long-only position file, simulates it internally, exports aligned
|
||||||
|
JoinQuant target files, writes a configured wrapper strategy, and creates
|
||||||
|
`joinquant_smoke_manifest.json` with all output paths.
|
||||||
|
|
||||||
|
## Browser Backtest Automation
|
||||||
|
|
||||||
|
JoinQuant's public `jqdatasdk` is a data SDK. It supports authenticated data
|
||||||
|
calls such as `auth(username, password)` and `get_price(...)`, but cloud
|
||||||
|
strategy upload, backtest execution, and result export are web-application
|
||||||
|
workflows. The plugin therefore automates those remote steps through Playwright
|
||||||
|
with a saved browser session.
|
||||||
|
|
||||||
|
Install the optional browser runner in the uv environment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv sync --extra joinquant-browser
|
||||||
|
uv run playwright install chromium
|
||||||
|
```
|
||||||
|
|
||||||
|
Save a reusable login state. This opens a browser; log in normally, including
|
||||||
|
any CAPTCHA or 2FA, then press Enter in the terminal to save state:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant browser-login \
|
||||||
|
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Create a selector/action config for a historical backtest:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant write-browser-config \
|
||||||
|
--out-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
|
||||||
|
--strategy-url "https://www.joinquant.com/<your strategy page>" \
|
||||||
|
--flow backtest
|
||||||
|
```
|
||||||
|
|
||||||
|
If selectors need tuning, capture the logged-in page:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant browser-snapshot \
|
||||||
|
--url "https://www.joinquant.com/<your strategy page>" \
|
||||||
|
--out-dir /tmp/chinese-equity-quant-realdata/browser_snapshot
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the remote backtest automation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant run-browser-backtest \
|
||||||
|
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
|
||||||
|
--config-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
|
||||||
|
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
|
||||||
|
--headed
|
||||||
|
```
|
||||||
|
|
||||||
|
The config is declarative: actions can navigate, paste the generated wrapper,
|
||||||
|
upload all target CSV files, fill dates, click run, wait for completion,
|
||||||
|
download result CSVs, and take screenshots. When the configured downloads
|
||||||
|
produce fills, positions, and PnL CSVs, the runner automatically calls
|
||||||
|
`joinquant ingest` and `joinquant reconcile`.
|
||||||
|
|
||||||
|
For forward testing / 模拟盘, create the config with `--flow sim-trade` and run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant run-browser-sim \
|
||||||
|
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
|
||||||
|
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
|
||||||
|
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
|
||||||
|
--headed
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not store raw JoinQuant passwords in this repository. The browser state file
|
||||||
|
is created with `0600` permissions and should live under `~/.config`, outside
|
||||||
|
the repo.
|
||||||
|
|
||||||
|
## Recommended First Sanity Checks
|
||||||
|
|
||||||
|
1. One liquid stock with a fixed target share count.
|
||||||
|
2. A 10-stock equal-weight long-only portfolio.
|
||||||
|
3. A forced suspension, limit-up, and limit-down sample.
|
||||||
|
4. A short target in long-only mode to confirm `SHORT_NOT_SUPPORTED`.
|
||||||
|
5. A 5-day reversal portfolio after the mechanical checks pass.
|
||||||
|
|
||||||
|
## Known Limitations
|
||||||
|
|
||||||
|
- JoinQuant internal execution details may differ from the reference simulator.
|
||||||
|
- External file loading depends on the JoinQuant environment.
|
||||||
|
- Short selling may not be supported.
|
||||||
|
- Fee, tax, slippage, and minimum-fee models may differ.
|
||||||
|
- Corporate actions may need special handling and should not be hidden.
|
||||||
|
- The internal simulator does not currently emit execution price in
|
||||||
|
`FILL_COLUMNS`; price reconciliation uses explicit price columns if supplied.
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
# JoinQuant Cost Model Findings
|
||||||
|
|
||||||
|
Generated: 2026-07-06
|
||||||
|
|
||||||
|
This report summarizes the JoinQuant trading-cost behavior observed from the
|
||||||
|
browser-automated real-data backtests and compares it with the current internal
|
||||||
|
simulator model. The JoinQuant cost formula below is inferred from rendered
|
||||||
|
transaction tables and strategy logs for this account. Treat it as an observed
|
||||||
|
platform default, not as a guaranteed external contract.
|
||||||
|
|
||||||
|
## Runs Used
|
||||||
|
|
||||||
|
### Longer Comparison Run
|
||||||
|
|
||||||
|
- Portfolio: `jq_long_one_stock_long`
|
||||||
|
- Window: `2024-01-11` to `2024-02-29`
|
||||||
|
- Booksize: `1,000,000 CNY`
|
||||||
|
- Instrument: `600000.XSHG`
|
||||||
|
- Target: buy and hold `1,000` shares
|
||||||
|
- JoinQuant rendered result: completed
|
||||||
|
- Local total PnL: `546.22 CNY`
|
||||||
|
- JoinQuant total PnL from positions tab: `575.00 CNY`
|
||||||
|
- Difference: `28.78 CNY`, or `0.002878` percentage points on the book
|
||||||
|
|
||||||
|
The first JoinQuant transaction was:
|
||||||
|
|
||||||
|
| Date | Side | Shares | Price | Turnover | Fee |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| `2024-01-11` | Buy | `1,000` | `6.57` | `6,570.00` | `5.00` |
|
||||||
|
|
||||||
|
That trade hit a minimum fee. The local simulator charged `6.1415 CNY` because
|
||||||
|
the smoke runner used a flat `10 bps` cash cost on adjusted-price turnover.
|
||||||
|
|
||||||
|
### Cost Probe Run
|
||||||
|
|
||||||
|
- Portfolio: `jq_cost_probe_buy_sell`
|
||||||
|
- Window: `2024-01-11` to `2024-01-12`
|
||||||
|
- Booksize: `1,000,000 CNY`
|
||||||
|
- Instrument: `600000.XSHG`
|
||||||
|
- Targets:
|
||||||
|
- `2024-01-11`: buy `100,000` shares
|
||||||
|
- `2024-01-12`: sell to `0` shares
|
||||||
|
|
||||||
|
Observed JoinQuant transactions:
|
||||||
|
|
||||||
|
| Date | Side | Shares | Price | Turnover | Fee | Implied Fee |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| `2024-01-11` | Buy | `100,000` | `6.57` | `657,000.00` | `197.10` | `3.0000 bps` |
|
||||||
|
| `2024-01-12` | Sell | `-100,000` | `6.51` | `651,000.00` | `846.30` | `13.0000 bps` |
|
||||||
|
|
||||||
|
The transaction-table numbers match this formula exactly:
|
||||||
|
|
||||||
|
```text
|
||||||
|
buy fee = max(5 CNY, turnover * 0.0003)
|
||||||
|
sell fee = max(5 CNY, turnover * 0.0003) + turnover * 0.001
|
||||||
|
```
|
||||||
|
|
||||||
|
In basis points:
|
||||||
|
|
||||||
|
- Buy commission: `3 bps`
|
||||||
|
- Sell commission: `3 bps`
|
||||||
|
- Sell stamp tax: `10 bps`
|
||||||
|
- Minimum commission: `5 CNY`
|
||||||
|
|
||||||
|
No separate transfer fee was visible in this probe. If a separate transfer fee
|
||||||
|
was present as an additional charge, the observed fees would not match the
|
||||||
|
formula above exactly. It may still be folded into JoinQuant's displayed
|
||||||
|
commission field, so this finding should be read as "not separately observable"
|
||||||
|
rather than "impossible".
|
||||||
|
|
||||||
|
No slippage was visible. The wrapper submitted open-time market orders with
|
||||||
|
`run_daily(..., time="open")`, and JoinQuant filled them at the displayed open
|
||||||
|
prices.
|
||||||
|
|
||||||
|
## Evidence From Strategy Logs
|
||||||
|
|
||||||
|
The JoinQuant order log for the cost probe showed:
|
||||||
|
|
||||||
|
```text
|
||||||
|
2024-01-11 ... trade price: 6.57, amount:100000, commission: 197.1
|
||||||
|
2024-01-12 ... trade price: 6.51, amount:100000, commission: 846.3
|
||||||
|
```
|
||||||
|
|
||||||
|
The normal transaction tab showed the same fee values. However, the generated
|
||||||
|
wrapper's `JOINQUANT_FILL` log records had `trade_cost: 0.0`, even for those
|
||||||
|
same fills. That means `get_trades()` did not expose the usable commission
|
||||||
|
value through the field the wrapper currently reads.
|
||||||
|
|
||||||
|
For reconciliation, use the transaction table or JoinQuant order logs for fee
|
||||||
|
details. Do not rely on the wrapper's current `JOINQUANT_FILL.trade_cost`.
|
||||||
|
|
||||||
|
## Difference From The Internal Simulator
|
||||||
|
|
||||||
|
The current internal simulator cost model is
|
||||||
|
`SimpleProportionalCostModel` in `pipeline/portfolio/costs.py`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
trade_cost = abs(traded_shares * execution_price)
|
||||||
|
* (cost_bps + slippage_bps) / 10000
|
||||||
|
```
|
||||||
|
|
||||||
|
The smoke runner used:
|
||||||
|
|
||||||
|
- `cost_bps = 5`
|
||||||
|
- `slippage_bps = 5`
|
||||||
|
- combined one-way cash cost: `10 bps`
|
||||||
|
|
||||||
|
Important differences:
|
||||||
|
|
||||||
|
- The internal simulator uses the same rate for buys and sells.
|
||||||
|
- It has no minimum commission.
|
||||||
|
- It has no sell-only stamp tax.
|
||||||
|
- Slippage is modeled as an extra cash cost.
|
||||||
|
- JoinQuant did not show slippage in the observed open-time fills.
|
||||||
|
- The local smoke download used `adjust="qfq"`, while the JoinQuant wrapper set
|
||||||
|
`set_option("use_real_price", True)`. That price-scale mismatch also affects
|
||||||
|
PnL and cost comparisons.
|
||||||
|
|
||||||
|
## Practical Implications
|
||||||
|
|
||||||
|
For a buy-only smoke test, JoinQuant may charge less than the local model when
|
||||||
|
the trade is large enough for `3 bps` to apply, but it may charge more on small
|
||||||
|
orders because of the `5 CNY` minimum.
|
||||||
|
|
||||||
|
For any test with sells, JoinQuant's default sell fee is materially higher than
|
||||||
|
the current local flat model because of the inferred `10 bps` stamp tax.
|
||||||
|
|
||||||
|
The earlier 30-day buy-and-hold discrepancy was small because only one buy was
|
||||||
|
executed. A rebalancing strategy with many sells will show a larger cost-model
|
||||||
|
difference unless the local simulator is configured to match JoinQuant.
|
||||||
|
|
||||||
|
## Recommended Follow-Ups
|
||||||
|
|
||||||
|
1. Add a JoinQuant-style cost model to the internal simulator:
|
||||||
|
|
||||||
|
```text
|
||||||
|
commission = max(min_commission, turnover * commission_bps / 10000)
|
||||||
|
stamp_tax = turnover * sell_stamp_tax_bps / 10000 for sells only
|
||||||
|
trade_cost = commission + stamp_tax
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Add a CLI option or preset for `portfolio simulate`, for example
|
||||||
|
`--cost-model joinquant-stock`.
|
||||||
|
|
||||||
|
3. Update JoinQuant reconciliation to parse fee values from the transaction
|
||||||
|
table or order logs when CSV exports are unavailable.
|
||||||
|
|
||||||
|
4. Run a second local-vs-JoinQuant comparison with:
|
||||||
|
|
||||||
|
- raw or real-price local bars, not adjusted-price bars
|
||||||
|
- JoinQuant-style costs
|
||||||
|
- slippage disabled locally
|
||||||
|
|
||||||
|
That test should isolate remaining differences to data alignment, price source,
|
||||||
|
rounding, and JoinQuant internal execution behavior.
|
||||||
|
|
||||||
|
## Local Artifacts
|
||||||
|
|
||||||
|
The temporary artifacts from the investigation are:
|
||||||
|
|
||||||
|
- `/tmp/chinese-equity-quant-jq-long/comparison_report.md`
|
||||||
|
- `/tmp/chinese-equity-quant-jq-long/parsed_joinquant/daily_pnl_compare_from_positions_tab.csv`
|
||||||
|
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_analysis_report.md`
|
||||||
|
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_analysis_summary.json`
|
||||||
|
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_probe_transactions_parsed.csv`
|
||||||
|
- `/tmp/chinese-equity-quant-jq-cost-probe/detail_tabs/transactions.txt`
|
||||||
|
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Minute Bar Data Notes
|
||||||
|
|
||||||
|
The minute-bar path downloads raw Baostock intraday bars and stores them as a
|
||||||
|
Hive-partitioned dataset:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py data download-minute \
|
||||||
|
--universe sh600000 \
|
||||||
|
--start-date 2024-01-02 --end-date 2024-01-05 \
|
||||||
|
--frequency 5
|
||||||
|
```
|
||||||
|
|
||||||
|
The default layout is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
data/minute_bars/{universe}/frequency=5m/month=YYYY-MM/*.pq
|
||||||
|
```
|
||||||
|
|
||||||
|
Derived-data plugins can aggregate those bars to daily `symbol_id,date` numeric
|
||||||
|
files, for example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py derived compute \
|
||||||
|
--minute-path data/minute_bars/sh600000 \
|
||||||
|
--daily-path data/daily_bars/sh600000 \
|
||||||
|
--derived-type minute_daily_summary \
|
||||||
|
--derived-name minute_summary
|
||||||
|
```
|
||||||
|
|
||||||
|
The legacy `feature compute` command delegates to the same derived-data
|
||||||
|
registry and remains available for existing scripts.
|
||||||
|
|
||||||
|
## Daily vs Minute Reconciliation
|
||||||
|
|
||||||
|
Baostock's daily raw bars and 5-minute raw bars are close, but they should not
|
||||||
|
be treated as perfectly reconstructible from each other.
|
||||||
|
|
||||||
|
When checking consistency, compare daily raw bars (`data download --adjust none`)
|
||||||
|
against minute bars on the same raw price scale. The minute aggregation should
|
||||||
|
use:
|
||||||
|
|
||||||
|
- `open`: first minute open
|
||||||
|
- `high`: max minute high
|
||||||
|
- `low`: min minute low
|
||||||
|
- `close`: last minute close
|
||||||
|
- `volume`: sum minute volume
|
||||||
|
- `amount`: sum minute amount
|
||||||
|
- `vwap`: `sum(amount) / sum(volume)`
|
||||||
|
|
||||||
|
In a sanity check for `sh600000` from `2024-01-02` through `2024-01-05`, Baostock
|
||||||
|
returned 4 daily rows and 192 5-minute bars, exactly 48 bars per day. Open, low,
|
||||||
|
and close matched daily exactly on all 4 days. High matched on 3 of 4 days; on
|
||||||
|
`2024-01-04`, the daily high was `6.67` while the max 5-minute high was `6.66`.
|
||||||
|
Minute-summed volume and amount were higher than daily by roughly `0.16%` to
|
||||||
|
`1.23%`. VWAP remained very close, with max relative difference around
|
||||||
|
`0.0043%`.
|
||||||
|
|
||||||
|
This appears to be a source-level Baostock reconciliation caveat, not a parser
|
||||||
|
or ordering issue: the minute bars covered the regular `09:35:00` through
|
||||||
|
`15:00:00` range and sorted correctly by timestamp.
|
||||||
|
|
||||||
|
Practical guidance:
|
||||||
|
|
||||||
|
- Use tolerance-based daily-vs-minute checks; do not require exact equality for
|
||||||
|
high, volume, or amount.
|
||||||
|
- Expect open/close alignment to be a stronger sanity check than exact volume
|
||||||
|
reconstruction.
|
||||||
|
- Use minute-derived values as separate daily features, not as replacements for
|
||||||
|
the canonical daily bar dataset unless a strategy explicitly wants that
|
||||||
|
source convention.
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
# Portfolio Trading Cost Model
|
||||||
|
|
||||||
|
This document describes the trading cost model used by `portfolio simulate`.
|
||||||
|
The current implementation is a simplified open-execution proportional cost
|
||||||
|
model. It is intentionally small, explicit, and easy to audit.
|
||||||
|
|
||||||
|
## Open-Execution Timeline
|
||||||
|
|
||||||
|
The simulator runs once per trading day:
|
||||||
|
|
||||||
|
1. A constructed portfolio row provides the target book for an execution date.
|
||||||
|
In the current file layout, a target dated `t` is executed at the next
|
||||||
|
available market date `d = next(t)`.
|
||||||
|
2. Trades are executed at `open[d]`.
|
||||||
|
3. Realized positions are held during the trading day.
|
||||||
|
4. Daily PnL is marked from `open[d]` to `close[d]` on the newly realized book,
|
||||||
|
plus any overnight gap from the previous realized holdings.
|
||||||
|
5. Trading cost is charged only on actually realized `traded_shares`, after all
|
||||||
|
constraints have clipped the desired trade.
|
||||||
|
|
||||||
|
This means a fully blocked order has `traded_shares = 0` and therefore zero
|
||||||
|
trading cost.
|
||||||
|
|
||||||
|
## Current Formula
|
||||||
|
|
||||||
|
For each symbol:
|
||||||
|
|
||||||
|
```text
|
||||||
|
trade_value_i = abs(traded_shares_i * execution_price_i)
|
||||||
|
trade_cost_i = trade_value_i * (cost_bps + slippage_bps) / 10000
|
||||||
|
```
|
||||||
|
|
||||||
|
where:
|
||||||
|
|
||||||
|
```text
|
||||||
|
execution_price_i = open_price_i
|
||||||
|
```
|
||||||
|
|
||||||
|
`cost_bps` is the proportional explicit trading-cost rate in basis points.
|
||||||
|
`slippage_bps` is modeled as an additional cash cost in basis points. The two
|
||||||
|
rates are added linearly. The CLI options `--cost-bps` and `--slippage-bps`
|
||||||
|
both default to `0.0`.
|
||||||
|
|
||||||
|
Both rates are **one-way, per-trade**: the combined `(cost_bps + slippage_bps)`
|
||||||
|
is charged on the traded notional of *each* fill, buy and sell alike. A full
|
||||||
|
round trip (enter then exit a position) is therefore charged twice — e.g.
|
||||||
|
`5 + 5` bps becomes ~20 bps over a complete round trip, not 10. Quote any
|
||||||
|
round-trip figure by doubling, or convert a round-trip budget to a per-trade
|
||||||
|
rate by halving before passing it in.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```text
|
||||||
|
traded_shares = 1000
|
||||||
|
execution_price = 20 yuan
|
||||||
|
cost_bps = 10
|
||||||
|
slippage_bps = 5
|
||||||
|
|
||||||
|
abs(1000 * 20) * 15 / 10000 = 30 yuan
|
||||||
|
```
|
||||||
|
|
||||||
|
## Slippage Convention
|
||||||
|
|
||||||
|
Slippage is not applied by changing the execution price. It is charged only as
|
||||||
|
a cash cost through `trade_cost`.
|
||||||
|
|
||||||
|
Do not double-count slippage by doing both:
|
||||||
|
|
||||||
|
```text
|
||||||
|
execution_price = open * (1 +/- slippage_bps / 10000)
|
||||||
|
trade_cost += trade_value * slippage_bps / 10000
|
||||||
|
```
|
||||||
|
|
||||||
|
The simulator should execute at the open price and subtract the slippage cash
|
||||||
|
cost from PnL.
|
||||||
|
|
||||||
|
## Relationship To The Simulator
|
||||||
|
|
||||||
|
`ReferenceSimulator.fill()` clips desired trades through constraints first, then
|
||||||
|
passes the actual `traded_shares` to the cost model. The per-name result is
|
||||||
|
stored in the fills parquet as `trade_cost`.
|
||||||
|
|
||||||
|
`ReferenceSimulator.run()` sums per-name `trade_cost` into the daily PnL row's
|
||||||
|
`cost` column and subtracts that total from daily PnL:
|
||||||
|
|
||||||
|
```text
|
||||||
|
pnl = overnight + intraday - cost_total
|
||||||
|
```
|
||||||
|
|
||||||
|
## What This Model Does Not Cover
|
||||||
|
|
||||||
|
The current model intentionally does not model:
|
||||||
|
|
||||||
|
- Minimum commissions.
|
||||||
|
- Buy/sell asymmetric fees.
|
||||||
|
- Sell-side stamp duty.
|
||||||
|
- Exchange handling fees.
|
||||||
|
- Regulatory fees.
|
||||||
|
- Transfer fees.
|
||||||
|
- Date-aware fee schedule changes.
|
||||||
|
- Nonlinear price impact.
|
||||||
|
- Auction liquidity / queue effects.
|
||||||
|
- Partial fills caused by open auction depth.
|
||||||
|
|
||||||
|
These omissions are deliberate. The current model is the default reference
|
||||||
|
model, not a detailed brokerage fee simulator.
|
||||||
|
|
||||||
|
## Future Extension
|
||||||
|
|
||||||
|
The simulator is structured around a cost model abstraction:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class CostModel:
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
traded_shares,
|
||||||
|
execution_price,
|
||||||
|
side,
|
||||||
|
date,
|
||||||
|
metadata,
|
||||||
|
):
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
The current implementation is `SimpleProportionalCostModel`.
|
||||||
|
|
||||||
|
A future `AShareDetailedCostModel` can add:
|
||||||
|
|
||||||
|
- Commission, optionally subject to minimum commission.
|
||||||
|
- Sell-side stamp duty.
|
||||||
|
- Transfer fee.
|
||||||
|
- Exchange handling fee.
|
||||||
|
- Regulatory fee.
|
||||||
|
- Date-aware fee rates.
|
||||||
|
- Separate buy-side and sell-side rates.
|
||||||
|
- Optional nonlinear slippage / market-impact model.
|
||||||
|
|
||||||
|
Any future model must preserve the same high-level simulator contract: costs
|
||||||
|
are computed from realized trades after constraints, and slippage must not be
|
||||||
|
counted both through execution-price adjustment and cash cost.
|
||||||
@@ -0,0 +1,509 @@
|
|||||||
|
# Tutorial: Testing a 5-Day Reversal Alpha
|
||||||
|
|
||||||
|
This document is a teaching walkthrough for someone who is new to this research
|
||||||
|
framework and only lightly familiar with quant research. We will use one
|
||||||
|
concrete experiment, a 5-day reversal alpha on the full downloaded Chinese
|
||||||
|
A-share universe, to learn how the framework defines an alpha, stores it, tests
|
||||||
|
it, turns it into a portfolio, and explains the gap between a research result
|
||||||
|
and simulated trading PnL.
|
||||||
|
|
||||||
|
This generated version was refreshed at 2026-06-12T22:52:56.
|
||||||
|
The important point is not the timestamp; it is the research method.
|
||||||
|
|
||||||
|
## The Research Question
|
||||||
|
|
||||||
|
A quant research project starts with a hypothesis:
|
||||||
|
|
||||||
|
> If a stock fell a lot over the last few trading days, it may rebound soon; if
|
||||||
|
> it rose a lot, it may cool off soon.
|
||||||
|
|
||||||
|
This is called **short-horizon reversal**. It is a simple idea: recent losers
|
||||||
|
are candidates to buy, and recent winners are candidates to sell or underweight.
|
||||||
|
In this repo, the tested version looks back 5 trading days.
|
||||||
|
|
||||||
|
The central research question is:
|
||||||
|
|
||||||
|
> Does this 5-day reversal rule create useful portfolio returns after the
|
||||||
|
> framework applies realistic storage, portfolio construction, execution
|
||||||
|
> constraints, and trading costs?
|
||||||
|
|
||||||
|
The answer from this run is nuanced:
|
||||||
|
|
||||||
|
- The naive built-in version is positive under the tradable
|
||||||
|
next-open-to-next-open research convention (**41.40%**),
|
||||||
|
but its stored weights still show that raw z-score weighting is too sensitive
|
||||||
|
to A-share outliers.
|
||||||
|
- A rank-weighted version on a liquid, non-ST, tradable universe has a positive
|
||||||
|
costless research result: **209.58%**
|
||||||
|
at Sharpe **1.44**.
|
||||||
|
- The daily-traded implementation is still not tradable after costs because
|
||||||
|
turnover is too high.
|
||||||
|
|
||||||
|
That is a normal research outcome. Good research is not just asking "did the
|
||||||
|
backtest go up?" It is asking **which layer explains the result**: signal,
|
||||||
|
weighting, universe, construction, execution, or cost.
|
||||||
|
|
||||||
|
## How This Framework Defines An Alpha
|
||||||
|
|
||||||
|
In many quant textbooks, an alpha is described as a **prediction** of future
|
||||||
|
returns. This framework uses a stricter and more practical convention:
|
||||||
|
|
||||||
|
> An alpha is a signed cross-sectional position weight.
|
||||||
|
|
||||||
|
That sentence is the key to the whole repo.
|
||||||
|
|
||||||
|
- **Signed** means positive values are long exposure and negative values are
|
||||||
|
short exposure.
|
||||||
|
- **Cross-sectional** means the alpha compares stocks to other stocks on the
|
||||||
|
same date.
|
||||||
|
- **Position weight** means the output is already an instruction about what the
|
||||||
|
portfolio wants to own. It is not merely a score to correlate with future
|
||||||
|
returns.
|
||||||
|
|
||||||
|
The stored alpha file always has this schema:
|
||||||
|
|
||||||
|
| column | meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `symbol_id` | Stock identifier such as `sh600000` or `sz000001`. |
|
||||||
|
| `date` | The signal date. The alpha is formed using information known by this date's close. |
|
||||||
|
| `alpha_name` | A label for this particular run, such as `reversal_5d_all`. |
|
||||||
|
| `weight` | Signed desired exposure. Positive means long; negative means short. |
|
||||||
|
|
||||||
|
Because the framework treats alphas as position weights, it evaluates them with
|
||||||
|
portfolio metrics: return, Sharpe, turnover, drawdown, and hit rate. It does
|
||||||
|
**not** use IC/IR, because IC/IR would treat the alpha as a return predictor.
|
||||||
|
|
||||||
|
## The Pipeline In One Picture
|
||||||
|
|
||||||
|
Every phase reads parquet files and writes parquet files. That makes the system
|
||||||
|
easy to inspect and rerun one layer at a time.
|
||||||
|
|
||||||
|
```text
|
||||||
|
daily bars
|
||||||
|
-> alpha weights
|
||||||
|
-> combined weights
|
||||||
|
-> portfolio targets and integer positions
|
||||||
|
-> simulated fills and PnL
|
||||||
|
-> evaluation metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
For this experiment, the important phases are:
|
||||||
|
|
||||||
|
| phase | command family | what it teaches you |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Data | `cli.py data download` | What market data is available. |
|
||||||
|
| Alpha compute | `cli.py alpha compute` | How a raw research idea becomes stored weights. |
|
||||||
|
| Alpha eval | `cli.py alpha eval` | How close-formed weights perform over the tradable next-open-to-next-open interval. |
|
||||||
|
| Combo | `cli.py combo combine` | How one or more alphas become one combined book. |
|
||||||
|
| Portfolio build | `cli.py portfolio build` | How weights become target values and integer shares. |
|
||||||
|
| Portfolio simulate | `cli.py portfolio simulate` | How the integer book trades at next open with constraints and costs. |
|
||||||
|
| Portfolio eval | `cli.py portfolio eval` | How the continuous target portfolio behaves over the same costless open-to-open research interval. |
|
||||||
|
|
||||||
|
In a real research workflow, you should learn to pause after every phase and
|
||||||
|
inspect the parquet output. Most mistakes are easier to find at the interface
|
||||||
|
between two phases than at the final PnL line.
|
||||||
|
|
||||||
|
## Step 1: Define The Raw Reversal Signal
|
||||||
|
|
||||||
|
The built-in 5-day reversal alpha is implemented as:
|
||||||
|
|
||||||
|
```python
|
||||||
|
signal = -close.pct_change(5, fill_method=None)
|
||||||
|
```
|
||||||
|
|
||||||
|
For stock `i` on date `t`, this is approximately:
|
||||||
|
|
||||||
|
```text
|
||||||
|
signal[i, t] = -(close[i, t] / close[i, t-5] - 1)
|
||||||
|
```
|
||||||
|
|
||||||
|
So:
|
||||||
|
|
||||||
|
- If a stock rose by 10% over the last 5 trading days, the raw signal is `-10%`.
|
||||||
|
It becomes a candidate short or underweight.
|
||||||
|
- If a stock fell by 10% over the last 5 trading days, the raw signal is `+10%`.
|
||||||
|
It becomes a candidate long or overweight.
|
||||||
|
|
||||||
|
Notice the timing. The signal uses prices through date `t`. It must not use the
|
||||||
|
return from `t` to `t+1`, because that is the future. The costless alpha
|
||||||
|
evaluator tests the weight formed on date `t` over the tradable interval from
|
||||||
|
`open[t+1]` to `open[t+2]`; the later execution simulator is the separate layer
|
||||||
|
that trades the constructed integer book at the next open.
|
||||||
|
|
||||||
|
The code lives in `pipeline/alpha/library/reversal.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class ReversalAlpha(BaseAlpha):
|
||||||
|
name = "reversal"
|
||||||
|
|
||||||
|
def __init__(self, lookback: int = 5):
|
||||||
|
self.lookback = lookback
|
||||||
|
|
||||||
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
return -close.pct_change(self.lookback, fill_method=None)
|
||||||
|
```
|
||||||
|
|
||||||
|
The alpha class only defines the raw signal. The base class then turns that
|
||||||
|
signal into weights.
|
||||||
|
|
||||||
|
## Step 2: Turn A Signal Into Cross-Sectional Weights
|
||||||
|
|
||||||
|
By default, `BaseAlpha.to_weights()` does a cross-sectional z-score each date:
|
||||||
|
|
||||||
|
```text
|
||||||
|
weight[i, t] = (signal[i, t] - mean_signal[t]) / std_signal[t]
|
||||||
|
```
|
||||||
|
|
||||||
|
This means the framework asks:
|
||||||
|
|
||||||
|
> On this date, which stocks have stronger reversal scores than the rest of the
|
||||||
|
> market, and by how much?
|
||||||
|
|
||||||
|
That is useful, but it has a weakness. If a few stocks have extreme trailing
|
||||||
|
returns because they are newly listed, suspended, illiquid, or limit-constrained,
|
||||||
|
z-scoring can put a very large amount of relative exposure into exactly those
|
||||||
|
names.
|
||||||
|
|
||||||
|
That is visible in the naive full-universe run. Stored weights reached about
|
||||||
|
`-52` standard deviations. The
|
||||||
|
result is positive under the open-to-open convention, but it is much weaker and
|
||||||
|
less robust than the rank-weighted versions:
|
||||||
|
|
||||||
|
| run | weighting | research cumulative return | research Sharpe | research turnover/year |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| naive z-score, full universe | z-score | 41.40% | 0.4514 | 160x |
|
||||||
|
|
||||||
|
The lesson is not "reversal is solved." The lesson is:
|
||||||
|
|
||||||
|
> The same raw signal can become a fragile portfolio if the weighting method
|
||||||
|
> reacts badly to outliers.
|
||||||
|
|
||||||
|
## Step 3: Make The Weighting More Robust
|
||||||
|
|
||||||
|
The repo also has a rank-weighted version, `reversal_rank`. It uses the same raw
|
||||||
|
5-day reversal signal, but converts the cross-section to ranks instead of
|
||||||
|
z-scores:
|
||||||
|
|
||||||
|
```python
|
||||||
|
ranks = signal.rank(axis=1)
|
||||||
|
weights = ranks.subtract(ranks.mean(axis=1), axis=0)
|
||||||
|
```
|
||||||
|
|
||||||
|
Rank weighting keeps the ordering of stocks but removes the importance of the
|
||||||
|
exact outlier magnitude. A stock can be "the worst recent loser" or "the best
|
||||||
|
recent winner," but it cannot become dozens of standard deviations important
|
||||||
|
just because its raw percentage move is unusual.
|
||||||
|
|
||||||
|
The full-universe rank version was much less pathological, but still not a
|
||||||
|
clean signal:
|
||||||
|
|
||||||
|
| run | weighting | research cumulative return | research Sharpe | research turnover/year |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| rank, full universe | rank | 73.44% | 0.8860 | 143x |
|
||||||
|
|
||||||
|
That tells us the weighting fix helped, but the universe still contains many
|
||||||
|
names that are poor candidates for a daily reversal strategy.
|
||||||
|
|
||||||
|
## Step 4: Define The Investable Universe
|
||||||
|
|
||||||
|
An alpha should be tested on stocks that could plausibly be traded. The liquid
|
||||||
|
run applies a per-date mask before weights are created. A stock must be:
|
||||||
|
|
||||||
|
- seasoned, with at least 60 observed closes;
|
||||||
|
- currently tradable, using `tradestatus == 1`;
|
||||||
|
- not ST, using `isST == 0`;
|
||||||
|
- inside the top 1000 names by trailing 20-day average traded amount.
|
||||||
|
|
||||||
|
This mask is applied to the signal, not to the price history used to compute the
|
||||||
|
5-day return. That distinction matters. We still compute `pct_change(5)` on the
|
||||||
|
full contiguous price history, then decide which names are eligible to hold on
|
||||||
|
each signal date.
|
||||||
|
|
||||||
|
The liquid rank result is the cleanest research result:
|
||||||
|
|
||||||
|
| run | weighting | universe | research cumulative return | research Sharpe | hit rate |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| rank, liquid subset | rank | top 1000 liquid, tradable, non-ST | 209.58% | 1.4422 | 55.68% |
|
||||||
|
|
||||||
|
This is the first point where a researcher can say:
|
||||||
|
|
||||||
|
> There appears to be a real 5-day reversal effect in a cleaner A-share
|
||||||
|
> universe, before trading costs.
|
||||||
|
|
||||||
|
That last phrase, **before trading costs**, is essential.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
When reading this chart, focus on the shape and relative behavior:
|
||||||
|
|
||||||
|
- The naive z-score line shows why outlier-sensitive weighting is fragile.
|
||||||
|
- The rank full-universe line shows that robust weighting helps, but the full
|
||||||
|
universe still contains noisy and hard-to-trade names.
|
||||||
|
- The liquid rank line shows the signal-level edge before execution costs.
|
||||||
|
|
||||||
|
## Step 5: Check That The Alpha File Is Sane
|
||||||
|
|
||||||
|
Before trusting any metric, inspect the stored alpha artifact. The run checked:
|
||||||
|
|
||||||
|
- The columns match `ALPHA_COLUMNS`.
|
||||||
|
- There are no null weights.
|
||||||
|
- There are no non-finite weights.
|
||||||
|
- There are no duplicate `(symbol_id, date)` rows.
|
||||||
|
- The daily cross-sectional mean is approximately zero.
|
||||||
|
- A one-alpha combo is an exact identity transform.
|
||||||
|
|
||||||
|
| run | schema ok | null w | non-finite w | dup keys | max \|daily mean\| | weight range | combo identity Δ |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| naive z-score (full) | True | 0 | 0 | 0 | 3.32e-16 | [-52.2, 19.2] | 0.00e+00 |
|
||||||
|
| rank (full) | True | 0 | 0 | 0 | 0.00e+00 | [-2603.0, 2603.0] | 0.00e+00 |
|
||||||
|
| rank (liquid subset) | True | 0 | 0 | 0 | 0.00e+00 | [-498.5, 498.5] | 0.00e+00 |
|
||||||
|
|
||||||
|
The rank ranges look numerically large because rank weights scale with the
|
||||||
|
number of names. That is fine: later evaluation divides by gross exposure, and
|
||||||
|
portfolio construction normalizes by `sum(abs(weight))`. The important
|
||||||
|
difference is that rank weights are bounded by cross-sectional rank, not by the
|
||||||
|
raw size of an abnormal stock move.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
This is a good habit: when a backtest looks strange, plot the weights before
|
||||||
|
debugging the PnL. A broken or concentrated weight distribution often explains
|
||||||
|
the result.
|
||||||
|
|
||||||
|
## Step 6: Understand The Alpha Evaluation Formula
|
||||||
|
|
||||||
|
The costless alpha evaluator now asks:
|
||||||
|
|
||||||
|
> If we compute alpha weights after close on date `t`, trade them at `open[t+1]`,
|
||||||
|
> and hold them until `open[t+2]`, what return would we earn before costs?
|
||||||
|
|
||||||
|
This is still a **research-layer approximation**, not the trading simulator. At
|
||||||
|
this stage the framework has only an alpha weight file. It has not yet rounded
|
||||||
|
shares, checked limits, clipped trades, or paid costs. The purpose is to answer
|
||||||
|
a clean signal question: "Do these close-formed weights line up with returns
|
||||||
|
over the interval we could actually own after next-open execution?"
|
||||||
|
|
||||||
|
The daily research return is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
R[t] = sum_i(weight[i, t] * (open[i, t+2] / open[i, t+1] - 1)) / sum_i(abs(weight[i, t]))
|
||||||
|
```
|
||||||
|
|
||||||
|
This has three important consequences:
|
||||||
|
|
||||||
|
- The alpha is normalized by its gross exposure, so the scale of raw weights
|
||||||
|
does not by itself create a higher return.
|
||||||
|
- The new signal does not receive credit for the overnight gap from `close[t]`
|
||||||
|
to `open[t+1]`, because it cannot be traded until `open[t+1]`.
|
||||||
|
- The final two signal dates are dropped from performance metrics because they
|
||||||
|
do not have a complete next-open-to-next-open holding interval.
|
||||||
|
|
||||||
|
Turnover is still measured from the weights:
|
||||||
|
|
||||||
|
```text
|
||||||
|
turnover[t] = sum_i(abs(weight[i, t] - weight[i, t-1])) / sum_i(abs(weight[i, t-1]))
|
||||||
|
```
|
||||||
|
|
||||||
|
The annualized turnover numbers are a warning. Even a positive signal can be
|
||||||
|
hard to monetize if it asks the portfolio to trade too much every day.
|
||||||
|
|
||||||
|
## Step 7: Build A Portfolio From The Alpha
|
||||||
|
|
||||||
|
The alpha file is still an abstract research book. `portfolio build` turns it
|
||||||
|
into target exposures and integer shares.
|
||||||
|
|
||||||
|
The main normalization is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
target_weight[i, t] = weight[i, t] / sum_i(abs(weight[i, t]))
|
||||||
|
target_value[i, t] = booksize * target_weight[i, t]
|
||||||
|
target_shares[i, t] = target_value[i, t] / construction_price[i, t]
|
||||||
|
```
|
||||||
|
|
||||||
|
Then the framework creates an integer A-share book using lot rules and repair
|
||||||
|
logic. This is where a research portfolio starts to become a tradable portfolio.
|
||||||
|
|
||||||
|
The continuous target portfolio matched the stored alpha almost exactly:
|
||||||
|
|
||||||
|
| run | target_value identity max\|Δ\| | alpha→target max\|Δ\| | research corr(alpha,portfolio) | mean integer gross | mean L1 tracking |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| naive z-score (full) | 0.0000 | 0.00e+00 | 1.000000 | 9,138,331 | 2,542,655 |
|
||||||
|
| rank (full) | 0.0000 | 0.00e+00 | 1.000000 | 8,984,098 | 2,678,278 |
|
||||||
|
| rank (liquid subset) | 0.0000 | 0.00e+00 | 1.000000 | 9,810,256 | 862,303 |
|
||||||
|
|
||||||
|
The integer book is not exact because small target positions can be rounded
|
||||||
|
away. The liquid subset has lower tracking error because it spreads the book
|
||||||
|
over fewer and more tradable names.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
When you research a new alpha, ask two separate questions:
|
||||||
|
|
||||||
|
- Does the continuous target portfolio match the alpha? It should.
|
||||||
|
- Does the integer tradable portfolio still resemble the target? It may not,
|
||||||
|
especially for small books or very broad universes.
|
||||||
|
|
||||||
|
## Step 8: Simulate Execution And Costs
|
||||||
|
|
||||||
|
Research returns are not the same as tradable PnL. The simulator executes the
|
||||||
|
integer `position_shares` at the next available open and applies constraints:
|
||||||
|
|
||||||
|
- suspension;
|
||||||
|
- price limit;
|
||||||
|
- volume cap;
|
||||||
|
- proportional trading cost.
|
||||||
|
|
||||||
|
The cost model is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
cost = abs(traded_shares * open) * (cost_bps + slippage_bps) / 10000
|
||||||
|
```
|
||||||
|
|
||||||
|
For this run, cost is 5 bps commission plus 5 bps slippage. Slippage is treated
|
||||||
|
as cash cost, not as an additional execution price adjustment.
|
||||||
|
|
||||||
|
The execution results explain the final research conclusion:
|
||||||
|
|
||||||
|
| run | corr(alpha, exec net) | PnL before cost | total cost | net PnL | mean daily turnover |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| naive z-score (full) | 0.8956 | 1,838,974 | 13,032,720 | -11,193,746 | 0.5711 |
|
||||||
|
| rank (full) | 0.9126 | 5,052,067 | 11,713,451 | -6,661,383 | 0.5133 |
|
||||||
|
| rank (liquid subset) | 0.8884 | 11,017,842 | 12,733,803 | -1,715,960 | 0.5715 |
|
||||||
|
|
||||||
|
For the liquid rank run, simulated PnL before cost is about
|
||||||
|
11,017,842, but total cost is about
|
||||||
|
12,733,803. That is why the final net PnL is
|
||||||
|
weak or negative.
|
||||||
|
|
||||||
|
This is not a contradiction. It is exactly what a research pipeline should show:
|
||||||
|
|
||||||
|
> The signal can exist in the costless layer, but the daily implementation can
|
||||||
|
> still trade too much to keep the edge.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
## Step 9: Read The Headline Metrics Like A Researcher
|
||||||
|
|
||||||
|
The complete summary is:
|
||||||
|
|
||||||
|
| run | weighting | research cum | research Sharpe | research turn/yr | exec before cost | exec net | exec net Sharpe |
|
||||||
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||||
|
| naive z-score (full) | z-score | 41.40% | 0.4514 | 160× | 18.39% | -111.94% | -1.4508 |
|
||||||
|
| rank (full) | rank | 73.44% | 0.8860 | 143× | 50.52% | -66.61% | -1.1839 |
|
||||||
|
| rank (liquid subset) | rank | 209.58% | 1.4422 | 148× | 110.18% | -17.16% | -0.2226 |
|
||||||
|
|
||||||
|
*Research = costless, no-look-ahead weights over the next-open-to-next-open
|
||||||
|
holding interval. Execution = next-open fills on the discretized integer book
|
||||||
|
under suspension / price-limit / volume-cap constraints, 5 bps commission + 5
|
||||||
|
bps slippage.*
|
||||||
|
|
||||||
|
Here is the interpretation:
|
||||||
|
|
||||||
|
- **Naive z-score full universe**: positive under open-to-open research, but a
|
||||||
|
less reliable test of the reversal idea because the weighting scheme lets
|
||||||
|
outliers dominate parts of the book.
|
||||||
|
- **Rank full universe**: a better test of the same idea, but still noisy
|
||||||
|
because the universe includes too many problematic names.
|
||||||
|
- **Rank liquid subset**: the best signal-level test; it finds the cleanest
|
||||||
|
costless reversal effect.
|
||||||
|
- **Execution net**: daily rebalancing remains heavily constrained by cost.
|
||||||
|
|
||||||
|
A beginner might look only at the final net PnL and say "the alpha failed." A
|
||||||
|
researcher should be more precise:
|
||||||
|
|
||||||
|
> The raw 5-day reversal idea has signal value in a liquid universe, but the
|
||||||
|
> current daily trading rule has too much turnover for the assumed cost model.
|
||||||
|
|
||||||
|
## Step 10: Reproduce The Experiment
|
||||||
|
|
||||||
|
These commands reproduce the important artifacts, assuming the full daily-bar
|
||||||
|
dataset already exists at `data/daily_bars/all`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Naive z-score baseline: built-in reversal alpha, full universe.
|
||||||
|
uv run python cli.py alpha compute --data-path data/daily_bars/all \
|
||||||
|
--alpha-name reversal_5d_all --alpha-type reversal --lookback 5 \
|
||||||
|
--output-dir alphas
|
||||||
|
|
||||||
|
# Rank-weighted full and liquid runs.
|
||||||
|
bash scripts/run_reversal_rank_e2e.sh
|
||||||
|
|
||||||
|
# Regenerate figures, diagnostics, and this tutorial report.
|
||||||
|
uv run python scripts/generate_reversal_5d_report.py
|
||||||
|
```
|
||||||
|
|
||||||
|
If you are learning the framework, do not run the whole pipeline blindly. Run
|
||||||
|
one phase, inspect the output parquet, then continue.
|
||||||
|
|
||||||
|
## How To Research Your Own Alpha
|
||||||
|
|
||||||
|
Use this checklist for a new idea.
|
||||||
|
|
||||||
|
1. State the hypothesis in plain language.
|
||||||
|
Example: "Stocks with poor 5-day returns may rebound over the next day."
|
||||||
|
|
||||||
|
2. Write the raw signal.
|
||||||
|
Implement `signal(close) -> wide DataFrame` in an alpha class. Higher values
|
||||||
|
should mean stronger long preference.
|
||||||
|
|
||||||
|
3. Choose the weighting method.
|
||||||
|
The default z-score is useful, but it can be fragile. Consider rank weights,
|
||||||
|
caps, neutralization, or liquidity-aware filters if outliers dominate.
|
||||||
|
|
||||||
|
4. Define the investable universe before trusting results.
|
||||||
|
Make sure the strategy is not depending on suspended, ST, newly listed, or
|
||||||
|
illiquid names.
|
||||||
|
|
||||||
|
5. Evaluate the alpha as a portfolio, not as a prediction.
|
||||||
|
Check cumulative return, Sharpe, drawdown, hit rate, and turnover over the
|
||||||
|
next-open-to-next-open holding interval. Do not add IC/IR unless the
|
||||||
|
framework's alpha convention changes.
|
||||||
|
|
||||||
|
6. Build the portfolio and inspect tracking.
|
||||||
|
Confirm that target weights match the alpha, then check whether integer
|
||||||
|
shares still track the target book.
|
||||||
|
|
||||||
|
7. Simulate execution with costs.
|
||||||
|
The final research question is not only "is there a signal?" It is "is there
|
||||||
|
enough signal left after realistic trading?"
|
||||||
|
|
||||||
|
8. Diagnose the failure layer.
|
||||||
|
If results are bad, identify whether the problem is the raw signal, weighting,
|
||||||
|
universe, construction, execution constraints, turnover, or cost.
|
||||||
|
|
||||||
|
For this 5-day reversal study, the diagnosis is clear: **the signal-level result
|
||||||
|
is most promising after robust weighting and a liquid universe filter, but the
|
||||||
|
current implementation needs turnover control before it can be considered
|
||||||
|
tradable.**
|
||||||
|
|
||||||
|
## Next Research Directions
|
||||||
|
|
||||||
|
The natural next experiments are:
|
||||||
|
|
||||||
|
- Add turnover control: no-trade bands, slower rebalancing, or weight smoothing.
|
||||||
|
- Sweep the lookback window: compare 3-day, 5-day, 10-day, and 20-day reversal.
|
||||||
|
- Sweep liquidity filters: top 500, top 1000, top 1500 by traded amount.
|
||||||
|
- Add position caps so no single name can dominate after normalization.
|
||||||
|
- Compare rank weighting with volatility-scaled reversal.
|
||||||
|
|
||||||
|
The most important habit is to keep the layers separate. A good alpha research
|
||||||
|
workflow does not stop at a single performance number; it explains how the idea
|
||||||
|
travels from hypothesis, to signal, to weights, to portfolio, to executable PnL.
|
||||||
|
|
||||||
|
## Appendix: Phase Timings From This Rerun
|
||||||
|
|
||||||
|
| phase | rank full (s) | rank liquid (s) |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| alpha compute | 94.1 | 107.8 |
|
||||||
|
| alpha eval | 93.3 | 96.9 |
|
||||||
|
| combo combine | 21.6 | 21.7 |
|
||||||
|
| portfolio build | 537.6 | 236.7 |
|
||||||
|
| portfolio eval | 95.1 | 88.3 |
|
||||||
|
| portfolio simulate | 139.6 | 139.1 |
|
||||||
|
| total | 981.3 | 690.5 |
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
`portfolio build` usually dominates because it iterates per signal date and
|
||||||
|
repairs a multi-thousand-name integer book under lot rules. The liquid run is
|
||||||
|
faster because it carries fewer non-zero names per date.
|
||||||
+19
-5
@@ -5,7 +5,7 @@ position weights. Subclasses implement :meth:`signal` — the raw, unnormalized
|
|||||||
score. The base class turns a signal into cross-sectionally z-scored weights
|
score. The base class turns a signal into cross-sectionally z-scored weights
|
||||||
via :meth:`to_weights` (override it for a different normalization).
|
via :meth:`to_weights` (override it for a different normalization).
|
||||||
"""
|
"""
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
@@ -15,15 +15,14 @@ class BaseAlpha(ABC):
|
|||||||
"""A position-weight alpha over a cross-section of stocks.
|
"""A position-weight alpha over a cross-section of stocks.
|
||||||
|
|
||||||
Concrete subclasses must set a unique class-level :attr:`name` (the registry
|
Concrete subclasses must set a unique class-level :attr:`name` (the registry
|
||||||
key) and implement :meth:`signal`. Construct subclasses with their own typed
|
key) and implement either :meth:`signal` or :meth:`signal_from_data`.
|
||||||
parameters (e.g. ``lookback``); the factory passes only the parameters a
|
Construct subclasses with their own typed parameters (e.g. ``lookback``);
|
||||||
given ``__init__`` accepts.
|
the factory passes only the parameters a given ``__init__`` accepts.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
#: Unique registry key. Every concrete alpha must set this to a non-empty str.
|
#: Unique registry key. Every concrete alpha must set this to a non-empty str.
|
||||||
name: str = ""
|
name: str = ""
|
||||||
|
|
||||||
@abstractmethod
|
|
||||||
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
"""Compute the raw signal.
|
"""Compute the raw signal.
|
||||||
|
|
||||||
@@ -34,6 +33,21 @@ class BaseAlpha(ABC):
|
|||||||
A wide DataFrame aligned to ``close`` where higher values indicate a
|
A wide DataFrame aligned to ``close`` where higher values indicate a
|
||||||
stronger long. Use NaN where the signal is undefined.
|
stronger long. Use NaN where the signal is undefined.
|
||||||
"""
|
"""
|
||||||
|
raise NotImplementedError(
|
||||||
|
f"{type(self).__name__} must implement signal() or signal_from_data()"
|
||||||
|
)
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute the raw signal from long daily data plus wide closes.
|
||||||
|
|
||||||
|
Feature-aware alphas can override this to pivot joined feature columns
|
||||||
|
from ``data``. The default preserves the existing close-only alpha API.
|
||||||
|
"""
|
||||||
|
return self.signal(close)
|
||||||
|
|
||||||
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
|
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
|
||||||
"""Cross-sectionally z-score a signal into signed position weights.
|
"""Cross-sectionally z-score a signal into signed position weights.
|
||||||
|
|||||||
+22
-4
@@ -56,6 +56,10 @@ def list_(alpha_modules):
|
|||||||
@click.option("--output-dir", default="alphas", help="Directory to save alpha parquet")
|
@click.option("--output-dir", default="alphas", help="Directory to save alpha parquet")
|
||||||
@click.option("--lookback", default=5, type=int, help="Lookback days")
|
@click.option("--lookback", default=5, type=int, help="Lookback days")
|
||||||
@click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)")
|
@click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)")
|
||||||
|
@click.option(
|
||||||
|
"--feature-path", "feature_paths", multiple=True,
|
||||||
|
help="Daily derived/feature parquet file or dataset to left-join on symbol_id,date (repeatable)",
|
||||||
|
)
|
||||||
@click.option(
|
@click.option(
|
||||||
"--alpha-module", "alpha_modules", multiple=True,
|
"--alpha-module", "alpha_modules", multiple=True,
|
||||||
help="External module(s) to import so their alphas register (dotted path or .py file)",
|
help="External module(s) to import so their alphas register (dotted path or .py file)",
|
||||||
@@ -64,8 +68,17 @@ def list_(alpha_modules):
|
|||||||
"--param", "extra_params", multiple=True,
|
"--param", "extra_params", multiple=True,
|
||||||
help="Extra alpha constructor param as name=value (repeatable)",
|
help="Extra alpha constructor param as name=value (repeatable)",
|
||||||
)
|
)
|
||||||
|
@click.option(
|
||||||
|
"--liquid-universe", is_flag=True, default=False,
|
||||||
|
help="Mask weights to a per-date investable universe (tradable, non-ST, "
|
||||||
|
"seasoned, top liquidity) before normalization",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--universe-top-n", default=1000, type=int,
|
||||||
|
help="Most-liquid names kept per date when --liquid-universe is set",
|
||||||
|
)
|
||||||
def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
|
def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
|
||||||
alpha_modules, extra_params):
|
feature_paths, alpha_modules, extra_params, liquid_universe, universe_top_n):
|
||||||
"""Compute one alpha from raw data and save as parquet."""
|
"""Compute one alpha from raw data and save as parquet."""
|
||||||
for spec in alpha_modules:
|
for spec in alpha_modules:
|
||||||
load_alpha_module(spec)
|
load_alpha_module(spec)
|
||||||
@@ -81,6 +94,8 @@ def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
|
|||||||
params = {"lookback": lookback, "vol_window": vol_window}
|
params = {"lookback": lookback, "vol_window": vol_window}
|
||||||
params.update(_parse_params(extra_params))
|
params.update(_parse_params(extra_params))
|
||||||
|
|
||||||
|
universe = {"top_n": universe_top_n} if liquid_universe else None
|
||||||
|
|
||||||
data = pd.read_parquet(data_path)
|
data = pd.read_parquet(data_path)
|
||||||
click.echo(f"Loaded data: {len(data):,} rows from {data_path}")
|
click.echo(f"Loaded data: {len(data):,} rows from {data_path}")
|
||||||
|
|
||||||
@@ -88,6 +103,8 @@ def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
|
|||||||
data=data,
|
data=data,
|
||||||
alpha_name=alpha_name,
|
alpha_name=alpha_name,
|
||||||
alpha_type=alpha_type,
|
alpha_type=alpha_type,
|
||||||
|
universe=universe,
|
||||||
|
feature_paths=feature_paths,
|
||||||
**params,
|
**params,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -143,7 +160,8 @@ def reversal_vol(data_path, output_dir, lookback, vol_window):
|
|||||||
@alpha.command("eval")
|
@alpha.command("eval")
|
||||||
@click.option("--alpha-path", required=True, help="Path to alpha parquet file")
|
@click.option("--alpha-path", required=True, help="Path to alpha parquet file")
|
||||||
@click.option("--data-path", required=True, help="Path to data parquet (for price data)")
|
@click.option("--data-path", required=True, help="Path to data parquet (for price data)")
|
||||||
def eval_(alpha_path, data_path):
|
@click.option("--report-dir", default="reports", help="Directory to save JSON report")
|
||||||
|
def eval_(alpha_path, data_path, report_dir):
|
||||||
"""Evaluate an alpha's performance (return, Sharpe, turnover).
|
"""Evaluate an alpha's performance (return, Sharpe, turnover).
|
||||||
|
|
||||||
Alphas are interpreted as position WEIGHTS, not return predictors.
|
Alphas are interpreted as position WEIGHTS, not return predictors.
|
||||||
@@ -166,9 +184,9 @@ def eval_(alpha_path, data_path):
|
|||||||
click.echo("=" * 50)
|
click.echo("=" * 50)
|
||||||
|
|
||||||
# Also dump JSON
|
# Also dump JSON
|
||||||
os.makedirs("reports", exist_ok=True)
|
os.makedirs(report_dir, exist_ok=True)
|
||||||
alpha_name = alpha_df["alpha_name"].iloc[0]
|
alpha_name = alpha_df["alpha_name"].iloc[0]
|
||||||
json_path = f"reports/{alpha_name}_eval.json"
|
json_path = os.path.join(report_dir, f"{alpha_name}_eval.json")
|
||||||
with open(json_path, "w") as f:
|
with open(json_path, "w") as f:
|
||||||
json.dump(metrics, f, indent=2)
|
json.dump(metrics, f, indent=2)
|
||||||
click.echo(f"\nReport saved: {json_path}")
|
click.echo(f"\nReport saved: {json_path}")
|
||||||
|
|||||||
+173
-17
@@ -7,12 +7,19 @@ through :mod:`pipeline.alpha.registry`.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
|
|
||||||
from pipeline.alpha.registry import get_alpha
|
from pipeline.alpha.registry import get_alpha
|
||||||
from pipeline.common.schema import ALPHA_COLUMNS
|
from pipeline.common.schema import ALPHA_COLUMNS
|
||||||
|
from pipeline.derived.compute import (
|
||||||
|
DERIVED_KEY_COLUMNS,
|
||||||
|
read_derived_frames,
|
||||||
|
validate_derived_frame,
|
||||||
|
)
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -25,15 +32,120 @@ def _pivot_close(df: pd.DataFrame) -> pd.DataFrame:
|
|||||||
return pivot.sort_index()
|
return pivot.sort_index()
|
||||||
|
|
||||||
|
|
||||||
def _daily_returns(close: pd.DataFrame) -> pd.DataFrame:
|
def _pivot_open(df: pd.DataFrame) -> pd.DataFrame:
|
||||||
"""Compute daily returns from wide close DataFrame."""
|
"""Pivot data to wide format: date index, columns = symbol_id, values = open."""
|
||||||
return close.pct_change()
|
pivot = df.pivot_table(
|
||||||
|
index="date", columns="symbol_id", values="open", aggfunc="first"
|
||||||
|
)
|
||||||
|
return pivot.sort_index()
|
||||||
|
|
||||||
|
|
||||||
|
def join_feature_frames(
|
||||||
|
data: pd.DataFrame,
|
||||||
|
feature_frames: Iterable[pd.DataFrame],
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Left-join validated daily derived/feature frames onto long daily data."""
|
||||||
|
out = data.copy()
|
||||||
|
out["date"] = pd.to_datetime(out["date"])
|
||||||
|
existing = set(out.columns)
|
||||||
|
joined_cols: list[str] = []
|
||||||
|
|
||||||
|
for frame in feature_frames:
|
||||||
|
features = validate_derived_frame(frame)
|
||||||
|
feature_cols = [col for col in features.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
overlap = sorted(existing.intersection(feature_cols))
|
||||||
|
if overlap:
|
||||||
|
raise ValueError(
|
||||||
|
f"Feature columns conflict with existing daily data columns: {overlap}"
|
||||||
|
)
|
||||||
|
out = out.merge(
|
||||||
|
features,
|
||||||
|
on=DERIVED_KEY_COLUMNS,
|
||||||
|
how="left",
|
||||||
|
validate="many_to_one",
|
||||||
|
)
|
||||||
|
existing.update(feature_cols)
|
||||||
|
joined_cols.extend(feature_cols)
|
||||||
|
|
||||||
|
if joined_cols:
|
||||||
|
logger.info("Joined feature columns into daily data: %s", joined_cols)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _forward_open_to_open_returns(open_: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""Return earned by a close-formed signal after next-open execution.
|
||||||
|
|
||||||
|
A weight formed after close on date t can first be traded at open[t+1].
|
||||||
|
With daily retargeting it is then held until open[t+2], so the signal-date
|
||||||
|
forward return is open[t+2] / open[t+1] - 1.
|
||||||
|
"""
|
||||||
|
return open_.shift(-2).divide(open_.shift(-1)) - 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def investable_universe_mask(
|
||||||
|
data: pd.DataFrame,
|
||||||
|
template: pd.DataFrame,
|
||||||
|
*,
|
||||||
|
top_n: int = 1000,
|
||||||
|
min_history: int = 60,
|
||||||
|
require_tradable: bool = True,
|
||||||
|
exclude_st: bool = True,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Build a per-date investable-universe mask aligned to ``template``.
|
||||||
|
|
||||||
|
A ``(date, symbol_id)`` cell is ``True`` when the name is, on that date,
|
||||||
|
seasoned (at least ``min_history`` prior closes), currently tradable
|
||||||
|
(``tradestatus == 1``), not flagged ST (``isST == 0``), and inside the
|
||||||
|
``top_n`` most liquid names by trailing 20-day mean ``amount``. The mask is
|
||||||
|
applied to the *signal* (computed on full contiguous prices), so it
|
||||||
|
restricts only what is *held*, never the price history used to form the
|
||||||
|
signal — that keeps ``pct_change`` correct and look-ahead free.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
data: Long DataFrame with at least ``symbol_id``, ``date``, ``close``,
|
||||||
|
``amount``, ``isST``, ``tradestatus``.
|
||||||
|
template: Wide signal (date index × ``symbol_id`` columns) to align to.
|
||||||
|
top_n: Keep this many most-liquid names per date.
|
||||||
|
min_history: Minimum number of observed closes before a name is eligible.
|
||||||
|
require_tradable: Require ``tradestatus == 1`` on the date.
|
||||||
|
exclude_st: Drop names flagged ``isST == 1``.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Boolean wide DataFrame aligned to ``template``.
|
||||||
|
"""
|
||||||
|
def _wide(col: str) -> pd.DataFrame:
|
||||||
|
return (
|
||||||
|
data.pivot_table(index="date", columns="symbol_id", values=col, aggfunc="first")
|
||||||
|
.sort_index()
|
||||||
|
.reindex(index=template.index, columns=template.columns)
|
||||||
|
)
|
||||||
|
|
||||||
|
close = _wide("close")
|
||||||
|
mask = close.notna()
|
||||||
|
|
||||||
|
seasoned = close.notna().cumsum() >= min_history
|
||||||
|
mask &= seasoned
|
||||||
|
|
||||||
|
if exclude_st and "isST" in data.columns:
|
||||||
|
mask &= _wide("isST").fillna(1) == 0
|
||||||
|
if require_tradable and "tradestatus" in data.columns:
|
||||||
|
mask &= _wide("tradestatus").fillna(0) == 1
|
||||||
|
|
||||||
|
amount = _wide("amount")
|
||||||
|
amt_ma = amount.rolling(20, min_periods=10).mean()
|
||||||
|
liquid_rank = amt_ma.rank(axis=1, ascending=False)
|
||||||
|
mask &= liquid_rank <= top_n
|
||||||
|
|
||||||
|
return mask.fillna(False)
|
||||||
|
|
||||||
|
|
||||||
def compute_alpha(
|
def compute_alpha(
|
||||||
data: pd.DataFrame,
|
data: pd.DataFrame,
|
||||||
alpha_name: str,
|
alpha_name: str,
|
||||||
alpha_type: str,
|
alpha_type: str,
|
||||||
|
universe: dict | None = None,
|
||||||
|
feature_paths: Iterable[str | Path] | None = None,
|
||||||
|
feature_frames: Iterable[pd.DataFrame] | None = None,
|
||||||
**params,
|
**params,
|
||||||
) -> pd.DataFrame:
|
) -> pd.DataFrame:
|
||||||
"""Compute alpha weights from raw data.
|
"""Compute alpha weights from raw data.
|
||||||
@@ -42,6 +154,15 @@ def compute_alpha(
|
|||||||
data: DataFrame with DATA_COLUMNS.
|
data: DataFrame with DATA_COLUMNS.
|
||||||
alpha_name: Label stored in the ``alpha_name`` output column.
|
alpha_name: Label stored in the ``alpha_name`` output column.
|
||||||
alpha_type: Registry key of the alpha class (e.g. ``reversal``).
|
alpha_type: Registry key of the alpha class (e.g. ``reversal``).
|
||||||
|
universe: Optional investable-universe filter. When given, the alpha's
|
||||||
|
raw signal is masked to the investable set (see
|
||||||
|
:func:`investable_universe_mask`) *before* it is turned into
|
||||||
|
weights, so unheld names get weight 0. Keys are forwarded as keyword
|
||||||
|
arguments to :func:`investable_universe_mask`.
|
||||||
|
feature_paths: Optional parquet files/datasets keyed by ``symbol_id``
|
||||||
|
and ``date``. Their numeric feature columns are left-joined onto
|
||||||
|
``data`` before alpha logic runs.
|
||||||
|
feature_frames: Optional in-memory feature frames with the same schema.
|
||||||
**params: Constructor parameters for the alpha (e.g. ``lookback``,
|
**params: Constructor parameters for the alpha (e.g. ``lookback``,
|
||||||
``vol_window``). Only the params the alpha's ``__init__`` accepts are
|
``vol_window``). Only the params the alpha's ``__init__`` accepts are
|
||||||
used; extras are ignored.
|
used; extras are ignored.
|
||||||
@@ -52,9 +173,22 @@ def compute_alpha(
|
|||||||
Raises:
|
Raises:
|
||||||
KeyError: If ``alpha_type`` is not registered.
|
KeyError: If ``alpha_type`` is not registered.
|
||||||
"""
|
"""
|
||||||
|
feature_inputs: list[pd.DataFrame] = []
|
||||||
|
if feature_paths:
|
||||||
|
feature_inputs.extend(read_derived_frames(feature_paths))
|
||||||
|
if feature_frames:
|
||||||
|
feature_inputs.extend(feature_frames)
|
||||||
|
if feature_inputs:
|
||||||
|
data = join_feature_frames(data, feature_inputs)
|
||||||
|
|
||||||
alpha = get_alpha(alpha_type, **params)
|
alpha = get_alpha(alpha_type, **params)
|
||||||
close = _pivot_close(data)
|
close = _pivot_close(data)
|
||||||
weights = alpha.weights(close)
|
signal = alpha.signal_from_data(data, close)
|
||||||
|
if universe is None:
|
||||||
|
weights = alpha.to_weights(signal)
|
||||||
|
else:
|
||||||
|
mask = investable_universe_mask(data, signal, **universe)
|
||||||
|
weights = alpha.to_weights(signal.where(mask))
|
||||||
|
|
||||||
# Melt to long format
|
# Melt to long format
|
||||||
weights_melted = weights.reset_index().melt(
|
weights_melted = weights.reset_index().melt(
|
||||||
@@ -82,8 +216,11 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
|
|||||||
|
|
||||||
Computes return, annualized Sharpe, annualized turnover, max drawdown.
|
Computes return, annualized Sharpe, annualized turnover, max drawdown.
|
||||||
|
|
||||||
Alpha is interpreted as POSITION WEIGHTS, not predictions.
|
Alpha is interpreted as POSITION WEIGHTS, not predictions. A close-formed
|
||||||
Return on date t = sum(weight[s,t] * realized_return[s,t]) / sum(abs(weight[s,t]))
|
weight on date t is assumed tradable at open[t+1] and held until open[t+2].
|
||||||
|
Return on signal date t = sum(weight[s,t] * open_to_open_return[s,t]) /
|
||||||
|
sum(abs(weight[s,t])). This matches the execution convention without
|
||||||
|
crediting the new signal for the overnight gap before it can be traded.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
alpha_df: DataFrame with ALPHA_COLUMNS.
|
alpha_df: DataFrame with ALPHA_COLUMNS.
|
||||||
@@ -93,31 +230,50 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
|
|||||||
Dict with metrics: cumulative_return, sharpe_annual, turnover_annual,
|
Dict with metrics: cumulative_return, sharpe_annual, turnover_annual,
|
||||||
max_drawdown, hit_rate, n_dates.
|
max_drawdown, hit_rate, n_dates.
|
||||||
"""
|
"""
|
||||||
close = _pivot_close(data_df)
|
open_ = _pivot_open(data_df)
|
||||||
returns = _daily_returns(close)
|
fwd_returns_all = _forward_open_to_open_returns(open_)
|
||||||
|
|
||||||
# Pivot alpha weights to wide format
|
# Pivot alpha weights to wide format
|
||||||
weights = alpha_df.pivot_table(
|
weights = alpha_df.pivot_table(
|
||||||
index="date", columns="symbol_id", values="weight", aggfunc="first"
|
index="date", columns="symbol_id", values="weight", aggfunc="first"
|
||||||
).sort_index()
|
).sort_index()
|
||||||
|
|
||||||
# Align dates
|
# Align weights to signal dates that exist on the market calendar. Compute
|
||||||
common_dates = weights.index.intersection(returns.index)
|
# forward open-to-open returns on the full market calendar first, so sparse
|
||||||
|
# signal grids still earn the next available open-to-open interval instead
|
||||||
|
# of the next signal date.
|
||||||
|
common_dates = weights.index.intersection(open_.index)
|
||||||
weights = weights.loc[common_dates]
|
weights = weights.loc[common_dates]
|
||||||
returns = returns.loc[common_dates]
|
fwd_returns = fwd_returns_all.reindex(common_dates)
|
||||||
|
|
||||||
if len(common_dates) < 2:
|
if len(common_dates) < 1:
|
||||||
return {
|
return {
|
||||||
"cumulative_return": 0.0,
|
"cumulative_return": 0.0,
|
||||||
"sharpe_annual": 0.0,
|
"sharpe_annual": 0.0,
|
||||||
"turnover_annual": 0.0,
|
"turnover_annual": 0.0,
|
||||||
"max_drawdown": 0.0,
|
"max_drawdown": 0.0,
|
||||||
"hit_rate": 0.0,
|
"hit_rate": 0.0,
|
||||||
"n_dates": len(common_dates),
|
"n_dates": 0,
|
||||||
}
|
}
|
||||||
|
|
||||||
# Daily portfolio return = sum(w * r) / sum(|w|) — normalized by gross exposure
|
# Daily portfolio return = sum(w_t * r_open[t+1→t+2]) / sum(|w_t|).
|
||||||
daily_returns = (weights * returns).sum(axis=1) / weights.abs().sum(axis=1)
|
# The final two signal dates have no complete next-open holding interval
|
||||||
|
# and are dropped below.
|
||||||
|
gross = weights.abs().sum(axis=1)
|
||||||
|
daily_returns = (
|
||||||
|
(weights * fwd_returns).sum(axis=1, min_count=1)
|
||||||
|
/ gross.replace(0.0, np.nan)
|
||||||
|
)
|
||||||
|
daily_returns = daily_returns.dropna()
|
||||||
|
if len(daily_returns) < 2:
|
||||||
|
return {
|
||||||
|
"cumulative_return": 0.0,
|
||||||
|
"sharpe_annual": 0.0,
|
||||||
|
"turnover_annual": 0.0,
|
||||||
|
"max_drawdown": 0.0,
|
||||||
|
"hit_rate": 0.0,
|
||||||
|
"n_dates": int(len(daily_returns)),
|
||||||
|
}
|
||||||
|
|
||||||
# Cumulative return
|
# Cumulative return
|
||||||
cumulative_return = float((1.0 + daily_returns).prod() - 1.0)
|
cumulative_return = float((1.0 + daily_returns).prod() - 1.0)
|
||||||
@@ -130,7 +286,7 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
|
|||||||
# Annualized turnover: avg daily turnover * 252
|
# Annualized turnover: avg daily turnover * 252
|
||||||
# Daily turnover = sum(|w_t - w_{t-1}|) / sum(|w_{t-1}|)
|
# Daily turnover = sum(|w_t - w_{t-1}|) / sum(|w_{t-1}|)
|
||||||
weight_change = weights.diff().abs().sum(axis=1)
|
weight_change = weights.diff().abs().sum(axis=1)
|
||||||
gross_exposure = weights.abs().sum(axis=1).shift(1)
|
gross_exposure = gross.shift(1)
|
||||||
daily_turnover = weight_change / gross_exposure
|
daily_turnover = weight_change / gross_exposure
|
||||||
turnover_annual = float(daily_turnover.mean() * 252)
|
turnover_annual = float(daily_turnover.mean() * 252)
|
||||||
|
|
||||||
@@ -149,5 +305,5 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
|
|||||||
"turnover_annual": turnover_annual,
|
"turnover_annual": turnover_annual,
|
||||||
"max_drawdown": max_drawdown,
|
"max_drawdown": max_drawdown,
|
||||||
"hit_rate": hit_rate,
|
"hit_rate": hit_rate,
|
||||||
"n_dates": len(common_dates),
|
"n_dates": int(len(daily_returns)),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,4 +4,9 @@ Importing this package imports each alpha module, which registers the alpha via
|
|||||||
the ``@register_alpha`` decorator. Add a new built-in by dropping a module here
|
the ``@register_alpha`` decorator. Add a new built-in by dropping a module here
|
||||||
and importing it below.
|
and importing it below.
|
||||||
"""
|
"""
|
||||||
from pipeline.alpha.library import momentum, reversal, reversal_vol # noqa: F401
|
from pipeline.alpha.library import ( # noqa: F401
|
||||||
|
momentum,
|
||||||
|
reversal,
|
||||||
|
reversal_rank,
|
||||||
|
reversal_vol,
|
||||||
|
)
|
||||||
|
|||||||
@@ -15,4 +15,4 @@ class MomentumAlpha(BaseAlpha):
|
|||||||
self.lookback = lookback
|
self.lookback = lookback
|
||||||
|
|
||||||
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
return close.pct_change(self.lookback)
|
return close.pct_change(self.lookback, fill_method=None)
|
||||||
|
|||||||
@@ -15,4 +15,4 @@ class ReversalAlpha(BaseAlpha):
|
|||||||
self.lookback = lookback
|
self.lookback = lookback
|
||||||
|
|
||||||
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
return -close.pct_change(self.lookback)
|
return -close.pct_change(self.lookback, fill_method=None)
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
"""Outlier-robust short-horizon reversal alpha."""
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class ReversalRankAlpha(BaseAlpha):
|
||||||
|
"""Reversal weighted by cross-sectional rank instead of z-score.
|
||||||
|
|
||||||
|
The signal is the same trailing-return reversal as :class:`ReversalAlpha`,
|
||||||
|
but :meth:`to_weights` converts it with a cross-sectional rank that is then
|
||||||
|
demeaned. Rank weighting is bounded and monotone, so it does not dump the
|
||||||
|
book into a handful of extreme movers the way raw z-scoring does — the
|
||||||
|
failure mode that makes plain ``reversal`` collapse on the A-share universe,
|
||||||
|
where newly listed / post-suspension / limit-up names produce huge
|
||||||
|
``pct_change`` outliers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
name = "reversal_rank"
|
||||||
|
|
||||||
|
def __init__(self, lookback: int = 5):
|
||||||
|
self.lookback = lookback
|
||||||
|
|
||||||
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
return -close.pct_change(self.lookback, fill_method=None)
|
||||||
|
|
||||||
|
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
signal = signal.dropna(how="all")
|
||||||
|
ranks = signal.rank(axis=1)
|
||||||
|
weights = ranks.subtract(ranks.mean(axis=1), axis=0)
|
||||||
|
return weights.fillna(0.0)
|
||||||
@@ -21,6 +21,6 @@ class ReversalVolAlpha(BaseAlpha):
|
|||||||
self.vol_window = vol_window
|
self.vol_window = vol_window
|
||||||
|
|
||||||
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
reversal = -close.pct_change(self.lookback)
|
reversal = -close.pct_change(self.lookback, fill_method=None)
|
||||||
vol = close.pct_change().rolling(self.vol_window).std()
|
vol = close.pct_change(fill_method=None).rolling(self.vol_window).std()
|
||||||
return reversal / vol
|
return reversal / vol
|
||||||
|
|||||||
@@ -26,8 +26,8 @@ def combo():
|
|||||||
def combine(alpha_paths, combo_name, method, output_dir):
|
def combine(alpha_paths, combo_name, method, output_dir):
|
||||||
"""Combine multiple alphas and save as parquet."""
|
"""Combine multiple alphas and save as parquet."""
|
||||||
paths = [p.strip() for p in alpha_paths.split(",") if p.strip()]
|
paths = [p.strip() for p in alpha_paths.split(",") if p.strip()]
|
||||||
if len(paths) < 2:
|
if len(paths) < 1:
|
||||||
click.echo("Error: --alpha-paths requires at least 2 comma-separated paths", err=True)
|
click.echo("Error: --alpha-paths requires at least 1 path", err=True)
|
||||||
return
|
return
|
||||||
|
|
||||||
result = combine_alphas(
|
result = combine_alphas(
|
||||||
|
|||||||
@@ -26,6 +26,31 @@ DATA_COLUMNS: Final[list[str]] = [
|
|||||||
"pcfNcfTTM", # float64: P/CF (net cash flow, TTM)
|
"pcfNcfTTM", # float64: P/CF (net cash flow, TTM)
|
||||||
]
|
]
|
||||||
|
|
||||||
|
# Required columns for raw intraday minute bar parquet files.
|
||||||
|
MINUTE_BAR_COLUMNS: Final[list[str]] = [
|
||||||
|
"symbol_id", # str: internal code like 'sh600000'
|
||||||
|
"symbol_name", # str: stock name like '浦发银行'
|
||||||
|
"datetime", # datetime64: intraday bar timestamp
|
||||||
|
"date", # date component, aligned with daily DATA_COLUMNS date
|
||||||
|
"time", # str: HH:MM:SS bar time
|
||||||
|
"frequency", # str: e.g. '5m'
|
||||||
|
"open", # float64
|
||||||
|
"high", # float64
|
||||||
|
"low", # float64
|
||||||
|
"close", # float64
|
||||||
|
"volume", # float64 (shares)
|
||||||
|
"amount", # float64 (turnover in yuan, raw/unadjusted)
|
||||||
|
"vwap", # float64: amount / volume
|
||||||
|
"adjustflag", # str: baostock adjustment flag; '3' for raw/unadjusted
|
||||||
|
]
|
||||||
|
|
||||||
|
# Required key columns for daily derived-data parquet files. Value columns are
|
||||||
|
# user/plugin-defined and must be numeric.
|
||||||
|
DERIVED_KEY_COLUMNS: Final[list[str]] = [
|
||||||
|
"symbol_id", # str
|
||||||
|
"date", # date: normalized daily timestamp
|
||||||
|
]
|
||||||
|
|
||||||
# Required columns for alpha parquet files.
|
# Required columns for alpha parquet files.
|
||||||
# Alphas are position WEIGHTS: positive=long, negative=short.
|
# Alphas are position WEIGHTS: positive=long, negative=short.
|
||||||
ALPHA_COLUMNS: Final[list[str]] = [
|
ALPHA_COLUMNS: Final[list[str]] = [
|
||||||
|
|||||||
+35
-1
@@ -3,7 +3,7 @@
|
|||||||
import click
|
import click
|
||||||
from datetime import date
|
from datetime import date
|
||||||
|
|
||||||
from pipeline.data.downloader import download_universe
|
from pipeline.data.downloader import download_minute_universe, download_universe
|
||||||
|
|
||||||
|
|
||||||
@click.group(name="data")
|
@click.group(name="data")
|
||||||
@@ -42,3 +42,37 @@ def download(universe, start_date, end_date, output_dir, symbols, chunk_size, ad
|
|||||||
f"{stats['n_rows']:,} bars, {stats['date_min']} → {stats['date_max']}"
|
f"{stats['n_rows']:,} bars, {stats['date_min']} → {stats['date_max']}"
|
||||||
)
|
)
|
||||||
click.echo(f"Dataset: {stats['dataset_path']}")
|
click.echo(f"Dataset: {stats['dataset_path']}")
|
||||||
|
|
||||||
|
|
||||||
|
@data.command("download-minute")
|
||||||
|
@click.option(
|
||||||
|
"--universe", default="csi500",
|
||||||
|
help="Which universe: hs300, csi500, all (~5000 A-shares), file path, or comma-separated symbols",
|
||||||
|
)
|
||||||
|
@click.option("--start-date", default="2017-01-01", help="Start date YYYY-MM-DD")
|
||||||
|
@click.option("--end-date", default=str(date.today()), help="End date YYYY-MM-DD")
|
||||||
|
@click.option("--output-dir", default="data/minute_bars", help="Root for the partitioned dataset")
|
||||||
|
@click.option("--symbols", default=0, type=int, help="Max symbols (0=all)")
|
||||||
|
@click.option("--chunk-size", default=100, type=int, help="Symbols per durability flush")
|
||||||
|
@click.option("--frequency", default="5", help="Minute frequency: 5, 15, 30, or 60")
|
||||||
|
def download_minute(universe, start_date, end_date, output_dir, symbols, chunk_size, frequency):
|
||||||
|
"""Download raw Baostock minute bars into a partitioned parquet dataset.
|
||||||
|
|
||||||
|
Writes ``{output_dir}/{universe}/frequency=5m/month=YYYY-MM/*.pq`` for the
|
||||||
|
default 5-minute frequency.
|
||||||
|
"""
|
||||||
|
stats = download_minute_universe(
|
||||||
|
universe=universe,
|
||||||
|
start_date=start_date,
|
||||||
|
end_date=end_date,
|
||||||
|
output_dir=output_dir,
|
||||||
|
max_symbols=symbols,
|
||||||
|
chunk_size=chunk_size,
|
||||||
|
frequency=frequency,
|
||||||
|
)
|
||||||
|
click.echo(
|
||||||
|
f"\nSummary: {stats['n_symbols']}/{stats['n_requested']} symbols, "
|
||||||
|
f"{stats['n_rows']:,} bars, {stats['date_min']} → {stats['date_max']}, "
|
||||||
|
f"frequency={stats['frequency']}"
|
||||||
|
)
|
||||||
|
click.echo(f"Dataset: {stats['dataset_path']}")
|
||||||
|
|||||||
+126
-2
@@ -11,9 +11,13 @@ import pyarrow.dataset as pads
|
|||||||
|
|
||||||
# Reuse existing downloader and universe modules
|
# Reuse existing downloader and universe modules
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||||
from data.downloader import download_daily_batch
|
from data.downloader import (
|
||||||
|
_normalize_minute_frequency,
|
||||||
|
download_daily_batch,
|
||||||
|
download_minute_batch,
|
||||||
|
)
|
||||||
from data.universe import get_all_stocks, get_hs300_stocks, get_zz500_stocks
|
from data.universe import get_all_stocks, get_hs300_stocks, get_zz500_stocks
|
||||||
from pipeline.common.schema import DATA_COLUMNS
|
from pipeline.common.schema import DATA_COLUMNS, MINUTE_BAR_COLUMNS
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -89,6 +93,25 @@ def _write_month_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: s
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_minute_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: str) -> None:
|
||||||
|
"""Append rows to a Hive-partitioned minute dataset.
|
||||||
|
|
||||||
|
Layout: ``frequency=5m/month=YYYY-MM/*.pq``.
|
||||||
|
"""
|
||||||
|
out = df.copy()
|
||||||
|
out["month"] = pd.to_datetime(out["date"]).dt.strftime("%Y-%m")
|
||||||
|
table = pa.Table.from_pandas(out, preserve_index=False)
|
||||||
|
pads.write_dataset(
|
||||||
|
table,
|
||||||
|
str(base_dir),
|
||||||
|
format="parquet",
|
||||||
|
partitioning=["frequency", "month"],
|
||||||
|
partitioning_flavor="hive",
|
||||||
|
basename_template=f"{basename_prefix}-{{i}}.pq",
|
||||||
|
existing_data_behavior="overwrite_or_ignore",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def download_universe(
|
def download_universe(
|
||||||
universe: str = "csi500",
|
universe: str = "csi500",
|
||||||
start_date: str = "2017-01-01",
|
start_date: str = "2017-01-01",
|
||||||
@@ -177,3 +200,104 @@ def download_universe(
|
|||||||
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
|
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
|
||||||
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
|
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def download_minute_universe(
|
||||||
|
universe: str = "csi500",
|
||||||
|
start_date: str = "2017-01-01",
|
||||||
|
end_date: str = "2026-12-31",
|
||||||
|
output_dir: str = "data/minute_bars",
|
||||||
|
max_symbols: int = 0,
|
||||||
|
chunk_size: int = 100,
|
||||||
|
frequency: str | int = 5,
|
||||||
|
) -> dict:
|
||||||
|
"""Download raw minute bars into a frequency/month-partitioned dataset.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
universe: ``hs300``, ``csi500``, ``all``/``full``, a file path, or a
|
||||||
|
comma-separated symbol list.
|
||||||
|
start_date, end_date: ``YYYY-MM-DD`` bounds.
|
||||||
|
output_dir: Root under which ``{universe}/frequency=5m/month=YYYY-MM``
|
||||||
|
is written.
|
||||||
|
max_symbols: Cap on symbols (0 = all).
|
||||||
|
chunk_size: Symbols per durability flush.
|
||||||
|
frequency: Minute interval. ``5``/``"5"``/``"5m"`` are 5-minute bars.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Stats dict with dataset path, row count, symbol count, date range, and
|
||||||
|
frequency label.
|
||||||
|
"""
|
||||||
|
_, frequency_label = _normalize_minute_frequency(frequency)
|
||||||
|
constituents = _resolve_universe(universe, max_symbols)
|
||||||
|
symbols = constituents["symbol_id"].tolist()
|
||||||
|
names = dict(zip(constituents["symbol_id"], constituents["symbol_name"]))
|
||||||
|
n_requested = len(symbols)
|
||||||
|
logger.info(
|
||||||
|
"Minute universe %s: %d symbols, %s → %s, frequency=%s",
|
||||||
|
universe,
|
||||||
|
n_requested,
|
||||||
|
start_date,
|
||||||
|
end_date,
|
||||||
|
frequency,
|
||||||
|
)
|
||||||
|
|
||||||
|
base_dir = Path(output_dir) / universe
|
||||||
|
target_frequency_dir = base_dir / f"frequency={frequency_label}"
|
||||||
|
if target_frequency_dir.exists():
|
||||||
|
shutil.rmtree(target_frequency_dir)
|
||||||
|
base_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
buffer: list[pd.DataFrame] = []
|
||||||
|
chunk_idx = 0
|
||||||
|
succeeded = 0
|
||||||
|
n_rows = 0
|
||||||
|
date_min = None
|
||||||
|
date_max = None
|
||||||
|
|
||||||
|
def flush() -> None:
|
||||||
|
nonlocal buffer, chunk_idx, n_rows, date_min, date_max
|
||||||
|
if not buffer:
|
||||||
|
return
|
||||||
|
chunk = pd.concat(buffer, ignore_index=True)
|
||||||
|
_write_minute_partitions(chunk, base_dir, basename_prefix=f"chunk{chunk_idx:04d}")
|
||||||
|
n_rows += len(chunk)
|
||||||
|
cmin, cmax = chunk["date"].min(), chunk["date"].max()
|
||||||
|
date_min = cmin if date_min is None else min(date_min, cmin)
|
||||||
|
date_max = cmax if date_max is None else max(date_max, cmax)
|
||||||
|
logger.info(
|
||||||
|
"Flushed minute chunk %d: %d rows (%d symbols done)",
|
||||||
|
chunk_idx,
|
||||||
|
len(chunk),
|
||||||
|
succeeded,
|
||||||
|
)
|
||||||
|
buffer = []
|
||||||
|
chunk_idx += 1
|
||||||
|
|
||||||
|
for i, (symbol, df) in enumerate(
|
||||||
|
download_minute_batch(symbols, start_date, end_date, frequency=frequency), start=1
|
||||||
|
):
|
||||||
|
if df is None:
|
||||||
|
logger.warning(" %s: no minute data", symbol)
|
||||||
|
else:
|
||||||
|
df["symbol_id"] = symbol
|
||||||
|
df["symbol_name"] = names.get(symbol, symbol)
|
||||||
|
buffer.append(df[MINUTE_BAR_COLUMNS])
|
||||||
|
succeeded += 1
|
||||||
|
if len(buffer) >= chunk_size:
|
||||||
|
flush()
|
||||||
|
if i % 100 == 0:
|
||||||
|
logger.info("Minute progress: %d/%d symbols", i, n_requested)
|
||||||
|
flush()
|
||||||
|
|
||||||
|
if succeeded == 0:
|
||||||
|
raise RuntimeError("No minute data downloaded for any symbol")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"dataset_path": str(base_dir),
|
||||||
|
"frequency": frequency_label,
|
||||||
|
"n_symbols": succeeded,
|
||||||
|
"n_requested": n_requested,
|
||||||
|
"n_rows": n_rows,
|
||||||
|
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
|
||||||
|
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
"""Daily derived-data plugin package."""
|
||||||
|
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
"""Base class for daily derived-data plugins."""
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
class BaseDerivedData(ABC):
|
||||||
|
"""Compute daily, symbol-keyed numeric derived data.
|
||||||
|
|
||||||
|
Derived-data plugins may use daily bars, minute bars, or both as inputs, but
|
||||||
|
they must always return daily rows keyed by ``symbol_id,date``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
#: Unique registry key. Every concrete derived-data plugin must set this.
|
||||||
|
name: str = ""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
minute: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute daily derived data.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
daily: Optional daily market data.
|
||||||
|
minute: Optional raw minute bars.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
DataFrame with ``symbol_id``, ``date``, and one or more numeric
|
||||||
|
derived-data columns.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
params = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
|
||||||
|
return f"{type(self).__name__}({params})"
|
||||||
|
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
"""CLI for daily derived-data ingestion and computation."""
|
||||||
|
|
||||||
|
import click
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.compute import (
|
||||||
|
DERIVED_KEY_COLUMNS,
|
||||||
|
compute_derived,
|
||||||
|
read_derived_frame,
|
||||||
|
write_derived_frame,
|
||||||
|
)
|
||||||
|
from pipeline.derived.registry import (
|
||||||
|
available_derived,
|
||||||
|
load_derived_module,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@click.group(name="derived")
|
||||||
|
def derived():
|
||||||
|
"""Ingest, compute, and validate daily derived data."""
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce(value: str):
|
||||||
|
"""Best-effort coercion of a CLI string to int, then float, else str."""
|
||||||
|
for cast in (int, float):
|
||||||
|
try:
|
||||||
|
return cast(value)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_params(pairs: tuple[str, ...]) -> dict:
|
||||||
|
"""Parse repeated ``name=value`` options into a params dict."""
|
||||||
|
params: dict = {}
|
||||||
|
for pair in pairs:
|
||||||
|
if "=" not in pair:
|
||||||
|
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
|
||||||
|
key, value = pair.split("=", 1)
|
||||||
|
params[key.strip()] = _coerce(value.strip())
|
||||||
|
return params
|
||||||
|
|
||||||
|
|
||||||
|
def _read_optional_parquet(path: str | None) -> pd.DataFrame | None:
|
||||||
|
return None if path is None else pd.read_parquet(path)
|
||||||
|
|
||||||
|
|
||||||
|
def _summarize(result: pd.DataFrame) -> str:
|
||||||
|
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
return f"{len(result):,} rows, {len(value_cols)} columns"
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("list")
|
||||||
|
@click.option(
|
||||||
|
"--derived-module", "derived_modules", multiple=True,
|
||||||
|
help="External module(s) to import first (dotted path or .py file)",
|
||||||
|
)
|
||||||
|
def list_(derived_modules):
|
||||||
|
"""List the registered derived-data plugin types."""
|
||||||
|
for spec in derived_modules:
|
||||||
|
load_derived_module(spec)
|
||||||
|
for name in available_derived():
|
||||||
|
click.echo(name)
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("validate")
|
||||||
|
@click.option("--input-path", required=True, help="CSV/parquet file or parquet dataset to validate")
|
||||||
|
def validate(input_path):
|
||||||
|
"""Validate a daily derived-data file without writing output."""
|
||||||
|
try:
|
||||||
|
result = read_derived_frame(input_path)
|
||||||
|
except Exception as exc:
|
||||||
|
raise click.ClickException(str(exc)) from exc
|
||||||
|
click.echo(f"Valid derived data: {input_path} ({_summarize(result)})")
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("ingest")
|
||||||
|
@click.option("--input-path", required=True, help="CSV/parquet file to ingest")
|
||||||
|
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
|
||||||
|
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
|
||||||
|
def ingest(input_path, derived_name, output_dir):
|
||||||
|
"""Ingest a user-provided daily derived-data CSV/parquet file."""
|
||||||
|
try:
|
||||||
|
result = read_derived_frame(input_path)
|
||||||
|
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
|
||||||
|
except Exception as exc:
|
||||||
|
raise click.ClickException(str(exc)) from exc
|
||||||
|
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("compute")
|
||||||
|
@click.option("--daily-path", default=None, help="Optional daily data parquet/dataset")
|
||||||
|
@click.option("--minute-path", default=None, help="Optional minute parquet/dataset")
|
||||||
|
@click.option("--derived-type", required=True, help="Registry key of the derived-data plugin")
|
||||||
|
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
|
||||||
|
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
|
||||||
|
@click.option(
|
||||||
|
"--derived-module", "derived_modules", multiple=True,
|
||||||
|
help="External module(s) to import so their derived-data plugins register",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--param", "extra_params", multiple=True,
|
||||||
|
help="Extra derived-data constructor param as name=value (repeatable)",
|
||||||
|
)
|
||||||
|
def compute(
|
||||||
|
daily_path,
|
||||||
|
minute_path,
|
||||||
|
derived_type,
|
||||||
|
derived_name,
|
||||||
|
output_dir,
|
||||||
|
derived_modules,
|
||||||
|
extra_params,
|
||||||
|
):
|
||||||
|
"""Compute one daily derived-data file from daily and/or minute inputs."""
|
||||||
|
for spec in derived_modules:
|
||||||
|
load_derived_module(spec)
|
||||||
|
|
||||||
|
options = available_derived()
|
||||||
|
if derived_type not in options:
|
||||||
|
raise click.BadParameter(
|
||||||
|
f"Unknown derived-type '{derived_type}'. Available: {options}. "
|
||||||
|
f"Use --derived-module to register an external derived-data plugin.",
|
||||||
|
param_hint="--derived-type",
|
||||||
|
)
|
||||||
|
if daily_path is None and minute_path is None:
|
||||||
|
raise click.UsageError("At least one of --daily-path or --minute-path is required")
|
||||||
|
|
||||||
|
daily = _read_optional_parquet(daily_path)
|
||||||
|
if daily_path:
|
||||||
|
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
|
||||||
|
minute = _read_optional_parquet(minute_path)
|
||||||
|
if minute_path:
|
||||||
|
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = compute_derived(
|
||||||
|
derived_type=derived_type,
|
||||||
|
daily=daily,
|
||||||
|
minute=minute,
|
||||||
|
**_parse_params(extra_params),
|
||||||
|
)
|
||||||
|
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
|
||||||
|
except Exception as exc:
|
||||||
|
raise click.ClickException(str(exc)) from exc
|
||||||
|
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
"""Derived-data computation and validation."""
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from pandas.api.types import is_bool_dtype, is_numeric_dtype
|
||||||
|
|
||||||
|
from pipeline.common.schema import DERIVED_KEY_COLUMNS
|
||||||
|
from pipeline.derived.registry import get_derived
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_derived_frame(derived: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""Validate and normalize a daily derived-data frame.
|
||||||
|
|
||||||
|
A valid derived frame is keyed by unique ``symbol_id,date`` rows and has at
|
||||||
|
least one numeric value column beyond those keys. Dates are normalized to
|
||||||
|
daily timestamps before duplicate-key checks.
|
||||||
|
"""
|
||||||
|
duplicated = derived.columns[derived.columns.duplicated()].tolist()
|
||||||
|
if duplicated:
|
||||||
|
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
|
||||||
|
|
||||||
|
missing = [col for col in DERIVED_KEY_COLUMNS if col not in derived.columns]
|
||||||
|
if missing:
|
||||||
|
raise ValueError(f"Derived data missing required columns: {missing}")
|
||||||
|
|
||||||
|
out = derived.copy()
|
||||||
|
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
|
||||||
|
|
||||||
|
if out.duplicated(DERIVED_KEY_COLUMNS).any():
|
||||||
|
raise ValueError("Derived data has duplicate symbol_id,date rows")
|
||||||
|
|
||||||
|
value_cols = [col for col in out.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
if not value_cols:
|
||||||
|
raise ValueError("Derived data must include at least one value column")
|
||||||
|
|
||||||
|
non_numeric = [
|
||||||
|
col
|
||||||
|
for col in value_cols
|
||||||
|
if is_bool_dtype(out[col]) or not is_numeric_dtype(out[col])
|
||||||
|
]
|
||||||
|
if non_numeric:
|
||||||
|
raise ValueError(f"Derived data value columns must be numeric: {non_numeric}")
|
||||||
|
|
||||||
|
out = out[DERIVED_KEY_COLUMNS + value_cols].copy()
|
||||||
|
return out.sort_values(DERIVED_KEY_COLUMNS).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def compute_derived(
|
||||||
|
derived_type: str,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
minute: pd.DataFrame | None = None,
|
||||||
|
**params,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute one registered derived-data plugin."""
|
||||||
|
if daily is None and minute is None:
|
||||||
|
raise ValueError("Derived data computation requires --daily-path or --minute-path")
|
||||||
|
|
||||||
|
derived = get_derived(derived_type, **params)
|
||||||
|
result = validate_derived_frame(derived.compute(daily=daily, minute=minute))
|
||||||
|
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
logger.info(
|
||||||
|
"Derived data '%s' (%r): %d symbols × %d dates, columns=%s",
|
||||||
|
derived_type,
|
||||||
|
derived,
|
||||||
|
result["symbol_id"].nunique(),
|
||||||
|
result["date"].nunique(),
|
||||||
|
value_cols,
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def read_derived_frame(path: str | Path) -> pd.DataFrame:
|
||||||
|
"""Read and validate one derived CSV/parquet file or parquet dataset."""
|
||||||
|
path = Path(path)
|
||||||
|
if path.suffix.lower() == ".csv":
|
||||||
|
return validate_derived_frame(_read_csv_with_duplicate_header_check(path))
|
||||||
|
return validate_derived_frame(pd.read_parquet(path))
|
||||||
|
|
||||||
|
|
||||||
|
def read_derived_frames(derived_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
|
||||||
|
"""Read and validate derived-data files."""
|
||||||
|
return [read_derived_frame(path) for path in derived_paths]
|
||||||
|
|
||||||
|
|
||||||
|
def write_derived_frame(
|
||||||
|
derived: pd.DataFrame,
|
||||||
|
derived_name: str,
|
||||||
|
output_dir: str | Path = "derived",
|
||||||
|
) -> Path:
|
||||||
|
"""Validate and write derived data to ``{output_dir}/{derived_name}.pq``."""
|
||||||
|
result = validate_derived_frame(derived)
|
||||||
|
output_dir = Path(output_dir)
|
||||||
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
out_path = output_dir / f"{derived_name}.pq"
|
||||||
|
result.to_parquet(out_path, index=False)
|
||||||
|
return out_path
|
||||||
|
|
||||||
|
|
||||||
|
def _read_csv_with_duplicate_header_check(path: Path) -> pd.DataFrame:
|
||||||
|
with path.open(newline="") as fh:
|
||||||
|
reader = csv.reader(fh)
|
||||||
|
try:
|
||||||
|
header = next(reader)
|
||||||
|
except StopIteration as exc:
|
||||||
|
raise ValueError("CSV input is empty") from exc
|
||||||
|
duplicated = sorted({col for col in header if header.count(col) > 1})
|
||||||
|
if duplicated:
|
||||||
|
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
|
||||||
|
return pd.read_csv(path)
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
"""Built-in derived-data library."""
|
||||||
|
|
||||||
|
from pipeline.derived.library import minute_daily_summary # noqa: F401
|
||||||
|
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
"""Daily summary data derived from raw minute bars."""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import register_derived
|
||||||
|
|
||||||
|
|
||||||
|
@register_derived
|
||||||
|
class MinuteDailySummaryDerived(BaseDerivedData):
|
||||||
|
"""Aggregate intraday bars into daily summary columns."""
|
||||||
|
|
||||||
|
name = "minute_daily_summary"
|
||||||
|
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
minute: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
if minute is None:
|
||||||
|
raise ValueError("minute_daily_summary requires minute input")
|
||||||
|
|
||||||
|
minute = minute.copy()
|
||||||
|
minute["date"] = pd.to_datetime(minute["date"]).dt.normalize()
|
||||||
|
sort_cols = ["symbol_id", "date"]
|
||||||
|
if "datetime" in minute.columns:
|
||||||
|
minute["datetime"] = pd.to_datetime(minute["datetime"])
|
||||||
|
sort_cols.append("datetime")
|
||||||
|
elif "time" in minute.columns:
|
||||||
|
sort_cols.append("time")
|
||||||
|
minute = minute.sort_values(sort_cols)
|
||||||
|
|
||||||
|
grouped = minute.groupby(["symbol_id", "date"], sort=True)
|
||||||
|
summary = grouped.agg(
|
||||||
|
minute_bar_count=("close", "count"),
|
||||||
|
first_open=("open", "first"),
|
||||||
|
last_close=("close", "last"),
|
||||||
|
high=("high", "max"),
|
||||||
|
low=("low", "min"),
|
||||||
|
volume_sum=("volume", "sum"),
|
||||||
|
amount_sum=("amount", "sum"),
|
||||||
|
)
|
||||||
|
summary["minute_intraday_return"] = (
|
||||||
|
summary["last_close"] / summary["first_open"] - 1.0
|
||||||
|
)
|
||||||
|
summary["minute_intraday_range"] = summary["high"] / summary["low"] - 1.0
|
||||||
|
summary["minute_vwap"] = (
|
||||||
|
summary["amount_sum"] / summary["volume_sum"].where(summary["volume_sum"] > 0)
|
||||||
|
)
|
||||||
|
summary = summary.reset_index()
|
||||||
|
|
||||||
|
if daily is not None:
|
||||||
|
daily_keys = daily[["symbol_id", "date"]].copy()
|
||||||
|
daily_keys["date"] = pd.to_datetime(daily_keys["date"]).dt.normalize()
|
||||||
|
daily_keys = daily_keys.drop_duplicates(["symbol_id", "date"])
|
||||||
|
result = daily_keys.merge(summary, on=["symbol_id", "date"], how="left")
|
||||||
|
if "close" in daily.columns:
|
||||||
|
daily_close = daily[["symbol_id", "date", "close"]].copy()
|
||||||
|
daily_close["date"] = pd.to_datetime(daily_close["date"]).dt.normalize()
|
||||||
|
daily_close = daily_close.drop_duplicates(["symbol_id", "date"])
|
||||||
|
result = result.merge(
|
||||||
|
daily_close.rename(columns={"close": "daily_close"}),
|
||||||
|
on=["symbol_id", "date"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
reference_close = result["daily_close"].fillna(result["last_close"])
|
||||||
|
else:
|
||||||
|
reference_close = result["last_close"]
|
||||||
|
else:
|
||||||
|
result = summary
|
||||||
|
reference_close = result["last_close"]
|
||||||
|
|
||||||
|
result["minute_vwap_deviation"] = (
|
||||||
|
result["minute_vwap"] / reference_close.replace(0.0, np.nan) - 1.0
|
||||||
|
)
|
||||||
|
return result[
|
||||||
|
[
|
||||||
|
"symbol_id",
|
||||||
|
"date",
|
||||||
|
"minute_bar_count",
|
||||||
|
"minute_intraday_return",
|
||||||
|
"minute_intraday_range",
|
||||||
|
"minute_vwap",
|
||||||
|
"minute_vwap_deviation",
|
||||||
|
]
|
||||||
|
]
|
||||||
|
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
"""Registry and factory for daily derived-data plugins."""
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import importlib.util
|
||||||
|
import inspect
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional, Type
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
|
||||||
|
_REGISTRY: dict[str, Type[BaseDerivedData]] = {}
|
||||||
|
_builtins_loaded = False
|
||||||
|
|
||||||
|
|
||||||
|
def register_derived(cls: Type[BaseDerivedData]) -> Type[BaseDerivedData]:
|
||||||
|
"""Class decorator that registers derived data under ``BaseDerivedData.name``."""
|
||||||
|
if not (isinstance(cls, type) and issubclass(cls, BaseDerivedData)):
|
||||||
|
raise TypeError(f"{cls!r} is not a BaseDerivedData subclass")
|
||||||
|
key = getattr(cls, "name", "")
|
||||||
|
if not key:
|
||||||
|
raise ValueError(f"{cls.__name__} must set a non-empty class attribute `name`")
|
||||||
|
existing = _REGISTRY.get(key)
|
||||||
|
if existing is not None and existing is not cls:
|
||||||
|
raise ValueError(
|
||||||
|
f"Derived data name '{key}' already registered by {existing.__name__}"
|
||||||
|
)
|
||||||
|
_REGISTRY[key] = cls
|
||||||
|
return cls
|
||||||
|
|
||||||
|
|
||||||
|
def available_derived() -> list[str]:
|
||||||
|
"""Sorted names of all registered derived-data plugins."""
|
||||||
|
_ensure_builtins()
|
||||||
|
return sorted(_REGISTRY)
|
||||||
|
|
||||||
|
|
||||||
|
def get_derived(name: str, **params) -> BaseDerivedData:
|
||||||
|
"""Instantiate a registered derived-data plugin by name.
|
||||||
|
|
||||||
|
Only parameters accepted by the plugin class's ``__init__`` are forwarded.
|
||||||
|
"""
|
||||||
|
_ensure_builtins()
|
||||||
|
if name not in _REGISTRY:
|
||||||
|
raise KeyError(f"Unknown derived data '{name}'. Available: {sorted(_REGISTRY)}")
|
||||||
|
cls = _REGISTRY[name]
|
||||||
|
accepted = _accepted_params(cls)
|
||||||
|
kwargs = params if accepted is None else {k: v for k, v in params.items() if k in accepted}
|
||||||
|
return cls(**kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def load_derived_module(spec: str) -> None:
|
||||||
|
"""Import an external module so its ``@register_derived`` classes register."""
|
||||||
|
looks_like_file = spec.endswith(".py") or Path(spec).expanduser().exists()
|
||||||
|
if looks_like_file:
|
||||||
|
path = Path(spec).expanduser().resolve()
|
||||||
|
if not path.exists():
|
||||||
|
raise FileNotFoundError(f"Derived data module not found: {path}")
|
||||||
|
module_spec = importlib.util.spec_from_file_location(path.stem, path)
|
||||||
|
if module_spec is None or module_spec.loader is None:
|
||||||
|
raise ImportError(f"Cannot load derived data module from {path}")
|
||||||
|
module = importlib.util.module_from_spec(module_spec)
|
||||||
|
module_spec.loader.exec_module(module)
|
||||||
|
else:
|
||||||
|
importlib.import_module(spec)
|
||||||
|
|
||||||
|
|
||||||
|
def _accepted_params(cls: Type[BaseDerivedData]) -> Optional[set[str]]:
|
||||||
|
"""Param names ``cls.__init__`` accepts, or None if it takes ``**kwargs``."""
|
||||||
|
sig = inspect.signature(cls.__init__)
|
||||||
|
if any(p.kind is p.VAR_KEYWORD for p in sig.parameters.values()):
|
||||||
|
return None
|
||||||
|
return {name for name in sig.parameters if name != "self"}
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_builtins() -> None:
|
||||||
|
global _builtins_loaded
|
||||||
|
if not _builtins_loaded:
|
||||||
|
import pipeline.derived.library # noqa: F401
|
||||||
|
_builtins_loaded = True
|
||||||
|
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Daily feature plugin package."""
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
"""Compatibility alias for daily feature plugins.
|
||||||
|
|
||||||
|
The canonical plugin API is ``pipeline.derived``. ``BaseFeature`` remains as an
|
||||||
|
alias so existing external feature modules continue to register unchanged.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData as BaseFeature
|
||||||
|
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
"""CLI for daily feature computation."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
import click
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.features.compute import compute_feature
|
||||||
|
from pipeline.features.registry import available_features, load_feature_module
|
||||||
|
|
||||||
|
|
||||||
|
@click.group(name="feature")
|
||||||
|
def feature():
|
||||||
|
"""Compute daily feature parquet files from minute bars."""
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce(value: str):
|
||||||
|
"""Best-effort coercion of a CLI string to int, then float, else str."""
|
||||||
|
for cast in (int, float):
|
||||||
|
try:
|
||||||
|
return cast(value)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_params(pairs: tuple[str, ...]) -> dict:
|
||||||
|
"""Parse repeated ``name=value`` options into a params dict."""
|
||||||
|
params: dict = {}
|
||||||
|
for pair in pairs:
|
||||||
|
if "=" not in pair:
|
||||||
|
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
|
||||||
|
key, value = pair.split("=", 1)
|
||||||
|
params[key.strip()] = _coerce(value.strip())
|
||||||
|
return params
|
||||||
|
|
||||||
|
|
||||||
|
@feature.command("list")
|
||||||
|
@click.option(
|
||||||
|
"--feature-module", "feature_modules", multiple=True,
|
||||||
|
help="External module(s) to import first (dotted path or .py file)",
|
||||||
|
)
|
||||||
|
def list_(feature_modules):
|
||||||
|
"""List the registered feature types."""
|
||||||
|
for spec in feature_modules:
|
||||||
|
load_feature_module(spec)
|
||||||
|
for name in available_features():
|
||||||
|
click.echo(name)
|
||||||
|
|
||||||
|
|
||||||
|
@feature.command("compute")
|
||||||
|
@click.option("--minute-path", required=True, help="Path to minute parquet dataset/file")
|
||||||
|
@click.option("--daily-path", default=None, help="Optional daily data parquet for alignment")
|
||||||
|
@click.option("--feature-type", required=True, help="Registry key of the feature class")
|
||||||
|
@click.option("--feature-name", required=True, help="Name for this feature run/output file")
|
||||||
|
@click.option("--output-dir", default="features", help="Directory to save feature parquet")
|
||||||
|
@click.option(
|
||||||
|
"--feature-module", "feature_modules", multiple=True,
|
||||||
|
help="External module(s) to import so their features register (dotted path or .py file)",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--param", "extra_params", multiple=True,
|
||||||
|
help="Extra feature constructor param as name=value (repeatable)",
|
||||||
|
)
|
||||||
|
def compute(
|
||||||
|
minute_path,
|
||||||
|
daily_path,
|
||||||
|
feature_type,
|
||||||
|
feature_name,
|
||||||
|
output_dir,
|
||||||
|
feature_modules,
|
||||||
|
extra_params,
|
||||||
|
):
|
||||||
|
"""Compute one daily feature file from raw minute bars."""
|
||||||
|
for spec in feature_modules:
|
||||||
|
load_feature_module(spec)
|
||||||
|
|
||||||
|
options = available_features()
|
||||||
|
if feature_type not in options:
|
||||||
|
raise click.BadParameter(
|
||||||
|
f"Unknown feature-type '{feature_type}'. Available: {options}. "
|
||||||
|
f"Use --feature-module to register an external feature.",
|
||||||
|
param_hint="--feature-type",
|
||||||
|
)
|
||||||
|
|
||||||
|
minute = pd.read_parquet(minute_path)
|
||||||
|
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
|
||||||
|
|
||||||
|
daily = None
|
||||||
|
if daily_path:
|
||||||
|
daily = pd.read_parquet(daily_path)
|
||||||
|
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
|
||||||
|
|
||||||
|
result = compute_feature(
|
||||||
|
minute=minute,
|
||||||
|
daily=daily,
|
||||||
|
feature_type=feature_type,
|
||||||
|
**_parse_params(extra_params),
|
||||||
|
)
|
||||||
|
|
||||||
|
os.makedirs(output_dir, exist_ok=True)
|
||||||
|
out_path = f"{output_dir}/{feature_name}.pq"
|
||||||
|
result.to_parquet(out_path, index=False)
|
||||||
|
feature_cols = [col for col in result.columns if col not in ("symbol_id", "date")]
|
||||||
|
click.echo(
|
||||||
|
f"Saved feature: {out_path} ({len(result):,} rows, "
|
||||||
|
f"{len(feature_cols)} columns)"
|
||||||
|
)
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
"""Compatibility wrappers for daily feature computation and validation."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.compute import (
|
||||||
|
DERIVED_KEY_COLUMNS,
|
||||||
|
compute_derived,
|
||||||
|
read_derived_frames,
|
||||||
|
validate_derived_frame,
|
||||||
|
)
|
||||||
|
|
||||||
|
FEATURE_KEY_COLUMNS = DERIVED_KEY_COLUMNS
|
||||||
|
|
||||||
|
|
||||||
|
def validate_feature_frame(features: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""Validate and normalize a legacy daily feature frame."""
|
||||||
|
return validate_derived_frame(features)
|
||||||
|
|
||||||
|
|
||||||
|
def compute_feature(
|
||||||
|
minute: pd.DataFrame,
|
||||||
|
feature_type: str,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
**params,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute one registered feature through the derived-data registry."""
|
||||||
|
return compute_derived(
|
||||||
|
derived_type=feature_type,
|
||||||
|
daily=daily,
|
||||||
|
minute=minute,
|
||||||
|
**params,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def read_feature_frames(feature_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
|
||||||
|
"""Read and validate feature/derived-data parquet files."""
|
||||||
|
return read_derived_frames(feature_paths)
|
||||||
|
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
"""Built-in feature library."""
|
||||||
|
|
||||||
|
from pipeline.features.library import minute_daily_summary # noqa: F401
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
"""Compatibility wrapper for the built-in minute daily summary plugin."""
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.library.minute_daily_summary import MinuteDailySummaryDerived
|
||||||
|
|
||||||
|
|
||||||
|
class MinuteDailySummaryFeature(MinuteDailySummaryDerived):
|
||||||
|
"""Legacy minute-first wrapper around the derived-data implementation."""
|
||||||
|
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
minute: pd.DataFrame,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
return super().compute(daily=daily, minute=minute)
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""Compatibility registry wrappers for daily feature plugins."""
|
||||||
|
|
||||||
|
from typing import Type
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import (
|
||||||
|
available_derived,
|
||||||
|
get_derived,
|
||||||
|
load_derived_module,
|
||||||
|
register_derived,
|
||||||
|
)
|
||||||
|
from pipeline.features.base import BaseFeature
|
||||||
|
|
||||||
|
|
||||||
|
def register_feature(cls: Type[BaseFeature]) -> Type[BaseFeature]:
|
||||||
|
"""Register a legacy feature plugin in the derived-data registry."""
|
||||||
|
return register_derived(cls)
|
||||||
|
|
||||||
|
|
||||||
|
def available_features() -> list[str]:
|
||||||
|
"""Sorted names of all registered feature/derived-data plugins."""
|
||||||
|
return available_derived()
|
||||||
|
|
||||||
|
|
||||||
|
def get_feature(name: str, **params) -> BaseDerivedData:
|
||||||
|
"""Instantiate a registered feature/derived-data plugin by name."""
|
||||||
|
if name == "minute_daily_summary":
|
||||||
|
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
|
||||||
|
|
||||||
|
return MinuteDailySummaryFeature(**params)
|
||||||
|
return get_derived(name, **params)
|
||||||
|
|
||||||
|
|
||||||
|
def load_feature_module(spec: str) -> None:
|
||||||
|
"""Import an external module so its ``@register_feature`` classes register."""
|
||||||
|
load_derived_module(spec)
|
||||||
@@ -7,9 +7,9 @@ across dates (positions are stateful, unlike alphas/combos), discretizing and
|
|||||||
repairing each day's target into a tradable integer book.
|
repairing each day's target into a tradable integer book.
|
||||||
|
|
||||||
Return-convention note: weights here are *target allocations*. The research
|
Return-convention note: weights here are *target allocations*. The research
|
||||||
evaluation in :mod:`pipeline.portfolio.research` marks them close-to-close on the
|
evaluation in :mod:`pipeline.portfolio.research` marks them from next open to
|
||||||
*next* period (no look-ahead); the execution simulator marks the actually-filled
|
the following open (no look-ahead); the execution simulator marks the
|
||||||
book at the next open. See those modules for details.
|
actually-filled book at the next open. See those modules for details.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|||||||
@@ -0,0 +1,50 @@
|
|||||||
|
"""Trading cost models for portfolio execution simulation."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Mapping
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
class CostModel(ABC):
|
||||||
|
"""Interface for per-name execution cost models."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
traded_shares: np.ndarray,
|
||||||
|
execution_price: np.ndarray,
|
||||||
|
side: np.ndarray,
|
||||||
|
date,
|
||||||
|
metadata: Mapping[str, object] | None = None,
|
||||||
|
) -> np.ndarray:
|
||||||
|
"""Return per-name trading cost in yuan."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SimpleProportionalCostModel(CostModel):
|
||||||
|
"""Simplified open-execution proportional cost model.
|
||||||
|
|
||||||
|
Slippage is represented as an additional cash cost. The execution price is
|
||||||
|
not adjusted by slippage, which avoids double-counting.
|
||||||
|
"""
|
||||||
|
|
||||||
|
cost_bps: float = 0.0
|
||||||
|
slippage_bps: float = 0.0
|
||||||
|
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
traded_shares: np.ndarray,
|
||||||
|
execution_price: np.ndarray,
|
||||||
|
side: np.ndarray,
|
||||||
|
date,
|
||||||
|
metadata: Mapping[str, object] | None = None,
|
||||||
|
) -> np.ndarray:
|
||||||
|
shares = np.asarray(traded_shares, dtype=np.float64)
|
||||||
|
price = np.asarray(execution_price, dtype=np.float64)
|
||||||
|
open_price = np.where(np.isfinite(price), price, 0.0)
|
||||||
|
trade_value = np.abs(shares * open_price)
|
||||||
|
return trade_value * (self.cost_bps + self.slippage_bps) / 1e4
|
||||||
@@ -6,9 +6,10 @@ trading constraints. Metrics are return / Sharpe / turnover / max-drawdown /
|
|||||||
convention that an alpha is a position weight, not a return predictor.
|
convention that an alpha is a position weight, not a return predictor.
|
||||||
|
|
||||||
Return convention (documented): the target weight formed from information at
|
Return convention (documented): the target weight formed from information at
|
||||||
date ``t`` earns the *next* period's close-to-close return, i.e. weights are
|
date ``t`` is assumed tradable at ``open[t+1]`` and held until ``open[t+2]``.
|
||||||
shifted one day relative to realized returns, so there is no look-ahead:
|
This is a costless approximation of the next-open execution path: no lots,
|
||||||
``R_t = sum_i w_{i,t} · r_{i,t+1}`` normalized by gross exposure.
|
constraints, or costs, but no credit for an overnight gap that the new signal
|
||||||
|
could not have owned.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -26,39 +27,46 @@ def evaluate_portfolio(positions_df: pd.DataFrame, data_df: pd.DataFrame) -> dic
|
|||||||
Args:
|
Args:
|
||||||
positions_df: POSITION_COLUMNS (uses ``target_weight``; zero-gross
|
positions_df: POSITION_COLUMNS (uses ``target_weight``; zero-gross
|
||||||
construction carry dates remain flat in this research view).
|
construction carry dates remain flat in this research view).
|
||||||
data_df: DATA_COLUMNS (uses ``close`` for returns).
|
data_df: DATA_COLUMNS (uses ``open`` for returns).
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Dict with ``cumulative_return, sharpe_annual, turnover_annual,
|
Dict with ``cumulative_return, sharpe_annual, turnover_annual,
|
||||||
max_drawdown, fitness, hit_rate, n_dates``. No IC key.
|
max_drawdown, fitness, hit_rate, n_dates``. No IC key.
|
||||||
"""
|
"""
|
||||||
close = data_df.pivot_table(
|
open_ = data_df.pivot_table(
|
||||||
index="date", columns="symbol_id", values="close", aggfunc="first"
|
index="date", columns="symbol_id", values="open", aggfunc="first"
|
||||||
).sort_index()
|
).sort_index()
|
||||||
returns = close.pct_change()
|
fwd = open_.shift(-2).divide(open_.shift(-1)) - 1.0
|
||||||
|
|
||||||
weights = positions_df.pivot_table(
|
weights = positions_df.pivot_table(
|
||||||
index="date", columns="symbol_id", values="target_weight", aggfunc="first"
|
index="date", columns="symbol_id", values="target_weight", aggfunc="first"
|
||||||
).sort_index()
|
).sort_index()
|
||||||
|
|
||||||
common = weights.index.intersection(returns.index)
|
common = weights.index.intersection(open_.index)
|
||||||
weights = weights.loc[common]
|
weights = weights.loc[common]
|
||||||
returns = returns.loc[common]
|
# Compute forward returns on the full market calendar before selecting
|
||||||
|
# signal dates. This preserves the next available open-to-open holding
|
||||||
|
# interval when the signal grid is sparser than the data grid.
|
||||||
|
fwd = fwd.reindex(common)
|
||||||
|
|
||||||
empty = {
|
empty = {
|
||||||
"cumulative_return": 0.0, "sharpe_annual": 0.0, "turnover_annual": 0.0,
|
"cumulative_return": 0.0, "sharpe_annual": 0.0, "turnover_annual": 0.0,
|
||||||
"max_drawdown": 0.0, "fitness": 0.0, "hit_rate": 0.0,
|
"max_drawdown": 0.0, "fitness": 0.0, "hit_rate": 0.0,
|
||||||
"n_dates": len(common),
|
"n_dates": len(common),
|
||||||
}
|
}
|
||||||
if len(common) < 3:
|
if len(common) < 1:
|
||||||
|
empty["n_dates"] = 0
|
||||||
return empty
|
return empty
|
||||||
|
|
||||||
gross = weights.abs().sum(axis=1)
|
gross = weights.abs().sum(axis=1)
|
||||||
# Weights at t earn the return from t to t+1: shift returns back by one.
|
# Weights at t earn the costless tradable interval open[t+1] -> open[t+2].
|
||||||
fwd = returns.shift(-1)
|
daily = (
|
||||||
daily = (weights * fwd).sum(axis=1) / gross.replace(0.0, np.nan)
|
(weights * fwd).sum(axis=1, min_count=1)
|
||||||
|
/ gross.replace(0.0, np.nan)
|
||||||
|
)
|
||||||
daily = daily.dropna()
|
daily = daily.dropna()
|
||||||
if len(daily) < 2:
|
if len(daily) < 2:
|
||||||
|
empty["n_dates"] = int(len(daily))
|
||||||
return empty
|
return empty
|
||||||
|
|
||||||
cumulative_return = float((1.0 + daily).prod() - 1.0)
|
cumulative_return = float((1.0 + daily).prod() - 1.0)
|
||||||
|
|||||||
@@ -4,7 +4,8 @@ Execution model (documented convention): a position book targeted from
|
|||||||
information available on date ``t`` is executed at ``open[t+1]``. Trades that
|
information available on date ``t`` is executed at ``open[t+1]``. Trades that
|
||||||
violate a :class:`~pipeline.portfolio.constraints.TradeConstraint` (suspension,
|
violate a :class:`~pipeline.portfolio.constraints.TradeConstraint` (suspension,
|
||||||
price limit, volume cap, …) are clipped; a fully blocked buy leaves the position
|
price limit, volume cap, …) are clipped; a fully blocked buy leaves the position
|
||||||
at its previous level. Realized PnL marks the *actually filled* book.
|
at its previous level. Realized PnL marks the *actually filled* book. Trading
|
||||||
|
cost defaults to a simplified open-execution proportional cash-cost model.
|
||||||
|
|
||||||
The simulator is an ABC + a :class:`ReferenceSimulator`; constraints compose by
|
The simulator is an ABC + a :class:`ReferenceSimulator`; constraints compose by
|
||||||
intersecting their per-name signed delta bounds.
|
intersecting their per-name signed delta bounds.
|
||||||
@@ -21,6 +22,7 @@ import pandas as pd
|
|||||||
|
|
||||||
from pipeline.common.schema import FILL_COLUMNS, PNL_COLUMNS
|
from pipeline.common.schema import FILL_COLUMNS, PNL_COLUMNS
|
||||||
from pipeline.portfolio.constraints import TradeConstraint
|
from pipeline.portfolio.constraints import TradeConstraint
|
||||||
|
from pipeline.portfolio.costs import CostModel, SimpleProportionalCostModel
|
||||||
from pipeline.portfolio.market_rules import MarketRule, compute_limit_status
|
from pipeline.portfolio.market_rules import MarketRule, compute_limit_status
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@@ -65,10 +67,12 @@ class ExecutionSimulator(ABC):
|
|||||||
"""Abstract execution layer. Subclasses define how a target gets filled."""
|
"""Abstract execution layer. Subclasses define how a target gets filled."""
|
||||||
|
|
||||||
def __init__(self, constraints: list[TradeConstraint] | None = None,
|
def __init__(self, constraints: list[TradeConstraint] | None = None,
|
||||||
cost_bps: float = 0.0, slippage_bps: float = 0.0):
|
cost_bps: float = 0.0, slippage_bps: float = 0.0,
|
||||||
|
cost_model: CostModel | None = None):
|
||||||
self.constraints = constraints or []
|
self.constraints = constraints or []
|
||||||
self.cost_bps = cost_bps
|
self.cost_model = cost_model or SimpleProportionalCostModel(
|
||||||
self.slippage_bps = slippage_bps
|
cost_bps=cost_bps, slippage_bps=slippage_bps
|
||||||
|
)
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def fill(self, ctx: TradeContext) -> FillResult:
|
def fill(self, ctx: TradeContext) -> FillResult:
|
||||||
@@ -104,9 +108,17 @@ class ReferenceSimulator(ExecutionSimulator):
|
|||||||
blocked = (traded != desired).astype(np.int64)
|
blocked = (traded != desired).astype(np.int64)
|
||||||
|
|
||||||
realized = prev + traded
|
realized = prev + traded
|
||||||
open_px = np.where(np.isfinite(ctx.slice.price), ctx.slice.price, 0.0)
|
cost = self.cost_model.compute(
|
||||||
trade_value = np.abs(traded.astype(np.float64) * open_px)
|
traded_shares=traded,
|
||||||
cost = trade_value * (self.cost_bps + self.slippage_bps) / 1e4
|
execution_price=ctx.slice.price,
|
||||||
|
side=np.sign(traded),
|
||||||
|
date=ctx.slice.date,
|
||||||
|
metadata={
|
||||||
|
"symbol_ids": ctx.slice.symbol_ids,
|
||||||
|
"booksize": ctx.booksize,
|
||||||
|
"market_slice": ctx.slice,
|
||||||
|
},
|
||||||
|
)
|
||||||
return FillResult(realized, traded, cost, blocked)
|
return FillResult(realized, traded, cost, blocked)
|
||||||
|
|
||||||
def run(
|
def run(
|
||||||
@@ -154,13 +166,14 @@ class ReferenceSimulator(ExecutionSimulator):
|
|||||||
st = wide(data_df, "isST") if "isST" in data_df.columns else opn * 0.0
|
st = wide(data_df, "isST") if "isST" in data_df.columns else opn * 0.0
|
||||||
|
|
||||||
symbols = sorted(set(tgt.columns) | set(opn.columns))
|
symbols = sorted(set(tgt.columns) | set(opn.columns))
|
||||||
|
data_index = close.index
|
||||||
tgt = tgt.reindex(columns=symbols)
|
tgt = tgt.reindex(columns=symbols)
|
||||||
opn = opn.reindex(columns=symbols)
|
opn = opn.reindex(index=data_index, columns=symbols)
|
||||||
close = close.reindex(columns=symbols)
|
close = close.reindex(columns=symbols)
|
||||||
preclose = preclose.reindex(columns=symbols)
|
preclose = preclose.reindex(index=data_index, columns=symbols)
|
||||||
amount = amount.reindex(columns=symbols)
|
amount = amount.reindex(index=data_index, columns=symbols)
|
||||||
tstat = tstat.reindex(columns=symbols)
|
tstat = tstat.reindex(index=data_index, columns=symbols)
|
||||||
st = st.reindex(columns=symbols)
|
st = st.reindex(index=data_index, columns=symbols)
|
||||||
|
|
||||||
sym_arr = np.asarray(symbols, dtype=object)
|
sym_arr = np.asarray(symbols, dtype=object)
|
||||||
n = len(symbols)
|
n = len(symbols)
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
"""Optional plugin packages for the research pipeline."""
|
||||||
|
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
# JoinQuant Comparison Plugin
|
||||||
|
|
||||||
|
This plugin exports frozen targets from the internal A-share research pipeline,
|
||||||
|
drives a standalone JoinQuant wrapper strategy, ingests JoinQuant output files,
|
||||||
|
and reconciles them against the internal reference simulator.
|
||||||
|
|
||||||
|
The plugin validates system mechanics, not alpha quality:
|
||||||
|
|
||||||
|
- date alignment
|
||||||
|
- symbol mapping
|
||||||
|
- target position generation
|
||||||
|
- open execution timing
|
||||||
|
- lot rounding and filled shares
|
||||||
|
- position carry
|
||||||
|
- trading cost and PnL accounting
|
||||||
|
- blocked trades from suspension and price limits
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py joinquant prepare-smoke \
|
||||||
|
--out-dir /tmp/chinese-equity-quant-realdata
|
||||||
|
|
||||||
|
uv sync --extra joinquant-browser
|
||||||
|
uv run playwright install chromium
|
||||||
|
|
||||||
|
uv run python cli.py joinquant browser-login \
|
||||||
|
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
|
||||||
|
|
||||||
|
uv run python cli.py joinquant write-browser-config \
|
||||||
|
--out-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
|
||||||
|
--strategy-url "https://www.joinquant.com/..." \
|
||||||
|
--flow backtest
|
||||||
|
|
||||||
|
uv run python cli.py joinquant run-browser-backtest \
|
||||||
|
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
|
||||||
|
--config-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
|
||||||
|
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
|
||||||
|
|
||||||
|
uv run python cli.py joinquant write-browser-config \
|
||||||
|
--out-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
|
||||||
|
--strategy-url "https://www.joinquant.com/<模拟盘 page>" \
|
||||||
|
--flow sim-trade
|
||||||
|
|
||||||
|
uv run python cli.py joinquant run-browser-sim \
|
||||||
|
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
|
||||||
|
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
|
||||||
|
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
|
||||||
|
|
||||||
|
uv run python cli.py joinquant export-targets \
|
||||||
|
--positions-path portfolio/run1.pq \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--mode target_shares \
|
||||||
|
--execution-calendar-path data/daily_bars/csi500 \
|
||||||
|
--start-date 2026-07-01 \
|
||||||
|
--end-date 2026-07-31 \
|
||||||
|
--out-dir plugins_output/joinquant/targets
|
||||||
|
|
||||||
|
uv run python cli.py joinquant write-wrapper \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--mode target_shares \
|
||||||
|
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
|
||||||
|
|
||||||
|
uv run python cli.py joinquant ingest \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--fills-csv path/to/jq_fills.csv \
|
||||||
|
--positions-csv path/to/jq_positions.csv \
|
||||||
|
--pnl-csv path/to/jq_pnl.csv \
|
||||||
|
--out-dir plugins_output/joinquant/ingested
|
||||||
|
|
||||||
|
uv run python cli.py joinquant reconcile \
|
||||||
|
--portfolio-name run1 \
|
||||||
|
--targets-dir plugins_output/joinquant/targets/run1 \
|
||||||
|
--our-fills-path fills/run1.pq \
|
||||||
|
--our-positions-path portfolio/run1.pq \
|
||||||
|
--our-pnl-path pnl/run1.pq \
|
||||||
|
--jq-fills-path plugins_output/joinquant/ingested/run1/fills.pq \
|
||||||
|
--jq-positions-path plugins_output/joinquant/ingested/run1/positions.pq \
|
||||||
|
--jq-pnl-path plugins_output/joinquant/ingested/run1/pnl.pq \
|
||||||
|
--out-dir plugins_output/joinquant/reconcile
|
||||||
|
```
|
||||||
|
|
||||||
|
`target_shares` is the default and uses the built integer `position_shares`
|
||||||
|
from `portfolio build`, matching what the internal simulator executes.
|
||||||
|
For strict simulator-vs-JoinQuant comparison, pass `--execution-calendar-path`
|
||||||
|
so position dates are shifted to the next session open, matching the internal
|
||||||
|
simulator's next-open convention.
|
||||||
|
|
||||||
|
`prepare-smoke` automates the local side of the first sanity check: tiny real
|
||||||
|
data download, one-stock long-only position file, internal simulation, aligned
|
||||||
|
target export, wrapper generation, and a manifest with expected JoinQuant CSV
|
||||||
|
export paths.
|
||||||
|
|
||||||
|
`run-browser-backtest` automates the remote JoinQuant web run through
|
||||||
|
Playwright. It reuses a saved browser login state, executes the configured UI
|
||||||
|
actions, downloads JoinQuant CSVs when configured, and runs ingest/reconcile
|
||||||
|
automatically once all three CSVs are present.
|
||||||
|
|
||||||
|
`run-browser-sim` is the forward-test / 模拟盘 equivalent. Use a `--flow
|
||||||
|
sim-trade` config to upload the frozen next-session target file, save the
|
||||||
|
strategy, and start or restart the JoinQuant simulated-trading job. After close,
|
||||||
|
run it again with download actions or use `ingest` / `reconcile` directly on
|
||||||
|
exported CSVs.
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
"""JoinQuant comparison plugin.
|
||||||
|
|
||||||
|
This package keeps JoinQuant-specific export, ingest, and reconciliation code
|
||||||
|
outside the core portfolio modules.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from plugins.joinquant.symbols import from_joinquant_symbol, to_joinquant_symbol
|
||||||
|
|
||||||
|
__all__ = ["from_joinquant_symbol", "to_joinquant_symbol"]
|
||||||
|
|
||||||
@@ -0,0 +1,710 @@
|
|||||||
|
"""Browser automation for JoinQuant cloud backtest and simulated-trading runs.
|
||||||
|
|
||||||
|
JoinQuant's public ``jqdatasdk`` is a data API; cloud strategy upload/run/export
|
||||||
|
is exposed through the web application. This module keeps that automation
|
||||||
|
optional and configurable so UI selector drift does not affect the core test
|
||||||
|
suite.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from plugins.joinquant.ingest import ingest_joinquant_outputs
|
||||||
|
from plugins.joinquant.reconcile import reconcile_joinquant
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_LOGIN_URL = "https://www.joinquant.com/user/login/index"
|
||||||
|
|
||||||
|
|
||||||
|
def _require_playwright():
|
||||||
|
try:
|
||||||
|
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
except ImportError as exc:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Playwright is required for JoinQuant browser automation. Install it "
|
||||||
|
"in this uv environment, then install Chromium:\n"
|
||||||
|
" uv sync --extra joinquant-browser\n"
|
||||||
|
" uv run playwright install chromium"
|
||||||
|
) from exc
|
||||||
|
return sync_playwright, PlaywrightTimeoutError
|
||||||
|
|
||||||
|
|
||||||
|
def _backtest_actions() -> list[dict[str, Any]]:
|
||||||
|
return [
|
||||||
|
{"type": "goto", "url": "{strategy_url}"},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": "text=不再提示",
|
||||||
|
"timeout_ms": 5000,
|
||||||
|
"force": True,
|
||||||
|
"optional": True,
|
||||||
|
"description": "Dismiss JoinQuant first-run editor guide.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": "text=确定",
|
||||||
|
"timeout_ms": 5000,
|
||||||
|
"force": True,
|
||||||
|
"optional": True,
|
||||||
|
"description": "Close JoinQuant first-run editor guide.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": "text=跳过",
|
||||||
|
"timeout_ms": 5000,
|
||||||
|
"force": True,
|
||||||
|
"optional": True,
|
||||||
|
"description": "Skip JoinQuant first-run editor guide.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "evaluate",
|
||||||
|
"script": (
|
||||||
|
"() => {"
|
||||||
|
"document.querySelectorAll("
|
||||||
|
"'.introjs-overlay,.introjs-helperLayer,.introjs-tooltipReferenceLayer,"
|
||||||
|
".introjs-tooltip,.introjs-disableInteraction'"
|
||||||
|
").forEach((node) => node.remove());"
|
||||||
|
"document.body.classList.remove('introjs-noscroll');"
|
||||||
|
"}"
|
||||||
|
),
|
||||||
|
"optional": True,
|
||||||
|
"description": "Remove any remaining JoinQuant intro overlay.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "set_ace_editor_file",
|
||||||
|
"selector": ".ace_editor",
|
||||||
|
"path": "{wrapper_path}",
|
||||||
|
"description": "Set generated wrapper strategy in the Ace code editor.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "set_input_files",
|
||||||
|
"selector": "input[type=file]",
|
||||||
|
"paths": "{target_csvs}",
|
||||||
|
"description": "Upload all aligned daily target CSV files.",
|
||||||
|
"timeout_ms": 5000,
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "evaluate",
|
||||||
|
"script": (
|
||||||
|
"(arg) => {"
|
||||||
|
"const setValue = (selector, value) => {"
|
||||||
|
"const el = document.querySelector(selector);"
|
||||||
|
"if (!el) return false;"
|
||||||
|
"el.value = value;"
|
||||||
|
"el.setAttribute('value', value);"
|
||||||
|
"el.dispatchEvent(new Event('input', {bubbles: true}));"
|
||||||
|
"el.dispatchEvent(new Event('change', {bubbles: true}));"
|
||||||
|
"return true;"
|
||||||
|
"};"
|
||||||
|
"return {"
|
||||||
|
"start: setValue('#startTime', arg.start),"
|
||||||
|
"end: setValue('#endTime', arg.end),"
|
||||||
|
"capital: setValue('#daily_backtest_capital_base_box', arg.capital)"
|
||||||
|
"};"
|
||||||
|
"}"
|
||||||
|
),
|
||||||
|
"arg": {
|
||||||
|
"start": "{backtest_start_date}",
|
||||||
|
"end": "{backtest_end_date}",
|
||||||
|
"capital": "{booksize}",
|
||||||
|
},
|
||||||
|
"description": "Set JoinQuant backtest dates and starting capital.",
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": "#algo-save-button",
|
||||||
|
"timeout_ms": 5000,
|
||||||
|
"description": "Save the JoinQuant strategy code.",
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": ".bootstrap-dialog .btn-primary, .modal.in button:has-text(\"确定\")",
|
||||||
|
"timeout_ms": 5000,
|
||||||
|
"force": True,
|
||||||
|
"description": "Confirm any JoinQuant save dialog.",
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{"type": "wait_for_timeout", "timeout_ms": 2000},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": "#daily-new-backtest-button",
|
||||||
|
"description": "Start the JoinQuant backtest.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "wait_for_selector",
|
||||||
|
"selector": "text=/回测完成|运行完成|Backtest complete|Finished/i",
|
||||||
|
"timeout_ms": 600_000,
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "download",
|
||||||
|
"selector": "text=/导出成交|下载成交|fills|trades/i",
|
||||||
|
"save_as": "{expected_joinquant_csvs.fills}",
|
||||||
|
"timeout_ms": 15000,
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "download",
|
||||||
|
"selector": "text=/导出持仓|下载持仓|positions/i",
|
||||||
|
"save_as": "{expected_joinquant_csvs.positions}",
|
||||||
|
"timeout_ms": 15000,
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "download",
|
||||||
|
"selector": "text=/导出收益|下载收益|pnl|收益/i",
|
||||||
|
"save_as": "{expected_joinquant_csvs.pnl}",
|
||||||
|
"timeout_ms": 15000,
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{"type": "screenshot", "path": "{run_artifact_dir}/final.png"},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _sim_trade_actions() -> list[dict[str, Any]]:
|
||||||
|
return [
|
||||||
|
{"type": "goto", "url": "{strategy_url}"},
|
||||||
|
{
|
||||||
|
"type": "paste_text_file",
|
||||||
|
"selector": "textarea, .ace_text-input, .cm-content, .CodeMirror textarea",
|
||||||
|
"path": "{wrapper_path}",
|
||||||
|
"description": "Paste generated wrapper strategy into the simulated-trading strategy editor.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "set_input_files",
|
||||||
|
"selector": "input[type=file]",
|
||||||
|
"paths": "{target_csvs}",
|
||||||
|
"description": "Upload frozen target CSV files for 模拟盘.",
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": "text=/保存|Save/i",
|
||||||
|
"description": "Save the strategy code and uploaded files.",
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "wait_for_selector",
|
||||||
|
"selector": "text=/保存成功|已保存|Saved/i",
|
||||||
|
"timeout_ms": 120_000,
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "click",
|
||||||
|
"selector": "text=/模拟盘|模拟交易|启动模拟|运行模拟|启动|重启|Run Sim|Start|Restart/i",
|
||||||
|
"description": "Start or restart the JoinQuant simulated-trading job.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "wait_for_selector",
|
||||||
|
"selector": "text=/运行中|已启动|模拟交易运行|Started|Running/i",
|
||||||
|
"timeout_ms": 180_000,
|
||||||
|
"optional": True,
|
||||||
|
},
|
||||||
|
{"type": "screenshot", "path": "{run_artifact_dir}/sim_trade_final.png"},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def default_browser_config(strategy_url: str = "", *, flow: str = "backtest") -> dict[str, Any]:
|
||||||
|
"""Return a selector/action template for JoinQuant browser automation."""
|
||||||
|
if flow not in {"backtest", "sim-trade", "sim_trade"}:
|
||||||
|
raise ValueError("flow must be 'backtest' or 'sim-trade'")
|
||||||
|
normalized_flow = "sim-trade" if flow == "sim_trade" else flow
|
||||||
|
actions = _sim_trade_actions() if normalized_flow == "sim-trade" else _backtest_actions()
|
||||||
|
return {
|
||||||
|
"flow": normalized_flow,
|
||||||
|
"strategy_url": strategy_url,
|
||||||
|
"login_url": DEFAULT_LOGIN_URL,
|
||||||
|
"headless": False,
|
||||||
|
"timeout_ms": 120_000,
|
||||||
|
"notes": [
|
||||||
|
"Fill strategy_url and selectors after inspecting your JoinQuant strategy page.",
|
||||||
|
"Use `joinquant browser-snapshot` to capture HTML/screenshots for selector tuning.",
|
||||||
|
"The action list is declarative and runs in order.",
|
||||||
|
],
|
||||||
|
"actions": actions,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def write_browser_config_template(
|
||||||
|
path: str | Path,
|
||||||
|
*,
|
||||||
|
strategy_url: str = "",
|
||||||
|
flow: str = "backtest",
|
||||||
|
) -> Path:
|
||||||
|
"""Write a JSON config template for browser automation."""
|
||||||
|
out_path = Path(path)
|
||||||
|
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
out_path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
default_browser_config(strategy_url, flow=flow),
|
||||||
|
indent=2,
|
||||||
|
ensure_ascii=False,
|
||||||
|
) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return out_path
|
||||||
|
|
||||||
|
|
||||||
|
def load_json(path: str | Path) -> dict[str, Any]:
|
||||||
|
"""Load a JSON object from disk."""
|
||||||
|
data = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
raise ValueError(f"Expected JSON object in {path}")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def load_env_file(path: str | Path) -> dict[str, str]:
|
||||||
|
"""Load simple KEY=VALUE env files without requiring shell-safe quoting."""
|
||||||
|
values: dict[str, str] = {}
|
||||||
|
for raw in Path(path).expanduser().read_text(encoding="utf-8").splitlines():
|
||||||
|
line = raw.strip()
|
||||||
|
if not line or line.startswith("#") or "=" not in line:
|
||||||
|
continue
|
||||||
|
key, value = line.split("=", 1)
|
||||||
|
clean = value.strip()
|
||||||
|
if len(clean) >= 2 and clean[0] == clean[-1] and clean[0] in {"'", '"'}:
|
||||||
|
clean = clean[1:-1]
|
||||||
|
else:
|
||||||
|
clean = clean.strip("'\"")
|
||||||
|
values[key.strip()] = clean
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def _get_dotted(data: dict[str, Any], dotted: str) -> Any:
|
||||||
|
current: Any = data
|
||||||
|
for part in dotted.split("."):
|
||||||
|
if isinstance(current, dict) and part in current:
|
||||||
|
current = current[part]
|
||||||
|
else:
|
||||||
|
raise KeyError(dotted)
|
||||||
|
return current
|
||||||
|
|
||||||
|
|
||||||
|
def _manifest_context(manifest: dict[str, Any], artifact_dir: Path, config: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
targets_dir = Path(str(manifest["targets_dir"]))
|
||||||
|
target_csvs = [str(path) for path in sorted(targets_dir.glob("*.csv"))]
|
||||||
|
if not target_csvs:
|
||||||
|
raise ValueError(f"No target CSV files found under {targets_dir}")
|
||||||
|
|
||||||
|
target_dates = [
|
||||||
|
datetime.strptime(Path(path).stem, "%Y%m%d").strftime("%Y-%m-%d")
|
||||||
|
for path in target_csvs
|
||||||
|
]
|
||||||
|
context = dict(manifest)
|
||||||
|
context.update({
|
||||||
|
"strategy_url": config.get("strategy_url", ""),
|
||||||
|
"target_csvs": target_csvs,
|
||||||
|
"target_csvs_csv": ",".join(target_csvs),
|
||||||
|
"backtest_start_date": config.get("backtest_start_date") or min(target_dates),
|
||||||
|
"backtest_end_date": config.get("backtest_end_date") or max(target_dates),
|
||||||
|
"run_artifact_dir": str(artifact_dir),
|
||||||
|
})
|
||||||
|
return context
|
||||||
|
|
||||||
|
|
||||||
|
_TOKEN_RE = re.compile(r"^\{([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)\}$")
|
||||||
|
_PARTIAL_TOKEN_RE = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)\}")
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_template(value: Any, context: dict[str, Any]) -> Any:
|
||||||
|
"""Resolve ``{tokens}`` in config values against the manifest context."""
|
||||||
|
if isinstance(value, str):
|
||||||
|
whole = _TOKEN_RE.match(value)
|
||||||
|
if whole:
|
||||||
|
return _get_dotted(context, whole.group(1))
|
||||||
|
|
||||||
|
def replace(match: re.Match[str]) -> str:
|
||||||
|
resolved = _get_dotted(context, match.group(1))
|
||||||
|
return str(resolved)
|
||||||
|
|
||||||
|
return _PARTIAL_TOKEN_RE.sub(replace, value)
|
||||||
|
if isinstance(value, list):
|
||||||
|
return [resolve_template(item, context) for item in value]
|
||||||
|
if isinstance(value, dict):
|
||||||
|
return {key: resolve_template(val, context) for key, val in value.items()}
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def save_login_state(
|
||||||
|
*,
|
||||||
|
storage_state: str | Path,
|
||||||
|
login_url: str = DEFAULT_LOGIN_URL,
|
||||||
|
headless: bool = False,
|
||||||
|
wait_seconds: int = 0,
|
||||||
|
) -> Path:
|
||||||
|
"""Open a browser for manual login and save the authenticated state."""
|
||||||
|
if headless and wait_seconds <= 0:
|
||||||
|
raise ValueError("headless browser-login requires --wait-seconds > 0")
|
||||||
|
sync_playwright, _ = _require_playwright()
|
||||||
|
state_path = Path(storage_state).expanduser()
|
||||||
|
state_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with sync_playwright() as pw:
|
||||||
|
browser = pw.chromium.launch(headless=headless)
|
||||||
|
context = browser.new_context()
|
||||||
|
page = context.new_page()
|
||||||
|
page.goto(login_url, wait_until="domcontentloaded")
|
||||||
|
if wait_seconds > 0:
|
||||||
|
page.wait_for_timeout(wait_seconds * 1000)
|
||||||
|
else:
|
||||||
|
input("Log in to JoinQuant in the opened browser, then press Enter here to save state...")
|
||||||
|
context.storage_state(path=str(state_path))
|
||||||
|
browser.close()
|
||||||
|
state_path.chmod(0o600)
|
||||||
|
return state_path
|
||||||
|
|
||||||
|
|
||||||
|
def save_login_state_from_env(
|
||||||
|
*,
|
||||||
|
env_path: str | Path,
|
||||||
|
storage_state: str | Path,
|
||||||
|
login_url: str = DEFAULT_LOGIN_URL,
|
||||||
|
headless: bool = True,
|
||||||
|
out_dir: str | Path | None = None,
|
||||||
|
timeout_ms: int = 120_000,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Try to log in with env-file credentials and save browser state.
|
||||||
|
|
||||||
|
This handles the normal password-login form. If JoinQuant presents CAPTCHA,
|
||||||
|
slide verification, SMS verification, or 2FA, the report will mark the run
|
||||||
|
as not logged in and save a screenshot for manual diagnosis.
|
||||||
|
"""
|
||||||
|
env = load_env_file(env_path)
|
||||||
|
username = env.get("JOINQUANT_USERNAME")
|
||||||
|
password = env.get("JOINQUANT_PASSWORD")
|
||||||
|
if not username or not password:
|
||||||
|
raise ValueError("JOINQUANT_USERNAME and JOINQUANT_PASSWORD are required")
|
||||||
|
|
||||||
|
sync_playwright, _ = _require_playwright()
|
||||||
|
state_path = Path(storage_state).expanduser()
|
||||||
|
state_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
artifact_dir = Path(out_dir or state_path.parent / "joinquant_login_artifacts")
|
||||||
|
artifact_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
screenshot_path = artifact_dir / "login_after_submit.png"
|
||||||
|
html_path = artifact_dir / "login_after_submit.html"
|
||||||
|
|
||||||
|
with sync_playwright() as pw:
|
||||||
|
browser = pw.chromium.launch(headless=headless)
|
||||||
|
context = browser.new_context()
|
||||||
|
page = context.new_page()
|
||||||
|
page.set_default_timeout(timeout_ms)
|
||||||
|
page.goto(login_url, wait_until="networkidle", timeout=timeout_ms)
|
||||||
|
page.locator('input[name="username"], input.pwd-phone').first.fill(username)
|
||||||
|
page.locator('input[name="pwd"], input.jq-login__password').first.fill(password)
|
||||||
|
checkbox = page.locator("#agreementBox, input.agreement-box").first
|
||||||
|
if checkbox.count():
|
||||||
|
checkbox.check(force=True)
|
||||||
|
page.locator("button.btnPwdSubmit, button.login-submit").first.click()
|
||||||
|
page.wait_for_timeout(5000)
|
||||||
|
|
||||||
|
html = page.content()
|
||||||
|
html_path.write_text(html, encoding="utf-8")
|
||||||
|
page.screenshot(path=str(screenshot_path), full_page=True)
|
||||||
|
login_inputs = page.locator('input[name="username"], input[name="pwd"]').count()
|
||||||
|
current_url = page.url
|
||||||
|
logged_in = login_inputs == 0 and "/user/login" not in current_url
|
||||||
|
if logged_in:
|
||||||
|
context.storage_state(path=str(state_path))
|
||||||
|
state_path.chmod(0o600)
|
||||||
|
browser.close()
|
||||||
|
|
||||||
|
report = {
|
||||||
|
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"logged_in": bool(logged_in),
|
||||||
|
"storage_state": str(state_path) if logged_in else "",
|
||||||
|
"artifact_dir": str(artifact_dir),
|
||||||
|
"screenshot": str(screenshot_path),
|
||||||
|
"html": str(html_path),
|
||||||
|
"current_url": current_url,
|
||||||
|
"notes": (
|
||||||
|
"Logged in and saved browser state."
|
||||||
|
if logged_in
|
||||||
|
else "Login did not complete. CAPTCHA/SMS/2FA or invalid credentials may be required."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
report_path = artifact_dir / "login_report.json"
|
||||||
|
report["report_path"] = str(report_path)
|
||||||
|
report_path.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def browser_snapshot(
|
||||||
|
*,
|
||||||
|
url: str,
|
||||||
|
storage_state: str | Path,
|
||||||
|
out_dir: str | Path,
|
||||||
|
headless: bool = True,
|
||||||
|
timeout_ms: int = 60_000,
|
||||||
|
) -> dict[str, Path]:
|
||||||
|
"""Save a logged-in page screenshot and HTML for selector discovery."""
|
||||||
|
sync_playwright, _ = _require_playwright()
|
||||||
|
root = Path(out_dir)
|
||||||
|
root.mkdir(parents=True, exist_ok=True)
|
||||||
|
html_path = root / "page.html"
|
||||||
|
screenshot_path = root / "page.png"
|
||||||
|
|
||||||
|
with sync_playwright() as pw:
|
||||||
|
browser = pw.chromium.launch(headless=headless)
|
||||||
|
context = browser.new_context(storage_state=str(Path(storage_state).expanduser()))
|
||||||
|
page = context.new_page()
|
||||||
|
page.set_default_timeout(timeout_ms)
|
||||||
|
page.goto(url, wait_until="networkidle")
|
||||||
|
html_path.write_text(page.content(), encoding="utf-8")
|
||||||
|
page.screenshot(path=str(screenshot_path), full_page=True)
|
||||||
|
browser.close()
|
||||||
|
return {"html": html_path, "screenshot": screenshot_path}
|
||||||
|
|
||||||
|
|
||||||
|
def _locator(page: Any, selector: str):
|
||||||
|
return page.locator(selector).first
|
||||||
|
|
||||||
|
|
||||||
|
def _action_fail(action: dict[str, Any], exc: Exception) -> None:
|
||||||
|
if action.get("optional", False):
|
||||||
|
return
|
||||||
|
raise exc
|
||||||
|
|
||||||
|
|
||||||
|
def _run_action(page: Any, action: dict[str, Any], context: dict[str, Any], timeout_default: int) -> dict[str, Any]:
|
||||||
|
resolved = resolve_template(action, context)
|
||||||
|
kind = resolved["type"]
|
||||||
|
timeout_ms = int(resolved.get("timeout_ms", timeout_default))
|
||||||
|
record: dict[str, Any] = {"type": kind, "status": "ok"}
|
||||||
|
try:
|
||||||
|
if kind == "goto":
|
||||||
|
page.goto(str(resolved["url"]), wait_until=resolved.get("wait_until", "domcontentloaded"), timeout=timeout_ms)
|
||||||
|
elif kind == "click":
|
||||||
|
_locator(page, str(resolved["selector"])).click(
|
||||||
|
timeout=timeout_ms,
|
||||||
|
force=bool(resolved.get("force", False)),
|
||||||
|
)
|
||||||
|
elif kind == "fill":
|
||||||
|
_locator(page, str(resolved["selector"])).fill(str(resolved.get("text", "")), timeout=timeout_ms)
|
||||||
|
elif kind == "press":
|
||||||
|
_locator(page, str(resolved["selector"])).press(str(resolved["key"]), timeout=timeout_ms)
|
||||||
|
elif kind == "set_input_files":
|
||||||
|
paths = resolved.get("paths", [])
|
||||||
|
if isinstance(paths, str):
|
||||||
|
paths = [path for path in paths.split(",") if path]
|
||||||
|
_locator(page, str(resolved["selector"])).set_input_files(paths, timeout=timeout_ms)
|
||||||
|
record["n_files"] = len(paths)
|
||||||
|
elif kind == "paste_text_file":
|
||||||
|
text = Path(str(resolved["path"])).read_text(encoding="utf-8")
|
||||||
|
loc = _locator(page, str(resolved["selector"]))
|
||||||
|
loc.click(timeout=timeout_ms)
|
||||||
|
page.keyboard.press("Control+A")
|
||||||
|
page.keyboard.insert_text(text)
|
||||||
|
record["n_chars"] = len(text)
|
||||||
|
elif kind == "set_ace_editor_file":
|
||||||
|
text = Path(str(resolved["path"])).read_text(encoding="utf-8")
|
||||||
|
selector = str(resolved.get("selector", ".ace_editor"))
|
||||||
|
page.wait_for_function(
|
||||||
|
"(selector) => Boolean(window.ace && document.querySelector(selector))",
|
||||||
|
arg=selector,
|
||||||
|
timeout=timeout_ms,
|
||||||
|
)
|
||||||
|
page.evaluate(
|
||||||
|
"""(arg) => {
|
||||||
|
const node = document.querySelector(arg.selector);
|
||||||
|
if (!node || !window.ace) {
|
||||||
|
throw new Error(`Ace editor not found for ${arg.selector}`);
|
||||||
|
}
|
||||||
|
const editor = window.ace.edit(node);
|
||||||
|
editor.setValue(arg.text, -1);
|
||||||
|
editor.clearSelection();
|
||||||
|
editor.focus();
|
||||||
|
return editor.getValue().length;
|
||||||
|
}""",
|
||||||
|
{"selector": selector, "text": text},
|
||||||
|
)
|
||||||
|
record["n_chars"] = len(text)
|
||||||
|
elif kind == "evaluate":
|
||||||
|
page.evaluate(str(resolved["script"]), resolved.get("arg"))
|
||||||
|
elif kind == "wait_for_selector":
|
||||||
|
page.wait_for_selector(str(resolved["selector"]), timeout=timeout_ms)
|
||||||
|
elif kind == "wait_for_timeout":
|
||||||
|
page.wait_for_timeout(int(resolved.get("timeout_ms", 1000)))
|
||||||
|
elif kind == "download":
|
||||||
|
save_as = Path(str(resolved["save_as"]))
|
||||||
|
save_as.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with page.expect_download(timeout=timeout_ms) as download_info:
|
||||||
|
_locator(page, str(resolved["selector"])).click(timeout=timeout_ms)
|
||||||
|
download = download_info.value
|
||||||
|
download.save_as(str(save_as))
|
||||||
|
record["path"] = str(save_as)
|
||||||
|
elif kind == "screenshot":
|
||||||
|
path = Path(str(resolved["path"]))
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
page.screenshot(path=str(path), full_page=bool(resolved.get("full_page", True)))
|
||||||
|
record["path"] = str(path)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unsupported browser action type: {kind}")
|
||||||
|
except Exception as exc: # pragma: no cover - exercised only with Playwright.
|
||||||
|
record["status"] = "skipped" if resolved.get("optional", False) else "failed"
|
||||||
|
record["error"] = str(exc)
|
||||||
|
_action_fail(resolved, exc)
|
||||||
|
return record
|
||||||
|
|
||||||
|
|
||||||
|
def run_browser_flow(
|
||||||
|
*,
|
||||||
|
manifest_path: str | Path,
|
||||||
|
config_path: str | Path,
|
||||||
|
storage_state: str | Path,
|
||||||
|
out_dir: str | Path | None = None,
|
||||||
|
headless: bool | None = None,
|
||||||
|
auto_reconcile: bool = True,
|
||||||
|
flow_name: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Run configured browser automation for a JoinQuant web workflow."""
|
||||||
|
sync_playwright, _ = _require_playwright()
|
||||||
|
manifest = load_json(manifest_path)
|
||||||
|
config = load_json(config_path)
|
||||||
|
flow = flow_name or str(config.get("flow") or "browser")
|
||||||
|
artifact_dir = Path(out_dir or Path(str(manifest["joinquant_export_dir"])).parent / f"browser_{flow}")
|
||||||
|
artifact_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
context_vars = _manifest_context(manifest, artifact_dir, config)
|
||||||
|
|
||||||
|
timeout_ms = int(config.get("timeout_ms", 120_000))
|
||||||
|
actions = config.get("actions") or []
|
||||||
|
if not actions:
|
||||||
|
raise ValueError("Browser config contains no actions")
|
||||||
|
|
||||||
|
records: list[dict[str, Any]] = []
|
||||||
|
failure: Exception | None = None
|
||||||
|
failure_artifacts: dict[str, str] = {}
|
||||||
|
with sync_playwright() as pw:
|
||||||
|
browser = pw.chromium.launch(
|
||||||
|
headless=bool(config.get("headless", False) if headless is None else headless)
|
||||||
|
)
|
||||||
|
context = browser.new_context(
|
||||||
|
storage_state=str(Path(storage_state).expanduser()),
|
||||||
|
accept_downloads=True,
|
||||||
|
)
|
||||||
|
page = context.new_page()
|
||||||
|
page.set_default_timeout(timeout_ms)
|
||||||
|
try:
|
||||||
|
for action in actions:
|
||||||
|
records.append(_run_action(page, action, context_vars, timeout_ms))
|
||||||
|
except Exception as exc: # pragma: no cover - requires live browser UI.
|
||||||
|
failure = exc
|
||||||
|
screenshot_path = artifact_dir / "failure.png"
|
||||||
|
html_path = artifact_dir / "failure.html"
|
||||||
|
try:
|
||||||
|
page.screenshot(path=str(screenshot_path), full_page=True)
|
||||||
|
html_path.write_text(page.content(), encoding="utf-8")
|
||||||
|
failure_artifacts = {
|
||||||
|
"screenshot": str(screenshot_path),
|
||||||
|
"html": str(html_path),
|
||||||
|
}
|
||||||
|
except Exception as artifact_exc:
|
||||||
|
failure_artifacts = {"artifact_error": str(artifact_exc)}
|
||||||
|
records.append({
|
||||||
|
"type": "browser_flow",
|
||||||
|
"status": "failed",
|
||||||
|
"error": str(exc),
|
||||||
|
**failure_artifacts,
|
||||||
|
})
|
||||||
|
browser.close()
|
||||||
|
|
||||||
|
expected = manifest.get("expected_joinquant_csvs", {})
|
||||||
|
downloaded = {
|
||||||
|
key: str(path)
|
||||||
|
for key, value in expected.items()
|
||||||
|
if (path := Path(str(value))).exists()
|
||||||
|
}
|
||||||
|
reconcile_paths: dict[str, str] = {}
|
||||||
|
if auto_reconcile and {"fills", "positions", "pnl"}.issubset(downloaded):
|
||||||
|
ingested = ingest_joinquant_outputs(
|
||||||
|
portfolio_name=str(manifest["portfolio_name"]),
|
||||||
|
fills_csv=downloaded["fills"],
|
||||||
|
positions_csv=downloaded["positions"],
|
||||||
|
pnl_csv=downloaded["pnl"],
|
||||||
|
out_dir=Path(str(manifest["joinquant_export_dir"])).parent / "ingested",
|
||||||
|
)
|
||||||
|
reconciled = reconcile_joinquant(
|
||||||
|
portfolio_name=str(manifest["portfolio_name"]),
|
||||||
|
targets_dir=str(manifest["targets_dir"]),
|
||||||
|
our_fills_path=str(manifest["fills_path"]),
|
||||||
|
our_positions_path=str(manifest["positions_path"]),
|
||||||
|
our_pnl_path=str(manifest["pnl_path"]),
|
||||||
|
jq_fills_path=str(ingested["fills"]),
|
||||||
|
jq_positions_path=str(ingested["positions"]),
|
||||||
|
jq_pnl_path=str(ingested["pnl"]),
|
||||||
|
out_dir=Path(str(manifest["joinquant_export_dir"])).parent / "reconcile",
|
||||||
|
)
|
||||||
|
reconcile_paths = {key: str(value) for key, value in reconciled.items()}
|
||||||
|
|
||||||
|
report = {
|
||||||
|
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"manifest_path": str(manifest_path),
|
||||||
|
"config_path": str(config_path),
|
||||||
|
"flow": flow,
|
||||||
|
"status": "failed" if failure else "ok",
|
||||||
|
"storage_state": str(storage_state),
|
||||||
|
"artifact_dir": str(artifact_dir),
|
||||||
|
"actions": records,
|
||||||
|
"downloaded": downloaded,
|
||||||
|
"reconcile_paths": reconcile_paths,
|
||||||
|
}
|
||||||
|
report_path = artifact_dir / "browser_run_report.json"
|
||||||
|
report["report_path"] = str(report_path)
|
||||||
|
report_path.write_text(
|
||||||
|
json.dumps(report, indent=2, ensure_ascii=False) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
if failure is not None:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Browser flow failed; report saved to {report_path}: {failure}"
|
||||||
|
) from failure
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def run_browser_backtest(
|
||||||
|
*,
|
||||||
|
manifest_path: str | Path,
|
||||||
|
config_path: str | Path,
|
||||||
|
storage_state: str | Path,
|
||||||
|
out_dir: str | Path | None = None,
|
||||||
|
headless: bool | None = None,
|
||||||
|
auto_reconcile: bool = True,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Run configured browser automation for a JoinQuant backtest."""
|
||||||
|
return run_browser_flow(
|
||||||
|
manifest_path=manifest_path,
|
||||||
|
config_path=config_path,
|
||||||
|
storage_state=storage_state,
|
||||||
|
out_dir=out_dir,
|
||||||
|
headless=headless,
|
||||||
|
auto_reconcile=auto_reconcile,
|
||||||
|
flow_name="backtest",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run_browser_sim_trade(
|
||||||
|
*,
|
||||||
|
manifest_path: str | Path,
|
||||||
|
config_path: str | Path,
|
||||||
|
storage_state: str | Path,
|
||||||
|
out_dir: str | Path | None = None,
|
||||||
|
headless: bool | None = None,
|
||||||
|
auto_reconcile: bool = True,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Run configured browser automation for JoinQuant 模拟盘."""
|
||||||
|
return run_browser_flow(
|
||||||
|
manifest_path=manifest_path,
|
||||||
|
config_path=config_path,
|
||||||
|
storage_state=storage_state,
|
||||||
|
out_dir=out_dir,
|
||||||
|
headless=headless,
|
||||||
|
auto_reconcile=auto_reconcile,
|
||||||
|
flow_name="sim-trade",
|
||||||
|
)
|
||||||
@@ -0,0 +1,426 @@
|
|||||||
|
"""CLI commands for the JoinQuant comparison plugin."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import click
|
||||||
|
|
||||||
|
from plugins.joinquant.browser import (
|
||||||
|
browser_snapshot,
|
||||||
|
load_env_file,
|
||||||
|
run_browser_backtest,
|
||||||
|
run_browser_sim_trade,
|
||||||
|
save_login_state,
|
||||||
|
save_login_state_from_env,
|
||||||
|
write_browser_config_template,
|
||||||
|
)
|
||||||
|
from plugins.joinquant.export_targets import export_targets
|
||||||
|
from plugins.joinquant.ingest import ingest_joinquant_outputs
|
||||||
|
from plugins.joinquant.reconcile import reconcile_joinquant
|
||||||
|
from plugins.joinquant.smoke import prepare_smoke_test
|
||||||
|
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
|
||||||
|
|
||||||
|
|
||||||
|
@click.group(name="joinquant")
|
||||||
|
def joinquant():
|
||||||
|
"""Compare internal portfolio simulation with JoinQuant output."""
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("export-targets")
|
||||||
|
@click.option("--positions-path", required=True, help="Portfolio positions parquet from `portfolio build`")
|
||||||
|
@click.option("--portfolio-name", required=True, help="Portfolio run to export")
|
||||||
|
@click.option(
|
||||||
|
"--mode",
|
||||||
|
"mode",
|
||||||
|
type=click.Choice(["target_shares", "target_value"]),
|
||||||
|
default="target_shares",
|
||||||
|
show_default=True,
|
||||||
|
help="JoinQuant target order mode",
|
||||||
|
)
|
||||||
|
@click.option("--start-date", default=None, help="Inclusive YYYY-MM-DD start date")
|
||||||
|
@click.option("--end-date", default=None, help="Inclusive YYYY-MM-DD end date")
|
||||||
|
@click.option(
|
||||||
|
"--execution-calendar-path",
|
||||||
|
default=None,
|
||||||
|
help="Daily data parquet/dataset used to shift position dates to next execution session",
|
||||||
|
)
|
||||||
|
@click.option("--out-dir", default="plugins_output/joinquant/targets", show_default=True)
|
||||||
|
@click.option("--force", is_flag=True, help="Overwrite frozen target/snapshot files")
|
||||||
|
def export_targets_cmd(
|
||||||
|
positions_path,
|
||||||
|
portfolio_name,
|
||||||
|
mode,
|
||||||
|
start_date,
|
||||||
|
end_date,
|
||||||
|
execution_calendar_path,
|
||||||
|
out_dir,
|
||||||
|
force,
|
||||||
|
):
|
||||||
|
"""Export frozen daily target files for JoinQuant."""
|
||||||
|
snapshots = export_targets(
|
||||||
|
positions_path=positions_path,
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
mode=mode,
|
||||||
|
start_date=start_date,
|
||||||
|
end_date=end_date,
|
||||||
|
execution_calendar_path=execution_calendar_path,
|
||||||
|
out_dir=out_dir,
|
||||||
|
force=force,
|
||||||
|
)
|
||||||
|
click.echo(f"Exported JoinQuant targets: {len(snapshots)} day(s)")
|
||||||
|
for snapshot in snapshots:
|
||||||
|
click.echo(
|
||||||
|
f" {snapshot['date']}: {snapshot['n_symbols']} symbols, "
|
||||||
|
f"sha256={str(snapshot['file_sha256'])[:12]}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("ingest")
|
||||||
|
@click.option("--portfolio-name", required=True, help="Portfolio run name")
|
||||||
|
@click.option("--fills-csv", required=True, help="JoinQuant fills CSV")
|
||||||
|
@click.option("--positions-csv", required=True, help="JoinQuant positions CSV")
|
||||||
|
@click.option("--pnl-csv", required=True, help="JoinQuant daily PnL CSV")
|
||||||
|
@click.option("--out-dir", default="plugins_output/joinquant/ingested", show_default=True)
|
||||||
|
def ingest_cmd(portfolio_name, fills_csv, positions_csv, pnl_csv, out_dir):
|
||||||
|
"""Normalize JoinQuant CSV exports to parquet."""
|
||||||
|
paths = ingest_joinquant_outputs(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
fills_csv=fills_csv,
|
||||||
|
positions_csv=positions_csv,
|
||||||
|
pnl_csv=pnl_csv,
|
||||||
|
out_dir=out_dir,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved JoinQuant fills: {paths['fills']}")
|
||||||
|
click.echo(f"Saved JoinQuant positions: {paths['positions']}")
|
||||||
|
click.echo(f"Saved JoinQuant pnl: {paths['pnl']}")
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("reconcile")
|
||||||
|
@click.option("--portfolio-name", required=True, help="Portfolio run name")
|
||||||
|
@click.option("--targets-dir", required=True, help="Directory containing exported daily target files")
|
||||||
|
@click.option("--our-fills-path", required=True, help="Internal simulator fills parquet")
|
||||||
|
@click.option("--our-positions-path", required=True, help="Internal portfolio positions parquet")
|
||||||
|
@click.option("--our-pnl-path", required=True, help="Internal simulator PnL parquet")
|
||||||
|
@click.option("--jq-fills-path", required=True, help="Normalized JoinQuant fills parquet")
|
||||||
|
@click.option("--jq-positions-path", required=True, help="Normalized JoinQuant positions parquet")
|
||||||
|
@click.option("--jq-pnl-path", required=True, help="Normalized JoinQuant PnL parquet")
|
||||||
|
@click.option("--out-dir", default="plugins_output/joinquant/reconcile", show_default=True)
|
||||||
|
@click.option("--share-tolerance", default=0.0, show_default=True, type=float)
|
||||||
|
@click.option("--price-rel-tolerance", default=1e-4, show_default=True, type=float)
|
||||||
|
@click.option("--pnl-tolerance", default=1.0, show_default=True, type=float)
|
||||||
|
@click.option("--booksize", default=None, type=float, help="Booksize for value tolerance inference")
|
||||||
|
def reconcile_cmd(
|
||||||
|
portfolio_name,
|
||||||
|
targets_dir,
|
||||||
|
our_fills_path,
|
||||||
|
our_positions_path,
|
||||||
|
our_pnl_path,
|
||||||
|
jq_fills_path,
|
||||||
|
jq_positions_path,
|
||||||
|
jq_pnl_path,
|
||||||
|
out_dir,
|
||||||
|
share_tolerance,
|
||||||
|
price_rel_tolerance,
|
||||||
|
pnl_tolerance,
|
||||||
|
booksize,
|
||||||
|
):
|
||||||
|
"""Write per-symbol and daily JoinQuant reconciliation reports."""
|
||||||
|
paths = reconcile_joinquant(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
targets_dir=targets_dir,
|
||||||
|
our_fills_path=our_fills_path,
|
||||||
|
our_positions_path=our_positions_path,
|
||||||
|
our_pnl_path=our_pnl_path,
|
||||||
|
jq_fills_path=jq_fills_path,
|
||||||
|
jq_positions_path=jq_positions_path,
|
||||||
|
jq_pnl_path=jq_pnl_path,
|
||||||
|
out_dir=out_dir,
|
||||||
|
share_tolerance=share_tolerance,
|
||||||
|
price_rel_tolerance=price_rel_tolerance,
|
||||||
|
pnl_tolerance=pnl_tolerance,
|
||||||
|
booksize=booksize,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved reconciliation parquet: {paths['daily_reconcile']}")
|
||||||
|
click.echo(f"Saved reconciliation summary: {paths['summary_md']}")
|
||||||
|
click.echo(f"Saved reconciliation CSV: {paths['summary_csv']}")
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("write-wrapper")
|
||||||
|
@click.option("--portfolio-name", required=True, help="Portfolio run name")
|
||||||
|
@click.option(
|
||||||
|
"--mode",
|
||||||
|
"mode",
|
||||||
|
type=click.Choice(["target_shares", "target_value"]),
|
||||||
|
default="target_shares",
|
||||||
|
show_default=True,
|
||||||
|
)
|
||||||
|
@click.option("--out-path", required=True, help="Path for generated standalone strategy")
|
||||||
|
@click.option("--allow-short", is_flag=True, help="Do not clip negative targets in the generated wrapper")
|
||||||
|
@click.option(
|
||||||
|
"--embed-targets-dir",
|
||||||
|
default=None,
|
||||||
|
help="Embed CSV target files from this directory into the strategy source",
|
||||||
|
)
|
||||||
|
def write_wrapper_cmd(portfolio_name, mode, out_path, allow_short, embed_targets_dir):
|
||||||
|
"""Generate a standalone JoinQuant wrapper strategy."""
|
||||||
|
path = write_wrapper_strategy(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
mode=mode,
|
||||||
|
out_path=out_path,
|
||||||
|
allow_short=allow_short,
|
||||||
|
embedded_targets_dir=embed_targets_dir,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved JoinQuant wrapper strategy: {path}")
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("prepare-smoke")
|
||||||
|
@click.option("--out-dir", required=True, help="Root directory for generated smoke-test artifacts")
|
||||||
|
@click.option(
|
||||||
|
"--universe",
|
||||||
|
default="sh600000,sz000001,sh600519,sz002594,sz300750",
|
||||||
|
show_default=True,
|
||||||
|
help="Universe for the tiny real-data download",
|
||||||
|
)
|
||||||
|
@click.option("--trade-symbol", default="sh600000", show_default=True)
|
||||||
|
@click.option("--start-date", default="2024-01-02", show_default=True)
|
||||||
|
@click.option("--end-date", default="2024-01-12", show_default=True)
|
||||||
|
@click.option("--portfolio-name", default="jq_smoke_one_stock_long", show_default=True)
|
||||||
|
@click.option("--shares", default=1000, show_default=True, type=int)
|
||||||
|
@click.option("--booksize", default=1_000_000.0, show_default=True, type=float)
|
||||||
|
@click.option("--max-signal-dates", default=3, show_default=True, type=int)
|
||||||
|
@click.option("--cost-bps", default=5.0, show_default=True, type=float)
|
||||||
|
@click.option("--slippage-bps", default=5.0, show_default=True, type=float)
|
||||||
|
@click.option("--volume-frac", default=0.02, show_default=True, type=float)
|
||||||
|
@click.option("--force", is_flag=True, help="Overwrite existing frozen target files")
|
||||||
|
def prepare_smoke_cmd(
|
||||||
|
out_dir,
|
||||||
|
universe,
|
||||||
|
trade_symbol,
|
||||||
|
start_date,
|
||||||
|
end_date,
|
||||||
|
portfolio_name,
|
||||||
|
shares,
|
||||||
|
booksize,
|
||||||
|
max_signal_dates,
|
||||||
|
cost_bps,
|
||||||
|
slippage_bps,
|
||||||
|
volume_frac,
|
||||||
|
force,
|
||||||
|
):
|
||||||
|
"""Prepare a one-command local real-data JoinQuant smoke test."""
|
||||||
|
manifest = prepare_smoke_test(
|
||||||
|
out_dir=out_dir,
|
||||||
|
universe=universe,
|
||||||
|
trade_symbol=trade_symbol,
|
||||||
|
start_date=start_date,
|
||||||
|
end_date=end_date,
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
shares=shares,
|
||||||
|
booksize=booksize,
|
||||||
|
max_signal_dates=max_signal_dates,
|
||||||
|
cost_bps=cost_bps,
|
||||||
|
slippage_bps=slippage_bps,
|
||||||
|
volume_frac=volume_frac,
|
||||||
|
force=force,
|
||||||
|
)
|
||||||
|
click.echo(f"Prepared JoinQuant smoke manifest: {manifest['manifest_path']}")
|
||||||
|
click.echo(f"Wrapper: {manifest['wrapper_path']}")
|
||||||
|
click.echo(f"Targets: {manifest['targets_dir']}")
|
||||||
|
click.echo(f"Expected JoinQuant exports: {manifest['joinquant_export_dir']}")
|
||||||
|
summary = manifest["local_summary"]
|
||||||
|
click.echo(
|
||||||
|
f"Local simulator: {summary['n_pnl_rows']} days, "
|
||||||
|
f"PnL={summary['total_pnl']:,.2f}, cost={summary['total_cost']:,.2f}, "
|
||||||
|
f"blocked={summary['blocked_trades']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("browser-login")
|
||||||
|
@click.option(
|
||||||
|
"--storage-state",
|
||||||
|
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
|
||||||
|
show_default=True,
|
||||||
|
help="Path where the authenticated browser state is stored",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--login-url",
|
||||||
|
default="https://www.joinquant.com/user/login/index",
|
||||||
|
show_default=True,
|
||||||
|
)
|
||||||
|
@click.option("--headless", is_flag=True, help="Use a headless browser")
|
||||||
|
@click.option(
|
||||||
|
"--wait-seconds",
|
||||||
|
default=0,
|
||||||
|
show_default=True,
|
||||||
|
type=int,
|
||||||
|
help="Seconds to wait before saving state; 0 prompts for Enter after login",
|
||||||
|
)
|
||||||
|
def browser_login_cmd(storage_state, login_url, headless, wait_seconds):
|
||||||
|
"""Open JoinQuant login and save reusable browser session state."""
|
||||||
|
path = save_login_state(
|
||||||
|
storage_state=storage_state,
|
||||||
|
login_url=login_url,
|
||||||
|
headless=headless,
|
||||||
|
wait_seconds=wait_seconds,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved JoinQuant browser state: {path}")
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("browser-login-env")
|
||||||
|
@click.option(
|
||||||
|
"--env-path",
|
||||||
|
default="~/.config/chinese-equity-quant/joinquant.env",
|
||||||
|
show_default=True,
|
||||||
|
help="Env file with JOINQUANT_USERNAME and JOINQUANT_PASSWORD",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--storage-state",
|
||||||
|
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
|
||||||
|
show_default=True,
|
||||||
|
help="Path where the authenticated browser state is stored",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--login-url",
|
||||||
|
default="https://www.joinquant.com/user/login/index",
|
||||||
|
show_default=True,
|
||||||
|
)
|
||||||
|
@click.option("--headed", is_flag=True, help="Run with a visible browser")
|
||||||
|
@click.option("--out-dir", default=None, help="Login artifact directory")
|
||||||
|
@click.option("--timeout-ms", default=120_000, show_default=True, type=int)
|
||||||
|
def browser_login_env_cmd(env_path, storage_state, login_url, headed, out_dir, timeout_ms):
|
||||||
|
"""Try credential-based JoinQuant login from an env file."""
|
||||||
|
env = load_env_file(env_path)
|
||||||
|
missing = [
|
||||||
|
key for key in ["JOINQUANT_USERNAME", "JOINQUANT_PASSWORD"]
|
||||||
|
if not env.get(key)
|
||||||
|
]
|
||||||
|
if missing:
|
||||||
|
raise click.ClickException(f"Missing required env keys: {', '.join(missing)}")
|
||||||
|
report = save_login_state_from_env(
|
||||||
|
env_path=env_path,
|
||||||
|
storage_state=storage_state,
|
||||||
|
login_url=login_url,
|
||||||
|
headless=not headed,
|
||||||
|
out_dir=out_dir,
|
||||||
|
timeout_ms=timeout_ms,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved login report: {report['report_path']}")
|
||||||
|
if report["logged_in"]:
|
||||||
|
click.echo(f"Saved JoinQuant browser state: {report['storage_state']}")
|
||||||
|
else:
|
||||||
|
raise click.ClickException(
|
||||||
|
"JoinQuant login did not complete. Check the saved screenshot/HTML; "
|
||||||
|
"CAPTCHA/SMS/2FA may require `joinquant browser-login` once."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("write-browser-config")
|
||||||
|
@click.option("--out-path", required=True, help="Path for JSON browser automation config")
|
||||||
|
@click.option("--strategy-url", default="", help="JoinQuant strategy edit/backtest/模拟盘 URL")
|
||||||
|
@click.option(
|
||||||
|
"--flow",
|
||||||
|
type=click.Choice(["backtest", "sim-trade"]),
|
||||||
|
default="backtest",
|
||||||
|
show_default=True,
|
||||||
|
help="Browser automation template to write",
|
||||||
|
)
|
||||||
|
def write_browser_config_cmd(out_path, strategy_url, flow):
|
||||||
|
"""Write a browser automation selector/action config template."""
|
||||||
|
path = write_browser_config_template(out_path, strategy_url=strategy_url, flow=flow)
|
||||||
|
click.echo(f"Saved JoinQuant browser config template: {path}")
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("browser-snapshot")
|
||||||
|
@click.option("--url", required=True, help="Logged-in JoinQuant page URL to inspect")
|
||||||
|
@click.option(
|
||||||
|
"--storage-state",
|
||||||
|
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
|
||||||
|
show_default=True,
|
||||||
|
)
|
||||||
|
@click.option("--out-dir", required=True, help="Directory for page.html and page.png")
|
||||||
|
@click.option("--headed", is_flag=True, help="Run with a visible browser")
|
||||||
|
@click.option("--timeout-ms", default=60_000, show_default=True, type=int)
|
||||||
|
def browser_snapshot_cmd(url, storage_state, out_dir, headed, timeout_ms):
|
||||||
|
"""Save a page screenshot and HTML for selector discovery."""
|
||||||
|
paths = browser_snapshot(
|
||||||
|
url=url,
|
||||||
|
storage_state=storage_state,
|
||||||
|
out_dir=out_dir,
|
||||||
|
headless=not headed,
|
||||||
|
timeout_ms=timeout_ms,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved HTML: {paths['html']}")
|
||||||
|
click.echo(f"Saved screenshot: {paths['screenshot']}")
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("run-browser-backtest")
|
||||||
|
@click.option("--manifest-path", required=True, help="Manifest from `joinquant prepare-smoke`")
|
||||||
|
@click.option("--config-path", required=True, help="JSON browser automation config")
|
||||||
|
@click.option(
|
||||||
|
"--storage-state",
|
||||||
|
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
|
||||||
|
show_default=True,
|
||||||
|
)
|
||||||
|
@click.option("--out-dir", default=None, help="Browser-run artifact directory")
|
||||||
|
@click.option("--headed", is_flag=True, help="Run with a visible browser")
|
||||||
|
@click.option("--no-auto-reconcile", is_flag=True, help="Skip ingest/reconcile after downloads")
|
||||||
|
def run_browser_backtest_cmd(
|
||||||
|
manifest_path,
|
||||||
|
config_path,
|
||||||
|
storage_state,
|
||||||
|
out_dir,
|
||||||
|
headed,
|
||||||
|
no_auto_reconcile,
|
||||||
|
):
|
||||||
|
"""Run JoinQuant browser automation from manifest and config."""
|
||||||
|
report = run_browser_backtest(
|
||||||
|
manifest_path=manifest_path,
|
||||||
|
config_path=config_path,
|
||||||
|
storage_state=storage_state,
|
||||||
|
out_dir=out_dir,
|
||||||
|
headless=not headed,
|
||||||
|
auto_reconcile=not no_auto_reconcile,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved browser run report: {report['report_path']}")
|
||||||
|
if report["downloaded"]:
|
||||||
|
click.echo("Downloaded JoinQuant CSVs:")
|
||||||
|
for key, path in report["downloaded"].items():
|
||||||
|
click.echo(f" {key}: {path}")
|
||||||
|
if report["reconcile_paths"]:
|
||||||
|
click.echo(f"Saved reconciliation summary: {report['reconcile_paths']['summary_md']}")
|
||||||
|
|
||||||
|
|
||||||
|
@joinquant.command("run-browser-sim")
|
||||||
|
@click.option("--manifest-path", required=True, help="Manifest from target preparation")
|
||||||
|
@click.option("--config-path", required=True, help="JSON simulated-trading browser config")
|
||||||
|
@click.option(
|
||||||
|
"--storage-state",
|
||||||
|
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
|
||||||
|
show_default=True,
|
||||||
|
)
|
||||||
|
@click.option("--out-dir", default=None, help="Browser-run artifact directory")
|
||||||
|
@click.option("--headed", is_flag=True, help="Run with a visible browser")
|
||||||
|
@click.option("--no-auto-reconcile", is_flag=True, help="Skip ingest/reconcile after downloads")
|
||||||
|
def run_browser_sim_cmd(
|
||||||
|
manifest_path,
|
||||||
|
config_path,
|
||||||
|
storage_state,
|
||||||
|
out_dir,
|
||||||
|
headed,
|
||||||
|
no_auto_reconcile,
|
||||||
|
):
|
||||||
|
"""Run JoinQuant 模拟盘 browser automation from manifest and config."""
|
||||||
|
report = run_browser_sim_trade(
|
||||||
|
manifest_path=manifest_path,
|
||||||
|
config_path=config_path,
|
||||||
|
storage_state=storage_state,
|
||||||
|
out_dir=out_dir,
|
||||||
|
headless=not headed,
|
||||||
|
auto_reconcile=not no_auto_reconcile,
|
||||||
|
)
|
||||||
|
click.echo(f"Saved browser sim-trade report: {report['report_path']}")
|
||||||
|
if report["downloaded"]:
|
||||||
|
click.echo("Downloaded JoinQuant CSVs:")
|
||||||
|
for key, path in report["downloaded"].items():
|
||||||
|
click.echo(f" {key}: {path}")
|
||||||
|
if report["reconcile_paths"]:
|
||||||
|
click.echo(f"Saved reconciliation summary: {report['reconcile_paths']['summary_md']}")
|
||||||
@@ -0,0 +1,248 @@
|
|||||||
|
"""Export portfolio positions as frozen JoinQuant target files."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable, Literal
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.common.schema import POSITION_COLUMNS
|
||||||
|
from plugins.joinquant.schema import JOINQUANT_TARGET_COLUMNS
|
||||||
|
from plugins.joinquant.symbols import to_joinquant_symbol
|
||||||
|
|
||||||
|
|
||||||
|
ExportMode = Literal["target_shares", "target_value"]
|
||||||
|
|
||||||
|
|
||||||
|
def _date_text(value: object) -> str:
|
||||||
|
return pd.Timestamp(value).strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
|
||||||
|
def _date_file_stem(date_text: str) -> str:
|
||||||
|
return pd.Timestamp(date_text).strftime("%Y%m%d")
|
||||||
|
|
||||||
|
|
||||||
|
def _snapshot_root_for(targets_root: Path) -> Path:
|
||||||
|
if targets_root.name == "targets":
|
||||||
|
return targets_root.parent / "snapshots"
|
||||||
|
return targets_root / "snapshots"
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256_file(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as fh:
|
||||||
|
for chunk in iter(lambda: fh.read(1024 * 1024), b""):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _check_position_columns(df: pd.DataFrame) -> None:
|
||||||
|
missing = [col for col in POSITION_COLUMNS if col not in df.columns]
|
||||||
|
if missing:
|
||||||
|
raise ValueError(f"Positions input missing required columns: {missing}")
|
||||||
|
|
||||||
|
|
||||||
|
def _filter_dates(
|
||||||
|
df: pd.DataFrame,
|
||||||
|
start_date: str | None,
|
||||||
|
end_date: str | None,
|
||||||
|
*,
|
||||||
|
date_column: str = "date",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
out = df.copy()
|
||||||
|
out[date_column] = pd.to_datetime(out[date_column]).dt.normalize()
|
||||||
|
if start_date:
|
||||||
|
out = out[out[date_column] >= pd.Timestamp(start_date).normalize()]
|
||||||
|
if end_date:
|
||||||
|
out = out[out[date_column] <= pd.Timestamp(end_date).normalize()]
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _read_execution_calendar(path: str | Path) -> pd.DatetimeIndex:
|
||||||
|
data = pd.read_parquet(path)
|
||||||
|
if "date" not in data.columns:
|
||||||
|
raise ValueError("execution calendar parquet must contain a 'date' column")
|
||||||
|
dates = pd.to_datetime(data["date"], errors="coerce").dropna().dt.normalize()
|
||||||
|
return pd.DatetimeIndex(sorted(dates.unique()))
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_execution_calendar(df: pd.DataFrame, calendar_path: str | Path) -> pd.DataFrame:
|
||||||
|
calendar = _read_execution_calendar(calendar_path)
|
||||||
|
if calendar.empty:
|
||||||
|
raise ValueError("execution calendar contains no dates")
|
||||||
|
|
||||||
|
source_dates = pd.to_datetime(df["date"]).dt.normalize()
|
||||||
|
positions = calendar.searchsorted(source_dates, side="right")
|
||||||
|
out = df.copy()
|
||||||
|
out["source_date"] = source_dates
|
||||||
|
out["export_date"] = pd.NaT
|
||||||
|
valid = positions < len(calendar)
|
||||||
|
out.loc[valid, "export_date"] = calendar.take(positions[valid])
|
||||||
|
return out[out["export_date"].notna()].copy()
|
||||||
|
|
||||||
|
|
||||||
|
def build_target_frame(
|
||||||
|
positions: pd.DataFrame,
|
||||||
|
*,
|
||||||
|
portfolio_name: str | None = None,
|
||||||
|
mode: ExportMode = "target_shares",
|
||||||
|
start_date: str | None = None,
|
||||||
|
end_date: str | None = None,
|
||||||
|
snapshot_ids: dict[str, str] | None = None,
|
||||||
|
execution_calendar_path: str | Path | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Build normalized JoinQuant target rows from portfolio positions.
|
||||||
|
|
||||||
|
``target_shares`` is populated from ``position_shares`` because the core
|
||||||
|
simulator executes the discretized book, not continuous research shares.
|
||||||
|
"""
|
||||||
|
if mode not in {"target_shares", "target_value"}:
|
||||||
|
raise ValueError("mode must be 'target_shares' or 'target_value'")
|
||||||
|
_check_position_columns(positions)
|
||||||
|
|
||||||
|
df = positions.copy()
|
||||||
|
df["date"] = pd.to_datetime(df["date"]).dt.normalize()
|
||||||
|
if execution_calendar_path is not None:
|
||||||
|
df = _apply_execution_calendar(df, execution_calendar_path)
|
||||||
|
df = df.drop(columns=["date"]).rename(columns={"export_date": "date"})
|
||||||
|
df["source_date"] = pd.to_datetime(df["source_date"]).dt.strftime("%Y-%m-%d")
|
||||||
|
df = _filter_dates(df, start_date, end_date)
|
||||||
|
else:
|
||||||
|
df = _filter_dates(df, start_date, end_date)
|
||||||
|
if portfolio_name is not None:
|
||||||
|
df = df[df["portfolio_name"].astype(str) == portfolio_name]
|
||||||
|
if df.empty:
|
||||||
|
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
|
||||||
|
|
||||||
|
out = pd.DataFrame({
|
||||||
|
"date": df["date"].map(_date_text),
|
||||||
|
"portfolio_name": df["portfolio_name"].astype(str),
|
||||||
|
"symbol_id": df["symbol_id"].astype(str),
|
||||||
|
"jq_symbol": df["symbol_id"].map(to_joinquant_symbol),
|
||||||
|
"target_shares": pd.to_numeric(df["position_shares"], errors="coerce").fillna(0).astype("int64"),
|
||||||
|
"target_value": pd.to_numeric(df["target_value"], errors="coerce").fillna(0.0),
|
||||||
|
"target_weight": pd.to_numeric(df["target_weight"], errors="coerce").fillna(0.0),
|
||||||
|
"export_mode": mode,
|
||||||
|
"snapshot_id": "",
|
||||||
|
})
|
||||||
|
|
||||||
|
if snapshot_ids:
|
||||||
|
out["snapshot_id"] = out["date"].map(snapshot_ids).fillna("")
|
||||||
|
|
||||||
|
return out[JOINQUANT_TARGET_COLUMNS].sort_values(
|
||||||
|
["date", "portfolio_name", "symbol_id"]
|
||||||
|
).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def export_targets(
|
||||||
|
positions_path: str | Path,
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
mode: ExportMode = "target_shares",
|
||||||
|
out_dir: str | Path = "plugins_output/joinquant/targets",
|
||||||
|
start_date: str | None = None,
|
||||||
|
end_date: str | None = None,
|
||||||
|
execution_calendar_path: str | Path | None = None,
|
||||||
|
force: bool = False,
|
||||||
|
) -> list[dict[str, object]]:
|
||||||
|
"""Export one daily CSV/parquet target file plus a snapshot JSON per date.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
positions_path: Parquet file produced by ``portfolio build``.
|
||||||
|
portfolio_name: Portfolio run to export.
|
||||||
|
mode: ``target_shares`` or ``target_value``.
|
||||||
|
out_dir: Target root. Files are written to ``out_dir/portfolio_name``.
|
||||||
|
If the root is named ``targets``, snapshots are written to the
|
||||||
|
sibling ``snapshots`` directory.
|
||||||
|
start_date: Optional inclusive start date.
|
||||||
|
end_date: Optional inclusive end date.
|
||||||
|
execution_calendar_path: Optional daily-bar parquet dataset used to
|
||||||
|
shift each position date to the next available execution session.
|
||||||
|
This matches the internal simulator's next-open convention.
|
||||||
|
force: If false, existing target or snapshot files are treated as
|
||||||
|
frozen and cause ``FileExistsError``.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Snapshot metadata dictionaries, one per exported date.
|
||||||
|
"""
|
||||||
|
positions_path = Path(positions_path)
|
||||||
|
targets_root = Path(out_dir)
|
||||||
|
snapshot_root = _snapshot_root_for(targets_root)
|
||||||
|
targets_portfolio_dir = targets_root / portfolio_name
|
||||||
|
snapshots_portfolio_dir = snapshot_root / portfolio_name
|
||||||
|
targets_portfolio_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
snapshots_portfolio_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
positions = pd.read_parquet(positions_path)
|
||||||
|
filtered = positions.copy()
|
||||||
|
filtered["date"] = pd.to_datetime(filtered["date"]).dt.normalize()
|
||||||
|
if execution_calendar_path is not None:
|
||||||
|
filtered = _apply_execution_calendar(filtered, execution_calendar_path)
|
||||||
|
filtered = filtered.drop(columns=["date"]).rename(columns={"export_date": "date"})
|
||||||
|
filtered["source_date"] = pd.to_datetime(filtered["source_date"]).dt.strftime("%Y-%m-%d")
|
||||||
|
filtered = _filter_dates(filtered, start_date, end_date)
|
||||||
|
else:
|
||||||
|
filtered = _filter_dates(filtered, start_date, end_date)
|
||||||
|
filtered = filtered[filtered["portfolio_name"].astype(str) == portfolio_name]
|
||||||
|
if filtered.empty:
|
||||||
|
return []
|
||||||
|
|
||||||
|
date_texts = sorted(filtered["date"].map(_date_text).unique())
|
||||||
|
snapshot_ids = {
|
||||||
|
date_text: f"jq-{portfolio_name}-{date_text}-{uuid.uuid4().hex[:12]}"
|
||||||
|
for date_text in date_texts
|
||||||
|
}
|
||||||
|
targets = build_target_frame(
|
||||||
|
filtered,
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
mode=mode,
|
||||||
|
snapshot_ids=snapshot_ids,
|
||||||
|
execution_calendar_path=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
snapshots: list[dict[str, object]] = []
|
||||||
|
for date_text, daily in targets.groupby("date", sort=True):
|
||||||
|
stem = _date_file_stem(date_text)
|
||||||
|
csv_path = targets_portfolio_dir / f"{stem}.csv"
|
||||||
|
parquet_path = targets_portfolio_dir / f"{stem}.parquet"
|
||||||
|
snapshot_path = snapshots_portfolio_dir / f"{stem}.json"
|
||||||
|
existing: Iterable[Path] = (csv_path, parquet_path, snapshot_path)
|
||||||
|
if not force:
|
||||||
|
conflicts = [str(path) for path in existing if path.exists()]
|
||||||
|
if conflicts:
|
||||||
|
raise FileExistsError(
|
||||||
|
"Frozen JoinQuant target already exists; use --force to overwrite: "
|
||||||
|
+ ", ".join(conflicts)
|
||||||
|
)
|
||||||
|
|
||||||
|
daily = daily[JOINQUANT_TARGET_COLUMNS].reset_index(drop=True)
|
||||||
|
daily.to_csv(csv_path, index=False)
|
||||||
|
daily.to_parquet(parquet_path, index=False)
|
||||||
|
file_hash = _sha256_file(csv_path)
|
||||||
|
|
||||||
|
snapshot = {
|
||||||
|
"snapshot_id": snapshot_ids[date_text],
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"date": date_text,
|
||||||
|
"export_mode": mode,
|
||||||
|
"source_positions_path": str(positions_path),
|
||||||
|
"execution_calendar_path": str(execution_calendar_path) if execution_calendar_path else None,
|
||||||
|
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"n_symbols": int(len(daily)),
|
||||||
|
"file_sha256": file_hash,
|
||||||
|
"notes": "Frozen JoinQuant target file.",
|
||||||
|
"target_csv_path": str(csv_path),
|
||||||
|
"target_parquet_path": str(parquet_path),
|
||||||
|
}
|
||||||
|
snapshot_path.write_text(
|
||||||
|
json.dumps(snapshot, indent=2, ensure_ascii=False) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
snapshots.append(snapshot)
|
||||||
|
|
||||||
|
return snapshots
|
||||||
@@ -0,0 +1,225 @@
|
|||||||
|
"""Normalize JoinQuant CSV exports into plugin parquet schemas."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from plugins.joinquant.schema import (
|
||||||
|
JOINQUANT_FILL_COLUMNS,
|
||||||
|
JOINQUANT_PNL_COLUMNS,
|
||||||
|
JOINQUANT_POSITION_COLUMNS,
|
||||||
|
)
|
||||||
|
from plugins.joinquant.symbols import normalize_symbol_pair, to_joinquant_symbol
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_name(name: object) -> str:
|
||||||
|
text = str(name).strip().lower()
|
||||||
|
text = re.sub(r"[\s\-.()/]+", "_", text)
|
||||||
|
return re.sub(r"[^0-9a-z_]+", "", text).strip("_")
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_columns(df: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
out = df.copy()
|
||||||
|
out.columns = [_clean_name(col) for col in out.columns]
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _pick(df: pd.DataFrame, candidates: list[str]) -> str | None:
|
||||||
|
for candidate in candidates:
|
||||||
|
clean = _clean_name(candidate)
|
||||||
|
if clean in df.columns:
|
||||||
|
return clean
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _series_or_default(df: pd.DataFrame, candidates: list[str], default: object) -> pd.Series:
|
||||||
|
col = _pick(df, candidates)
|
||||||
|
if col is None:
|
||||||
|
return pd.Series([default] * len(df), index=df.index)
|
||||||
|
return df[col]
|
||||||
|
|
||||||
|
|
||||||
|
def _date_series(df: pd.DataFrame) -> pd.Series:
|
||||||
|
values = _series_or_default(
|
||||||
|
df,
|
||||||
|
["date", "trade_date", "datetime", "time", "created_at"],
|
||||||
|
pd.NaT,
|
||||||
|
)
|
||||||
|
parsed = pd.to_datetime(values, errors="coerce").dt.normalize()
|
||||||
|
return parsed.dt.strftime("%Y-%m-%d").fillna("")
|
||||||
|
|
||||||
|
|
||||||
|
def _numeric(values: pd.Series, default: float = 0.0) -> pd.Series:
|
||||||
|
return pd.to_numeric(values, errors="coerce").replace([np.inf, -np.inf], np.nan).fillna(default)
|
||||||
|
|
||||||
|
|
||||||
|
def _text(values: pd.Series, default: str = "") -> pd.Series:
|
||||||
|
return values.fillna(default).astype(str)
|
||||||
|
|
||||||
|
|
||||||
|
def _portfolio_series(df: pd.DataFrame, portfolio_name: str) -> pd.Series:
|
||||||
|
return _text(_series_or_default(df, ["portfolio_name", "portfolio", "strategy"], portfolio_name), portfolio_name)
|
||||||
|
|
||||||
|
|
||||||
|
def _symbol_frame(df: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
internal_col = _pick(df, ["symbol_id", "internal_symbol"])
|
||||||
|
jq_col = _pick(df, ["jq_symbol", "security", "stock", "symbol", "code", "order_book_id"])
|
||||||
|
|
||||||
|
symbol_ids: list[str] = []
|
||||||
|
jq_symbols: list[str] = []
|
||||||
|
for idx in df.index:
|
||||||
|
internal = df.at[idx, internal_col] if internal_col else None
|
||||||
|
jq_value = df.at[idx, jq_col] if jq_col else None
|
||||||
|
value = internal if internal is not None and str(internal).strip() else jq_value
|
||||||
|
if value is None or not str(value).strip():
|
||||||
|
symbol_ids.append("")
|
||||||
|
jq_symbols.append("")
|
||||||
|
continue
|
||||||
|
symbol_id, jq_symbol = normalize_symbol_pair(value)
|
||||||
|
if jq_value is not None and str(jq_value).strip():
|
||||||
|
try:
|
||||||
|
_, jq_symbol = normalize_symbol_pair(jq_value)
|
||||||
|
except ValueError:
|
||||||
|
jq_symbol = to_joinquant_symbol(symbol_id)
|
||||||
|
symbol_ids.append(symbol_id)
|
||||||
|
jq_symbols.append(jq_symbol)
|
||||||
|
|
||||||
|
return pd.DataFrame({"symbol_id": symbol_ids, "jq_symbol": jq_symbols}, index=df.index)
|
||||||
|
|
||||||
|
|
||||||
|
def _signed_shares(shares: pd.Series, side: pd.Series) -> pd.Series:
|
||||||
|
signed = _numeric(shares, 0.0)
|
||||||
|
side_text = side.fillna("").astype(str).str.lower()
|
||||||
|
sell = side_text.str.contains("sell|short|close|reduce|-", regex=True)
|
||||||
|
buy = side_text.str.contains("buy|long|open|add|\\+", regex=True)
|
||||||
|
signed = signed.abs()
|
||||||
|
signed = signed.mask(sell, -signed)
|
||||||
|
signed = signed.mask(~(sell | buy), _numeric(shares, 0.0))
|
||||||
|
return signed
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_fills_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
"""Read a JoinQuant fills CSV and return ``JOINQUANT_FILL_COLUMNS``."""
|
||||||
|
raw = _clean_columns(pd.read_csv(path))
|
||||||
|
symbols = _symbol_frame(raw)
|
||||||
|
side = _text(_series_or_default(raw, ["side", "action", "direction"], ""))
|
||||||
|
requested = _signed_shares(
|
||||||
|
_series_or_default(raw, ["requested_shares", "target_shares", "amount", "order_amount"], 0),
|
||||||
|
side,
|
||||||
|
)
|
||||||
|
filled = _signed_shares(
|
||||||
|
_series_or_default(raw, ["filled_shares", "filled", "filled_amount", "deal_amount", "traded_shares"], 0),
|
||||||
|
side,
|
||||||
|
)
|
||||||
|
price = _numeric(_series_or_default(raw, ["fill_price", "price", "avg_cost", "avg_price"], np.nan), np.nan)
|
||||||
|
trade_value = _numeric(
|
||||||
|
_series_or_default(raw, ["trade_value", "value", "filled_value", "turnover"], np.nan),
|
||||||
|
np.nan,
|
||||||
|
)
|
||||||
|
trade_value = trade_value.fillna((filled * price).abs()).fillna(0.0)
|
||||||
|
|
||||||
|
out = pd.DataFrame({
|
||||||
|
"date": _date_series(raw),
|
||||||
|
"portfolio_name": _portfolio_series(raw, portfolio_name),
|
||||||
|
"symbol_id": symbols["symbol_id"],
|
||||||
|
"jq_symbol": symbols["jq_symbol"],
|
||||||
|
"order_id": _text(_series_or_default(raw, ["order_id", "id"], "")),
|
||||||
|
"side": side,
|
||||||
|
"requested_shares": requested.astype(float),
|
||||||
|
"filled_shares": filled.astype(float),
|
||||||
|
"fill_price": price.astype(float),
|
||||||
|
"trade_value": trade_value.astype(float),
|
||||||
|
"trade_cost": _numeric(_series_or_default(raw, ["trade_cost", "cost", "commission", "fee"], 0.0), 0.0),
|
||||||
|
"blocked": _numeric(_series_or_default(raw, ["blocked", "is_blocked"], 0), 0).astype("int64"),
|
||||||
|
"raw_status": _text(_series_or_default(raw, ["raw_status", "status", "order_status"], "")),
|
||||||
|
})
|
||||||
|
return out[JOINQUANT_FILL_COLUMNS]
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_positions_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
"""Read a JoinQuant positions CSV and return ``JOINQUANT_POSITION_COLUMNS``."""
|
||||||
|
raw = _clean_columns(pd.read_csv(path))
|
||||||
|
symbols = _symbol_frame(raw)
|
||||||
|
out = pd.DataFrame({
|
||||||
|
"date": _date_series(raw),
|
||||||
|
"portfolio_name": _portfolio_series(raw, portfolio_name),
|
||||||
|
"symbol_id": symbols["symbol_id"],
|
||||||
|
"jq_symbol": symbols["jq_symbol"],
|
||||||
|
"position_shares": _numeric(
|
||||||
|
_series_or_default(raw, ["position_shares", "shares", "amount", "quantity", "total_amount"], 0),
|
||||||
|
0,
|
||||||
|
),
|
||||||
|
"position_value": _numeric(
|
||||||
|
_series_or_default(raw, ["position_value", "market_value", "value"], 0.0),
|
||||||
|
0.0,
|
||||||
|
),
|
||||||
|
"cash": _numeric(_series_or_default(raw, ["cash", "available_cash"], np.nan), np.nan),
|
||||||
|
"total_value": _numeric(
|
||||||
|
_series_or_default(raw, ["total_value", "portfolio_value", "total_asset"], np.nan),
|
||||||
|
np.nan,
|
||||||
|
),
|
||||||
|
})
|
||||||
|
return out[JOINQUANT_POSITION_COLUMNS]
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_pnl_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
"""Read a JoinQuant daily PnL CSV and return ``JOINQUANT_PNL_COLUMNS``."""
|
||||||
|
raw = _clean_columns(pd.read_csv(path))
|
||||||
|
total_value = _numeric(
|
||||||
|
_series_or_default(raw, ["total_value", "portfolio_value", "total_asset"], np.nan),
|
||||||
|
np.nan,
|
||||||
|
)
|
||||||
|
pnl = _numeric(_series_or_default(raw, ["pnl", "daily_pnl", "profit", "returns_value"], np.nan), np.nan)
|
||||||
|
if pnl.isna().all() and total_value.notna().any():
|
||||||
|
pnl = total_value.diff().fillna(0.0)
|
||||||
|
|
||||||
|
out = pd.DataFrame({
|
||||||
|
"date": _date_series(raw),
|
||||||
|
"portfolio_name": _portfolio_series(raw, portfolio_name),
|
||||||
|
"gross_exposure": _numeric(
|
||||||
|
_series_or_default(raw, ["gross_exposure", "gross", "positions_value", "market_value"], np.nan),
|
||||||
|
np.nan,
|
||||||
|
),
|
||||||
|
"net_exposure": _numeric(
|
||||||
|
_series_or_default(raw, ["net_exposure", "net"], np.nan),
|
||||||
|
np.nan,
|
||||||
|
),
|
||||||
|
"cash": _numeric(_series_or_default(raw, ["cash", "available_cash"], np.nan), np.nan),
|
||||||
|
"total_value": total_value,
|
||||||
|
"pnl": pnl.fillna(0.0),
|
||||||
|
"cost": _numeric(_series_or_default(raw, ["cost", "trade_cost", "commission", "fee"], 0.0), 0.0),
|
||||||
|
"turnover": _numeric(_series_or_default(raw, ["turnover"], 0.0), 0.0),
|
||||||
|
})
|
||||||
|
return out[JOINQUANT_PNL_COLUMNS]
|
||||||
|
|
||||||
|
|
||||||
|
def ingest_joinquant_outputs(
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
fills_csv: str | Path,
|
||||||
|
positions_csv: str | Path,
|
||||||
|
pnl_csv: str | Path,
|
||||||
|
out_dir: str | Path = "plugins_output/joinquant/ingested",
|
||||||
|
) -> dict[str, Path]:
|
||||||
|
"""Normalize JoinQuant CSV exports and write parquet outputs."""
|
||||||
|
out_root = Path(out_dir) / portfolio_name
|
||||||
|
out_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
fills = normalize_fills_csv(fills_csv, portfolio_name)
|
||||||
|
positions = normalize_positions_csv(positions_csv, portfolio_name)
|
||||||
|
pnl = normalize_pnl_csv(pnl_csv, portfolio_name)
|
||||||
|
|
||||||
|
paths = {
|
||||||
|
"fills": out_root / "fills.pq",
|
||||||
|
"positions": out_root / "positions.pq",
|
||||||
|
"pnl": out_root / "pnl.pq",
|
||||||
|
}
|
||||||
|
fills.to_parquet(paths["fills"], index=False)
|
||||||
|
positions.to_parquet(paths["positions"], index=False)
|
||||||
|
pnl.to_parquet(paths["pnl"], index=False)
|
||||||
|
return paths
|
||||||
|
|
||||||
@@ -0,0 +1,607 @@
|
|||||||
|
"""Reconcile internal simulator output against normalized JoinQuant output."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from plugins.joinquant.schema import (
|
||||||
|
JOINQUANT_FILL_COLUMNS,
|
||||||
|
JOINQUANT_PNL_COLUMNS,
|
||||||
|
JOINQUANT_POSITION_COLUMNS,
|
||||||
|
JOINQUANT_TARGET_COLUMNS,
|
||||||
|
RECONCILE_COLUMNS,
|
||||||
|
)
|
||||||
|
from plugins.joinquant.symbols import to_joinquant_symbol
|
||||||
|
|
||||||
|
|
||||||
|
def _date_text(value: object) -> str:
|
||||||
|
if pd.isna(value):
|
||||||
|
return ""
|
||||||
|
return pd.Timestamp(value).strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
|
||||||
|
def _read_parquet(path: str | Path | None) -> pd.DataFrame:
|
||||||
|
if path is None:
|
||||||
|
return pd.DataFrame()
|
||||||
|
return pd.read_parquet(path)
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_common(df: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
out = df.copy()
|
||||||
|
if "date" in out.columns:
|
||||||
|
out["date"] = out["date"].map(_date_text)
|
||||||
|
else:
|
||||||
|
out["date"] = ""
|
||||||
|
if "portfolio_name" not in out.columns:
|
||||||
|
out["portfolio_name"] = portfolio_name
|
||||||
|
out["portfolio_name"] = out["portfolio_name"].fillna(portfolio_name).astype(str)
|
||||||
|
if "symbol_id" in out.columns:
|
||||||
|
out["symbol_id"] = out["symbol_id"].fillna("").astype(str)
|
||||||
|
if "jq_symbol" not in out.columns and "symbol_id" in out.columns:
|
||||||
|
out["jq_symbol"] = out["symbol_id"].map(
|
||||||
|
lambda s: to_joinquant_symbol(s) if s else ""
|
||||||
|
)
|
||||||
|
elif "jq_symbol" in out.columns:
|
||||||
|
out["jq_symbol"] = out["jq_symbol"].fillna("").astype(str)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _numeric(df: pd.DataFrame, column: str, default: float = 0.0) -> pd.Series:
|
||||||
|
if column not in df.columns:
|
||||||
|
return pd.Series([default] * len(df), index=df.index, dtype=float)
|
||||||
|
return pd.to_numeric(df[column], errors="coerce").replace([np.inf, -np.inf], np.nan)
|
||||||
|
|
||||||
|
|
||||||
|
def _weighted_price(group: pd.DataFrame, price_col: str, shares_col: str) -> float:
|
||||||
|
prices = pd.to_numeric(group[price_col], errors="coerce")
|
||||||
|
shares = pd.to_numeric(group[shares_col], errors="coerce").abs()
|
||||||
|
valid = prices.notna() & shares.notna() & (shares > 0)
|
||||||
|
if not valid.any():
|
||||||
|
return np.nan
|
||||||
|
return float(np.average(prices[valid], weights=shares[valid]))
|
||||||
|
|
||||||
|
|
||||||
|
def _load_targets(targets_dir: str | Path, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
root = Path(targets_dir)
|
||||||
|
if not root.exists():
|
||||||
|
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
|
||||||
|
|
||||||
|
files_by_stem: dict[str, Path] = {}
|
||||||
|
for path in sorted(root.glob("*.csv")):
|
||||||
|
files_by_stem[path.stem] = path
|
||||||
|
for path in sorted(root.glob("*.parquet")):
|
||||||
|
files_by_stem[path.stem] = path
|
||||||
|
|
||||||
|
frames: list[pd.DataFrame] = []
|
||||||
|
for path in files_by_stem.values():
|
||||||
|
if path.suffix == ".parquet":
|
||||||
|
frame = pd.read_parquet(path)
|
||||||
|
else:
|
||||||
|
frame = pd.read_csv(path)
|
||||||
|
frames.append(frame)
|
||||||
|
|
||||||
|
if not frames:
|
||||||
|
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
|
||||||
|
targets = pd.concat(frames, ignore_index=True)
|
||||||
|
targets = _normalize_common(targets, portfolio_name)
|
||||||
|
targets = targets[targets["portfolio_name"].astype(str) == portfolio_name]
|
||||||
|
if "target_shares" not in targets.columns:
|
||||||
|
targets["target_shares"] = 0
|
||||||
|
return targets.reindex(columns=JOINQUANT_TARGET_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def _aggregate_targets(targets: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
if targets.empty:
|
||||||
|
return pd.DataFrame(columns=["date", "portfolio_name", "symbol_id", "jq_symbol", "target_shares"])
|
||||||
|
targets = _normalize_common(targets, portfolio_name)
|
||||||
|
targets["target_shares"] = _numeric(targets, "target_shares", 0.0)
|
||||||
|
grouped = (
|
||||||
|
targets.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
|
||||||
|
.agg(jq_symbol=("jq_symbol", "last"), target_shares=("target_shares", "last"))
|
||||||
|
)
|
||||||
|
return grouped
|
||||||
|
|
||||||
|
|
||||||
|
def _aggregate_our_fills(fills: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
if fills.empty:
|
||||||
|
return pd.DataFrame(columns=[
|
||||||
|
"date", "portfolio_name", "symbol_id", "our_filled_shares",
|
||||||
|
"our_position_shares", "our_cost", "our_trade_price", "our_blocked",
|
||||||
|
"our_target_shares",
|
||||||
|
])
|
||||||
|
fills = _normalize_common(fills, portfolio_name)
|
||||||
|
fills["traded_shares"] = _numeric(fills, "traded_shares", 0.0)
|
||||||
|
fills["realized_shares"] = _numeric(fills, "realized_shares", np.nan)
|
||||||
|
fills["trade_cost"] = _numeric(fills, "trade_cost", 0.0).fillna(0.0)
|
||||||
|
fills["target_shares"] = _numeric(fills, "target_shares", np.nan)
|
||||||
|
fills["blocked"] = _numeric(fills, "blocked", 0.0).fillna(0.0)
|
||||||
|
price_col = next(
|
||||||
|
(col for col in ["trade_price", "fill_price", "execution_price", "price"] if col in fills.columns),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
for key, group in fills.groupby(["date", "portfolio_name", "symbol_id"], sort=False):
|
||||||
|
row = {
|
||||||
|
"date": key[0],
|
||||||
|
"portfolio_name": key[1],
|
||||||
|
"symbol_id": key[2],
|
||||||
|
"our_filled_shares": float(group["traded_shares"].sum()),
|
||||||
|
"our_position_shares": float(group["realized_shares"].dropna().iloc[-1])
|
||||||
|
if group["realized_shares"].notna().any() else np.nan,
|
||||||
|
"our_cost": float(group["trade_cost"].sum()),
|
||||||
|
"our_trade_price": _weighted_price(group, price_col, "traded_shares")
|
||||||
|
if price_col else np.nan,
|
||||||
|
"our_blocked": int(group["blocked"].max()),
|
||||||
|
"our_target_shares": float(group["target_shares"].dropna().iloc[-1])
|
||||||
|
if group["target_shares"].notna().any() else np.nan,
|
||||||
|
}
|
||||||
|
rows.append(row)
|
||||||
|
return pd.DataFrame(rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _aggregate_our_positions(positions: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
if positions.empty:
|
||||||
|
return pd.DataFrame(columns=[
|
||||||
|
"date", "portfolio_name", "symbol_id", "jq_symbol",
|
||||||
|
"our_position_fallback", "our_position_price",
|
||||||
|
])
|
||||||
|
positions = _normalize_common(positions, portfolio_name)
|
||||||
|
positions["position_shares"] = _numeric(positions, "position_shares", np.nan)
|
||||||
|
positions["price"] = _numeric(positions, "price", np.nan)
|
||||||
|
return (
|
||||||
|
positions.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
|
||||||
|
.agg(
|
||||||
|
jq_symbol=("jq_symbol", "last"),
|
||||||
|
our_position_fallback=("position_shares", "last"),
|
||||||
|
our_position_price=("price", "last"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _aggregate_jq_fills(fills: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
if fills.empty:
|
||||||
|
return pd.DataFrame(columns=[
|
||||||
|
"date", "portfolio_name", "symbol_id", "jq_filled_shares",
|
||||||
|
"jq_trade_price", "jq_cost", "jq_blocked", "jq_requested_shares",
|
||||||
|
"raw_status",
|
||||||
|
])
|
||||||
|
fills = _normalize_common(fills, portfolio_name)
|
||||||
|
for col in JOINQUANT_FILL_COLUMNS:
|
||||||
|
if col not in fills.columns:
|
||||||
|
fills[col] = np.nan
|
||||||
|
fills["filled_shares"] = _numeric(fills, "filled_shares", 0.0).fillna(0.0)
|
||||||
|
fills["requested_shares"] = _numeric(fills, "requested_shares", np.nan)
|
||||||
|
fills["fill_price"] = _numeric(fills, "fill_price", np.nan)
|
||||||
|
fills["trade_cost"] = _numeric(fills, "trade_cost", 0.0).fillna(0.0)
|
||||||
|
fills["blocked"] = _numeric(fills, "blocked", 0.0).fillna(0.0)
|
||||||
|
fills["raw_status"] = fills["raw_status"].fillna("").astype(str)
|
||||||
|
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
for key, group in fills.groupby(["date", "portfolio_name", "symbol_id"], sort=False):
|
||||||
|
row = {
|
||||||
|
"date": key[0],
|
||||||
|
"portfolio_name": key[1],
|
||||||
|
"symbol_id": key[2],
|
||||||
|
"jq_filled_shares": float(group["filled_shares"].sum()),
|
||||||
|
"jq_trade_price": _weighted_price(group, "fill_price", "filled_shares"),
|
||||||
|
"jq_cost": float(group["trade_cost"].sum()),
|
||||||
|
"jq_blocked": int(group["blocked"].max()),
|
||||||
|
"jq_requested_shares": float(group["requested_shares"].dropna().iloc[-1])
|
||||||
|
if group["requested_shares"].notna().any() else np.nan,
|
||||||
|
"raw_status": ";".join([s for s in group["raw_status"].astype(str) if s]),
|
||||||
|
}
|
||||||
|
rows.append(row)
|
||||||
|
return pd.DataFrame(rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _aggregate_jq_positions(positions: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
|
||||||
|
if positions.empty:
|
||||||
|
return pd.DataFrame(columns=[
|
||||||
|
"date", "portfolio_name", "symbol_id", "jq_symbol", "jq_position_shares",
|
||||||
|
])
|
||||||
|
positions = _normalize_common(positions, portfolio_name)
|
||||||
|
for col in JOINQUANT_POSITION_COLUMNS:
|
||||||
|
if col not in positions.columns:
|
||||||
|
positions[col] = np.nan
|
||||||
|
positions["position_shares"] = _numeric(positions, "position_shares", np.nan)
|
||||||
|
return (
|
||||||
|
positions.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
|
||||||
|
.agg(jq_symbol=("jq_symbol", "last"), jq_position_shares=("position_shares", "last"))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _portfolio_frame(df: pd.DataFrame, portfolio_name: str, prefix: str) -> pd.DataFrame:
|
||||||
|
if df.empty:
|
||||||
|
return pd.DataFrame(columns=["date", "portfolio_name"])
|
||||||
|
out = _normalize_common(df, portfolio_name)
|
||||||
|
for col in JOINQUANT_PNL_COLUMNS:
|
||||||
|
if col not in out.columns:
|
||||||
|
out[col] = np.nan
|
||||||
|
keep = ["date", "portfolio_name", "gross_exposure", "net_exposure", "cash", "total_value", "pnl", "cost", "turnover"]
|
||||||
|
out = out[keep].copy()
|
||||||
|
for col in keep[2:]:
|
||||||
|
out[col] = pd.to_numeric(out[col], errors="coerce")
|
||||||
|
out = out.groupby(["date", "portfolio_name"], as_index=False).last()
|
||||||
|
return out.rename(columns={col: f"{prefix}_{col}" for col in keep[2:]})
|
||||||
|
|
||||||
|
|
||||||
|
def _infer_booksize(targets: pd.DataFrame, our_pnl: pd.DataFrame, jq_pnl: pd.DataFrame) -> float:
|
||||||
|
candidates: list[float] = []
|
||||||
|
if "target_value" in targets.columns:
|
||||||
|
gross = (
|
||||||
|
pd.to_numeric(targets["target_value"], errors="coerce")
|
||||||
|
.abs()
|
||||||
|
.replace([np.inf, -np.inf], np.nan)
|
||||||
|
.dropna()
|
||||||
|
.sum()
|
||||||
|
)
|
||||||
|
if gross > 0:
|
||||||
|
candidates.append(float(gross))
|
||||||
|
for df in (our_pnl, jq_pnl):
|
||||||
|
if "gross_exposure" in df.columns:
|
||||||
|
val = pd.to_numeric(df["gross_exposure"], errors="coerce").max()
|
||||||
|
if pd.notna(val) and val > 0:
|
||||||
|
candidates.append(float(val))
|
||||||
|
return max(candidates) if candidates else 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def _status_reason(raw_status: object) -> str | None:
|
||||||
|
text = str(raw_status or "").lower()
|
||||||
|
if "suspend" in text or "halt" in text:
|
||||||
|
return "SUSPENSION"
|
||||||
|
if "limit_up" in text or "limit up" in text or "up_limit" in text:
|
||||||
|
return "LIMIT_UP_BLOCK"
|
||||||
|
if "limit_down" in text or "limit down" in text or "down_limit" in text:
|
||||||
|
return "LIMIT_DOWN_BLOCK"
|
||||||
|
if "volume" in text or "liquid" in text:
|
||||||
|
return "VOLUME_OR_LIQUIDITY"
|
||||||
|
if "cash" in text or "margin" in text:
|
||||||
|
return "CASH_CONSTRAINT"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _classify_symbol_row(
|
||||||
|
row: pd.Series,
|
||||||
|
*,
|
||||||
|
share_tol: float,
|
||||||
|
price_rel_tol: float,
|
||||||
|
value_tol: float,
|
||||||
|
pnl_tol: float,
|
||||||
|
) -> str:
|
||||||
|
target = row.get("target_shares", np.nan)
|
||||||
|
filled_diff = abs(row.get("filled_share_diff", 0.0))
|
||||||
|
position_diff = abs(row.get("position_share_diff", 0.0))
|
||||||
|
our_present = bool(row.get("_our_present", False))
|
||||||
|
jq_present = bool(row.get("_jq_present", False))
|
||||||
|
|
||||||
|
if pd.notna(target) and target < 0 and (filled_diff > share_tol or position_diff > share_tol or not jq_present):
|
||||||
|
return "SHORT_NOT_SUPPORTED"
|
||||||
|
if not jq_present and (our_present or pd.notna(target)):
|
||||||
|
return "MISSING_IN_JOINQUANT"
|
||||||
|
if not our_present and jq_present:
|
||||||
|
return "MISSING_IN_OUR_SYSTEM"
|
||||||
|
|
||||||
|
status_reason = _status_reason(row.get("raw_status", ""))
|
||||||
|
if (filled_diff > share_tol or position_diff > share_tol) and status_reason:
|
||||||
|
return status_reason
|
||||||
|
if filled_diff > share_tol or position_diff > share_tol:
|
||||||
|
return "UNKNOWN"
|
||||||
|
|
||||||
|
our_price = row.get("our_trade_price", np.nan)
|
||||||
|
jq_price = row.get("jq_trade_price", np.nan)
|
||||||
|
if pd.notna(our_price) and pd.notna(jq_price):
|
||||||
|
denom = max(abs(float(our_price)), abs(float(jq_price)), 1.0)
|
||||||
|
if abs(float(our_price) - float(jq_price)) > price_rel_tol * denom:
|
||||||
|
return "PRICE_MISMATCH"
|
||||||
|
|
||||||
|
if abs(row.get("cost_diff", 0.0)) > value_tol:
|
||||||
|
return "COST_MODEL"
|
||||||
|
if abs(row.get("pnl_diff", 0.0)) > pnl_tol:
|
||||||
|
return "UNKNOWN"
|
||||||
|
return "MATCH"
|
||||||
|
|
||||||
|
|
||||||
|
def _classify_portfolio_row(row: pd.Series, value_tol: float, pnl_tol: float) -> str:
|
||||||
|
our_present = bool(row.get("_our_present", False))
|
||||||
|
jq_present = bool(row.get("_jq_present", False))
|
||||||
|
if not jq_present and our_present:
|
||||||
|
return "MISSING_IN_JOINQUANT"
|
||||||
|
if not our_present and jq_present:
|
||||||
|
return "MISSING_IN_OUR_SYSTEM"
|
||||||
|
if abs(row.get("cost_diff", 0.0)) > value_tol:
|
||||||
|
return "COST_MODEL"
|
||||||
|
if abs(row.get("pnl_diff", 0.0)) > pnl_tol:
|
||||||
|
return "UNKNOWN"
|
||||||
|
return "MATCH"
|
||||||
|
|
||||||
|
|
||||||
|
def _build_symbol_reconcile(
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
targets: pd.DataFrame,
|
||||||
|
our_fills: pd.DataFrame,
|
||||||
|
our_positions: pd.DataFrame,
|
||||||
|
our_pnl: pd.DataFrame,
|
||||||
|
jq_fills: pd.DataFrame,
|
||||||
|
jq_positions: pd.DataFrame,
|
||||||
|
jq_pnl: pd.DataFrame,
|
||||||
|
share_tol: float,
|
||||||
|
price_rel_tol: float,
|
||||||
|
value_tol: float,
|
||||||
|
pnl_tol: float,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
target_agg = _aggregate_targets(targets, portfolio_name)
|
||||||
|
our_fill_agg = _aggregate_our_fills(our_fills, portfolio_name)
|
||||||
|
our_pos_agg = _aggregate_our_positions(our_positions, portfolio_name)
|
||||||
|
jq_fill_agg = _aggregate_jq_fills(jq_fills, portfolio_name)
|
||||||
|
jq_pos_agg = _aggregate_jq_positions(jq_positions, portfolio_name)
|
||||||
|
|
||||||
|
key_cols = ["date", "portfolio_name", "symbol_id"]
|
||||||
|
keys = []
|
||||||
|
for frame in [target_agg, our_fill_agg, our_pos_agg, jq_fill_agg, jq_pos_agg]:
|
||||||
|
if not frame.empty:
|
||||||
|
keys.append(frame[key_cols])
|
||||||
|
if not keys:
|
||||||
|
return pd.DataFrame(columns=RECONCILE_COLUMNS)
|
||||||
|
|
||||||
|
base = pd.concat(keys, ignore_index=True).drop_duplicates()
|
||||||
|
result = base.merge(target_agg, on=key_cols, how="left")
|
||||||
|
result = result.merge(our_fill_agg, on=key_cols, how="left")
|
||||||
|
result = result.merge(our_pos_agg, on=key_cols, how="left", suffixes=("", "_ourpos"))
|
||||||
|
result = result.merge(jq_fill_agg, on=key_cols, how="left")
|
||||||
|
result = result.merge(jq_pos_agg, on=key_cols, how="left", suffixes=("", "_jqpos"))
|
||||||
|
|
||||||
|
jq_symbol_cols = [col for col in result.columns if col.startswith("jq_symbol")]
|
||||||
|
jq_symbol_values = result[jq_symbol_cols].copy() if jq_symbol_cols else pd.DataFrame(index=result.index)
|
||||||
|
result["jq_symbol"] = ""
|
||||||
|
for col in jq_symbol_values.columns:
|
||||||
|
values = jq_symbol_values[col].fillna("").astype(str)
|
||||||
|
result["jq_symbol"] = result["jq_symbol"].mask(
|
||||||
|
result["jq_symbol"].eq("") & values.ne(""),
|
||||||
|
values,
|
||||||
|
)
|
||||||
|
result["jq_symbol"] = result.apply(
|
||||||
|
lambda row: row["jq_symbol"] or to_joinquant_symbol(row["symbol_id"]),
|
||||||
|
axis=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
result["_our_present"] = (
|
||||||
|
result[["our_filled_shares", "our_position_shares", "our_position_fallback"]]
|
||||||
|
.notna()
|
||||||
|
.any(axis=1)
|
||||||
|
)
|
||||||
|
result["_jq_present"] = (
|
||||||
|
result[["jq_filled_shares", "jq_position_shares"]].notna().any(axis=1)
|
||||||
|
)
|
||||||
|
|
||||||
|
target_shares = pd.to_numeric(result["target_shares"], errors="coerce")
|
||||||
|
our_target = pd.to_numeric(result["our_target_shares"], errors="coerce")
|
||||||
|
jq_target = pd.to_numeric(result["jq_requested_shares"], errors="coerce")
|
||||||
|
result["target_shares"] = target_shares.where(target_shares.notna(), our_target)
|
||||||
|
result["target_shares"] = result["target_shares"].where(
|
||||||
|
result["target_shares"].notna(),
|
||||||
|
jq_target,
|
||||||
|
)
|
||||||
|
|
||||||
|
our_position = pd.to_numeric(result["our_position_shares"], errors="coerce")
|
||||||
|
our_position_fallback = pd.to_numeric(result["our_position_fallback"], errors="coerce")
|
||||||
|
result["our_position_shares"] = our_position.where(
|
||||||
|
our_position.notna(),
|
||||||
|
our_position_fallback,
|
||||||
|
)
|
||||||
|
|
||||||
|
for col in [
|
||||||
|
"target_shares", "our_filled_shares", "jq_filled_shares",
|
||||||
|
"our_position_shares", "jq_position_shares", "our_cost", "jq_cost",
|
||||||
|
]:
|
||||||
|
result[col] = pd.to_numeric(result[col], errors="coerce").fillna(0.0)
|
||||||
|
|
||||||
|
result["filled_share_diff"] = result["our_filled_shares"] - result["jq_filled_shares"]
|
||||||
|
result["position_share_diff"] = result["our_position_shares"] - result["jq_position_shares"]
|
||||||
|
result["trade_price_diff"] = np.where(
|
||||||
|
result["our_trade_price"].notna() & result["jq_trade_price"].notna(),
|
||||||
|
result["our_trade_price"] - result["jq_trade_price"],
|
||||||
|
np.nan,
|
||||||
|
)
|
||||||
|
result["cost_diff"] = result["our_cost"] - result["jq_cost"]
|
||||||
|
|
||||||
|
our_daily = _portfolio_frame(our_pnl, portfolio_name, "our")
|
||||||
|
jq_daily = _portfolio_frame(jq_pnl, portfolio_name, "jq")
|
||||||
|
pnl_daily = our_daily.merge(jq_daily, on=["date", "portfolio_name"], how="outer")
|
||||||
|
if not pnl_daily.empty:
|
||||||
|
pnl_daily["our_pnl"] = pd.to_numeric(pnl_daily.get("our_pnl"), errors="coerce").fillna(0.0)
|
||||||
|
pnl_daily["jq_pnl"] = pd.to_numeric(pnl_daily.get("jq_pnl"), errors="coerce").fillna(0.0)
|
||||||
|
pnl_daily["pnl_diff"] = pnl_daily["our_pnl"] - pnl_daily["jq_pnl"]
|
||||||
|
result = result.merge(
|
||||||
|
pnl_daily[["date", "portfolio_name", "our_pnl", "jq_pnl", "pnl_diff"]],
|
||||||
|
on=["date", "portfolio_name"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
result["our_pnl"] = 0.0
|
||||||
|
result["jq_pnl"] = 0.0
|
||||||
|
result["pnl_diff"] = 0.0
|
||||||
|
for col in ["our_pnl", "jq_pnl", "pnl_diff"]:
|
||||||
|
result[col] = pd.to_numeric(result[col], errors="coerce").fillna(0.0)
|
||||||
|
|
||||||
|
result["raw_status"] = result.get("raw_status", "").fillna("")
|
||||||
|
result["diff_reason"] = result.apply(
|
||||||
|
_classify_symbol_row,
|
||||||
|
axis=1,
|
||||||
|
share_tol=share_tol,
|
||||||
|
price_rel_tol=price_rel_tol,
|
||||||
|
value_tol=value_tol,
|
||||||
|
pnl_tol=pnl_tol,
|
||||||
|
)
|
||||||
|
|
||||||
|
return result[RECONCILE_COLUMNS].sort_values(
|
||||||
|
["date", "portfolio_name", "symbol_id"]
|
||||||
|
).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_portfolio_summary(
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
our_pnl: pd.DataFrame,
|
||||||
|
jq_pnl: pd.DataFrame,
|
||||||
|
value_tol: float,
|
||||||
|
pnl_tol: float,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
our = _portfolio_frame(our_pnl, portfolio_name, "our")
|
||||||
|
jq = _portfolio_frame(jq_pnl, portfolio_name, "jq")
|
||||||
|
if our.empty and jq.empty:
|
||||||
|
return pd.DataFrame(columns=[
|
||||||
|
"date", "portfolio_name", "diff_reason",
|
||||||
|
"our_pnl", "jq_pnl", "pnl_diff",
|
||||||
|
])
|
||||||
|
summary = our.merge(jq, on=["date", "portfolio_name"], how="outer")
|
||||||
|
summary["_our_present"] = summary.filter(regex=r"^our_").notna().any(axis=1)
|
||||||
|
summary["_jq_present"] = summary.filter(regex=r"^jq_").notna().any(axis=1)
|
||||||
|
metrics = ["gross_exposure", "net_exposure", "cash", "total_value", "pnl", "cost", "turnover"]
|
||||||
|
for metric in metrics:
|
||||||
|
our_col = f"our_{metric}"
|
||||||
|
jq_col = f"jq_{metric}"
|
||||||
|
if our_col not in summary.columns:
|
||||||
|
summary[our_col] = np.nan
|
||||||
|
if jq_col not in summary.columns:
|
||||||
|
summary[jq_col] = np.nan
|
||||||
|
summary[f"{metric}_diff"] = (
|
||||||
|
pd.to_numeric(summary[our_col], errors="coerce").fillna(0.0)
|
||||||
|
- pd.to_numeric(summary[jq_col], errors="coerce").fillna(0.0)
|
||||||
|
)
|
||||||
|
summary["our_cumulative_pnl"] = (
|
||||||
|
pd.to_numeric(summary["our_pnl"], errors="coerce").fillna(0.0).cumsum()
|
||||||
|
)
|
||||||
|
summary["jq_cumulative_pnl"] = (
|
||||||
|
pd.to_numeric(summary["jq_pnl"], errors="coerce").fillna(0.0).cumsum()
|
||||||
|
)
|
||||||
|
summary["cumulative_pnl_diff"] = summary["our_cumulative_pnl"] - summary["jq_cumulative_pnl"]
|
||||||
|
summary["diff_reason"] = summary.apply(
|
||||||
|
_classify_portfolio_row,
|
||||||
|
axis=1,
|
||||||
|
value_tol=value_tol,
|
||||||
|
pnl_tol=pnl_tol,
|
||||||
|
)
|
||||||
|
summary = summary.drop(columns=["_our_present", "_jq_present"])
|
||||||
|
return summary.sort_values(["date", "portfolio_name"]).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_summary_md(
|
||||||
|
path: Path,
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
symbol_report: pd.DataFrame,
|
||||||
|
portfolio_summary: pd.DataFrame,
|
||||||
|
) -> None:
|
||||||
|
symbol_counts = (
|
||||||
|
symbol_report["diff_reason"].value_counts().sort_index()
|
||||||
|
if not symbol_report.empty else pd.Series(dtype=int)
|
||||||
|
)
|
||||||
|
portfolio_counts = (
|
||||||
|
portfolio_summary["diff_reason"].value_counts().sort_index()
|
||||||
|
if not portfolio_summary.empty else pd.Series(dtype=int)
|
||||||
|
)
|
||||||
|
lines = [
|
||||||
|
"# JoinQuant Reconciliation Summary",
|
||||||
|
"",
|
||||||
|
f"Portfolio: `{portfolio_name}`",
|
||||||
|
"",
|
||||||
|
"## Per-symbol Difference Counts",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
if symbol_counts.empty:
|
||||||
|
lines.append("No per-symbol rows were produced.")
|
||||||
|
else:
|
||||||
|
for reason, count in symbol_counts.items():
|
||||||
|
lines.append(f"- {reason}: {int(count)}")
|
||||||
|
lines.extend(["", "## Daily Portfolio Difference Counts", ""])
|
||||||
|
if portfolio_counts.empty:
|
||||||
|
lines.append("No daily portfolio rows were produced.")
|
||||||
|
else:
|
||||||
|
for reason, count in portfolio_counts.items():
|
||||||
|
lines.append(f"- {reason}: {int(count)}")
|
||||||
|
|
||||||
|
if not portfolio_summary.empty:
|
||||||
|
lines.extend(["", "## Daily Portfolio Preview", ""])
|
||||||
|
preview_cols = [
|
||||||
|
"date", "diff_reason", "our_pnl", "jq_pnl", "pnl_diff",
|
||||||
|
"our_cost", "jq_cost", "cost_diff",
|
||||||
|
]
|
||||||
|
preview_cols = [col for col in preview_cols if col in portfolio_summary.columns]
|
||||||
|
lines.append(",".join(preview_cols))
|
||||||
|
for row in portfolio_summary[preview_cols].head(20).itertuples(index=False):
|
||||||
|
lines.append(",".join(str(value) for value in row))
|
||||||
|
|
||||||
|
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def reconcile_joinquant(
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
targets_dir: str | Path,
|
||||||
|
our_fills_path: str | Path,
|
||||||
|
our_positions_path: str | Path,
|
||||||
|
our_pnl_path: str | Path,
|
||||||
|
jq_fills_path: str | Path,
|
||||||
|
jq_positions_path: str | Path,
|
||||||
|
jq_pnl_path: str | Path,
|
||||||
|
out_dir: str | Path = "plugins_output/joinquant/reconcile",
|
||||||
|
share_tolerance: float = 0.0,
|
||||||
|
price_rel_tolerance: float = 1e-4,
|
||||||
|
pnl_tolerance: float = 1.0,
|
||||||
|
booksize: float | None = None,
|
||||||
|
) -> dict[str, Path]:
|
||||||
|
"""Reconcile JoinQuant output against internal simulator output."""
|
||||||
|
targets = _load_targets(targets_dir, portfolio_name)
|
||||||
|
our_fills = _read_parquet(our_fills_path)
|
||||||
|
our_positions = _read_parquet(our_positions_path)
|
||||||
|
our_pnl = _read_parquet(our_pnl_path)
|
||||||
|
jq_fills = _read_parquet(jq_fills_path)
|
||||||
|
jq_positions = _read_parquet(jq_positions_path)
|
||||||
|
jq_pnl = _read_parquet(jq_pnl_path)
|
||||||
|
|
||||||
|
inferred_booksize = booksize or _infer_booksize(targets, our_pnl, jq_pnl)
|
||||||
|
value_tol = max(1.0, 1e-6 * float(inferred_booksize))
|
||||||
|
|
||||||
|
symbol_report = _build_symbol_reconcile(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
targets=targets,
|
||||||
|
our_fills=our_fills,
|
||||||
|
our_positions=our_positions,
|
||||||
|
our_pnl=our_pnl,
|
||||||
|
jq_fills=jq_fills,
|
||||||
|
jq_positions=jq_positions,
|
||||||
|
jq_pnl=jq_pnl,
|
||||||
|
share_tol=share_tolerance,
|
||||||
|
price_rel_tol=price_rel_tolerance,
|
||||||
|
value_tol=value_tol,
|
||||||
|
pnl_tol=pnl_tolerance,
|
||||||
|
)
|
||||||
|
portfolio_summary = _build_portfolio_summary(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
our_pnl=our_pnl,
|
||||||
|
jq_pnl=jq_pnl,
|
||||||
|
value_tol=value_tol,
|
||||||
|
pnl_tol=pnl_tolerance,
|
||||||
|
)
|
||||||
|
|
||||||
|
root = Path(out_dir) / portfolio_name
|
||||||
|
root.mkdir(parents=True, exist_ok=True)
|
||||||
|
paths = {
|
||||||
|
"daily_reconcile": root / "daily_reconcile.pq",
|
||||||
|
"summary_csv": root / "summary.csv",
|
||||||
|
"summary_md": root / "summary.md",
|
||||||
|
}
|
||||||
|
symbol_report.to_parquet(paths["daily_reconcile"], index=False)
|
||||||
|
portfolio_summary.to_csv(paths["summary_csv"], index=False)
|
||||||
|
_write_summary_md(
|
||||||
|
paths["summary_md"],
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
symbol_report=symbol_report,
|
||||||
|
portfolio_summary=portfolio_summary,
|
||||||
|
)
|
||||||
|
return paths
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
"""Column contracts for the JoinQuant comparison plugin."""
|
||||||
|
|
||||||
|
from typing import Final
|
||||||
|
|
||||||
|
|
||||||
|
JOINQUANT_TARGET_COLUMNS: Final[list[str]] = [
|
||||||
|
"date",
|
||||||
|
"portfolio_name",
|
||||||
|
"symbol_id",
|
||||||
|
"jq_symbol",
|
||||||
|
"target_shares",
|
||||||
|
"target_value",
|
||||||
|
"target_weight",
|
||||||
|
"export_mode",
|
||||||
|
"snapshot_id",
|
||||||
|
]
|
||||||
|
|
||||||
|
JOINQUANT_FILL_COLUMNS: Final[list[str]] = [
|
||||||
|
"date",
|
||||||
|
"portfolio_name",
|
||||||
|
"symbol_id",
|
||||||
|
"jq_symbol",
|
||||||
|
"order_id",
|
||||||
|
"side",
|
||||||
|
"requested_shares",
|
||||||
|
"filled_shares",
|
||||||
|
"fill_price",
|
||||||
|
"trade_value",
|
||||||
|
"trade_cost",
|
||||||
|
"blocked",
|
||||||
|
"raw_status",
|
||||||
|
]
|
||||||
|
|
||||||
|
JOINQUANT_POSITION_COLUMNS: Final[list[str]] = [
|
||||||
|
"date",
|
||||||
|
"portfolio_name",
|
||||||
|
"symbol_id",
|
||||||
|
"jq_symbol",
|
||||||
|
"position_shares",
|
||||||
|
"position_value",
|
||||||
|
"cash",
|
||||||
|
"total_value",
|
||||||
|
]
|
||||||
|
|
||||||
|
JOINQUANT_PNL_COLUMNS: Final[list[str]] = [
|
||||||
|
"date",
|
||||||
|
"portfolio_name",
|
||||||
|
"gross_exposure",
|
||||||
|
"net_exposure",
|
||||||
|
"cash",
|
||||||
|
"total_value",
|
||||||
|
"pnl",
|
||||||
|
"cost",
|
||||||
|
"turnover",
|
||||||
|
]
|
||||||
|
|
||||||
|
RECONCILE_COLUMNS: Final[list[str]] = [
|
||||||
|
"date",
|
||||||
|
"portfolio_name",
|
||||||
|
"symbol_id",
|
||||||
|
"jq_symbol",
|
||||||
|
"target_shares",
|
||||||
|
"our_filled_shares",
|
||||||
|
"jq_filled_shares",
|
||||||
|
"filled_share_diff",
|
||||||
|
"our_position_shares",
|
||||||
|
"jq_position_shares",
|
||||||
|
"position_share_diff",
|
||||||
|
"our_trade_price",
|
||||||
|
"jq_trade_price",
|
||||||
|
"trade_price_diff",
|
||||||
|
"our_cost",
|
||||||
|
"jq_cost",
|
||||||
|
"cost_diff",
|
||||||
|
"our_pnl",
|
||||||
|
"jq_pnl",
|
||||||
|
"pnl_diff",
|
||||||
|
"diff_reason",
|
||||||
|
]
|
||||||
|
|
||||||
|
DIFF_REASONS: Final[list[str]] = [
|
||||||
|
"MATCH",
|
||||||
|
"SYMBOL_MAPPING",
|
||||||
|
"PRICE_MISMATCH",
|
||||||
|
"LOT_ROUNDING",
|
||||||
|
"SUSPENSION",
|
||||||
|
"LIMIT_UP_BLOCK",
|
||||||
|
"LIMIT_DOWN_BLOCK",
|
||||||
|
"VOLUME_OR_LIQUIDITY",
|
||||||
|
"COST_MODEL",
|
||||||
|
"CASH_CONSTRAINT",
|
||||||
|
"SHORT_NOT_SUPPORTED",
|
||||||
|
"CORPORATE_ACTION",
|
||||||
|
"JOINQUANT_INTERNAL_ROUNDING",
|
||||||
|
"MISSING_IN_OUR_SYSTEM",
|
||||||
|
"MISSING_IN_JOINQUANT",
|
||||||
|
"UNKNOWN",
|
||||||
|
]
|
||||||
|
|
||||||
@@ -0,0 +1,191 @@
|
|||||||
|
"""End-to-end local smoke preparation for JoinQuant comparison."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.common.schema import POSITION_COLUMNS
|
||||||
|
from pipeline.data.downloader import download_universe
|
||||||
|
from pipeline.portfolio.constraints import get_constraint
|
||||||
|
from pipeline.portfolio.simulator import ReferenceSimulator
|
||||||
|
from plugins.joinquant.export_targets import export_targets
|
||||||
|
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
|
||||||
|
|
||||||
|
|
||||||
|
def build_fixed_share_positions(
|
||||||
|
data: pd.DataFrame,
|
||||||
|
*,
|
||||||
|
trade_symbol: str,
|
||||||
|
portfolio_name: str,
|
||||||
|
shares: int,
|
||||||
|
booksize: float,
|
||||||
|
max_signal_dates: int | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Create a deterministic long-only fixed-share position book.
|
||||||
|
|
||||||
|
The final available data date is excluded because the internal simulator
|
||||||
|
executes each signal date at the next available open.
|
||||||
|
"""
|
||||||
|
data = data.copy()
|
||||||
|
data["date"] = pd.to_datetime(data["date"]).dt.normalize()
|
||||||
|
symbol_data = (
|
||||||
|
data[data["symbol_id"].astype(str) == trade_symbol]
|
||||||
|
.sort_values("date")
|
||||||
|
.reset_index(drop=True)
|
||||||
|
)
|
||||||
|
if len(symbol_data) < 2:
|
||||||
|
raise ValueError(f"Need at least two daily bars for {trade_symbol}")
|
||||||
|
|
||||||
|
signal_data = symbol_data.iloc[:-1].copy()
|
||||||
|
if max_signal_dates is not None and max_signal_dates > 0:
|
||||||
|
signal_data = signal_data.tail(max_signal_dates)
|
||||||
|
if signal_data.empty:
|
||||||
|
raise ValueError("No signal dates available after excluding final data date")
|
||||||
|
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
for row in signal_data.itertuples(index=False):
|
||||||
|
price = float(row.close)
|
||||||
|
target_value = float(shares * price)
|
||||||
|
rows.append({
|
||||||
|
"symbol_id": trade_symbol,
|
||||||
|
"date": pd.Timestamp(row.date),
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"target_weight": target_value / float(booksize),
|
||||||
|
"target_value": target_value,
|
||||||
|
"target_shares": float(shares),
|
||||||
|
"position_shares": int(shares),
|
||||||
|
"position_value": target_value,
|
||||||
|
"price": price,
|
||||||
|
})
|
||||||
|
return pd.DataFrame(rows, columns=POSITION_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def prepare_smoke_test(
|
||||||
|
*,
|
||||||
|
out_dir: str | Path,
|
||||||
|
universe: str = "sh600000,sz000001,sh600519,sz002594,sz300750",
|
||||||
|
trade_symbol: str = "sh600000",
|
||||||
|
start_date: str = "2024-01-02",
|
||||||
|
end_date: str = "2024-01-12",
|
||||||
|
portfolio_name: str = "jq_smoke_one_stock_long",
|
||||||
|
shares: int = 1000,
|
||||||
|
booksize: float = 1_000_000.0,
|
||||||
|
max_signal_dates: int = 3,
|
||||||
|
cost_bps: float = 5.0,
|
||||||
|
slippage_bps: float = 5.0,
|
||||||
|
volume_frac: float = 0.02,
|
||||||
|
force: bool = False,
|
||||||
|
) -> dict[str, object]:
|
||||||
|
"""Run the local side of a tiny real-data JoinQuant smoke test."""
|
||||||
|
root = Path(out_dir)
|
||||||
|
root.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
stats = download_universe(
|
||||||
|
universe=universe,
|
||||||
|
start_date=start_date,
|
||||||
|
end_date=end_date,
|
||||||
|
output_dir=root / "daily_bars",
|
||||||
|
max_symbols=0,
|
||||||
|
chunk_size=100,
|
||||||
|
adjust="qfq",
|
||||||
|
)
|
||||||
|
data_path = Path(stats["dataset_path"])
|
||||||
|
data = pd.read_parquet(data_path)
|
||||||
|
|
||||||
|
positions = build_fixed_share_positions(
|
||||||
|
data,
|
||||||
|
trade_symbol=trade_symbol,
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
shares=shares,
|
||||||
|
booksize=booksize,
|
||||||
|
max_signal_dates=max_signal_dates,
|
||||||
|
)
|
||||||
|
portfolio_dir = root / "portfolio"
|
||||||
|
portfolio_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
positions_path = portfolio_dir / f"{portfolio_name}.pq"
|
||||||
|
positions.to_parquet(positions_path, index=False)
|
||||||
|
|
||||||
|
constraints = [
|
||||||
|
get_constraint("suspension"),
|
||||||
|
get_constraint("price_limit"),
|
||||||
|
get_constraint("volume_cap", max_frac=volume_frac),
|
||||||
|
]
|
||||||
|
sim = ReferenceSimulator(
|
||||||
|
constraints=constraints,
|
||||||
|
cost_bps=cost_bps,
|
||||||
|
slippage_bps=slippage_bps,
|
||||||
|
)
|
||||||
|
fills, pnl = sim.run(positions, data)
|
||||||
|
execution_dir = root / "execution"
|
||||||
|
fills_dir = execution_dir / "fills"
|
||||||
|
pnl_dir = execution_dir / "pnl"
|
||||||
|
fills_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
pnl_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
fills_path = fills_dir / f"{portfolio_name}.pq"
|
||||||
|
pnl_path = pnl_dir / f"{portfolio_name}.pq"
|
||||||
|
fills.to_parquet(fills_path, index=False)
|
||||||
|
pnl.to_parquet(pnl_path, index=False)
|
||||||
|
|
||||||
|
target_root = root / "plugins_output" / "joinquant" / "targets_aligned"
|
||||||
|
snapshots = export_targets(
|
||||||
|
positions_path=positions_path,
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
mode="target_shares",
|
||||||
|
out_dir=target_root,
|
||||||
|
execution_calendar_path=data_path,
|
||||||
|
force=force,
|
||||||
|
)
|
||||||
|
wrapper_path = root / "plugins_output" / "joinquant" / f"wrapper_strategy_{portfolio_name}.py"
|
||||||
|
write_wrapper_strategy(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
mode="target_shares",
|
||||||
|
out_path=wrapper_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
export_dir = root / "joinquant_exports"
|
||||||
|
export_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
manifest = {
|
||||||
|
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"universe": universe,
|
||||||
|
"trade_symbol": trade_symbol,
|
||||||
|
"start_date": start_date,
|
||||||
|
"end_date": end_date,
|
||||||
|
"shares": shares,
|
||||||
|
"booksize": booksize,
|
||||||
|
"data_path": str(data_path),
|
||||||
|
"positions_path": str(positions_path),
|
||||||
|
"fills_path": str(fills_path),
|
||||||
|
"pnl_path": str(pnl_path),
|
||||||
|
"targets_dir": str(target_root / portfolio_name),
|
||||||
|
"wrapper_path": str(wrapper_path),
|
||||||
|
"joinquant_export_dir": str(export_dir),
|
||||||
|
"expected_joinquant_csvs": {
|
||||||
|
"fills": str(export_dir / "jq_fills.csv"),
|
||||||
|
"positions": str(export_dir / "jq_positions.csv"),
|
||||||
|
"pnl": str(export_dir / "jq_pnl.csv"),
|
||||||
|
},
|
||||||
|
"target_snapshots": snapshots,
|
||||||
|
"local_summary": {
|
||||||
|
"n_data_rows": int(len(data)),
|
||||||
|
"n_position_rows": int(len(positions)),
|
||||||
|
"n_fill_rows": int(len(fills)),
|
||||||
|
"n_pnl_rows": int(len(pnl)),
|
||||||
|
"total_pnl": float(pnl["pnl"].sum()) if len(pnl) else 0.0,
|
||||||
|
"total_cost": float(pnl["cost"].sum()) if len(pnl) else 0.0,
|
||||||
|
"blocked_trades": int(fills["blocked"].sum()) if len(fills) else 0,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
manifest_path = root / "joinquant_smoke_manifest.json"
|
||||||
|
manifest["manifest_path"] = str(manifest_path)
|
||||||
|
manifest_path.write_text(
|
||||||
|
json.dumps(manifest, indent=2, ensure_ascii=False) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return manifest
|
||||||
|
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
"""Symbol conversion between internal A-share ids and JoinQuant ids."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
|
||||||
|
_INTERNAL_RE = re.compile(r"^(?P<prefix>sh|sz)(?P<code>\d{6})$", re.IGNORECASE)
|
||||||
|
_JOINQUANT_RE = re.compile(
|
||||||
|
r"^(?P<code>\d{6})\.(?P<exchange>XSHG|XSHE)$", re.IGNORECASE
|
||||||
|
)
|
||||||
|
_BARE_RE = re.compile(r"^\d{6}$")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_internal(prefix: str, code: str) -> None:
|
||||||
|
if prefix == "sh" and code.startswith("6"):
|
||||||
|
return
|
||||||
|
if prefix == "sz" and code.startswith(("0", "3")):
|
||||||
|
return
|
||||||
|
raise ValueError(f"Unsupported A-share symbol: {prefix}{code}")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_joinquant(code: str, exchange: str) -> None:
|
||||||
|
if exchange == "XSHG" and code.startswith("6"):
|
||||||
|
return
|
||||||
|
if exchange == "XSHE" and code.startswith(("0", "3")):
|
||||||
|
return
|
||||||
|
raise ValueError(f"Unsupported JoinQuant A-share symbol: {code}.{exchange}")
|
||||||
|
|
||||||
|
|
||||||
|
def to_joinquant_symbol(symbol_id: str) -> str:
|
||||||
|
"""Convert an internal symbol like ``sh600000`` to ``600000.XSHG``.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
symbol_id: Internal A-share id with ``sh`` or ``sz`` prefix.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
JoinQuant security id.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: If the symbol is malformed or outside supported A-share
|
||||||
|
Shanghai/Shenzhen equity prefixes.
|
||||||
|
"""
|
||||||
|
text = str(symbol_id).strip().lower()
|
||||||
|
match = _INTERNAL_RE.match(text)
|
||||||
|
if not match:
|
||||||
|
raise ValueError(f"Invalid internal symbol: {symbol_id!r}")
|
||||||
|
|
||||||
|
prefix = match.group("prefix").lower()
|
||||||
|
code = match.group("code")
|
||||||
|
_validate_internal(prefix, code)
|
||||||
|
exchange = "XSHG" if prefix == "sh" else "XSHE"
|
||||||
|
return f"{code}.{exchange}"
|
||||||
|
|
||||||
|
|
||||||
|
def from_joinquant_symbol(jq_symbol: str) -> str:
|
||||||
|
"""Convert a JoinQuant symbol like ``600000.XSHG`` to ``sh600000``."""
|
||||||
|
text = str(jq_symbol).strip().upper()
|
||||||
|
match = _JOINQUANT_RE.match(text)
|
||||||
|
if not match:
|
||||||
|
raise ValueError(f"Invalid JoinQuant symbol: {jq_symbol!r}")
|
||||||
|
|
||||||
|
code = match.group("code")
|
||||||
|
exchange = match.group("exchange").upper()
|
||||||
|
_validate_joinquant(code, exchange)
|
||||||
|
prefix = "sh" if exchange == "XSHG" else "sz"
|
||||||
|
return f"{prefix}{code}"
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_symbol_pair(value: object) -> tuple[str, str]:
|
||||||
|
"""Return ``(symbol_id, jq_symbol)`` from any supported symbol spelling."""
|
||||||
|
text = str(value).strip()
|
||||||
|
if not text or text.lower() == "nan":
|
||||||
|
raise ValueError("Missing symbol")
|
||||||
|
|
||||||
|
if _INTERNAL_RE.match(text):
|
||||||
|
symbol_id = text.lower()
|
||||||
|
return symbol_id, to_joinquant_symbol(symbol_id)
|
||||||
|
|
||||||
|
if _JOINQUANT_RE.match(text):
|
||||||
|
jq_symbol = text.upper()
|
||||||
|
return from_joinquant_symbol(jq_symbol), jq_symbol
|
||||||
|
|
||||||
|
if _BARE_RE.match(text):
|
||||||
|
if text.startswith("6"):
|
||||||
|
symbol_id = f"sh{text}"
|
||||||
|
elif text.startswith(("0", "3")):
|
||||||
|
symbol_id = f"sz{text}"
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unsupported bare A-share code: {text}")
|
||||||
|
return symbol_id, to_joinquant_symbol(symbol_id)
|
||||||
|
|
||||||
|
raise ValueError(f"Unsupported symbol: {value!r}")
|
||||||
|
|
||||||
@@ -0,0 +1,271 @@
|
|||||||
|
"""Generate a standalone JoinQuant wrapper strategy.
|
||||||
|
|
||||||
|
The module also defines default JoinQuant strategy hooks by executing the same
|
||||||
|
template with ``run1`` / ``target_shares`` defaults. That means this file can be
|
||||||
|
copied directly into JoinQuant for a quick smoke test, while the CLI can still
|
||||||
|
write a configured standalone file for a real run.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from string import Template
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
|
||||||
|
WrapperMode = Literal["target_shares", "target_value"]
|
||||||
|
|
||||||
|
|
||||||
|
_WRAPPER_TEMPLATE = Template(
|
||||||
|
r'''# Standalone JoinQuant target wrapper generated by chinese-equity-quant.
|
||||||
|
# Copy this file and the exported daily CSV target files into JoinQuant.
|
||||||
|
#
|
||||||
|
# The file loader is isolated in _read_target_file(). The default implementation
|
||||||
|
# uses JoinQuant's read_file API for uploaded files. If your JoinQuant runtime
|
||||||
|
# allows HTTP or another storage backend, replace only that function.
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
|
||||||
|
|
||||||
|
PORTFOLIO_NAME = "${portfolio_name}"
|
||||||
|
TARGET_MODE = "${mode}"
|
||||||
|
ALLOW_SHORT = ${allow_short}
|
||||||
|
TARGET_FILE_PREFIX = "" # Optional uploaded-file prefix, for example "run1/"
|
||||||
|
_EMBEDDED_TARGETS = ${embedded_targets}
|
||||||
|
|
||||||
|
|
||||||
|
def initialize(context):
|
||||||
|
set_benchmark("000300.XSHG")
|
||||||
|
set_option("use_real_price", True)
|
||||||
|
g.portfolio_name = PORTFOLIO_NAME
|
||||||
|
g.target_mode = TARGET_MODE
|
||||||
|
g.targets_by_date = {}
|
||||||
|
run_daily(load_targets, time="before_open")
|
||||||
|
run_daily(rebalance_at_open, time="open")
|
||||||
|
run_daily(record_after_close, time="after_close")
|
||||||
|
|
||||||
|
|
||||||
|
def _today_text(context):
|
||||||
|
return context.current_dt.strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
|
||||||
|
def _today_file_name(context):
|
||||||
|
return context.current_dt.strftime("%Y%m%d") + ".csv"
|
||||||
|
|
||||||
|
|
||||||
|
def _read_target_file(file_name):
|
||||||
|
${embedded_target_read}
|
||||||
|
data = read_file(TARGET_FILE_PREFIX + file_name)
|
||||||
|
if isinstance(data, bytes):
|
||||||
|
data = data.decode("utf-8")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def _load_target_rows(context):
|
||||||
|
file_name = _today_file_name(context)
|
||||||
|
text = _read_target_file(file_name)
|
||||||
|
rows = list(csv.DictReader(io.StringIO(text)))
|
||||||
|
clean_rows = []
|
||||||
|
for row in rows:
|
||||||
|
if row.get("portfolio_name") and row["portfolio_name"] != PORTFOLIO_NAME:
|
||||||
|
continue
|
||||||
|
jq_symbol = row.get("jq_symbol") or row.get("security") or row.get("symbol")
|
||||||
|
if not jq_symbol:
|
||||||
|
log.warn("Skipping target row with no jq_symbol: %s" % row)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if TARGET_MODE == "target_shares":
|
||||||
|
target = int(float(row.get("target_shares") or 0))
|
||||||
|
elif TARGET_MODE == "target_value":
|
||||||
|
target = float(row.get("target_value") or 0.0)
|
||||||
|
else:
|
||||||
|
raise ValueError("Unsupported TARGET_MODE: %s" % TARGET_MODE)
|
||||||
|
|
||||||
|
if not ALLOW_SHORT and target < 0:
|
||||||
|
log.warn(
|
||||||
|
"SHORT_NOT_SUPPORTED clipping %s target from %s to 0" %
|
||||||
|
(jq_symbol, target)
|
||||||
|
)
|
||||||
|
target = 0
|
||||||
|
|
||||||
|
clean_rows.append({"jq_symbol": jq_symbol, "target": target, "raw": row})
|
||||||
|
return clean_rows
|
||||||
|
|
||||||
|
|
||||||
|
def load_targets(context):
|
||||||
|
date_text = _today_text(context)
|
||||||
|
try:
|
||||||
|
rows = _load_target_rows(context)
|
||||||
|
except Exception as exc:
|
||||||
|
log.error("Failed to load JoinQuant target file for %s: %s" % (date_text, exc))
|
||||||
|
rows = []
|
||||||
|
g.targets_by_date[date_text] = rows
|
||||||
|
log.info("JOINQUANT_TARGET_LOAD|%s" % json.dumps({
|
||||||
|
"date": date_text,
|
||||||
|
"portfolio_name": PORTFOLIO_NAME,
|
||||||
|
"target_mode": TARGET_MODE,
|
||||||
|
"n_targets": len(rows),
|
||||||
|
}, sort_keys=True))
|
||||||
|
|
||||||
|
|
||||||
|
def rebalance_at_open(context):
|
||||||
|
date_text = _today_text(context)
|
||||||
|
rows = g.targets_by_date.get(date_text, [])
|
||||||
|
target_symbols = set()
|
||||||
|
for row in rows:
|
||||||
|
security = row["jq_symbol"]
|
||||||
|
target_symbols.add(security)
|
||||||
|
if TARGET_MODE == "target_shares":
|
||||||
|
order_target(security, int(row["target"]))
|
||||||
|
else:
|
||||||
|
order_target_value(security, float(row["target"]))
|
||||||
|
log.info("JOINQUANT_ORDER_SUBMIT|%s" % json.dumps({
|
||||||
|
"date": date_text,
|
||||||
|
"portfolio_name": PORTFOLIO_NAME,
|
||||||
|
"jq_symbol": security,
|
||||||
|
"target_mode": TARGET_MODE,
|
||||||
|
"target": row["target"],
|
||||||
|
}, sort_keys=True))
|
||||||
|
|
||||||
|
for security in list(context.portfolio.positions.keys()):
|
||||||
|
if security not in target_symbols:
|
||||||
|
order_target(security, 0)
|
||||||
|
log.info("JOINQUANT_ORDER_CLOSE|%s" % json.dumps({
|
||||||
|
"date": date_text,
|
||||||
|
"portfolio_name": PORTFOLIO_NAME,
|
||||||
|
"jq_symbol": security,
|
||||||
|
}, sort_keys=True))
|
||||||
|
|
||||||
|
|
||||||
|
def _position_records(context):
|
||||||
|
records = []
|
||||||
|
cash = float(context.portfolio.available_cash)
|
||||||
|
total_value = float(context.portfolio.total_value)
|
||||||
|
for security, position in context.portfolio.positions.items():
|
||||||
|
records.append({
|
||||||
|
"date": _today_text(context),
|
||||||
|
"portfolio_name": PORTFOLIO_NAME,
|
||||||
|
"jq_symbol": security,
|
||||||
|
"position_shares": int(position.total_amount),
|
||||||
|
"position_value": float(position.value),
|
||||||
|
"cash": cash,
|
||||||
|
"total_value": total_value,
|
||||||
|
})
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def _trade_records(context):
|
||||||
|
records = []
|
||||||
|
try:
|
||||||
|
trades = get_trades()
|
||||||
|
except Exception:
|
||||||
|
trades = {}
|
||||||
|
for trade_id, trade in trades.items():
|
||||||
|
amount = int(getattr(trade, "amount", 0))
|
||||||
|
price = float(getattr(trade, "price", 0.0))
|
||||||
|
security = getattr(trade, "security", "")
|
||||||
|
side = "buy" if amount >= 0 else "sell"
|
||||||
|
records.append({
|
||||||
|
"date": _today_text(context),
|
||||||
|
"portfolio_name": PORTFOLIO_NAME,
|
||||||
|
"jq_symbol": security,
|
||||||
|
"order_id": str(getattr(trade, "order_id", trade_id)),
|
||||||
|
"side": side,
|
||||||
|
"filled_shares": amount,
|
||||||
|
"fill_price": price,
|
||||||
|
"trade_value": abs(amount * price),
|
||||||
|
"trade_cost": float(getattr(trade, "commission", 0.0)),
|
||||||
|
"raw_status": "filled",
|
||||||
|
})
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def record_after_close(context):
|
||||||
|
date_text = _today_text(context)
|
||||||
|
for record in _trade_records(context):
|
||||||
|
log.info("JOINQUANT_FILL|%s" % json.dumps(record, sort_keys=True))
|
||||||
|
for record in _position_records(context):
|
||||||
|
log.info("JOINQUANT_POSITION|%s" % json.dumps(record, sort_keys=True))
|
||||||
|
log.info("JOINQUANT_PNL|%s" % json.dumps({
|
||||||
|
"date": date_text,
|
||||||
|
"portfolio_name": PORTFOLIO_NAME,
|
||||||
|
"cash": float(context.portfolio.available_cash),
|
||||||
|
"total_value": float(context.portfolio.total_value),
|
||||||
|
}, sort_keys=True))
|
||||||
|
'''
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Make this module itself usable as a JoinQuant strategy with defaults.
|
||||||
|
exec(_WRAPPER_TEMPLATE.substitute(
|
||||||
|
portfolio_name="run1",
|
||||||
|
mode="target_shares",
|
||||||
|
allow_short="False",
|
||||||
|
embedded_targets="{}",
|
||||||
|
embedded_target_read="",
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def render_wrapper_strategy(
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
mode: WrapperMode = "target_shares",
|
||||||
|
allow_short: bool = False,
|
||||||
|
embedded_targets: dict[str, str] | None = None,
|
||||||
|
) -> str:
|
||||||
|
"""Render the standalone JoinQuant wrapper strategy source."""
|
||||||
|
if mode not in {"target_shares", "target_value"}:
|
||||||
|
raise ValueError("mode must be 'target_shares' or 'target_value'")
|
||||||
|
embedded_targets = embedded_targets or {}
|
||||||
|
embedded_target_read = ""
|
||||||
|
if embedded_targets:
|
||||||
|
embedded_target_read = (
|
||||||
|
" if file_name in _EMBEDDED_TARGETS:\n"
|
||||||
|
" return _EMBEDDED_TARGETS[file_name]\n"
|
||||||
|
)
|
||||||
|
return _WRAPPER_TEMPLATE.substitute(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
mode=mode,
|
||||||
|
allow_short="True" if allow_short else "False",
|
||||||
|
embedded_targets=json.dumps(embedded_targets, ensure_ascii=False, indent=4),
|
||||||
|
embedded_target_read=embedded_target_read,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_embedded_targets(targets_dir: str | Path | None) -> dict[str, str]:
|
||||||
|
if targets_dir is None:
|
||||||
|
return {}
|
||||||
|
root = Path(targets_dir)
|
||||||
|
targets = {
|
||||||
|
path.name: path.read_text(encoding="utf-8")
|
||||||
|
for path in sorted(root.glob("*.csv"))
|
||||||
|
}
|
||||||
|
if not targets:
|
||||||
|
raise ValueError(f"No CSV target files found under {root}")
|
||||||
|
return targets
|
||||||
|
|
||||||
|
|
||||||
|
def write_wrapper_strategy(
|
||||||
|
*,
|
||||||
|
portfolio_name: str,
|
||||||
|
mode: WrapperMode = "target_shares",
|
||||||
|
out_path: str | Path,
|
||||||
|
allow_short: bool = False,
|
||||||
|
embedded_targets_dir: str | Path | None = None,
|
||||||
|
) -> Path:
|
||||||
|
"""Write a configured standalone JoinQuant wrapper strategy."""
|
||||||
|
path = Path(out_path)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(
|
||||||
|
render_wrapper_strategy(
|
||||||
|
portfolio_name=portfolio_name,
|
||||||
|
mode=mode,
|
||||||
|
allow_short=allow_short,
|
||||||
|
embedded_targets=_load_embedded_targets(embedded_targets_dir),
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return path
|
||||||
+32
-1
@@ -4,7 +4,6 @@ version = "0.1.0"
|
|||||||
description = "A modular Chinese A-share quant research framework (daily frequency)."
|
description = "A modular Chinese A-share quant research framework (daily frequency)."
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"backtrader>=1.9.76.123",
|
|
||||||
"akshare>=1.14.0",
|
"akshare>=1.14.0",
|
||||||
"baostock>=0.8.8",
|
"baostock>=0.8.8",
|
||||||
"pandas>=2.0.0",
|
"pandas>=2.0.0",
|
||||||
@@ -13,10 +12,42 @@ dependencies = [
|
|||||||
"pyarrow>=14.0.0",
|
"pyarrow>=14.0.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
backtrader = [
|
||||||
|
"backtrader>=1.9.76.123",
|
||||||
|
]
|
||||||
|
joinquant-browser = [
|
||||||
|
"playwright>=1.61.0",
|
||||||
|
]
|
||||||
|
|
||||||
[dependency-groups]
|
[dependency-groups]
|
||||||
dev = [
|
dev = [
|
||||||
|
"coverage>=7.14.1",
|
||||||
"pytest>=7.0.0",
|
"pytest>=7.0.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.uv]
|
[tool.uv]
|
||||||
package = false
|
package = false
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
markers = [
|
||||||
|
"network: tests that call live external data providers and are skipped unless explicitly enabled",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.coverage.run]
|
||||||
|
branch = true
|
||||||
|
relative_files = true
|
||||||
|
source = [
|
||||||
|
".",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.coverage.report]
|
||||||
|
fail_under = 95
|
||||||
|
show_missing = true
|
||||||
|
skip_covered = false
|
||||||
|
omit = [
|
||||||
|
"tests/*",
|
||||||
|
"docs/*",
|
||||||
|
"scripts/*",
|
||||||
|
".venv/*",
|
||||||
|
]
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,40 @@
|
|||||||
|
"""Run pytest under coverage in this environment.
|
||||||
|
|
||||||
|
The plain ``coverage run -m pytest`` path reloads NumPy under the current VS Code
|
||||||
|
Python startup environment, which breaks pandas/numpy reductions. Import NumPy
|
||||||
|
before starting coverage so the measured test run uses one stable NumPy module.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import numpy # noqa: F401
|
||||||
|
import coverage
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str]) -> int:
|
||||||
|
pytest_args = argv or ["tests/", "-v"]
|
||||||
|
|
||||||
|
cov = coverage.Coverage(config_file=True)
|
||||||
|
cov.erase()
|
||||||
|
cov.start()
|
||||||
|
test_status = pytest.main(pytest_args)
|
||||||
|
cov.stop()
|
||||||
|
cov.save()
|
||||||
|
|
||||||
|
if test_status != 0:
|
||||||
|
return int(test_status)
|
||||||
|
|
||||||
|
total = cov.report()
|
||||||
|
print(f"\nCoverage total: {total:.2f}%")
|
||||||
|
fail_under = float(cov.config.fail_under)
|
||||||
|
if total < fail_under:
|
||||||
|
print(f"Coverage failure: total {total:.2f}% is below {fail_under:.2f}%")
|
||||||
|
return 2
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main(sys.argv[1:]))
|
||||||
Executable
+62
@@ -0,0 +1,62 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# End-to-end run of the outlier-robust reversal_rank alpha on the full
|
||||||
|
# all-universe dataset and on a per-date liquid subset. Records per-phase
|
||||||
|
# wall-clock time to reports/reversal_rank_timings.json.
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
DATA=data/daily_bars/all
|
||||||
|
BOOK=10000000
|
||||||
|
TIMINGS=reports/reversal_rank_timings.json
|
||||||
|
mkdir -p reports
|
||||||
|
echo "{" > "$TIMINGS"
|
||||||
|
|
||||||
|
run() { # run <json_key> <cmd...>
|
||||||
|
local key="$1"; shift
|
||||||
|
local t0 t1
|
||||||
|
t0=$(date +%s.%N)
|
||||||
|
"$@"
|
||||||
|
t1=$(date +%s.%N)
|
||||||
|
printf ' "%s": %.2f,\n' "$key" "$(echo "$t1 - $t0" | bc)" >> "$TIMINGS"
|
||||||
|
echo ">>> $key took $(echo "$t1 - $t0" | bc)s"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---- full all-universe, robust rank weighting ----
|
||||||
|
run full_alpha_compute uv run python cli.py alpha compute --data-path "$DATA" \
|
||||||
|
--alpha-name reversal_rank_all --alpha-type reversal_rank --lookback 5 --output-dir alphas
|
||||||
|
run full_alpha_eval uv run python cli.py alpha eval \
|
||||||
|
--alpha-path alphas/reversal_rank_all.pq --data-path "$DATA"
|
||||||
|
run full_combo uv run python cli.py combo combine \
|
||||||
|
--alpha-paths alphas/reversal_rank_all.pq --combo-name reversal_rank_all_combo \
|
||||||
|
--method equal_weight --output-dir combos
|
||||||
|
run full_portfolio_build uv run python cli.py portfolio build \
|
||||||
|
--weights-path combos/reversal_rank_all_combo.pq --data-path "$DATA" \
|
||||||
|
--booksize "$BOOK" --portfolio-name reversal_rank_all_10m --output-dir portfolio
|
||||||
|
run full_portfolio_eval uv run python cli.py portfolio eval \
|
||||||
|
--positions-path portfolio/reversal_rank_all_10m.pq --data-path "$DATA"
|
||||||
|
run full_portfolio_simulate uv run python cli.py portfolio simulate \
|
||||||
|
--positions-path portfolio/reversal_rank_all_10m.pq --data-path "$DATA" \
|
||||||
|
--constraint suspension --constraint price_limit --constraint volume_cap \
|
||||||
|
--cost-bps 5 --slippage-bps 5 --output-dir portfolio
|
||||||
|
|
||||||
|
# ---- liquid subset (per-date investable universe), robust rank weighting ----
|
||||||
|
run liq_alpha_compute uv run python cli.py alpha compute --data-path "$DATA" \
|
||||||
|
--alpha-name reversal_rank_liq --alpha-type reversal_rank --lookback 5 \
|
||||||
|
--liquid-universe --universe-top-n 1000 --output-dir alphas
|
||||||
|
run liq_alpha_eval uv run python cli.py alpha eval \
|
||||||
|
--alpha-path alphas/reversal_rank_liq.pq --data-path "$DATA"
|
||||||
|
run liq_combo uv run python cli.py combo combine \
|
||||||
|
--alpha-paths alphas/reversal_rank_liq.pq --combo-name reversal_rank_liq_combo \
|
||||||
|
--method equal_weight --output-dir combos
|
||||||
|
run liq_portfolio_build uv run python cli.py portfolio build \
|
||||||
|
--weights-path combos/reversal_rank_liq_combo.pq --data-path "$DATA" \
|
||||||
|
--booksize "$BOOK" --portfolio-name reversal_rank_liq_10m --output-dir portfolio
|
||||||
|
run liq_portfolio_eval uv run python cli.py portfolio eval \
|
||||||
|
--positions-path portfolio/reversal_rank_liq_10m.pq --data-path "$DATA"
|
||||||
|
run liq_portfolio_simulate uv run python cli.py portfolio simulate \
|
||||||
|
--positions-path portfolio/reversal_rank_liq_10m.pq --data-path "$DATA" \
|
||||||
|
--constraint suspension --constraint price_limit --constraint volume_cap \
|
||||||
|
--cost-bps 5 --slippage-bps 5 --output-dir portfolio
|
||||||
|
|
||||||
|
printf ' "_done": true\n}\n' >> "$TIMINGS"
|
||||||
|
echo "Wrote $TIMINGS"
|
||||||
Binary file not shown.
@@ -0,0 +1,261 @@
|
|||||||
|
"""Shared deterministic test data for offline workflow tests."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.common.schema import (
|
||||||
|
ALPHA_COLUMNS,
|
||||||
|
COMBO_COLUMNS,
|
||||||
|
DATA_COLUMNS,
|
||||||
|
MINUTE_BAR_COLUMNS,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
GENERATED_SYMBOLS: tuple[str, ...] = (
|
||||||
|
"sh600000",
|
||||||
|
"sz000001",
|
||||||
|
"sh600519",
|
||||||
|
"sz300750",
|
||||||
|
)
|
||||||
|
|
||||||
|
GENERATED_SYMBOL_NAMES: dict[str, str] = {
|
||||||
|
"sh600000": "PF Bank",
|
||||||
|
"sz000001": "Ping An Bank",
|
||||||
|
"sh600519": "Kweichow Moutai",
|
||||||
|
"sz300750": "CATL",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def generated_sessions(n_sessions: int = 12) -> pd.DatetimeIndex:
|
||||||
|
"""Return a fixed business-day calendar used by generated fixtures."""
|
||||||
|
return pd.bdate_range("2024-01-02", periods=n_sessions)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_daily_bars(
|
||||||
|
n_sessions: int = 12,
|
||||||
|
include_missing: bool = True,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Build daily bars with explicit edge cases and no randomness.
|
||||||
|
|
||||||
|
The panel covers four A-share symbols and includes a suspended row, an ST
|
||||||
|
flag, a zero-volume row, a missing symbol-date, and limit-style open/close
|
||||||
|
moves. Values are deterministic so tests can assert exact identities.
|
||||||
|
"""
|
||||||
|
dates = generated_sessions(n_sessions)
|
||||||
|
base_close = {
|
||||||
|
"sh600000": 10.00,
|
||||||
|
"sz000001": 15.00,
|
||||||
|
"sh600519": 1200.00,
|
||||||
|
"sz300750": 180.00,
|
||||||
|
}
|
||||||
|
returns = {
|
||||||
|
"sh600000": [0.000, 0.012, -0.006, 0.018, 0.100, -0.014, 0.006, 0.000, 0.008, -0.011, 0.004, 0.009],
|
||||||
|
"sz000001": [0.000, -0.008, 0.011, -0.004, 0.006, 0.000, -0.012, 0.009, 0.005, -0.007, 0.010, -0.003],
|
||||||
|
"sh600519": [0.000, 0.006, 0.004, -0.010, 0.012, -0.006, 0.005, 0.003, -0.009, 0.007, -0.004, 0.006],
|
||||||
|
"sz300750": [0.000, -0.010, 0.014, 0.006, -0.008, 0.011, -0.004, 0.009, -0.200, 0.012, -0.006, 0.008],
|
||||||
|
}
|
||||||
|
base_volume = {
|
||||||
|
"sh600000": 1_200_000.0,
|
||||||
|
"sz000001": 900_000.0,
|
||||||
|
"sh600519": 80_000.0,
|
||||||
|
"sz300750": 240_000.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
for sym in GENERATED_SYMBOLS:
|
||||||
|
closes = [base_close[sym]]
|
||||||
|
pattern = returns[sym]
|
||||||
|
for step in range(1, n_sessions):
|
||||||
|
ret = pattern[step % len(pattern)]
|
||||||
|
closes.append(closes[-1] * (1.0 + ret))
|
||||||
|
closes_arr = np.asarray(closes, dtype=float)
|
||||||
|
precloses = np.concatenate([[closes_arr[0]], closes_arr[:-1]])
|
||||||
|
|
||||||
|
for i, date in enumerate(dates):
|
||||||
|
preclose = float(precloses[i])
|
||||||
|
close = float(closes_arr[i])
|
||||||
|
open_price = preclose * (1.0 + 0.25 * (close / preclose - 1.0))
|
||||||
|
high = max(open_price, close) * 1.01
|
||||||
|
low = min(open_price, close) * 0.99
|
||||||
|
volume = base_volume[sym] + 10_000.0 * i
|
||||||
|
tradestatus = 1
|
||||||
|
is_st = 0
|
||||||
|
|
||||||
|
if sym == "sh600000" and i == 4:
|
||||||
|
open_price = preclose * 1.10
|
||||||
|
close = open_price
|
||||||
|
high = open_price
|
||||||
|
low = open_price
|
||||||
|
if sym == "sh600000" and i == 7:
|
||||||
|
volume = 0.0
|
||||||
|
if sym == "sz000001" and i == 5:
|
||||||
|
open_price = preclose
|
||||||
|
close = preclose
|
||||||
|
high = preclose
|
||||||
|
low = preclose
|
||||||
|
volume = 0.0
|
||||||
|
tradestatus = 0
|
||||||
|
if sym == "sh600519" and i == 7:
|
||||||
|
is_st = 1
|
||||||
|
if sym == "sz300750" and i == 8:
|
||||||
|
open_price = preclose * 0.80
|
||||||
|
close = open_price
|
||||||
|
high = open_price
|
||||||
|
low = open_price
|
||||||
|
|
||||||
|
amount = volume * ((open_price + close) / 2.0)
|
||||||
|
vwap = amount / volume if volume > 0 else np.nan
|
||||||
|
pct_chg = (close / preclose - 1.0) * 100.0 if preclose else 0.0
|
||||||
|
rows.append({
|
||||||
|
"symbol_id": sym,
|
||||||
|
"symbol_name": GENERATED_SYMBOL_NAMES[sym],
|
||||||
|
"date": date,
|
||||||
|
"open": open_price,
|
||||||
|
"high": high,
|
||||||
|
"low": low,
|
||||||
|
"close": close,
|
||||||
|
"preclose": preclose,
|
||||||
|
"volume": volume,
|
||||||
|
"amount": amount,
|
||||||
|
"vwap": vwap,
|
||||||
|
"turn": volume / 1_000_000.0,
|
||||||
|
"pctChg": pct_chg,
|
||||||
|
"tradestatus": tradestatus,
|
||||||
|
"isST": is_st,
|
||||||
|
"peTTM": 8.0 + i,
|
||||||
|
"pbMRQ": 1.0 + 0.05 * i,
|
||||||
|
"psTTM": 2.0 + 0.03 * i,
|
||||||
|
"pcfNcfTTM": 5.0 + 0.1 * i,
|
||||||
|
})
|
||||||
|
|
||||||
|
result = pd.DataFrame(rows)
|
||||||
|
if include_missing and n_sessions > 6:
|
||||||
|
missing_mask = (
|
||||||
|
(result["symbol_id"] == "sz300750")
|
||||||
|
& (result["date"] == dates[6])
|
||||||
|
)
|
||||||
|
result = result.loc[~missing_mask].copy()
|
||||||
|
|
||||||
|
result = result[DATA_COLUMNS]
|
||||||
|
return result.sort_values(["date", "symbol_id"]).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_minute_bars(
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Expand generated daily bars into a tiny deterministic intraday panel."""
|
||||||
|
daily = make_generated_daily_bars() if daily is None else daily.copy()
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
bar_times = ("09:35:00", "10:30:00", "14:55:00")
|
||||||
|
|
||||||
|
for daily_row in daily.sort_values(["date", "symbol_id"]).itertuples(index=False):
|
||||||
|
if int(getattr(daily_row, "tradestatus", 1)) == 0:
|
||||||
|
continue
|
||||||
|
|
||||||
|
volume = float(daily_row.volume)
|
||||||
|
volume_slices = [0.25 * volume, 0.35 * volume, 0.40 * volume]
|
||||||
|
prices = np.linspace(float(daily_row.open), float(daily_row.close), len(bar_times))
|
||||||
|
for j, time_text in enumerate(bar_times):
|
||||||
|
dt = pd.Timestamp(daily_row.date) + pd.Timedelta(time_text)
|
||||||
|
open_price = prices[j - 1] if j else float(daily_row.open)
|
||||||
|
close_price = float(prices[j])
|
||||||
|
high = max(open_price, close_price) * 1.002
|
||||||
|
low = min(open_price, close_price) * 0.998
|
||||||
|
minute_volume = float(volume_slices[j])
|
||||||
|
amount = minute_volume * ((open_price + close_price) / 2.0)
|
||||||
|
rows.append({
|
||||||
|
"symbol_id": daily_row.symbol_id,
|
||||||
|
"symbol_name": daily_row.symbol_name,
|
||||||
|
"datetime": dt,
|
||||||
|
"date": pd.Timestamp(daily_row.date).normalize(),
|
||||||
|
"time": time_text,
|
||||||
|
"frequency": "5m",
|
||||||
|
"open": open_price,
|
||||||
|
"high": high,
|
||||||
|
"low": low,
|
||||||
|
"close": close_price,
|
||||||
|
"volume": minute_volume,
|
||||||
|
"amount": amount,
|
||||||
|
"vwap": amount / minute_volume if minute_volume > 0 else np.nan,
|
||||||
|
"adjustflag": "3",
|
||||||
|
})
|
||||||
|
|
||||||
|
return pd.DataFrame(rows, columns=MINUTE_BAR_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_derived_features(
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Return numeric daily derived values, including NaN and infinity cells."""
|
||||||
|
daily = make_generated_daily_bars() if daily is None else daily.copy()
|
||||||
|
keys = (
|
||||||
|
daily[["symbol_id", "date"]]
|
||||||
|
.drop_duplicates()
|
||||||
|
.sort_values(["date", "symbol_id"])
|
||||||
|
.reset_index(drop=True)
|
||||||
|
)
|
||||||
|
date_rank = keys["date"].rank(method="dense").astype(float)
|
||||||
|
symbol_rank = keys["symbol_id"].map({
|
||||||
|
"sh600000": 1.0,
|
||||||
|
"sz000001": 2.0,
|
||||||
|
"sh600519": 3.0,
|
||||||
|
"sz300750": 4.0,
|
||||||
|
})
|
||||||
|
out = keys.copy()
|
||||||
|
out["toy_feature"] = symbol_rank + date_rank / 100.0
|
||||||
|
out["finite_feature"] = symbol_rank * date_rank
|
||||||
|
out["nan_feature"] = out["toy_feature"]
|
||||||
|
out["inf_feature"] = out["toy_feature"]
|
||||||
|
if len(out) >= 2:
|
||||||
|
out.loc[out.index[0], "nan_feature"] = np.nan
|
||||||
|
out.loc[out.index[1], "inf_feature"] = np.inf
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_alpha_weights(
|
||||||
|
alpha_name: str = "alpha_a",
|
||||||
|
*,
|
||||||
|
scale: float = 1.0,
|
||||||
|
offset: float = 0.0,
|
||||||
|
zero_date_index: int | None = None,
|
||||||
|
n_sessions: int = 10,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Create a deterministic long alpha grid with optional zero-gross date."""
|
||||||
|
dates = generated_sessions(n_sessions)
|
||||||
|
even = np.array([1.20, -0.80, 0.40, -0.80], dtype=float)
|
||||||
|
odd = np.array([-0.60, 1.10, -0.90, 0.40], dtype=float)
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
for i, date in enumerate(dates):
|
||||||
|
vector = even.copy() if i % 2 == 0 else odd.copy()
|
||||||
|
vector = vector + offset
|
||||||
|
vector = vector - vector.mean()
|
||||||
|
if zero_date_index is not None and i == zero_date_index:
|
||||||
|
vector = np.zeros_like(vector)
|
||||||
|
for sym, weight in zip(GENERATED_SYMBOLS, scale * vector):
|
||||||
|
rows.append({
|
||||||
|
"symbol_id": sym,
|
||||||
|
"date": date,
|
||||||
|
"alpha_name": alpha_name,
|
||||||
|
"weight": float(weight),
|
||||||
|
})
|
||||||
|
result = pd.DataFrame(rows, columns=ALPHA_COLUMNS)
|
||||||
|
return result.sort_values(["symbol_id", "date"]).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_combo_weights(
|
||||||
|
combo_name: str = "combo",
|
||||||
|
*,
|
||||||
|
zero_date_index: int | None = 2,
|
||||||
|
n_sessions: int = 10,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Create deterministic combo weights for portfolio construction tests."""
|
||||||
|
alpha = make_generated_alpha_weights(
|
||||||
|
"combo_source",
|
||||||
|
zero_date_index=zero_date_index,
|
||||||
|
n_sessions=n_sessions,
|
||||||
|
)
|
||||||
|
combo = alpha.rename(columns={"alpha_name": "combo_name"}).copy()
|
||||||
|
combo["combo_name"] = combo_name
|
||||||
|
return combo[COMBO_COLUMNS].sort_values(["symbol_id", "date"]).reset_index(drop=True)
|
||||||
+407
-4
@@ -6,7 +6,11 @@ import pandas as pd
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from pipeline.alpha.base import BaseAlpha
|
from pipeline.alpha.base import BaseAlpha
|
||||||
from pipeline.alpha.compute import compute_alpha, evaluate_alpha
|
from pipeline.alpha.compute import (
|
||||||
|
compute_alpha,
|
||||||
|
evaluate_alpha,
|
||||||
|
investable_universe_mask,
|
||||||
|
)
|
||||||
from pipeline.alpha.registry import (
|
from pipeline.alpha.registry import (
|
||||||
available_alphas,
|
available_alphas,
|
||||||
get_alpha,
|
get_alpha,
|
||||||
@@ -51,7 +55,7 @@ def test_reversal_sign_matches_negative_trailing_return():
|
|||||||
data = _make_data()
|
data = _make_data()
|
||||||
alpha = compute_alpha(data, "rev5", "reversal", lookback=5)
|
alpha = compute_alpha(data, "rev5", "reversal", lookback=5)
|
||||||
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
|
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
|
||||||
raw = -close.pct_change(5)
|
raw = -close.pct_change(5, fill_method=None)
|
||||||
last = raw.index[-1]
|
last = raw.index[-1]
|
||||||
expected_top = raw.loc[last].idxmax()
|
expected_top = raw.loc[last].idxmax()
|
||||||
got = alpha[alpha["date"] == last].set_index("symbol_id")["weight"].idxmax()
|
got = alpha[alpha["date"] == last].set_index("symbol_id")["weight"].idxmax()
|
||||||
@@ -74,6 +78,83 @@ def test_evaluate_alpha_keys():
|
|||||||
assert key in metrics
|
assert key in metrics
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_alpha_uses_next_open_to_next_open_returns():
|
||||||
|
dates = pd.date_range("2024-01-01", periods=5)
|
||||||
|
data = pd.concat([
|
||||||
|
pd.DataFrame({
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "sh600000",
|
||||||
|
"date": dates,
|
||||||
|
"open": [100.0, 100.0, 100.0, 100.0, 200.0],
|
||||||
|
"high": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
|
||||||
|
"low": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
|
||||||
|
"close": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
|
||||||
|
"volume": 1_000.0,
|
||||||
|
"amount": 1_000.0,
|
||||||
|
}),
|
||||||
|
pd.DataFrame({
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "sz000001",
|
||||||
|
"date": dates,
|
||||||
|
"open": [100.0, 100.0, 100.0, 200.0, 200.0],
|
||||||
|
"high": [100.0, 10.0, 10.0, 10.0, 10.0],
|
||||||
|
"low": [100.0, 10.0, 10.0, 10.0, 10.0],
|
||||||
|
"close": [100.0, 10.0, 10.0, 10.0, 10.0],
|
||||||
|
"volume": 1_000.0,
|
||||||
|
"amount": 1_000.0,
|
||||||
|
}),
|
||||||
|
], ignore_index=True)
|
||||||
|
alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[1], dates[1], dates[2], dates[2]],
|
||||||
|
"alpha_name": ["toy"] * 4,
|
||||||
|
"weight": [-1.0, 1.0, 1.0, -1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
metrics = evaluate_alpha(alpha, data)
|
||||||
|
|
||||||
|
assert metrics["n_dates"] == 2
|
||||||
|
assert np.isclose(metrics["cumulative_return"], 1.25)
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_alpha_excludes_signal_without_forward_return():
|
||||||
|
dates = pd.date_range("2024-01-01", periods=3)
|
||||||
|
data = pd.concat([
|
||||||
|
pd.DataFrame({
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "sh600000",
|
||||||
|
"date": dates,
|
||||||
|
"open": [100.0, 100.0, 200.0],
|
||||||
|
"high": [100.0, 100.0, 200.0],
|
||||||
|
"low": [100.0, 100.0, 200.0],
|
||||||
|
"close": [100.0, 100.0, 200.0],
|
||||||
|
"volume": 1_000.0,
|
||||||
|
"amount": 1_000.0,
|
||||||
|
}),
|
||||||
|
pd.DataFrame({
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "sz000001",
|
||||||
|
"date": dates,
|
||||||
|
"open": [100.0, 100.0, 100.0],
|
||||||
|
"high": [100.0, 100.0, 100.0],
|
||||||
|
"low": [100.0, 100.0, 100.0],
|
||||||
|
"close": [100.0, 100.0, 100.0],
|
||||||
|
"volume": 1_000.0,
|
||||||
|
"amount": 1_000.0,
|
||||||
|
}),
|
||||||
|
], ignore_index=True)
|
||||||
|
alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1], dates[1]],
|
||||||
|
"alpha_name": ["toy"] * 4,
|
||||||
|
"weight": [1.0, -1.0, -1.0, 1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
metrics = evaluate_alpha(alpha, data)
|
||||||
|
|
||||||
|
assert metrics["n_dates"] == 1
|
||||||
|
|
||||||
|
|
||||||
def test_equal_weight_is_mean_of_alphas():
|
def test_equal_weight_is_mean_of_alphas():
|
||||||
data = _make_data()
|
data = _make_data()
|
||||||
a = compute_alpha(data, "rev", "reversal", lookback=5)
|
a = compute_alpha(data, "rev", "reversal", lookback=5)
|
||||||
@@ -95,10 +176,34 @@ def test_combine_alphas_schema(tmp_path):
|
|||||||
assert (combo["combo_name"] == "eq").all()
|
assert (combo["combo_name"] == "eq").all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_combine_single_alpha_is_identity(tmp_path):
|
||||||
|
data = _make_data()
|
||||||
|
a = compute_alpha(data, "rev", "reversal", lookback=5)
|
||||||
|
a_path = tmp_path / "a.pq"
|
||||||
|
a.to_parquet(a_path, index=False)
|
||||||
|
|
||||||
|
combo = combine_alphas([str(a_path)], "rev_combo", method="equal_weight")
|
||||||
|
|
||||||
|
expected = a[["symbol_id", "date", "weight"]].reset_index(drop=True)
|
||||||
|
got = combo[["symbol_id", "date", "weight"]].reset_index(drop=True)
|
||||||
|
pd.testing.assert_frame_equal(got, expected)
|
||||||
|
assert list(combo.columns) == COMBO_COLUMNS
|
||||||
|
assert (combo["combo_name"] == "rev_combo").all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_combine_alphas_rejects_unknown_method(tmp_path):
|
||||||
|
data = _make_data()
|
||||||
|
alpha_path = tmp_path / "alpha.pq"
|
||||||
|
compute_alpha(data, "rev", "reversal", lookback=5).to_parquet(alpha_path, index=False)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="Unknown combo method"):
|
||||||
|
combine_alphas([str(alpha_path)], "bad_combo", method="does_not_exist")
|
||||||
|
|
||||||
|
|
||||||
# --- registry / factory -----------------------------------------------------
|
# --- registry / factory -----------------------------------------------------
|
||||||
|
|
||||||
def test_builtins_are_registered():
|
def test_builtins_are_registered():
|
||||||
assert {"reversal", "reversal_vol", "momentum"} <= set(available_alphas())
|
assert {"reversal", "reversal_vol", "momentum", "reversal_rank"} <= set(available_alphas())
|
||||||
|
|
||||||
|
|
||||||
def test_get_alpha_filters_unaccepted_params():
|
def test_get_alpha_filters_unaccepted_params():
|
||||||
@@ -109,6 +214,22 @@ def test_get_alpha_filters_unaccepted_params():
|
|||||||
assert not hasattr(alpha, "vol_window")
|
assert not hasattr(alpha, "vol_window")
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_alpha_forwards_kwargs_to_flexible_alpha():
|
||||||
|
@register_alpha
|
||||||
|
class _FlexibleAlpha(BaseAlpha):
|
||||||
|
name = "_flexible_alpha_kwargs"
|
||||||
|
|
||||||
|
def __init__(self, **kwargs):
|
||||||
|
self.kwargs = kwargs
|
||||||
|
|
||||||
|
def signal(self, close):
|
||||||
|
return close
|
||||||
|
|
||||||
|
alpha = get_alpha("_flexible_alpha_kwargs", decay=0.5, label="demo")
|
||||||
|
|
||||||
|
assert alpha.kwargs == {"decay": 0.5, "label": "demo"}
|
||||||
|
|
||||||
|
|
||||||
def test_get_alpha_unknown_raises():
|
def test_get_alpha_unknown_raises():
|
||||||
with pytest.raises(KeyError):
|
with pytest.raises(KeyError):
|
||||||
get_alpha("does_not_exist")
|
get_alpha("does_not_exist")
|
||||||
@@ -131,6 +252,31 @@ def test_register_rejects_non_basealpha():
|
|||||||
register_alpha(object) # type: ignore[arg-type]
|
register_alpha(object) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_register_rejects_empty_alpha_name():
|
||||||
|
with pytest.raises(ValueError, match="non-empty"):
|
||||||
|
@register_alpha
|
||||||
|
class NoNameAlpha(BaseAlpha):
|
||||||
|
def signal(self, close):
|
||||||
|
return close
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_alpha_module_error_paths(tmp_path, monkeypatch):
|
||||||
|
missing_path = tmp_path / "missing_alpha.py"
|
||||||
|
with pytest.raises(FileNotFoundError):
|
||||||
|
load_alpha_module(str(missing_path))
|
||||||
|
|
||||||
|
bad_path = tmp_path / "bad_alpha.py"
|
||||||
|
bad_path.write_text("x = 1\n")
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"pipeline.alpha.registry.importlib.util.spec_from_file_location",
|
||||||
|
lambda *args, **kwargs: None,
|
||||||
|
)
|
||||||
|
with pytest.raises(ImportError, match="Cannot load alpha module"):
|
||||||
|
load_alpha_module(str(bad_path))
|
||||||
|
|
||||||
|
load_alpha_module("math")
|
||||||
|
|
||||||
|
|
||||||
# --- base class --------------------------------------------------------------
|
# --- base class --------------------------------------------------------------
|
||||||
|
|
||||||
def test_to_weights_are_per_date_zscore():
|
def test_to_weights_are_per_date_zscore():
|
||||||
@@ -146,6 +292,20 @@ def test_to_weights_are_per_date_zscore():
|
|||||||
assert (weights.mean(axis=1).abs() < 1e-9).all()
|
assert (weights.mean(axis=1).abs() < 1e-9).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_base_alpha_default_signal_and_repr():
|
||||||
|
alpha = BaseAlpha()
|
||||||
|
alpha.example = 3
|
||||||
|
|
||||||
|
with pytest.raises(NotImplementedError, match="signal"):
|
||||||
|
alpha.signal(pd.DataFrame({"x": [1.0]}))
|
||||||
|
with pytest.raises(NotImplementedError, match="signal"):
|
||||||
|
alpha.signal_from_data(
|
||||||
|
pd.DataFrame({"symbol_id": ["sh600000"]}),
|
||||||
|
pd.DataFrame({"sh600000": [1.0]}),
|
||||||
|
)
|
||||||
|
assert repr(alpha) == "BaseAlpha(example=3)"
|
||||||
|
|
||||||
|
|
||||||
# --- external plugin loading -------------------------------------------------
|
# --- external plugin loading -------------------------------------------------
|
||||||
|
|
||||||
def test_load_external_alpha_module(tmp_path):
|
def test_load_external_alpha_module(tmp_path):
|
||||||
@@ -163,7 +323,7 @@ def test_load_external_alpha_module(tmp_path):
|
|||||||
self.span = span
|
self.span = span
|
||||||
|
|
||||||
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
return -close.pct_change(self.span)
|
return -close.pct_change(self.span, fill_method=None)
|
||||||
'''))
|
'''))
|
||||||
|
|
||||||
load_alpha_module(str(module_path))
|
load_alpha_module(str(module_path))
|
||||||
@@ -178,3 +338,246 @@ def test_load_external_alpha_module(tmp_path):
|
|||||||
assert list(result.columns) == ALPHA_COLUMNS
|
assert list(result.columns) == ALPHA_COLUMNS
|
||||||
assert (result["alpha_name"] == "ext").all()
|
assert (result["alpha_name"] == "ext").all()
|
||||||
|
|
||||||
|
|
||||||
|
# --- rank reversal + investable universe filter ------------------------------
|
||||||
|
|
||||||
|
def _make_rich_data(n_days: int = 70, symbols=("sh600000", "sz000001", "sh600519", "sz300750")):
|
||||||
|
"""Long-format data with the columns the universe filter needs."""
|
||||||
|
dates = pd.date_range("2024-01-01", periods=n_days)
|
||||||
|
rng = np.random.default_rng(1)
|
||||||
|
frames = []
|
||||||
|
for i, sym in enumerate(symbols):
|
||||||
|
close = 100.0 + i * 5 + np.cumsum(rng.standard_normal(n_days))
|
||||||
|
frames.append(pd.DataFrame({
|
||||||
|
"symbol_id": sym,
|
||||||
|
"symbol_name": sym,
|
||||||
|
"date": dates,
|
||||||
|
"open": close, "high": close, "low": close, "close": close,
|
||||||
|
"volume": 1_000.0,
|
||||||
|
"amount": (1_000.0 + i * 5_000.0) * close, # higher i = more liquid
|
||||||
|
"isST": 0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
}))
|
||||||
|
return pd.concat(frames, ignore_index=True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_reversal_rank_registered_and_bounded():
|
||||||
|
data = _make_data(n_days=30)
|
||||||
|
alpha = compute_alpha(data, "rr", "reversal_rank", lookback=5)
|
||||||
|
assert list(alpha.columns) == ALPHA_COLUMNS
|
||||||
|
# Rank-demeaned weights are per-date zero-mean and bounded by the
|
||||||
|
# cross-section size, never blowing up the way a z-score outlier can.
|
||||||
|
per_date_mean = alpha.groupby("date")["weight"].mean().abs()
|
||||||
|
assert (per_date_mean < 1e-9).all()
|
||||||
|
assert alpha["weight"].abs().max() <= len(data["symbol_id"].unique())
|
||||||
|
|
||||||
|
|
||||||
|
def test_investable_universe_mask_excludes_st_and_suspended():
|
||||||
|
data = _make_rich_data()
|
||||||
|
# Flag one name ST throughout, and suspend another on the last date.
|
||||||
|
data.loc[data["symbol_id"] == "sh600000", "isST"] = 1
|
||||||
|
last = data["date"].max()
|
||||||
|
data.loc[(data["symbol_id"] == "sz000001") & (data["date"] == last), "tradestatus"] = 0
|
||||||
|
|
||||||
|
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
|
||||||
|
mask = investable_universe_mask(data, close, top_n=10, min_history=5)
|
||||||
|
|
||||||
|
assert not mask["sh600000"].any() # ST excluded on every date
|
||||||
|
assert not bool(mask.loc[last, "sz000001"]) # suspended on the last date
|
||||||
|
assert bool(mask.loc[last, "sh600519"]) # a normal name stays investable
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_alpha_universe_filter_zeros_excluded_names():
|
||||||
|
data = _make_rich_data()
|
||||||
|
data.loc[data["symbol_id"] == "sh600000", "isST"] = 1
|
||||||
|
|
||||||
|
alpha = compute_alpha(
|
||||||
|
data, "rr_liq", "reversal_rank", lookback=5,
|
||||||
|
universe={"top_n": 10, "min_history": 5},
|
||||||
|
)
|
||||||
|
# The ST name is never held; an investable name is.
|
||||||
|
st_w = alpha.loc[alpha["symbol_id"] == "sh600000", "weight"]
|
||||||
|
assert (st_w.fillna(0.0) == 0.0).all()
|
||||||
|
assert alpha.loc[alpha["symbol_id"] == "sz300750", "weight"].abs().sum() > 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_universe_filter_does_not_corrupt_signal_history():
|
||||||
|
# Masking happens on the signal, not the price history, so weights on
|
||||||
|
# investable names match the unfiltered weights restricted to that set.
|
||||||
|
data = _make_rich_data()
|
||||||
|
universe = {"top_n": 2, "min_history": 5} # keep only the 2 most liquid names
|
||||||
|
|
||||||
|
filtered = compute_alpha(data, "f", "reversal_rank", lookback=5, universe=universe)
|
||||||
|
held = set(filtered.loc[filtered["weight"] != 0.0, "symbol_id"].unique())
|
||||||
|
# The two most liquid names (highest amount) are sh600519, sz300750.
|
||||||
|
assert held == {"sh600519", "sz300750"}
|
||||||
|
|
||||||
|
|
||||||
|
# --- feature-aware alpha integration ----------------------------------------
|
||||||
|
|
||||||
|
def test_compute_alpha_without_feature_path_matches_empty_feature_paths():
|
||||||
|
data = _make_data()
|
||||||
|
|
||||||
|
base = compute_alpha(data, "rev5", "reversal", lookback=5)
|
||||||
|
with_empty_features = compute_alpha(
|
||||||
|
data,
|
||||||
|
"rev5",
|
||||||
|
"reversal",
|
||||||
|
lookback=5,
|
||||||
|
feature_paths=[],
|
||||||
|
)
|
||||||
|
|
||||||
|
pd.testing.assert_frame_equal(base, with_empty_features)
|
||||||
|
|
||||||
|
|
||||||
|
def test_feature_aware_alpha_reads_joined_feature_column(tmp_path):
|
||||||
|
module_path = tmp_path / "feature_aware_alpha.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class FeatureAwareAlpha(BaseAlpha):
|
||||||
|
name = "feature_aware_test_alpha"
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="toy_feature",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
return signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
'''))
|
||||||
|
|
||||||
|
data = _make_data()
|
||||||
|
feature = data[["symbol_id", "date"]].copy()
|
||||||
|
feature["toy_feature"] = feature["symbol_id"].map({
|
||||||
|
"sh600000": 1.0,
|
||||||
|
"sz000001": 2.0,
|
||||||
|
"sh600519": 3.0,
|
||||||
|
})
|
||||||
|
feature_path = tmp_path / "toy_feature.pq"
|
||||||
|
feature.to_parquet(feature_path, index=False)
|
||||||
|
|
||||||
|
load_alpha_module(str(module_path))
|
||||||
|
result = compute_alpha(
|
||||||
|
data,
|
||||||
|
"feature_run",
|
||||||
|
"feature_aware_test_alpha",
|
||||||
|
feature_paths=[str(feature_path)],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(result.columns) == ALPHA_COLUMNS
|
||||||
|
assert (result["alpha_name"] == "feature_run").all()
|
||||||
|
last = result[result["date"] == result["date"].max()]
|
||||||
|
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
|
||||||
|
|
||||||
|
|
||||||
|
def test_feature_paths_join_multiple_files_and_normalize_dates(tmp_path):
|
||||||
|
module_path = tmp_path / "multi_feature_alpha.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class MultiFeatureAlpha(BaseAlpha):
|
||||||
|
name = "multi_feature_test_alpha"
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
data = data.copy()
|
||||||
|
data["combined_feature"] = data["toy_a"] + data["toy_b"]
|
||||||
|
signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="combined_feature",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
return signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
'''))
|
||||||
|
|
||||||
|
data = _make_data(n_days=8)
|
||||||
|
symbol_score = {"sh600000": 1.0, "sz000001": 2.0, "sh600519": 3.0}
|
||||||
|
|
||||||
|
feature_a = data[["symbol_id", "date"]].copy()
|
||||||
|
feature_a["date"] = feature_a["date"] + pd.Timedelta(hours=15)
|
||||||
|
feature_a["toy_a"] = feature_a["symbol_id"].map(symbol_score)
|
||||||
|
|
||||||
|
feature_b = data[["symbol_id", "date"]].copy()
|
||||||
|
feature_b["date"] = feature_b["date"].dt.strftime("%Y-%m-%d 09:30:00")
|
||||||
|
feature_b["toy_b"] = feature_b["symbol_id"].map(symbol_score) * 10.0
|
||||||
|
|
||||||
|
feature_a_path = tmp_path / "toy_a.pq"
|
||||||
|
feature_b_path = tmp_path / "toy_b.pq"
|
||||||
|
feature_a.to_parquet(feature_a_path, index=False)
|
||||||
|
feature_b.to_parquet(feature_b_path, index=False)
|
||||||
|
|
||||||
|
load_alpha_module(str(module_path))
|
||||||
|
result = compute_alpha(
|
||||||
|
data,
|
||||||
|
"multi_feature_run",
|
||||||
|
"multi_feature_test_alpha",
|
||||||
|
feature_paths=[str(feature_a_path), str(feature_b_path)],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(result.columns) == ALPHA_COLUMNS
|
||||||
|
assert (result["alpha_name"] == "multi_feature_run").all()
|
||||||
|
last = result[result["date"] == result["date"].max()]
|
||||||
|
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_alpha_rejects_duplicate_feature_frame_columns():
|
||||||
|
data = _make_data()
|
||||||
|
duplicate_columns = pd.DataFrame(
|
||||||
|
[["sh600000", pd.Timestamp("2024-01-01"), 1.0, 2.0]],
|
||||||
|
columns=["symbol_id", "date", "toy_feature", "toy_feature"],
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="duplicate columns"):
|
||||||
|
compute_alpha(
|
||||||
|
data,
|
||||||
|
"bad_features",
|
||||||
|
"reversal",
|
||||||
|
feature_frames=[duplicate_columns],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_alpha_rejects_feature_path_collision_with_daily_data(tmp_path):
|
||||||
|
data = _make_data()
|
||||||
|
close_collision = data[["symbol_id", "date"]].copy()
|
||||||
|
close_collision["close"] = 1.0
|
||||||
|
close_collision_path = tmp_path / "close_collision.pq"
|
||||||
|
close_collision.to_parquet(close_collision_path, index=False)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="conflict"):
|
||||||
|
compute_alpha(
|
||||||
|
data,
|
||||||
|
"close_collision",
|
||||||
|
"reversal",
|
||||||
|
feature_paths=[str(close_collision_path)],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_alpha_empty_when_signal_dates_not_on_market_calendar():
|
||||||
|
data = _make_data(n_days=3)
|
||||||
|
alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2030-01-01")],
|
||||||
|
"alpha_name": ["future"],
|
||||||
|
"weight": [1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
metrics = evaluate_alpha(alpha, data)
|
||||||
|
|
||||||
|
assert metrics["n_dates"] == 0
|
||||||
|
assert metrics["cumulative_return"] == 0.0
|
||||||
|
|||||||
@@ -0,0 +1,771 @@
|
|||||||
|
"""CLI handoff tests for the offline daily workflow."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import textwrap
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import click
|
||||||
|
import pandas as pd
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from cli import cli
|
||||||
|
import pipeline.derived.cli as derived_cli
|
||||||
|
import pipeline.features.cli as features_cli
|
||||||
|
from tests.helpers import (
|
||||||
|
make_generated_daily_bars,
|
||||||
|
make_generated_derived_features,
|
||||||
|
make_generated_minute_bars,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_ok(runner: CliRunner, args: list[str]):
|
||||||
|
result = runner.invoke(cli, args)
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_error(runner: CliRunner, args: list[str]):
|
||||||
|
result = runner.invoke(cli, args)
|
||||||
|
assert result.exit_code != 0, result.output
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_daily_workflow_handoffs_stay_in_tmp_path(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars()
|
||||||
|
minute_bars = make_generated_minute_bars(daily_bars)
|
||||||
|
derived_features = make_generated_derived_features(daily_bars)
|
||||||
|
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
minute_path = tmp_path / "minute_bars.pq"
|
||||||
|
derived_input_path = tmp_path / "derived_input.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
minute_bars.to_parquet(minute_path, index=False)
|
||||||
|
derived_features.to_parquet(derived_input_path, index=False)
|
||||||
|
|
||||||
|
ingest_dir = tmp_path / "derived_ingested"
|
||||||
|
ingest_result = _invoke_ok(runner, [
|
||||||
|
"derived", "ingest",
|
||||||
|
"--input-path", str(derived_input_path),
|
||||||
|
"--derived-name", "toy_features",
|
||||||
|
"--output-dir", str(ingest_dir),
|
||||||
|
])
|
||||||
|
ingested_feature_path = ingest_dir / "toy_features.pq"
|
||||||
|
assert "Saved derived data:" in ingest_result.output
|
||||||
|
assert ingested_feature_path.exists()
|
||||||
|
|
||||||
|
validate_result = _invoke_ok(runner, [
|
||||||
|
"derived", "validate",
|
||||||
|
"--input-path", str(ingested_feature_path),
|
||||||
|
])
|
||||||
|
assert "Valid derived data:" in validate_result.output
|
||||||
|
assert "rows" in validate_result.output
|
||||||
|
|
||||||
|
computed_derived_dir = tmp_path / "derived_computed"
|
||||||
|
derived_compute_result = _invoke_ok(runner, [
|
||||||
|
"derived", "compute",
|
||||||
|
"--daily-path", str(daily_path),
|
||||||
|
"--minute-path", str(minute_path),
|
||||||
|
"--derived-type", "minute_daily_summary",
|
||||||
|
"--derived-name", "minute_summary",
|
||||||
|
"--output-dir", str(computed_derived_dir),
|
||||||
|
])
|
||||||
|
minute_summary_path = computed_derived_dir / "minute_summary.pq"
|
||||||
|
assert "Loaded daily data:" in derived_compute_result.output
|
||||||
|
assert "Loaded minute bars:" in derived_compute_result.output
|
||||||
|
assert "Saved derived data:" in derived_compute_result.output
|
||||||
|
assert minute_summary_path.exists()
|
||||||
|
assert "minute_vwap" in pd.read_parquet(minute_summary_path).columns
|
||||||
|
|
||||||
|
alpha_module_path = tmp_path / "cli_feature_alpha.py"
|
||||||
|
alpha_module_path.write_text(textwrap.dedent("""
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class CliFeatureAlpha(BaseAlpha):
|
||||||
|
name = "cli_feature_alpha_workflow"
|
||||||
|
|
||||||
|
def __init__(self, **kwargs):
|
||||||
|
self.kwargs = kwargs
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="minute_intraday_return",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
fallback = close.pct_change(1, fill_method=None)
|
||||||
|
feature_signal = signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
toy_signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="toy_feature",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
toy_signal = toy_signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
return feature_signal.fillna(fallback) + toy_signal / 1000.0
|
||||||
|
"""))
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
alpha_result = _invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--feature-path", str(minute_summary_path),
|
||||||
|
"--feature-path", str(ingested_feature_path),
|
||||||
|
"--alpha-module", str(alpha_module_path),
|
||||||
|
"--alpha-type", "cli_feature_alpha_workflow",
|
||||||
|
"--alpha-name", "cli_feature_alpha",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
alpha_path = alpha_dir / "cli_feature_alpha.pq"
|
||||||
|
assert "Loaded data:" in alpha_result.output
|
||||||
|
assert "Saved alpha:" in alpha_result.output
|
||||||
|
assert "Weight stats" in alpha_result.output
|
||||||
|
assert alpha_path.exists()
|
||||||
|
assert not pd.read_parquet(alpha_path).empty
|
||||||
|
|
||||||
|
alpha_report_dir = tmp_path / "alpha_reports"
|
||||||
|
alpha_eval_result = _invoke_ok(runner, [
|
||||||
|
"alpha", "eval",
|
||||||
|
"--alpha-path", str(alpha_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--report-dir", str(alpha_report_dir),
|
||||||
|
])
|
||||||
|
alpha_report_path = alpha_report_dir / "cli_feature_alpha_eval.json"
|
||||||
|
assert "ALPHA EVALUATION" in alpha_eval_result.output
|
||||||
|
assert "Report saved:" in alpha_eval_result.output
|
||||||
|
assert alpha_report_path.exists()
|
||||||
|
|
||||||
|
combo_dir = tmp_path / "combos"
|
||||||
|
combo_result = _invoke_ok(runner, [
|
||||||
|
"combo", "combine",
|
||||||
|
"--alpha-paths", f"{alpha_path},{alpha_path}",
|
||||||
|
"--combo-name", "cli_combo",
|
||||||
|
"--method", "equal_weight",
|
||||||
|
"--output-dir", str(combo_dir),
|
||||||
|
])
|
||||||
|
combo_path = combo_dir / "cli_combo.pq"
|
||||||
|
assert "Saved combo:" in combo_result.output
|
||||||
|
assert "Weight stats" in combo_result.output
|
||||||
|
assert combo_path.exists()
|
||||||
|
|
||||||
|
portfolio_dir = tmp_path / "portfolio"
|
||||||
|
build_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "build",
|
||||||
|
"--weights-path", str(combo_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--booksize", "2000000",
|
||||||
|
"--portfolio-name", "cli_portfolio",
|
||||||
|
"--output-dir", str(portfolio_dir),
|
||||||
|
])
|
||||||
|
positions_path = portfolio_dir / "cli_portfolio.pq"
|
||||||
|
assert "Saved positions:" in build_result.output
|
||||||
|
assert "Gross exposure" in build_result.output
|
||||||
|
assert positions_path.exists()
|
||||||
|
|
||||||
|
execution_dir = tmp_path / "execution"
|
||||||
|
simulate_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "simulate",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--constraint", "suspension",
|
||||||
|
"--constraint", "price_limit",
|
||||||
|
"--constraint", "volume_cap",
|
||||||
|
"--cost-bps", "5",
|
||||||
|
"--slippage-bps", "5",
|
||||||
|
"--volume-frac", "0.02",
|
||||||
|
"--output-dir", str(execution_dir),
|
||||||
|
])
|
||||||
|
fills_path = execution_dir / "fills" / "cli_portfolio.pq"
|
||||||
|
pnl_path = execution_dir / "pnl" / "cli_portfolio.pq"
|
||||||
|
assert "Saved fills:" in simulate_result.output
|
||||||
|
assert "Saved pnl:" in simulate_result.output
|
||||||
|
assert "Total PnL:" in simulate_result.output
|
||||||
|
assert fills_path.exists()
|
||||||
|
assert pnl_path.exists()
|
||||||
|
|
||||||
|
eval_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "eval",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
])
|
||||||
|
assert "Research-portfolio metrics:" in eval_result.output
|
||||||
|
assert "cumulative_return" in eval_result.output
|
||||||
|
assert "fitness" in eval_result.output
|
||||||
|
|
||||||
|
pqcat_result = _invoke_ok(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(positions_path),
|
||||||
|
"--info",
|
||||||
|
])
|
||||||
|
assert "shape:" in pqcat_result.output
|
||||||
|
assert "dtypes:" in pqcat_result.output
|
||||||
|
assert "position_shares" in pqcat_result.output
|
||||||
|
|
||||||
|
alphaview_result = _invoke_ok(runner, [
|
||||||
|
"alphaview",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-path", str(alpha_path),
|
||||||
|
"--symbol", "sh600000",
|
||||||
|
"--start-date", "2024-01-02",
|
||||||
|
"--end-date", "2024-01-12",
|
||||||
|
"--columns", "close,volume",
|
||||||
|
])
|
||||||
|
assert "symbol: sh600000" in alphaview_result.output
|
||||||
|
assert "cli_feature_alpha" in alphaview_result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_pipeline_accepts_partitioned_daily_dataset(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(include_missing=False)
|
||||||
|
dataset_dir = tmp_path / "daily_dataset"
|
||||||
|
dataset_frame = daily_bars.copy()
|
||||||
|
dataset_frame["month"] = dataset_frame["date"].dt.strftime("%Y-%m")
|
||||||
|
dataset_frame.to_parquet(dataset_dir, partition_cols=["month"], index=False)
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
alpha_result = _invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(dataset_dir),
|
||||||
|
"--alpha-type", "reversal",
|
||||||
|
"--alpha-name", "dataset_reversal",
|
||||||
|
"--lookback", "3",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
alpha_path = alpha_dir / "dataset_reversal.pq"
|
||||||
|
assert "Loaded data:" in alpha_result.output
|
||||||
|
assert alpha_path.exists()
|
||||||
|
|
||||||
|
combo_dir = tmp_path / "combos"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"combo", "combine",
|
||||||
|
"--alpha-paths", str(alpha_path),
|
||||||
|
"--combo-name", "dataset_combo",
|
||||||
|
"--output-dir", str(combo_dir),
|
||||||
|
])
|
||||||
|
combo_path = combo_dir / "dataset_combo.pq"
|
||||||
|
assert combo_path.exists()
|
||||||
|
|
||||||
|
portfolio_dir = tmp_path / "portfolio"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"portfolio", "build",
|
||||||
|
"--weights-path", str(combo_path),
|
||||||
|
"--data-path", str(dataset_dir),
|
||||||
|
"--booksize", "1000000",
|
||||||
|
"--portfolio-name", "dataset_portfolio",
|
||||||
|
"--output-dir", str(portfolio_dir),
|
||||||
|
])
|
||||||
|
positions_path = portfolio_dir / "dataset_portfolio.pq"
|
||||||
|
assert positions_path.exists()
|
||||||
|
|
||||||
|
execution_dir = tmp_path / "execution"
|
||||||
|
simulate_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "simulate",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(dataset_dir),
|
||||||
|
"--constraint", "suspension",
|
||||||
|
"--output-dir", str(execution_dir),
|
||||||
|
])
|
||||||
|
assert "Saved fills:" in simulate_result.output
|
||||||
|
assert (execution_dir / "fills" / "dataset_portfolio.pq").exists()
|
||||||
|
assert (execution_dir / "pnl" / "dataset_portfolio.pq").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_liquid_universe_masks_to_top_liquid_names(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=75, include_missing=False)
|
||||||
|
daily_path = tmp_path / "daily_bars_75d.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
result = _invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-type", "reversal_rank",
|
||||||
|
"--alpha-name", "liquid_rank",
|
||||||
|
"--lookback", "3",
|
||||||
|
"--liquid-universe",
|
||||||
|
"--universe-top-n", "2",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
|
||||||
|
alpha_path = alpha_dir / "liquid_rank.pq"
|
||||||
|
alpha = pd.read_parquet(alpha_path)
|
||||||
|
nonzero = alpha[alpha["weight"] != 0.0]
|
||||||
|
assert "Saved alpha:" in result.output
|
||||||
|
assert alpha_path.exists()
|
||||||
|
assert not nonzero.empty
|
||||||
|
assert nonzero.groupby("date")["symbol_id"].nunique().max() <= 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_real_fixture_round_trips_through_portfolio(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(FIXTURE_PATH),
|
||||||
|
"--alpha-type", "reversal_vol",
|
||||||
|
"--alpha-name", "real_cli_reversal_vol",
|
||||||
|
"--lookback", "3",
|
||||||
|
"--vol-window", "3",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
alpha_path = alpha_dir / "real_cli_reversal_vol.pq"
|
||||||
|
assert alpha_path.exists()
|
||||||
|
assert not pd.read_parquet(alpha_path).empty
|
||||||
|
|
||||||
|
combo_dir = tmp_path / "combos"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"combo", "combine",
|
||||||
|
"--alpha-paths", str(alpha_path),
|
||||||
|
"--combo-name", "real_cli_combo",
|
||||||
|
"--output-dir", str(combo_dir),
|
||||||
|
])
|
||||||
|
combo_path = combo_dir / "real_cli_combo.pq"
|
||||||
|
assert combo_path.exists()
|
||||||
|
|
||||||
|
portfolio_dir = tmp_path / "portfolio"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"portfolio", "build",
|
||||||
|
"--weights-path", str(combo_path),
|
||||||
|
"--data-path", str(FIXTURE_PATH),
|
||||||
|
"--booksize", "1000000",
|
||||||
|
"--portfolio-name", "real_cli_portfolio",
|
||||||
|
"--output-dir", str(portfolio_dir),
|
||||||
|
])
|
||||||
|
positions_path = portfolio_dir / "real_cli_portfolio.pq"
|
||||||
|
positions = pd.read_parquet(positions_path)
|
||||||
|
assert not positions.empty
|
||||||
|
|
||||||
|
eval_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "eval",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(FIXTURE_PATH),
|
||||||
|
])
|
||||||
|
assert "Research-portfolio metrics:" in eval_result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_error_paths_are_clear_for_bad_user_inputs(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars()
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
|
||||||
|
unknown_alpha = _invoke_error(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-type", "does_not_exist",
|
||||||
|
"--alpha-name", "bad",
|
||||||
|
"--output-dir", str(tmp_path / "alphas"),
|
||||||
|
])
|
||||||
|
assert "Unknown alpha-type" in unknown_alpha.output
|
||||||
|
|
||||||
|
malformed_param = _invoke_error(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-type", "reversal",
|
||||||
|
"--alpha-name", "bad_param",
|
||||||
|
"--param", "not-an-assignment",
|
||||||
|
"--output-dir", str(tmp_path / "alphas"),
|
||||||
|
])
|
||||||
|
assert "--param must be name=value" in malformed_param.output
|
||||||
|
|
||||||
|
unknown_derived = _invoke_error(runner, [
|
||||||
|
"derived", "compute",
|
||||||
|
"--daily-path", str(daily_path),
|
||||||
|
"--derived-type", "does_not_exist",
|
||||||
|
"--derived-name", "bad",
|
||||||
|
"--output-dir", str(tmp_path / "derived"),
|
||||||
|
])
|
||||||
|
assert "Unknown derived-type" in unknown_derived.output
|
||||||
|
|
||||||
|
bad_constraint_positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"portfolio_name": ["bad_constraint"],
|
||||||
|
"target_weight": [1.0],
|
||||||
|
"target_value": [1000.0],
|
||||||
|
"target_shares": [100.0],
|
||||||
|
"position_shares": [100],
|
||||||
|
"position_value": [1000.0],
|
||||||
|
"price": [10.0],
|
||||||
|
})
|
||||||
|
positions_path = tmp_path / "positions.pq"
|
||||||
|
bad_constraint_positions.to_parquet(positions_path, index=False)
|
||||||
|
unknown_constraint = _invoke_error(runner, [
|
||||||
|
"portfolio", "simulate",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--constraint", "not_a_constraint",
|
||||||
|
"--output-dir", str(tmp_path / "execution"),
|
||||||
|
])
|
||||||
|
assert isinstance(unknown_constraint.exception, KeyError)
|
||||||
|
assert "not_a_constraint" in str(unknown_constraint.exception)
|
||||||
|
|
||||||
|
pqcat_missing_column = _invoke_error(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(daily_path),
|
||||||
|
"--columns", "close,not_a_column",
|
||||||
|
])
|
||||||
|
assert "Columns not found: not_a_column" in pqcat_missing_column.output
|
||||||
|
|
||||||
|
alphaview_missing_symbol = _invoke_error(runner, [
|
||||||
|
"alphaview",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-path", str(positions_path),
|
||||||
|
"--symbol", "sh999999",
|
||||||
|
])
|
||||||
|
assert "Symbol 'sh999999' not found" in alphaview_missing_symbol.output
|
||||||
|
|
||||||
|
alphaview_missing_column = _invoke_error(runner, [
|
||||||
|
"alphaview",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-path", str(positions_path),
|
||||||
|
"--symbol", "sh600000",
|
||||||
|
"--columns", "close,missing_bar_col",
|
||||||
|
])
|
||||||
|
assert "Bar columns not found: missing_bar_col" in alphaview_missing_column.output
|
||||||
|
|
||||||
|
alphaview_alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [daily_bars["date"].min()],
|
||||||
|
"alpha_name": ["toy_alpha"],
|
||||||
|
"weight": [1.0],
|
||||||
|
})
|
||||||
|
alphaview_alpha_path = tmp_path / "alphaview_alpha.pq"
|
||||||
|
alphaview_alpha.to_parquet(alphaview_alpha_path, index=False)
|
||||||
|
alphaview_empty_range = _invoke_error(runner, [
|
||||||
|
"alphaview",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-path", str(alphaview_alpha_path),
|
||||||
|
"--symbol", "sh600000",
|
||||||
|
"--start-date", "2030-01-01",
|
||||||
|
])
|
||||||
|
assert "No rows in the requested date range" in alphaview_empty_range.output
|
||||||
|
|
||||||
|
empty_combo_paths = _invoke_ok(runner, [
|
||||||
|
"combo", "combine",
|
||||||
|
"--alpha-paths", " , ",
|
||||||
|
"--combo-name", "empty",
|
||||||
|
"--output-dir", str(tmp_path / "combos"),
|
||||||
|
])
|
||||||
|
assert "requires at least 1 path" in empty_combo_paths.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_parser_helpers_cover_string_coercion_and_bad_params():
|
||||||
|
assert derived_cli._parse_params(("n=7", "scale=2.5", "label=demo")) == {
|
||||||
|
"n": 7,
|
||||||
|
"scale": 2.5,
|
||||||
|
"label": "demo",
|
||||||
|
}
|
||||||
|
assert features_cli._parse_params(("n=7", "scale=2.5", "label=demo")) == {
|
||||||
|
"n": 7,
|
||||||
|
"scale": 2.5,
|
||||||
|
"label": "demo",
|
||||||
|
}
|
||||||
|
|
||||||
|
for module in (derived_cli, features_cli):
|
||||||
|
try:
|
||||||
|
module._parse_params(("not-an-assignment",))
|
||||||
|
except click.BadParameter as exc:
|
||||||
|
assert "--param must be name=value" in str(exc)
|
||||||
|
else:
|
||||||
|
raise AssertionError("expected BadParameter")
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_list_and_legacy_feature_paths(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
|
||||||
|
minute_bars = make_generated_minute_bars(daily_bars)
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
minute_path = tmp_path / "minute_bars.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
minute_bars.to_parquet(minute_path, index=False)
|
||||||
|
|
||||||
|
derived_list = _invoke_ok(runner, ["derived", "list"])
|
||||||
|
feature_list = _invoke_ok(runner, ["feature", "list"])
|
||||||
|
assert "minute_daily_summary" in derived_list.output
|
||||||
|
assert "minute_daily_summary" in feature_list.output
|
||||||
|
|
||||||
|
feature_dir = tmp_path / "features"
|
||||||
|
feature_compute = _invoke_ok(runner, [
|
||||||
|
"feature",
|
||||||
|
"compute",
|
||||||
|
"--minute-path",
|
||||||
|
str(minute_path),
|
||||||
|
"--daily-path",
|
||||||
|
str(daily_path),
|
||||||
|
"--feature-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--feature-name",
|
||||||
|
"legacy_summary",
|
||||||
|
"--output-dir",
|
||||||
|
str(feature_dir),
|
||||||
|
])
|
||||||
|
feature_path = feature_dir / "legacy_summary.pq"
|
||||||
|
assert "Loaded minute bars:" in feature_compute.output
|
||||||
|
assert "Loaded daily data:" in feature_compute.output
|
||||||
|
assert "Saved feature:" in feature_compute.output
|
||||||
|
assert feature_path.exists()
|
||||||
|
|
||||||
|
no_input = _invoke_error(runner, [
|
||||||
|
"derived",
|
||||||
|
"compute",
|
||||||
|
"--derived-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--derived-name",
|
||||||
|
"missing_inputs",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "derived"),
|
||||||
|
])
|
||||||
|
assert "At least one of --daily-path or --minute-path is required" in no_input.output
|
||||||
|
|
||||||
|
missing_minute = _invoke_error(runner, [
|
||||||
|
"derived",
|
||||||
|
"compute",
|
||||||
|
"--daily-path",
|
||||||
|
str(daily_path),
|
||||||
|
"--derived-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--derived-name",
|
||||||
|
"daily_only",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "derived"),
|
||||||
|
])
|
||||||
|
assert "minute_daily_summary requires minute input" in missing_minute.output
|
||||||
|
|
||||||
|
malformed_feature_param = _invoke_error(runner, [
|
||||||
|
"feature",
|
||||||
|
"compute",
|
||||||
|
"--minute-path",
|
||||||
|
str(minute_path),
|
||||||
|
"--feature-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--feature-name",
|
||||||
|
"bad_param",
|
||||||
|
"--param",
|
||||||
|
"not-an-assignment",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "features_bad"),
|
||||||
|
])
|
||||||
|
assert "--param must be name=value" in malformed_feature_param.output
|
||||||
|
|
||||||
|
unknown_feature = _invoke_error(runner, [
|
||||||
|
"feature",
|
||||||
|
"compute",
|
||||||
|
"--minute-path",
|
||||||
|
str(minute_path),
|
||||||
|
"--feature-type",
|
||||||
|
"does_not_exist",
|
||||||
|
"--feature-name",
|
||||||
|
"bad_feature",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "features_unknown"),
|
||||||
|
])
|
||||||
|
assert "Unknown feature-type" in unknown_feature.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_shortcuts_and_external_module_loading(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=8, include_missing=False)
|
||||||
|
minute_bars = make_generated_minute_bars(daily_bars)
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
minute_path = tmp_path / "minute_bars.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
minute_bars.to_parquet(minute_path, index=False)
|
||||||
|
|
||||||
|
alpha_module = tmp_path / "listed_alpha.py"
|
||||||
|
alpha_module.write_text(textwrap.dedent("""
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class ListedAlpha(BaseAlpha):
|
||||||
|
name = "listed_alpha_cli"
|
||||||
|
|
||||||
|
def __init__(self, scale: float = 1.0, label: str = "x"):
|
||||||
|
self.scale = scale
|
||||||
|
self.label = label
|
||||||
|
|
||||||
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
return close.pct_change(1, fill_method=None) * self.scale
|
||||||
|
"""))
|
||||||
|
derived_module = tmp_path / "listed_derived.py"
|
||||||
|
derived_module.write_text(textwrap.dedent("""
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import register_derived
|
||||||
|
|
||||||
|
@register_derived
|
||||||
|
class ListedDerived(BaseDerivedData):
|
||||||
|
name = "listed_derived_cli"
|
||||||
|
|
||||||
|
def __init__(self, scale: float = 1.0):
|
||||||
|
self.scale = scale
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None) -> pd.DataFrame:
|
||||||
|
out = daily[["symbol_id", "date", "close"]].copy()
|
||||||
|
out["listed_value"] = out.pop("close") * self.scale
|
||||||
|
return out
|
||||||
|
"""))
|
||||||
|
derived_compute_module = tmp_path / "computed_derived.py"
|
||||||
|
derived_compute_module.write_text(textwrap.dedent("""
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import register_derived
|
||||||
|
|
||||||
|
@register_derived
|
||||||
|
class ComputedDerived(BaseDerivedData):
|
||||||
|
name = "computed_derived_cli"
|
||||||
|
|
||||||
|
def __init__(self, scale: float = 1.0):
|
||||||
|
self.scale = scale
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None) -> pd.DataFrame:
|
||||||
|
out = daily[["symbol_id", "date", "close"]].copy()
|
||||||
|
out["computed_value"] = out.pop("close") * self.scale
|
||||||
|
return out
|
||||||
|
"""))
|
||||||
|
feature_module = tmp_path / "listed_feature.py"
|
||||||
|
feature_module.write_text(textwrap.dedent("""
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.features.base import BaseFeature
|
||||||
|
from pipeline.features.registry import register_feature
|
||||||
|
|
||||||
|
@register_feature
|
||||||
|
class ListedFeature(BaseFeature):
|
||||||
|
name = "listed_feature_cli"
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None) -> pd.DataFrame:
|
||||||
|
out = minute[["symbol_id", "date", "close"]].copy()
|
||||||
|
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
|
||||||
|
out = out.groupby(["symbol_id", "date"], as_index=False)["close"].mean()
|
||||||
|
return out.rename(columns={"close": "listed_feature_value"})
|
||||||
|
"""))
|
||||||
|
feature_compute_module = tmp_path / "computed_feature.py"
|
||||||
|
feature_compute_module.write_text(textwrap.dedent("""
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.features.base import BaseFeature
|
||||||
|
from pipeline.features.registry import register_feature
|
||||||
|
|
||||||
|
@register_feature
|
||||||
|
class ComputedFeature(BaseFeature):
|
||||||
|
name = "computed_feature_cli"
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None) -> pd.DataFrame:
|
||||||
|
out = minute[["symbol_id", "date", "close"]].copy()
|
||||||
|
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
|
||||||
|
out = out.groupby(["symbol_id", "date"], as_index=False)["close"].mean()
|
||||||
|
return out.rename(columns={"close": "computed_feature_value"})
|
||||||
|
"""))
|
||||||
|
|
||||||
|
alpha_list = _invoke_ok(runner, [
|
||||||
|
"alpha", "list",
|
||||||
|
"--alpha-module", str(alpha_module),
|
||||||
|
])
|
||||||
|
assert "listed_alpha_cli" in alpha_list.output
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
reversal = _invoke_ok(runner, [
|
||||||
|
"alpha", "reversal",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
"--lookback", "3",
|
||||||
|
])
|
||||||
|
reversal_vol = _invoke_ok(runner, [
|
||||||
|
"alpha", "reversal-vol",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
"--lookback", "3",
|
||||||
|
"--vol-window", "3",
|
||||||
|
])
|
||||||
|
external_alpha = _invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-type", "listed_alpha_cli",
|
||||||
|
"--alpha-name", "listed_alpha_run",
|
||||||
|
"--param", "scale=2.5",
|
||||||
|
"--param", "label=demo",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
|
||||||
|
assert "Saved alpha:" in reversal.output
|
||||||
|
assert "Saved alpha:" in reversal_vol.output
|
||||||
|
assert "Saved alpha:" in external_alpha.output
|
||||||
|
assert (alpha_dir / "reversal_3d.pq").exists()
|
||||||
|
assert (alpha_dir / "reversal_vol_3d_3d.pq").exists()
|
||||||
|
assert (alpha_dir / "listed_alpha_run.pq").exists()
|
||||||
|
|
||||||
|
derived_list = _invoke_ok(runner, [
|
||||||
|
"derived", "list",
|
||||||
|
"--derived-module", str(derived_module),
|
||||||
|
])
|
||||||
|
assert "listed_derived_cli" in derived_list.output
|
||||||
|
derived_dir = tmp_path / "derived_external"
|
||||||
|
derived_compute = _invoke_ok(runner, [
|
||||||
|
"derived", "compute",
|
||||||
|
"--daily-path", str(daily_path),
|
||||||
|
"--derived-module", str(derived_compute_module),
|
||||||
|
"--derived-type", "computed_derived_cli",
|
||||||
|
"--derived-name", "listed_derived_run",
|
||||||
|
"--param", "scale=3",
|
||||||
|
"--output-dir", str(derived_dir),
|
||||||
|
])
|
||||||
|
assert "Saved derived data:" in derived_compute.output
|
||||||
|
assert (derived_dir / "listed_derived_run.pq").exists()
|
||||||
|
|
||||||
|
feature_list = _invoke_ok(runner, [
|
||||||
|
"feature", "list",
|
||||||
|
"--feature-module", str(feature_module),
|
||||||
|
])
|
||||||
|
assert "listed_feature_cli" in feature_list.output
|
||||||
|
feature_dir = tmp_path / "features_external"
|
||||||
|
feature_compute = _invoke_ok(runner, [
|
||||||
|
"feature", "compute",
|
||||||
|
"--minute-path", str(minute_path),
|
||||||
|
"--feature-module", str(feature_compute_module),
|
||||||
|
"--feature-type", "computed_feature_cli",
|
||||||
|
"--feature-name", "listed_feature_run",
|
||||||
|
"--output-dir", str(feature_dir),
|
||||||
|
])
|
||||||
|
assert "Saved feature:" in feature_compute.output
|
||||||
|
assert (feature_dir / "listed_feature_run.pq").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_pqcat_row_modes(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
|
||||||
|
head_result = _invoke_ok(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(daily_path),
|
||||||
|
"--head",
|
||||||
|
"2",
|
||||||
|
"--columns",
|
||||||
|
"symbol_id,close",
|
||||||
|
])
|
||||||
|
tail_result = _invoke_ok(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(daily_path),
|
||||||
|
"--tail",
|
||||||
|
"1",
|
||||||
|
])
|
||||||
|
|
||||||
|
assert "symbol_id" in head_result.output
|
||||||
|
assert "close" in head_result.output
|
||||||
|
assert "date" in tail_result.output
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
"""Offline coverage for data CLI and universe resolution glue."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from cli import cli
|
||||||
|
import pipeline.data.cli as data_cli
|
||||||
|
import pipeline.data.downloader as pipeline_downloader
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_universe_handles_named_file_all_and_symbol_list(tmp_path, monkeypatch):
|
||||||
|
hs300_raw = pd.DataFrame({
|
||||||
|
"updateDate": ["2024-01-12", "2024-01-12"],
|
||||||
|
"stockName": ["浦发银行", "平安银行"],
|
||||||
|
"stockCode": ["sh.600000", "sz.000001"],
|
||||||
|
})
|
||||||
|
zz500_raw = pd.DataFrame({
|
||||||
|
"name": ["东风汽车"],
|
||||||
|
"code": ["sh.600006"],
|
||||||
|
"date": ["2024-01-12"],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "get_hs300_stocks", lambda: hs300_raw)
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "get_zz500_stocks", lambda: zz500_raw)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"get_all_stocks",
|
||||||
|
lambda: pd.DataFrame({
|
||||||
|
"code": ["sh600000", "sz000001", "sh600519"],
|
||||||
|
"name": ["浦发银行", "平安银行", "贵州茅台"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
symbol_file = tmp_path / "symbols.txt"
|
||||||
|
symbol_file.write_text("sh600000\n\nsz000001\n")
|
||||||
|
|
||||||
|
hs300 = pipeline_downloader._resolve_universe("hs300")
|
||||||
|
zz500 = pipeline_downloader._resolve_universe("csi500")
|
||||||
|
all_capped = pipeline_downloader._resolve_universe("all", max_symbols=2)
|
||||||
|
from_file = pipeline_downloader._resolve_universe(str(symbol_file))
|
||||||
|
from_list = pipeline_downloader._resolve_universe("sh600000, sz000001")
|
||||||
|
|
||||||
|
assert hs300.to_dict("list") == {
|
||||||
|
"symbol_name": ["浦发银行", "平安银行"],
|
||||||
|
"symbol_id": ["sh600000", "sz000001"],
|
||||||
|
}
|
||||||
|
assert zz500.to_dict("list") == {
|
||||||
|
"symbol_name": ["东风汽车"],
|
||||||
|
"symbol_id": ["sh600006"],
|
||||||
|
}
|
||||||
|
assert all_capped["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert from_file["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert from_file["symbol_name"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert from_list["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_data_cli_download_commands_print_summaries_without_network(monkeypatch, tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_calls: list[dict] = []
|
||||||
|
minute_calls: list[dict] = []
|
||||||
|
|
||||||
|
def fake_daily(**kwargs):
|
||||||
|
daily_calls.append(kwargs)
|
||||||
|
return {
|
||||||
|
"dataset_path": str(tmp_path / "daily" / kwargs["universe"]),
|
||||||
|
"n_symbols": 2,
|
||||||
|
"n_requested": 3,
|
||||||
|
"n_rows": 18,
|
||||||
|
"date_min": "2024-01-02",
|
||||||
|
"date_max": "2024-01-12",
|
||||||
|
}
|
||||||
|
|
||||||
|
def fake_minute(**kwargs):
|
||||||
|
minute_calls.append(kwargs)
|
||||||
|
return {
|
||||||
|
"dataset_path": str(tmp_path / "minute" / kwargs["universe"]),
|
||||||
|
"frequency": "15m",
|
||||||
|
"n_symbols": 1,
|
||||||
|
"n_requested": 1,
|
||||||
|
"n_rows": 32,
|
||||||
|
"date_min": "2024-01-02",
|
||||||
|
"date_max": "2024-01-03",
|
||||||
|
}
|
||||||
|
|
||||||
|
monkeypatch.setattr(data_cli, "download_universe", fake_daily)
|
||||||
|
monkeypatch.setattr(data_cli, "download_minute_universe", fake_minute)
|
||||||
|
|
||||||
|
daily_result = runner.invoke(cli, [
|
||||||
|
"data",
|
||||||
|
"download",
|
||||||
|
"--universe",
|
||||||
|
"sh600000,sz000001",
|
||||||
|
"--start-date",
|
||||||
|
"2024-01-02",
|
||||||
|
"--end-date",
|
||||||
|
"2024-01-12",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "daily"),
|
||||||
|
"--symbols",
|
||||||
|
"3",
|
||||||
|
"--chunk-size",
|
||||||
|
"2",
|
||||||
|
"--adjust",
|
||||||
|
"none",
|
||||||
|
])
|
||||||
|
minute_result = runner.invoke(cli, [
|
||||||
|
"data",
|
||||||
|
"download-minute",
|
||||||
|
"--universe",
|
||||||
|
"toy",
|
||||||
|
"--start-date",
|
||||||
|
"2024-01-02",
|
||||||
|
"--end-date",
|
||||||
|
"2024-01-03",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "minute"),
|
||||||
|
"--symbols",
|
||||||
|
"1",
|
||||||
|
"--chunk-size",
|
||||||
|
"1",
|
||||||
|
"--frequency",
|
||||||
|
"15",
|
||||||
|
])
|
||||||
|
|
||||||
|
assert daily_result.exit_code == 0, daily_result.output
|
||||||
|
assert "Summary: 2/3 symbols, 18 bars" in daily_result.output
|
||||||
|
assert daily_calls == [{
|
||||||
|
"universe": "sh600000,sz000001",
|
||||||
|
"start_date": "2024-01-02",
|
||||||
|
"end_date": "2024-01-12",
|
||||||
|
"output_dir": str(tmp_path / "daily"),
|
||||||
|
"max_symbols": 3,
|
||||||
|
"chunk_size": 2,
|
||||||
|
"adjust": "none",
|
||||||
|
}]
|
||||||
|
assert minute_result.exit_code == 0, minute_result.output
|
||||||
|
assert "frequency=15m" in minute_result.output
|
||||||
|
assert minute_calls == [{
|
||||||
|
"universe": "toy",
|
||||||
|
"start_date": "2024-01-02",
|
||||||
|
"end_date": "2024-01-03",
|
||||||
|
"output_dir": str(tmp_path / "minute"),
|
||||||
|
"max_symbols": 1,
|
||||||
|
"chunk_size": 1,
|
||||||
|
"frequency": "15",
|
||||||
|
}]
|
||||||
@@ -0,0 +1,376 @@
|
|||||||
|
"""Tests for daily derived-data ingestion and plugins."""
|
||||||
|
|
||||||
|
import textwrap
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from cli import cli
|
||||||
|
from pipeline.alpha.compute import join_feature_frames
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.compute import compute_derived, read_derived_frame, validate_derived_frame
|
||||||
|
from pipeline.derived.registry import (
|
||||||
|
available_derived,
|
||||||
|
get_derived,
|
||||||
|
load_derived_module,
|
||||||
|
register_derived,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _daily_bars() -> pd.DataFrame:
|
||||||
|
return pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
|
||||||
|
"open": [10.0, 20.0, 11.0],
|
||||||
|
"close": [10.5, 20.5, 11.5],
|
||||||
|
"volume": [1000.0, 2000.0, 1200.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def _minute_bars() -> pd.DataFrame:
|
||||||
|
return pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001"],
|
||||||
|
"datetime": pd.to_datetime([
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
"2024-01-02 09:40:00",
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
]),
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
|
||||||
|
"time": ["09:35:00", "09:40:00", "09:35:00"],
|
||||||
|
"open": [10.0, 10.5, 20.0],
|
||||||
|
"high": [11.0, 12.0, 21.0],
|
||||||
|
"low": [9.0, 10.0, 19.0],
|
||||||
|
"close": [10.5, 11.0, 20.5],
|
||||||
|
"volume": [100.0, 300.0, 200.0],
|
||||||
|
"amount": [1000.0, 3300.0, 4100.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_normalizes_and_sorts():
|
||||||
|
result = validate_derived_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sz000001", "sh600000"],
|
||||||
|
"date": ["2024-01-02 15:00:00", "2024-01-02 09:30:00"],
|
||||||
|
"custom_value": [2.0, 1.0],
|
||||||
|
}))
|
||||||
|
|
||||||
|
assert result["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert result["date"].tolist() == [
|
||||||
|
pd.Timestamp("2024-01-02"),
|
||||||
|
pd.Timestamp("2024-01-02"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_missing_keys():
|
||||||
|
with pytest.raises(ValueError, match="missing required"):
|
||||||
|
validate_derived_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_duplicate_normalized_keys():
|
||||||
|
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
|
||||||
|
validate_derived_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
|
||||||
|
"x": [1.0, 2.0],
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_duplicate_columns():
|
||||||
|
bad = pd.DataFrame(
|
||||||
|
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
|
||||||
|
columns=["symbol_id", "date", "dup", "dup"],
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError, match="duplicate columns"):
|
||||||
|
validate_derived_frame(bad)
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_non_numeric_values():
|
||||||
|
with pytest.raises(ValueError, match="numeric"):
|
||||||
|
validate_derived_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"bad": ["not numeric"],
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_missing_value_columns():
|
||||||
|
with pytest.raises(ValueError, match="at least one value column"):
|
||||||
|
validate_derived_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_derived_frame_rejects_empty_csv(tmp_path):
|
||||||
|
empty_csv = tmp_path / "empty.csv"
|
||||||
|
empty_csv.write_text("")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="CSV input is empty"):
|
||||||
|
read_derived_frame(empty_csv)
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_derived_rejects_missing_inputs():
|
||||||
|
with pytest.raises(ValueError, match="requires --daily-path or --minute-path"):
|
||||||
|
compute_derived("minute_daily_summary")
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_ingest_cli_accepts_csv_and_parquet(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
source = pd.DataFrame({
|
||||||
|
"symbol_id": ["sz000001", "sh600000"],
|
||||||
|
"date": ["2024-01-02", "2024-01-02"],
|
||||||
|
"custom_value": [2.0, 1.0],
|
||||||
|
})
|
||||||
|
csv_path = tmp_path / "custom.csv"
|
||||||
|
parquet_path = tmp_path / "custom.pq"
|
||||||
|
out_dir = tmp_path / "derived"
|
||||||
|
source.to_csv(csv_path, index=False)
|
||||||
|
source.to_parquet(parquet_path, index=False)
|
||||||
|
|
||||||
|
csv_result = runner.invoke(cli, [
|
||||||
|
"derived", "ingest",
|
||||||
|
"--input-path", str(csv_path),
|
||||||
|
"--derived-name", "csv_custom",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
assert csv_result.exit_code == 0, csv_result.output
|
||||||
|
|
||||||
|
parquet_result = runner.invoke(cli, [
|
||||||
|
"derived", "ingest",
|
||||||
|
"--input-path", str(parquet_path),
|
||||||
|
"--derived-name", "parquet_custom",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
assert parquet_result.exit_code == 0, parquet_result.output
|
||||||
|
|
||||||
|
written = pd.read_parquet(out_dir / "csv_custom.pq")
|
||||||
|
assert written["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert (out_dir / "parquet_custom.pq").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_validate_cli_rejects_duplicate_csv_columns(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
csv_path = tmp_path / "bad.csv"
|
||||||
|
csv_path.write_text("symbol_id,date,x,x\nsh600000,2024-01-02,1.0,2.0\n")
|
||||||
|
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"derived", "validate",
|
||||||
|
"--input-path", str(csv_path),
|
||||||
|
])
|
||||||
|
|
||||||
|
assert result.exit_code != 0
|
||||||
|
assert "duplicate columns" in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_ingest_cli_wraps_validation_errors(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
csv_path = tmp_path / "bad_ingest.csv"
|
||||||
|
csv_path.write_text("symbol_id,date\nsh600000,2024-01-02\n")
|
||||||
|
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"derived",
|
||||||
|
"ingest",
|
||||||
|
"--input-path",
|
||||||
|
str(csv_path),
|
||||||
|
"--derived-name",
|
||||||
|
"bad",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "derived"),
|
||||||
|
])
|
||||||
|
|
||||||
|
assert result.exit_code != 0
|
||||||
|
assert "at least one value column" in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_external_derived_plugin_loads_filters_params_and_uses_inputs(tmp_path):
|
||||||
|
module_path = tmp_path / "external_derived.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import register_derived
|
||||||
|
|
||||||
|
@register_derived
|
||||||
|
class FlexibleDerived(BaseDerivedData):
|
||||||
|
name = "flexible_derived_test"
|
||||||
|
|
||||||
|
def __init__(self, scale: float = 1.0):
|
||||||
|
self.scale = scale
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None) -> pd.DataFrame:
|
||||||
|
result = None
|
||||||
|
if daily is not None:
|
||||||
|
result = daily[["symbol_id", "date", "close"]].copy()
|
||||||
|
result["daily_scaled_close"] = result.pop("close") * self.scale
|
||||||
|
if minute is not None:
|
||||||
|
minute_out = (
|
||||||
|
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
|
||||||
|
.sum()
|
||||||
|
.rename(columns={"volume": "minute_volume_sum"})
|
||||||
|
)
|
||||||
|
minute_out["minute_volume_sum"] *= self.scale
|
||||||
|
result = minute_out if result is None else result.merge(
|
||||||
|
minute_out, on=["symbol_id", "date"], how="left"
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
'''))
|
||||||
|
|
||||||
|
load_derived_module(str(module_path))
|
||||||
|
assert "flexible_derived_test" in available_derived()
|
||||||
|
|
||||||
|
instance = get_derived("flexible_derived_test", scale=2.0, ignored=99)
|
||||||
|
assert instance.scale == 2.0
|
||||||
|
assert not hasattr(instance, "ignored")
|
||||||
|
|
||||||
|
daily_result = compute_derived(
|
||||||
|
"flexible_derived_test",
|
||||||
|
daily=_daily_bars(),
|
||||||
|
scale=2.0,
|
||||||
|
ignored=99,
|
||||||
|
)
|
||||||
|
assert "daily_scaled_close" in daily_result.columns
|
||||||
|
assert np.isclose(daily_result["daily_scaled_close"].iloc[0], 21.0)
|
||||||
|
|
||||||
|
minute_result = compute_derived(
|
||||||
|
"flexible_derived_test",
|
||||||
|
minute=_minute_bars(),
|
||||||
|
scale=2.0,
|
||||||
|
)
|
||||||
|
assert "minute_volume_sum" in minute_result.columns
|
||||||
|
assert np.isclose(
|
||||||
|
minute_result.loc[minute_result["symbol_id"] == "sh600000", "minute_volume_sum"].iloc[0],
|
||||||
|
800.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
both_result = compute_derived(
|
||||||
|
"flexible_derived_test",
|
||||||
|
daily=_daily_bars(),
|
||||||
|
minute=_minute_bars(),
|
||||||
|
scale=1.0,
|
||||||
|
)
|
||||||
|
assert {"daily_scaled_close", "minute_volume_sum"}.issubset(both_result.columns)
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_registry_rejects_bad_plugins_and_load_failures(tmp_path, monkeypatch):
|
||||||
|
with pytest.raises(TypeError):
|
||||||
|
register_derived(object) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="non-empty"):
|
||||||
|
@register_derived
|
||||||
|
class NoNameDerived(BaseDerivedData):
|
||||||
|
def compute(self, daily=None, minute=None):
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
@register_derived
|
||||||
|
class _CoverageDerived(BaseDerivedData):
|
||||||
|
name = "_coverage_derived_registry"
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None):
|
||||||
|
return pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"x": [1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="already registered"):
|
||||||
|
@register_derived
|
||||||
|
class _CoverageDerivedDuplicate(BaseDerivedData):
|
||||||
|
name = "_coverage_derived_registry"
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None):
|
||||||
|
return pd.DataFrame()
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="Unknown derived data"):
|
||||||
|
get_derived("does_not_exist")
|
||||||
|
|
||||||
|
missing_path = tmp_path / "missing_derived.py"
|
||||||
|
with pytest.raises(FileNotFoundError):
|
||||||
|
load_derived_module(str(missing_path))
|
||||||
|
|
||||||
|
bad_path = tmp_path / "bad_derived.py"
|
||||||
|
bad_path.write_text("x = 1\n")
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"pipeline.derived.registry.importlib.util.spec_from_file_location",
|
||||||
|
lambda *args, **kwargs: None,
|
||||||
|
)
|
||||||
|
with pytest.raises(ImportError, match="Cannot load derived data module"):
|
||||||
|
load_derived_module(str(bad_path))
|
||||||
|
|
||||||
|
load_derived_module("math")
|
||||||
|
instance = _CoverageDerived()
|
||||||
|
instance.scale = 2
|
||||||
|
assert repr(instance) == "_CoverageDerived(scale=2)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_compute_cli_writes_builtin_minute_summary(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
minute_path = tmp_path / "minute.pq"
|
||||||
|
out_dir = tmp_path / "derived"
|
||||||
|
_minute_bars().to_parquet(minute_path, index=False)
|
||||||
|
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"derived", "compute",
|
||||||
|
"--minute-path", str(minute_path),
|
||||||
|
"--derived-type", "minute_daily_summary",
|
||||||
|
"--derived-name", "minute_summary",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
written = pd.read_parquet(out_dir / "minute_summary.pq")
|
||||||
|
assert "minute_vwap" in written.columns
|
||||||
|
|
||||||
|
|
||||||
|
def test_minute_summary_uses_time_sort_and_daily_without_close():
|
||||||
|
minute = _minute_bars().drop(columns=["datetime"]).sample(frac=1.0, random_state=0)
|
||||||
|
daily = _daily_bars()[["symbol_id", "date", "open"]]
|
||||||
|
|
||||||
|
result = compute_derived(
|
||||||
|
"minute_daily_summary",
|
||||||
|
daily=daily,
|
||||||
|
minute=minute,
|
||||||
|
)
|
||||||
|
|
||||||
|
by_symbol = result.set_index(["symbol_id", "date"])
|
||||||
|
assert np.isclose(
|
||||||
|
by_symbol.loc[("sh600000", pd.Timestamp("2024-01-02")), "minute_intraday_return"],
|
||||||
|
0.10,
|
||||||
|
)
|
||||||
|
assert "minute_vwap_deviation" in result.columns
|
||||||
|
|
||||||
|
|
||||||
|
def test_alpha_feature_join_rejects_derived_column_collisions():
|
||||||
|
data = _daily_bars()
|
||||||
|
derived_a = data[["symbol_id", "date"]].copy()
|
||||||
|
derived_a["custom_value"] = 1.0
|
||||||
|
derived_b = data[["symbol_id", "date"]].copy()
|
||||||
|
derived_b["custom_value"] = 2.0
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="conflict"):
|
||||||
|
join_feature_frames(data, [derived_a, derived_b])
|
||||||
|
|
||||||
|
close_collision = data[["symbol_id", "date"]].copy()
|
||||||
|
close_collision["close"] = 1.0
|
||||||
|
with pytest.raises(ValueError, match="conflict"):
|
||||||
|
join_feature_frames(data, [close_collision])
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_feature_cli_delegates_to_derived_registry(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
minute_path = tmp_path / "minute.pq"
|
||||||
|
out_dir = tmp_path / "features"
|
||||||
|
_minute_bars().to_parquet(minute_path, index=False)
|
||||||
|
|
||||||
|
list_result = runner.invoke(cli, ["feature", "list"])
|
||||||
|
assert list_result.exit_code == 0, list_result.output
|
||||||
|
assert "minute_daily_summary" in list_result.output
|
||||||
|
|
||||||
|
compute_result = runner.invoke(cli, [
|
||||||
|
"feature", "compute",
|
||||||
|
"--minute-path", str(minute_path),
|
||||||
|
"--feature-type", "minute_daily_summary",
|
||||||
|
"--feature-name", "minute_summary",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
assert compute_result.exit_code == 0, compute_result.output
|
||||||
|
assert (out_dir / "minute_summary.pq").exists()
|
||||||
@@ -1,6 +1,16 @@
|
|||||||
|
import os
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from data.downloader import download_daily
|
from data.downloader import download_daily
|
||||||
|
|
||||||
|
pytestmark = [
|
||||||
|
pytest.mark.network,
|
||||||
|
pytest.mark.skipif(
|
||||||
|
os.environ.get("CEQ_RUN_LIVE_DOWNLOADER") != "1",
|
||||||
|
reason="set CEQ_RUN_LIVE_DOWNLOADER=1 to run live baostock/akshare smoke tests",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_download_single_stock():
|
def test_download_single_stock():
|
||||||
"""Smoke test: download data for 浦发银行 for a short window."""
|
"""Smoke test: download data for 浦发银行 for a short window."""
|
||||||
|
|||||||
@@ -0,0 +1,631 @@
|
|||||||
|
"""Offline downloader contract tests with mocked data providers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import data.downloader as downloader
|
||||||
|
import pipeline.data.downloader as pipeline_downloader
|
||||||
|
from data.downloader import download_daily, download_daily_batch
|
||||||
|
from pipeline.common.schema import DATA_COLUMNS
|
||||||
|
from pipeline.data.downloader import download_universe
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResult:
|
||||||
|
def __init__(self, rows, error_code="0", error_msg=""):
|
||||||
|
self.rows = rows
|
||||||
|
self.error_code = error_code
|
||||||
|
self.error_msg = error_msg
|
||||||
|
self._idx = -1
|
||||||
|
|
||||||
|
def next(self):
|
||||||
|
self._idx += 1
|
||||||
|
return self._idx < len(self.rows)
|
||||||
|
|
||||||
|
def get_row_data(self):
|
||||||
|
return self.rows[self._idx]
|
||||||
|
|
||||||
|
|
||||||
|
def _daily_batch_row(
|
||||||
|
date: str = "2024-01-02",
|
||||||
|
open_: str = "10",
|
||||||
|
high: str = "11",
|
||||||
|
low: str = "9",
|
||||||
|
close: str = "10.5",
|
||||||
|
preclose: str = "10",
|
||||||
|
volume: str = "1000",
|
||||||
|
amount: str = "10500",
|
||||||
|
) -> list[str]:
|
||||||
|
return [
|
||||||
|
date,
|
||||||
|
open_,
|
||||||
|
high,
|
||||||
|
low,
|
||||||
|
close,
|
||||||
|
preclose,
|
||||||
|
volume,
|
||||||
|
amount,
|
||||||
|
"1.23",
|
||||||
|
"5.0",
|
||||||
|
"1",
|
||||||
|
"0",
|
||||||
|
"8.0",
|
||||||
|
"1.1",
|
||||||
|
"2.2",
|
||||||
|
"3.3",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_uses_baostock_before_akshare_in_auto(monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
expected = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000"],
|
||||||
|
"date": ["2024-01-02"],
|
||||||
|
"open": [10.0],
|
||||||
|
"high": [11.0],
|
||||||
|
"low": [9.0],
|
||||||
|
"close": [10.5],
|
||||||
|
"volume": [1000.0],
|
||||||
|
"amount": [10500.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
def fake_baostock(symbol, start, end, adjust):
|
||||||
|
calls.append("baostock")
|
||||||
|
return expected
|
||||||
|
|
||||||
|
def fake_akshare(symbol, start, end, adjust):
|
||||||
|
calls.append("akshare")
|
||||||
|
raise AssertionError("akshare should not be called after baostock succeeds")
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader, "_download_baostock", fake_baostock)
|
||||||
|
monkeypatch.setattr(downloader, "_download_akshare", fake_akshare)
|
||||||
|
|
||||||
|
result = download_daily("sh600000", "2024-01-02", "2024-01-02", source="auto")
|
||||||
|
|
||||||
|
assert calls == ["baostock"]
|
||||||
|
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
assert result["close"].tolist() == [10.5]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_falls_back_to_akshare_when_baostock_empty(monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
fallback = pd.DataFrame({
|
||||||
|
"symbol": ["sz000001"],
|
||||||
|
"date": ["2024-01-02"],
|
||||||
|
"open": [20.0],
|
||||||
|
"high": [21.0],
|
||||||
|
"low": [19.0],
|
||||||
|
"close": [20.5],
|
||||||
|
"volume": [2000.0],
|
||||||
|
"amount": [41000.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_baostock",
|
||||||
|
lambda symbol, start, end, adjust: calls.append("baostock") or None,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_akshare",
|
||||||
|
lambda symbol, start, end, adjust: calls.append("akshare") or fallback,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = download_daily("sz000001", "2024-01-02", "2024-01-02", source="auto")
|
||||||
|
|
||||||
|
assert calls == ["baostock", "akshare"]
|
||||||
|
assert result["symbol"].tolist() == ["sz000001"]
|
||||||
|
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_raises_when_requested_source_has_no_data(monkeypatch):
|
||||||
|
monkeypatch.setattr(downloader, "_download_baostock", lambda *args: None)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="Failed to download data for sh600000"):
|
||||||
|
download_daily(
|
||||||
|
"sh600000",
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
source="baostock",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_akshare_source_skips_baostock(monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
fallback = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000"],
|
||||||
|
"date": ["2024-01-02"],
|
||||||
|
"open": [10.0],
|
||||||
|
"high": [11.0],
|
||||||
|
"low": [9.0],
|
||||||
|
"close": [10.5],
|
||||||
|
"volume": [1000.0],
|
||||||
|
"amount": [10500.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_baostock",
|
||||||
|
lambda *args: calls.append("baostock") or None,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_akshare",
|
||||||
|
lambda *args: calls.append("akshare") or fallback,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = download_daily(
|
||||||
|
"sh600000",
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
source="akshare",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert calls == ["akshare"]
|
||||||
|
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_akshare_daily_downloader_maps_columns_and_failures(monkeypatch):
|
||||||
|
calls: list[dict] = []
|
||||||
|
raw = pd.DataFrame({
|
||||||
|
"日期": ["2024-01-02"],
|
||||||
|
"开盘": [10.0],
|
||||||
|
"最高": [11.0],
|
||||||
|
"最低": [9.0],
|
||||||
|
"收盘": [10.5],
|
||||||
|
"成交量": [1000.0],
|
||||||
|
"成交额": [10500.0],
|
||||||
|
"换手率": [1.2],
|
||||||
|
})
|
||||||
|
|
||||||
|
def fake_hist(**kwargs):
|
||||||
|
calls.append(kwargs)
|
||||||
|
return raw.copy()
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", fake_hist)
|
||||||
|
|
||||||
|
result = downloader._download_akshare(
|
||||||
|
"sh600000",
|
||||||
|
"20240102",
|
||||||
|
"20240102",
|
||||||
|
adjust="",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert calls == [{
|
||||||
|
"symbol": "600000",
|
||||||
|
"period": "daily",
|
||||||
|
"start_date": "20240102",
|
||||||
|
"end_date": "20240102",
|
||||||
|
"adjust": "",
|
||||||
|
}]
|
||||||
|
assert result is not None
|
||||||
|
assert result.columns.tolist() == [
|
||||||
|
"symbol", "date", "open", "high", "low", "close", "volume", "amount",
|
||||||
|
]
|
||||||
|
assert result["symbol"].tolist() == ["sh600000"]
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", lambda **kwargs: pd.DataFrame())
|
||||||
|
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.ak,
|
||||||
|
"stock_zh_a_hist",
|
||||||
|
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("provider down")),
|
||||||
|
)
|
||||||
|
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_baostock_daily_downloader_maps_errors_and_logout_failures(monkeypatch):
|
||||||
|
query_calls: list[dict] = []
|
||||||
|
row = ["2024-01-02", "10", "11", "9", "10.5", "1000", "10500"]
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"logout",
|
||||||
|
lambda: (_ for _ in ()).throw(RuntimeError("logout failed")),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_query(**kwargs):
|
||||||
|
query_calls.append(kwargs)
|
||||||
|
return _FakeResult([row])
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
|
||||||
|
|
||||||
|
result = downloader._download_baostock(
|
||||||
|
"sz000001",
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
adjust="none",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert query_calls[0]["code"] == "sz.000001"
|
||||||
|
assert query_calls[0]["adjustflag"] == "3"
|
||||||
|
assert result is not None
|
||||||
|
assert result["symbol"].tolist() == ["sz000001"]
|
||||||
|
assert pd.api.types.is_numeric_dtype(result["close"])
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([], error_code="1", error_msg="bad symbol"),
|
||||||
|
)
|
||||||
|
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([]),
|
||||||
|
)
|
||||||
|
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("query failed")),
|
||||||
|
)
|
||||||
|
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_maps_rich_schema_and_vwap(monkeypatch):
|
||||||
|
query_calls: list[dict] = []
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
def fake_query(**kwargs):
|
||||||
|
query_calls.append(kwargs)
|
||||||
|
rows = [
|
||||||
|
_daily_batch_row(volume="1000", amount="10500"),
|
||||||
|
_daily_batch_row(date="2024-01-03", volume="0", amount="0"),
|
||||||
|
]
|
||||||
|
return _FakeResult(rows)
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
|
||||||
|
|
||||||
|
[(symbol, frame)] = list(
|
||||||
|
download_daily_batch(
|
||||||
|
["sh600000"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-03",
|
||||||
|
adjust="hfq",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert query_calls[0]["code"] == "sh.600000"
|
||||||
|
assert query_calls[0]["adjustflag"] == "1"
|
||||||
|
assert login_count == 1
|
||||||
|
assert logout_count == 1
|
||||||
|
assert frame is not None
|
||||||
|
assert frame.columns.tolist() == [
|
||||||
|
"symbol", "date", "open", "high", "low", "close", "preclose",
|
||||||
|
"volume", "amount", "vwap", "turn", "pctChg", "tradestatus", "isST",
|
||||||
|
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
|
||||||
|
]
|
||||||
|
assert np.isclose(frame["vwap"].iloc[0], 10.5)
|
||||||
|
assert pd.isna(frame["vwap"].iloc[1])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(frame["date"])
|
||||||
|
assert pd.api.types.is_numeric_dtype(frame["tradestatus"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_periodic_relogin_and_none_result(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="1", error_msg="bad symbol"),
|
||||||
|
_FakeResult([_daily_batch_row(date="2024-01-03")]),
|
||||||
|
]
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
results = list(
|
||||||
|
download_daily_batch(
|
||||||
|
["sh600000", "sz000001"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-03",
|
||||||
|
relogin_every=1,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert results[0] == ("sh600000", None)
|
||||||
|
assert results[1][0] == "sz000001"
|
||||||
|
assert results[1][1] is not None
|
||||||
|
assert login_count == 2
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_empty_rows_yields_none(monkeypatch):
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([]),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_generic_exception_yields_none(monkeypatch):
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("query failed")),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_relogs_and_retries_session_loss(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
_FakeResult([_daily_batch_row()]),
|
||||||
|
]
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
[(symbol, frame)] = list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02"))
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert frame is not None
|
||||||
|
assert len(frame) == 1
|
||||||
|
assert login_count == 2
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_second_session_loss_and_logout_failure(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
]
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
raise RuntimeError("logout failed")
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_uses_akshare_fallback_when_enabled(monkeypatch):
|
||||||
|
fallback = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000"],
|
||||||
|
"date": ["2024-01-02"],
|
||||||
|
"open": [10.0],
|
||||||
|
"high": [11.0],
|
||||||
|
"low": [9.0],
|
||||||
|
"close": [10.5],
|
||||||
|
"volume": [1000.0],
|
||||||
|
"amount": [10500.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([], error_code="1", error_msg="no data"),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_akshare",
|
||||||
|
lambda symbol, start, end, adjust: fallback.copy(),
|
||||||
|
)
|
||||||
|
|
||||||
|
[(symbol, frame)] = list(
|
||||||
|
download_daily_batch(
|
||||||
|
["sh600000"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
akshare_fallback=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert frame is not None
|
||||||
|
assert frame["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
assert frame["close"].tolist() == [10.5]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_universe_writes_daily_partitions_from_mock_batch(tmp_path, monkeypatch):
|
||||||
|
batch_frame = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-02-01"]),
|
||||||
|
"open": [10.0, 11.0],
|
||||||
|
"high": [11.0, 12.0],
|
||||||
|
"low": [9.0, 10.0],
|
||||||
|
"close": [10.5, 11.5],
|
||||||
|
"preclose": [10.0, 10.5],
|
||||||
|
"volume": [1000.0, 1200.0],
|
||||||
|
"amount": [10500.0, 13800.0],
|
||||||
|
"vwap": [10.5, 11.5],
|
||||||
|
"turn": [1.0, 1.1],
|
||||||
|
"pctChg": [5.0, 9.5],
|
||||||
|
"tradestatus": [1, 1],
|
||||||
|
"isST": [0, 0],
|
||||||
|
"peTTM": [8.0, 8.1],
|
||||||
|
"pbMRQ": [1.1, 1.2],
|
||||||
|
"psTTM": [2.1, 2.2],
|
||||||
|
"pcfNcfTTM": [3.1, 3.2],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001"],
|
||||||
|
"symbol_name": ["PF Bank", "Ping An Bank"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_batch(symbols, start, end, adjust="qfq"):
|
||||||
|
assert symbols == ["sh600000", "sz000001"]
|
||||||
|
assert adjust == "qfq"
|
||||||
|
yield "sh600000", batch_frame
|
||||||
|
yield "sz000001", None
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "download_daily_batch", fake_batch)
|
||||||
|
|
||||||
|
stale_file = tmp_path / "toy" / "month=2024-01" / "stale.pq"
|
||||||
|
stale_file.parent.mkdir(parents=True)
|
||||||
|
batch_frame.iloc[[0]][DATA_COLUMNS[2:]].to_parquet(stale_file, index=False)
|
||||||
|
|
||||||
|
stats = download_universe(
|
||||||
|
universe="toy",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-02-01",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
chunk_size=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
dataset_path = tmp_path / "toy"
|
||||||
|
written = pd.read_parquet(dataset_path).sort_values(["date", "symbol_id"]).reset_index(drop=True)
|
||||||
|
assert stats == {
|
||||||
|
"dataset_path": str(dataset_path),
|
||||||
|
"n_symbols": 1,
|
||||||
|
"n_requested": 2,
|
||||||
|
"n_rows": 2,
|
||||||
|
"date_min": "2024-01-02",
|
||||||
|
"date_max": "2024-02-01",
|
||||||
|
}
|
||||||
|
assert (dataset_path / "month=2024-01").exists()
|
||||||
|
assert (dataset_path / "month=2024-02").exists()
|
||||||
|
assert not stale_file.exists()
|
||||||
|
assert written[DATA_COLUMNS].columns.tolist() == DATA_COLUMNS
|
||||||
|
assert written["symbol_name"].tolist() == ["PF Bank", "PF Bank"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_universe_raises_when_all_daily_symbols_empty(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"symbol_name": ["PF Bank"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"download_daily_batch",
|
||||||
|
lambda symbols, start, end, adjust="qfq": iter([("sh600000", None)]),
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="No data downloaded"):
|
||||||
|
download_universe(
|
||||||
|
universe="toy",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-01-02",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_universe_progress_branch_at_100_symbols(tmp_path, monkeypatch):
|
||||||
|
symbols = [f"sh6{i:05d}" for i in range(100)]
|
||||||
|
batch_frame = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"open": [10.0],
|
||||||
|
"high": [11.0],
|
||||||
|
"low": [9.0],
|
||||||
|
"close": [10.5],
|
||||||
|
"preclose": [10.0],
|
||||||
|
"volume": [1000.0],
|
||||||
|
"amount": [10500.0],
|
||||||
|
"vwap": [10.5],
|
||||||
|
"turn": [1.0],
|
||||||
|
"pctChg": [5.0],
|
||||||
|
"tradestatus": [1],
|
||||||
|
"isST": [0],
|
||||||
|
"peTTM": [8.0],
|
||||||
|
"pbMRQ": [1.1],
|
||||||
|
"psTTM": [2.1],
|
||||||
|
"pcfNcfTTM": [3.1],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": symbols,
|
||||||
|
"symbol_name": symbols,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_batch(requested_symbols, start, end, adjust="qfq"):
|
||||||
|
assert requested_symbols == symbols
|
||||||
|
for symbol in requested_symbols:
|
||||||
|
yield symbol, batch_frame.copy()
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "download_daily_batch", fake_batch)
|
||||||
|
|
||||||
|
stats = download_universe(
|
||||||
|
universe="toy100",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-01-02",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
chunk_size=200,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert stats["n_symbols"] == 100
|
||||||
|
assert stats["n_rows"] == 100
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
"""Tests for minute-derived daily feature plugins."""
|
||||||
|
|
||||||
|
import textwrap
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pipeline.features.compute import compute_feature, validate_feature_frame
|
||||||
|
from pipeline.features.compute import read_feature_frames
|
||||||
|
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
|
||||||
|
from pipeline.features.registry import (
|
||||||
|
available_features,
|
||||||
|
get_feature,
|
||||||
|
load_feature_module,
|
||||||
|
)
|
||||||
|
from pipeline.derived.compute import compute_derived
|
||||||
|
|
||||||
|
|
||||||
|
def _minute_bars() -> pd.DataFrame:
|
||||||
|
return pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001"],
|
||||||
|
"symbol_name": ["PF Bank", "PF Bank", "Ping An"],
|
||||||
|
"datetime": pd.to_datetime([
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
"2024-01-02 09:40:00",
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
]),
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
|
||||||
|
"time": ["09:35:00", "09:40:00", "09:35:00"],
|
||||||
|
"frequency": ["5m", "5m", "5m"],
|
||||||
|
"open": [10.0, 10.5, 20.0],
|
||||||
|
"high": [11.0, 12.0, 21.0],
|
||||||
|
"low": [9.0, 10.0, 19.0],
|
||||||
|
"close": [10.5, 11.0, 20.5],
|
||||||
|
"volume": [100.0, 300.0, 200.0],
|
||||||
|
"amount": [1000.0, 3300.0, 4100.0],
|
||||||
|
"vwap": [10.0, 11.0, 20.5],
|
||||||
|
"adjustflag": ["3", "3", "3"],
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def test_built_in_minute_daily_summary():
|
||||||
|
daily = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
|
||||||
|
"close": [11.0, 20.5, 12.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
result = compute_feature(
|
||||||
|
minute=_minute_bars(),
|
||||||
|
daily=daily,
|
||||||
|
feature_type="minute_daily_summary",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert "minute_daily_summary" in available_features()
|
||||||
|
row = result[
|
||||||
|
(result["symbol_id"] == "sh600000")
|
||||||
|
& (result["date"] == pd.Timestamp("2024-01-02"))
|
||||||
|
].iloc[0]
|
||||||
|
assert row["minute_bar_count"] == 2
|
||||||
|
assert np.isclose(row["minute_intraday_return"], 11.0 / 10.0 - 1.0)
|
||||||
|
assert np.isclose(row["minute_intraday_range"], 12.0 / 9.0 - 1.0)
|
||||||
|
assert np.isclose(row["minute_vwap"], 4300.0 / 400.0)
|
||||||
|
assert np.isclose(row["minute_vwap_deviation"], (4300.0 / 400.0) / 11.0 - 1.0)
|
||||||
|
|
||||||
|
missing = result[
|
||||||
|
(result["symbol_id"] == "sh600000")
|
||||||
|
& (result["date"] == pd.Timestamp("2024-01-03"))
|
||||||
|
].iloc[0]
|
||||||
|
assert pd.isna(missing["minute_vwap"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_minute_daily_summary_feature_preserves_legacy_positional_compute():
|
||||||
|
direct = MinuteDailySummaryFeature().compute(_minute_bars())
|
||||||
|
via_registry = get_feature("minute_daily_summary").compute(_minute_bars())
|
||||||
|
|
||||||
|
assert "minute_vwap" in direct.columns
|
||||||
|
pd.testing.assert_frame_equal(direct, via_registry)
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_feature_compute_matches_canonical_derived_compute():
|
||||||
|
daily = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
|
||||||
|
"close": [11.0, 20.5, 12.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
legacy_feature = compute_feature(
|
||||||
|
minute=_minute_bars(),
|
||||||
|
daily=daily,
|
||||||
|
feature_type="minute_daily_summary",
|
||||||
|
)
|
||||||
|
canonical_derived = compute_derived(
|
||||||
|
"minute_daily_summary",
|
||||||
|
daily=daily,
|
||||||
|
minute=_minute_bars(),
|
||||||
|
)
|
||||||
|
|
||||||
|
pd.testing.assert_frame_equal(legacy_feature, canonical_derived)
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_feature_frames_delegates_to_derived_validation(tmp_path):
|
||||||
|
feature = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": ["2024-01-02 15:00:00"],
|
||||||
|
"toy_feature": [1.5],
|
||||||
|
})
|
||||||
|
feature_path = tmp_path / "feature.pq"
|
||||||
|
feature.to_parquet(feature_path, index=False)
|
||||||
|
|
||||||
|
[result] = read_feature_frames([feature_path])
|
||||||
|
|
||||||
|
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
assert result["toy_feature"].tolist() == [1.5]
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_external_feature_module_and_filter_params(tmp_path):
|
||||||
|
module_path = tmp_path / "external_feature.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.features.base import BaseFeature
|
||||||
|
from pipeline.features.registry import register_feature
|
||||||
|
|
||||||
|
@register_feature
|
||||||
|
class ExternalVolumeFeature(BaseFeature):
|
||||||
|
name = "external_volume_feature"
|
||||||
|
|
||||||
|
def __init__(self, scale: float = 1.0):
|
||||||
|
self.scale = scale
|
||||||
|
|
||||||
|
def compute(self, minute: pd.DataFrame, daily=None) -> pd.DataFrame:
|
||||||
|
out = (
|
||||||
|
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
|
||||||
|
.sum()
|
||||||
|
.rename(columns={"volume": "scaled_volume"})
|
||||||
|
)
|
||||||
|
out["scaled_volume"] *= self.scale
|
||||||
|
return out
|
||||||
|
'''))
|
||||||
|
|
||||||
|
load_feature_module(str(module_path))
|
||||||
|
assert "external_volume_feature" in available_features()
|
||||||
|
|
||||||
|
instance = get_feature("external_volume_feature", scale=2.0, ignored=99)
|
||||||
|
assert instance.scale == 2.0
|
||||||
|
assert not hasattr(instance, "ignored")
|
||||||
|
|
||||||
|
result = compute_feature(
|
||||||
|
minute=_minute_bars(),
|
||||||
|
feature_type="external_volume_feature",
|
||||||
|
scale=2.0,
|
||||||
|
ignored=99,
|
||||||
|
)
|
||||||
|
row = result[result["symbol_id"] == "sh600000"].iloc[0]
|
||||||
|
assert np.isclose(row["scaled_volume"], 800.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_missing_keys():
|
||||||
|
with pytest.raises(ValueError, match="missing required"):
|
||||||
|
validate_feature_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_duplicate_keys_after_date_normalization():
|
||||||
|
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
|
||||||
|
validate_feature_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": ["2024-01-02", pd.Timestamp("2024-01-02")],
|
||||||
|
"x": [1.0, 2.0],
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_duplicate_columns():
|
||||||
|
bad = pd.DataFrame(
|
||||||
|
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
|
||||||
|
columns=["symbol_id", "date", "dup", "dup"],
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError, match="duplicate columns"):
|
||||||
|
validate_feature_frame(bad)
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_non_numeric_feature_columns():
|
||||||
|
with pytest.raises(ValueError, match="numeric"):
|
||||||
|
validate_feature_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"bad": ["not numeric"],
|
||||||
|
}))
|
||||||
@@ -0,0 +1,650 @@
|
|||||||
|
"""Tests for the JoinQuant comparison plugin (network-free)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from cli import cli
|
||||||
|
from pipeline.common.schema import FILL_COLUMNS, PNL_COLUMNS, POSITION_COLUMNS
|
||||||
|
from plugins.joinquant.browser import (
|
||||||
|
default_browser_config,
|
||||||
|
load_env_file,
|
||||||
|
resolve_template,
|
||||||
|
write_browser_config_template,
|
||||||
|
)
|
||||||
|
from plugins.joinquant.export_targets import export_targets
|
||||||
|
from plugins.joinquant.ingest import (
|
||||||
|
ingest_joinquant_outputs,
|
||||||
|
normalize_fills_csv,
|
||||||
|
)
|
||||||
|
from plugins.joinquant.reconcile import reconcile_joinquant
|
||||||
|
from plugins.joinquant.schema import (
|
||||||
|
JOINQUANT_FILL_COLUMNS,
|
||||||
|
JOINQUANT_PNL_COLUMNS,
|
||||||
|
JOINQUANT_POSITION_COLUMNS,
|
||||||
|
JOINQUANT_TARGET_COLUMNS,
|
||||||
|
RECONCILE_COLUMNS,
|
||||||
|
)
|
||||||
|
from plugins.joinquant.smoke import build_fixed_share_positions
|
||||||
|
from plugins.joinquant.symbols import from_joinquant_symbol, to_joinquant_symbol
|
||||||
|
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
|
||||||
|
|
||||||
|
|
||||||
|
def _positions(
|
||||||
|
*,
|
||||||
|
symbol: str = "sh600000",
|
||||||
|
date: str = "2026-07-01",
|
||||||
|
shares: int = 1000,
|
||||||
|
price: float = 10.0,
|
||||||
|
portfolio_name: str = "run1",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
target_value = float(shares * price)
|
||||||
|
weight = target_value / 1_000_000.0
|
||||||
|
return pd.DataFrame([{
|
||||||
|
"symbol_id": symbol,
|
||||||
|
"date": pd.Timestamp(date),
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"target_weight": weight,
|
||||||
|
"target_value": target_value,
|
||||||
|
"target_shares": float(shares) + 0.25,
|
||||||
|
"position_shares": shares,
|
||||||
|
"position_value": target_value,
|
||||||
|
"price": price,
|
||||||
|
}], columns=POSITION_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def _our_fills(
|
||||||
|
*,
|
||||||
|
symbol: str = "sh600000",
|
||||||
|
date: str = "2026-07-01",
|
||||||
|
shares: int = 1000,
|
||||||
|
price: float = 10.0,
|
||||||
|
cost: float = 5.0,
|
||||||
|
portfolio_name: str = "run1",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
fills = pd.DataFrame([{
|
||||||
|
"symbol_id": symbol,
|
||||||
|
"date": pd.Timestamp(date),
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"prev_shares": 0,
|
||||||
|
"target_shares": shares,
|
||||||
|
"traded_shares": shares,
|
||||||
|
"realized_shares": shares,
|
||||||
|
"blocked": 0,
|
||||||
|
"trade_cost": cost,
|
||||||
|
"trade_price": price,
|
||||||
|
}])
|
||||||
|
return fills
|
||||||
|
|
||||||
|
|
||||||
|
def _our_pnl(
|
||||||
|
*,
|
||||||
|
date: str = "2026-07-01",
|
||||||
|
pnl: float = 100.0,
|
||||||
|
cost: float = 5.0,
|
||||||
|
portfolio_name: str = "run1",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
return pd.DataFrame([{
|
||||||
|
"date": pd.Timestamp(date),
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"gross_exposure": 10_000.0,
|
||||||
|
"net_exposure": 10_000.0,
|
||||||
|
"pnl": pnl,
|
||||||
|
"cost": cost,
|
||||||
|
"turnover": 1.0,
|
||||||
|
"n_positions": 1,
|
||||||
|
}], columns=PNL_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def _jq_fills(
|
||||||
|
*,
|
||||||
|
symbol: str = "sh600000",
|
||||||
|
date: str = "2026-07-01",
|
||||||
|
shares: int = 1000,
|
||||||
|
price: float = 10.0,
|
||||||
|
cost: float = 5.0,
|
||||||
|
portfolio_name: str = "run1",
|
||||||
|
raw_status: str = "filled",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
return pd.DataFrame([{
|
||||||
|
"date": date,
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"symbol_id": symbol,
|
||||||
|
"jq_symbol": to_joinquant_symbol(symbol),
|
||||||
|
"order_id": "ord-1",
|
||||||
|
"side": "buy" if shares >= 0 else "sell",
|
||||||
|
"requested_shares": shares,
|
||||||
|
"filled_shares": shares,
|
||||||
|
"fill_price": price,
|
||||||
|
"trade_value": abs(shares * price),
|
||||||
|
"trade_cost": cost,
|
||||||
|
"blocked": 0,
|
||||||
|
"raw_status": raw_status,
|
||||||
|
}], columns=JOINQUANT_FILL_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def _jq_positions(
|
||||||
|
*,
|
||||||
|
symbol: str = "sh600000",
|
||||||
|
date: str = "2026-07-01",
|
||||||
|
shares: int = 1000,
|
||||||
|
price: float = 10.0,
|
||||||
|
portfolio_name: str = "run1",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
return pd.DataFrame([{
|
||||||
|
"date": date,
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"symbol_id": symbol,
|
||||||
|
"jq_symbol": to_joinquant_symbol(symbol),
|
||||||
|
"position_shares": shares,
|
||||||
|
"position_value": shares * price,
|
||||||
|
"cash": 990_000.0,
|
||||||
|
"total_value": 1_000_000.0,
|
||||||
|
}], columns=JOINQUANT_POSITION_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def _jq_pnl(
|
||||||
|
*,
|
||||||
|
date: str = "2026-07-01",
|
||||||
|
pnl: float = 100.0,
|
||||||
|
cost: float = 5.0,
|
||||||
|
portfolio_name: str = "run1",
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
return pd.DataFrame([{
|
||||||
|
"date": date,
|
||||||
|
"portfolio_name": portfolio_name,
|
||||||
|
"gross_exposure": 10_000.0,
|
||||||
|
"net_exposure": 10_000.0,
|
||||||
|
"cash": 990_000.0,
|
||||||
|
"total_value": 1_000_000.0,
|
||||||
|
"pnl": pnl,
|
||||||
|
"cost": cost,
|
||||||
|
"turnover": 1.0,
|
||||||
|
}], columns=JOINQUANT_PNL_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_parquets(tmp_path: Path, frames: dict[str, pd.DataFrame]) -> dict[str, Path]:
|
||||||
|
paths = {}
|
||||||
|
for name, frame in frames.items():
|
||||||
|
path = tmp_path / f"{name}.pq"
|
||||||
|
frame.to_parquet(path, index=False)
|
||||||
|
paths[name] = path
|
||||||
|
return paths
|
||||||
|
|
||||||
|
|
||||||
|
def _export_targets_for(tmp_path: Path, positions: pd.DataFrame) -> tuple[Path, Path]:
|
||||||
|
positions_path = tmp_path / "positions.pq"
|
||||||
|
positions.to_parquet(positions_path, index=False)
|
||||||
|
targets_root = tmp_path / "targets"
|
||||||
|
export_targets(
|
||||||
|
positions_path,
|
||||||
|
portfolio_name="run1",
|
||||||
|
out_dir=targets_root,
|
||||||
|
mode="target_shares",
|
||||||
|
)
|
||||||
|
return positions_path, targets_root / "run1"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("internal", "joinquant"),
|
||||||
|
[
|
||||||
|
("sh600000", "600000.XSHG"),
|
||||||
|
("sh688001", "688001.XSHG"),
|
||||||
|
("sz000001", "000001.XSHE"),
|
||||||
|
("sz001001", "001001.XSHE"),
|
||||||
|
("sz002594", "002594.XSHE"),
|
||||||
|
("sz300001", "300001.XSHE"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_symbol_mapping_both_directions(internal, joinquant):
|
||||||
|
assert to_joinquant_symbol(internal) == joinquant
|
||||||
|
assert from_joinquant_symbol(joinquant) == internal
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bad", ["600000", "bj830000", "sh000001", "sz600000", "abc"])
|
||||||
|
def test_symbol_mapping_rejects_invalid_symbols(bad):
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
to_joinquant_symbol(bad)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bad", ["600000", "600000.XSHE", "000001.XSHG", "abc.XSHG"])
|
||||||
|
def test_reverse_symbol_mapping_rejects_invalid_symbols(bad):
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
from_joinquant_symbol(bad)
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_targets_schema_snapshot_hash_and_no_overwrite(tmp_path):
|
||||||
|
positions_path = tmp_path / "positions.pq"
|
||||||
|
_positions().to_parquet(positions_path, index=False)
|
||||||
|
|
||||||
|
snapshots = export_targets(
|
||||||
|
positions_path,
|
||||||
|
portfolio_name="run1",
|
||||||
|
out_dir=tmp_path / "targets",
|
||||||
|
mode="target_shares",
|
||||||
|
)
|
||||||
|
|
||||||
|
csv_path = tmp_path / "targets" / "run1" / "20260701.csv"
|
||||||
|
parquet_path = tmp_path / "targets" / "run1" / "20260701.parquet"
|
||||||
|
snapshot_path = tmp_path / "snapshots" / "run1" / "20260701.json"
|
||||||
|
assert csv_path.exists()
|
||||||
|
assert parquet_path.exists()
|
||||||
|
assert snapshot_path.exists()
|
||||||
|
|
||||||
|
target = pd.read_csv(csv_path)
|
||||||
|
assert list(target.columns) == JOINQUANT_TARGET_COLUMNS
|
||||||
|
assert int(target.loc[0, "target_shares"]) == 1000
|
||||||
|
assert float(target.loc[0, "target_value"]) == 10_000.0
|
||||||
|
assert target.loc[0, "export_mode"] == "target_shares"
|
||||||
|
|
||||||
|
snapshot = json.loads(snapshot_path.read_text())
|
||||||
|
actual_hash = hashlib.sha256(csv_path.read_bytes()).hexdigest()
|
||||||
|
assert snapshots[0]["file_sha256"] == actual_hash
|
||||||
|
assert snapshot["file_sha256"] == actual_hash
|
||||||
|
assert snapshot["n_symbols"] == 1
|
||||||
|
|
||||||
|
with pytest.raises(FileExistsError):
|
||||||
|
export_targets(
|
||||||
|
positions_path,
|
||||||
|
portfolio_name="run1",
|
||||||
|
out_dir=tmp_path / "targets",
|
||||||
|
mode="target_shares",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_targets_target_value_mode_from_position_columns(tmp_path):
|
||||||
|
positions_path = tmp_path / "positions.pq"
|
||||||
|
_positions(shares=250, price=20.0).to_parquet(positions_path, index=False)
|
||||||
|
|
||||||
|
export_targets(
|
||||||
|
positions_path,
|
||||||
|
portfolio_name="run1",
|
||||||
|
out_dir=tmp_path / "targets_value",
|
||||||
|
mode="target_value",
|
||||||
|
)
|
||||||
|
|
||||||
|
target = pd.read_parquet(tmp_path / "targets_value" / "run1" / "20260701.parquet")
|
||||||
|
assert list(target.columns) == JOINQUANT_TARGET_COLUMNS
|
||||||
|
assert target.loc[0, "export_mode"] == "target_value"
|
||||||
|
assert target.loc[0, "target_value"] == 5_000.0
|
||||||
|
assert target.loc[0, "target_shares"] == 250
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_targets_can_shift_to_next_execution_session(tmp_path):
|
||||||
|
positions_path = tmp_path / "positions.pq"
|
||||||
|
_positions(date="2024-01-09").to_parquet(positions_path, index=False)
|
||||||
|
calendar_path = tmp_path / "daily.pq"
|
||||||
|
pd.DataFrame({
|
||||||
|
"date": pd.to_datetime(["2024-01-09", "2024-01-10", "2024-01-11"]),
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sh600000"],
|
||||||
|
}).to_parquet(calendar_path, index=False)
|
||||||
|
|
||||||
|
snapshots = export_targets(
|
||||||
|
positions_path,
|
||||||
|
portfolio_name="run1",
|
||||||
|
out_dir=tmp_path / "targets_shifted",
|
||||||
|
mode="target_shares",
|
||||||
|
start_date="2024-01-10",
|
||||||
|
end_date="2024-01-10",
|
||||||
|
execution_calendar_path=calendar_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert len(snapshots) == 1
|
||||||
|
assert snapshots[0]["date"] == "2024-01-10"
|
||||||
|
assert (tmp_path / "targets_shifted" / "run1" / "20240110.csv").exists()
|
||||||
|
target = pd.read_csv(tmp_path / "targets_shifted" / "run1" / "20240110.csv")
|
||||||
|
assert target.loc[0, "date"] == "2024-01-10"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ingest_permissive_csv_column_mapping_and_output_schemas(tmp_path):
|
||||||
|
fills_csv = tmp_path / "jq_fills.csv"
|
||||||
|
positions_csv = tmp_path / "jq_positions.csv"
|
||||||
|
pnl_csv = tmp_path / "jq_pnl.csv"
|
||||||
|
pd.DataFrame([{
|
||||||
|
"Trade Date": "2026-07-01 09:31:00",
|
||||||
|
"Security": "600000.XSHG",
|
||||||
|
"Direction": "buy",
|
||||||
|
"Order Amount": 1000,
|
||||||
|
"Filled Amount": 1000,
|
||||||
|
"Price": 10.0,
|
||||||
|
"Status": "filled",
|
||||||
|
}]).to_csv(fills_csv, index=False)
|
||||||
|
pd.DataFrame([{
|
||||||
|
"Date": "2026-07-01",
|
||||||
|
"Security": "600000.XSHG",
|
||||||
|
"Shares": 1000,
|
||||||
|
"Market Value": 10_000.0,
|
||||||
|
"Cash": 990_000.0,
|
||||||
|
"Portfolio Value": 1_000_000.0,
|
||||||
|
}]).to_csv(positions_csv, index=False)
|
||||||
|
pd.DataFrame([{
|
||||||
|
"Date": "2026-07-01",
|
||||||
|
"Portfolio Value": 1_000_000.0,
|
||||||
|
"Daily PnL": 100.0,
|
||||||
|
"Turnover": 1.0,
|
||||||
|
}]).to_csv(pnl_csv, index=False)
|
||||||
|
|
||||||
|
fills = normalize_fills_csv(fills_csv, "run1")
|
||||||
|
assert list(fills.columns) == JOINQUANT_FILL_COLUMNS
|
||||||
|
assert fills.loc[0, "symbol_id"] == "sh600000"
|
||||||
|
assert fills.loc[0, "jq_symbol"] == "600000.XSHG"
|
||||||
|
assert fills.loc[0, "trade_cost"] == 0.0
|
||||||
|
assert fills.loc[0, "blocked"] == 0
|
||||||
|
|
||||||
|
paths = ingest_joinquant_outputs(
|
||||||
|
portfolio_name="run1",
|
||||||
|
fills_csv=fills_csv,
|
||||||
|
positions_csv=positions_csv,
|
||||||
|
pnl_csv=pnl_csv,
|
||||||
|
out_dir=tmp_path / "ingested",
|
||||||
|
)
|
||||||
|
assert list(pd.read_parquet(paths["fills"]).columns) == JOINQUANT_FILL_COLUMNS
|
||||||
|
assert list(pd.read_parquet(paths["positions"]).columns) == JOINQUANT_POSITION_COLUMNS
|
||||||
|
assert list(pd.read_parquet(paths["pnl"]).columns) == JOINQUANT_PNL_COLUMNS
|
||||||
|
|
||||||
|
|
||||||
|
def _run_reconcile_case(
|
||||||
|
tmp_path: Path,
|
||||||
|
*,
|
||||||
|
positions: pd.DataFrame | None = None,
|
||||||
|
our_fills: pd.DataFrame | None = None,
|
||||||
|
jq_fills: pd.DataFrame | None = None,
|
||||||
|
jq_positions: pd.DataFrame | None = None,
|
||||||
|
our_pnl: pd.DataFrame | None = None,
|
||||||
|
jq_pnl: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
positions = _positions() if positions is None else positions
|
||||||
|
_, targets_dir = _export_targets_for(tmp_path, positions)
|
||||||
|
paths = _write_parquets(tmp_path, {
|
||||||
|
"our_fills": _our_fills() if our_fills is None else our_fills,
|
||||||
|
"our_positions": positions,
|
||||||
|
"our_pnl": _our_pnl() if our_pnl is None else our_pnl,
|
||||||
|
"jq_fills": _jq_fills() if jq_fills is None else jq_fills,
|
||||||
|
"jq_positions": _jq_positions() if jq_positions is None else jq_positions,
|
||||||
|
"jq_pnl": _jq_pnl() if jq_pnl is None else jq_pnl,
|
||||||
|
})
|
||||||
|
out_paths = reconcile_joinquant(
|
||||||
|
portfolio_name="run1",
|
||||||
|
targets_dir=targets_dir,
|
||||||
|
our_fills_path=paths["our_fills"],
|
||||||
|
our_positions_path=paths["our_positions"],
|
||||||
|
our_pnl_path=paths["our_pnl"],
|
||||||
|
jq_fills_path=paths["jq_fills"],
|
||||||
|
jq_positions_path=paths["jq_positions"],
|
||||||
|
jq_pnl_path=paths["jq_pnl"],
|
||||||
|
out_dir=tmp_path / "reconcile",
|
||||||
|
)
|
||||||
|
report = pd.read_parquet(out_paths["daily_reconcile"])
|
||||||
|
assert list(report.columns) == RECONCILE_COLUMNS
|
||||||
|
assert out_paths["summary_md"].exists()
|
||||||
|
assert out_paths["summary_csv"].exists()
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_exact_match(tmp_path):
|
||||||
|
report = _run_reconcile_case(tmp_path)
|
||||||
|
assert report.loc[0, "diff_reason"] == "MATCH"
|
||||||
|
assert report.loc[0, "filled_share_diff"] == 0
|
||||||
|
assert report.loc[0, "position_share_diff"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_price_mismatch(tmp_path):
|
||||||
|
report = _run_reconcile_case(tmp_path, jq_fills=_jq_fills(price=10.5))
|
||||||
|
assert report.loc[0, "diff_reason"] == "PRICE_MISMATCH"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_cost_mismatch(tmp_path):
|
||||||
|
report = _run_reconcile_case(
|
||||||
|
tmp_path,
|
||||||
|
jq_fills=_jq_fills(cost=8.0),
|
||||||
|
jq_pnl=_jq_pnl(cost=8.0),
|
||||||
|
)
|
||||||
|
assert report.loc[0, "diff_reason"] == "COST_MODEL"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_missing_symbol_in_joinquant(tmp_path):
|
||||||
|
empty_jq_fills = pd.DataFrame(columns=JOINQUANT_FILL_COLUMNS)
|
||||||
|
empty_jq_positions = pd.DataFrame(columns=JOINQUANT_POSITION_COLUMNS)
|
||||||
|
report = _run_reconcile_case(
|
||||||
|
tmp_path,
|
||||||
|
jq_fills=empty_jq_fills,
|
||||||
|
jq_positions=empty_jq_positions,
|
||||||
|
)
|
||||||
|
assert report.loc[0, "diff_reason"] == "MISSING_IN_JOINQUANT"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_short_target_with_long_only_joinquant_output(tmp_path):
|
||||||
|
positions = _positions(shares=-100, price=10.0)
|
||||||
|
our_fills = _our_fills(shares=-100, price=10.0)
|
||||||
|
jq_fills = _jq_fills(shares=0, price=10.0, cost=0.0, raw_status="short clipped")
|
||||||
|
jq_positions = _jq_positions(shares=0, price=10.0)
|
||||||
|
|
||||||
|
report = _run_reconcile_case(
|
||||||
|
tmp_path,
|
||||||
|
positions=positions,
|
||||||
|
our_fills=our_fills,
|
||||||
|
jq_fills=jq_fills,
|
||||||
|
jq_positions=jq_positions,
|
||||||
|
)
|
||||||
|
assert report.loc[0, "diff_reason"] == "SHORT_NOT_SUPPORTED"
|
||||||
|
|
||||||
|
|
||||||
|
def test_joinquant_cli_smoke_export_ingest_reconcile_and_wrapper(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
positions_path = tmp_path / "positions.pq"
|
||||||
|
_positions().to_parquet(positions_path, index=False)
|
||||||
|
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"joinquant", "export-targets",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--portfolio-name", "run1",
|
||||||
|
"--mode", "target_shares",
|
||||||
|
"--out-dir", str(tmp_path / "targets"),
|
||||||
|
])
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert "Exported JoinQuant targets" in result.output
|
||||||
|
|
||||||
|
fills_csv = tmp_path / "jq_fills.csv"
|
||||||
|
positions_csv = tmp_path / "jq_positions.csv"
|
||||||
|
pnl_csv = tmp_path / "jq_pnl.csv"
|
||||||
|
_jq_fills().to_csv(fills_csv, index=False)
|
||||||
|
_jq_positions().to_csv(positions_csv, index=False)
|
||||||
|
_jq_pnl().to_csv(pnl_csv, index=False)
|
||||||
|
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"joinquant", "ingest",
|
||||||
|
"--portfolio-name", "run1",
|
||||||
|
"--fills-csv", str(fills_csv),
|
||||||
|
"--positions-csv", str(positions_csv),
|
||||||
|
"--pnl-csv", str(pnl_csv),
|
||||||
|
"--out-dir", str(tmp_path / "ingested"),
|
||||||
|
])
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert "Saved JoinQuant fills" in result.output
|
||||||
|
|
||||||
|
paths = _write_parquets(tmp_path, {
|
||||||
|
"our_fills": _our_fills(),
|
||||||
|
"our_pnl": _our_pnl(),
|
||||||
|
})
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"joinquant", "reconcile",
|
||||||
|
"--portfolio-name", "run1",
|
||||||
|
"--targets-dir", str(tmp_path / "targets" / "run1"),
|
||||||
|
"--our-fills-path", str(paths["our_fills"]),
|
||||||
|
"--our-positions-path", str(positions_path),
|
||||||
|
"--our-pnl-path", str(paths["our_pnl"]),
|
||||||
|
"--jq-fills-path", str(tmp_path / "ingested" / "run1" / "fills.pq"),
|
||||||
|
"--jq-positions-path", str(tmp_path / "ingested" / "run1" / "positions.pq"),
|
||||||
|
"--jq-pnl-path", str(tmp_path / "ingested" / "run1" / "pnl.pq"),
|
||||||
|
"--out-dir", str(tmp_path / "reconcile"),
|
||||||
|
])
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert "Saved reconciliation parquet" in result.output
|
||||||
|
|
||||||
|
wrapper_path = tmp_path / "wrapper_strategy_run1.py"
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"joinquant", "write-wrapper",
|
||||||
|
"--portfolio-name", "run1",
|
||||||
|
"--mode", "target_shares",
|
||||||
|
"--out-path", str(wrapper_path),
|
||||||
|
])
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert "Saved JoinQuant wrapper strategy" in result.output
|
||||||
|
text = wrapper_path.read_text()
|
||||||
|
assert 'PORTFOLIO_NAME = "run1"' in text
|
||||||
|
assert 'TARGET_MODE = "target_shares"' in text
|
||||||
|
assert "ALLOW_SHORT = False" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_wrapper_strategy_generation_smoke(tmp_path):
|
||||||
|
path = write_wrapper_strategy(
|
||||||
|
portfolio_name="run2",
|
||||||
|
mode="target_value",
|
||||||
|
out_path=tmp_path / "wrapper.py",
|
||||||
|
)
|
||||||
|
text = path.read_text()
|
||||||
|
assert 'PORTFOLIO_NAME = "run2"' in text
|
||||||
|
assert 'TARGET_MODE = "target_value"' in text
|
||||||
|
assert "order_target_value" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_wrapper_strategy_can_embed_target_csvs(tmp_path):
|
||||||
|
targets_dir = tmp_path / "targets"
|
||||||
|
targets_dir.mkdir()
|
||||||
|
(targets_dir / "20260701.csv").write_text(
|
||||||
|
"date,portfolio_name,symbol_id,jq_symbol,target_shares,target_value,export_mode\n"
|
||||||
|
"2026-07-01,run1,sh600000,600000.XSHG,1000,10000,target_shares\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
path = write_wrapper_strategy(
|
||||||
|
portfolio_name="run1",
|
||||||
|
mode="target_shares",
|
||||||
|
out_path=tmp_path / "wrapper_embedded.py",
|
||||||
|
embedded_targets_dir=targets_dir,
|
||||||
|
)
|
||||||
|
|
||||||
|
text = path.read_text()
|
||||||
|
assert '"20260701.csv"' in text
|
||||||
|
assert "600000.XSHG" in text
|
||||||
|
assert "if file_name in _EMBEDDED_TARGETS" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_fixed_share_positions_excludes_final_executionless_date():
|
||||||
|
data = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-09", "2024-01-10", "2024-01-11"]),
|
||||||
|
"close": [10.0, 10.5, 11.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
positions = build_fixed_share_positions(
|
||||||
|
data,
|
||||||
|
trade_symbol="sh600000",
|
||||||
|
portfolio_name="run1",
|
||||||
|
shares=1000,
|
||||||
|
booksize=1_000_000.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(positions.columns) == POSITION_COLUMNS
|
||||||
|
assert positions["date"].dt.strftime("%Y-%m-%d").tolist() == [
|
||||||
|
"2024-01-09",
|
||||||
|
"2024-01-10",
|
||||||
|
]
|
||||||
|
assert positions["position_shares"].tolist() == [1000, 1000]
|
||||||
|
assert positions["target_value"].tolist() == [10_000.0, 10_500.0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_browser_config_template_and_placeholder_resolution(tmp_path):
|
||||||
|
config_path = write_browser_config_template(
|
||||||
|
tmp_path / "browser_config.json",
|
||||||
|
strategy_url="https://www.joinquant.com/example",
|
||||||
|
)
|
||||||
|
config = json.loads(config_path.read_text())
|
||||||
|
assert config["strategy_url"] == "https://www.joinquant.com/example"
|
||||||
|
assert config["actions"][0]["type"] == "goto"
|
||||||
|
|
||||||
|
context = {
|
||||||
|
"wrapper_path": "/tmp/wrapper.py",
|
||||||
|
"target_csvs": ["/tmp/20240110.csv", "/tmp/20240111.csv"],
|
||||||
|
"expected_joinquant_csvs": {"fills": "/tmp/jq_fills.csv"},
|
||||||
|
}
|
||||||
|
assert resolve_template("{wrapper_path}", context) == "/tmp/wrapper.py"
|
||||||
|
assert resolve_template("{target_csvs}", context) == [
|
||||||
|
"/tmp/20240110.csv",
|
||||||
|
"/tmp/20240111.csv",
|
||||||
|
]
|
||||||
|
assert resolve_template("save:{expected_joinquant_csvs.fills}", context) == "save:/tmp/jq_fills.csv"
|
||||||
|
script = "(arg) => { return new Event('change', {bubbles: true}); }"
|
||||||
|
assert resolve_template(script, context) == script
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_env_file_handles_quotes_without_shell_sourcing(tmp_path):
|
||||||
|
env_path = tmp_path / "joinquant.env"
|
||||||
|
env_path.write_text(
|
||||||
|
"JOINQUANT_USERNAME=alice\n"
|
||||||
|
"JOINQUANT_PASSWORD=\"secret\"\n"
|
||||||
|
"JOINQUANT_STRATEGY_URL='https://example.test/path?x=1&y=2'\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
env = load_env_file(env_path)
|
||||||
|
assert env["JOINQUANT_USERNAME"] == "alice"
|
||||||
|
assert env["JOINQUANT_PASSWORD"] == "secret"
|
||||||
|
assert env["JOINQUANT_STRATEGY_URL"] == "https://example.test/path?x=1&y=2"
|
||||||
|
|
||||||
|
|
||||||
|
def test_sim_trade_browser_config_template(tmp_path):
|
||||||
|
config_path = write_browser_config_template(
|
||||||
|
tmp_path / "sim_config.json",
|
||||||
|
strategy_url="https://www.joinquant.com/sim",
|
||||||
|
flow="sim-trade",
|
||||||
|
)
|
||||||
|
config = json.loads(config_path.read_text())
|
||||||
|
|
||||||
|
assert config["flow"] == "sim-trade"
|
||||||
|
selectors = " ".join(
|
||||||
|
action.get("selector", "") for action in config["actions"]
|
||||||
|
)
|
||||||
|
assert "模拟盘" in selectors
|
||||||
|
assert "模拟交易" in selectors
|
||||||
|
assert any(
|
||||||
|
action["type"] == "screenshot"
|
||||||
|
and action["path"] == "{run_artifact_dir}/sim_trade_final.png"
|
||||||
|
for action in config["actions"]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_joinquant_cli_browser_config_smoke(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
config_path = tmp_path / "browser_config.json"
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"joinquant",
|
||||||
|
"write-browser-config",
|
||||||
|
"--out-path",
|
||||||
|
str(config_path),
|
||||||
|
"--strategy-url",
|
||||||
|
"https://www.joinquant.com/example",
|
||||||
|
])
|
||||||
|
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert config_path.exists()
|
||||||
|
assert default_browser_config()["actions"]
|
||||||
|
|
||||||
|
sim_config_path = tmp_path / "sim_browser_config.json"
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"joinquant",
|
||||||
|
"write-browser-config",
|
||||||
|
"--out-path",
|
||||||
|
str(sim_config_path),
|
||||||
|
"--strategy-url",
|
||||||
|
"https://www.joinquant.com/sim",
|
||||||
|
"--flow",
|
||||||
|
"sim-trade",
|
||||||
|
])
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
assert json.loads(sim_config_path.read_text())["flow"] == "sim-trade"
|
||||||
@@ -0,0 +1,369 @@
|
|||||||
|
"""Tests for raw Baostock minute bar download plumbing."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import data.downloader as low_level_downloader
|
||||||
|
import pipeline.data.downloader as pipeline_downloader
|
||||||
|
from data.downloader import download_minute_batch
|
||||||
|
from pipeline.common.schema import MINUTE_BAR_COLUMNS
|
||||||
|
from pipeline.data.downloader import download_minute_universe
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResult:
|
||||||
|
def __init__(self, rows, error_code="0", error_msg=""):
|
||||||
|
self.rows = rows
|
||||||
|
self.error_code = error_code
|
||||||
|
self.error_msg = error_msg
|
||||||
|
self._idx = -1
|
||||||
|
|
||||||
|
def next(self):
|
||||||
|
self._idx += 1
|
||||||
|
return self._idx < len(self.rows)
|
||||||
|
|
||||||
|
def get_row_data(self):
|
||||||
|
return self.rows[self._idx]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_maps_and_parses_baostock_rows(monkeypatch):
|
||||||
|
rows = [
|
||||||
|
[
|
||||||
|
"2024-01-02",
|
||||||
|
"20240102093500000",
|
||||||
|
"sh.600000",
|
||||||
|
"10",
|
||||||
|
"11",
|
||||||
|
"9",
|
||||||
|
"10.5",
|
||||||
|
"1000",
|
||||||
|
"10500",
|
||||||
|
"3",
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"2024-01-02",
|
||||||
|
"20240102094000000",
|
||||||
|
"sh.600000",
|
||||||
|
"10.5",
|
||||||
|
"12",
|
||||||
|
"10",
|
||||||
|
"11",
|
||||||
|
"2000",
|
||||||
|
"22000",
|
||||||
|
"3",
|
||||||
|
],
|
||||||
|
]
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake_query(**kwargs):
|
||||||
|
calls.append(kwargs)
|
||||||
|
return _FakeResult(rows)
|
||||||
|
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
fake_query,
|
||||||
|
)
|
||||||
|
|
||||||
|
[(symbol, df)] = list(
|
||||||
|
download_minute_batch(
|
||||||
|
["sh600000"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
frequency=5,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert calls[0]["code"] == "sh.600000"
|
||||||
|
assert calls[0]["frequency"] == "5"
|
||||||
|
assert calls[0]["adjustflag"] == "3"
|
||||||
|
assert df is not None
|
||||||
|
assert df["datetime"].iloc[0] == pd.Timestamp("2024-01-02 09:35:00")
|
||||||
|
assert df["time"].tolist() == ["09:35:00", "09:40:00"]
|
||||||
|
assert (df["frequency"] == "5m").all()
|
||||||
|
assert np.isclose(df["open"].iloc[0], 10.0)
|
||||||
|
assert np.isclose(df["vwap"].iloc[0], 10.5)
|
||||||
|
assert pd.api.types.is_numeric_dtype(df["volume"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_minute_frequency_and_timestamp_parsing_edge_cases():
|
||||||
|
frequency, label = low_level_downloader._normalize_minute_frequency("15m")
|
||||||
|
assert (frequency, label) == ("15", "15m")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="Unsupported minute frequency"):
|
||||||
|
low_level_downloader._normalize_minute_frequency("1m")
|
||||||
|
|
||||||
|
parsed = low_level_downloader._parse_minute_datetime(
|
||||||
|
pd.Series(["2024-01-02", "2024-01-02"]),
|
||||||
|
pd.Series(["0935", "09:40:00"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert parsed.tolist() == [
|
||||||
|
pd.Timestamp("2024-01-02 09:35:00"),
|
||||||
|
pd.Timestamp("2024-01-02 09:40:00"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_empty_result_yields_none(monkeypatch):
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([]),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_non_login_error_and_periodic_relogin(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="1", error_msg="bad symbol"),
|
||||||
|
_FakeResult([]),
|
||||||
|
]
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(
|
||||||
|
download_minute_batch(
|
||||||
|
["sh600000", "sz000001"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
relogin_every=1,
|
||||||
|
)
|
||||||
|
) == [("sh600000", None), ("sz000001", None)]
|
||||||
|
assert login_count == 2
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_second_session_loss_yields_none(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
]
|
||||||
|
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_ignores_relogin_and_final_logout_failures(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="1", error_msg="bad symbol"),
|
||||||
|
_FakeResult([]),
|
||||||
|
]
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
raise RuntimeError("logout failed")
|
||||||
|
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(
|
||||||
|
download_minute_batch(
|
||||||
|
["sh600000", "sz000001"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
relogin_every=1,
|
||||||
|
)
|
||||||
|
) == [("sh600000", None), ("sz000001", None)]
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_rejects_unparsed_timestamps(monkeypatch):
|
||||||
|
bad_rows = [[
|
||||||
|
"2024-01-02",
|
||||||
|
"not-a-time",
|
||||||
|
"sh.600000",
|
||||||
|
"10",
|
||||||
|
"11",
|
||||||
|
"9",
|
||||||
|
"10.5",
|
||||||
|
"1000",
|
||||||
|
"10500",
|
||||||
|
"3",
|
||||||
|
]]
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult(bad_rows),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_universe_writes_frequency_month_partitions(tmp_path, monkeypatch):
|
||||||
|
minute = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000", "sh600000"],
|
||||||
|
"datetime": pd.to_datetime(["2024-01-02 09:35:00", "2024-01-02 09:40:00"]),
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02"]),
|
||||||
|
"time": ["09:35:00", "09:40:00"],
|
||||||
|
"frequency": ["5m", "5m"],
|
||||||
|
"open": [10.0, 10.5],
|
||||||
|
"high": [11.0, 12.0],
|
||||||
|
"low": [9.0, 10.0],
|
||||||
|
"close": [10.5, 11.0],
|
||||||
|
"volume": [1000.0, 2000.0],
|
||||||
|
"amount": [10500.0, 22000.0],
|
||||||
|
"vwap": [10.5, 11.0],
|
||||||
|
"adjustflag": ["3", "3"],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"symbol_name": ["PF Bank"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_batch(symbols, start, end, frequency=5):
|
||||||
|
assert symbols == ["sh600000"]
|
||||||
|
assert frequency == "5"
|
||||||
|
yield "sh600000", minute
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "download_minute_batch", fake_batch)
|
||||||
|
|
||||||
|
preserved = tmp_path / "toy" / "frequency=15m" / "month=2024-01" / "old.pq"
|
||||||
|
preserved.parent.mkdir(parents=True)
|
||||||
|
preserved_minute = minute.copy()
|
||||||
|
preserved_minute["frequency"] = "15m"
|
||||||
|
preserved_minute["symbol_id"] = "sh600000"
|
||||||
|
preserved_minute["symbol_name"] = "PF Bank"
|
||||||
|
preserved_minute[MINUTE_BAR_COLUMNS].to_parquet(preserved, index=False)
|
||||||
|
stale = tmp_path / "toy" / "frequency=5m" / "month=2024-01" / "stale.pq"
|
||||||
|
stale.parent.mkdir(parents=True)
|
||||||
|
preserved_minute.assign(frequency="5m")[MINUTE_BAR_COLUMNS].to_parquet(stale, index=False)
|
||||||
|
|
||||||
|
stats = download_minute_universe(
|
||||||
|
universe="toy",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-01-02",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
chunk_size=1,
|
||||||
|
frequency="5",
|
||||||
|
)
|
||||||
|
|
||||||
|
dataset_path = Path(stats["dataset_path"])
|
||||||
|
assert (dataset_path / "frequency=5m" / "month=2024-01").is_dir()
|
||||||
|
assert preserved.exists()
|
||||||
|
assert not stale.exists()
|
||||||
|
out = pd.read_parquet(dataset_path / "frequency=5m")
|
||||||
|
assert (set(MINUTE_BAR_COLUMNS) - {"frequency"}) <= set(out.columns)
|
||||||
|
assert set(out["symbol_id"]) == {"sh600000"}
|
||||||
|
assert set(out["symbol_name"]) == {"PF Bank"}
|
||||||
|
assert stats["n_rows"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_universe_raises_when_all_symbols_empty(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"symbol_name": ["PF Bank"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"download_minute_batch",
|
||||||
|
lambda symbols, start, end, frequency=5: iter([("sh600000", None)]),
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="No minute data"):
|
||||||
|
download_minute_universe(
|
||||||
|
universe="toy",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-01-02",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_universe_progress_branch_at_100_symbols(tmp_path, monkeypatch):
|
||||||
|
symbols = [f"sh6{i:05d}" for i in range(100)]
|
||||||
|
minute = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000"],
|
||||||
|
"datetime": [pd.Timestamp("2024-01-02 09:35:00")],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"time": ["09:35:00"],
|
||||||
|
"frequency": ["5m"],
|
||||||
|
"open": [10.0],
|
||||||
|
"high": [11.0],
|
||||||
|
"low": [9.0],
|
||||||
|
"close": [10.5],
|
||||||
|
"volume": [1000.0],
|
||||||
|
"amount": [10500.0],
|
||||||
|
"vwap": [10.5],
|
||||||
|
"adjustflag": ["3"],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": symbols,
|
||||||
|
"symbol_name": symbols,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_batch(requested_symbols, start, end, frequency=5):
|
||||||
|
assert requested_symbols == symbols
|
||||||
|
for symbol in requested_symbols:
|
||||||
|
yield symbol, minute.copy()
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "download_minute_batch", fake_batch)
|
||||||
|
|
||||||
|
stats = download_minute_universe(
|
||||||
|
universe="toy100",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-01-02",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
chunk_size=200,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert stats["n_symbols"] == 100
|
||||||
|
assert stats["n_rows"] == 100
|
||||||
@@ -0,0 +1,305 @@
|
|||||||
|
"""End-to-end correctness invariants for the reversal_5d pipeline (no network).
|
||||||
|
|
||||||
|
Each test maps 1:1 to one of the ten review checks. Naming convention: the
|
||||||
|
*execution date* ``d`` is the market session on which a target is actually
|
||||||
|
filled at the open; the *signal date* ``t`` is the session whose close formed
|
||||||
|
that target. The documented convention is ``d = next(t)`` (see
|
||||||
|
``docs/portfolio_trading_cost_model.md``), so ``close[d-1] == close[t]``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.alpha.compute import compute_alpha, investable_universe_mask
|
||||||
|
from pipeline.portfolio.construct import construct_positions
|
||||||
|
from pipeline.portfolio.discretize import repair_exposure, round_to_valid_lot
|
||||||
|
from pipeline.portfolio.market_rules import MarketRule
|
||||||
|
from pipeline.portfolio.costs import SimpleProportionalCostModel
|
||||||
|
from pipeline.portfolio.constraints import (
|
||||||
|
SuspensionConstraint,
|
||||||
|
VolumeCapConstraint,
|
||||||
|
)
|
||||||
|
from pipeline.portfolio.simulator import ReferenceSimulator
|
||||||
|
|
||||||
|
|
||||||
|
_SYMBOLS = ("sh600000", "sz000001", "sh688981", "sz300750")
|
||||||
|
|
||||||
|
|
||||||
|
def _panel(n_days=12, symbols=_SYMBOLS, start="2024-01-01", seed=0,
|
||||||
|
distinct_open=True):
|
||||||
|
"""Contiguous long-format DATA frame with all columns the pipeline needs.
|
||||||
|
|
||||||
|
Open and close differ (so overnight vs intraday PnL terms are separable),
|
||||||
|
and the calendar is gap-free so each session is the next session's ``d-1``.
|
||||||
|
"""
|
||||||
|
dates = pd.date_range(start, periods=n_days)
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
frames = []
|
||||||
|
for i, sym in enumerate(symbols):
|
||||||
|
close = np.abs(50.0 + i * 10 + np.cumsum(rng.standard_normal(n_days))) + 5.0
|
||||||
|
open_ = close * (0.99 + 0.02 * rng.random(n_days)) if distinct_open else close.copy()
|
||||||
|
preclose = np.concatenate([[close[0]], close[:-1]])
|
||||||
|
frames.append(pd.DataFrame({
|
||||||
|
"symbol_id": sym,
|
||||||
|
"symbol_name": sym,
|
||||||
|
"date": dates,
|
||||||
|
"open": open_,
|
||||||
|
"high": np.maximum(open_, close),
|
||||||
|
"low": np.minimum(open_, close),
|
||||||
|
"close": close,
|
||||||
|
"preclose": preclose,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 1_000_000.0 * close,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
}))
|
||||||
|
return pd.concat(frames, ignore_index=True)
|
||||||
|
|
||||||
|
|
||||||
|
# --- 1. reversal signal uses only close[d-1] and earlier ---------------------
|
||||||
|
|
||||||
|
def test_reversal_signal_does_not_peek_at_future_closes():
|
||||||
|
data = _panel(n_days=12)
|
||||||
|
base = compute_alpha(data, "rev", "reversal_rank", lookback=5)
|
||||||
|
|
||||||
|
# Perturb every close strictly AFTER an interior signal date t; the weight
|
||||||
|
# dated t (executed at d = t+1) must be unchanged — it may use close[t]
|
||||||
|
# (== close[d-1]) and earlier only.
|
||||||
|
t = sorted(data["date"].unique())[6]
|
||||||
|
future = data.copy()
|
||||||
|
mask = future["date"] > t
|
||||||
|
future.loc[mask, ["open", "high", "low", "close"]] *= 1.5
|
||||||
|
perturbed = compute_alpha(future, "rev", "reversal_rank", lookback=5)
|
||||||
|
|
||||||
|
b = base[base["date"] <= t].set_index(["symbol_id", "date"])["weight"]
|
||||||
|
p = perturbed[perturbed["date"] <= t].set_index(["symbol_id", "date"])["weight"]
|
||||||
|
pd.testing.assert_series_equal(b.sort_index(), p.sort_index())
|
||||||
|
|
||||||
|
|
||||||
|
# --- 2. the executed (fill/PnL) date is the open-execution date d = next(t) --
|
||||||
|
|
||||||
|
def test_fill_date_is_next_session_open_execution_date():
|
||||||
|
data = _panel(n_days=8)
|
||||||
|
weights = compute_alpha(data, "c", "reversal_rank", lookback=3)
|
||||||
|
weights = weights.rename(columns={"alpha_name": "combo_name"})
|
||||||
|
weights["combo_name"] = "c"
|
||||||
|
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(pos, data)
|
||||||
|
|
||||||
|
sessions = sorted(data["date"].unique())
|
||||||
|
nxt = {s: sessions[i + 1] for i, s in enumerate(sessions[:-1])}
|
||||||
|
# Every executed date equals the session AFTER some position (signal) date.
|
||||||
|
pos_dates = set(pos["date"].unique())
|
||||||
|
exec_dates = set(pnl["date"].unique())
|
||||||
|
assert exec_dates == {nxt[t] for t in pos_dates if t in nxt}
|
||||||
|
|
||||||
|
# Execution price is the open of the execution date, not the signal close.
|
||||||
|
opn = data.pivot_table(index="date", columns="symbol_id", values="open",
|
||||||
|
aggfunc="first").sort_index()
|
||||||
|
d = sorted(exec_dates)[1]
|
||||||
|
row = fills[(fills["date"] == d) & (fills["traded_shares"] != 0)].iloc[0]
|
||||||
|
sym = row["symbol_id"]
|
||||||
|
expected_cost = abs(row["traded_shares"]) * opn.loc[d, sym] * (5 + 5) / 1e4
|
||||||
|
assert np.isclose(row["trade_cost"], expected_cost)
|
||||||
|
|
||||||
|
|
||||||
|
# --- 3. PnL identity: overnight(old book) + intraday(new book) - cost --------
|
||||||
|
|
||||||
|
def test_daily_pnl_matches_overnight_plus_intraday_minus_cost():
|
||||||
|
data = _panel(n_days=8)
|
||||||
|
weights = compute_alpha(data, "c", "reversal_rank", lookback=3)
|
||||||
|
weights = weights.rename(columns={"alpha_name": "combo_name"})
|
||||||
|
weights["combo_name"] = "c"
|
||||||
|
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(pos, data)
|
||||||
|
|
||||||
|
opn = data.pivot_table(index="date", columns="symbol_id", values="open", aggfunc="first").sort_index()
|
||||||
|
cls = data.pivot_table(index="date", columns="symbol_id", values="close", aggfunc="first").sort_index()
|
||||||
|
sessions = list(cls.index)
|
||||||
|
|
||||||
|
prev_close_of = {sessions[i]: sessions[i - 1] for i in range(1, len(sessions))}
|
||||||
|
|
||||||
|
for d in sorted(pnl["date"].unique()):
|
||||||
|
day = fills[fills["date"] == d]
|
||||||
|
prev = day.set_index("symbol_id")["prev_shares"]
|
||||||
|
realized = day.set_index("symbol_id")["realized_shares"]
|
||||||
|
cost = day["trade_cost"].sum()
|
||||||
|
|
||||||
|
intraday = float((realized * (cls.loc[d] - opn.loc[d]).reindex(realized.index)).sum())
|
||||||
|
# Overnight gap on the OLD book is taken from the previous *executed*
|
||||||
|
# date's close. With a gap-free calendar and daily execution that is the
|
||||||
|
# immediately preceding session; the first executed date has no prior
|
||||||
|
# book so the term is naturally zero (prev_shares == 0 there).
|
||||||
|
pc = prev_close_of.get(d)
|
||||||
|
if pc is not None and (prev != 0).any():
|
||||||
|
overnight = float((prev * (opn.loc[d] - cls.loc[pc]).reindex(prev.index)).sum())
|
||||||
|
else:
|
||||||
|
overnight = 0.0
|
||||||
|
expected = overnight + intraday - cost
|
||||||
|
got = float(pnl[pnl["date"] == d]["pnl"].iloc[0])
|
||||||
|
assert np.isclose(got, expected, rtol=1e-6, atol=1e-3), (d, got, expected)
|
||||||
|
|
||||||
|
|
||||||
|
# --- 4. realized shares (not target shares) are threaded into the next day ---
|
||||||
|
|
||||||
|
def test_realized_not_target_threaded_forward():
|
||||||
|
data = _panel(n_days=6)
|
||||||
|
weights = compute_alpha(data, "c", "reversal_rank", lookback=2)
|
||||||
|
weights = weights.rename(columns={"alpha_name": "combo_name"})
|
||||||
|
weights["combo_name"] = "c"
|
||||||
|
pos = construct_positions(weights, data, booksize=1e8, portfolio_name="run1")
|
||||||
|
|
||||||
|
# A tight volume cap forces partial fills, so realized != target on most
|
||||||
|
# names — exactly the case where threading target vs realized diverges.
|
||||||
|
fills, _ = ReferenceSimulator(
|
||||||
|
constraints=[VolumeCapConstraint(max_frac=1e-6)]
|
||||||
|
).run(pos, data)
|
||||||
|
|
||||||
|
wide_prev = fills.pivot_table(index="date", columns="symbol_id", values="prev_shares", aggfunc="first")
|
||||||
|
wide_real = fills.pivot_table(index="date", columns="symbol_id", values="realized_shares", aggfunc="first")
|
||||||
|
exec_dates = list(wide_prev.index)
|
||||||
|
assert len(exec_dates) >= 2
|
||||||
|
# Today's prev_shares == yesterday's realized_shares for every name.
|
||||||
|
for a, b in zip(exec_dates[:-1], exec_dates[1:]):
|
||||||
|
prev_today = wide_prev.loc[b].dropna()
|
||||||
|
real_yest = wide_real.loc[a].reindex(prev_today.index).fillna(0.0)
|
||||||
|
pd.testing.assert_series_equal(
|
||||||
|
prev_today.astype(float), real_yest.astype(float), check_names=False
|
||||||
|
)
|
||||||
|
# And realized actually diverged from target (cap bit), so the test is real.
|
||||||
|
assert (fills["realized_shares"] != fills["target_shares"]).any()
|
||||||
|
|
||||||
|
|
||||||
|
# --- 5. blocked trades create zero traded_shares and zero trade_cost ---------
|
||||||
|
|
||||||
|
def test_blocked_trade_has_zero_shares_and_zero_cost():
|
||||||
|
data = _panel(n_days=6)
|
||||||
|
# Suspend one name on every session so any attempt to trade it is blocked.
|
||||||
|
data.loc[data["symbol_id"] == "sz000001", "tradestatus"] = 0
|
||||||
|
weights = compute_alpha(data, "c", "reversal_rank", lookback=2)
|
||||||
|
weights = weights.rename(columns={"alpha_name": "combo_name"})
|
||||||
|
weights["combo_name"] = "c"
|
||||||
|
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
|
||||||
|
|
||||||
|
fills, _ = ReferenceSimulator(
|
||||||
|
constraints=[SuspensionConstraint()], cost_bps=5, slippage_bps=5
|
||||||
|
).run(pos, data)
|
||||||
|
|
||||||
|
blocked = fills[fills["blocked"] == 1]
|
||||||
|
assert (blocked["traded_shares"] == 0).all()
|
||||||
|
assert (blocked["trade_cost"] == 0.0).all()
|
||||||
|
# The suspended name never trades and never accrues cost.
|
||||||
|
susp = fills[fills["symbol_id"] == "sz000001"]
|
||||||
|
assert (susp["traded_shares"] == 0).all()
|
||||||
|
assert (susp["trade_cost"] == 0.0).all()
|
||||||
|
|
||||||
|
|
||||||
|
# --- 6. liquid universe uses only information known before open[d] -----------
|
||||||
|
|
||||||
|
def test_investable_universe_mask_is_causal():
|
||||||
|
data = _panel(n_days=14)
|
||||||
|
close = data.pivot_table(index="date", columns="symbol_id", values="close", aggfunc="first").sort_index()
|
||||||
|
full = investable_universe_mask(data, close, top_n=10, min_history=3)
|
||||||
|
|
||||||
|
t = sorted(data["date"].unique())[8]
|
||||||
|
# Recompute the mask from data truncated at the signal date t: the mask row
|
||||||
|
# for t must be identical, proving it never reads dates > t (i.e. nothing
|
||||||
|
# from open[d=t+1] onward).
|
||||||
|
trunc = data[data["date"] <= t]
|
||||||
|
close_t = close.loc[:t]
|
||||||
|
mask_t = investable_universe_mask(trunc, close_t, top_n=10, min_history=3)
|
||||||
|
pd.testing.assert_series_equal(
|
||||||
|
full.loc[t].sort_index(), mask_t.loc[t].sort_index(), check_names=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- 7. cost bps is one-way per-trade (a round trip is charged twice) --------
|
||||||
|
|
||||||
|
def test_cost_bps_is_one_way_per_trade():
|
||||||
|
model = SimpleProportionalCostModel(cost_bps=5, slippage_bps=5)
|
||||||
|
price = np.array([20.0])
|
||||||
|
|
||||||
|
buy = model.compute(np.array([1000]), price, np.array([1]), date=None)
|
||||||
|
sell = model.compute(np.array([-1000]), price, np.array([-1]), date=None)
|
||||||
|
|
||||||
|
one_way = 1000 * 20 * (5 + 5) / 1e4
|
||||||
|
assert np.isclose(buy[0], one_way) # charged once on the buy leg
|
||||||
|
assert np.isclose(sell[0], one_way) # charged again on the sell leg
|
||||||
|
# A full round trip (enter then exit) therefore costs ~2x the one-way rate.
|
||||||
|
assert np.isclose(buy[0] + sell[0], 2 * one_way)
|
||||||
|
|
||||||
|
|
||||||
|
# --- 8. execution & PnL use raw tradable prices on the same scale as shares --
|
||||||
|
|
||||||
|
def test_position_value_is_shares_times_raw_price():
|
||||||
|
data = _panel(n_days=10)
|
||||||
|
weights = compute_alpha(data, "c", "reversal_rank", lookback=3)
|
||||||
|
weights = weights.rename(columns={"alpha_name": "combo_name"})
|
||||||
|
weights["combo_name"] = "c"
|
||||||
|
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
|
||||||
|
|
||||||
|
finite = pos["price"] > 0
|
||||||
|
# The stored value is exactly integer shares × the raw construction price —
|
||||||
|
# no adjusted-price factor is mixed into the share→value accounting.
|
||||||
|
expected = pos.loc[finite, "position_shares"] * pos.loc[finite, "price"]
|
||||||
|
pd.testing.assert_series_equal(
|
||||||
|
pos.loc[finite, "position_value"].astype(float),
|
||||||
|
expected.astype(float),
|
||||||
|
check_names=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- 9. alpha is scale-free (adjusted prices ok); accounting uses raw units --
|
||||||
|
|
||||||
|
def test_alpha_weights_invariant_to_per_symbol_price_scaling():
|
||||||
|
data = _panel(n_days=12)
|
||||||
|
base = compute_alpha(data, "rev", "reversal_rank", lookback=5)
|
||||||
|
|
||||||
|
# A qfq/hfq adjustment is (per symbol) a multiplicative rescaling of the
|
||||||
|
# price series; pct_change is scale-free, so the alpha weights must not move.
|
||||||
|
scaled = data.copy()
|
||||||
|
factor = {"sh600000": 2.0, "sz000001": 0.5, "sh688981": 3.0, "sz300750": 1.25}
|
||||||
|
for sym, f in factor.items():
|
||||||
|
m = scaled["symbol_id"] == sym
|
||||||
|
scaled.loc[m, ["open", "high", "low", "close"]] *= f
|
||||||
|
scaled_alpha = compute_alpha(scaled, "rev", "reversal_rank", lookback=5)
|
||||||
|
|
||||||
|
b = base.set_index(["symbol_id", "date"])["weight"].sort_index()
|
||||||
|
s = scaled_alpha.set_index(["symbol_id", "date"])["weight"].sort_index()
|
||||||
|
pd.testing.assert_series_equal(b, s)
|
||||||
|
|
||||||
|
|
||||||
|
# --- 10. repaired book stays on valid A-share lot lattices -------------------
|
||||||
|
|
||||||
|
def _on_lattice(q, min_open, increment):
|
||||||
|
q = np.abs(np.asarray(q, dtype=np.int64))
|
||||||
|
on = (q == 0) | ((q >= min_open) & ((q - min_open) % increment == 0))
|
||||||
|
return bool(on.all())
|
||||||
|
|
||||||
|
|
||||||
|
def test_repair_output_stays_on_lot_lattice():
|
||||||
|
# Pre-2023 main board has a 100-share increment (the strongest lattice
|
||||||
|
# constraint); STAR uses min 200 / increment 1.
|
||||||
|
symbols = np.array(["sh600000", "sz000001", "sh688981", "sz300750"], dtype=object)
|
||||||
|
rule = MarketRule()
|
||||||
|
on = "2022-06-01" # pre 2023-08-10 → main-board increment is 100
|
||||||
|
min_open, increment, odd_full, _ = rule.get_rules_vectorized(
|
||||||
|
symbols, on, np.zeros(len(symbols), dtype=bool)
|
||||||
|
)
|
||||||
|
assert min_open[0] == 100 and increment[0] == 100 # main board pre-2023
|
||||||
|
|
||||||
|
price = np.array([12.3, 8.7, 45.0, 230.0])
|
||||||
|
prev = np.zeros(len(symbols), dtype=np.int64)
|
||||||
|
q_target = np.array([3251.0, -7777.0, 640.0, -415.0])
|
||||||
|
|
||||||
|
q_round = round_to_valid_lot(q_target, prev, min_open, increment, odd_full)
|
||||||
|
assert _on_lattice(q_round, min_open, increment)
|
||||||
|
|
||||||
|
repaired = repair_exposure(
|
||||||
|
q_round, q_target, price, increment, min_open, prev, odd_full,
|
||||||
|
booksize=float(np.abs(q_target * price).sum()),
|
||||||
|
)
|
||||||
|
assert _on_lattice(repaired, min_open, increment)
|
||||||
|
# Repair never flips a name's sign relative to the rounded book.
|
||||||
|
nz = q_round != 0
|
||||||
|
assert np.all(np.sign(repaired[nz]) * np.sign(q_round[nz]) >= 0)
|
||||||
+488
-1
@@ -13,15 +13,21 @@ from pipeline.portfolio.market_rules import (
|
|||||||
Board,
|
Board,
|
||||||
LimitStatus,
|
LimitStatus,
|
||||||
MarketRule,
|
MarketRule,
|
||||||
|
_to_date,
|
||||||
compute_limit_status,
|
compute_limit_status,
|
||||||
detect_board,
|
detect_board,
|
||||||
)
|
)
|
||||||
from pipeline.portfolio.research import evaluate_portfolio
|
from pipeline.portfolio.research import evaluate_portfolio
|
||||||
from pipeline.portfolio.constraints import (
|
from pipeline.portfolio.constraints import (
|
||||||
|
TradeConstraint,
|
||||||
|
available_constraints,
|
||||||
|
get_constraint,
|
||||||
PriceLimitConstraint,
|
PriceLimitConstraint,
|
||||||
|
register_constraint,
|
||||||
SuspensionConstraint,
|
SuspensionConstraint,
|
||||||
VolumeCapConstraint,
|
VolumeCapConstraint,
|
||||||
)
|
)
|
||||||
|
from pipeline.portfolio.costs import SimpleProportionalCostModel
|
||||||
from pipeline.portfolio.simulator import (
|
from pipeline.portfolio.simulator import (
|
||||||
MarketSlice,
|
MarketSlice,
|
||||||
ReferenceSimulator,
|
ReferenceSimulator,
|
||||||
@@ -64,7 +70,7 @@ def _make_data(n_days: int = 40, symbols=_SYMBOLS, start="2024-01-01",
|
|||||||
def _make_weights(data: pd.DataFrame, name="combo") -> pd.DataFrame:
|
def _make_weights(data: pd.DataFrame, name="combo") -> pd.DataFrame:
|
||||||
"""Demeaned per-date signed weights so the cross-section is dollar-neutral."""
|
"""Demeaned per-date signed weights so the cross-section is dollar-neutral."""
|
||||||
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
|
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
|
||||||
raw = -close.pct_change(5)
|
raw = -close.pct_change(5, fill_method=None)
|
||||||
demeaned = raw.sub(raw.mean(axis=1), axis=0)
|
demeaned = raw.sub(raw.mean(axis=1), axis=0)
|
||||||
long = demeaned.reset_index().melt(id_vars="date", var_name="symbol_id",
|
long = demeaned.reset_index().melt(id_vars="date", var_name="symbol_id",
|
||||||
value_name="weight").dropna()
|
value_name="weight").dropna()
|
||||||
@@ -81,6 +87,7 @@ def test_detect_board():
|
|||||||
assert detect_board("sh688981") == Board.STAR
|
assert detect_board("sh688981") == Board.STAR
|
||||||
assert detect_board("sz300750") == Board.CHINEXT
|
assert detect_board("sz300750") == Board.CHINEXT
|
||||||
assert detect_board("bj830000") == Board.UNKNOWN
|
assert detect_board("bj830000") == Board.UNKNOWN
|
||||||
|
assert detect_board("sh") == Board.UNKNOWN
|
||||||
|
|
||||||
|
|
||||||
# --- MarketRule date transitions ---------------------------------------------
|
# --- MarketRule date transitions ---------------------------------------------
|
||||||
@@ -119,6 +126,26 @@ def test_get_rules_vectorized():
|
|||||||
assert list(limit) == [0.10, 0.20, 0.20]
|
assert list(limit) == [0.10, 0.20, 0.20]
|
||||||
|
|
||||||
|
|
||||||
|
def test_market_rule_date_coercion_unknown_board_and_st_vector_override():
|
||||||
|
rules = MarketRule()
|
||||||
|
|
||||||
|
assert _to_date(dt.datetime(2024, 1, 2, 15, 0)) == dt.date(2024, 1, 2)
|
||||||
|
assert _to_date(pd.Timestamp("2024-01-03 09:30")) == dt.date(2024, 1, 3)
|
||||||
|
assert _to_date("2024-01-04 10:00:00") == dt.date(2024, 1, 4)
|
||||||
|
|
||||||
|
unknown_rule = rules.get_rule("xx999999", "2024-01-02")
|
||||||
|
assert unknown_rule.minimum_open_size == 100
|
||||||
|
assert unknown_rule.share_increment == 100
|
||||||
|
assert unknown_rule.price_limit_pct == 0.10
|
||||||
|
|
||||||
|
_, _, _, limit = rules.get_rules_vectorized(
|
||||||
|
np.array(["sh600000", "sh600000"], dtype=object),
|
||||||
|
"2024-01-02",
|
||||||
|
np.array([0, 1]),
|
||||||
|
)
|
||||||
|
assert limit.tolist() == [0.10, 0.05]
|
||||||
|
|
||||||
|
|
||||||
def test_compute_limit_status():
|
def test_compute_limit_status():
|
||||||
price = np.array([110.0, 90.0, 100.0])
|
price = np.array([110.0, 90.0, 100.0])
|
||||||
preclose = np.array([100.0, 100.0, 100.0])
|
preclose = np.array([100.0, 100.0, 100.0])
|
||||||
@@ -129,6 +156,20 @@ def test_compute_limit_status():
|
|||||||
assert status[2] == LimitStatus.NORMAL.value
|
assert status[2] == LimitStatus.NORMAL.value
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_limit_status_treats_bad_preclose_as_normal():
|
||||||
|
status = compute_limit_status(
|
||||||
|
price=np.array([np.nan, 110.0, 90.0]),
|
||||||
|
preclose=np.array([100.0, np.nan, 0.0]),
|
||||||
|
limit_pct=np.array([0.10, 0.10, 0.10]),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert status.tolist() == [
|
||||||
|
LimitStatus.NORMAL.value,
|
||||||
|
LimitStatus.NORMAL.value,
|
||||||
|
LimitStatus.NORMAL.value,
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
# --- continuous targets ------------------------------------------------------
|
# --- continuous targets ------------------------------------------------------
|
||||||
|
|
||||||
def test_continuous_targets_normalization():
|
def test_continuous_targets_normalization():
|
||||||
@@ -294,6 +335,70 @@ def test_repair_scales_to_4000_names():
|
|||||||
assert abs(gross - B) <= 0.03 * B
|
assert abs(gross - B) <= 0.03 * B
|
||||||
|
|
||||||
|
|
||||||
|
def test_repair_handles_empty_input():
|
||||||
|
result = repair_exposure(
|
||||||
|
np.array([], dtype=np.int64),
|
||||||
|
np.array([], dtype=float),
|
||||||
|
np.array([], dtype=float),
|
||||||
|
np.array([], dtype=np.int64),
|
||||||
|
np.array([], dtype=np.int64),
|
||||||
|
np.array([], dtype=np.int64),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.dtype == np.int64
|
||||||
|
assert result.tolist() == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_repair_respects_max_iters_and_zero_increment_noop():
|
||||||
|
q_round = np.array([100, -100], dtype=np.int64)
|
||||||
|
q_target = np.array([300.0, -300.0])
|
||||||
|
price = np.array([10.0, 10.0])
|
||||||
|
min_open = np.array([100, 100])
|
||||||
|
prev = np.zeros(2, dtype=np.int64)
|
||||||
|
|
||||||
|
capped = repair_exposure(
|
||||||
|
q_round,
|
||||||
|
q_target,
|
||||||
|
price,
|
||||||
|
increment=np.array([1, 1]),
|
||||||
|
min_open=min_open,
|
||||||
|
prev_shares=prev,
|
||||||
|
booksize=10_000.0,
|
||||||
|
gross_tol=0.0,
|
||||||
|
max_iters=0,
|
||||||
|
)
|
||||||
|
zero_increment = repair_exposure(
|
||||||
|
q_round,
|
||||||
|
q_target,
|
||||||
|
price,
|
||||||
|
increment=np.array([0, 0]),
|
||||||
|
min_open=min_open,
|
||||||
|
prev_shares=prev,
|
||||||
|
booksize=0.0,
|
||||||
|
net_tol=0.0,
|
||||||
|
gross_tol=0.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert capped.tolist() == [100, -100]
|
||||||
|
assert zero_increment.tolist() == [100, -100]
|
||||||
|
|
||||||
|
|
||||||
|
def test_repair_gross_growth_obeys_net_band():
|
||||||
|
pos = repair_exposure(
|
||||||
|
q_round=np.array([100], dtype=np.int64),
|
||||||
|
q_target=np.array([300.0]),
|
||||||
|
price=np.array([10.0]),
|
||||||
|
increment=np.array([1]),
|
||||||
|
min_open=np.array([100]),
|
||||||
|
prev_shares=np.array([0], dtype=np.int64),
|
||||||
|
booksize=2_000.0,
|
||||||
|
net_tol=0.5,
|
||||||
|
gross_tol=0.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert pos.tolist() == [100]
|
||||||
|
|
||||||
|
|
||||||
# --- construct_positions -----------------------------------------------------
|
# --- construct_positions -----------------------------------------------------
|
||||||
|
|
||||||
def test_construct_positions_schema():
|
def test_construct_positions_schema():
|
||||||
@@ -305,6 +410,47 @@ def test_construct_positions_schema():
|
|||||||
assert pos["position_shares"].dtype == np.int64
|
assert pos["position_shares"].dtype == np.int64
|
||||||
|
|
||||||
|
|
||||||
|
def test_construct_positions_empty_weights_returns_schema():
|
||||||
|
data = _make_data(n_days=3)
|
||||||
|
empty_weights = pd.DataFrame(columns=["symbol_id", "date", "combo_name", "weight"])
|
||||||
|
|
||||||
|
pos = construct_positions(
|
||||||
|
empty_weights,
|
||||||
|
data,
|
||||||
|
booksize=1e6,
|
||||||
|
portfolio_name="empty",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(pos.columns) == POSITION_COLUMNS
|
||||||
|
assert pos.empty
|
||||||
|
|
||||||
|
|
||||||
|
def test_construct_positions_ignores_absent_or_bad_prices():
|
||||||
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
||||||
|
data = pd.DataFrame([
|
||||||
|
{"symbol_id": "sh600000", "date": dates[0], "close": np.nan, "isST": 0},
|
||||||
|
{"symbol_id": "sz000001", "date": dates[0], "close": 20.0, "isST": 0},
|
||||||
|
{"symbol_id": "sh600000", "date": dates[1], "close": 10.0, "isST": 0},
|
||||||
|
{"symbol_id": "sz000001", "date": dates[1], "close": 20.0, "isST": 0},
|
||||||
|
])
|
||||||
|
weights = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1], dates[1]],
|
||||||
|
"combo_name": ["combo"] * 4,
|
||||||
|
"weight": [1.0, -1.0, 1.0, -1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
pos = construct_positions(weights, data, booksize=10000.0, portfolio_name="bad_price")
|
||||||
|
|
||||||
|
bad_price_rows = pos[
|
||||||
|
(pos["date"] == dates[0])
|
||||||
|
& (pos["symbol_id"] == "sh600000")
|
||||||
|
]
|
||||||
|
assert bad_price_rows.empty or (bad_price_rows["target_weight"] == 0.0).all()
|
||||||
|
assert np.isfinite(pos["target_value"]).all()
|
||||||
|
assert np.isfinite(pos["position_value"]).all()
|
||||||
|
|
||||||
|
|
||||||
def test_construct_positions_threads_state_and_closes_absent():
|
def test_construct_positions_threads_state_and_closes_absent():
|
||||||
data = _make_data()
|
data = _make_data()
|
||||||
weights = _make_weights(data)
|
weights = _make_weights(data)
|
||||||
@@ -320,6 +466,34 @@ def test_construct_positions_threads_state_and_closes_absent():
|
|||||||
assert final.empty or (final["position_shares"] == 0).all()
|
assert final.empty or (final["position_shares"] == 0).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_construct_positions_closes_absent_short_position():
|
||||||
|
dates = pd.to_datetime(["2024-01-02", "2024-01-03"])
|
||||||
|
data = pd.DataFrame([
|
||||||
|
{"symbol_id": sym, "date": d, "close": price, "isST": 0}
|
||||||
|
for d in dates
|
||||||
|
for sym, price in (("sh600000", 10.0), ("sz000001", 20.0))
|
||||||
|
])
|
||||||
|
weights = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1]],
|
||||||
|
"combo_name": ["combo", "combo", "combo"],
|
||||||
|
"weight": [-1.0, 1.0, 1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
pos = construct_positions(weights, data, booksize=20000.0, portfolio_name="absent_short")
|
||||||
|
|
||||||
|
first_day_short = pos[
|
||||||
|
(pos["date"] == dates[0])
|
||||||
|
& (pos["symbol_id"] == "sh600000")
|
||||||
|
]
|
||||||
|
final_day_short = pos[
|
||||||
|
(pos["date"] == dates[1])
|
||||||
|
& (pos["symbol_id"] == "sh600000")
|
||||||
|
]
|
||||||
|
assert (first_day_short["position_shares"] < 0).all()
|
||||||
|
assert final_day_short.empty or (final_day_short["position_shares"] == 0).all()
|
||||||
|
|
||||||
|
|
||||||
def test_construct_positions_carries_book_on_zero_gross(caplog):
|
def test_construct_positions_carries_book_on_zero_gross(caplog):
|
||||||
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
||||||
symbols = ["sh600000", "sz000001"]
|
symbols = ["sh600000", "sz000001"]
|
||||||
@@ -391,8 +565,117 @@ def test_volume_cap_uses_traded_value():
|
|||||||
assert low[0] == -10000.0
|
assert low[0] == -10000.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_constraints_compose_repeatably_regardless_of_order():
|
||||||
|
n = 1
|
||||||
|
sl = _slice(
|
||||||
|
n,
|
||||||
|
tradestatus=np.array([0.0]),
|
||||||
|
limit_status=np.array([LimitStatus.UP_LIMIT.value], dtype=np.int8),
|
||||||
|
amount=np.array([1_000.0]),
|
||||||
|
price=np.array([10.0]),
|
||||||
|
)
|
||||||
|
ctx = TradeContext(np.zeros(n, np.int64), np.array([500]), sl, 1e6)
|
||||||
|
first_order = ReferenceSimulator(
|
||||||
|
constraints=[
|
||||||
|
SuspensionConstraint(),
|
||||||
|
PriceLimitConstraint(),
|
||||||
|
VolumeCapConstraint(max_frac=0.1),
|
||||||
|
],
|
||||||
|
cost_bps=10,
|
||||||
|
).fill(ctx)
|
||||||
|
reversed_order = ReferenceSimulator(
|
||||||
|
constraints=[
|
||||||
|
VolumeCapConstraint(max_frac=0.1),
|
||||||
|
PriceLimitConstraint(),
|
||||||
|
SuspensionConstraint(),
|
||||||
|
],
|
||||||
|
cost_bps=10,
|
||||||
|
).fill(ctx)
|
||||||
|
|
||||||
|
assert first_order.traded_shares.tolist() == [0]
|
||||||
|
assert first_order.realized_shares.tolist() == [0]
|
||||||
|
assert first_order.blocked.tolist() == [1]
|
||||||
|
assert np.array_equal(first_order.traded_shares, reversed_order.traded_shares)
|
||||||
|
assert np.array_equal(first_order.realized_shares, reversed_order.realized_shares)
|
||||||
|
assert np.array_equal(first_order.blocked, reversed_order.blocked)
|
||||||
|
assert np.array_equal(first_order.cost, reversed_order.cost)
|
||||||
|
|
||||||
|
|
||||||
|
def test_constraint_registry_and_default_adjust_targets():
|
||||||
|
class _NoopConstraint(TradeConstraint):
|
||||||
|
name = "_coverage_noop_constraint"
|
||||||
|
|
||||||
|
def delta_bounds(self, ctx):
|
||||||
|
return np.zeros(1), np.ones(1)
|
||||||
|
|
||||||
|
registered = register_constraint(_NoopConstraint)
|
||||||
|
|
||||||
|
assert registered is _NoopConstraint
|
||||||
|
assert "_coverage_noop_constraint" in available_constraints()
|
||||||
|
assert isinstance(get_constraint("_coverage_noop_constraint"), _NoopConstraint)
|
||||||
|
assert get_constraint("_coverage_noop_constraint").adjust_targets(object()) is None
|
||||||
|
with np.testing.assert_raises(KeyError):
|
||||||
|
get_constraint("_missing_constraint")
|
||||||
|
with np.testing.assert_raises(TypeError):
|
||||||
|
register_constraint(object) # type: ignore[arg-type]
|
||||||
|
with np.testing.assert_raises(ValueError):
|
||||||
|
class _NoNameConstraint(TradeConstraint):
|
||||||
|
def delta_bounds(self, ctx):
|
||||||
|
return np.zeros(1), np.ones(1)
|
||||||
|
|
||||||
|
register_constraint(_NoNameConstraint)
|
||||||
|
with np.testing.assert_raises(ValueError):
|
||||||
|
class _DuplicateConstraint(TradeConstraint):
|
||||||
|
name = "_coverage_noop_constraint"
|
||||||
|
|
||||||
|
def delta_bounds(self, ctx):
|
||||||
|
return np.zeros(1), np.ones(1)
|
||||||
|
|
||||||
|
register_constraint(_DuplicateConstraint)
|
||||||
|
|
||||||
|
|
||||||
# --- ReferenceSimulator ------------------------------------------------------
|
# --- ReferenceSimulator ------------------------------------------------------
|
||||||
|
|
||||||
|
def test_simulator_applies_constraint_target_adjustment():
|
||||||
|
class _HalveTarget(TradeConstraint):
|
||||||
|
name = "_halve_target"
|
||||||
|
|
||||||
|
def adjust_targets(self, ctx):
|
||||||
|
return ctx.target_shares // 2
|
||||||
|
|
||||||
|
def delta_bounds(self, ctx):
|
||||||
|
return np.full(len(ctx.target_shares), -np.inf), np.full(len(ctx.target_shares), np.inf)
|
||||||
|
|
||||||
|
sl = _slice(1, price=np.array([10.0]))
|
||||||
|
ctx = TradeContext(np.array([0], np.int64), np.array([100], np.int64), sl, 1e6)
|
||||||
|
|
||||||
|
result = ReferenceSimulator(constraints=[_HalveTarget()]).fill(ctx)
|
||||||
|
|
||||||
|
assert result.traded_shares.tolist() == [50]
|
||||||
|
assert result.realized_shares.tolist() == [50]
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_empty_positions_uses_default_booksize():
|
||||||
|
data = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"open": [10.0],
|
||||||
|
"close": [10.0],
|
||||||
|
"preclose": [10.0],
|
||||||
|
"amount": [1e9],
|
||||||
|
"tradestatus": [1],
|
||||||
|
"isST": [0],
|
||||||
|
})
|
||||||
|
positions = pd.DataFrame(columns=POSITION_COLUMNS)
|
||||||
|
|
||||||
|
fills, pnl = ReferenceSimulator().run(positions, data)
|
||||||
|
|
||||||
|
assert list(fills.columns) == FILL_COLUMNS
|
||||||
|
assert list(pnl.columns) == PNL_COLUMNS
|
||||||
|
assert fills.empty
|
||||||
|
assert pnl.empty
|
||||||
|
|
||||||
|
|
||||||
def test_simulator_next_open_and_blocked_buy_holds_prev():
|
def test_simulator_next_open_and_blocked_buy_holds_prev():
|
||||||
data = _make_data(n_days=15)
|
data = _make_data(n_days=15)
|
||||||
weights = _make_weights(data)
|
weights = _make_weights(data)
|
||||||
@@ -445,6 +728,7 @@ def test_simulator_blocked_buy_when_suspended():
|
|||||||
assert res.traded_shares[0] == 0
|
assert res.traded_shares[0] == 0
|
||||||
assert res.realized_shares[0] == 0
|
assert res.realized_shares[0] == 0
|
||||||
assert res.blocked[0] == 1
|
assert res.blocked[0] == 1
|
||||||
|
assert res.cost[0] == 0.0
|
||||||
|
|
||||||
|
|
||||||
def test_simulator_cost_is_positive_when_trading():
|
def test_simulator_cost_is_positive_when_trading():
|
||||||
@@ -458,6 +742,154 @@ def test_simulator_cost_is_positive_when_trading():
|
|||||||
assert np.isclose(res.cost[0], 1000 * 20 * 15 / 1e4)
|
assert np.isclose(res.cost[0], 1000 * 20 * 15 / 1e4)
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_cost_only_on_nonzero_realized_trades():
|
||||||
|
n = 2
|
||||||
|
sim = ReferenceSimulator(constraints=[], cost_bps=10)
|
||||||
|
sl = _slice(n, price=np.array([10.0, 20.0]))
|
||||||
|
ctx = TradeContext(np.array([100, 100], np.int64),
|
||||||
|
np.array([100, 150], np.int64), sl, 1e6)
|
||||||
|
|
||||||
|
res = sim.fill(ctx)
|
||||||
|
|
||||||
|
assert res.traded_shares.tolist() == [0, 50]
|
||||||
|
assert res.cost[0] == 0.0
|
||||||
|
assert np.isclose(res.cost[1], 50 * 20 * 10 / 1e4)
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_short_to_long_flip_trades_full_delta():
|
||||||
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"])
|
||||||
|
positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": [dates[0], dates[1]],
|
||||||
|
"portfolio_name": ["flip", "flip"],
|
||||||
|
"target_weight": [-1.0, 1.0],
|
||||||
|
"target_value": [-1000.0, 1000.0],
|
||||||
|
"target_shares": [-100.0, 100.0],
|
||||||
|
"position_shares": [-100, 100],
|
||||||
|
"position_value": [-1000.0, 1000.0],
|
||||||
|
"price": [10.0, 10.0],
|
||||||
|
})
|
||||||
|
data = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"] * 3,
|
||||||
|
"date": dates,
|
||||||
|
"open": [10.0, 10.0, 10.0],
|
||||||
|
"close": [10.0, 10.0, 10.0],
|
||||||
|
"preclose": [10.0, 10.0, 10.0],
|
||||||
|
"amount": [1e9, 1e9, 1e9],
|
||||||
|
"tradestatus": [1, 1, 1],
|
||||||
|
"isST": [0, 0, 0],
|
||||||
|
})
|
||||||
|
|
||||||
|
fills, _ = ReferenceSimulator().run(positions, data)
|
||||||
|
|
||||||
|
by_date = fills.set_index("date")
|
||||||
|
assert by_date.loc[dates[1], "traded_shares"] == -100
|
||||||
|
assert by_date.loc[dates[1], "realized_shares"] == -100
|
||||||
|
assert by_date.loc[dates[2], "prev_shares"] == -100
|
||||||
|
assert by_date.loc[dates[2], "traded_shares"] == 200
|
||||||
|
assert by_date.loc[dates[2], "realized_shares"] == 100
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_volume_cap_partially_fills_sell():
|
||||||
|
sl = _slice(1, amount=np.array([10_000.0]), price=np.array([10.0]))
|
||||||
|
ctx = TradeContext(
|
||||||
|
np.array([1000], np.int64),
|
||||||
|
np.array([0], np.int64),
|
||||||
|
sl,
|
||||||
|
1_000_000.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = ReferenceSimulator(
|
||||||
|
constraints=[VolumeCapConstraint(max_frac=0.10)]
|
||||||
|
).fill(ctx)
|
||||||
|
|
||||||
|
assert result.traded_shares.tolist() == [-100]
|
||||||
|
assert result.realized_shares.tolist() == [900]
|
||||||
|
assert result.blocked.tolist() == [1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_missing_next_open_has_zero_cost_and_turnover():
|
||||||
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
||||||
|
positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [dates[0]],
|
||||||
|
"portfolio_name": ["missing_open"],
|
||||||
|
"target_weight": [1.0],
|
||||||
|
"target_value": [1000.0],
|
||||||
|
"target_shares": [100.0],
|
||||||
|
"position_shares": [100],
|
||||||
|
"position_value": [1000.0],
|
||||||
|
"price": [10.0],
|
||||||
|
})
|
||||||
|
data = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": dates,
|
||||||
|
"open": [10.0, np.nan],
|
||||||
|
"close": [10.0, 10.0],
|
||||||
|
"preclose": [10.0, 10.0],
|
||||||
|
"amount": [1e9, 1e9],
|
||||||
|
"tradestatus": [1, 1],
|
||||||
|
"isST": [0, 0],
|
||||||
|
})
|
||||||
|
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=10, slippage_bps=5).run(positions, data)
|
||||||
|
|
||||||
|
assert fills["traded_shares"].iloc[0] == 100
|
||||||
|
assert fills["trade_cost"].iloc[0] == 0.0
|
||||||
|
assert pnl["cost"].iloc[0] == 0.0
|
||||||
|
assert pnl["turnover"].iloc[0] == 0.0
|
||||||
|
assert pnl["gross_exposure"].iloc[0] == 1000.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_simple_cost_model_adds_cost_and_slippage_without_price_adjustment():
|
||||||
|
model = SimpleProportionalCostModel(cost_bps=10, slippage_bps=5)
|
||||||
|
|
||||||
|
cost = model.compute(
|
||||||
|
traded_shares=np.array([1000, -1000]),
|
||||||
|
execution_price=np.array([20.0, 20.0]),
|
||||||
|
side=np.array([1, -1]),
|
||||||
|
date=dt.date(2024, 1, 2),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert np.allclose(cost, np.array([30.0, 30.0]))
|
||||||
|
|
||||||
|
|
||||||
|
def test_daily_pnl_cost_matches_fill_trade_cost_sum():
|
||||||
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
||||||
|
positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0]],
|
||||||
|
"portfolio_name": ["run1", "run1"],
|
||||||
|
"target_weight": [0.5, -0.5],
|
||||||
|
"target_value": [1000.0, -1000.0],
|
||||||
|
"target_shares": [100.0, -50.0],
|
||||||
|
"position_shares": [100, -50],
|
||||||
|
"position_value": [1000.0, -1000.0],
|
||||||
|
"price": [10.0, 20.0],
|
||||||
|
})
|
||||||
|
data = pd.DataFrame([
|
||||||
|
{
|
||||||
|
"symbol_id": sym,
|
||||||
|
"date": d,
|
||||||
|
"open": price,
|
||||||
|
"close": price,
|
||||||
|
"preclose": price,
|
||||||
|
"amount": 1e9,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
}
|
||||||
|
for d in dates
|
||||||
|
for sym, price in (("sh600000", 10.0), ("sz000001", 20.0))
|
||||||
|
])
|
||||||
|
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=10, slippage_bps=5).run(positions, data)
|
||||||
|
|
||||||
|
total_fill_cost = fills["trade_cost"].sum()
|
||||||
|
assert np.isclose(total_fill_cost, 3.0)
|
||||||
|
assert np.isclose(pnl["cost"].iloc[0], total_fill_cost)
|
||||||
|
assert np.isclose(pnl["pnl"].iloc[0], -total_fill_cost)
|
||||||
|
|
||||||
|
|
||||||
# --- evaluate_portfolio ------------------------------------------------------
|
# --- evaluate_portfolio ------------------------------------------------------
|
||||||
|
|
||||||
def test_evaluate_portfolio_keys_no_ic():
|
def test_evaluate_portfolio_keys_no_ic():
|
||||||
@@ -470,3 +902,58 @@ def test_evaluate_portfolio_keys_no_ic():
|
|||||||
assert key in metrics
|
assert key in metrics
|
||||||
assert "ic" not in metrics
|
assert "ic" not in metrics
|
||||||
assert "rank_ic" not in metrics
|
assert "rank_ic" not in metrics
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_portfolio_excludes_signal_without_forward_return():
|
||||||
|
dates = pd.date_range("2024-01-01", periods=3)
|
||||||
|
data = pd.DataFrame([
|
||||||
|
{"symbol_id": sym, "date": d, "open": price, "close": price}
|
||||||
|
for d, prices in zip(dates, [(100.0, 100.0), (100.0, 100.0), (200.0, 100.0)])
|
||||||
|
for sym, price in zip(("sh600000", "sz000001"), prices)
|
||||||
|
])
|
||||||
|
positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1], dates[1]],
|
||||||
|
"portfolio_name": ["run1"] * 4,
|
||||||
|
"target_weight": [0.5, -0.5, -0.5, 0.5],
|
||||||
|
"target_value": [500.0, -500.0, -500.0, 500.0],
|
||||||
|
"target_shares": [5.0, -5.0, -2.5, 5.0],
|
||||||
|
"position_shares": [5, -5, -2, 5],
|
||||||
|
"position_value": [500.0, -500.0, -400.0, 500.0],
|
||||||
|
"price": [100.0, 100.0, 200.0, 100.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
metrics = evaluate_portfolio(positions, data)
|
||||||
|
|
||||||
|
assert metrics["n_dates"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_portfolio_empty_and_single_return_paths():
|
||||||
|
empty_metrics = evaluate_portfolio(
|
||||||
|
pd.DataFrame(columns=POSITION_COLUMNS),
|
||||||
|
pd.DataFrame(columns=["symbol_id", "date", "open"]),
|
||||||
|
)
|
||||||
|
assert empty_metrics["n_dates"] == 0
|
||||||
|
assert empty_metrics["cumulative_return"] == 0.0
|
||||||
|
|
||||||
|
dates = pd.date_range("2024-01-01", periods=3)
|
||||||
|
data = pd.DataFrame([
|
||||||
|
{"symbol_id": "sh600000", "date": d, "open": price}
|
||||||
|
for d, price in zip(dates, [100.0, 100.0, 110.0])
|
||||||
|
])
|
||||||
|
positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [dates[0]],
|
||||||
|
"portfolio_name": ["single"],
|
||||||
|
"target_weight": [1.0],
|
||||||
|
"target_value": [1000.0],
|
||||||
|
"target_shares": [10.0],
|
||||||
|
"position_shares": [10],
|
||||||
|
"position_value": [1000.0],
|
||||||
|
"price": [100.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
single_metrics = evaluate_portfolio(positions, data)
|
||||||
|
|
||||||
|
assert single_metrics["n_dates"] == 1
|
||||||
|
assert single_metrics["cumulative_return"] == 0.0
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
"""Malformed parquet/input tests for phase boundary contracts."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pipeline.alpha.compute import compute_alpha
|
||||||
|
from pipeline.combo.combine import combine_alphas
|
||||||
|
from pipeline.derived.compute import validate_derived_frame
|
||||||
|
from pipeline.portfolio.construct import construct_positions
|
||||||
|
from pipeline.portfolio.simulator import ReferenceSimulator
|
||||||
|
from tests.helpers import make_generated_daily_bars
|
||||||
|
|
||||||
|
|
||||||
|
def test_alpha_compute_rejects_daily_data_without_close():
|
||||||
|
daily = make_generated_daily_bars().drop(columns=["close"])
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="close"):
|
||||||
|
compute_alpha(daily, "bad", "reversal", lookback=3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_alpha_feature_path_rejects_duplicate_symbol_dates(tmp_path):
|
||||||
|
daily = make_generated_daily_bars()
|
||||||
|
feature = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
|
||||||
|
"toy_feature": [1.0, 2.0],
|
||||||
|
})
|
||||||
|
feature_path = tmp_path / "duplicate_feature_keys.pq"
|
||||||
|
feature.to_parquet(feature_path, index=False)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
|
||||||
|
compute_alpha(
|
||||||
|
daily,
|
||||||
|
"bad_features",
|
||||||
|
"reversal",
|
||||||
|
lookback=3,
|
||||||
|
feature_paths=[str(feature_path)],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_validation_rejects_bool_value_columns():
|
||||||
|
derived = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"is_good": [True],
|
||||||
|
})
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="numeric"):
|
||||||
|
validate_derived_frame(derived)
|
||||||
|
|
||||||
|
|
||||||
|
def test_combo_combine_rejects_missing_weight_column(tmp_path):
|
||||||
|
bad_alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"alpha_name": ["bad"],
|
||||||
|
})
|
||||||
|
bad_alpha_path = tmp_path / "bad_alpha.pq"
|
||||||
|
bad_alpha.to_parquet(bad_alpha_path, index=False)
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="weight"):
|
||||||
|
combine_alphas([str(bad_alpha_path)], "bad_combo")
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_build_rejects_weights_without_symbol_id():
|
||||||
|
daily = make_generated_daily_bars()
|
||||||
|
bad_weights = pd.DataFrame({
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"combo_name": ["bad"],
|
||||||
|
"weight": [1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="symbol_id"):
|
||||||
|
construct_positions(
|
||||||
|
bad_weights,
|
||||||
|
daily,
|
||||||
|
booksize=1_000_000.0,
|
||||||
|
portfolio_name="bad_portfolio",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_simulate_rejects_positions_without_position_shares():
|
||||||
|
daily = make_generated_daily_bars()
|
||||||
|
bad_positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"portfolio_name": ["bad"],
|
||||||
|
"target_weight": [1.0],
|
||||||
|
"target_value": [1000.0],
|
||||||
|
"target_shares": [100.0],
|
||||||
|
"position_value": [1000.0],
|
||||||
|
"price": [10.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="position_shares"):
|
||||||
|
ReferenceSimulator().run(bad_positions, daily)
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
"""Offline tests for baostock-backed universe helpers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
import data.universe as universe
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResult:
|
||||||
|
def __init__(self, rows, fields=None):
|
||||||
|
self.rows = rows
|
||||||
|
self.fields = fields or ["code", "name", "date"]
|
||||||
|
self._idx = -1
|
||||||
|
|
||||||
|
def next(self):
|
||||||
|
self._idx += 1
|
||||||
|
return self._idx < len(self.rows)
|
||||||
|
|
||||||
|
def get_row_data(self):
|
||||||
|
return self.rows[self._idx]
|
||||||
|
|
||||||
|
|
||||||
|
def test_index_constituent_helpers_normalize_dotted_codes(monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
|
||||||
|
monkeypatch.setattr(universe.bs, "login", lambda: calls.append("login"))
|
||||||
|
monkeypatch.setattr(universe.bs, "logout", lambda: calls.append("logout"))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
universe.bs,
|
||||||
|
"query_hs300_stocks",
|
||||||
|
lambda: _FakeResult([
|
||||||
|
["sh.600000", "浦发银行", "2024-01-12"],
|
||||||
|
["sz.000001", "平安银行", "2024-01-12"],
|
||||||
|
]),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
universe.bs,
|
||||||
|
"query_zz500_stocks",
|
||||||
|
lambda: _FakeResult([
|
||||||
|
["sh.600006", "东风汽车", "2024-01-12"],
|
||||||
|
]),
|
||||||
|
)
|
||||||
|
|
||||||
|
hs300 = universe.get_hs300_stocks()
|
||||||
|
zz500 = universe.get_zz500_stocks()
|
||||||
|
|
||||||
|
assert calls == ["login", "logout", "login", "logout"]
|
||||||
|
assert hs300["code"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert zz500["code"].tolist() == ["sh600006"]
|
||||||
|
assert hs300["name"].tolist() == ["浦发银行", "平安银行"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_all_stocks_walks_back_and_filters_to_listed_a_shares(monkeypatch):
|
||||||
|
fields = ["code", "tradeStatus", "code_name"]
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], fields=fields),
|
||||||
|
_FakeResult(
|
||||||
|
[
|
||||||
|
["sh.600000", "1", "浦发银行"],
|
||||||
|
["sh.688001", "1", "华兴源创"],
|
||||||
|
["sz.000001", "1", "平安银行"],
|
||||||
|
["sz.300750", "1", "宁德时代"],
|
||||||
|
["sz.399001", "1", "深证成指"],
|
||||||
|
["sz.200001", "1", "深物业B"],
|
||||||
|
["bj.430047", "1", "北交所样本"],
|
||||||
|
],
|
||||||
|
fields=fields,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
query_days: list[str] = []
|
||||||
|
|
||||||
|
monkeypatch.setattr(universe.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(universe.bs, "logout", lambda: None)
|
||||||
|
|
||||||
|
def fake_query_all_stock(day):
|
||||||
|
query_days.append(day)
|
||||||
|
return responses.pop(0)
|
||||||
|
|
||||||
|
monkeypatch.setattr(universe.bs, "query_all_stock", fake_query_all_stock)
|
||||||
|
|
||||||
|
result = universe.get_all_stocks("2024-01-07")
|
||||||
|
|
||||||
|
assert query_days == ["2024-01-07", "2024-01-06"]
|
||||||
|
assert result.columns.tolist() == ["code", "name"]
|
||||||
|
assert result["code"].tolist() == [
|
||||||
|
"sh600000",
|
||||||
|
"sh688001",
|
||||||
|
"sz000001",
|
||||||
|
"sz300750",
|
||||||
|
]
|
||||||
|
assert result["name"].tolist() == ["浦发银行", "华兴源创", "平安银行", "宁德时代"]
|
||||||
@@ -0,0 +1,508 @@
|
|||||||
|
"""Verbose offline checks for the daily research workflow."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.alpha.compute import compute_alpha
|
||||||
|
from pipeline.combo.combine import combine_alphas
|
||||||
|
from pipeline.common.schema import (
|
||||||
|
ALPHA_COLUMNS,
|
||||||
|
COMBO_COLUMNS,
|
||||||
|
FILL_COLUMNS,
|
||||||
|
PNL_COLUMNS,
|
||||||
|
POSITION_COLUMNS,
|
||||||
|
)
|
||||||
|
from pipeline.portfolio.constraints import (
|
||||||
|
PriceLimitConstraint,
|
||||||
|
SuspensionConstraint,
|
||||||
|
VolumeCapConstraint,
|
||||||
|
)
|
||||||
|
from pipeline.portfolio.construct import construct_positions
|
||||||
|
from pipeline.portfolio.research import evaluate_portfolio
|
||||||
|
from pipeline.portfolio.simulator import ReferenceSimulator
|
||||||
|
from tests.helpers import (
|
||||||
|
GENERATED_SYMBOLS,
|
||||||
|
generated_sessions,
|
||||||
|
make_generated_alpha_weights,
|
||||||
|
make_generated_combo_weights,
|
||||||
|
make_generated_daily_bars,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_sorted_by_symbol_date(frame: pd.DataFrame) -> None:
|
||||||
|
expected = frame.sort_values(["symbol_id", "date"]).reset_index(drop=True)
|
||||||
|
pd.testing.assert_frame_equal(frame.reset_index(drop=True), expected)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_metric_dict_is_finite(metrics: dict[str, float]) -> None:
|
||||||
|
for key in (
|
||||||
|
"cumulative_return",
|
||||||
|
"sharpe_annual",
|
||||||
|
"turnover_annual",
|
||||||
|
"max_drawdown",
|
||||||
|
"hit_rate",
|
||||||
|
"n_dates",
|
||||||
|
):
|
||||||
|
assert key in metrics
|
||||||
|
assert np.isfinite(metrics[key])
|
||||||
|
assert "ic" not in metrics
|
||||||
|
assert "rank_ic" not in metrics
|
||||||
|
assert "ir" not in metrics
|
||||||
|
|
||||||
|
|
||||||
|
def test_tiny_workflow_golden_outputs_are_stable(tmp_path):
|
||||||
|
dates = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"])
|
||||||
|
daily_bars = pd.DataFrame([
|
||||||
|
{
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "A",
|
||||||
|
"date": dates[0],
|
||||||
|
"open": 10.0,
|
||||||
|
"high": 10.0,
|
||||||
|
"low": 10.0,
|
||||||
|
"close": 10.0,
|
||||||
|
"preclose": 10.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 10_000_000.0,
|
||||||
|
"vwap": 10.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 0.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "B",
|
||||||
|
"date": dates[0],
|
||||||
|
"open": 20.0,
|
||||||
|
"high": 20.0,
|
||||||
|
"low": 20.0,
|
||||||
|
"close": 20.0,
|
||||||
|
"preclose": 20.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 20_000_000.0,
|
||||||
|
"vwap": 20.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 0.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "A",
|
||||||
|
"date": dates[1],
|
||||||
|
"open": 10.0,
|
||||||
|
"high": 12.0,
|
||||||
|
"low": 10.0,
|
||||||
|
"close": 12.0,
|
||||||
|
"preclose": 10.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 10_000_000.0,
|
||||||
|
"vwap": 10.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 20.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "B",
|
||||||
|
"date": dates[1],
|
||||||
|
"open": 20.0,
|
||||||
|
"high": 20.0,
|
||||||
|
"low": 18.0,
|
||||||
|
"close": 18.0,
|
||||||
|
"preclose": 20.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 20_000_000.0,
|
||||||
|
"vwap": 20.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": -10.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "A",
|
||||||
|
"date": dates[2],
|
||||||
|
"open": 12.0,
|
||||||
|
"high": 13.0,
|
||||||
|
"low": 12.0,
|
||||||
|
"close": 13.0,
|
||||||
|
"preclose": 12.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 12_000_000.0,
|
||||||
|
"vwap": 12.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 8.33,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "B",
|
||||||
|
"date": dates[2],
|
||||||
|
"open": 18.0,
|
||||||
|
"high": 21.0,
|
||||||
|
"low": 18.0,
|
||||||
|
"close": 21.0,
|
||||||
|
"preclose": 18.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 18_000_000.0,
|
||||||
|
"vwap": 18.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 16.67,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
])
|
||||||
|
alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1], dates[1]],
|
||||||
|
"alpha_name": ["gold_alpha"] * 4,
|
||||||
|
"weight": [1.0, -1.0, -1.0, 1.0],
|
||||||
|
})
|
||||||
|
alpha_path = tmp_path / "gold_alpha.pq"
|
||||||
|
alpha.to_parquet(alpha_path, index=False)
|
||||||
|
|
||||||
|
combo = combine_alphas([str(alpha_path)], "gold_combo")
|
||||||
|
positions = construct_positions(combo, daily_bars, booksize=20_000.0, portfolio_name="gold_port")
|
||||||
|
fills, pnl = ReferenceSimulator().run(positions, daily_bars)
|
||||||
|
|
||||||
|
expected_combo = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
|
||||||
|
"date": [dates[0], dates[1], dates[0], dates[1]],
|
||||||
|
"combo_name": ["gold_combo"] * 4,
|
||||||
|
"weight": [1.0, -1.0, -1.0, 1.0],
|
||||||
|
})
|
||||||
|
expected_positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
|
||||||
|
"date": [dates[0], dates[1], dates[0], dates[1]],
|
||||||
|
"portfolio_name": ["gold_port"] * 4,
|
||||||
|
"target_weight": [0.5, -0.5, -0.5, 0.5],
|
||||||
|
"target_value": [10000.0, -10000.0, -10000.0, 10000.0],
|
||||||
|
"target_shares": [1000.0, -10000.0 / 12.0, -500.0, 10000.0 / 18.0],
|
||||||
|
"position_shares": [1000, -833, -500, 556],
|
||||||
|
"position_value": [10000.0, -9996.0, -10000.0, 10008.0],
|
||||||
|
"price": [10.0, 12.0, 20.0, 18.0],
|
||||||
|
})
|
||||||
|
expected_fills = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[1], dates[1], dates[2], dates[2]],
|
||||||
|
"portfolio_name": ["gold_port"] * 4,
|
||||||
|
"prev_shares": [0, 0, 1000, -500],
|
||||||
|
"target_shares": [1000, -500, -833, 556],
|
||||||
|
"traded_shares": [1000, -500, -1833, 1056],
|
||||||
|
"realized_shares": [1000, -500, -833, 556],
|
||||||
|
"blocked": [0, 0, 0, 0],
|
||||||
|
"trade_cost": [0.0, 0.0, 0.0, 0.0],
|
||||||
|
})
|
||||||
|
expected_pnl = pd.DataFrame({
|
||||||
|
"date": [dates[1], dates[2]],
|
||||||
|
"portfolio_name": ["gold_port", "gold_port"],
|
||||||
|
"gross_exposure": [21000.0, 22505.0],
|
||||||
|
"net_exposure": [3000.0, 847.0],
|
||||||
|
"pnl": [3000.0, 835.0],
|
||||||
|
"cost": [0.0, 0.0],
|
||||||
|
"turnover": [1.0, 2.0502],
|
||||||
|
"n_positions": [2, 2],
|
||||||
|
})
|
||||||
|
|
||||||
|
pd.testing.assert_frame_equal(combo, expected_combo)
|
||||||
|
pd.testing.assert_frame_equal(positions, expected_positions)
|
||||||
|
pd.testing.assert_frame_equal(fills, expected_fills)
|
||||||
|
pd.testing.assert_frame_equal(pnl, expected_pnl)
|
||||||
|
|
||||||
|
|
||||||
|
def test_generated_alpha_combo_portfolio_execution_workflow(tmp_path):
|
||||||
|
daily_bars = make_generated_daily_bars()
|
||||||
|
|
||||||
|
computed_alpha = compute_alpha(
|
||||||
|
data=daily_bars,
|
||||||
|
alpha_name="generated_reversal_3d",
|
||||||
|
alpha_type="reversal",
|
||||||
|
lookback=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(computed_alpha.columns) == ALPHA_COLUMNS
|
||||||
|
assert not computed_alpha.empty
|
||||||
|
assert set(computed_alpha["symbol_id"]).issubset(set(GENERATED_SYMBOLS))
|
||||||
|
assert computed_alpha["date"].min() > daily_bars["date"].min()
|
||||||
|
assert computed_alpha["weight"].notna().all()
|
||||||
|
assert computed_alpha["weight"].abs().sum() > 0.0
|
||||||
|
assert {"ic", "rank_ic", "ir"}.isdisjoint(computed_alpha.columns)
|
||||||
|
_assert_sorted_by_symbol_date(computed_alpha)
|
||||||
|
|
||||||
|
alpha_a = make_generated_alpha_weights("alpha_a", zero_date_index=2)
|
||||||
|
alpha_b = make_generated_alpha_weights(
|
||||||
|
"alpha_b",
|
||||||
|
scale=0.5,
|
||||||
|
offset=0.25,
|
||||||
|
zero_date_index=2,
|
||||||
|
)
|
||||||
|
alpha_a_path = tmp_path / "alpha_a.pq"
|
||||||
|
alpha_b_path = tmp_path / "alpha_b.pq"
|
||||||
|
alpha_a.to_parquet(alpha_a_path, index=False)
|
||||||
|
alpha_b.to_parquet(alpha_b_path, index=False)
|
||||||
|
|
||||||
|
identity_combo = combine_alphas([str(alpha_a_path)], "identity_combo")
|
||||||
|
assert list(identity_combo.columns) == COMBO_COLUMNS
|
||||||
|
assert (identity_combo["combo_name"] == "identity_combo").all()
|
||||||
|
pd.testing.assert_frame_equal(
|
||||||
|
identity_combo[["symbol_id", "date", "weight"]],
|
||||||
|
alpha_a[["symbol_id", "date", "weight"]],
|
||||||
|
)
|
||||||
|
|
||||||
|
equal_combo = combine_alphas([str(alpha_a_path), str(alpha_b_path)], "equal_combo")
|
||||||
|
expected_equal_weights = (
|
||||||
|
pd.concat([alpha_a, alpha_b], ignore_index=True)
|
||||||
|
.groupby(["symbol_id", "date"], as_index=False)["weight"]
|
||||||
|
.mean()
|
||||||
|
.sort_values(["symbol_id", "date"])
|
||||||
|
.reset_index(drop=True)
|
||||||
|
)
|
||||||
|
pd.testing.assert_frame_equal(
|
||||||
|
equal_combo[["symbol_id", "date", "weight"]],
|
||||||
|
expected_equal_weights,
|
||||||
|
)
|
||||||
|
|
||||||
|
portfolio_weights = make_generated_combo_weights("workflow_combo", zero_date_index=2)
|
||||||
|
positions = construct_positions(
|
||||||
|
weights_df=portfolio_weights,
|
||||||
|
data_df=daily_bars,
|
||||||
|
booksize=2_000_000.0,
|
||||||
|
portfolio_name="workflow_portfolio",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(positions.columns) == POSITION_COLUMNS
|
||||||
|
assert not positions.empty
|
||||||
|
assert (positions["portfolio_name"] == "workflow_portfolio").all()
|
||||||
|
assert pd.api.types.is_integer_dtype(positions["position_shares"])
|
||||||
|
assert np.allclose(
|
||||||
|
positions["position_value"],
|
||||||
|
positions["position_shares"].astype(float) * positions["price"].fillna(0.0),
|
||||||
|
)
|
||||||
|
|
||||||
|
target_gross_by_date = positions.groupby("date")["target_weight"].apply(lambda s: s.abs().sum())
|
||||||
|
nonzero_target_dates = target_gross_by_date[target_gross_by_date > 0.0]
|
||||||
|
assert np.allclose(nonzero_target_dates, 1.0)
|
||||||
|
|
||||||
|
nonzero_share_counts = positions.loc[positions["position_shares"] != 0, "position_shares"].abs()
|
||||||
|
assert (nonzero_share_counts >= 100).all()
|
||||||
|
|
||||||
|
zero_gross_date = generated_sessions(10)[2]
|
||||||
|
previous_date = generated_sessions(10)[1]
|
||||||
|
zero_gross_positions = positions[positions["date"] == zero_gross_date].set_index("symbol_id")
|
||||||
|
previous_positions = positions[positions["date"] == previous_date].set_index("symbol_id")
|
||||||
|
common_symbols = zero_gross_positions.index.intersection(previous_positions.index)
|
||||||
|
assert not common_symbols.empty
|
||||||
|
assert (zero_gross_positions.loc[common_symbols, "target_weight"] == 0.0).all()
|
||||||
|
pd.testing.assert_series_equal(
|
||||||
|
zero_gross_positions.loc[common_symbols, "position_shares"],
|
||||||
|
previous_positions.loc[common_symbols, "position_shares"],
|
||||||
|
check_names=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
simulator = ReferenceSimulator(
|
||||||
|
constraints=[
|
||||||
|
SuspensionConstraint(),
|
||||||
|
PriceLimitConstraint(),
|
||||||
|
VolumeCapConstraint(max_frac=0.02),
|
||||||
|
],
|
||||||
|
cost_bps=5,
|
||||||
|
slippage_bps=5,
|
||||||
|
)
|
||||||
|
fills, pnl = simulator.run(positions, daily_bars)
|
||||||
|
|
||||||
|
assert list(fills.columns) == FILL_COLUMNS
|
||||||
|
assert list(pnl.columns) == PNL_COLUMNS
|
||||||
|
assert not fills.empty
|
||||||
|
assert not pnl.empty
|
||||||
|
assert (fills["realized_shares"] == fills["prev_shares"] + fills["traded_shares"]).all()
|
||||||
|
assert fills["blocked"].sum() > 0
|
||||||
|
|
||||||
|
fill_prices = fills.merge(
|
||||||
|
daily_bars[["symbol_id", "date", "open"]],
|
||||||
|
on=["symbol_id", "date"],
|
||||||
|
how="left",
|
||||||
|
validate="many_to_one",
|
||||||
|
)
|
||||||
|
expected_trade_cost = (
|
||||||
|
fill_prices["traded_shares"].abs()
|
||||||
|
* fill_prices["open"].fillna(0.0)
|
||||||
|
* 10
|
||||||
|
/ 10_000
|
||||||
|
)
|
||||||
|
assert np.allclose(fill_prices["trade_cost"], expected_trade_cost)
|
||||||
|
|
||||||
|
cost_by_date = fills.groupby("date")["trade_cost"].sum()
|
||||||
|
assert np.allclose(
|
||||||
|
pnl.set_index("date")["cost"],
|
||||||
|
cost_by_date.reindex(pnl["date"], fill_value=0.0),
|
||||||
|
)
|
||||||
|
|
||||||
|
booksize_used_by_simulator = positions.groupby("date")["target_value"].apply(lambda s: s.abs().sum()).max()
|
||||||
|
traded_value_by_date = (
|
||||||
|
fill_prices.assign(traded_value=fill_prices["traded_shares"].abs() * fill_prices["open"])
|
||||||
|
.groupby("date")["traded_value"]
|
||||||
|
.sum()
|
||||||
|
)
|
||||||
|
assert np.allclose(
|
||||||
|
pnl.set_index("date")["turnover"],
|
||||||
|
traded_value_by_date.reindex(pnl["date"], fill_value=0.0) / booksize_used_by_simulator,
|
||||||
|
)
|
||||||
|
|
||||||
|
metrics = evaluate_portfolio(positions, daily_bars)
|
||||||
|
_assert_metric_dict_is_finite(metrics)
|
||||||
|
|
||||||
|
|
||||||
|
def test_generated_workflow_outputs_keep_parquet_schema_contracts(tmp_path):
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=10, include_missing=False)
|
||||||
|
alpha = compute_alpha(
|
||||||
|
data=daily_bars,
|
||||||
|
alpha_name="schema_reversal",
|
||||||
|
alpha_type="reversal",
|
||||||
|
lookback=3,
|
||||||
|
)
|
||||||
|
alpha_path = tmp_path / "schema_reversal.pq"
|
||||||
|
alpha.to_parquet(alpha_path, index=False)
|
||||||
|
|
||||||
|
combo = combine_alphas([str(alpha_path)], "schema_combo")
|
||||||
|
positions = construct_positions(
|
||||||
|
weights_df=combo,
|
||||||
|
data_df=daily_bars,
|
||||||
|
booksize=1_000_000.0,
|
||||||
|
portfolio_name="schema_portfolio",
|
||||||
|
)
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, daily_bars)
|
||||||
|
|
||||||
|
assert list(alpha.columns) == ALPHA_COLUMNS
|
||||||
|
assert pd.api.types.is_object_dtype(alpha["symbol_id"])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(alpha["date"])
|
||||||
|
assert pd.api.types.is_object_dtype(alpha["alpha_name"])
|
||||||
|
assert pd.api.types.is_float_dtype(alpha["weight"])
|
||||||
|
assert not alpha.isna().any().any()
|
||||||
|
assert np.isfinite(alpha["weight"]).all()
|
||||||
|
|
||||||
|
assert list(combo.columns) == COMBO_COLUMNS
|
||||||
|
assert pd.api.types.is_object_dtype(combo["symbol_id"])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(combo["date"])
|
||||||
|
assert pd.api.types.is_object_dtype(combo["combo_name"])
|
||||||
|
assert pd.api.types.is_float_dtype(combo["weight"])
|
||||||
|
assert not combo.isna().any().any()
|
||||||
|
assert np.isfinite(combo["weight"]).all()
|
||||||
|
|
||||||
|
assert list(positions.columns) == POSITION_COLUMNS
|
||||||
|
assert pd.api.types.is_integer_dtype(positions["position_shares"])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(positions["date"])
|
||||||
|
assert not positions.isna().any().any()
|
||||||
|
position_numeric_columns = [
|
||||||
|
"target_weight",
|
||||||
|
"target_value",
|
||||||
|
"target_shares",
|
||||||
|
"position_value",
|
||||||
|
"price",
|
||||||
|
]
|
||||||
|
assert np.isfinite(positions[position_numeric_columns]).all().all()
|
||||||
|
|
||||||
|
assert list(fills.columns) == FILL_COLUMNS
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["prev_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["target_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["traded_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["realized_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["blocked"])
|
||||||
|
assert not fills.isna().any().any()
|
||||||
|
assert np.isfinite(fills["trade_cost"]).all()
|
||||||
|
|
||||||
|
assert list(pnl.columns) == PNL_COLUMNS
|
||||||
|
assert pd.api.types.is_integer_dtype(pnl["n_positions"])
|
||||||
|
assert not pnl.isna().any().any()
|
||||||
|
pnl_numeric_columns = [
|
||||||
|
"gross_exposure",
|
||||||
|
"net_exposure",
|
||||||
|
"pnl",
|
||||||
|
"cost",
|
||||||
|
"turnover",
|
||||||
|
]
|
||||||
|
assert np.isfinite(pnl[pnl_numeric_columns]).all().all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_frozen_real_fixture_runs_high_level_workflow(tmp_path):
|
||||||
|
real_daily_bars = pd.read_parquet(FIXTURE_PATH)
|
||||||
|
|
||||||
|
assert real_daily_bars.shape == (36, 19)
|
||||||
|
assert set(real_daily_bars["symbol_id"]) == set(GENERATED_SYMBOLS)
|
||||||
|
assert real_daily_bars["date"].min() == pd.Timestamp("2024-01-02")
|
||||||
|
assert real_daily_bars["date"].max() == pd.Timestamp("2024-01-12")
|
||||||
|
assert real_daily_bars.groupby("date")["symbol_id"].nunique().eq(4).all()
|
||||||
|
|
||||||
|
reversal_alpha = compute_alpha(
|
||||||
|
data=real_daily_bars,
|
||||||
|
alpha_name="real_reversal_3d",
|
||||||
|
alpha_type="reversal",
|
||||||
|
lookback=3,
|
||||||
|
)
|
||||||
|
reversal_vol_alpha = compute_alpha(
|
||||||
|
data=real_daily_bars,
|
||||||
|
alpha_name="real_reversal_vol_3d",
|
||||||
|
alpha_type="reversal_vol",
|
||||||
|
lookback=3,
|
||||||
|
vol_window=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
reversal_path = tmp_path / "real_reversal.pq"
|
||||||
|
reversal_vol_path = tmp_path / "real_reversal_vol.pq"
|
||||||
|
reversal_alpha.to_parquet(reversal_path, index=False)
|
||||||
|
reversal_vol_alpha.to_parquet(reversal_vol_path, index=False)
|
||||||
|
|
||||||
|
combo = combine_alphas([str(reversal_path), str(reversal_vol_path)], "real_equal_combo")
|
||||||
|
positions = construct_positions(
|
||||||
|
weights_df=combo,
|
||||||
|
data_df=real_daily_bars,
|
||||||
|
booksize=1_000_000.0,
|
||||||
|
portfolio_name="real_fixture_portfolio",
|
||||||
|
)
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, real_daily_bars)
|
||||||
|
metrics = evaluate_portfolio(positions, real_daily_bars)
|
||||||
|
|
||||||
|
assert not reversal_alpha.empty
|
||||||
|
assert not reversal_vol_alpha.empty
|
||||||
|
assert not combo.empty
|
||||||
|
assert not positions.empty
|
||||||
|
assert not fills.empty
|
||||||
|
assert not pnl.empty
|
||||||
|
assert np.isfinite(combo["weight"]).all()
|
||||||
|
assert np.isfinite(positions["target_weight"]).all()
|
||||||
|
assert np.isfinite(pnl[["gross_exposure", "net_exposure", "pnl", "cost", "turnover"]]).all().all()
|
||||||
|
_assert_metric_dict_is_finite(metrics)
|
||||||
@@ -284,7 +284,6 @@ version = "0.1.0"
|
|||||||
source = { virtual = "." }
|
source = { virtual = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "akshare" },
|
{ name = "akshare" },
|
||||||
{ name = "backtrader" },
|
|
||||||
{ name = "baostock" },
|
{ name = "baostock" },
|
||||||
{ name = "click" },
|
{ name = "click" },
|
||||||
{ name = "matplotlib" },
|
{ name = "matplotlib" },
|
||||||
@@ -293,24 +292,38 @@ dependencies = [
|
|||||||
{ name = "pyarrow" },
|
{ name = "pyarrow" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[package.optional-dependencies]
|
||||||
|
backtrader = [
|
||||||
|
{ name = "backtrader" },
|
||||||
|
]
|
||||||
|
joinquant-browser = [
|
||||||
|
{ name = "playwright" },
|
||||||
|
]
|
||||||
|
|
||||||
[package.dev-dependencies]
|
[package.dev-dependencies]
|
||||||
dev = [
|
dev = [
|
||||||
|
{ name = "coverage" },
|
||||||
{ name = "pytest" },
|
{ name = "pytest" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
requires-dist = [
|
requires-dist = [
|
||||||
{ name = "akshare", specifier = ">=1.14.0" },
|
{ name = "akshare", specifier = ">=1.14.0" },
|
||||||
{ name = "backtrader", specifier = ">=1.9.76.123" },
|
{ name = "backtrader", marker = "extra == 'backtrader'", specifier = ">=1.9.76.123" },
|
||||||
{ name = "baostock", specifier = ">=0.8.8" },
|
{ name = "baostock", specifier = ">=0.8.8" },
|
||||||
{ name = "click", specifier = ">=8.0.0" },
|
{ name = "click", specifier = ">=8.0.0" },
|
||||||
{ name = "matplotlib", specifier = ">=3.7.0" },
|
{ name = "matplotlib", specifier = ">=3.7.0" },
|
||||||
{ name = "pandas", specifier = ">=2.0.0" },
|
{ name = "pandas", specifier = ">=2.0.0" },
|
||||||
|
{ name = "playwright", marker = "extra == 'joinquant-browser'", specifier = ">=1.61.0" },
|
||||||
{ name = "pyarrow", specifier = ">=14.0.0" },
|
{ name = "pyarrow", specifier = ">=14.0.0" },
|
||||||
]
|
]
|
||||||
|
provides-extras = ["backtrader", "joinquant-browser"]
|
||||||
|
|
||||||
[package.metadata.requires-dev]
|
[package.metadata.requires-dev]
|
||||||
dev = [{ name = "pytest", specifier = ">=7.0.0" }]
|
dev = [
|
||||||
|
{ name = "coverage", specifier = ">=7.14.1" },
|
||||||
|
{ name = "pytest", specifier = ">=7.0.0" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "click"
|
name = "click"
|
||||||
@@ -493,6 +506,119 @@ wheels = [
|
|||||||
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" },
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "coverage"
|
||||||
|
version = "7.14.1"
|
||||||
|
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
|
||||||
|
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/54/fd/0ab2772530e946e1be1abd0bc09e647ec9b02e88f0867857601fefca8953/coverage-7.14.1.tar.gz", hash = "sha256:30c08f7d90415aa98b3c990385dea2939b0da55f38515e5b369b83655f8523be", size = 920132, upload-time = "2026-05-26T20:41:36.783Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/92/69/0d2ef01ff4b8fcecd4cba920d11e92fa4f96ae412441d3b56a90a258e69b/coverage-7.14.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:3e3680291c4a1d0dadfa84a2c459576a4af5133abb617905714339a0c73138cf", size = 219722, upload-time = "2026-05-26T20:38:14.002Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f8/ae/9afdeaa31b9d9ce98124b6abf8bb49119bf71aecae04f8567c189d91299f/coverage-7.14.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a5274669f37f2343635a347b91a60777621341ab3378e9c6ac9335eee704bddf", size = 220240, upload-time = "2026-05-26T20:38:17.424Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/69/c998589871df7ea7dba865cc5ee32b5a3e1d47ba6c68ef91104c7c46fa5e/coverage-7.14.1-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cfe5a5fec635799ef33428f1e5e61bafa45a92a96190ba731561ba558ccc214d", size = 246981, upload-time = "2026-05-26T20:38:19.266Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/10/1c7d04c13040dac531d21b712bbe08f902e6dd9b58f5d77875c4d030f8f2/coverage-7.14.1-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:62a9f70b52e0b5a95cfef4a5c5641b06983cadc5e538a3feeb5c00211f523ac2", size = 248812, upload-time = "2026-05-26T20:38:20.75Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c1/65/2a38a4607ef27cadcfbcee034dba5830ae2569f90144a0f4c7dbf47d30b0/coverage-7.14.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3c18ebc343e15be53049b3a2dce38fe82d58f37e20ab9094b3a39c0aa4f6bb47", size = 250675, upload-time = "2026-05-26T20:38:22.159Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/a2/a446ed9752a4a59b79e0fb6cbb319f6facb2183045c0725462625e66f87e/coverage-7.14.1-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b84ffdf877644e7096aa936991efeed873f7f3df57b9cd001312b7668ab08550", size = 252590, upload-time = "2026-05-26T20:38:23.63Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9e/fd/e81fbd7ba752365546e9842b1cbdaad3d6919d2a522c590aef16a281ec5e/coverage-7.14.1-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e854312c4103f2ad4c0dc023b69b77ebfd2c89db5f86c4c94dc2353f9a92167e", size = 247691, upload-time = "2026-05-26T20:38:25.057Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/53/35/f3c26fdaae9ea937d154ca4d372e5ea0a4167ff70d36c6074ac2eacb2f83/coverage-7.14.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:c643734307300234fafa36bf2a040a7235f8f177ea1fd6ec1423aea6fb7b929f", size = 248716, upload-time = "2026-05-26T20:38:26.406Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2e/14/940b6c49551fd343e8507ee2b0ba7af5d0aa04ed5bf768285cb7c72a9884/coverage-7.14.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:84ac9499e48700399a5dd0ea7085b5091961fec52c68d66b4ec0d3cf7f4441b1", size = 246721, upload-time = "2026-05-26T20:38:28.282Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/aa/2c/40fc0634186c28292a662dff578866b3913983d6c375a3c2a74020938719/coverage-7.14.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:7f02d09f70776579b926d889a4c9c235070a1f47c40458aeaca563fae5acfdb5", size = 250533, upload-time = "2026-05-26T20:38:29.753Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/e3/2c26bf1e811f9df991ff2a9bdddebdd13ee0665d564df7d05979f9146297/coverage-7.14.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:ce66d8e46da2bb5ee313a745cbd2e391d319176c1f7a9451bfcd3a2fb920859b", size = 246990, upload-time = "2026-05-26T20:38:31.516Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a8/b0/060260ef56bd92363ebdce0c7095ce422b06e69aae71828efeca473ab1ca/coverage-7.14.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:c912c259304cfb5ee584481cfb7ce1ff932b4d61e6c9140b8f19cb7b5ed82332", size = 247593, upload-time = "2026-05-26T20:38:33.065Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/63/f3/501502046efeb0d6d94b5ca54941d95f1184183dd6bdb7f283985783bb4a/coverage-7.14.1-cp310-cp310-win32.whl", hash = "sha256:1238cb94638e610e972c60dac68e813f868dc7d6e982535270558443058d9d59", size = 222330, upload-time = "2026-05-26T20:38:35.36Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/5d/1bf99f2c558f128faf7906817ccbdb576ba815d3b41ce2ac1719b70a3663/coverage-7.14.1-cp310-cp310-win_amd64.whl", hash = "sha256:fc459e5d73be2d6332fcfe8dbf3d8994671fe33c700f4565988ecfa511547253", size = 223261, upload-time = "2026-05-26T20:38:37.196Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/d7/477ad149490e6cb849f28abea1dabb9c823cea72e7500c81b4240ce619c0/coverage-7.14.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:478b5bcd63c2e1357c5c7e16c070690df7b07f676b1c114d7b93e533c664309f", size = 219848, upload-time = "2026-05-26T20:38:38.715Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/91/82/a5eb47257c50601bb7b9a9d2857c67b7a3a85ad74180eb2c98bb1fbe0ce5/coverage-7.14.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a24a81f9715ee42ef59a316cc11611c98fe23920f7c81861315c9f3ff4a230f4", size = 220354, upload-time = "2026-05-26T20:38:40.232Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/8b/78419b5391a5cb706b6544390507e469d83ffc9a8248b02c4011aceb9365/coverage-7.14.1-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:196a13319ad88d6d8ef5ab489ec4f44ddde2143c0c7d5b27786f6c3ffd56a7e1", size = 250771, upload-time = "2026-05-26T20:38:41.782Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/77/63/e77aaacd491182210d639636b7a8bba23ffffa9b82aa3762da9431855fa9/coverage-7.14.1-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3d452fd08b5c72c5167c93e6867b5c08500bd40f2a21e1e854a500550b6cc36f", size = 252683, upload-time = "2026-05-26T20:38:43.305Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/65/1c/a022e3cfbec2ac241640003cb3a817e161d9c7f5aa9b49173756cdc03204/coverage-7.14.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:23bf7fa51ac02e07fc7c96849b82946da47ae862dc8f86d183b2a4864fc38129", size = 254791, upload-time = "2026-05-26T20:38:45.361Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/61/d6/967e408aca4c1ceb88cb0cc677169110ae7f5995fb5eaf5fb1f5a1bb8f5d/coverage-7.14.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bcaa50684dcaadfa599ac48f81103c756d791cfd85c97203d2217c593d48b860", size = 256748, upload-time = "2026-05-26T20:38:46.91Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/be/869188f7fe28638078ec479331ace6dc5f7b40b7153eb616f47ab79404d8/coverage-7.14.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4ea1c034f95c9b056e856b794630b17f9fa3d57e4800ff1e503d3be0f9c9078c", size = 250907, upload-time = "2026-05-26T20:38:48.493Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/07/aa/adb7d3b4278d690e68703abcd76ab1b948242e3668d921711551b78f9ddb/coverage-7.14.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c7e057326434e441306226fbeb5d1aaf14a2637efe97ba668306635835f32ad7", size = 252483, upload-time = "2026-05-26T20:38:50.074Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/61/331c74103c62dcb0c4b9b3a0de9a61aca016208b0a90f109592a9f9ecc28/coverage-7.14.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:59baf88468dbc8d63b1887afd92bda52e40bb1561696e5819670601403810cec", size = 250545, upload-time = "2026-05-26T20:38:51.613Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/b6/c5dae3c104d89be04828f61810e6b3473825482e4c288cc4ed04553e08ae/coverage-7.14.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:d34d75f892b3ab73ba11cab5442cce7b3e168fd64162b16f0e1e0d09c508edef", size = 254310, upload-time = "2026-05-26T20:38:53.503Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ad/a1/2b9d5863e3b83c01ad8199e3c597802fbb3a9dc90b058885804c20296d31/coverage-7.14.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:3a56abc20a472baf0304c455721bc601477440d28ecfde8a03dde79ede07e0df", size = 250266, upload-time = "2026-05-26T20:38:55.414Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/5e/0e511fbdb269359be26fe678a1c3fa1f2aa2a01573cc3f54268c8d6d4797/coverage-7.14.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:6a3cb83d1552c0cd1b4906655b6a33fd4a8473229633a901c6b73bf86914dee9", size = 251174, upload-time = "2026-05-26T20:38:57.141Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/85/10/e55307b622b3dd9671cb321824502dc10f93e72f2802b9946159a8edadeb/coverage-7.14.1-cp311-cp311-win32.whl", hash = "sha256:10274a1fbeb8ec5d72966e17bb198a3104257aca4ac09d98667c5f8aca8c8548", size = 222354, upload-time = "2026-05-26T20:38:58.727Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/cf/107421693cfb71e4f1ca5bf70443f64d4161878068d07a3e51c7ad21d17b/coverage-7.14.1-cp311-cp311-win_amd64.whl", hash = "sha256:87ebdf787d4888e3f3f2d523eadc6e18c6d18c6d0eb173801a189641627fb37e", size = 223290, upload-time = "2026-05-26T20:39:00.413Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/1d/3e3644585eb29e9dafefb19555078529a4d7cce12bd21929664eea989277/coverage-7.14.1-cp311-cp311-win_arm64.whl", hash = "sha256:dd34767fa19848d35659ffc0a75314f58c7af3f1cd87ec521e8292a1238398a3", size = 221953, upload-time = "2026-05-26T20:39:02.159Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/b7/bdbb725ba02c5b42825b200c940f38b7a54fcad24627b7192f78f8110d76/coverage-7.14.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a06c76364a9360e33d6d23769aefdf7f66f38e2ffb60ceb1baaa4989d83b695c", size = 220022, upload-time = "2026-05-26T20:39:03.702Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/72/81/fdc0898a55c6219223291ec1a1fe89966ef212ce82276aa0899df84b5de0/coverage-7.14.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fad54e871165f6ec2f536063ac74c3104508a12963e64072ba44bd822de52b0c", size = 220379, upload-time = "2026-05-26T20:39:05.381Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/72/de048c4a25e13bce59ac6a339351c10bdf2515e07459afcdaf04dc3143a2/coverage-7.14.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:84b535f00655ecafe1d929d1fb00ed5d6fa3051ea643ab2c161a3887b86f294b", size = 251888, upload-time = "2026-05-26T20:39:07.367Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/28/30/300c343f68beb9d4cbb64ec81e58c5b6b80b56927f72d2b38654ac26e013/coverage-7.14.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6b6b0853b895fe0e98cbfc580d1ec3393d9302b4b1e96a77b3f5c91fdab899e6", size = 254624, upload-time = "2026-05-26T20:39:09.037Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b1/ed/7b25642496e8170b6bac14adce00537c6e5fa2d586159401a4de3e8b49e6/coverage-7.14.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:442cc9c952b2df400cda54bb04ab87330cf2cd08a8692cbbea36773531eb6f37", size = 255739, upload-time = "2026-05-26T20:39:10.889Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/a2/abd210b8c4e29c24e4624916db97bb519097a91034aaeb767f937e7da794/coverage-7.14.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8270544c361ed405a27a060dbc9ed2c124b084d96dfdc2d9a2510482aef981ad", size = 257998, upload-time = "2026-05-26T20:39:12.722Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/24/7c50beed3792fe62f6ce0545c6686ce83379719e2c0276179333d97eae92/coverage-7.14.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:48b283b1dd6372e8de2a7a9a4c4d5dc06f4d4fd209b876f3c88a7a205a0c8f84", size = 252296, upload-time = "2026-05-26T20:39:14.259Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/15/05/0f874628ebcbfc77ead559ff210281ef06a97db08481832e7dd39274a135/coverage-7.14.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5b0c99ba93a07d56f6df340bb79be53202a082b2fdb81bfe6190b741a3470d54", size = 253658, upload-time = "2026-05-26T20:39:15.923Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/99/6f/ca6ad067364b337ef997802115e7ecad2abd2248b05471464b0dea02b4d4/coverage-7.14.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:e471bc5769ff073b058cfadb0d736b56ce067c8560eabeb0da88462df98c23e7", size = 251803, upload-time = "2026-05-26T20:39:17.537Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c0/30/b9b4d377cd9f40baf228068f5a81faf8450c6228503011bd499708483a50/coverage-7.14.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f497a1ea81d4cd7c10ddcaa685135b9aabd291af3d55775a9ddf3cb7a364cdd9", size = 255873, upload-time = "2026-05-26T20:39:19.414Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3c/21/7c721a9e5e6bb88547d30a787aefb97512d3f54c1324c7488d9b3743f7f9/coverage-7.14.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2222be86d0b54f5dd5a38f45f17f315f737245e857bf0bdedc70734f84a13c02", size = 251372, upload-time = "2026-05-26T20:39:21.169Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9d/8c/f8ae5a2200130e1503cd7661a6cd3b2b7bacef98277fbf3571fb13f8b766/coverage-7.14.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:85e85586565842f6932abebd4c18bcb1074223dc0b3576e7d173ca710622813a", size = 253245, upload-time = "2026-05-26T20:39:23.097Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/34/62/70a9024672a5f6910517d9628c52c9afbdd3cf8f46426af52bb148a56fff/coverage-7.14.1-cp312-cp312-win32.whl", hash = "sha256:4a28fd227808366b196a75476dced2eb35b351d6766ba9c858dc93319e87f4f1", size = 222567, upload-time = "2026-05-26T20:39:24.868Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/81/8b7cd386839b039ebe1855733b9f9449a8dec5d79564018234f185a7fa70/coverage-7.14.1-cp312-cp312-win_amd64.whl", hash = "sha256:54acdb6674a4661768d7bf7db32dfb9f46ab1d764f8aba6df75ce1a6a088724e", size = 223372, upload-time = "2026-05-26T20:39:26.603Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ae/ba/b44d472022f620d289d95fa830143235c0c36461c6f2437ea8d51e5481ed/coverage-7.14.1-cp312-cp312-win_arm64.whl", hash = "sha256:99cd41ff91afd94896fea3bc002706b6ae4ce95727d06e4a0f39c0a8d8bd8b1a", size = 221989, upload-time = "2026-05-26T20:39:28.242Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/9e/5f6d56327c62b185225d145191c607e07515294a0aa6338e58805cd4a5ac/coverage-7.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:be9f2c802dcfce3f71298303aa5dad0dce440a76c52f2f60dacd8656dab78793", size = 220044, upload-time = "2026-05-26T20:39:29.902Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/75/92/e82aca356744cbbc0f77a0b623e38918c1872361963413a3bab5d0340393/coverage-7.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6223a72fd0e4c7156353ec0f08a5f93623e1d3034d0e2683b9bb8ea674131b1d", size = 220412, upload-time = "2026-05-26T20:39:31.561Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/c9/385bde0bf7ed0f4bf3a7ee5367060a86b5d218718cfd6fb943c0f836b34f/coverage-7.14.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:7279d2110a28cebc738b6459ecda2771735a4c18465fbbd36b3288fe5ed92247", size = 251412, upload-time = "2026-05-26T20:39:33.337Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/8c/23faf6a2343a0d17f960a4bd56c43bc7eb4cf312f774dd6ceebd82c7d8fc/coverage-7.14.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9eeb3fcbc13ba40dfbdb22d01d196a28e9cef9ed4c29b60061a1e0e823a9929d", size = 254008, upload-time = "2026-05-26T20:39:35.009Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/42/06/36f4aa9ca8a815e6036156e80706a67828bb97bd826948244f6996dda957/coverage-7.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f0cfc27c539f07cf5c0a4cfe211d0b6cae039f8f40526dbaa71944e64b50a7b", size = 255241, upload-time = "2026-05-26T20:39:36.71Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/79/95266316352f90f6b1c6736bb413302edfde2453fb32422d3911642691b3/coverage-7.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:221c70f316241a78e77e607c227cefc8808d4e08f28d99c04f35694690e940be", size = 257373, upload-time = "2026-05-26T20:39:38.412Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e3/9c/58316d1f66c488b5fca8a0eb3e98348807813efa8a0d0833b9021be27488/coverage-7.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:da028256b04ec30e5e0114b6f76172938c313991f0a2d3d894271315cf5d5e43", size = 251635, upload-time = "2026-05-26T20:39:40.268Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ef/5a/ca2398a568e16fed7bb713e84ba3603a7164fb65779abe645c565ec890d5/coverage-7.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:76a085d7005236a767e3426148b2c407e53ad61695c562f8a81da2d373324901", size = 253373, upload-time = "2026-05-26T20:39:42.145Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6e/2c/0396562c32deaebe7be51d865b3a41e9a87d7561acafe1a28f53b07e019a/coverage-7.14.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b553d04b5e778a8e56d57eb134aff42a92718ecba45e79c4764ecfa40efd92ff", size = 251341, upload-time = "2026-05-26T20:39:43.907Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fd/8f/a94f9221184c9cae1ee115820e3798e48b6b17777a9f19e46fb9a0c8dc74/coverage-7.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:46f714d2fb8ae2f4f29f23ada7f1e79b759fff5a70f94a1dac23af204c3ec9e4", size = 255497, upload-time = "2026-05-26T20:39:46.166Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/69/505d70e47db1eaebcd002c39759707621ef184cd6b1ae084d9f41293f323/coverage-7.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:1896f5e19ff3f0431c7ce2172adc54890fd97f86b59ced8ca1649145d9ffe35d", size = 251159, upload-time = "2026-05-26T20:39:48.03Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e0/aa/58681c383aa33a9d2ed40a02d7a22fbf780d1fa4d575396365777828198c/coverage-7.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:62fd185ef9df3c33d1c8178c5af105f762afbad96038de9a4ae100aa6297ca33", size = 252934, upload-time = "2026-05-26T20:39:49.872Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/eb/fd/11c928cd6bdffc7074bb5965c173d9ebf517fb00205e1da524b98d29ef92/coverage-7.14.1-cp313-cp313-win32.whl", hash = "sha256:ab4af6352741a604c431c6072fce5bee33bf0f20dc7a56618d6bf6bb89e9810c", size = 222584, upload-time = "2026-05-26T20:39:51.68Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6f/92/fb416fc26d340dcba19518c418d6048e913186e17243982c5e435e41fa7a/coverage-7.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:7af486dabe8954d03b087f0021540897afe084f04e16ff5579e08cc46f871416", size = 223394, upload-time = "2026-05-26T20:39:53.472Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/73/c6/02d56e3867972f77d5036de924643f26c056e848f00452cafb4dbc3c29b4/coverage-7.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:2224f89ffd0c5605ccce1ed7a584da162bc7c55f601ab1c946bc9de31a486b42", size = 222015, upload-time = "2026-05-26T20:39:55.374Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/4d/9e/fcc77914050df73f7662fa1f00902774c79c075a8388ab334074574bf77e/coverage-7.14.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:de286598cc65d2b489411174b1faec2f5a7775fb3201fd925db2a76b4030f37d", size = 220733, upload-time = "2026-05-26T20:39:57.189Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f7/67/2963cbdaf5cbadec44efa3a1e39eaa1f02df4079585f05387607a221e126/coverage-7.14.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:042c46ded7c288aeb07cf14a28b6c1e10b78fcba40171c3fa1e939377eeef0b5", size = 221086, upload-time = "2026-05-26T20:39:59.019Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c8/c5/8701645574e11881f2f47d8930f98bc48b5d43b25eb5b4430dfc4a2f9f48/coverage-7.14.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:f4ddbe407477f04c45115d1a4e5bc480f753553b534d338d4c3358b1cdd0ea52", size = 262381, upload-time = "2026-05-26T20:40:00.822Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7c/28/7a64d73598263e0c5abd5084211a8474488d31b3c552ff531c719dfcff62/coverage-7.14.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d13e6725992e2d2fd7d81d4f5241952d13740121dfd501da09201be39b2c003a", size = 264458, upload-time = "2026-05-26T20:40:02.506Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fa/d8/4969179db9f7eb4df218e69540adf829d1c835f59452513d065d15446802/coverage-7.14.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f747dc8edcfe740130f28f32f3995e955494285717e86ee25af51db2219df08a", size = 266884, upload-time = "2026-05-26T20:40:04.421Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a6/78/a45d5794dbc9bafd97afc96a4377c86c7820d78b6cf51b89bc1d4e919275/coverage-7.14.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ced2f09ef276fd58611a1ef502164ad266d2b75174e5a40cabbdb4033f9f6cf2", size = 268022, upload-time = "2026-05-26T20:40:06.298Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/21/cb/4f5e354e9e3e67af96bd4e57113e6db6b22298c7168b13eec408a549903d/coverage-7.14.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b84800013769a78ccb9ef4659402e26d06867e337b61ec365f77ad008adea80e", size = 261631, upload-time = "2026-05-26T20:40:08.226Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ec/49/eced49af4cb996d5d8b7e94e736175c513e4facd3398507b89892b4326d8/coverage-7.14.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:ea8cd6ca0ee9f616aaef3afc6882e32c2cbf18b00d96313ffd76af650574034d", size = 264443, upload-time = "2026-05-26T20:40:10.137Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f1/d8/5603a88a7c5913a6b54f6cb1a8c46f7b39cbb30f27cd3f492908da09b2d7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:aa5e304a873fabddc11e484e9b6b738bd38bd7bed17b09aa84eecf5332e8b8bb", size = 262069, upload-time = "2026-05-26T20:40:11.999Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f0/59/2ae3cb79da554a06c8619d6c88ea19dd1e4aed4b834b6a83bb1fa243bdc5/coverage-7.14.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:5a1c5215be81035e629d5bc756650634d0bf31991038db7a0eccb90f025ce16d", size = 265780, upload-time = "2026-05-26T20:40:13.858Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/af/5f/b130c1dc999031f2648bd25317fbce505ad8d5562079b4ed81e736a84967/coverage-7.14.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:79058c47dae6788504b5effb319961bcd72d7240551464b91d474bc0ed186d69", size = 260970, upload-time = "2026-05-26T20:40:16.142Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/87/d1/ec13ccddeb48ec963bdfa72a11224bac2584bd045ba13beca82f8113e9c7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:370c5afae3fa0658e11694a32b24c2778f6bc2d17718121f94ee185e69f26b54", size = 263157, upload-time = "2026-05-26T20:40:18.382Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cf/c2/cd91ead503045161092d3845f7bb95ea2f25131ce96d3e314dd835d91b9c/coverage-7.14.1-cp313-cp313t-win32.whl", hash = "sha256:3758dd0a7f1fa57365ef2e781df0f0731d38b6e3772259d13dae4bd8a958d4b1", size = 223259, upload-time = "2026-05-26T20:40:20.381Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/9f/1e28d97e6bd2c76b07f38b7c02870f1371255ff6717f54eca578fcbbdd0e/coverage-7.14.1-cp313-cp313t-win_amd64.whl", hash = "sha256:6ff665fb023a77386fe11685190cee1f60a7d635994a30d9b0a061533d470fce", size = 224320, upload-time = "2026-05-26T20:40:22.316Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a9/e0/d936e908f0e1efa55e52b91e01b52f1055cef5e1ab2718493390ed8e2fb8/coverage-7.14.1-cp313-cp313t-win_arm64.whl", hash = "sha256:17a5a241e5997621a956a7f402a7433ef4221e5152809b785bec79e2323799f1", size = 222577, upload-time = "2026-05-26T20:40:24.894Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d6/34/fc2f101b151af3799a101f0550b0454aa008afdc0add677394ec4aa8ea10/coverage-7.14.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d5ed429d0b8edaac649e889b4ffcedb6c80b06629a3f93050e3dddfb99235bee", size = 220091, upload-time = "2026-05-26T20:40:27.249Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/a7/1ebae2ab5b961b5c79bb09fe7b3ac99edb190d8be4a8c510b2cf66f46468/coverage-7.14.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8011224a62280e50dab346960c03cf47aca1a1e09e608c0fb33fd6e0cc8e9500", size = 220421, upload-time = "2026-05-26T20:40:30.084Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5e/90/92aca9cf0acc95123c96cd1eb1f08917897a7f5dee01e15738922971ec31/coverage-7.14.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:12c42ec1e14f553c4f817e989365982e646e27211f10a0f717855b94a79c8906", size = 251466, upload-time = "2026-05-26T20:40:32.542Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/26/2b/78048cbe3b999f6cbf9cc0d90abba6a88a3e0863a8c1c6cbc762f3f8802f/coverage-7.14.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:06144cd511cf2624873a035c5069cf297144f6e77a73ee3d7a55b605ec5efb42", size = 253973, upload-time = "2026-05-26T20:40:34.473Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/21/c2e33b29d1cfde484a19d437afc343c6cd30b08d78cbbf9f5aff14e57b2b/coverage-7.14.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a311d8e1da24be5c1ccf85cbfb06315dbaa1703d5a1eab3f6432c72b837917c8", size = 255318, upload-time = "2026-05-26T20:40:38.154Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/ee/aad2f108d63b769121005302f16bf66db8625c88ceaba466942e09a2607e/coverage-7.14.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c79cead5b5bc584d9c71451cb984d0e3a84e0c0937379c8efcbf27c8d661b851", size = 257633, upload-time = "2026-05-26T20:40:40.164Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c2/f8/11a2c29b4fd76d9849f81d0bb812ec0017a9396df3217214e38934a8c837/coverage-7.14.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:dcbf65f1f66a26cdd88c35cf68fb4729c5d1cd2e88added72420541dfb212034", size = 251488, upload-time = "2026-05-26T20:40:42.631Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/b8/9a5820de4b8ac2b71d85e3b5fb49108d7469c665f0e2ad0dd7569023e305/coverage-7.14.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fd86572566fb40189a8260446158235159bc7a82dfbc87a3b39cf4fb57fcec1c", size = 253329, upload-time = "2026-05-26T20:40:45.208Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6b/ff/f33e4823667e27548e8fd8df44217515303f9808d0ff29817db56f87d990/coverage-7.14.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:7771b601718fdde84832c3a434ca9bbf4ae9adbc49d84198b4110700c3c77c36", size = 251291, upload-time = "2026-05-26T20:40:47.502Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/68/9b/489db0ebb209054766b90a9014a45f6d26eb724c02ec21311c3733b5a644/coverage-7.14.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:39b21e212c55af06fa375e3dbf90a8a8e38792f3a910c580066d23563830ddd5", size = 255564, upload-time = "2026-05-26T20:40:49.372Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/b5/16bc2d4c2409b23c7737edb68c83bc89e345f378050549fe1d75ac7d34d5/coverage-7.14.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:f2302660e32562a532b442480121aef8aa61a5bdb20b30bf0adab29f10a5a4b4", size = 251107, upload-time = "2026-05-26T20:40:51.677Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/0c/2629997469a00cd069d588a41c9dc887610f2775ae89d250c4791e65272a/coverage-7.14.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:03a6f93c1ec3b7f2e77b5dbcc5573a2c21f12529a5c6bbe0f16f72303cc2fa4d", size = 252764, upload-time = "2026-05-26T20:40:54.267Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d2/ee/f78d63c8f079e0d7211c7e2401fa17e311514534ba61bae03e4b287ce4ab/coverage-7.14.1-cp314-cp314-win32.whl", hash = "sha256:8a3ce026d73290f42f08dafecbd82c193a74df280461fbf97300fec51fd133ee", size = 222837, upload-time = "2026-05-26T20:40:56.496Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/dc/b9/be539854f93a70dfbeec69117f33ec70dc42ff0b65b5b07ab8d40d04228e/coverage-7.14.1-cp314-cp314-win_amd64.whl", hash = "sha256:114c95ef29302423b87d159075805f4ab973254a2638a5d7d046c94887cc87d7", size = 223650, upload-time = "2026-05-26T20:40:58.351Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fe/9e/24e2842fef40f35ac82ba3a7719c8023d011bf3bf652d0675316a9d088a1/coverage-7.14.1-cp314-cp314-win_arm64.whl", hash = "sha256:a07891c3f4805442b31b71e84ba3cf29ed1aa9a428284e06deeb4b23e5b46343", size = 222218, upload-time = "2026-05-26T20:41:00.321Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0a/1d/ac0a9df5fe31c1e8bdd658074905fc12844a05c1a7e3fdb8417e97c31e23/coverage-7.14.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:1101a5ebb083aecb625ebb6209d4105b58f647b093cb2dc8122d7b33f743cfe1", size = 220822, upload-time = "2026-05-26T20:41:02.281Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/32/cf/f964fd9aff20323f9f1a726c97135f8a76bcd87b92dad141a456a43f3c64/coverage-7.14.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:851b9e1e4e8a4608e77c79714b2e77c0970d2ed7202a05e92ae407817481887b", size = 221084, upload-time = "2026-05-26T20:41:04.593Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d8/5e/7e5ef2aba844de2b80d678619fcf0841b42e3f37f16411226f3fe4c1016f/coverage-7.14.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d5b89cdfb2ee051b71e8c3c70bd81a9eff81100f736a269136fe1a68efe00474", size = 262454, upload-time = "2026-05-26T20:41:06.641Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/64/62/75809bded87015cc4935524218a2a8ed8dd1a8498bfed30a2f4f7a4b4d34/coverage-7.14.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0177614a0370f227888b4e436a7c55686d6a9f90eb1ade2b624ba685a1686e86", size = 264578, upload-time = "2026-05-26T20:41:08.556Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f3/42/d33392dc14633525012d2d504fa1a33b05538bf535f5c1d64675e5754b78/coverage-7.14.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d69af5dea2de76fc485a83032a630523f985198b7e25be901ec60181587b01e", size = 266981, upload-time = "2026-05-26T20:41:10.824Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2a/49/0157c4428c2aca7f1e09d5565930586fd5ae36f1655f08b0daa7cf1fcae1/coverage-7.14.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:35ab22d91de736e8966b980dc355cbcdd2c6dbbcfe275f9a2991bc8a91b3df65", size = 268112, upload-time = "2026-05-26T20:41:12.966Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/96/26/86b9ce71f4092b1ed325ce1421698081df1286b833400b6836912834d6e0/coverage-7.14.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:357d4e32935c36588aaba057d734fa32428c360c9fc2e4442afbf1b646beee6e", size = 261558, upload-time = "2026-05-26T20:41:15Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/20/4c/c311210c5472cf5401d8422b0d7812cdd520f24417673afabda6c323faca/coverage-7.14.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:51bd64741cc6fa065abd300ede1afe5a5291ece9c31da8b24884deda48bcc3f8", size = 264447, upload-time = "2026-05-26T20:41:17.369Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/71/59513f8710ed3e6b0ac0a050a5b7e977bb9c9e880354863b5d00d8809256/coverage-7.14.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:9132cd363a68a4c3daa7c8704a654b1e39d3360f6f5b8ddd470608a945236c07", size = 262048, upload-time = "2026-05-26T20:41:19.309Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/84/8d/bceed32dc494f5bbf50f775cd2e78ca814953942b5ea28d3c1c3ac316f14/coverage-7.14.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:07c6290b1697b862c0478eab545eec949a0d0e4d6d03497f446d706da3b4f2de", size = 265781, upload-time = "2026-05-26T20:41:21.559Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e7/c5/9348fe40dbfd4991aaf78df2c6c3098bfb2cc834d1fd362a64b4efef855a/coverage-7.14.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:5ea0c297e27133853b4d8a3eb799bff5a2dbd9f2f41537a240d337ac9b4df890", size = 260896, upload-time = "2026-05-26T20:41:23.428Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/92/1ea0f03929da7cf87206b1fa24f4c8e9c158be0455481af29ec0a1f3503f/coverage-7.14.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:01b7733daad0237daa01ef80fe2dfceffc911e6a17fa7b55d14aa8214eaaaecd", size = 263214, upload-time = "2026-05-26T20:41:25.419Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/a9/b2493c054c0e01a643266742ab45e15744e60743f9260cd930c7142b1124/coverage-7.14.1-cp314-cp314t-win32.whl", hash = "sha256:6adc5a36984624a70bf11d7184e20fa0a49aa7c47ffab43804106a1a695ea22e", size = 223624, upload-time = "2026-05-26T20:41:27.795Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/bd/3e1e6a57fccd2d7c83fcdf338e93ba98eb85c6e877dd34731ac585375490/coverage-7.14.1-cp314-cp314t-win_amd64.whl", hash = "sha256:ddf799247318f34dbcd2efa8c95a8d0642674e926bb1774cf9b63dfd2a389d1c", size = 224728, upload-time = "2026-05-26T20:41:30.098Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bb/d7/31066cf1d2f0c6c797fce911bcfa01dd35642dc6da992a950256097c5860/coverage-7.14.1-cp314-cp314t-win_arm64.whl", hash = "sha256:145986fe66647eb489f18d9a997567a3fd358584c4b5a808769113abc07466af", size = 222752, upload-time = "2026-05-26T20:41:32.123Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/3c/1a983b9a745d7f83d53f057bcc5bf79ba6a2bbc08266b3f0c7d6fe630c9b/coverage-7.14.1-py3-none-any.whl", hash = "sha256:a252f21c27e38347e60111a3266b03827422a7d5525951aceee313aa68bab1d2", size = 211815, upload-time = "2026-05-26T20:41:34.078Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "curl-cffi"
|
name = "curl-cffi"
|
||||||
version = "0.15.0"
|
version = "0.15.0"
|
||||||
@@ -622,6 +748,92 @@ wheels = [
|
|||||||
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" },
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "greenlet"
|
||||||
|
version = "3.5.3"
|
||||||
|
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
|
||||||
|
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e2/f1/fbbfef6af0bad0548f09bc28948ea3c275b4edb19e17fc5ca9900a6a634d/greenlet-3.5.3.tar.gz", hash = "sha256:a61efc018fd3eb317eeca31aba90ee9e7f26f22884a79b6c6ec715bf71bb62f1", size = 200270, upload-time = "2026-06-26T19:28:24.832Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/a1/1f7f0c555f5858fd2906fe9f7b0a3554fddb85cb70df7a6aaec41dc292c2/greenlet-3.5.3-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:c180d22d325fb613956b443c3c6f4406eb70e6defc70d3974da2a7b59e06f48c", size = 285838, upload-time = "2026-06-26T18:21:05.167Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0a/29/be9f43ed61677a5759b38c8a9389248133c8c731bbfc0574ecdff66c99fc/greenlet-3.5.3-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:483d08c11181c83a6ce1a7a61df0f624a208ec40817a3bb2302714592eee4f04", size = 602342, upload-time = "2026-06-26T19:07:06.908Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b9/42/ba41c97ec36aa4b3ec25e5aa691d79561254805fad7f2f826dd6770587e2/greenlet-3.5.3-cp310-cp310-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1dae6e0091eae084317e411f047f0b7cb241c6db570f7c45fd6b900a274914ce", size = 615541, upload-time = "2026-06-26T19:10:04.909Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2e/8c/231ca675b0df779816950ca66b40b1fa14dbff4a0ed9814a9a29ec399140/greenlet-3.5.3-cp310-cp310-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0f6ff50ff8dbd51fae9b37f4101648b04ea0df19b3f50ab2beb5061e7716a5c8", size = 622473, upload-time = "2026-06-26T19:24:12.786Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/c7/28747042e1df8a9cd120a1ebe15529fc4be3b486e13e8d551ff307a82412/greenlet-3.5.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9bcd2d72ccd70a1ec68ba6ef93e7fbb4420ef9997dabc7010d893bd4015e0bec", size = 615675, upload-time = "2026-06-26T18:32:14.444Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/81/fe/dd97c483a3ff82849196ccd07851600edd3ac9de74669ca8a6022ada9ea1/greenlet-3.5.3-cp310-cp310-manylinux_2_39_riscv64.whl", hash = "sha256:37bf9c538f5ae6e63d643f88dec37c0c83bdf0e2ebc62961dedcf458822f7b71", size = 418421, upload-time = "2026-06-26T19:25:34.503Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cc/a8/b85525a6c8fba9f009a5f7c8df1545de8fb0f0bf3e0179194ef4e500317f/greenlet-3.5.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:73f152c895e09907e0dbe24f6c2db37beb085cd63db91c3825a0fcd0064124a8", size = 1575057, upload-time = "2026-06-26T19:09:00.264Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/03/79/fb76edb218fe6735ab0edeba176c7ab80df9618f7c02ce4208979f3ae7db/greenlet-3.5.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8bdb43e1a1d1873721acab2be99c5befd4d2044ddfd52e4d610801019880a702", size = 1641692, upload-time = "2026-06-26T18:31:41.454Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6b/79/86fe3ee50ed55d9b3907eecd3208b5c3fe8a79515519aae98b4753c3fa1d/greenlet-3.5.3-cp310-cp310-win_amd64.whl", hash = "sha256:0909f9355a9f24845d3299f3112e266a06afb68302041989fd26bd68894933db", size = 238742, upload-time = "2026-06-26T18:20:40.758Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/58/5404031044f55afad7aad1aff8be3f22b1bed03e237cfeabbc7e5c8cfde0/greenlet-3.5.3-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:aca9b4ce85b152b5524ef7d88170efdff80dc0032aa8b75f9aaf7f3479ea95b4", size = 287424, upload-time = "2026-06-26T18:20:31.469Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/bf/1c65e9b94a54d547068fa5b5a8a06f221f3316b48908e08668d29c77cb50/greenlet-3.5.3-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0f71be4920368fe1fabeeaa53d1e3548337e2b223d9565f8ad5e392a75ba23fc", size = 606523, upload-time = "2026-06-26T19:07:08.859Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/c7/b66baacc95775ad511287acb0137b95574a9ce5491902372b7564799d790/greenlet-3.5.3-cp311-cp311-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4d77e67f65f98449e3fb83f795b5d0a8437aead2f874ca89c96576caf4be3af6", size = 618315, upload-time = "2026-06-26T19:10:06.055Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b0/a0/68afd1ebad40db87dac0a28ffa120726b98bf9c7c40c481b0f63c105d298/greenlet-3.5.3-cp311-cp311-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e18619ba655ac05d78d80fc83cac4ba892bd6927b99e3b8237aee861aaacc8bb", size = 626155, upload-time = "2026-06-26T19:24:14.44Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/78/2b/28ed29463522fdbe4c15b1f63922041626a7478316b34ab4adda3f0a4aba/greenlet-3.5.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8540f1e6205bd13ca0ce685581037219ca54a1b41a0a15d228c6c9b8ad5903d7", size = 617381, upload-time = "2026-06-26T18:32:16.077Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/07/7f/e327d912239ec4b3b49999e3967389bcf1ee8722b9ee9194d2752ecd558a/greenlet-3.5.3-cp311-cp311-manylinux_2_39_riscv64.whl", hash = "sha256:d27c0c653a60d9535f690226474a5cc1036a8b0d7b57504d1c4f89c44a07a80c", size = 421083, upload-time = "2026-06-26T19:25:35.804Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2a/7b/ad04e9d1337fc04965dc9fc616b6a72cb65a24b800a014c011ec812f5489/greenlet-3.5.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7ef56fe650f50575bf843acde967b9c567687f3c22340941a899b7bc56e956a8", size = 1577771, upload-time = "2026-06-26T19:09:01.537Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d8/33/6c87ab7ba663f70ca21f3022aad1ffe56d3f3e0521e836c2415e13abcc3c/greenlet-3.5.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5121af01cf911e70056c00d4b46d5e9b5d1415550038573d744138bacb59e6b8", size = 1644048, upload-time = "2026-06-26T18:31:42.996Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/1c/35/f0d8ee998b422cf8693b270f098e55d8d4ec8006b061b333f54f177d28d9/greenlet-3.5.3-cp311-cp311-win_amd64.whl", hash = "sha256:0f41e4a05a3c0cb31b17023eff28dd111e1d16bf7d7d00406cd7df23f31398a7", size = 239137, upload-time = "2026-06-26T18:23:21.664Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/96/b9820295576ef18c9edc404f10e260ae7215ceaf3781a54b720ed2627862/greenlet-3.5.3-cp311-cp311-win_arm64.whl", hash = "sha256:ec6f1af59f6b5f3fc9678e2ea062d8377d22ac644f7844cb7a292910cf12ff44", size = 237630, upload-time = "2026-06-26T18:24:00.281Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5d/6e/4c37d51a2b7f82d2ff11bb6b5f7d766d9a011726624af255e843727627a3/greenlet-3.5.3-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:719757059f5a53fd0dde23f78cffeafcdd97b21c850ddb7ca684a3c1a1f122e2", size = 288685, upload-time = "2026-06-26T18:22:08.977Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/73/815dd90131c1b71ebdf53dbc7c276cafec2a1173b97559f97aba72724a87/greenlet-3.5.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:efa9f765dd09f9d0cdac651ffdf631ee59ec5dc6ee7a73e0c012ba9c52fbdf5b", size = 604761, upload-time = "2026-06-26T19:07:10.114Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9f/57/079cfe76bcef36b153b25607ee91c6fcb58f17f8b23c86bbbeabe0c88d72/greenlet-3.5.3-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7faba15ac005376e02a0384504e0243be3370ce010296a44a820feb342b505ab", size = 617044, upload-time = "2026-06-26T19:10:07.25Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/fb/d97dc261209c80744b7c8132693a30d70ec6e7315e632cb0a10b3fec94dd/greenlet-3.5.3-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5795cd1101371140551c645f2d408b8d3c01a5a29cf8a9bce6e759c983682d23", size = 622351, upload-time = "2026-06-26T19:24:16.32Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/37/87/b4d095775a3fb1bcafbb483fc206b27ebb785724c83051447737085dc54e/greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:87142215824be6ac05e2e8e2786eec307ccbc27c36723c3881959df654af6861", size = 614244, upload-time = "2026-06-26T18:32:17.594Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/ac/e5fee13cbbd0e8de312d9a146584b8a51891c68847330ef9dc8b5109d23f/greenlet-3.5.3-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:af4923b3096e26a36d7e9cf24ab88083a20f97d191e3b97f253731ce9b41b28c", size = 425395, upload-time = "2026-06-26T19:25:37.144Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/70/7559b609683650fa2b95b8ab84b4ab0b26556a635d19675e12aa832d826d/greenlet-3.5.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:215275b1b49320987352e6c1b054acca0064f965a2c66992bed9a6f7d913f149", size = 1574210, upload-time = "2026-06-26T19:09:03.077Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ae/73/be55392074c60fc37655ca40fa6022457bfbf6718e9e342a7b0b41f96dd2/greenlet-3.5.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6b1b0eed82364b0e32c4ea0f221452d33e6bb17ae094d9f72aed9851812747ea", size = 1638627, upload-time = "2026-06-26T18:31:44.748Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/14/40/c57489acf8e37d74e2913d4eff63aa0dba17acccc4bdeef874dde2dbbec9/greenlet-3.5.3-cp312-cp312-win_amd64.whl", hash = "sha256:cde8adafa2365676f74a979744629589999093bc86e2484214f58e61df08902c", size = 239882, upload-time = "2026-06-26T18:23:27.518Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/fd/6fea0e3d6600f785069481ee637e09378dd4118acdfd38ad88ae2db31c98/greenlet-3.5.3-cp312-cp312-win_arm64.whl", hash = "sha256:c4e7b79d83805475f0102008843f6eb45fd3bb0b2e88c774adab5fbaab27117d", size = 238211, upload-time = "2026-06-26T18:22:37.671Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9b/ff/a620267401db30a50cc8450ee90730e2d4a85658c055c0e760d4ed47fb13/greenlet-3.5.3-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:c8d87c2134d871df96ecdea9cec7cbaab286dadab0f56476e57aaf9e8ac11550", size = 287609, upload-time = "2026-06-26T18:21:14.724Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d6/fa/5401ac78021c826a25b6dde0c705e0a8f29b617509f9185a31dac15fbe1b/greenlet-3.5.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a2d185dd1621757e70c3861cceffd5317ab4e7ed7eb09c82994828468527ade5", size = 607435, upload-time = "2026-06-26T19:07:11.412Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e9/76/1dc144a2e56e65d36405078ed774224375ea520a1870a6e46e08bb4ac7bf/greenlet-3.5.3-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1c514a468149bf8fbbab874188a3535cd8a48a3e353eb53a3d424296f8dbacd3", size = 619787, upload-time = "2026-06-26T19:10:08.396Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/57/61/2f5b1adf256d039f5dab8005de8d3d7ad2b0070a3219c0e036b3fbfeb440/greenlet-3.5.3-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9ad04dd75458c6300b047c61b8639092433d205a25a14e310d6582a480efcca1", size = 625580, upload-time = "2026-06-26T19:24:18.344Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bf/87/c298cee62df1de4ad7fec32abda73526cff347fd143a6ed4ac369246668a/greenlet-3.5.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:915f887cf2682b66419b879423a2e072634aa7b7dce6f3ada4957cfced3f1e9a", size = 616786, upload-time = "2026-06-26T18:32:19.128Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3e/d9/ab7fc9e543e44d6879b0a6ef9a4b2188940fd180cc65d6f646883ddf7201/greenlet-3.5.3-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:afaabdd554cd7ae9bbb3ca070b0d7fdfd207dbf1d16865f7233837709d354bda", size = 427933, upload-time = "2026-06-26T19:25:38.219Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9e/2e/e6f009885ed0705ccf33fe0583c117cfd03cde77e31a596dd5785a30762b/greenlet-3.5.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:766cfd421c13e450feb340cd472a3ed9957d438727b7b4593ad7c76c5d2b0deb", size = 1574316, upload-time = "2026-06-26T19:09:04.273Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ef/fe/43fd110b01e40da0adb7c90ac7ea744bef2d43dca00de5095fd2351c2a68/greenlet-3.5.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2ecda9ec22edf38fa389369eaed8c3d37c05f3c54e69f69438dbb2cc1de1458b", size = 1638614, upload-time = "2026-06-26T18:31:46.297Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0f/7c/062447147a61f8b4337b156fe70d32a165fcf2f89d7ca6255e572806705c/greenlet-3.5.3-cp313-cp313-win_amd64.whl", hash = "sha256:c82304750f057167ff60d188df1d0cc1764ce9567eadf03e6a7443bcedd0b30b", size = 239850, upload-time = "2026-06-26T18:21:54.613Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c7/7e/220a7f5824a64a60443fc03b39dfac4ea63a7fb6d481efa27eafa928e7f4/greenlet-3.5.3-cp313-cp313-win_arm64.whl", hash = "sha256:dc133a1569ee667b2a6ef56ce551084aeefd87a5acbc4736d336d1e2edc6cfc4", size = 238141, upload-time = "2026-06-26T18:22:48.507Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c3/93/43e116ee114b28737ba7e12952a0d4e2f55944d0f84e42bc91ba7192a3c9/greenlet-3.5.3-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:fd2e02fa07485778536a036222d616ab957b1d533f36b3ed98ce725d9c9d3117", size = 288202, upload-time = "2026-06-26T18:23:49.604Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/82/2f/146d218299046a43d1f029fd544b3d110d0f175a09c715c7e8da4a4a345d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df0a0628d1597eb0897b62f55d1343f772405fd25f3b2a796c76874b0c2e22e8", size = 654096, upload-time = "2026-06-26T19:07:12.71Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/cc/04738cafb3f45fa991ea44f9de94c47dcec964f5a972300988a6751f49d9/greenlet-3.5.3-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ebd933a6adabc298bab47731a130fe6bfb888bd934eee37810f151159544540d", size = 666304, upload-time = "2026-06-26T19:10:09.503Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/86/a9/73fa62893d5b84b4205544e6b673c654cc43aa5b9899bac00f04d64af73d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8d19fe6c39ebff9259f07bcc685d3290f8fa4ea2278e51dd0008e4d6b0f2d814", size = 670657, upload-time = "2026-06-26T19:24:19.967Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ce/aa/4e0dad5e605c270c784ab911c43da6adb136ccd4d81180f763ca429a723d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4b9d501b40e80b70e32323c799dd9b420a5577a9601469d362ae1ffb690f3a7c", size = 663635, upload-time = "2026-06-26T18:32:20.802Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/29/7e/2ffce64929fb3cab7b65d5a0b20aaf9764e227681d731b041077fc9a525a/greenlet-3.5.3-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:962c5df2db8cb446da51edf1ca5296c389d93b99c9d8aa2ee4c7d0d8f1218260", size = 473497, upload-time = "2026-06-26T19:25:39.421Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d1/50/13efdbea246fe3d3b735e191fec08fb50809f53cd2383ebe123d0809e44b/greenlet-3.5.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a1fad1d11e7d6aab184107baa8e4ece11ccba3ec9599cd7efa5ff4d70d43256a", size = 1621252, upload-time = "2026-06-26T19:09:05.647Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f7/22/c0a336ae4a1410fd5f5121098e5bfbf1865f64c5ef80b4b5412886c4a332/greenlet-3.5.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:fad5aec764399f1b5cc347ad250a59660f20c8f8888ea6bae1f93b769cce1154", size = 1684824, upload-time = "2026-06-26T18:31:47.738Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/94/91aec0030bea75c4b3244251d0de60a1f3432d1ecb53ab6c437fb5c3ba61/greenlet-3.5.3-cp314-cp314-win_amd64.whl", hash = "sha256:7669aa24cf2a1041d6f7899575b494a3ab4cf68bfcc8609b1dc0be7272db835e", size = 240754, upload-time = "2026-06-26T18:22:15.669Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e5/06/68d0983e79e02138f64b4d303c500c27ddb48e5e77f3debb80888a921eae/greenlet-3.5.3-cp314-cp314-win_arm64.whl", hash = "sha256:5b4807c4082c9d1b6d9eed56fcd041863e37f2228106eef24c30ca096e238605", size = 239549, upload-time = "2026-06-26T18:22:42.996Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/91/95/3e161213d7f1d378d15aa9e792093e9bfe01844680d04b7fd6e0107c9098/greenlet-3.5.3-cp314-cp314t-macosx_11_0_universal2.whl", hash = "sha256:271a8ea7c1024e8a0d7dd2be66dd66dda8a07193f41a17b9e924f7600f5b62be", size = 296389, upload-time = "2026-06-26T18:22:20.657Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/00/92/715c44721abe2b4d1ae9abde4179411868a5bff312479f54e105d372f131/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:19131729ae0ddc3c2e1ef85e650169b5e37ee32e400f215f78b94d7b0d567310", size = 653382, upload-time = "2026-06-26T19:07:14.209Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/83/37a10372a1090a6624cca8e74c12df1a36c2dc36429ed0255b7fb1aeee23/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1540dd8e5fc2a5aec40fbb98ef8e149fa47c89a4b4a1cf2575a14d3d1869d7a8", size = 659401, upload-time = "2026-06-26T19:10:10.876Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cb/73/8faec206b851c22b1733545fda900829a1f3f5b1c78ae7e0fb3dba57d9f4/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b897d97759425953f69a9c0fac67f8fe333ec0ce7377ef186fb2b0c3ad5e354d", size = 659582, upload-time = "2026-06-26T19:24:21.357Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/db/e2/d1509cad4207da559cc42986ecdd8fc67ad0d1bba2bf03023c467fd5e0f3/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e81fa194a1d20967877bdf9c7794db2bc99063e5be36aee710c08f04c5bb087f", size = 656969, upload-time = "2026-06-26T18:32:22.272Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/55/50c19e49f8045834ada71ef12f8ad048eba8517c6aa41161bed676328fae/greenlet-3.5.3-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:3236754d423955ea08e9bb5f6c04a7895f9e22c290b66aa7653fcb922d839eb0", size = 491037, upload-time = "2026-06-26T19:25:40.672Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/86/7d/eaf70de20aadca3a5884aec58362861c64ce45e7b277f47ed026926a3b89/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:55cf4d777485d43110e47133cbba6d74a8885a87ec1227ef0267f9ee80c5aa21", size = 1617822, upload-time = "2026-06-26T19:09:06.893Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/f9/414d38fc400ae4350d4185eaad1827676f7cf5287b9136e0ed1cbbe20a7f/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:12a248ba75f6a9a236375f52296c498c89ff1d8badf32deb9eca7abd5853f7da", size = 1677983, upload-time = "2026-06-26T18:31:49.396Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e4/15/7edb977e08f9bff702fe42d6c902702786ff6b9694058b4e6a2a6ac90e57/greenlet-3.5.3-cp314-cp314t-win_amd64.whl", hash = "sha256:efc6bd60ea02e085862c74a3ef64b147ffc6f1a5ea7d9f26e7a939943f68c1e3", size = 243626, upload-time = "2026-06-26T18:24:41.485Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/8a/93928dce91e6b3598b5e779e8d1fd6576a504640c58e78627077f6a7a91a/greenlet-3.5.3-cp315-cp315-macosx_11_0_universal2.whl", hash = "sha256:ea03f2f04367845d6b58eeed276e1e56e51f0b97d8ad5a88a7d20a91dc9056cc", size = 288860, upload-time = "2026-06-26T18:22:48.07Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/4f/ca/69db42d447a1378043e2c8f19c09cbbd1263371505053c496b49066d3d16/greenlet-3.5.3-cp315-cp315-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:78dbef602fda6d97d957eb7937f70c9ce9e9527330347f8f6b6f9e554a9e7a47", size = 659747, upload-time = "2026-06-26T19:07:15.565Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a8/0b/af7ac2ef8dd41e3da1a40dda6305c23b9a03e13ba975ec916357b50f8575/greenlet-3.5.3-cp315-cp315-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6f73857adb8fee13fa56c172bd11262f888c0c648f9fea113e777bb2c7904a81", size = 670419, upload-time = "2026-06-26T19:10:12.293Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/25/aa/952cf28c2ff949a8c971134fb43854dd7eaa737218723aaef758f8c9aead/greenlet-3.5.3-cp315-cp315-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:cefa9cef4b371f9844c6053db71f1138bc6807bab1578b0dae5149c1f1141357", size = 674261, upload-time = "2026-06-26T19:24:22.79Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/1e/1d51640cacbfc455dbe9f9a9f594c49e4e244f63b9971a2f4764e46cc53d/greenlet-3.5.3-cp315-cp315-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:232fec92e823addaf02d9472cf7381e24a1d046a6ced1103c5caa4c21b9dfc1d", size = 668787, upload-time = "2026-06-26T18:32:24.298Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/dc/f2/b00d6f5e63e531a93562b2ec1a4c320fbee91f580fc42e6417af69d706e5/greenlet-3.5.3-cp315-cp315-manylinux_2_39_riscv64.whl", hash = "sha256:6219b6d04dbf6ba6084d77dc609e8473060dc55f759cbf626d512122781fa128", size = 480322, upload-time = "2026-06-26T19:25:41.852Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/21/66/4030d5b0b5894500023f003bb054d9bb354dfbd1e186c3a296759172f5f5/greenlet-3.5.3-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:2421c3564da9429d5586d46ca31ebb26516b5498a802cf65c041a8e8a8980d34", size = 1626305, upload-time = "2026-06-26T19:09:08.281Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0e/50/5221371c7550108dfa3c378debc41d032aa9c78e89abb01d8011cfc93289/greenlet-3.5.3-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e0f0d160f0b2e558e6c75f7930967183255dc9735e5f5b8cae58ee09c9576d8b", size = 1688631, upload-time = "2026-06-26T18:31:51.278Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/68/5d/00d469daae3c65d2bf620b10eee82eb022127d483c6bc8c69fae6f3fbf17/greenlet-3.5.3-cp315-cp315-win_amd64.whl", hash = "sha256:dd99329bbc15ca78dcc583dba05d0b1b0bae01ab6c2174989f5aaee3e41ac930", size = 241027, upload-time = "2026-06-26T18:22:38.203Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e7/e8/883785b44c5780ed71e83d3e4437e710470be17a2e181e8b601e2da0dc4a/greenlet-3.5.3-cp315-cp315-win_arm64.whl", hash = "sha256:499fef2acede88c1864a57bb586b4bf533c81e1b82df7ab93451cdb47dfec227", size = 240085, upload-time = "2026-06-26T18:23:54.217Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/1c/da/4f4a8450962fad137c1c8981a3f1b8919d06c829993d4d476f9c525d5173/greenlet-3.5.3-cp315-cp315t-macosx_11_0_universal2.whl", hash = "sha256:176bc16a721fa5fc294d70b87b4dfa5fbdd251b3da5d5372735ecef9bd7d6d0c", size = 297221, upload-time = "2026-06-26T18:23:27.176Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/57/66/b3bfae3e220a9b63ea539a0eea681800c69ab1aada757eae8789f183e7ce/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:629b614d2b786e89c50440e246f33eea78f58a962d0bdbbcc809e6d13605903f", size = 657221, upload-time = "2026-06-26T19:07:16.973Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7b/81/b6d4d73a709684fc77e7fa034d7c2fe82cffa9fc920fadcaa659c2626213/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2b2e857ae16f5f72142edf75f9f176fe7526ba19a2841df1420516f83831c9f2", size = 663226, upload-time = "2026-06-26T19:10:13.723Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e9/39/0e0938a75115b939d42733a2a12e1d349653c9531fe6fe563e8a681f04e6/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:16d192579ed281051396dddd7f7754dac6259e6b1fb26378c87b66622f8e3f91", size = 663706, upload-time = "2026-06-26T19:24:24.312Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/07/e210b02b589f16e74ff48b730690e4a34ffe984219fce4f3c1a0e7ec8545/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e515757e2e36bcbf1fad09a46e1557e8b1ae1797d4b44d09da7deed88ad28608", size = 660802, upload-time = "2026-06-26T18:32:26.081Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5b/41/35d1c678cdb3c3b9e6bee691728e563cfb294202b23c7a4c3c2ccc343589/greenlet-3.5.3-cp315-cp315t-manylinux_2_39_riscv64.whl", hash = "sha256:4399eb8d041f20b68d943918bc55502a93d6fdc0a37c14da7881c04139acee9d", size = 498803, upload-time = "2026-06-26T19:25:43.063Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/eb/2e/5303eb3fa06bca089060f479707182a93e360683bc252acf846c3090d34e/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:b363d46ed1ea431825fdb01471bb024fc08399bad1572a616e853c7684415adb", size = 1622157, upload-time = "2026-06-26T19:09:09.527Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/54/70/50de47a488f14df260b50ae34fb5d56016e308b098eab02c878b5223c26a/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:e44da2f5bbdaabaf7d80b73dbb430c7035771e9f244e3c8b769715c9d8fa0a16", size = 1681159, upload-time = "2026-06-26T18:31:52.986Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a7/13/1055e1dda7882073eda533e2b96c62e55bbd2db7fda6d5ece992febc7071/greenlet-3.5.3-cp315-cp315t-win_amd64.whl", hash = "sha256:8ff8bed3e3baa20a3ea261ce00526f1898ad4801d4886fd2220580ee0ad8fadf", size = 244007, upload-time = "2026-06-26T18:22:04.353Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/0d/ca7d15afbdc397e3401134c9e1800d51d12b829661786187a4ad08fe484f/greenlet-3.5.3-cp315-cp315t-win_arm64.whl", hash = "sha256:b7068bd09f761f3f5b4d214c2bed063186b2a86148c740b3873e3f56d79bac31", size = 242586, upload-time = "2026-06-26T18:23:37.93Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "html5lib"
|
name = "html5lib"
|
||||||
version = "1.1"
|
version = "1.1"
|
||||||
@@ -1412,6 +1624,25 @@ wheels = [
|
|||||||
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bc/60/5382c03e1970de634027cee8e1b7d39776b778b81812aaf45b694dfe9e28/pillow-12.2.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:bfa9c230d2fe991bed5318a5f119bd6780cda2915cca595393649fc118ab895e", size = 7080946, upload-time = "2026-04-01T14:46:11.734Z" },
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bc/60/5382c03e1970de634027cee8e1b7d39776b778b81812aaf45b694dfe9e28/pillow-12.2.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:bfa9c230d2fe991bed5318a5f119bd6780cda2915cca595393649fc118ab895e", size = 7080946, upload-time = "2026-04-01T14:46:11.734Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "playwright"
|
||||||
|
version = "1.61.0"
|
||||||
|
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "greenlet" },
|
||||||
|
{ name = "pyee" },
|
||||||
|
]
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/44/ee/31e4e0db36588b817a10b299a0285082545fde7d36543c2abe498bb3d61a/playwright-1.61.0-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:ff138c3a604f69911e9d42fd036e55c2a171e5616edf04c1e7f60a2a285540b0", size = 43421877, upload-time = "2026-06-29T10:32:48.428Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/42/35/71395dd3ecc798965be4a3ef8c443217d4abca168e7cb34536304f9489e6/playwright-1.61.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:009588c2a7e499bc5a8b425b61fa65490968bbda9cd69e0cf2cff10f8304659a", size = 42205016, upload-time = "2026-06-29T10:32:52.104Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f4/44/323164cf5cd1647bdefce76ffce27651aadb959d089b48f53ea40918276e/playwright-1.61.0-py3-none-macosx_11_0_universal2.whl", hash = "sha256:9f7de4536088d12037c13a52b7ea34b59270b78926bb56935070597ffac6b1af", size = 43421884, upload-time = "2026-06-29T10:32:55.773Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ab/f8/a35bf179e4ba2522c1893635094a64e407572547bd61528820fc0abc87fe/playwright-1.61.0-py3-none-manylinux1_x86_64.whl", hash = "sha256:54f3b39f6eab832e33458c1dd7da0b5682aedab3b09ae731b5c59fa12fd2024e", size = 47421381, upload-time = "2026-06-29T10:32:59.903Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b7/eb/e3f922348ec17c315f98c463f72faa1181a1c3de0bfe31a8d2edf6561723/playwright-1.61.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:93454322ade8c11d5d6c211bfd91bdfb9ffb4810e3e026371bcbc4bec1b7ee4c", size = 47120545, upload-time = "2026-06-29T10:33:03.574Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c2/a6/5be4e52b40a9c0c8a073e7c5b0785c05cf5a9ea8f8a7b5b260e32d970342/playwright-1.61.0-py3-none-win32.whl", hash = "sha256:372d55a6f1248fa1dd47599686980cb8fb5bbe6fcda59eab793eb657c11d8a9b", size = 37844841, upload-time = "2026-06-29T10:33:07.361Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6c/fd/2b78036e5fbe9d5f5645bbe08a1eac7160c51243c0093963edbcf67c35d9/playwright-1.61.0-py3-none-win_amd64.whl", hash = "sha256:35c6cc4589a5d00964a59d7b3e59641e0aac0c02f15479a7af77d20f6bc79597", size = 37844846, upload-time = "2026-06-29T10:33:10.637Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/0d/1b0f3c4ee4eb0514bc805b5c2f9a223e5b6de4f11a926f5235d51d0fc81b/playwright-1.61.0-py3-none-win_arm64.whl", hash = "sha256:e9fcbffcf557a8620fdedd92491eb59a32d18e23d6f3b4f6214b952be324fe51", size = 33955127, upload-time = "2026-06-29T10:33:14.008Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "pluggy"
|
name = "pluggy"
|
||||||
version = "1.6.0"
|
version = "1.6.0"
|
||||||
@@ -1496,6 +1727,18 @@ wheels = [
|
|||||||
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "pyee"
|
||||||
|
version = "13.0.1"
|
||||||
|
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "typing-extensions" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8b/04/e7c1fe4dc78a6fdbfd6c337b1c3732ff543b8a397683ab38378447baa331/pyee-13.0.1.tar.gz", hash = "sha256:0b931f7c14535667ed4c7e0d531716368715e860b988770fc7eb8578d1f67fc8", size = 31655, upload-time = "2026-02-14T21:12:28.044Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/c4/b4d4827c93ef43c01f599ef31453ccc1c132b353284fc6c87d535c233129/pyee-13.0.1-py3-none-any.whl", hash = "sha256:af2f8fede4171ef667dfded53f96e2ed0d6e6bd7ee3bb46437f77e3b57689228", size = 15659, upload-time = "2026-02-14T21:12:26.263Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "pygments"
|
name = "pygments"
|
||||||
version = "2.20.0"
|
version = "2.20.0"
|
||||||
|
|||||||
Reference in New Issue
Block a user