Compare commits

...

23 Commits

Author SHA1 Message Date
Yuxuan Yan c91415aff8 Document JoinQuant cost model findings 2026-07-06 15:35:41 +08:00
Yuxuan Yan efa9d24c73 Harden JoinQuant browser smoke automation 2026-07-04 19:27:48 +08:00
Yuxuan Yan 5fada75d23 Add JoinQuant env browser login helper 2026-07-04 18:37:34 +08:00
Yuxuan Yan d05931f41a Add JoinQuant simulated trading automation 2026-07-04 18:12:01 +08:00
Yuxuan Yan 39a93259e1 Add JoinQuant browser backtest automation 2026-07-04 18:08:31 +08:00
Yuxuan Yan 5388359dc8 Automate local JoinQuant smoke prep 2026-07-04 17:55:19 +08:00
Yuxuan Yan c200508f9e Align JoinQuant targets to execution dates 2026-07-04 17:50:55 +08:00
Yuxuan Yan f25db279bf Add JoinQuant comparison plugin 2026-07-04 17:43:09 +08:00
Yuxuan Yan 528620b271 Raise coverage threshold to 95% and expand test coverage
- pyproject.toml: fail_under 80 → 95
- test_alpha: +79 lines
- test_cli_workflow: +226 lines
- test_derived: +121 lines
- test_downloader_contracts: +169 lines
- test_features: +16 lines
- test_minute_downloader: +81 lines
- test_portfolio: +208 lines
2026-06-16 21:10:30 +08:00
Yuxuan Yan b5c8c0b8da Improve offline coverage for data boundaries 2026-06-16 17:42:20 +08:00
Yuxuan Yan 31baa18ce5 Add offline workflow and coverage tests 2026-06-16 17:37:16 +08:00
Yuxuan Yan 8d908477e2 Add daily derived data pipeline 2026-06-16 15:55:30 +08:00
Yuxuan Yan 83a006bbe4 Add minute bar feature pipeline 2026-06-16 13:57:17 +08:00
Yuxuan Yan 17fa75495d Restore reversal tutorial wording 2026-06-12 22:58:22 +08:00
Yuxuan Yan 3c58a1372e Use next-open returns for research eval 2026-06-12 18:41:18 +08:00
Yuxuan Yan 16b4988f16 Rewrite reversal report as tutorial 2026-06-12 17:12:29 +08:00
Yuxuan Yan 2c0ca53bd6 Document cost bps as one-way per-trade, not round-trip
The simulator charges (cost_bps + slippage_bps) on each fill, so a full
round trip is charged twice. Correct the cost-model doc, the reversal_5d
report, and the report generator to state the rate is one-way per-trade
(~20 bps round trip for 5+5), rather than mislabeling it round-trip.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 21:46:41 +08:00
Yuxuan Yan 2ceac82325 Add pipeline invariant checks for look-ahead, execution, PnL, and lot rules
Ten network-free correctness tests mapping 1:1 to the review checks:
reversal look-ahead, next-open execution date, PnL decomposition,
realized-not-target threading, blocked-trade zero cost, causal universe
mask, one-way cost bps, raw-price accounting, adjustment-invariant alpha,
and lot-lattice repair.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 21:46:29 +08:00
Yuxuan Yan b7dd94b032 Add 5-day reversal end-to-end pipeline report and repro scripts
Runs the 5-day reversal signal through data→alpha→combo→portfolio on the
full A-share universe and documents the finding: the naive z-score book
loses to outlier concentration, rank weighting on a liquid universe
recovers a real edge, and turnover-driven cost is the binding constraint.
Includes the e2e driver and figure generator that produce the report.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 17:40:52 +08:00
Yuxuan Yan 07ed6ad917 Add outlier-robust reversal_rank alpha and investable-universe filter
reversal_rank weights the 5-day reversal signal by bounded cross-sectional
rank instead of z-score, so a few extreme A-share pct_change outliers
(newly listed / post-suspension / limit-up names) can no longer dominate
the book. compute_alpha gains an optional per-date investable-universe
mask (tradable, non-ST, seasoned, top-liquidity) applied to the signal
before weighting, exposed via --liquid-universe/--universe-top-n.

combo combine now accepts a single alpha as an identity passthrough so a
one-alpha pipeline run needs no synthetic second input.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 17:40:28 +08:00
Yuxuan Yan 0a6f367fbf Evaluate weights against next-period returns to avoid look-ahead
Weights formed from close[t] now earn the t→t+1 return: forward returns
are computed on the full market calendar before selecting signal dates,
so a sparse signal grid earns the next available return rather than the
next signal date, and the final signal date (no forward return) is
dropped. Signal pct_change uses fill_method=None so suspended names do
not inherit stale prices.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-11 17:39:55 +08:00
Yuxuan Yan 534b91aaa4 Document and abstract portfolio trading costs 2026-06-10 15:41:38 +08:00
Yuxuan Yan 4a477b8f75 Make Backtrader an optional extra 2026-06-10 15:13:11 +08:00
78 changed files with 12555 additions and 74 deletions
+1
View File
@@ -1,6 +1,7 @@
__pycache__/
*.py[cod]
.pytest_cache/
.coverage
*.egg-info/
.venv/
venv/
+6
View File
@@ -26,6 +26,10 @@ uv run python cli.py portfolio eval --positions-path portfolio/eq_10m.pq --data-
Add a runtime dep with `uv add <pkg>`, a dev/test dep with `uv add --dev <pkg>` (both update `pyproject.toml` + `uv.lock`).
Backtrader is optional (`uv sync --extra backtrader`) and is not used by the
current pipeline. Keep `portfolio simulate` as the canonical backtest/execution
path unless an explicit future adapter is requested.
Note: `tests/test_downloader.py` hits the network (live baostock/akshare); `tests/test_alpha.py` and `tests/test_portfolio.py` are pure and fast.
## Architecture: one decoupled pipeline
@@ -65,6 +69,8 @@ Data is stored **long/tidy**, not wide, as a Hive-partitioned dataset keyed by `
`portfolio simulate` must execute `position_shares`, not continuous `target_shares`. It fills at the next available open and clips desired deltas through repeatable constraints (`suspension`, `price_limit`, `volume_cap`). `portfolio eval` uses `target_weight` for a continuous research view, so zero-gross carry dates remain flat there. Keep IC/IR out of portfolio metrics too.
Trading cost uses the simplified open-execution proportional cash-cost model in `docs/portfolio_trading_cost_model.md`: `abs(traded_shares * open) * (cost_bps + slippage_bps) / 10000`. Slippage is cash cost only; do not also adjust execution prices for slippage.
## Alphas: factory + plugin pattern
Each alpha is a class subclassing `BaseAlpha` (`pipeline/alpha/base.py`), living in its own module. It implements `signal(close) -> wide DataFrame` (the raw score); the base class's `to_weights` cross-sectionally z-scores that into position weights (override for custom normalization). Subclasses declare their own typed `__init__` params (e.g. `lookback`, `vol_window`, or anything custom).
+91 -1
View File
@@ -48,6 +48,13 @@ The env is managed with [uv](https://docs.astral.sh/uv/). `uv sync` builds `.ven
uv sync
```
Backtrader is an optional dependency and is **not used by the current pipeline**.
Install it only for future experiments or adapter work:
```bash
uv sync --extra backtrader
```
## Quick start
```bash
@@ -113,6 +120,49 @@ partitions already written. Pass the **dataset directory** (`{output_dir}/{unive
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
### `derived` — daily custom/derived data
Derived data is daily-only v1 research data keyed by `symbol_id,date`, with one
or more numeric value columns. It can come from user CSV/parquet files or Python
plugins, and is written as a single parquet file at `derived/{name}.pq`.
The validator normalizes `date` to the trading day, requires unique
`symbol_id,date` keys, rejects duplicate columns, and rejects non-numeric value
columns. Alpha computation consumes derived data through the existing
`--feature-path` flag.
```bash
# Validate a user file without writing output.
uv run python cli.py derived validate --input-path vendor_factor.csv
# Ingest CSV/parquet into the canonical derived/ layout.
uv run python cli.py derived ingest \
--input-path vendor_factor.csv \
--derived-name vendor_factor
# List built-in and external derived-data plugin types.
uv run python cli.py derived list
uv run python cli.py derived list --derived-module path/to/my_derived.py
# Compute a derived file from daily and/or minute inputs.
uv run python cli.py derived compute \
--minute-path data/minute_bars/sh600000 \
--daily-path data/daily_bars/sh600000 \
--derived-type minute_daily_summary \
--derived-name minute_summary
# Join derived columns into a feature-aware alpha.
uv run python cli.py alpha compute \
--data-path data/daily_bars/sh600000 \
--feature-path derived/minute_summary.pq \
--alpha-type my_feature_aware_alpha \
--alpha-name my_run
```
For compatibility, `feature list` and `feature compute` remain available and
delegate to the same derived-data registry. Existing `features/*.pq` files are
still valid `--feature-path` inputs when they satisfy the daily numeric contract.
### `alpha list` — show registered alpha types
```bash
@@ -130,6 +180,7 @@ uv run python cli.py alpha list --alpha-module path/to/my_alpha.py # include a
| `--output-dir` | `alphas` | Output directory |
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
| `--feature-path` | — | Daily derived/feature parquet file or dataset to left-join on `symbol_id,date`; repeatable |
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
| `--param` | — | Extra constructor param as `name=value`; repeatable |
@@ -203,7 +254,9 @@ uv run python cli.py portfolio build \
Executes the constructed `position_shares` book at the next available open,
clipping trades through repeatable constraints. It writes `fills/<name>.pq` and
`pnl/<name>.pq`.
`pnl/<name>.pq`. Trading costs use the simplified open-execution proportional
cash-cost model documented in
[`docs/portfolio_trading_cost_model.md`](docs/portfolio_trading_cost_model.md).
| Option | Default | Description |
| --- | --- | --- |
@@ -361,6 +414,8 @@ between phases (data is stored long/tidy):
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
baostock valuation ratios.)
- **derived** (`DERIVED_KEY_COLUMNS` + values): required keys `symbol_id, date`;
value columns are user/plugin-defined and must be numeric in v1.
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
@@ -378,8 +433,11 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
- `cli.py` — entry point wiring the file-based phases together
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
- `pipeline/derived/` — daily derived-data ingestion, validation, plugin registry,
and built-in derived computations → `derived/*.pq`
- `pipeline/alpha/``base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
- `pipeline/features/` — compatibility wrappers for the derived-data registry
- `pipeline/combo/` — alpha combination → `combos/*.pq`
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
reference next-open simulator, and research metrics
@@ -401,9 +459,41 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
- [x] **Reference execution simulation** — next-open fills over constructed
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
and slippage controls.
- [x] **Derived/custom daily data ("Level 2")** — ingest user CSV/parquet files
or compute plugin outputs as validated numeric daily datasets under
`derived/{name}.pq`; alpha joins continue through `--feature-path`.
- [ ] **Optional Backtrader adapter** — Backtrader is available as the
`backtrader` extra for possible future event-driven/broker-style experiments,
but it is not part of the current canonical portfolio workflow.
- [ ] **Forward / paper trading** — run the same construction logic on live
daily data, track simulated fills and a running P&L without real capital.
- [ ] **Intraday / microstructure data** — bid/ask prices & sizes, mid-price,
and intraday VWAP. These need a tick / L1L2 quote feed (typically a paid or
brokerage data tier); the free daily sources here only expose daily bars, so
this is a separate data phase rather than extra columns on the daily schema.
### Additional TODOs
The following items are intended extensions beyond the current daily
alpha-to-portfolio pipeline:
- **Long-only portfolio mode** — add a construction option that converts
alpha/combo weights into a long-only book while preserving existing lot,
price, suspension, and volume-cap handling.
- **Index-short hedging mode** — support portfolios that hold long A-share
names while shorting an index or index proxy for market exposure control.
- **Expanded universe presets** — add explicit universe aliases for CSI 300,
CSI 500, CSI 1000, and CSI 1800, while keeping file-based and comma-separated
custom universes available.
- **Categorical derived data** — extend the numeric-only derived-data v1 contract
to support categorical inputs such as industry classifications. In this
project, "Level 2" means customized second-level research data produced by
users or plugins; it does not necessarily mean exchange order-book/L2 quote
feeds.
- **Minute bar data** — continue extending the raw minute-bar and feature
workflow. The initial Baostock 5-minute download and daily feature plugin path
exist; intraday execution and replacing canonical daily bars remain out of
scope unless explicitly added later.
- **Industry data** — add industry classification inputs for filtering,
grouping, exposure reporting, neutralization, or industry-aware portfolio
construction.
+8
View File
@@ -3,7 +3,9 @@
Phases:
data — Download daily bars to parquet
derived — Ingest or compute daily derived data
alpha — Compute alpha weights from data
feature — Compute daily features from minute bars
combo — Combine alphas into a single weight
portfolio — Build tradable positions and simulate execution
"""
@@ -13,9 +15,12 @@ import logging
import click
from pipeline.data.cli import data
from pipeline.derived.cli import derived
from pipeline.alpha.cli import alpha
from pipeline.features.cli import feature
from pipeline.combo.cli import combo
from pipeline.portfolio.cli import portfolio
from plugins.joinquant.cli import joinquant
from tools.pqcat import pqcat
from tools.alphaview import alphaview
@@ -39,9 +44,12 @@ def cli(log_level):
cli.add_command(data)
cli.add_command(derived)
cli.add_command(alpha)
cli.add_command(feature)
cli.add_command(combo)
cli.add_command(portfolio)
cli.add_command(joinquant)
cli.add_command(pqcat)
cli.add_command(alphaview)
+158
View File
@@ -31,11 +31,68 @@ _BATCH_COLUMNS = [
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
]
# Raw Baostock minute bars. The ``time`` field is usually compact
# YYYYMMDDHHMMSSmmm; parsing below also tolerates HH:MM:SS strings in tests.
_MINUTE_FIELDS = "date,time,code,open,high,low,close,volume,amount,adjustflag"
_MINUTE_NUMERIC = ["open", "high", "low", "close", "volume", "amount"]
_MINUTE_COLUMNS = [
"symbol", "datetime", "date", "time", "frequency",
"open", "high", "low", "close", "volume", "amount", "vwap", "adjustflag",
]
_MINUTE_FREQUENCIES = {"5", "15", "30", "60"}
class _SessionLost(Exception):
"""baostock reported the session was dropped (``用户未登录``)."""
def _normalize_minute_frequency(frequency: str | int) -> tuple[str, str]:
"""Return Baostock frequency and partition label for a minute interval."""
raw = str(frequency).strip().lower()
if raw.endswith("m"):
raw = raw[:-1]
if raw not in _MINUTE_FREQUENCIES:
raise ValueError(
f"Unsupported minute frequency '{frequency}'. "
f"Expected one of {sorted(_MINUTE_FREQUENCIES)} minutes."
)
return raw, f"{raw}m"
def _parse_minute_datetime(date: pd.Series, time: pd.Series) -> pd.Series:
"""Parse Baostock minute timestamps into pandas datetimes."""
date_dt = pd.to_datetime(date, errors="coerce")
date_compact = date_dt.dt.strftime("%Y%m%d")
time_text = time.astype(str).str.strip()
time_digits = time_text.str.replace(r"\D", "", regex=True)
full_digits = time_digits.str.slice(0, 14)
from_full = pd.to_datetime(full_digits, format="%Y%m%d%H%M%S", errors="coerce")
from_short = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
short_time = time_digits.str.len().between(1, 6)
if short_time.any():
short_digits = (
time_digits.loc[short_time]
.str.pad(6, side="right", fillchar="0")
.str.slice(0, 6)
)
from_short.loc[short_time] = pd.to_datetime(
date_compact.loc[short_time] + short_digits,
format="%Y%m%d%H%M%S",
errors="coerce",
)
from_text = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
text_time = time_text.str.contains(":", regex=False)
if text_time.any():
from_text.loc[text_time] = pd.to_datetime(
date.astype(str).loc[text_time] + " " + time_text.loc[text_time],
errors="coerce",
)
return from_full.fillna(from_short).fillna(from_text)
def _download_akshare(symbol: str, start: str, end: str, adjust: str = "qfq") -> Optional[pd.DataFrame]:
"""Download daily bars from akshare. Returns DataFrame with OHLCV columns."""
try:
@@ -239,3 +296,104 @@ def download_daily_batch(
except Exception:
pass
def download_minute_batch(
symbols: Iterable[str],
start: str,
end: str,
frequency: str | int = 5,
relogin_every: int = 200,
) -> Iterator[Tuple[str, Optional[pd.DataFrame]]]:
"""Download raw Baostock minute bars for many symbols.
Minute bars are intentionally unadjusted (`adjustflag='3'`) because the
output is raw intraday market data for downstream feature aggregation, not a
tradable daily price series.
Args:
symbols: Internal-form symbols (``sh600000`` / ``sz000001``).
start, end: ``YYYY-MM-DD`` bounds.
frequency: Baostock minute frequency. ``5``/``"5"``/``"5m"`` all mean
5-minute bars.
relogin_every: Proactively refresh the baostock session every N symbols.
Yields:
``(symbol, df)`` where ``df`` has raw minute bars or ``None`` when no
data is available.
"""
query_frequency, frequency_label = _normalize_minute_frequency(frequency)
adjustflag = _BAOSTOCK_ADJUST["none"]
def _relogin() -> None:
try:
bs.logout()
except Exception:
pass
bs.login()
def _fetch(symbol: str) -> Optional[pd.DataFrame]:
"""One Baostock minute query; returns df, None, or raises _SessionLost."""
code = f"{symbol[:2]}.{symbol[2:]}"
rs = bs.query_history_k_data_plus(
code=code,
fields=_MINUTE_FIELDS,
start_date=start,
end_date=end,
frequency=query_frequency,
adjustflag=adjustflag,
)
if rs.error_code != "0":
if "未登录" in (rs.error_msg or ""):
raise _SessionLost(rs.error_msg)
logger.warning("baostock minute error for %s: %s", symbol, rs.error_msg)
return None
rows = []
while rs.next():
rows.append(rs.get_row_data())
if not rows:
return None
df = pd.DataFrame(rows, columns=_MINUTE_FIELDS.split(","))
df[_MINUTE_NUMERIC] = df[_MINUTE_NUMERIC].apply(pd.to_numeric, errors="coerce")
df["datetime"] = _parse_minute_datetime(df["date"], df["time"])
bad_timestamps = df["datetime"].isna()
if bad_timestamps.any():
raise ValueError(
f"Could not parse {int(bad_timestamps.sum())} minute timestamp(s)"
)
df["date"] = pd.to_datetime(df["date"], errors="coerce")
df["time"] = df["datetime"].dt.strftime("%H:%M:%S")
df["frequency"] = frequency_label
df["vwap"] = (df["amount"] / df["volume"]).where(df["volume"] > 0)
df["symbol"] = symbol
return df[_MINUTE_COLUMNS].sort_values("datetime").reset_index(drop=True)
bs.login()
try:
for i, symbol in enumerate(symbols):
if i and relogin_every and i % relogin_every == 0:
_relogin()
df: Optional[pd.DataFrame] = None
for attempt in (1, 2):
try:
df = _fetch(symbol)
break
except _SessionLost:
if attempt == 1:
_relogin()
continue
logger.warning("baostock minute session lost for %s after relogin", symbol)
except Exception as e:
logger.warning("baostock minute download failed for %s: %s", symbol, e)
break
if df is not None and not df.empty:
yield symbol, df
else:
yield symbol, None
finally:
try:
bs.logout()
except Exception:
pass
Binary file not shown.

After

Width:  |  Height:  |  Size: 156 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 107 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

+352
View File
@@ -0,0 +1,352 @@
# JoinQuant Comparison Plugin
## Why a Plugin
JoinQuant is an external execution and simulation reference. Keeping this code
under `plugins/joinquant/` prevents vendor-specific assumptions from entering
`pipeline/portfolio/`, where the internal reference simulator remains the
canonical implementation.
## What It Validates
The comparison is for system correctness:
- date alignment
- internal to JoinQuant symbol mapping
- target position generation
- once-per-day open execution timing
- lot rounding and filled shares
- position carry
- trading cost
- PnL accounting
- blocked trades from suspension, limit-up, and limit-down conditions
## What It Does Not Validate
It does not validate alpha quality, IC, IR, forecast skill, or whether the
strategy is economically useful. Differences can be expected when JoinQuant
uses different fee, slippage, cash, corporate-action, or internal rounding
rules.
## Historical Backtest Workflow
```bash
# 1. Build internal portfolio targets.
uv run python cli.py portfolio build ...
# 2. Export JoinQuant-compatible frozen targets.
uv run python cli.py joinquant export-targets \
--positions-path portfolio/run1.pq \
--portfolio-name run1 \
--mode target_shares \
--execution-calendar-path data/daily_bars/<universe> \
--out-dir plugins_output/joinquant/targets
# 3. Generate and copy the wrapper strategy and target files into JoinQuant.
uv run python cli.py joinquant write-wrapper \
--portfolio-name run1 \
--mode target_shares \
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
# 4. Run the JoinQuant backtest or simulated trading job.
# 5. Export JoinQuant fills, positions, and daily PnL to CSV.
# 6. Ingest JoinQuant output.
uv run python cli.py joinquant ingest \
--portfolio-name run1 \
--fills-csv path/to/jq_fills.csv \
--positions-csv path/to/jq_positions.csv \
--pnl-csv path/to/jq_pnl.csv
# 7. Reconcile.
uv run python cli.py joinquant reconcile \
--portfolio-name run1 \
--targets-dir plugins_output/joinquant/targets/run1 \
--our-fills-path fills/run1.pq \
--our-positions-path portfolio/run1.pq \
--our-pnl-path pnl/run1.pq \
--jq-fills-path plugins_output/joinquant/ingested/run1/fills.pq \
--jq-positions-path plugins_output/joinquant/ingested/run1/positions.pq \
--jq-pnl-path plugins_output/joinquant/ingested/run1/pnl.pq
```
## Forward-Testing Workflow
After the T-1 close and after the data update:
```bash
uv run python cli.py portfolio build ...
uv run python cli.py joinquant export-targets \
--positions-path portfolio/run1.pq \
--portfolio-name run1 \
--mode target_shares \
--execution-calendar-path data/daily_bars/<universe> \
--start-date T \
--end-date T
```
Before the T open, upload or expose the frozen target file to JoinQuant. During
the T open, the JoinQuant wrapper reads that file and submits orders, while the
internal simulator should run against the same frozen target. After T close or
after JoinQuant results are available, ingest the JoinQuant CSV files and run
`joinquant reconcile`.
Forward target files must be frozen before execution. Do not regenerate a
target file after observing open or close data for the same trading date. The
exporter writes a snapshot JSON with a SHA-256 hash for this reason and refuses
to overwrite existing target/snapshot files unless `--force` is passed.
For comparisons against the internal simulator, pass
`--execution-calendar-path` to `joinquant export-targets`. The positions file is
dated by construction/signal date, while the simulator executes at the next
available open. The calendar option shifts exported target files to that next
trading date, so JoinQuant reads the same target on the same execution session.
For JoinQuant 模拟盘, the browser automation has two operational phases:
```bash
# Before T open: upload frozen target(s), save strategy, and start/restart 模拟盘.
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--strategy-url "https://www.joinquant.com/<your 模拟盘 page>" \
--flow sim-trade
uv run python cli.py joinquant run-browser-sim \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
--headed
# After T close: download/export JoinQuant fills, positions, and pnl, then
# ingest/reconcile. The same run-browser-sim command can do this if the config
# includes download actions, otherwise use the ingest/reconcile commands.
```
The default simulated-trading template includes selectors for saving the
strategy and clicking simulated-trading controls such as `模拟盘`, `模拟交易`,
`启动`, and `重启`. These selectors are intentionally configurable because the
JoinQuant web UI can differ by account and page version.
## Target-Shares Mode
`target_shares` is the default and preferred correctness mode. The exported
`target_shares` field comes from the internal `position_shares` column produced
by `portfolio build`, because the internal simulator executes that discretized
integer book. The generated wrapper calls:
```python
order_target(jq_symbol, target_shares)
```
This mode makes filled shares, position carry, and blocked trades easiest to
compare.
## Target-Value Mode
`target_value` mode exports `target_value` and `target_weight` from the
portfolio file. The generated wrapper calls:
```python
order_target_value(jq_symbol, target_value)
```
This can be useful for portfolio-level comparisons, but JoinQuant may apply its
own rounding, cash, and lot rules. Differences are often classified as
`JOINQUANT_INTERNAL_ROUNDING`, `LOT_ROUNDING`, or `CASH_CONSTRAINT` depending
on the observed output.
## Symbol Mapping
Internal symbols are converted as follows:
```text
sh600000 -> 600000.XSHG
sh688001 -> 688001.XSHG
sz000001 -> 000001.XSHE
sz300001 -> 300001.XSHE
```
Reverse mapping is also supported. Invalid exchanges or unsupported A-share
prefixes raise `ValueError` instead of silently guessing.
## Wrapper Strategy Usage
Generate a configured wrapper:
```bash
uv run python cli.py joinquant write-wrapper \
--portfolio-name run1 \
--mode target_shares \
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
```
Copy the generated file and daily CSV target files into JoinQuant. The default
loader uses JoinQuant `read_file`, which works for uploaded files. If your
JoinQuant runtime allows HTTP or another storage backend, replace only
`_read_target_file()` in the generated strategy.
The wrapper is long-only by default:
```python
ALLOW_SHORT = False
```
Negative targets are clipped to zero and logged. Use `--allow-short` only if
the target JoinQuant account supports the required shorting mechanics.
## Ingesting JoinQuant Outputs
The ingest command accepts permissive CSV column names and writes strict plugin
schemas:
```text
plugins_output/joinquant/ingested/{portfolio_name}/fills.pq
plugins_output/joinquant/ingested/{portfolio_name}/positions.pq
plugins_output/joinquant/ingested/{portfolio_name}/pnl.pq
```
Missing cost fields default to zero. Missing blocked status defaults to zero.
Symbols and dates are normalized.
## Reading Reconciliation Reports
The reconcile command writes:
```text
plugins_output/joinquant/reconcile/{portfolio_name}/daily_reconcile.pq
plugins_output/joinquant/reconcile/{portfolio_name}/summary.csv
plugins_output/joinquant/reconcile/{portfolio_name}/summary.md
```
`daily_reconcile.pq` is per-symbol and includes target shares, internal filled
shares, JoinQuant filled shares, realized positions, trade prices, costs, PnL,
and a `diff_reason`. `summary.csv` is the daily portfolio-level view for gross
exposure, net exposure, cash, total value, PnL, cumulative PnL, turnover, and
cost.
Difference reasons include:
```text
MATCH SYMBOL_MAPPING PRICE_MISMATCH LOT_ROUNDING SUSPENSION LIMIT_UP_BLOCK
LIMIT_DOWN_BLOCK VOLUME_OR_LIQUIDITY COST_MODEL CASH_CONSTRAINT
SHORT_NOT_SUPPORTED CORPORATE_ACTION JOINQUANT_INTERNAL_ROUNDING
MISSING_IN_OUR_SYSTEM MISSING_IN_JOINQUANT UNKNOWN
```
Default tolerances are exact share matching, `1e-4` relative trade-price
tolerance, and value tolerance `max(1 yuan, 1e-6 * booksize)`. PnL tolerance is
configurable with `--pnl-tolerance`.
## Minimal Example
Create a 5-stock equal-weight or fixed-share test portfolio:
```text
sh600000, sz000001, sh600519, sz002594, sz300750
```
Build positions for a small date range, export `target_shares`, upload the CSV
files and wrapper to JoinQuant, run the JoinQuant backtest, export fills,
positions, and PnL, then run ingest and reconcile. Start with one or two days
before expanding the sample.
For the first one-stock long-only smoke test, the local side can be prepared in
one command:
```bash
uv run python cli.py joinquant prepare-smoke \
--out-dir /tmp/chinese-equity-quant-realdata
```
The command downloads a tiny public daily-bar sample, builds a fixed-share
`sh600000` long-only position file, simulates it internally, exports aligned
JoinQuant target files, writes a configured wrapper strategy, and creates
`joinquant_smoke_manifest.json` with all output paths.
## Browser Backtest Automation
JoinQuant's public `jqdatasdk` is a data SDK. It supports authenticated data
calls such as `auth(username, password)` and `get_price(...)`, but cloud
strategy upload, backtest execution, and result export are web-application
workflows. The plugin therefore automates those remote steps through Playwright
with a saved browser session.
Install the optional browser runner in the uv environment:
```bash
uv sync --extra joinquant-browser
uv run playwright install chromium
```
Save a reusable login state. This opens a browser; log in normally, including
any CAPTCHA or 2FA, then press Enter in the terminal to save state:
```bash
uv run python cli.py joinquant browser-login \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
```
Create a selector/action config for a historical backtest:
```bash
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--strategy-url "https://www.joinquant.com/<your strategy page>" \
--flow backtest
```
If selectors need tuning, capture the logged-in page:
```bash
uv run python cli.py joinquant browser-snapshot \
--url "https://www.joinquant.com/<your strategy page>" \
--out-dir /tmp/chinese-equity-quant-realdata/browser_snapshot
```
Then run the remote backtest automation:
```bash
uv run python cli.py joinquant run-browser-backtest \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
--headed
```
The config is declarative: actions can navigate, paste the generated wrapper,
upload all target CSV files, fill dates, click run, wait for completion,
download result CSVs, and take screenshots. When the configured downloads
produce fills, positions, and PnL CSVs, the runner automatically calls
`joinquant ingest` and `joinquant reconcile`.
For forward testing / 模拟盘, create the config with `--flow sim-trade` and run:
```bash
uv run python cli.py joinquant run-browser-sim \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
--headed
```
Do not store raw JoinQuant passwords in this repository. The browser state file
is created with `0600` permissions and should live under `~/.config`, outside
the repo.
## Recommended First Sanity Checks
1. One liquid stock with a fixed target share count.
2. A 10-stock equal-weight long-only portfolio.
3. A forced suspension, limit-up, and limit-down sample.
4. A short target in long-only mode to confirm `SHORT_NOT_SUPPORTED`.
5. A 5-day reversal portfolio after the mechanical checks pass.
## Known Limitations
- JoinQuant internal execution details may differ from the reference simulator.
- External file loading depends on the JoinQuant environment.
- Short selling may not be supported.
- Fee, tax, slippage, and minimum-fee models may differ.
- Corporate actions may need special handling and should not be hidden.
- The internal simulator does not currently emit execution price in
`FILL_COLUMNS`; price reconciliation uses explicit price columns if supplied.
+167
View File
@@ -0,0 +1,167 @@
# JoinQuant Cost Model Findings
Generated: 2026-07-06
This report summarizes the JoinQuant trading-cost behavior observed from the
browser-automated real-data backtests and compares it with the current internal
simulator model. The JoinQuant cost formula below is inferred from rendered
transaction tables and strategy logs for this account. Treat it as an observed
platform default, not as a guaranteed external contract.
## Runs Used
### Longer Comparison Run
- Portfolio: `jq_long_one_stock_long`
- Window: `2024-01-11` to `2024-02-29`
- Booksize: `1,000,000 CNY`
- Instrument: `600000.XSHG`
- Target: buy and hold `1,000` shares
- JoinQuant rendered result: completed
- Local total PnL: `546.22 CNY`
- JoinQuant total PnL from positions tab: `575.00 CNY`
- Difference: `28.78 CNY`, or `0.002878` percentage points on the book
The first JoinQuant transaction was:
| Date | Side | Shares | Price | Turnover | Fee |
|---|---:|---:|---:|---:|---:|
| `2024-01-11` | Buy | `1,000` | `6.57` | `6,570.00` | `5.00` |
That trade hit a minimum fee. The local simulator charged `6.1415 CNY` because
the smoke runner used a flat `10 bps` cash cost on adjusted-price turnover.
### Cost Probe Run
- Portfolio: `jq_cost_probe_buy_sell`
- Window: `2024-01-11` to `2024-01-12`
- Booksize: `1,000,000 CNY`
- Instrument: `600000.XSHG`
- Targets:
- `2024-01-11`: buy `100,000` shares
- `2024-01-12`: sell to `0` shares
Observed JoinQuant transactions:
| Date | Side | Shares | Price | Turnover | Fee | Implied Fee |
|---|---:|---:|---:|---:|---:|---:|
| `2024-01-11` | Buy | `100,000` | `6.57` | `657,000.00` | `197.10` | `3.0000 bps` |
| `2024-01-12` | Sell | `-100,000` | `6.51` | `651,000.00` | `846.30` | `13.0000 bps` |
The transaction-table numbers match this formula exactly:
```text
buy fee = max(5 CNY, turnover * 0.0003)
sell fee = max(5 CNY, turnover * 0.0003) + turnover * 0.001
```
In basis points:
- Buy commission: `3 bps`
- Sell commission: `3 bps`
- Sell stamp tax: `10 bps`
- Minimum commission: `5 CNY`
No separate transfer fee was visible in this probe. If a separate transfer fee
was present as an additional charge, the observed fees would not match the
formula above exactly. It may still be folded into JoinQuant's displayed
commission field, so this finding should be read as "not separately observable"
rather than "impossible".
No slippage was visible. The wrapper submitted open-time market orders with
`run_daily(..., time="open")`, and JoinQuant filled them at the displayed open
prices.
## Evidence From Strategy Logs
The JoinQuant order log for the cost probe showed:
```text
2024-01-11 ... trade price: 6.57, amount:100000, commission: 197.1
2024-01-12 ... trade price: 6.51, amount:100000, commission: 846.3
```
The normal transaction tab showed the same fee values. However, the generated
wrapper's `JOINQUANT_FILL` log records had `trade_cost: 0.0`, even for those
same fills. That means `get_trades()` did not expose the usable commission
value through the field the wrapper currently reads.
For reconciliation, use the transaction table or JoinQuant order logs for fee
details. Do not rely on the wrapper's current `JOINQUANT_FILL.trade_cost`.
## Difference From The Internal Simulator
The current internal simulator cost model is
`SimpleProportionalCostModel` in `pipeline/portfolio/costs.py`:
```text
trade_cost = abs(traded_shares * execution_price)
* (cost_bps + slippage_bps) / 10000
```
The smoke runner used:
- `cost_bps = 5`
- `slippage_bps = 5`
- combined one-way cash cost: `10 bps`
Important differences:
- The internal simulator uses the same rate for buys and sells.
- It has no minimum commission.
- It has no sell-only stamp tax.
- Slippage is modeled as an extra cash cost.
- JoinQuant did not show slippage in the observed open-time fills.
- The local smoke download used `adjust="qfq"`, while the JoinQuant wrapper set
`set_option("use_real_price", True)`. That price-scale mismatch also affects
PnL and cost comparisons.
## Practical Implications
For a buy-only smoke test, JoinQuant may charge less than the local model when
the trade is large enough for `3 bps` to apply, but it may charge more on small
orders because of the `5 CNY` minimum.
For any test with sells, JoinQuant's default sell fee is materially higher than
the current local flat model because of the inferred `10 bps` stamp tax.
The earlier 30-day buy-and-hold discrepancy was small because only one buy was
executed. A rebalancing strategy with many sells will show a larger cost-model
difference unless the local simulator is configured to match JoinQuant.
## Recommended Follow-Ups
1. Add a JoinQuant-style cost model to the internal simulator:
```text
commission = max(min_commission, turnover * commission_bps / 10000)
stamp_tax = turnover * sell_stamp_tax_bps / 10000 for sells only
trade_cost = commission + stamp_tax
```
2. Add a CLI option or preset for `portfolio simulate`, for example
`--cost-model joinquant-stock`.
3. Update JoinQuant reconciliation to parse fee values from the transaction
table or order logs when CSV exports are unavailable.
4. Run a second local-vs-JoinQuant comparison with:
- raw or real-price local bars, not adjusted-price bars
- JoinQuant-style costs
- slippage disabled locally
That test should isolate remaining differences to data alignment, price source,
rounding, and JoinQuant internal execution behavior.
## Local Artifacts
The temporary artifacts from the investigation are:
- `/tmp/chinese-equity-quant-jq-long/comparison_report.md`
- `/tmp/chinese-equity-quant-jq-long/parsed_joinquant/daily_pnl_compare_from_positions_tab.csv`
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_analysis_report.md`
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_analysis_summary.json`
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_probe_transactions_parsed.csv`
- `/tmp/chinese-equity-quant-jq-cost-probe/detail_tabs/transactions.txt`
+70
View File
@@ -0,0 +1,70 @@
# Minute Bar Data Notes
The minute-bar path downloads raw Baostock intraday bars and stores them as a
Hive-partitioned dataset:
```bash
uv run python cli.py data download-minute \
--universe sh600000 \
--start-date 2024-01-02 --end-date 2024-01-05 \
--frequency 5
```
The default layout is:
```text
data/minute_bars/{universe}/frequency=5m/month=YYYY-MM/*.pq
```
Derived-data plugins can aggregate those bars to daily `symbol_id,date` numeric
files, for example:
```bash
uv run python cli.py derived compute \
--minute-path data/minute_bars/sh600000 \
--daily-path data/daily_bars/sh600000 \
--derived-type minute_daily_summary \
--derived-name minute_summary
```
The legacy `feature compute` command delegates to the same derived-data
registry and remains available for existing scripts.
## Daily vs Minute Reconciliation
Baostock's daily raw bars and 5-minute raw bars are close, but they should not
be treated as perfectly reconstructible from each other.
When checking consistency, compare daily raw bars (`data download --adjust none`)
against minute bars on the same raw price scale. The minute aggregation should
use:
- `open`: first minute open
- `high`: max minute high
- `low`: min minute low
- `close`: last minute close
- `volume`: sum minute volume
- `amount`: sum minute amount
- `vwap`: `sum(amount) / sum(volume)`
In a sanity check for `sh600000` from `2024-01-02` through `2024-01-05`, Baostock
returned 4 daily rows and 192 5-minute bars, exactly 48 bars per day. Open, low,
and close matched daily exactly on all 4 days. High matched on 3 of 4 days; on
`2024-01-04`, the daily high was `6.67` while the max 5-minute high was `6.66`.
Minute-summed volume and amount were higher than daily by roughly `0.16%` to
`1.23%`. VWAP remained very close, with max relative difference around
`0.0043%`.
This appears to be a source-level Baostock reconciliation caveat, not a parser
or ordering issue: the minute bars covered the regular `09:35:00` through
`15:00:00` range and sorted correctly by timestamp.
Practical guidance:
- Use tolerance-based daily-vs-minute checks; do not require exact equality for
high, volume, or amount.
- Expect open/close alignment to be a stronger sanity check than exact volume
reconstruction.
- Use minute-derived values as separate daily features, not as replacements for
the canonical daily bar dataset unless a strategy explicitly wants that
source convention.
+140
View File
@@ -0,0 +1,140 @@
# Portfolio Trading Cost Model
This document describes the trading cost model used by `portfolio simulate`.
The current implementation is a simplified open-execution proportional cost
model. It is intentionally small, explicit, and easy to audit.
## Open-Execution Timeline
The simulator runs once per trading day:
1. A constructed portfolio row provides the target book for an execution date.
In the current file layout, a target dated `t` is executed at the next
available market date `d = next(t)`.
2. Trades are executed at `open[d]`.
3. Realized positions are held during the trading day.
4. Daily PnL is marked from `open[d]` to `close[d]` on the newly realized book,
plus any overnight gap from the previous realized holdings.
5. Trading cost is charged only on actually realized `traded_shares`, after all
constraints have clipped the desired trade.
This means a fully blocked order has `traded_shares = 0` and therefore zero
trading cost.
## Current Formula
For each symbol:
```text
trade_value_i = abs(traded_shares_i * execution_price_i)
trade_cost_i = trade_value_i * (cost_bps + slippage_bps) / 10000
```
where:
```text
execution_price_i = open_price_i
```
`cost_bps` is the proportional explicit trading-cost rate in basis points.
`slippage_bps` is modeled as an additional cash cost in basis points. The two
rates are added linearly. The CLI options `--cost-bps` and `--slippage-bps`
both default to `0.0`.
Both rates are **one-way, per-trade**: the combined `(cost_bps + slippage_bps)`
is charged on the traded notional of *each* fill, buy and sell alike. A full
round trip (enter then exit a position) is therefore charged twice — e.g.
`5 + 5` bps becomes ~20 bps over a complete round trip, not 10. Quote any
round-trip figure by doubling, or convert a round-trip budget to a per-trade
rate by halving before passing it in.
Example:
```text
traded_shares = 1000
execution_price = 20 yuan
cost_bps = 10
slippage_bps = 5
abs(1000 * 20) * 15 / 10000 = 30 yuan
```
## Slippage Convention
Slippage is not applied by changing the execution price. It is charged only as
a cash cost through `trade_cost`.
Do not double-count slippage by doing both:
```text
execution_price = open * (1 +/- slippage_bps / 10000)
trade_cost += trade_value * slippage_bps / 10000
```
The simulator should execute at the open price and subtract the slippage cash
cost from PnL.
## Relationship To The Simulator
`ReferenceSimulator.fill()` clips desired trades through constraints first, then
passes the actual `traded_shares` to the cost model. The per-name result is
stored in the fills parquet as `trade_cost`.
`ReferenceSimulator.run()` sums per-name `trade_cost` into the daily PnL row's
`cost` column and subtracts that total from daily PnL:
```text
pnl = overnight + intraday - cost_total
```
## What This Model Does Not Cover
The current model intentionally does not model:
- Minimum commissions.
- Buy/sell asymmetric fees.
- Sell-side stamp duty.
- Exchange handling fees.
- Regulatory fees.
- Transfer fees.
- Date-aware fee schedule changes.
- Nonlinear price impact.
- Auction liquidity / queue effects.
- Partial fills caused by open auction depth.
These omissions are deliberate. The current model is the default reference
model, not a detailed brokerage fee simulator.
## Future Extension
The simulator is structured around a cost model abstraction:
```python
class CostModel:
def compute(
self,
traded_shares,
execution_price,
side,
date,
metadata,
):
...
```
The current implementation is `SimpleProportionalCostModel`.
A future `AShareDetailedCostModel` can add:
- Commission, optionally subject to minimum commission.
- Sell-side stamp duty.
- Transfer fee.
- Exchange handling fee.
- Regulatory fee.
- Date-aware fee rates.
- Separate buy-side and sell-side rates.
- Optional nonlinear slippage / market-impact model.
Any future model must preserve the same high-level simulator contract: costs
are computed from realized trades after constraints, and slippage must not be
counted both through execution-price adjustment and cash cost.
@@ -0,0 +1,509 @@
# Tutorial: Testing a 5-Day Reversal Alpha
This document is a teaching walkthrough for someone who is new to this research
framework and only lightly familiar with quant research. We will use one
concrete experiment, a 5-day reversal alpha on the full downloaded Chinese
A-share universe, to learn how the framework defines an alpha, stores it, tests
it, turns it into a portfolio, and explains the gap between a research result
and simulated trading PnL.
This generated version was refreshed at 2026-06-12T22:52:56.
The important point is not the timestamp; it is the research method.
## The Research Question
A quant research project starts with a hypothesis:
> If a stock fell a lot over the last few trading days, it may rebound soon; if
> it rose a lot, it may cool off soon.
This is called **short-horizon reversal**. It is a simple idea: recent losers
are candidates to buy, and recent winners are candidates to sell or underweight.
In this repo, the tested version looks back 5 trading days.
The central research question is:
> Does this 5-day reversal rule create useful portfolio returns after the
> framework applies realistic storage, portfolio construction, execution
> constraints, and trading costs?
The answer from this run is nuanced:
- The naive built-in version is positive under the tradable
next-open-to-next-open research convention (**41.40%**),
but its stored weights still show that raw z-score weighting is too sensitive
to A-share outliers.
- A rank-weighted version on a liquid, non-ST, tradable universe has a positive
costless research result: **209.58%**
at Sharpe **1.44**.
- The daily-traded implementation is still not tradable after costs because
turnover is too high.
That is a normal research outcome. Good research is not just asking "did the
backtest go up?" It is asking **which layer explains the result**: signal,
weighting, universe, construction, execution, or cost.
## How This Framework Defines An Alpha
In many quant textbooks, an alpha is described as a **prediction** of future
returns. This framework uses a stricter and more practical convention:
> An alpha is a signed cross-sectional position weight.
That sentence is the key to the whole repo.
- **Signed** means positive values are long exposure and negative values are
short exposure.
- **Cross-sectional** means the alpha compares stocks to other stocks on the
same date.
- **Position weight** means the output is already an instruction about what the
portfolio wants to own. It is not merely a score to correlate with future
returns.
The stored alpha file always has this schema:
| column | meaning |
| --- | --- |
| `symbol_id` | Stock identifier such as `sh600000` or `sz000001`. |
| `date` | The signal date. The alpha is formed using information known by this date's close. |
| `alpha_name` | A label for this particular run, such as `reversal_5d_all`. |
| `weight` | Signed desired exposure. Positive means long; negative means short. |
Because the framework treats alphas as position weights, it evaluates them with
portfolio metrics: return, Sharpe, turnover, drawdown, and hit rate. It does
**not** use IC/IR, because IC/IR would treat the alpha as a return predictor.
## The Pipeline In One Picture
Every phase reads parquet files and writes parquet files. That makes the system
easy to inspect and rerun one layer at a time.
```text
daily bars
-> alpha weights
-> combined weights
-> portfolio targets and integer positions
-> simulated fills and PnL
-> evaluation metrics
```
For this experiment, the important phases are:
| phase | command family | what it teaches you |
| --- | --- | --- |
| Data | `cli.py data download` | What market data is available. |
| Alpha compute | `cli.py alpha compute` | How a raw research idea becomes stored weights. |
| Alpha eval | `cli.py alpha eval` | How close-formed weights perform over the tradable next-open-to-next-open interval. |
| Combo | `cli.py combo combine` | How one or more alphas become one combined book. |
| Portfolio build | `cli.py portfolio build` | How weights become target values and integer shares. |
| Portfolio simulate | `cli.py portfolio simulate` | How the integer book trades at next open with constraints and costs. |
| Portfolio eval | `cli.py portfolio eval` | How the continuous target portfolio behaves over the same costless open-to-open research interval. |
In a real research workflow, you should learn to pause after every phase and
inspect the parquet output. Most mistakes are easier to find at the interface
between two phases than at the final PnL line.
## Step 1: Define The Raw Reversal Signal
The built-in 5-day reversal alpha is implemented as:
```python
signal = -close.pct_change(5, fill_method=None)
```
For stock `i` on date `t`, this is approximately:
```text
signal[i, t] = -(close[i, t] / close[i, t-5] - 1)
```
So:
- If a stock rose by 10% over the last 5 trading days, the raw signal is `-10%`.
It becomes a candidate short or underweight.
- If a stock fell by 10% over the last 5 trading days, the raw signal is `+10%`.
It becomes a candidate long or overweight.
Notice the timing. The signal uses prices through date `t`. It must not use the
return from `t` to `t+1`, because that is the future. The costless alpha
evaluator tests the weight formed on date `t` over the tradable interval from
`open[t+1]` to `open[t+2]`; the later execution simulator is the separate layer
that trades the constructed integer book at the next open.
The code lives in `pipeline/alpha/library/reversal.py`:
```python
class ReversalAlpha(BaseAlpha):
name = "reversal"
def __init__(self, lookback: int = 5):
self.lookback = lookback
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
return -close.pct_change(self.lookback, fill_method=None)
```
The alpha class only defines the raw signal. The base class then turns that
signal into weights.
## Step 2: Turn A Signal Into Cross-Sectional Weights
By default, `BaseAlpha.to_weights()` does a cross-sectional z-score each date:
```text
weight[i, t] = (signal[i, t] - mean_signal[t]) / std_signal[t]
```
This means the framework asks:
> On this date, which stocks have stronger reversal scores than the rest of the
> market, and by how much?
That is useful, but it has a weakness. If a few stocks have extreme trailing
returns because they are newly listed, suspended, illiquid, or limit-constrained,
z-scoring can put a very large amount of relative exposure into exactly those
names.
That is visible in the naive full-universe run. Stored weights reached about
`-52` standard deviations. The
result is positive under the open-to-open convention, but it is much weaker and
less robust than the rank-weighted versions:
| run | weighting | research cumulative return | research Sharpe | research turnover/year |
| --- | --- | --- | --- | --- |
| naive z-score, full universe | z-score | 41.40% | 0.4514 | 160x |
The lesson is not "reversal is solved." The lesson is:
> The same raw signal can become a fragile portfolio if the weighting method
> reacts badly to outliers.
## Step 3: Make The Weighting More Robust
The repo also has a rank-weighted version, `reversal_rank`. It uses the same raw
5-day reversal signal, but converts the cross-section to ranks instead of
z-scores:
```python
ranks = signal.rank(axis=1)
weights = ranks.subtract(ranks.mean(axis=1), axis=0)
```
Rank weighting keeps the ordering of stocks but removes the importance of the
exact outlier magnitude. A stock can be "the worst recent loser" or "the best
recent winner," but it cannot become dozens of standard deviations important
just because its raw percentage move is unusual.
The full-universe rank version was much less pathological, but still not a
clean signal:
| run | weighting | research cumulative return | research Sharpe | research turnover/year |
| --- | --- | --- | --- | --- |
| rank, full universe | rank | 73.44% | 0.8860 | 143x |
That tells us the weighting fix helped, but the universe still contains many
names that are poor candidates for a daily reversal strategy.
## Step 4: Define The Investable Universe
An alpha should be tested on stocks that could plausibly be traded. The liquid
run applies a per-date mask before weights are created. A stock must be:
- seasoned, with at least 60 observed closes;
- currently tradable, using `tradestatus == 1`;
- not ST, using `isST == 0`;
- inside the top 1000 names by trailing 20-day average traded amount.
This mask is applied to the signal, not to the price history used to compute the
5-day return. That distinction matters. We still compute `pct_change(5)` on the
full contiguous price history, then decide which names are eligible to hold on
each signal date.
The liquid rank result is the cleanest research result:
| run | weighting | universe | research cumulative return | research Sharpe | hit rate |
| --- | --- | --- | --- | --- | --- |
| rank, liquid subset | rank | top 1000 liquid, tradable, non-ST | 209.58% | 1.4422 | 55.68% |
This is the first point where a researcher can say:
> There appears to be a real 5-day reversal effect in a cleaner A-share
> universe, before trading costs.
That last phrase, **before trading costs**, is essential.
![Research equity](assets/reversal_5d_research_equity.png)
When reading this chart, focus on the shape and relative behavior:
- The naive z-score line shows why outlier-sensitive weighting is fragile.
- The rank full-universe line shows that robust weighting helps, but the full
universe still contains noisy and hard-to-trade names.
- The liquid rank line shows the signal-level edge before execution costs.
## Step 5: Check That The Alpha File Is Sane
Before trusting any metric, inspect the stored alpha artifact. The run checked:
- The columns match `ALPHA_COLUMNS`.
- There are no null weights.
- There are no non-finite weights.
- There are no duplicate `(symbol_id, date)` rows.
- The daily cross-sectional mean is approximately zero.
- A one-alpha combo is an exact identity transform.
| run | schema ok | null w | non-finite w | dup keys | max \|daily mean\| | weight range | combo identity Δ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| naive z-score (full) | True | 0 | 0 | 0 | 3.32e-16 | [-52.2, 19.2] | 0.00e+00 |
| rank (full) | True | 0 | 0 | 0 | 0.00e+00 | [-2603.0, 2603.0] | 0.00e+00 |
| rank (liquid subset) | True | 0 | 0 | 0 | 0.00e+00 | [-498.5, 498.5] | 0.00e+00 |
The rank ranges look numerically large because rank weights scale with the
number of names. That is fine: later evaluation divides by gross exposure, and
portfolio construction normalizes by `sum(abs(weight))`. The important
difference is that rank weights are bounded by cross-sectional rank, not by the
raw size of an abnormal stock move.
![Weight distributions](assets/reversal_5d_weight_distributions.png)
This is a good habit: when a backtest looks strange, plot the weights before
debugging the PnL. A broken or concentrated weight distribution often explains
the result.
## Step 6: Understand The Alpha Evaluation Formula
The costless alpha evaluator now asks:
> If we compute alpha weights after close on date `t`, trade them at `open[t+1]`,
> and hold them until `open[t+2]`, what return would we earn before costs?
This is still a **research-layer approximation**, not the trading simulator. At
this stage the framework has only an alpha weight file. It has not yet rounded
shares, checked limits, clipped trades, or paid costs. The purpose is to answer
a clean signal question: "Do these close-formed weights line up with returns
over the interval we could actually own after next-open execution?"
The daily research return is:
```text
R[t] = sum_i(weight[i, t] * (open[i, t+2] / open[i, t+1] - 1)) / sum_i(abs(weight[i, t]))
```
This has three important consequences:
- The alpha is normalized by its gross exposure, so the scale of raw weights
does not by itself create a higher return.
- The new signal does not receive credit for the overnight gap from `close[t]`
to `open[t+1]`, because it cannot be traded until `open[t+1]`.
- The final two signal dates are dropped from performance metrics because they
do not have a complete next-open-to-next-open holding interval.
Turnover is still measured from the weights:
```text
turnover[t] = sum_i(abs(weight[i, t] - weight[i, t-1])) / sum_i(abs(weight[i, t-1]))
```
The annualized turnover numbers are a warning. Even a positive signal can be
hard to monetize if it asks the portfolio to trade too much every day.
## Step 7: Build A Portfolio From The Alpha
The alpha file is still an abstract research book. `portfolio build` turns it
into target exposures and integer shares.
The main normalization is:
```text
target_weight[i, t] = weight[i, t] / sum_i(abs(weight[i, t]))
target_value[i, t] = booksize * target_weight[i, t]
target_shares[i, t] = target_value[i, t] / construction_price[i, t]
```
Then the framework creates an integer A-share book using lot rules and repair
logic. This is where a research portfolio starts to become a tradable portfolio.
The continuous target portfolio matched the stored alpha almost exactly:
| run | target_value identity max\|Δ\| | alpha→target max\|Δ\| | research corr(alpha,portfolio) | mean integer gross | mean L1 tracking |
| --- | --- | --- | --- | --- | --- |
| naive z-score (full) | 0.0000 | 0.00e+00 | 1.000000 | 9,138,331 | 2,542,655 |
| rank (full) | 0.0000 | 0.00e+00 | 1.000000 | 8,984,098 | 2,678,278 |
| rank (liquid subset) | 0.0000 | 0.00e+00 | 1.000000 | 9,810,256 | 862,303 |
The integer book is not exact because small target positions can be rounded
away. The liquid subset has lower tracking error because it spreads the book
over fewer and more tradable names.
![Portfolio tracking](assets/reversal_5d_portfolio_tracking.png)
When you research a new alpha, ask two separate questions:
- Does the continuous target portfolio match the alpha? It should.
- Does the integer tradable portfolio still resemble the target? It may not,
especially for small books or very broad universes.
## Step 8: Simulate Execution And Costs
Research returns are not the same as tradable PnL. The simulator executes the
integer `position_shares` at the next available open and applies constraints:
- suspension;
- price limit;
- volume cap;
- proportional trading cost.
The cost model is:
```text
cost = abs(traded_shares * open) * (cost_bps + slippage_bps) / 10000
```
For this run, cost is 5 bps commission plus 5 bps slippage. Slippage is treated
as cash cost, not as an additional execution price adjustment.
The execution results explain the final research conclusion:
| run | corr(alpha, exec net) | PnL before cost | total cost | net PnL | mean daily turnover |
| --- | --- | --- | --- | --- | --- |
| naive z-score (full) | 0.8956 | 1,838,974 | 13,032,720 | -11,193,746 | 0.5711 |
| rank (full) | 0.9126 | 5,052,067 | 11,713,451 | -6,661,383 | 0.5133 |
| rank (liquid subset) | 0.8884 | 11,017,842 | 12,733,803 | -1,715,960 | 0.5715 |
For the liquid rank run, simulated PnL before cost is about
11,017,842, but total cost is about
12,733,803. That is why the final net PnL is
weak or negative.
This is not a contradiction. It is exactly what a research pipeline should show:
> The signal can exist in the costless layer, but the daily implementation can
> still trade too much to keep the edge.
![Execution vs research](assets/reversal_5d_exec_vs_research.png)
## Step 9: Read The Headline Metrics Like A Researcher
The complete summary is:
| run | weighting | research cum | research Sharpe | research turn/yr | exec before cost | exec net | exec net Sharpe |
| --- | --- | --- | --- | --- | --- | --- | --- |
| naive z-score (full) | z-score | 41.40% | 0.4514 | 160× | 18.39% | -111.94% | -1.4508 |
| rank (full) | rank | 73.44% | 0.8860 | 143× | 50.52% | -66.61% | -1.1839 |
| rank (liquid subset) | rank | 209.58% | 1.4422 | 148× | 110.18% | -17.16% | -0.2226 |
*Research = costless, no-look-ahead weights over the next-open-to-next-open
holding interval. Execution = next-open fills on the discretized integer book
under suspension / price-limit / volume-cap constraints, 5 bps commission + 5
bps slippage.*
Here is the interpretation:
- **Naive z-score full universe**: positive under open-to-open research, but a
less reliable test of the reversal idea because the weighting scheme lets
outliers dominate parts of the book.
- **Rank full universe**: a better test of the same idea, but still noisy
because the universe includes too many problematic names.
- **Rank liquid subset**: the best signal-level test; it finds the cleanest
costless reversal effect.
- **Execution net**: daily rebalancing remains heavily constrained by cost.
A beginner might look only at the final net PnL and say "the alpha failed." A
researcher should be more precise:
> The raw 5-day reversal idea has signal value in a liquid universe, but the
> current daily trading rule has too much turnover for the assumed cost model.
## Step 10: Reproduce The Experiment
These commands reproduce the important artifacts, assuming the full daily-bar
dataset already exists at `data/daily_bars/all`.
```bash
# Naive z-score baseline: built-in reversal alpha, full universe.
uv run python cli.py alpha compute --data-path data/daily_bars/all \
--alpha-name reversal_5d_all --alpha-type reversal --lookback 5 \
--output-dir alphas
# Rank-weighted full and liquid runs.
bash scripts/run_reversal_rank_e2e.sh
# Regenerate figures, diagnostics, and this tutorial report.
uv run python scripts/generate_reversal_5d_report.py
```
If you are learning the framework, do not run the whole pipeline blindly. Run
one phase, inspect the output parquet, then continue.
## How To Research Your Own Alpha
Use this checklist for a new idea.
1. State the hypothesis in plain language.
Example: "Stocks with poor 5-day returns may rebound over the next day."
2. Write the raw signal.
Implement `signal(close) -> wide DataFrame` in an alpha class. Higher values
should mean stronger long preference.
3. Choose the weighting method.
The default z-score is useful, but it can be fragile. Consider rank weights,
caps, neutralization, or liquidity-aware filters if outliers dominate.
4. Define the investable universe before trusting results.
Make sure the strategy is not depending on suspended, ST, newly listed, or
illiquid names.
5. Evaluate the alpha as a portfolio, not as a prediction.
Check cumulative return, Sharpe, drawdown, hit rate, and turnover over the
next-open-to-next-open holding interval. Do not add IC/IR unless the
framework's alpha convention changes.
6. Build the portfolio and inspect tracking.
Confirm that target weights match the alpha, then check whether integer
shares still track the target book.
7. Simulate execution with costs.
The final research question is not only "is there a signal?" It is "is there
enough signal left after realistic trading?"
8. Diagnose the failure layer.
If results are bad, identify whether the problem is the raw signal, weighting,
universe, construction, execution constraints, turnover, or cost.
For this 5-day reversal study, the diagnosis is clear: **the signal-level result
is most promising after robust weighting and a liquid universe filter, but the
current implementation needs turnover control before it can be considered
tradable.**
## Next Research Directions
The natural next experiments are:
- Add turnover control: no-trade bands, slower rebalancing, or weight smoothing.
- Sweep the lookback window: compare 3-day, 5-day, 10-day, and 20-day reversal.
- Sweep liquidity filters: top 500, top 1000, top 1500 by traded amount.
- Add position caps so no single name can dominate after normalization.
- Compare rank weighting with volatility-scaled reversal.
The most important habit is to keep the layers separate. A good alpha research
workflow does not stop at a single performance number; it explains how the idea
travels from hypothesis, to signal, to weights, to portfolio, to executable PnL.
## Appendix: Phase Timings From This Rerun
| phase | rank full (s) | rank liquid (s) |
| --- | --- | --- |
| alpha compute | 94.1 | 107.8 |
| alpha eval | 93.3 | 96.9 |
| combo combine | 21.6 | 21.7 |
| portfolio build | 537.6 | 236.7 |
| portfolio eval | 95.1 | 88.3 |
| portfolio simulate | 139.6 | 139.1 |
| total | 981.3 | 690.5 |
![Phase timings](assets/reversal_5d_phase_timings.png)
`portfolio build` usually dominates because it iterates per signal date and
repairs a multi-thousand-name integer book under lot rules. The liquid run is
faster because it carries fewer non-zero names per date.
+19 -5
View File
@@ -5,7 +5,7 @@ position weights. Subclasses implement :meth:`signal` — the raw, unnormalized
score. The base class turns a signal into cross-sectionally z-scored weights
via :meth:`to_weights` (override it for a different normalization).
"""
from abc import ABC, abstractmethod
from abc import ABC
import numpy as np
import pandas as pd
@@ -15,15 +15,14 @@ class BaseAlpha(ABC):
"""A position-weight alpha over a cross-section of stocks.
Concrete subclasses must set a unique class-level :attr:`name` (the registry
key) and implement :meth:`signal`. Construct subclasses with their own typed
parameters (e.g. ``lookback``); the factory passes only the parameters a
given ``__init__`` accepts.
key) and implement either :meth:`signal` or :meth:`signal_from_data`.
Construct subclasses with their own typed parameters (e.g. ``lookback``);
the factory passes only the parameters a given ``__init__`` accepts.
"""
#: Unique registry key. Every concrete alpha must set this to a non-empty str.
name: str = ""
@abstractmethod
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
"""Compute the raw signal.
@@ -34,6 +33,21 @@ class BaseAlpha(ABC):
A wide DataFrame aligned to ``close`` where higher values indicate a
stronger long. Use NaN where the signal is undefined.
"""
raise NotImplementedError(
f"{type(self).__name__} must implement signal() or signal_from_data()"
)
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
"""Compute the raw signal from long daily data plus wide closes.
Feature-aware alphas can override this to pivot joined feature columns
from ``data``. The default preserves the existing close-only alpha API.
"""
return self.signal(close)
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
"""Cross-sectionally z-score a signal into signed position weights.
+22 -4
View File
@@ -56,6 +56,10 @@ def list_(alpha_modules):
@click.option("--output-dir", default="alphas", help="Directory to save alpha parquet")
@click.option("--lookback", default=5, type=int, help="Lookback days")
@click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)")
@click.option(
"--feature-path", "feature_paths", multiple=True,
help="Daily derived/feature parquet file or dataset to left-join on symbol_id,date (repeatable)",
)
@click.option(
"--alpha-module", "alpha_modules", multiple=True,
help="External module(s) to import so their alphas register (dotted path or .py file)",
@@ -64,8 +68,17 @@ def list_(alpha_modules):
"--param", "extra_params", multiple=True,
help="Extra alpha constructor param as name=value (repeatable)",
)
@click.option(
"--liquid-universe", is_flag=True, default=False,
help="Mask weights to a per-date investable universe (tradable, non-ST, "
"seasoned, top liquidity) before normalization",
)
@click.option(
"--universe-top-n", default=1000, type=int,
help="Most-liquid names kept per date when --liquid-universe is set",
)
def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
alpha_modules, extra_params):
feature_paths, alpha_modules, extra_params, liquid_universe, universe_top_n):
"""Compute one alpha from raw data and save as parquet."""
for spec in alpha_modules:
load_alpha_module(spec)
@@ -81,6 +94,8 @@ def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
params = {"lookback": lookback, "vol_window": vol_window}
params.update(_parse_params(extra_params))
universe = {"top_n": universe_top_n} if liquid_universe else None
data = pd.read_parquet(data_path)
click.echo(f"Loaded data: {len(data):,} rows from {data_path}")
@@ -88,6 +103,8 @@ def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
data=data,
alpha_name=alpha_name,
alpha_type=alpha_type,
universe=universe,
feature_paths=feature_paths,
**params,
)
@@ -143,7 +160,8 @@ def reversal_vol(data_path, output_dir, lookback, vol_window):
@alpha.command("eval")
@click.option("--alpha-path", required=True, help="Path to alpha parquet file")
@click.option("--data-path", required=True, help="Path to data parquet (for price data)")
def eval_(alpha_path, data_path):
@click.option("--report-dir", default="reports", help="Directory to save JSON report")
def eval_(alpha_path, data_path, report_dir):
"""Evaluate an alpha's performance (return, Sharpe, turnover).
Alphas are interpreted as position WEIGHTS, not return predictors.
@@ -166,9 +184,9 @@ def eval_(alpha_path, data_path):
click.echo("=" * 50)
# Also dump JSON
os.makedirs("reports", exist_ok=True)
os.makedirs(report_dir, exist_ok=True)
alpha_name = alpha_df["alpha_name"].iloc[0]
json_path = f"reports/{alpha_name}_eval.json"
json_path = os.path.join(report_dir, f"{alpha_name}_eval.json")
with open(json_path, "w") as f:
json.dump(metrics, f, indent=2)
click.echo(f"\nReport saved: {json_path}")
+173 -17
View File
@@ -7,12 +7,19 @@ through :mod:`pipeline.alpha.registry`.
"""
import logging
from pathlib import Path
from typing import Iterable
import numpy as np
import pandas as pd
from pipeline.alpha.registry import get_alpha
from pipeline.common.schema import ALPHA_COLUMNS
from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
read_derived_frames,
validate_derived_frame,
)
logger = logging.getLogger(__name__)
@@ -25,15 +32,120 @@ def _pivot_close(df: pd.DataFrame) -> pd.DataFrame:
return pivot.sort_index()
def _daily_returns(close: pd.DataFrame) -> pd.DataFrame:
"""Compute daily returns from wide close DataFrame."""
return close.pct_change()
def _pivot_open(df: pd.DataFrame) -> pd.DataFrame:
"""Pivot data to wide format: date index, columns = symbol_id, values = open."""
pivot = df.pivot_table(
index="date", columns="symbol_id", values="open", aggfunc="first"
)
return pivot.sort_index()
def join_feature_frames(
data: pd.DataFrame,
feature_frames: Iterable[pd.DataFrame],
) -> pd.DataFrame:
"""Left-join validated daily derived/feature frames onto long daily data."""
out = data.copy()
out["date"] = pd.to_datetime(out["date"])
existing = set(out.columns)
joined_cols: list[str] = []
for frame in feature_frames:
features = validate_derived_frame(frame)
feature_cols = [col for col in features.columns if col not in DERIVED_KEY_COLUMNS]
overlap = sorted(existing.intersection(feature_cols))
if overlap:
raise ValueError(
f"Feature columns conflict with existing daily data columns: {overlap}"
)
out = out.merge(
features,
on=DERIVED_KEY_COLUMNS,
how="left",
validate="many_to_one",
)
existing.update(feature_cols)
joined_cols.extend(feature_cols)
if joined_cols:
logger.info("Joined feature columns into daily data: %s", joined_cols)
return out
def _forward_open_to_open_returns(open_: pd.DataFrame) -> pd.DataFrame:
"""Return earned by a close-formed signal after next-open execution.
A weight formed after close on date t can first be traded at open[t+1].
With daily retargeting it is then held until open[t+2], so the signal-date
forward return is open[t+2] / open[t+1] - 1.
"""
return open_.shift(-2).divide(open_.shift(-1)) - 1.0
def investable_universe_mask(
data: pd.DataFrame,
template: pd.DataFrame,
*,
top_n: int = 1000,
min_history: int = 60,
require_tradable: bool = True,
exclude_st: bool = True,
) -> pd.DataFrame:
"""Build a per-date investable-universe mask aligned to ``template``.
A ``(date, symbol_id)`` cell is ``True`` when the name is, on that date,
seasoned (at least ``min_history`` prior closes), currently tradable
(``tradestatus == 1``), not flagged ST (``isST == 0``), and inside the
``top_n`` most liquid names by trailing 20-day mean ``amount``. The mask is
applied to the *signal* (computed on full contiguous prices), so it
restricts only what is *held*, never the price history used to form the
signal — that keeps ``pct_change`` correct and look-ahead free.
Args:
data: Long DataFrame with at least ``symbol_id``, ``date``, ``close``,
``amount``, ``isST``, ``tradestatus``.
template: Wide signal (date index × ``symbol_id`` columns) to align to.
top_n: Keep this many most-liquid names per date.
min_history: Minimum number of observed closes before a name is eligible.
require_tradable: Require ``tradestatus == 1`` on the date.
exclude_st: Drop names flagged ``isST == 1``.
Returns:
Boolean wide DataFrame aligned to ``template``.
"""
def _wide(col: str) -> pd.DataFrame:
return (
data.pivot_table(index="date", columns="symbol_id", values=col, aggfunc="first")
.sort_index()
.reindex(index=template.index, columns=template.columns)
)
close = _wide("close")
mask = close.notna()
seasoned = close.notna().cumsum() >= min_history
mask &= seasoned
if exclude_st and "isST" in data.columns:
mask &= _wide("isST").fillna(1) == 0
if require_tradable and "tradestatus" in data.columns:
mask &= _wide("tradestatus").fillna(0) == 1
amount = _wide("amount")
amt_ma = amount.rolling(20, min_periods=10).mean()
liquid_rank = amt_ma.rank(axis=1, ascending=False)
mask &= liquid_rank <= top_n
return mask.fillna(False)
def compute_alpha(
data: pd.DataFrame,
alpha_name: str,
alpha_type: str,
universe: dict | None = None,
feature_paths: Iterable[str | Path] | None = None,
feature_frames: Iterable[pd.DataFrame] | None = None,
**params,
) -> pd.DataFrame:
"""Compute alpha weights from raw data.
@@ -42,6 +154,15 @@ def compute_alpha(
data: DataFrame with DATA_COLUMNS.
alpha_name: Label stored in the ``alpha_name`` output column.
alpha_type: Registry key of the alpha class (e.g. ``reversal``).
universe: Optional investable-universe filter. When given, the alpha's
raw signal is masked to the investable set (see
:func:`investable_universe_mask`) *before* it is turned into
weights, so unheld names get weight 0. Keys are forwarded as keyword
arguments to :func:`investable_universe_mask`.
feature_paths: Optional parquet files/datasets keyed by ``symbol_id``
and ``date``. Their numeric feature columns are left-joined onto
``data`` before alpha logic runs.
feature_frames: Optional in-memory feature frames with the same schema.
**params: Constructor parameters for the alpha (e.g. ``lookback``,
``vol_window``). Only the params the alpha's ``__init__`` accepts are
used; extras are ignored.
@@ -52,9 +173,22 @@ def compute_alpha(
Raises:
KeyError: If ``alpha_type`` is not registered.
"""
feature_inputs: list[pd.DataFrame] = []
if feature_paths:
feature_inputs.extend(read_derived_frames(feature_paths))
if feature_frames:
feature_inputs.extend(feature_frames)
if feature_inputs:
data = join_feature_frames(data, feature_inputs)
alpha = get_alpha(alpha_type, **params)
close = _pivot_close(data)
weights = alpha.weights(close)
signal = alpha.signal_from_data(data, close)
if universe is None:
weights = alpha.to_weights(signal)
else:
mask = investable_universe_mask(data, signal, **universe)
weights = alpha.to_weights(signal.where(mask))
# Melt to long format
weights_melted = weights.reset_index().melt(
@@ -82,8 +216,11 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
Computes return, annualized Sharpe, annualized turnover, max drawdown.
Alpha is interpreted as POSITION WEIGHTS, not predictions.
Return on date t = sum(weight[s,t] * realized_return[s,t]) / sum(abs(weight[s,t]))
Alpha is interpreted as POSITION WEIGHTS, not predictions. A close-formed
weight on date t is assumed tradable at open[t+1] and held until open[t+2].
Return on signal date t = sum(weight[s,t] * open_to_open_return[s,t]) /
sum(abs(weight[s,t])). This matches the execution convention without
crediting the new signal for the overnight gap before it can be traded.
Args:
alpha_df: DataFrame with ALPHA_COLUMNS.
@@ -93,31 +230,50 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
Dict with metrics: cumulative_return, sharpe_annual, turnover_annual,
max_drawdown, hit_rate, n_dates.
"""
close = _pivot_close(data_df)
returns = _daily_returns(close)
open_ = _pivot_open(data_df)
fwd_returns_all = _forward_open_to_open_returns(open_)
# Pivot alpha weights to wide format
weights = alpha_df.pivot_table(
index="date", columns="symbol_id", values="weight", aggfunc="first"
).sort_index()
# Align dates
common_dates = weights.index.intersection(returns.index)
# Align weights to signal dates that exist on the market calendar. Compute
# forward open-to-open returns on the full market calendar first, so sparse
# signal grids still earn the next available open-to-open interval instead
# of the next signal date.
common_dates = weights.index.intersection(open_.index)
weights = weights.loc[common_dates]
returns = returns.loc[common_dates]
fwd_returns = fwd_returns_all.reindex(common_dates)
if len(common_dates) < 2:
if len(common_dates) < 1:
return {
"cumulative_return": 0.0,
"sharpe_annual": 0.0,
"turnover_annual": 0.0,
"max_drawdown": 0.0,
"hit_rate": 0.0,
"n_dates": len(common_dates),
"n_dates": 0,
}
# Daily portfolio return = sum(w * r) / sum(|w|) — normalized by gross exposure
daily_returns = (weights * returns).sum(axis=1) / weights.abs().sum(axis=1)
# Daily portfolio return = sum(w_t * r_open[t+1→t+2]) / sum(|w_t|).
# The final two signal dates have no complete next-open holding interval
# and are dropped below.
gross = weights.abs().sum(axis=1)
daily_returns = (
(weights * fwd_returns).sum(axis=1, min_count=1)
/ gross.replace(0.0, np.nan)
)
daily_returns = daily_returns.dropna()
if len(daily_returns) < 2:
return {
"cumulative_return": 0.0,
"sharpe_annual": 0.0,
"turnover_annual": 0.0,
"max_drawdown": 0.0,
"hit_rate": 0.0,
"n_dates": int(len(daily_returns)),
}
# Cumulative return
cumulative_return = float((1.0 + daily_returns).prod() - 1.0)
@@ -130,7 +286,7 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
# Annualized turnover: avg daily turnover * 252
# Daily turnover = sum(|w_t - w_{t-1}|) / sum(|w_{t-1}|)
weight_change = weights.diff().abs().sum(axis=1)
gross_exposure = weights.abs().sum(axis=1).shift(1)
gross_exposure = gross.shift(1)
daily_turnover = weight_change / gross_exposure
turnover_annual = float(daily_turnover.mean() * 252)
@@ -149,5 +305,5 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
"turnover_annual": turnover_annual,
"max_drawdown": max_drawdown,
"hit_rate": hit_rate,
"n_dates": len(common_dates),
"n_dates": int(len(daily_returns)),
}
+6 -1
View File
@@ -4,4 +4,9 @@ Importing this package imports each alpha module, which registers the alpha via
the ``@register_alpha`` decorator. Add a new built-in by dropping a module here
and importing it below.
"""
from pipeline.alpha.library import momentum, reversal, reversal_vol # noqa: F401
from pipeline.alpha.library import ( # noqa: F401
momentum,
reversal,
reversal_rank,
reversal_vol,
)
+1 -1
View File
@@ -15,4 +15,4 @@ class MomentumAlpha(BaseAlpha):
self.lookback = lookback
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
return close.pct_change(self.lookback)
return close.pct_change(self.lookback, fill_method=None)
+1 -1
View File
@@ -15,4 +15,4 @@ class ReversalAlpha(BaseAlpha):
self.lookback = lookback
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
return -close.pct_change(self.lookback)
return -close.pct_change(self.lookback, fill_method=None)
+33
View File
@@ -0,0 +1,33 @@
"""Outlier-robust short-horizon reversal alpha."""
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class ReversalRankAlpha(BaseAlpha):
"""Reversal weighted by cross-sectional rank instead of z-score.
The signal is the same trailing-return reversal as :class:`ReversalAlpha`,
but :meth:`to_weights` converts it with a cross-sectional rank that is then
demeaned. Rank weighting is bounded and monotone, so it does not dump the
book into a handful of extreme movers the way raw z-scoring does — the
failure mode that makes plain ``reversal`` collapse on the A-share universe,
where newly listed / post-suspension / limit-up names produce huge
``pct_change`` outliers.
"""
name = "reversal_rank"
def __init__(self, lookback: int = 5):
self.lookback = lookback
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
return -close.pct_change(self.lookback, fill_method=None)
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
signal = signal.dropna(how="all")
ranks = signal.rank(axis=1)
weights = ranks.subtract(ranks.mean(axis=1), axis=0)
return weights.fillna(0.0)
+2 -2
View File
@@ -21,6 +21,6 @@ class ReversalVolAlpha(BaseAlpha):
self.vol_window = vol_window
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
reversal = -close.pct_change(self.lookback)
vol = close.pct_change().rolling(self.vol_window).std()
reversal = -close.pct_change(self.lookback, fill_method=None)
vol = close.pct_change(fill_method=None).rolling(self.vol_window).std()
return reversal / vol
+2 -2
View File
@@ -26,8 +26,8 @@ def combo():
def combine(alpha_paths, combo_name, method, output_dir):
"""Combine multiple alphas and save as parquet."""
paths = [p.strip() for p in alpha_paths.split(",") if p.strip()]
if len(paths) < 2:
click.echo("Error: --alpha-paths requires at least 2 comma-separated paths", err=True)
if len(paths) < 1:
click.echo("Error: --alpha-paths requires at least 1 path", err=True)
return
result = combine_alphas(
+25
View File
@@ -26,6 +26,31 @@ DATA_COLUMNS: Final[list[str]] = [
"pcfNcfTTM", # float64: P/CF (net cash flow, TTM)
]
# Required columns for raw intraday minute bar parquet files.
MINUTE_BAR_COLUMNS: Final[list[str]] = [
"symbol_id", # str: internal code like 'sh600000'
"symbol_name", # str: stock name like '浦发银行'
"datetime", # datetime64: intraday bar timestamp
"date", # date component, aligned with daily DATA_COLUMNS date
"time", # str: HH:MM:SS bar time
"frequency", # str: e.g. '5m'
"open", # float64
"high", # float64
"low", # float64
"close", # float64
"volume", # float64 (shares)
"amount", # float64 (turnover in yuan, raw/unadjusted)
"vwap", # float64: amount / volume
"adjustflag", # str: baostock adjustment flag; '3' for raw/unadjusted
]
# Required key columns for daily derived-data parquet files. Value columns are
# user/plugin-defined and must be numeric.
DERIVED_KEY_COLUMNS: Final[list[str]] = [
"symbol_id", # str
"date", # date: normalized daily timestamp
]
# Required columns for alpha parquet files.
# Alphas are position WEIGHTS: positive=long, negative=short.
ALPHA_COLUMNS: Final[list[str]] = [
+35 -1
View File
@@ -3,7 +3,7 @@
import click
from datetime import date
from pipeline.data.downloader import download_universe
from pipeline.data.downloader import download_minute_universe, download_universe
@click.group(name="data")
@@ -42,3 +42,37 @@ def download(universe, start_date, end_date, output_dir, symbols, chunk_size, ad
f"{stats['n_rows']:,} bars, {stats['date_min']}{stats['date_max']}"
)
click.echo(f"Dataset: {stats['dataset_path']}")
@data.command("download-minute")
@click.option(
"--universe", default="csi500",
help="Which universe: hs300, csi500, all (~5000 A-shares), file path, or comma-separated symbols",
)
@click.option("--start-date", default="2017-01-01", help="Start date YYYY-MM-DD")
@click.option("--end-date", default=str(date.today()), help="End date YYYY-MM-DD")
@click.option("--output-dir", default="data/minute_bars", help="Root for the partitioned dataset")
@click.option("--symbols", default=0, type=int, help="Max symbols (0=all)")
@click.option("--chunk-size", default=100, type=int, help="Symbols per durability flush")
@click.option("--frequency", default="5", help="Minute frequency: 5, 15, 30, or 60")
def download_minute(universe, start_date, end_date, output_dir, symbols, chunk_size, frequency):
"""Download raw Baostock minute bars into a partitioned parquet dataset.
Writes ``{output_dir}/{universe}/frequency=5m/month=YYYY-MM/*.pq`` for the
default 5-minute frequency.
"""
stats = download_minute_universe(
universe=universe,
start_date=start_date,
end_date=end_date,
output_dir=output_dir,
max_symbols=symbols,
chunk_size=chunk_size,
frequency=frequency,
)
click.echo(
f"\nSummary: {stats['n_symbols']}/{stats['n_requested']} symbols, "
f"{stats['n_rows']:,} bars, {stats['date_min']}{stats['date_max']}, "
f"frequency={stats['frequency']}"
)
click.echo(f"Dataset: {stats['dataset_path']}")
+126 -2
View File
@@ -11,9 +11,13 @@ import pyarrow.dataset as pads
# Reuse existing downloader and universe modules
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from data.downloader import download_daily_batch
from data.downloader import (
_normalize_minute_frequency,
download_daily_batch,
download_minute_batch,
)
from data.universe import get_all_stocks, get_hs300_stocks, get_zz500_stocks
from pipeline.common.schema import DATA_COLUMNS
from pipeline.common.schema import DATA_COLUMNS, MINUTE_BAR_COLUMNS
logger = logging.getLogger(__name__)
@@ -89,6 +93,25 @@ def _write_month_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: s
)
def _write_minute_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: str) -> None:
"""Append rows to a Hive-partitioned minute dataset.
Layout: ``frequency=5m/month=YYYY-MM/*.pq``.
"""
out = df.copy()
out["month"] = pd.to_datetime(out["date"]).dt.strftime("%Y-%m")
table = pa.Table.from_pandas(out, preserve_index=False)
pads.write_dataset(
table,
str(base_dir),
format="parquet",
partitioning=["frequency", "month"],
partitioning_flavor="hive",
basename_template=f"{basename_prefix}-{{i}}.pq",
existing_data_behavior="overwrite_or_ignore",
)
def download_universe(
universe: str = "csi500",
start_date: str = "2017-01-01",
@@ -177,3 +200,104 @@ def download_universe(
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
}
def download_minute_universe(
universe: str = "csi500",
start_date: str = "2017-01-01",
end_date: str = "2026-12-31",
output_dir: str = "data/minute_bars",
max_symbols: int = 0,
chunk_size: int = 100,
frequency: str | int = 5,
) -> dict:
"""Download raw minute bars into a frequency/month-partitioned dataset.
Args:
universe: ``hs300``, ``csi500``, ``all``/``full``, a file path, or a
comma-separated symbol list.
start_date, end_date: ``YYYY-MM-DD`` bounds.
output_dir: Root under which ``{universe}/frequency=5m/month=YYYY-MM``
is written.
max_symbols: Cap on symbols (0 = all).
chunk_size: Symbols per durability flush.
frequency: Minute interval. ``5``/``"5"``/``"5m"`` are 5-minute bars.
Returns:
Stats dict with dataset path, row count, symbol count, date range, and
frequency label.
"""
_, frequency_label = _normalize_minute_frequency(frequency)
constituents = _resolve_universe(universe, max_symbols)
symbols = constituents["symbol_id"].tolist()
names = dict(zip(constituents["symbol_id"], constituents["symbol_name"]))
n_requested = len(symbols)
logger.info(
"Minute universe %s: %d symbols, %s%s, frequency=%s",
universe,
n_requested,
start_date,
end_date,
frequency,
)
base_dir = Path(output_dir) / universe
target_frequency_dir = base_dir / f"frequency={frequency_label}"
if target_frequency_dir.exists():
shutil.rmtree(target_frequency_dir)
base_dir.mkdir(parents=True, exist_ok=True)
buffer: list[pd.DataFrame] = []
chunk_idx = 0
succeeded = 0
n_rows = 0
date_min = None
date_max = None
def flush() -> None:
nonlocal buffer, chunk_idx, n_rows, date_min, date_max
if not buffer:
return
chunk = pd.concat(buffer, ignore_index=True)
_write_minute_partitions(chunk, base_dir, basename_prefix=f"chunk{chunk_idx:04d}")
n_rows += len(chunk)
cmin, cmax = chunk["date"].min(), chunk["date"].max()
date_min = cmin if date_min is None else min(date_min, cmin)
date_max = cmax if date_max is None else max(date_max, cmax)
logger.info(
"Flushed minute chunk %d: %d rows (%d symbols done)",
chunk_idx,
len(chunk),
succeeded,
)
buffer = []
chunk_idx += 1
for i, (symbol, df) in enumerate(
download_minute_batch(symbols, start_date, end_date, frequency=frequency), start=1
):
if df is None:
logger.warning(" %s: no minute data", symbol)
else:
df["symbol_id"] = symbol
df["symbol_name"] = names.get(symbol, symbol)
buffer.append(df[MINUTE_BAR_COLUMNS])
succeeded += 1
if len(buffer) >= chunk_size:
flush()
if i % 100 == 0:
logger.info("Minute progress: %d/%d symbols", i, n_requested)
flush()
if succeeded == 0:
raise RuntimeError("No minute data downloaded for any symbol")
return {
"dataset_path": str(base_dir),
"frequency": frequency_label,
"n_symbols": succeeded,
"n_requested": n_requested,
"n_rows": n_rows,
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
}
+2
View File
@@ -0,0 +1,2 @@
"""Daily derived-data plugin package."""
+38
View File
@@ -0,0 +1,38 @@
"""Base class for daily derived-data plugins."""
from abc import ABC, abstractmethod
import pandas as pd
class BaseDerivedData(ABC):
"""Compute daily, symbol-keyed numeric derived data.
Derived-data plugins may use daily bars, minute bars, or both as inputs, but
they must always return daily rows keyed by ``symbol_id,date``.
"""
#: Unique registry key. Every concrete derived-data plugin must set this.
name: str = ""
@abstractmethod
def compute(
self,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Compute daily derived data.
Args:
daily: Optional daily market data.
minute: Optional raw minute bars.
Returns:
DataFrame with ``symbol_id``, ``date``, and one or more numeric
derived-data columns.
"""
def __repr__(self) -> str:
params = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
return f"{type(self).__name__}({params})"
+145
View File
@@ -0,0 +1,145 @@
"""CLI for daily derived-data ingestion and computation."""
import click
import pandas as pd
from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
compute_derived,
read_derived_frame,
write_derived_frame,
)
from pipeline.derived.registry import (
available_derived,
load_derived_module,
)
@click.group(name="derived")
def derived():
"""Ingest, compute, and validate daily derived data."""
def _coerce(value: str):
"""Best-effort coercion of a CLI string to int, then float, else str."""
for cast in (int, float):
try:
return cast(value)
except ValueError:
continue
return value
def _parse_params(pairs: tuple[str, ...]) -> dict:
"""Parse repeated ``name=value`` options into a params dict."""
params: dict = {}
for pair in pairs:
if "=" not in pair:
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
key, value = pair.split("=", 1)
params[key.strip()] = _coerce(value.strip())
return params
def _read_optional_parquet(path: str | None) -> pd.DataFrame | None:
return None if path is None else pd.read_parquet(path)
def _summarize(result: pd.DataFrame) -> str:
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
return f"{len(result):,} rows, {len(value_cols)} columns"
@derived.command("list")
@click.option(
"--derived-module", "derived_modules", multiple=True,
help="External module(s) to import first (dotted path or .py file)",
)
def list_(derived_modules):
"""List the registered derived-data plugin types."""
for spec in derived_modules:
load_derived_module(spec)
for name in available_derived():
click.echo(name)
@derived.command("validate")
@click.option("--input-path", required=True, help="CSV/parquet file or parquet dataset to validate")
def validate(input_path):
"""Validate a daily derived-data file without writing output."""
try:
result = read_derived_frame(input_path)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Valid derived data: {input_path} ({_summarize(result)})")
@derived.command("ingest")
@click.option("--input-path", required=True, help="CSV/parquet file to ingest")
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
def ingest(input_path, derived_name, output_dir):
"""Ingest a user-provided daily derived-data CSV/parquet file."""
try:
result = read_derived_frame(input_path)
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
@derived.command("compute")
@click.option("--daily-path", default=None, help="Optional daily data parquet/dataset")
@click.option("--minute-path", default=None, help="Optional minute parquet/dataset")
@click.option("--derived-type", required=True, help="Registry key of the derived-data plugin")
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
@click.option(
"--derived-module", "derived_modules", multiple=True,
help="External module(s) to import so their derived-data plugins register",
)
@click.option(
"--param", "extra_params", multiple=True,
help="Extra derived-data constructor param as name=value (repeatable)",
)
def compute(
daily_path,
minute_path,
derived_type,
derived_name,
output_dir,
derived_modules,
extra_params,
):
"""Compute one daily derived-data file from daily and/or minute inputs."""
for spec in derived_modules:
load_derived_module(spec)
options = available_derived()
if derived_type not in options:
raise click.BadParameter(
f"Unknown derived-type '{derived_type}'. Available: {options}. "
f"Use --derived-module to register an external derived-data plugin.",
param_hint="--derived-type",
)
if daily_path is None and minute_path is None:
raise click.UsageError("At least one of --daily-path or --minute-path is required")
daily = _read_optional_parquet(daily_path)
if daily_path:
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
minute = _read_optional_parquet(minute_path)
if minute_path:
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
try:
result = compute_derived(
derived_type=derived_type,
daily=daily,
minute=minute,
**_parse_params(extra_params),
)
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
+115
View File
@@ -0,0 +1,115 @@
"""Derived-data computation and validation."""
import csv
import logging
from pathlib import Path
from typing import Iterable
import pandas as pd
from pandas.api.types import is_bool_dtype, is_numeric_dtype
from pipeline.common.schema import DERIVED_KEY_COLUMNS
from pipeline.derived.registry import get_derived
logger = logging.getLogger(__name__)
def validate_derived_frame(derived: pd.DataFrame) -> pd.DataFrame:
"""Validate and normalize a daily derived-data frame.
A valid derived frame is keyed by unique ``symbol_id,date`` rows and has at
least one numeric value column beyond those keys. Dates are normalized to
daily timestamps before duplicate-key checks.
"""
duplicated = derived.columns[derived.columns.duplicated()].tolist()
if duplicated:
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
missing = [col for col in DERIVED_KEY_COLUMNS if col not in derived.columns]
if missing:
raise ValueError(f"Derived data missing required columns: {missing}")
out = derived.copy()
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
if out.duplicated(DERIVED_KEY_COLUMNS).any():
raise ValueError("Derived data has duplicate symbol_id,date rows")
value_cols = [col for col in out.columns if col not in DERIVED_KEY_COLUMNS]
if not value_cols:
raise ValueError("Derived data must include at least one value column")
non_numeric = [
col
for col in value_cols
if is_bool_dtype(out[col]) or not is_numeric_dtype(out[col])
]
if non_numeric:
raise ValueError(f"Derived data value columns must be numeric: {non_numeric}")
out = out[DERIVED_KEY_COLUMNS + value_cols].copy()
return out.sort_values(DERIVED_KEY_COLUMNS).reset_index(drop=True)
def compute_derived(
derived_type: str,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
**params,
) -> pd.DataFrame:
"""Compute one registered derived-data plugin."""
if daily is None and minute is None:
raise ValueError("Derived data computation requires --daily-path or --minute-path")
derived = get_derived(derived_type, **params)
result = validate_derived_frame(derived.compute(daily=daily, minute=minute))
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
logger.info(
"Derived data '%s' (%r): %d symbols × %d dates, columns=%s",
derived_type,
derived,
result["symbol_id"].nunique(),
result["date"].nunique(),
value_cols,
)
return result
def read_derived_frame(path: str | Path) -> pd.DataFrame:
"""Read and validate one derived CSV/parquet file or parquet dataset."""
path = Path(path)
if path.suffix.lower() == ".csv":
return validate_derived_frame(_read_csv_with_duplicate_header_check(path))
return validate_derived_frame(pd.read_parquet(path))
def read_derived_frames(derived_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
"""Read and validate derived-data files."""
return [read_derived_frame(path) for path in derived_paths]
def write_derived_frame(
derived: pd.DataFrame,
derived_name: str,
output_dir: str | Path = "derived",
) -> Path:
"""Validate and write derived data to ``{output_dir}/{derived_name}.pq``."""
result = validate_derived_frame(derived)
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
out_path = output_dir / f"{derived_name}.pq"
result.to_parquet(out_path, index=False)
return out_path
def _read_csv_with_duplicate_header_check(path: Path) -> pd.DataFrame:
with path.open(newline="") as fh:
reader = csv.reader(fh)
try:
header = next(reader)
except StopIteration as exc:
raise ValueError("CSV input is empty") from exc
duplicated = sorted({col for col in header if header.count(col) > 1})
if duplicated:
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
return pd.read_csv(path)
+4
View File
@@ -0,0 +1,4 @@
"""Built-in derived-data library."""
from pipeline.derived.library import minute_daily_summary # noqa: F401
@@ -0,0 +1,88 @@
"""Daily summary data derived from raw minute bars."""
import numpy as np
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class MinuteDailySummaryDerived(BaseDerivedData):
"""Aggregate intraday bars into daily summary columns."""
name = "minute_daily_summary"
def compute(
self,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
) -> pd.DataFrame:
if minute is None:
raise ValueError("minute_daily_summary requires minute input")
minute = minute.copy()
minute["date"] = pd.to_datetime(minute["date"]).dt.normalize()
sort_cols = ["symbol_id", "date"]
if "datetime" in minute.columns:
minute["datetime"] = pd.to_datetime(minute["datetime"])
sort_cols.append("datetime")
elif "time" in minute.columns:
sort_cols.append("time")
minute = minute.sort_values(sort_cols)
grouped = minute.groupby(["symbol_id", "date"], sort=True)
summary = grouped.agg(
minute_bar_count=("close", "count"),
first_open=("open", "first"),
last_close=("close", "last"),
high=("high", "max"),
low=("low", "min"),
volume_sum=("volume", "sum"),
amount_sum=("amount", "sum"),
)
summary["minute_intraday_return"] = (
summary["last_close"] / summary["first_open"] - 1.0
)
summary["minute_intraday_range"] = summary["high"] / summary["low"] - 1.0
summary["minute_vwap"] = (
summary["amount_sum"] / summary["volume_sum"].where(summary["volume_sum"] > 0)
)
summary = summary.reset_index()
if daily is not None:
daily_keys = daily[["symbol_id", "date"]].copy()
daily_keys["date"] = pd.to_datetime(daily_keys["date"]).dt.normalize()
daily_keys = daily_keys.drop_duplicates(["symbol_id", "date"])
result = daily_keys.merge(summary, on=["symbol_id", "date"], how="left")
if "close" in daily.columns:
daily_close = daily[["symbol_id", "date", "close"]].copy()
daily_close["date"] = pd.to_datetime(daily_close["date"]).dt.normalize()
daily_close = daily_close.drop_duplicates(["symbol_id", "date"])
result = result.merge(
daily_close.rename(columns={"close": "daily_close"}),
on=["symbol_id", "date"],
how="left",
)
reference_close = result["daily_close"].fillna(result["last_close"])
else:
reference_close = result["last_close"]
else:
result = summary
reference_close = result["last_close"]
result["minute_vwap_deviation"] = (
result["minute_vwap"] / reference_close.replace(0.0, np.nan) - 1.0
)
return result[
[
"symbol_id",
"date",
"minute_bar_count",
"minute_intraday_return",
"minute_intraday_range",
"minute_vwap",
"minute_vwap_deviation",
]
]
+80
View File
@@ -0,0 +1,80 @@
"""Registry and factory for daily derived-data plugins."""
import importlib
import importlib.util
import inspect
from pathlib import Path
from typing import Optional, Type
from pipeline.derived.base import BaseDerivedData
_REGISTRY: dict[str, Type[BaseDerivedData]] = {}
_builtins_loaded = False
def register_derived(cls: Type[BaseDerivedData]) -> Type[BaseDerivedData]:
"""Class decorator that registers derived data under ``BaseDerivedData.name``."""
if not (isinstance(cls, type) and issubclass(cls, BaseDerivedData)):
raise TypeError(f"{cls!r} is not a BaseDerivedData subclass")
key = getattr(cls, "name", "")
if not key:
raise ValueError(f"{cls.__name__} must set a non-empty class attribute `name`")
existing = _REGISTRY.get(key)
if existing is not None and existing is not cls:
raise ValueError(
f"Derived data name '{key}' already registered by {existing.__name__}"
)
_REGISTRY[key] = cls
return cls
def available_derived() -> list[str]:
"""Sorted names of all registered derived-data plugins."""
_ensure_builtins()
return sorted(_REGISTRY)
def get_derived(name: str, **params) -> BaseDerivedData:
"""Instantiate a registered derived-data plugin by name.
Only parameters accepted by the plugin class's ``__init__`` are forwarded.
"""
_ensure_builtins()
if name not in _REGISTRY:
raise KeyError(f"Unknown derived data '{name}'. Available: {sorted(_REGISTRY)}")
cls = _REGISTRY[name]
accepted = _accepted_params(cls)
kwargs = params if accepted is None else {k: v for k, v in params.items() if k in accepted}
return cls(**kwargs)
def load_derived_module(spec: str) -> None:
"""Import an external module so its ``@register_derived`` classes register."""
looks_like_file = spec.endswith(".py") or Path(spec).expanduser().exists()
if looks_like_file:
path = Path(spec).expanduser().resolve()
if not path.exists():
raise FileNotFoundError(f"Derived data module not found: {path}")
module_spec = importlib.util.spec_from_file_location(path.stem, path)
if module_spec is None or module_spec.loader is None:
raise ImportError(f"Cannot load derived data module from {path}")
module = importlib.util.module_from_spec(module_spec)
module_spec.loader.exec_module(module)
else:
importlib.import_module(spec)
def _accepted_params(cls: Type[BaseDerivedData]) -> Optional[set[str]]:
"""Param names ``cls.__init__`` accepts, or None if it takes ``**kwargs``."""
sig = inspect.signature(cls.__init__)
if any(p.kind is p.VAR_KEYWORD for p in sig.parameters.values()):
return None
return {name for name in sig.parameters if name != "self"}
def _ensure_builtins() -> None:
global _builtins_loaded
if not _builtins_loaded:
import pipeline.derived.library # noqa: F401
_builtins_loaded = True
+1
View File
@@ -0,0 +1 @@
"""Daily feature plugin package."""
+8
View File
@@ -0,0 +1,8 @@
"""Compatibility alias for daily feature plugins.
The canonical plugin API is ``pipeline.derived``. ``BaseFeature`` remains as an
alias so existing external feature modules continue to register unchanged.
"""
from pipeline.derived.base import BaseDerivedData as BaseFeature
+108
View File
@@ -0,0 +1,108 @@
"""CLI for daily feature computation."""
import os
import click
import pandas as pd
from pipeline.features.compute import compute_feature
from pipeline.features.registry import available_features, load_feature_module
@click.group(name="feature")
def feature():
"""Compute daily feature parquet files from minute bars."""
def _coerce(value: str):
"""Best-effort coercion of a CLI string to int, then float, else str."""
for cast in (int, float):
try:
return cast(value)
except ValueError:
continue
return value
def _parse_params(pairs: tuple[str, ...]) -> dict:
"""Parse repeated ``name=value`` options into a params dict."""
params: dict = {}
for pair in pairs:
if "=" not in pair:
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
key, value = pair.split("=", 1)
params[key.strip()] = _coerce(value.strip())
return params
@feature.command("list")
@click.option(
"--feature-module", "feature_modules", multiple=True,
help="External module(s) to import first (dotted path or .py file)",
)
def list_(feature_modules):
"""List the registered feature types."""
for spec in feature_modules:
load_feature_module(spec)
for name in available_features():
click.echo(name)
@feature.command("compute")
@click.option("--minute-path", required=True, help="Path to minute parquet dataset/file")
@click.option("--daily-path", default=None, help="Optional daily data parquet for alignment")
@click.option("--feature-type", required=True, help="Registry key of the feature class")
@click.option("--feature-name", required=True, help="Name for this feature run/output file")
@click.option("--output-dir", default="features", help="Directory to save feature parquet")
@click.option(
"--feature-module", "feature_modules", multiple=True,
help="External module(s) to import so their features register (dotted path or .py file)",
)
@click.option(
"--param", "extra_params", multiple=True,
help="Extra feature constructor param as name=value (repeatable)",
)
def compute(
minute_path,
daily_path,
feature_type,
feature_name,
output_dir,
feature_modules,
extra_params,
):
"""Compute one daily feature file from raw minute bars."""
for spec in feature_modules:
load_feature_module(spec)
options = available_features()
if feature_type not in options:
raise click.BadParameter(
f"Unknown feature-type '{feature_type}'. Available: {options}. "
f"Use --feature-module to register an external feature.",
param_hint="--feature-type",
)
minute = pd.read_parquet(minute_path)
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
daily = None
if daily_path:
daily = pd.read_parquet(daily_path)
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
result = compute_feature(
minute=minute,
daily=daily,
feature_type=feature_type,
**_parse_params(extra_params),
)
os.makedirs(output_dir, exist_ok=True)
out_path = f"{output_dir}/{feature_name}.pq"
result.to_parquet(out_path, index=False)
feature_cols = [col for col in result.columns if col not in ("symbol_id", "date")]
click.echo(
f"Saved feature: {out_path} ({len(result):,} rows, "
f"{len(feature_cols)} columns)"
)
+41
View File
@@ -0,0 +1,41 @@
"""Compatibility wrappers for daily feature computation and validation."""
from pathlib import Path
from typing import Iterable
import pandas as pd
from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
compute_derived,
read_derived_frames,
validate_derived_frame,
)
FEATURE_KEY_COLUMNS = DERIVED_KEY_COLUMNS
def validate_feature_frame(features: pd.DataFrame) -> pd.DataFrame:
"""Validate and normalize a legacy daily feature frame."""
return validate_derived_frame(features)
def compute_feature(
minute: pd.DataFrame,
feature_type: str,
daily: pd.DataFrame | None = None,
**params,
) -> pd.DataFrame:
"""Compute one registered feature through the derived-data registry."""
return compute_derived(
derived_type=feature_type,
daily=daily,
minute=minute,
**params,
)
def read_feature_frames(feature_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
"""Read and validate feature/derived-data parquet files."""
return read_derived_frames(feature_paths)
+3
View File
@@ -0,0 +1,3 @@
"""Built-in feature library."""
from pipeline.features.library import minute_daily_summary # noqa: F401
@@ -0,0 +1,16 @@
"""Compatibility wrapper for the built-in minute daily summary plugin."""
import pandas as pd
from pipeline.derived.library.minute_daily_summary import MinuteDailySummaryDerived
class MinuteDailySummaryFeature(MinuteDailySummaryDerived):
"""Legacy minute-first wrapper around the derived-data implementation."""
def compute(
self,
minute: pd.DataFrame,
daily: pd.DataFrame | None = None,
) -> pd.DataFrame:
return super().compute(daily=daily, minute=minute)
+36
View File
@@ -0,0 +1,36 @@
"""Compatibility registry wrappers for daily feature plugins."""
from typing import Type
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import (
available_derived,
get_derived,
load_derived_module,
register_derived,
)
from pipeline.features.base import BaseFeature
def register_feature(cls: Type[BaseFeature]) -> Type[BaseFeature]:
"""Register a legacy feature plugin in the derived-data registry."""
return register_derived(cls)
def available_features() -> list[str]:
"""Sorted names of all registered feature/derived-data plugins."""
return available_derived()
def get_feature(name: str, **params) -> BaseDerivedData:
"""Instantiate a registered feature/derived-data plugin by name."""
if name == "minute_daily_summary":
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
return MinuteDailySummaryFeature(**params)
return get_derived(name, **params)
def load_feature_module(spec: str) -> None:
"""Import an external module so its ``@register_feature`` classes register."""
load_derived_module(spec)
+3 -3
View File
@@ -7,9 +7,9 @@ across dates (positions are stateful, unlike alphas/combos), discretizing and
repairing each day's target into a tradable integer book.
Return-convention note: weights here are *target allocations*. The research
evaluation in :mod:`pipeline.portfolio.research` marks them close-to-close on the
*next* period (no look-ahead); the execution simulator marks the actually-filled
book at the next open. See those modules for details.
evaluation in :mod:`pipeline.portfolio.research` marks them from next open to
the following open (no look-ahead); the execution simulator marks the
actually-filled book at the next open. See those modules for details.
"""
from __future__ import annotations
+50
View File
@@ -0,0 +1,50 @@
"""Trading cost models for portfolio execution simulation."""
from __future__ import annotations
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Mapping
import numpy as np
class CostModel(ABC):
"""Interface for per-name execution cost models."""
@abstractmethod
def compute(
self,
traded_shares: np.ndarray,
execution_price: np.ndarray,
side: np.ndarray,
date,
metadata: Mapping[str, object] | None = None,
) -> np.ndarray:
"""Return per-name trading cost in yuan."""
@dataclass(frozen=True)
class SimpleProportionalCostModel(CostModel):
"""Simplified open-execution proportional cost model.
Slippage is represented as an additional cash cost. The execution price is
not adjusted by slippage, which avoids double-counting.
"""
cost_bps: float = 0.0
slippage_bps: float = 0.0
def compute(
self,
traded_shares: np.ndarray,
execution_price: np.ndarray,
side: np.ndarray,
date,
metadata: Mapping[str, object] | None = None,
) -> np.ndarray:
shares = np.asarray(traded_shares, dtype=np.float64)
price = np.asarray(execution_price, dtype=np.float64)
open_price = np.where(np.isfinite(price), price, 0.0)
trade_value = np.abs(shares * open_price)
return trade_value * (self.cost_bps + self.slippage_bps) / 1e4
+21 -13
View File
@@ -6,9 +6,10 @@ trading constraints. Metrics are return / Sharpe / turnover / max-drawdown /
convention that an alpha is a position weight, not a return predictor.
Return convention (documented): the target weight formed from information at
date ``t`` earns the *next* period's close-to-close return, i.e. weights are
shifted one day relative to realized returns, so there is no look-ahead:
``R_t = sum_i w_{i,t} · r_{i,t+1}`` normalized by gross exposure.
date ``t`` is assumed tradable at ``open[t+1]`` and held until ``open[t+2]``.
This is a costless approximation of the next-open execution path: no lots,
constraints, or costs, but no credit for an overnight gap that the new signal
could not have owned.
"""
from __future__ import annotations
@@ -26,39 +27,46 @@ def evaluate_portfolio(positions_df: pd.DataFrame, data_df: pd.DataFrame) -> dic
Args:
positions_df: POSITION_COLUMNS (uses ``target_weight``; zero-gross
construction carry dates remain flat in this research view).
data_df: DATA_COLUMNS (uses ``close`` for returns).
data_df: DATA_COLUMNS (uses ``open`` for returns).
Returns:
Dict with ``cumulative_return, sharpe_annual, turnover_annual,
max_drawdown, fitness, hit_rate, n_dates``. No IC key.
"""
close = data_df.pivot_table(
index="date", columns="symbol_id", values="close", aggfunc="first"
open_ = data_df.pivot_table(
index="date", columns="symbol_id", values="open", aggfunc="first"
).sort_index()
returns = close.pct_change()
fwd = open_.shift(-2).divide(open_.shift(-1)) - 1.0
weights = positions_df.pivot_table(
index="date", columns="symbol_id", values="target_weight", aggfunc="first"
).sort_index()
common = weights.index.intersection(returns.index)
common = weights.index.intersection(open_.index)
weights = weights.loc[common]
returns = returns.loc[common]
# Compute forward returns on the full market calendar before selecting
# signal dates. This preserves the next available open-to-open holding
# interval when the signal grid is sparser than the data grid.
fwd = fwd.reindex(common)
empty = {
"cumulative_return": 0.0, "sharpe_annual": 0.0, "turnover_annual": 0.0,
"max_drawdown": 0.0, "fitness": 0.0, "hit_rate": 0.0,
"n_dates": len(common),
}
if len(common) < 3:
if len(common) < 1:
empty["n_dates"] = 0
return empty
gross = weights.abs().sum(axis=1)
# Weights at t earn the return from t to t+1: shift returns back by one.
fwd = returns.shift(-1)
daily = (weights * fwd).sum(axis=1) / gross.replace(0.0, np.nan)
# Weights at t earn the costless tradable interval open[t+1] -> open[t+2].
daily = (
(weights * fwd).sum(axis=1, min_count=1)
/ gross.replace(0.0, np.nan)
)
daily = daily.dropna()
if len(daily) < 2:
empty["n_dates"] = int(len(daily))
return empty
cumulative_return = float((1.0 + daily).prod() - 1.0)
+25 -12
View File
@@ -4,7 +4,8 @@ Execution model (documented convention): a position book targeted from
information available on date ``t`` is executed at ``open[t+1]``. Trades that
violate a :class:`~pipeline.portfolio.constraints.TradeConstraint` (suspension,
price limit, volume cap, ) are clipped; a fully blocked buy leaves the position
at its previous level. Realized PnL marks the *actually filled* book.
at its previous level. Realized PnL marks the *actually filled* book. Trading
cost defaults to a simplified open-execution proportional cash-cost model.
The simulator is an ABC + a :class:`ReferenceSimulator`; constraints compose by
intersecting their per-name signed delta bounds.
@@ -21,6 +22,7 @@ import pandas as pd
from pipeline.common.schema import FILL_COLUMNS, PNL_COLUMNS
from pipeline.portfolio.constraints import TradeConstraint
from pipeline.portfolio.costs import CostModel, SimpleProportionalCostModel
from pipeline.portfolio.market_rules import MarketRule, compute_limit_status
logger = logging.getLogger(__name__)
@@ -65,10 +67,12 @@ class ExecutionSimulator(ABC):
"""Abstract execution layer. Subclasses define how a target gets filled."""
def __init__(self, constraints: list[TradeConstraint] | None = None,
cost_bps: float = 0.0, slippage_bps: float = 0.0):
cost_bps: float = 0.0, slippage_bps: float = 0.0,
cost_model: CostModel | None = None):
self.constraints = constraints or []
self.cost_bps = cost_bps
self.slippage_bps = slippage_bps
self.cost_model = cost_model or SimpleProportionalCostModel(
cost_bps=cost_bps, slippage_bps=slippage_bps
)
@abstractmethod
def fill(self, ctx: TradeContext) -> FillResult:
@@ -104,9 +108,17 @@ class ReferenceSimulator(ExecutionSimulator):
blocked = (traded != desired).astype(np.int64)
realized = prev + traded
open_px = np.where(np.isfinite(ctx.slice.price), ctx.slice.price, 0.0)
trade_value = np.abs(traded.astype(np.float64) * open_px)
cost = trade_value * (self.cost_bps + self.slippage_bps) / 1e4
cost = self.cost_model.compute(
traded_shares=traded,
execution_price=ctx.slice.price,
side=np.sign(traded),
date=ctx.slice.date,
metadata={
"symbol_ids": ctx.slice.symbol_ids,
"booksize": ctx.booksize,
"market_slice": ctx.slice,
},
)
return FillResult(realized, traded, cost, blocked)
def run(
@@ -154,13 +166,14 @@ class ReferenceSimulator(ExecutionSimulator):
st = wide(data_df, "isST") if "isST" in data_df.columns else opn * 0.0
symbols = sorted(set(tgt.columns) | set(opn.columns))
data_index = close.index
tgt = tgt.reindex(columns=symbols)
opn = opn.reindex(columns=symbols)
opn = opn.reindex(index=data_index, columns=symbols)
close = close.reindex(columns=symbols)
preclose = preclose.reindex(columns=symbols)
amount = amount.reindex(columns=symbols)
tstat = tstat.reindex(columns=symbols)
st = st.reindex(columns=symbols)
preclose = preclose.reindex(index=data_index, columns=symbols)
amount = amount.reindex(index=data_index, columns=symbols)
tstat = tstat.reindex(index=data_index, columns=symbols)
st = st.reindex(index=data_index, columns=symbols)
sym_arr = np.asarray(symbols, dtype=object)
n = len(symbols)
+2
View File
@@ -0,0 +1,2 @@
"""Optional plugin packages for the research pipeline."""
+103
View File
@@ -0,0 +1,103 @@
# JoinQuant Comparison Plugin
This plugin exports frozen targets from the internal A-share research pipeline,
drives a standalone JoinQuant wrapper strategy, ingests JoinQuant output files,
and reconciles them against the internal reference simulator.
The plugin validates system mechanics, not alpha quality:
- date alignment
- symbol mapping
- target position generation
- open execution timing
- lot rounding and filled shares
- position carry
- trading cost and PnL accounting
- blocked trades from suspension and price limits
## Commands
```bash
uv run python cli.py joinquant prepare-smoke \
--out-dir /tmp/chinese-equity-quant-realdata
uv sync --extra joinquant-browser
uv run playwright install chromium
uv run python cli.py joinquant browser-login \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--strategy-url "https://www.joinquant.com/..." \
--flow backtest
uv run python cli.py joinquant run-browser-backtest \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--strategy-url "https://www.joinquant.com/<模拟盘 page>" \
--flow sim-trade
uv run python cli.py joinquant run-browser-sim \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
uv run python cli.py joinquant export-targets \
--positions-path portfolio/run1.pq \
--portfolio-name run1 \
--mode target_shares \
--execution-calendar-path data/daily_bars/csi500 \
--start-date 2026-07-01 \
--end-date 2026-07-31 \
--out-dir plugins_output/joinquant/targets
uv run python cli.py joinquant write-wrapper \
--portfolio-name run1 \
--mode target_shares \
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
uv run python cli.py joinquant ingest \
--portfolio-name run1 \
--fills-csv path/to/jq_fills.csv \
--positions-csv path/to/jq_positions.csv \
--pnl-csv path/to/jq_pnl.csv \
--out-dir plugins_output/joinquant/ingested
uv run python cli.py joinquant reconcile \
--portfolio-name run1 \
--targets-dir plugins_output/joinquant/targets/run1 \
--our-fills-path fills/run1.pq \
--our-positions-path portfolio/run1.pq \
--our-pnl-path pnl/run1.pq \
--jq-fills-path plugins_output/joinquant/ingested/run1/fills.pq \
--jq-positions-path plugins_output/joinquant/ingested/run1/positions.pq \
--jq-pnl-path plugins_output/joinquant/ingested/run1/pnl.pq \
--out-dir plugins_output/joinquant/reconcile
```
`target_shares` is the default and uses the built integer `position_shares`
from `portfolio build`, matching what the internal simulator executes.
For strict simulator-vs-JoinQuant comparison, pass `--execution-calendar-path`
so position dates are shifted to the next session open, matching the internal
simulator's next-open convention.
`prepare-smoke` automates the local side of the first sanity check: tiny real
data download, one-stock long-only position file, internal simulation, aligned
target export, wrapper generation, and a manifest with expected JoinQuant CSV
export paths.
`run-browser-backtest` automates the remote JoinQuant web run through
Playwright. It reuses a saved browser login state, executes the configured UI
actions, downloads JoinQuant CSVs when configured, and runs ingest/reconcile
automatically once all three CSVs are present.
`run-browser-sim` is the forward-test / 模拟盘 equivalent. Use a `--flow
sim-trade` config to upload the frozen next-session target file, save the
strategy, and start or restart the JoinQuant simulated-trading job. After close,
run it again with download actions or use `ingest` / `reconcile` directly on
exported CSVs.
+10
View File
@@ -0,0 +1,10 @@
"""JoinQuant comparison plugin.
This package keeps JoinQuant-specific export, ingest, and reconciliation code
outside the core portfolio modules.
"""
from plugins.joinquant.symbols import from_joinquant_symbol, to_joinquant_symbol
__all__ = ["from_joinquant_symbol", "to_joinquant_symbol"]
+710
View File
@@ -0,0 +1,710 @@
"""Browser automation for JoinQuant cloud backtest and simulated-trading runs.
JoinQuant's public ``jqdatasdk`` is a data API; cloud strategy upload/run/export
is exposed through the web application. This module keeps that automation
optional and configurable so UI selector drift does not affect the core test
suite.
"""
from __future__ import annotations
import json
import re
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from plugins.joinquant.ingest import ingest_joinquant_outputs
from plugins.joinquant.reconcile import reconcile_joinquant
DEFAULT_LOGIN_URL = "https://www.joinquant.com/user/login/index"
def _require_playwright():
try:
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError
from playwright.sync_api import sync_playwright
except ImportError as exc:
raise RuntimeError(
"Playwright is required for JoinQuant browser automation. Install it "
"in this uv environment, then install Chromium:\n"
" uv sync --extra joinquant-browser\n"
" uv run playwright install chromium"
) from exc
return sync_playwright, PlaywrightTimeoutError
def _backtest_actions() -> list[dict[str, Any]]:
return [
{"type": "goto", "url": "{strategy_url}"},
{
"type": "click",
"selector": "text=不再提示",
"timeout_ms": 5000,
"force": True,
"optional": True,
"description": "Dismiss JoinQuant first-run editor guide.",
},
{
"type": "click",
"selector": "text=确定",
"timeout_ms": 5000,
"force": True,
"optional": True,
"description": "Close JoinQuant first-run editor guide.",
},
{
"type": "click",
"selector": "text=跳过",
"timeout_ms": 5000,
"force": True,
"optional": True,
"description": "Skip JoinQuant first-run editor guide.",
},
{
"type": "evaluate",
"script": (
"() => {"
"document.querySelectorAll("
"'.introjs-overlay,.introjs-helperLayer,.introjs-tooltipReferenceLayer,"
".introjs-tooltip,.introjs-disableInteraction'"
").forEach((node) => node.remove());"
"document.body.classList.remove('introjs-noscroll');"
"}"
),
"optional": True,
"description": "Remove any remaining JoinQuant intro overlay.",
},
{
"type": "set_ace_editor_file",
"selector": ".ace_editor",
"path": "{wrapper_path}",
"description": "Set generated wrapper strategy in the Ace code editor.",
},
{
"type": "set_input_files",
"selector": "input[type=file]",
"paths": "{target_csvs}",
"description": "Upload all aligned daily target CSV files.",
"timeout_ms": 5000,
"optional": True,
},
{
"type": "evaluate",
"script": (
"(arg) => {"
"const setValue = (selector, value) => {"
"const el = document.querySelector(selector);"
"if (!el) return false;"
"el.value = value;"
"el.setAttribute('value', value);"
"el.dispatchEvent(new Event('input', {bubbles: true}));"
"el.dispatchEvent(new Event('change', {bubbles: true}));"
"return true;"
"};"
"return {"
"start: setValue('#startTime', arg.start),"
"end: setValue('#endTime', arg.end),"
"capital: setValue('#daily_backtest_capital_base_box', arg.capital)"
"};"
"}"
),
"arg": {
"start": "{backtest_start_date}",
"end": "{backtest_end_date}",
"capital": "{booksize}",
},
"description": "Set JoinQuant backtest dates and starting capital.",
"optional": True,
},
{
"type": "click",
"selector": "#algo-save-button",
"timeout_ms": 5000,
"description": "Save the JoinQuant strategy code.",
"optional": True,
},
{
"type": "click",
"selector": ".bootstrap-dialog .btn-primary, .modal.in button:has-text(\"确定\")",
"timeout_ms": 5000,
"force": True,
"description": "Confirm any JoinQuant save dialog.",
"optional": True,
},
{"type": "wait_for_timeout", "timeout_ms": 2000},
{
"type": "click",
"selector": "#daily-new-backtest-button",
"description": "Start the JoinQuant backtest.",
},
{
"type": "wait_for_selector",
"selector": "text=/回测完成|运行完成|Backtest complete|Finished/i",
"timeout_ms": 600_000,
"optional": True,
},
{
"type": "download",
"selector": "text=/导出成交|下载成交|fills|trades/i",
"save_as": "{expected_joinquant_csvs.fills}",
"timeout_ms": 15000,
"optional": True,
},
{
"type": "download",
"selector": "text=/导出持仓|下载持仓|positions/i",
"save_as": "{expected_joinquant_csvs.positions}",
"timeout_ms": 15000,
"optional": True,
},
{
"type": "download",
"selector": "text=/导出收益|下载收益|pnl|收益/i",
"save_as": "{expected_joinquant_csvs.pnl}",
"timeout_ms": 15000,
"optional": True,
},
{"type": "screenshot", "path": "{run_artifact_dir}/final.png"},
]
def _sim_trade_actions() -> list[dict[str, Any]]:
return [
{"type": "goto", "url": "{strategy_url}"},
{
"type": "paste_text_file",
"selector": "textarea, .ace_text-input, .cm-content, .CodeMirror textarea",
"path": "{wrapper_path}",
"description": "Paste generated wrapper strategy into the simulated-trading strategy editor.",
},
{
"type": "set_input_files",
"selector": "input[type=file]",
"paths": "{target_csvs}",
"description": "Upload frozen target CSV files for 模拟盘.",
"optional": True,
},
{
"type": "click",
"selector": "text=/保存|Save/i",
"description": "Save the strategy code and uploaded files.",
"optional": True,
},
{
"type": "wait_for_selector",
"selector": "text=/保存成功|已保存|Saved/i",
"timeout_ms": 120_000,
"optional": True,
},
{
"type": "click",
"selector": "text=/模拟盘|模拟交易|启动模拟|运行模拟|启动|重启|Run Sim|Start|Restart/i",
"description": "Start or restart the JoinQuant simulated-trading job.",
},
{
"type": "wait_for_selector",
"selector": "text=/运行中|已启动|模拟交易运行|Started|Running/i",
"timeout_ms": 180_000,
"optional": True,
},
{"type": "screenshot", "path": "{run_artifact_dir}/sim_trade_final.png"},
]
def default_browser_config(strategy_url: str = "", *, flow: str = "backtest") -> dict[str, Any]:
"""Return a selector/action template for JoinQuant browser automation."""
if flow not in {"backtest", "sim-trade", "sim_trade"}:
raise ValueError("flow must be 'backtest' or 'sim-trade'")
normalized_flow = "sim-trade" if flow == "sim_trade" else flow
actions = _sim_trade_actions() if normalized_flow == "sim-trade" else _backtest_actions()
return {
"flow": normalized_flow,
"strategy_url": strategy_url,
"login_url": DEFAULT_LOGIN_URL,
"headless": False,
"timeout_ms": 120_000,
"notes": [
"Fill strategy_url and selectors after inspecting your JoinQuant strategy page.",
"Use `joinquant browser-snapshot` to capture HTML/screenshots for selector tuning.",
"The action list is declarative and runs in order.",
],
"actions": actions,
}
def write_browser_config_template(
path: str | Path,
*,
strategy_url: str = "",
flow: str = "backtest",
) -> Path:
"""Write a JSON config template for browser automation."""
out_path = Path(path)
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text(
json.dumps(
default_browser_config(strategy_url, flow=flow),
indent=2,
ensure_ascii=False,
) + "\n",
encoding="utf-8",
)
return out_path
def load_json(path: str | Path) -> dict[str, Any]:
"""Load a JSON object from disk."""
data = json.loads(Path(path).read_text(encoding="utf-8"))
if not isinstance(data, dict):
raise ValueError(f"Expected JSON object in {path}")
return data
def load_env_file(path: str | Path) -> dict[str, str]:
"""Load simple KEY=VALUE env files without requiring shell-safe quoting."""
values: dict[str, str] = {}
for raw in Path(path).expanduser().read_text(encoding="utf-8").splitlines():
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
clean = value.strip()
if len(clean) >= 2 and clean[0] == clean[-1] and clean[0] in {"'", '"'}:
clean = clean[1:-1]
else:
clean = clean.strip("'\"")
values[key.strip()] = clean
return values
def _get_dotted(data: dict[str, Any], dotted: str) -> Any:
current: Any = data
for part in dotted.split("."):
if isinstance(current, dict) and part in current:
current = current[part]
else:
raise KeyError(dotted)
return current
def _manifest_context(manifest: dict[str, Any], artifact_dir: Path, config: dict[str, Any]) -> dict[str, Any]:
targets_dir = Path(str(manifest["targets_dir"]))
target_csvs = [str(path) for path in sorted(targets_dir.glob("*.csv"))]
if not target_csvs:
raise ValueError(f"No target CSV files found under {targets_dir}")
target_dates = [
datetime.strptime(Path(path).stem, "%Y%m%d").strftime("%Y-%m-%d")
for path in target_csvs
]
context = dict(manifest)
context.update({
"strategy_url": config.get("strategy_url", ""),
"target_csvs": target_csvs,
"target_csvs_csv": ",".join(target_csvs),
"backtest_start_date": config.get("backtest_start_date") or min(target_dates),
"backtest_end_date": config.get("backtest_end_date") or max(target_dates),
"run_artifact_dir": str(artifact_dir),
})
return context
_TOKEN_RE = re.compile(r"^\{([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)\}$")
_PARTIAL_TOKEN_RE = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)\}")
def resolve_template(value: Any, context: dict[str, Any]) -> Any:
"""Resolve ``{tokens}`` in config values against the manifest context."""
if isinstance(value, str):
whole = _TOKEN_RE.match(value)
if whole:
return _get_dotted(context, whole.group(1))
def replace(match: re.Match[str]) -> str:
resolved = _get_dotted(context, match.group(1))
return str(resolved)
return _PARTIAL_TOKEN_RE.sub(replace, value)
if isinstance(value, list):
return [resolve_template(item, context) for item in value]
if isinstance(value, dict):
return {key: resolve_template(val, context) for key, val in value.items()}
return value
def save_login_state(
*,
storage_state: str | Path,
login_url: str = DEFAULT_LOGIN_URL,
headless: bool = False,
wait_seconds: int = 0,
) -> Path:
"""Open a browser for manual login and save the authenticated state."""
if headless and wait_seconds <= 0:
raise ValueError("headless browser-login requires --wait-seconds > 0")
sync_playwright, _ = _require_playwright()
state_path = Path(storage_state).expanduser()
state_path.parent.mkdir(parents=True, exist_ok=True)
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=headless)
context = browser.new_context()
page = context.new_page()
page.goto(login_url, wait_until="domcontentloaded")
if wait_seconds > 0:
page.wait_for_timeout(wait_seconds * 1000)
else:
input("Log in to JoinQuant in the opened browser, then press Enter here to save state...")
context.storage_state(path=str(state_path))
browser.close()
state_path.chmod(0o600)
return state_path
def save_login_state_from_env(
*,
env_path: str | Path,
storage_state: str | Path,
login_url: str = DEFAULT_LOGIN_URL,
headless: bool = True,
out_dir: str | Path | None = None,
timeout_ms: int = 120_000,
) -> dict[str, Any]:
"""Try to log in with env-file credentials and save browser state.
This handles the normal password-login form. If JoinQuant presents CAPTCHA,
slide verification, SMS verification, or 2FA, the report will mark the run
as not logged in and save a screenshot for manual diagnosis.
"""
env = load_env_file(env_path)
username = env.get("JOINQUANT_USERNAME")
password = env.get("JOINQUANT_PASSWORD")
if not username or not password:
raise ValueError("JOINQUANT_USERNAME and JOINQUANT_PASSWORD are required")
sync_playwright, _ = _require_playwright()
state_path = Path(storage_state).expanduser()
state_path.parent.mkdir(parents=True, exist_ok=True)
artifact_dir = Path(out_dir or state_path.parent / "joinquant_login_artifacts")
artifact_dir.mkdir(parents=True, exist_ok=True)
screenshot_path = artifact_dir / "login_after_submit.png"
html_path = artifact_dir / "login_after_submit.html"
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=headless)
context = browser.new_context()
page = context.new_page()
page.set_default_timeout(timeout_ms)
page.goto(login_url, wait_until="networkidle", timeout=timeout_ms)
page.locator('input[name="username"], input.pwd-phone').first.fill(username)
page.locator('input[name="pwd"], input.jq-login__password').first.fill(password)
checkbox = page.locator("#agreementBox, input.agreement-box").first
if checkbox.count():
checkbox.check(force=True)
page.locator("button.btnPwdSubmit, button.login-submit").first.click()
page.wait_for_timeout(5000)
html = page.content()
html_path.write_text(html, encoding="utf-8")
page.screenshot(path=str(screenshot_path), full_page=True)
login_inputs = page.locator('input[name="username"], input[name="pwd"]').count()
current_url = page.url
logged_in = login_inputs == 0 and "/user/login" not in current_url
if logged_in:
context.storage_state(path=str(state_path))
state_path.chmod(0o600)
browser.close()
report = {
"created_at": datetime.now(timezone.utc).isoformat(),
"logged_in": bool(logged_in),
"storage_state": str(state_path) if logged_in else "",
"artifact_dir": str(artifact_dir),
"screenshot": str(screenshot_path),
"html": str(html_path),
"current_url": current_url,
"notes": (
"Logged in and saved browser state."
if logged_in
else "Login did not complete. CAPTCHA/SMS/2FA or invalid credentials may be required."
),
}
report_path = artifact_dir / "login_report.json"
report["report_path"] = str(report_path)
report_path.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
return report
def browser_snapshot(
*,
url: str,
storage_state: str | Path,
out_dir: str | Path,
headless: bool = True,
timeout_ms: int = 60_000,
) -> dict[str, Path]:
"""Save a logged-in page screenshot and HTML for selector discovery."""
sync_playwright, _ = _require_playwright()
root = Path(out_dir)
root.mkdir(parents=True, exist_ok=True)
html_path = root / "page.html"
screenshot_path = root / "page.png"
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=headless)
context = browser.new_context(storage_state=str(Path(storage_state).expanduser()))
page = context.new_page()
page.set_default_timeout(timeout_ms)
page.goto(url, wait_until="networkidle")
html_path.write_text(page.content(), encoding="utf-8")
page.screenshot(path=str(screenshot_path), full_page=True)
browser.close()
return {"html": html_path, "screenshot": screenshot_path}
def _locator(page: Any, selector: str):
return page.locator(selector).first
def _action_fail(action: dict[str, Any], exc: Exception) -> None:
if action.get("optional", False):
return
raise exc
def _run_action(page: Any, action: dict[str, Any], context: dict[str, Any], timeout_default: int) -> dict[str, Any]:
resolved = resolve_template(action, context)
kind = resolved["type"]
timeout_ms = int(resolved.get("timeout_ms", timeout_default))
record: dict[str, Any] = {"type": kind, "status": "ok"}
try:
if kind == "goto":
page.goto(str(resolved["url"]), wait_until=resolved.get("wait_until", "domcontentloaded"), timeout=timeout_ms)
elif kind == "click":
_locator(page, str(resolved["selector"])).click(
timeout=timeout_ms,
force=bool(resolved.get("force", False)),
)
elif kind == "fill":
_locator(page, str(resolved["selector"])).fill(str(resolved.get("text", "")), timeout=timeout_ms)
elif kind == "press":
_locator(page, str(resolved["selector"])).press(str(resolved["key"]), timeout=timeout_ms)
elif kind == "set_input_files":
paths = resolved.get("paths", [])
if isinstance(paths, str):
paths = [path for path in paths.split(",") if path]
_locator(page, str(resolved["selector"])).set_input_files(paths, timeout=timeout_ms)
record["n_files"] = len(paths)
elif kind == "paste_text_file":
text = Path(str(resolved["path"])).read_text(encoding="utf-8")
loc = _locator(page, str(resolved["selector"]))
loc.click(timeout=timeout_ms)
page.keyboard.press("Control+A")
page.keyboard.insert_text(text)
record["n_chars"] = len(text)
elif kind == "set_ace_editor_file":
text = Path(str(resolved["path"])).read_text(encoding="utf-8")
selector = str(resolved.get("selector", ".ace_editor"))
page.wait_for_function(
"(selector) => Boolean(window.ace && document.querySelector(selector))",
arg=selector,
timeout=timeout_ms,
)
page.evaluate(
"""(arg) => {
const node = document.querySelector(arg.selector);
if (!node || !window.ace) {
throw new Error(`Ace editor not found for ${arg.selector}`);
}
const editor = window.ace.edit(node);
editor.setValue(arg.text, -1);
editor.clearSelection();
editor.focus();
return editor.getValue().length;
}""",
{"selector": selector, "text": text},
)
record["n_chars"] = len(text)
elif kind == "evaluate":
page.evaluate(str(resolved["script"]), resolved.get("arg"))
elif kind == "wait_for_selector":
page.wait_for_selector(str(resolved["selector"]), timeout=timeout_ms)
elif kind == "wait_for_timeout":
page.wait_for_timeout(int(resolved.get("timeout_ms", 1000)))
elif kind == "download":
save_as = Path(str(resolved["save_as"]))
save_as.parent.mkdir(parents=True, exist_ok=True)
with page.expect_download(timeout=timeout_ms) as download_info:
_locator(page, str(resolved["selector"])).click(timeout=timeout_ms)
download = download_info.value
download.save_as(str(save_as))
record["path"] = str(save_as)
elif kind == "screenshot":
path = Path(str(resolved["path"]))
path.parent.mkdir(parents=True, exist_ok=True)
page.screenshot(path=str(path), full_page=bool(resolved.get("full_page", True)))
record["path"] = str(path)
else:
raise ValueError(f"Unsupported browser action type: {kind}")
except Exception as exc: # pragma: no cover - exercised only with Playwright.
record["status"] = "skipped" if resolved.get("optional", False) else "failed"
record["error"] = str(exc)
_action_fail(resolved, exc)
return record
def run_browser_flow(
*,
manifest_path: str | Path,
config_path: str | Path,
storage_state: str | Path,
out_dir: str | Path | None = None,
headless: bool | None = None,
auto_reconcile: bool = True,
flow_name: str | None = None,
) -> dict[str, Any]:
"""Run configured browser automation for a JoinQuant web workflow."""
sync_playwright, _ = _require_playwright()
manifest = load_json(manifest_path)
config = load_json(config_path)
flow = flow_name or str(config.get("flow") or "browser")
artifact_dir = Path(out_dir or Path(str(manifest["joinquant_export_dir"])).parent / f"browser_{flow}")
artifact_dir.mkdir(parents=True, exist_ok=True)
context_vars = _manifest_context(manifest, artifact_dir, config)
timeout_ms = int(config.get("timeout_ms", 120_000))
actions = config.get("actions") or []
if not actions:
raise ValueError("Browser config contains no actions")
records: list[dict[str, Any]] = []
failure: Exception | None = None
failure_artifacts: dict[str, str] = {}
with sync_playwright() as pw:
browser = pw.chromium.launch(
headless=bool(config.get("headless", False) if headless is None else headless)
)
context = browser.new_context(
storage_state=str(Path(storage_state).expanduser()),
accept_downloads=True,
)
page = context.new_page()
page.set_default_timeout(timeout_ms)
try:
for action in actions:
records.append(_run_action(page, action, context_vars, timeout_ms))
except Exception as exc: # pragma: no cover - requires live browser UI.
failure = exc
screenshot_path = artifact_dir / "failure.png"
html_path = artifact_dir / "failure.html"
try:
page.screenshot(path=str(screenshot_path), full_page=True)
html_path.write_text(page.content(), encoding="utf-8")
failure_artifacts = {
"screenshot": str(screenshot_path),
"html": str(html_path),
}
except Exception as artifact_exc:
failure_artifacts = {"artifact_error": str(artifact_exc)}
records.append({
"type": "browser_flow",
"status": "failed",
"error": str(exc),
**failure_artifacts,
})
browser.close()
expected = manifest.get("expected_joinquant_csvs", {})
downloaded = {
key: str(path)
for key, value in expected.items()
if (path := Path(str(value))).exists()
}
reconcile_paths: dict[str, str] = {}
if auto_reconcile and {"fills", "positions", "pnl"}.issubset(downloaded):
ingested = ingest_joinquant_outputs(
portfolio_name=str(manifest["portfolio_name"]),
fills_csv=downloaded["fills"],
positions_csv=downloaded["positions"],
pnl_csv=downloaded["pnl"],
out_dir=Path(str(manifest["joinquant_export_dir"])).parent / "ingested",
)
reconciled = reconcile_joinquant(
portfolio_name=str(manifest["portfolio_name"]),
targets_dir=str(manifest["targets_dir"]),
our_fills_path=str(manifest["fills_path"]),
our_positions_path=str(manifest["positions_path"]),
our_pnl_path=str(manifest["pnl_path"]),
jq_fills_path=str(ingested["fills"]),
jq_positions_path=str(ingested["positions"]),
jq_pnl_path=str(ingested["pnl"]),
out_dir=Path(str(manifest["joinquant_export_dir"])).parent / "reconcile",
)
reconcile_paths = {key: str(value) for key, value in reconciled.items()}
report = {
"created_at": datetime.now(timezone.utc).isoformat(),
"manifest_path": str(manifest_path),
"config_path": str(config_path),
"flow": flow,
"status": "failed" if failure else "ok",
"storage_state": str(storage_state),
"artifact_dir": str(artifact_dir),
"actions": records,
"downloaded": downloaded,
"reconcile_paths": reconcile_paths,
}
report_path = artifact_dir / "browser_run_report.json"
report["report_path"] = str(report_path)
report_path.write_text(
json.dumps(report, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
if failure is not None:
raise RuntimeError(
f"Browser flow failed; report saved to {report_path}: {failure}"
) from failure
return report
def run_browser_backtest(
*,
manifest_path: str | Path,
config_path: str | Path,
storage_state: str | Path,
out_dir: str | Path | None = None,
headless: bool | None = None,
auto_reconcile: bool = True,
) -> dict[str, Any]:
"""Run configured browser automation for a JoinQuant backtest."""
return run_browser_flow(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=headless,
auto_reconcile=auto_reconcile,
flow_name="backtest",
)
def run_browser_sim_trade(
*,
manifest_path: str | Path,
config_path: str | Path,
storage_state: str | Path,
out_dir: str | Path | None = None,
headless: bool | None = None,
auto_reconcile: bool = True,
) -> dict[str, Any]:
"""Run configured browser automation for JoinQuant 模拟盘."""
return run_browser_flow(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=headless,
auto_reconcile=auto_reconcile,
flow_name="sim-trade",
)
+426
View File
@@ -0,0 +1,426 @@
"""CLI commands for the JoinQuant comparison plugin."""
from __future__ import annotations
import click
from plugins.joinquant.browser import (
browser_snapshot,
load_env_file,
run_browser_backtest,
run_browser_sim_trade,
save_login_state,
save_login_state_from_env,
write_browser_config_template,
)
from plugins.joinquant.export_targets import export_targets
from plugins.joinquant.ingest import ingest_joinquant_outputs
from plugins.joinquant.reconcile import reconcile_joinquant
from plugins.joinquant.smoke import prepare_smoke_test
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
@click.group(name="joinquant")
def joinquant():
"""Compare internal portfolio simulation with JoinQuant output."""
@joinquant.command("export-targets")
@click.option("--positions-path", required=True, help="Portfolio positions parquet from `portfolio build`")
@click.option("--portfolio-name", required=True, help="Portfolio run to export")
@click.option(
"--mode",
"mode",
type=click.Choice(["target_shares", "target_value"]),
default="target_shares",
show_default=True,
help="JoinQuant target order mode",
)
@click.option("--start-date", default=None, help="Inclusive YYYY-MM-DD start date")
@click.option("--end-date", default=None, help="Inclusive YYYY-MM-DD end date")
@click.option(
"--execution-calendar-path",
default=None,
help="Daily data parquet/dataset used to shift position dates to next execution session",
)
@click.option("--out-dir", default="plugins_output/joinquant/targets", show_default=True)
@click.option("--force", is_flag=True, help="Overwrite frozen target/snapshot files")
def export_targets_cmd(
positions_path,
portfolio_name,
mode,
start_date,
end_date,
execution_calendar_path,
out_dir,
force,
):
"""Export frozen daily target files for JoinQuant."""
snapshots = export_targets(
positions_path=positions_path,
portfolio_name=portfolio_name,
mode=mode,
start_date=start_date,
end_date=end_date,
execution_calendar_path=execution_calendar_path,
out_dir=out_dir,
force=force,
)
click.echo(f"Exported JoinQuant targets: {len(snapshots)} day(s)")
for snapshot in snapshots:
click.echo(
f" {snapshot['date']}: {snapshot['n_symbols']} symbols, "
f"sha256={str(snapshot['file_sha256'])[:12]}"
)
@joinquant.command("ingest")
@click.option("--portfolio-name", required=True, help="Portfolio run name")
@click.option("--fills-csv", required=True, help="JoinQuant fills CSV")
@click.option("--positions-csv", required=True, help="JoinQuant positions CSV")
@click.option("--pnl-csv", required=True, help="JoinQuant daily PnL CSV")
@click.option("--out-dir", default="plugins_output/joinquant/ingested", show_default=True)
def ingest_cmd(portfolio_name, fills_csv, positions_csv, pnl_csv, out_dir):
"""Normalize JoinQuant CSV exports to parquet."""
paths = ingest_joinquant_outputs(
portfolio_name=portfolio_name,
fills_csv=fills_csv,
positions_csv=positions_csv,
pnl_csv=pnl_csv,
out_dir=out_dir,
)
click.echo(f"Saved JoinQuant fills: {paths['fills']}")
click.echo(f"Saved JoinQuant positions: {paths['positions']}")
click.echo(f"Saved JoinQuant pnl: {paths['pnl']}")
@joinquant.command("reconcile")
@click.option("--portfolio-name", required=True, help="Portfolio run name")
@click.option("--targets-dir", required=True, help="Directory containing exported daily target files")
@click.option("--our-fills-path", required=True, help="Internal simulator fills parquet")
@click.option("--our-positions-path", required=True, help="Internal portfolio positions parquet")
@click.option("--our-pnl-path", required=True, help="Internal simulator PnL parquet")
@click.option("--jq-fills-path", required=True, help="Normalized JoinQuant fills parquet")
@click.option("--jq-positions-path", required=True, help="Normalized JoinQuant positions parquet")
@click.option("--jq-pnl-path", required=True, help="Normalized JoinQuant PnL parquet")
@click.option("--out-dir", default="plugins_output/joinquant/reconcile", show_default=True)
@click.option("--share-tolerance", default=0.0, show_default=True, type=float)
@click.option("--price-rel-tolerance", default=1e-4, show_default=True, type=float)
@click.option("--pnl-tolerance", default=1.0, show_default=True, type=float)
@click.option("--booksize", default=None, type=float, help="Booksize for value tolerance inference")
def reconcile_cmd(
portfolio_name,
targets_dir,
our_fills_path,
our_positions_path,
our_pnl_path,
jq_fills_path,
jq_positions_path,
jq_pnl_path,
out_dir,
share_tolerance,
price_rel_tolerance,
pnl_tolerance,
booksize,
):
"""Write per-symbol and daily JoinQuant reconciliation reports."""
paths = reconcile_joinquant(
portfolio_name=portfolio_name,
targets_dir=targets_dir,
our_fills_path=our_fills_path,
our_positions_path=our_positions_path,
our_pnl_path=our_pnl_path,
jq_fills_path=jq_fills_path,
jq_positions_path=jq_positions_path,
jq_pnl_path=jq_pnl_path,
out_dir=out_dir,
share_tolerance=share_tolerance,
price_rel_tolerance=price_rel_tolerance,
pnl_tolerance=pnl_tolerance,
booksize=booksize,
)
click.echo(f"Saved reconciliation parquet: {paths['daily_reconcile']}")
click.echo(f"Saved reconciliation summary: {paths['summary_md']}")
click.echo(f"Saved reconciliation CSV: {paths['summary_csv']}")
@joinquant.command("write-wrapper")
@click.option("--portfolio-name", required=True, help="Portfolio run name")
@click.option(
"--mode",
"mode",
type=click.Choice(["target_shares", "target_value"]),
default="target_shares",
show_default=True,
)
@click.option("--out-path", required=True, help="Path for generated standalone strategy")
@click.option("--allow-short", is_flag=True, help="Do not clip negative targets in the generated wrapper")
@click.option(
"--embed-targets-dir",
default=None,
help="Embed CSV target files from this directory into the strategy source",
)
def write_wrapper_cmd(portfolio_name, mode, out_path, allow_short, embed_targets_dir):
"""Generate a standalone JoinQuant wrapper strategy."""
path = write_wrapper_strategy(
portfolio_name=portfolio_name,
mode=mode,
out_path=out_path,
allow_short=allow_short,
embedded_targets_dir=embed_targets_dir,
)
click.echo(f"Saved JoinQuant wrapper strategy: {path}")
@joinquant.command("prepare-smoke")
@click.option("--out-dir", required=True, help="Root directory for generated smoke-test artifacts")
@click.option(
"--universe",
default="sh600000,sz000001,sh600519,sz002594,sz300750",
show_default=True,
help="Universe for the tiny real-data download",
)
@click.option("--trade-symbol", default="sh600000", show_default=True)
@click.option("--start-date", default="2024-01-02", show_default=True)
@click.option("--end-date", default="2024-01-12", show_default=True)
@click.option("--portfolio-name", default="jq_smoke_one_stock_long", show_default=True)
@click.option("--shares", default=1000, show_default=True, type=int)
@click.option("--booksize", default=1_000_000.0, show_default=True, type=float)
@click.option("--max-signal-dates", default=3, show_default=True, type=int)
@click.option("--cost-bps", default=5.0, show_default=True, type=float)
@click.option("--slippage-bps", default=5.0, show_default=True, type=float)
@click.option("--volume-frac", default=0.02, show_default=True, type=float)
@click.option("--force", is_flag=True, help="Overwrite existing frozen target files")
def prepare_smoke_cmd(
out_dir,
universe,
trade_symbol,
start_date,
end_date,
portfolio_name,
shares,
booksize,
max_signal_dates,
cost_bps,
slippage_bps,
volume_frac,
force,
):
"""Prepare a one-command local real-data JoinQuant smoke test."""
manifest = prepare_smoke_test(
out_dir=out_dir,
universe=universe,
trade_symbol=trade_symbol,
start_date=start_date,
end_date=end_date,
portfolio_name=portfolio_name,
shares=shares,
booksize=booksize,
max_signal_dates=max_signal_dates,
cost_bps=cost_bps,
slippage_bps=slippage_bps,
volume_frac=volume_frac,
force=force,
)
click.echo(f"Prepared JoinQuant smoke manifest: {manifest['manifest_path']}")
click.echo(f"Wrapper: {manifest['wrapper_path']}")
click.echo(f"Targets: {manifest['targets_dir']}")
click.echo(f"Expected JoinQuant exports: {manifest['joinquant_export_dir']}")
summary = manifest["local_summary"]
click.echo(
f"Local simulator: {summary['n_pnl_rows']} days, "
f"PnL={summary['total_pnl']:,.2f}, cost={summary['total_cost']:,.2f}, "
f"blocked={summary['blocked_trades']}"
)
@joinquant.command("browser-login")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
help="Path where the authenticated browser state is stored",
)
@click.option(
"--login-url",
default="https://www.joinquant.com/user/login/index",
show_default=True,
)
@click.option("--headless", is_flag=True, help="Use a headless browser")
@click.option(
"--wait-seconds",
default=0,
show_default=True,
type=int,
help="Seconds to wait before saving state; 0 prompts for Enter after login",
)
def browser_login_cmd(storage_state, login_url, headless, wait_seconds):
"""Open JoinQuant login and save reusable browser session state."""
path = save_login_state(
storage_state=storage_state,
login_url=login_url,
headless=headless,
wait_seconds=wait_seconds,
)
click.echo(f"Saved JoinQuant browser state: {path}")
@joinquant.command("browser-login-env")
@click.option(
"--env-path",
default="~/.config/chinese-equity-quant/joinquant.env",
show_default=True,
help="Env file with JOINQUANT_USERNAME and JOINQUANT_PASSWORD",
)
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
help="Path where the authenticated browser state is stored",
)
@click.option(
"--login-url",
default="https://www.joinquant.com/user/login/index",
show_default=True,
)
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--out-dir", default=None, help="Login artifact directory")
@click.option("--timeout-ms", default=120_000, show_default=True, type=int)
def browser_login_env_cmd(env_path, storage_state, login_url, headed, out_dir, timeout_ms):
"""Try credential-based JoinQuant login from an env file."""
env = load_env_file(env_path)
missing = [
key for key in ["JOINQUANT_USERNAME", "JOINQUANT_PASSWORD"]
if not env.get(key)
]
if missing:
raise click.ClickException(f"Missing required env keys: {', '.join(missing)}")
report = save_login_state_from_env(
env_path=env_path,
storage_state=storage_state,
login_url=login_url,
headless=not headed,
out_dir=out_dir,
timeout_ms=timeout_ms,
)
click.echo(f"Saved login report: {report['report_path']}")
if report["logged_in"]:
click.echo(f"Saved JoinQuant browser state: {report['storage_state']}")
else:
raise click.ClickException(
"JoinQuant login did not complete. Check the saved screenshot/HTML; "
"CAPTCHA/SMS/2FA may require `joinquant browser-login` once."
)
@joinquant.command("write-browser-config")
@click.option("--out-path", required=True, help="Path for JSON browser automation config")
@click.option("--strategy-url", default="", help="JoinQuant strategy edit/backtest/模拟盘 URL")
@click.option(
"--flow",
type=click.Choice(["backtest", "sim-trade"]),
default="backtest",
show_default=True,
help="Browser automation template to write",
)
def write_browser_config_cmd(out_path, strategy_url, flow):
"""Write a browser automation selector/action config template."""
path = write_browser_config_template(out_path, strategy_url=strategy_url, flow=flow)
click.echo(f"Saved JoinQuant browser config template: {path}")
@joinquant.command("browser-snapshot")
@click.option("--url", required=True, help="Logged-in JoinQuant page URL to inspect")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
)
@click.option("--out-dir", required=True, help="Directory for page.html and page.png")
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--timeout-ms", default=60_000, show_default=True, type=int)
def browser_snapshot_cmd(url, storage_state, out_dir, headed, timeout_ms):
"""Save a page screenshot and HTML for selector discovery."""
paths = browser_snapshot(
url=url,
storage_state=storage_state,
out_dir=out_dir,
headless=not headed,
timeout_ms=timeout_ms,
)
click.echo(f"Saved HTML: {paths['html']}")
click.echo(f"Saved screenshot: {paths['screenshot']}")
@joinquant.command("run-browser-backtest")
@click.option("--manifest-path", required=True, help="Manifest from `joinquant prepare-smoke`")
@click.option("--config-path", required=True, help="JSON browser automation config")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
)
@click.option("--out-dir", default=None, help="Browser-run artifact directory")
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--no-auto-reconcile", is_flag=True, help="Skip ingest/reconcile after downloads")
def run_browser_backtest_cmd(
manifest_path,
config_path,
storage_state,
out_dir,
headed,
no_auto_reconcile,
):
"""Run JoinQuant browser automation from manifest and config."""
report = run_browser_backtest(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=not headed,
auto_reconcile=not no_auto_reconcile,
)
click.echo(f"Saved browser run report: {report['report_path']}")
if report["downloaded"]:
click.echo("Downloaded JoinQuant CSVs:")
for key, path in report["downloaded"].items():
click.echo(f" {key}: {path}")
if report["reconcile_paths"]:
click.echo(f"Saved reconciliation summary: {report['reconcile_paths']['summary_md']}")
@joinquant.command("run-browser-sim")
@click.option("--manifest-path", required=True, help="Manifest from target preparation")
@click.option("--config-path", required=True, help="JSON simulated-trading browser config")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
)
@click.option("--out-dir", default=None, help="Browser-run artifact directory")
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--no-auto-reconcile", is_flag=True, help="Skip ingest/reconcile after downloads")
def run_browser_sim_cmd(
manifest_path,
config_path,
storage_state,
out_dir,
headed,
no_auto_reconcile,
):
"""Run JoinQuant 模拟盘 browser automation from manifest and config."""
report = run_browser_sim_trade(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=not headed,
auto_reconcile=not no_auto_reconcile,
)
click.echo(f"Saved browser sim-trade report: {report['report_path']}")
if report["downloaded"]:
click.echo("Downloaded JoinQuant CSVs:")
for key, path in report["downloaded"].items():
click.echo(f" {key}: {path}")
if report["reconcile_paths"]:
click.echo(f"Saved reconciliation summary: {report['reconcile_paths']['summary_md']}")
+248
View File
@@ -0,0 +1,248 @@
"""Export portfolio positions as frozen JoinQuant target files."""
from __future__ import annotations
import hashlib
import json
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Iterable, Literal
import pandas as pd
from pipeline.common.schema import POSITION_COLUMNS
from plugins.joinquant.schema import JOINQUANT_TARGET_COLUMNS
from plugins.joinquant.symbols import to_joinquant_symbol
ExportMode = Literal["target_shares", "target_value"]
def _date_text(value: object) -> str:
return pd.Timestamp(value).strftime("%Y-%m-%d")
def _date_file_stem(date_text: str) -> str:
return pd.Timestamp(date_text).strftime("%Y%m%d")
def _snapshot_root_for(targets_root: Path) -> Path:
if targets_root.name == "targets":
return targets_root.parent / "snapshots"
return targets_root / "snapshots"
def _sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as fh:
for chunk in iter(lambda: fh.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def _check_position_columns(df: pd.DataFrame) -> None:
missing = [col for col in POSITION_COLUMNS if col not in df.columns]
if missing:
raise ValueError(f"Positions input missing required columns: {missing}")
def _filter_dates(
df: pd.DataFrame,
start_date: str | None,
end_date: str | None,
*,
date_column: str = "date",
) -> pd.DataFrame:
out = df.copy()
out[date_column] = pd.to_datetime(out[date_column]).dt.normalize()
if start_date:
out = out[out[date_column] >= pd.Timestamp(start_date).normalize()]
if end_date:
out = out[out[date_column] <= pd.Timestamp(end_date).normalize()]
return out
def _read_execution_calendar(path: str | Path) -> pd.DatetimeIndex:
data = pd.read_parquet(path)
if "date" not in data.columns:
raise ValueError("execution calendar parquet must contain a 'date' column")
dates = pd.to_datetime(data["date"], errors="coerce").dropna().dt.normalize()
return pd.DatetimeIndex(sorted(dates.unique()))
def _apply_execution_calendar(df: pd.DataFrame, calendar_path: str | Path) -> pd.DataFrame:
calendar = _read_execution_calendar(calendar_path)
if calendar.empty:
raise ValueError("execution calendar contains no dates")
source_dates = pd.to_datetime(df["date"]).dt.normalize()
positions = calendar.searchsorted(source_dates, side="right")
out = df.copy()
out["source_date"] = source_dates
out["export_date"] = pd.NaT
valid = positions < len(calendar)
out.loc[valid, "export_date"] = calendar.take(positions[valid])
return out[out["export_date"].notna()].copy()
def build_target_frame(
positions: pd.DataFrame,
*,
portfolio_name: str | None = None,
mode: ExportMode = "target_shares",
start_date: str | None = None,
end_date: str | None = None,
snapshot_ids: dict[str, str] | None = None,
execution_calendar_path: str | Path | None = None,
) -> pd.DataFrame:
"""Build normalized JoinQuant target rows from portfolio positions.
``target_shares`` is populated from ``position_shares`` because the core
simulator executes the discretized book, not continuous research shares.
"""
if mode not in {"target_shares", "target_value"}:
raise ValueError("mode must be 'target_shares' or 'target_value'")
_check_position_columns(positions)
df = positions.copy()
df["date"] = pd.to_datetime(df["date"]).dt.normalize()
if execution_calendar_path is not None:
df = _apply_execution_calendar(df, execution_calendar_path)
df = df.drop(columns=["date"]).rename(columns={"export_date": "date"})
df["source_date"] = pd.to_datetime(df["source_date"]).dt.strftime("%Y-%m-%d")
df = _filter_dates(df, start_date, end_date)
else:
df = _filter_dates(df, start_date, end_date)
if portfolio_name is not None:
df = df[df["portfolio_name"].astype(str) == portfolio_name]
if df.empty:
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
out = pd.DataFrame({
"date": df["date"].map(_date_text),
"portfolio_name": df["portfolio_name"].astype(str),
"symbol_id": df["symbol_id"].astype(str),
"jq_symbol": df["symbol_id"].map(to_joinquant_symbol),
"target_shares": pd.to_numeric(df["position_shares"], errors="coerce").fillna(0).astype("int64"),
"target_value": pd.to_numeric(df["target_value"], errors="coerce").fillna(0.0),
"target_weight": pd.to_numeric(df["target_weight"], errors="coerce").fillna(0.0),
"export_mode": mode,
"snapshot_id": "",
})
if snapshot_ids:
out["snapshot_id"] = out["date"].map(snapshot_ids).fillna("")
return out[JOINQUANT_TARGET_COLUMNS].sort_values(
["date", "portfolio_name", "symbol_id"]
).reset_index(drop=True)
def export_targets(
positions_path: str | Path,
*,
portfolio_name: str,
mode: ExportMode = "target_shares",
out_dir: str | Path = "plugins_output/joinquant/targets",
start_date: str | None = None,
end_date: str | None = None,
execution_calendar_path: str | Path | None = None,
force: bool = False,
) -> list[dict[str, object]]:
"""Export one daily CSV/parquet target file plus a snapshot JSON per date.
Args:
positions_path: Parquet file produced by ``portfolio build``.
portfolio_name: Portfolio run to export.
mode: ``target_shares`` or ``target_value``.
out_dir: Target root. Files are written to ``out_dir/portfolio_name``.
If the root is named ``targets``, snapshots are written to the
sibling ``snapshots`` directory.
start_date: Optional inclusive start date.
end_date: Optional inclusive end date.
execution_calendar_path: Optional daily-bar parquet dataset used to
shift each position date to the next available execution session.
This matches the internal simulator's next-open convention.
force: If false, existing target or snapshot files are treated as
frozen and cause ``FileExistsError``.
Returns:
Snapshot metadata dictionaries, one per exported date.
"""
positions_path = Path(positions_path)
targets_root = Path(out_dir)
snapshot_root = _snapshot_root_for(targets_root)
targets_portfolio_dir = targets_root / portfolio_name
snapshots_portfolio_dir = snapshot_root / portfolio_name
targets_portfolio_dir.mkdir(parents=True, exist_ok=True)
snapshots_portfolio_dir.mkdir(parents=True, exist_ok=True)
positions = pd.read_parquet(positions_path)
filtered = positions.copy()
filtered["date"] = pd.to_datetime(filtered["date"]).dt.normalize()
if execution_calendar_path is not None:
filtered = _apply_execution_calendar(filtered, execution_calendar_path)
filtered = filtered.drop(columns=["date"]).rename(columns={"export_date": "date"})
filtered["source_date"] = pd.to_datetime(filtered["source_date"]).dt.strftime("%Y-%m-%d")
filtered = _filter_dates(filtered, start_date, end_date)
else:
filtered = _filter_dates(filtered, start_date, end_date)
filtered = filtered[filtered["portfolio_name"].astype(str) == portfolio_name]
if filtered.empty:
return []
date_texts = sorted(filtered["date"].map(_date_text).unique())
snapshot_ids = {
date_text: f"jq-{portfolio_name}-{date_text}-{uuid.uuid4().hex[:12]}"
for date_text in date_texts
}
targets = build_target_frame(
filtered,
portfolio_name=portfolio_name,
mode=mode,
snapshot_ids=snapshot_ids,
execution_calendar_path=None,
)
snapshots: list[dict[str, object]] = []
for date_text, daily in targets.groupby("date", sort=True):
stem = _date_file_stem(date_text)
csv_path = targets_portfolio_dir / f"{stem}.csv"
parquet_path = targets_portfolio_dir / f"{stem}.parquet"
snapshot_path = snapshots_portfolio_dir / f"{stem}.json"
existing: Iterable[Path] = (csv_path, parquet_path, snapshot_path)
if not force:
conflicts = [str(path) for path in existing if path.exists()]
if conflicts:
raise FileExistsError(
"Frozen JoinQuant target already exists; use --force to overwrite: "
+ ", ".join(conflicts)
)
daily = daily[JOINQUANT_TARGET_COLUMNS].reset_index(drop=True)
daily.to_csv(csv_path, index=False)
daily.to_parquet(parquet_path, index=False)
file_hash = _sha256_file(csv_path)
snapshot = {
"snapshot_id": snapshot_ids[date_text],
"portfolio_name": portfolio_name,
"date": date_text,
"export_mode": mode,
"source_positions_path": str(positions_path),
"execution_calendar_path": str(execution_calendar_path) if execution_calendar_path else None,
"created_at": datetime.now(timezone.utc).isoformat(),
"n_symbols": int(len(daily)),
"file_sha256": file_hash,
"notes": "Frozen JoinQuant target file.",
"target_csv_path": str(csv_path),
"target_parquet_path": str(parquet_path),
}
snapshot_path.write_text(
json.dumps(snapshot, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
snapshots.append(snapshot)
return snapshots
+225
View File
@@ -0,0 +1,225 @@
"""Normalize JoinQuant CSV exports into plugin parquet schemas."""
from __future__ import annotations
import re
from pathlib import Path
import numpy as np
import pandas as pd
from plugins.joinquant.schema import (
JOINQUANT_FILL_COLUMNS,
JOINQUANT_PNL_COLUMNS,
JOINQUANT_POSITION_COLUMNS,
)
from plugins.joinquant.symbols import normalize_symbol_pair, to_joinquant_symbol
def _clean_name(name: object) -> str:
text = str(name).strip().lower()
text = re.sub(r"[\s\-.()/]+", "_", text)
return re.sub(r"[^0-9a-z_]+", "", text).strip("_")
def _clean_columns(df: pd.DataFrame) -> pd.DataFrame:
out = df.copy()
out.columns = [_clean_name(col) for col in out.columns]
return out
def _pick(df: pd.DataFrame, candidates: list[str]) -> str | None:
for candidate in candidates:
clean = _clean_name(candidate)
if clean in df.columns:
return clean
return None
def _series_or_default(df: pd.DataFrame, candidates: list[str], default: object) -> pd.Series:
col = _pick(df, candidates)
if col is None:
return pd.Series([default] * len(df), index=df.index)
return df[col]
def _date_series(df: pd.DataFrame) -> pd.Series:
values = _series_or_default(
df,
["date", "trade_date", "datetime", "time", "created_at"],
pd.NaT,
)
parsed = pd.to_datetime(values, errors="coerce").dt.normalize()
return parsed.dt.strftime("%Y-%m-%d").fillna("")
def _numeric(values: pd.Series, default: float = 0.0) -> pd.Series:
return pd.to_numeric(values, errors="coerce").replace([np.inf, -np.inf], np.nan).fillna(default)
def _text(values: pd.Series, default: str = "") -> pd.Series:
return values.fillna(default).astype(str)
def _portfolio_series(df: pd.DataFrame, portfolio_name: str) -> pd.Series:
return _text(_series_or_default(df, ["portfolio_name", "portfolio", "strategy"], portfolio_name), portfolio_name)
def _symbol_frame(df: pd.DataFrame) -> pd.DataFrame:
internal_col = _pick(df, ["symbol_id", "internal_symbol"])
jq_col = _pick(df, ["jq_symbol", "security", "stock", "symbol", "code", "order_book_id"])
symbol_ids: list[str] = []
jq_symbols: list[str] = []
for idx in df.index:
internal = df.at[idx, internal_col] if internal_col else None
jq_value = df.at[idx, jq_col] if jq_col else None
value = internal if internal is not None and str(internal).strip() else jq_value
if value is None or not str(value).strip():
symbol_ids.append("")
jq_symbols.append("")
continue
symbol_id, jq_symbol = normalize_symbol_pair(value)
if jq_value is not None and str(jq_value).strip():
try:
_, jq_symbol = normalize_symbol_pair(jq_value)
except ValueError:
jq_symbol = to_joinquant_symbol(symbol_id)
symbol_ids.append(symbol_id)
jq_symbols.append(jq_symbol)
return pd.DataFrame({"symbol_id": symbol_ids, "jq_symbol": jq_symbols}, index=df.index)
def _signed_shares(shares: pd.Series, side: pd.Series) -> pd.Series:
signed = _numeric(shares, 0.0)
side_text = side.fillna("").astype(str).str.lower()
sell = side_text.str.contains("sell|short|close|reduce|-", regex=True)
buy = side_text.str.contains("buy|long|open|add|\\+", regex=True)
signed = signed.abs()
signed = signed.mask(sell, -signed)
signed = signed.mask(~(sell | buy), _numeric(shares, 0.0))
return signed
def normalize_fills_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
"""Read a JoinQuant fills CSV and return ``JOINQUANT_FILL_COLUMNS``."""
raw = _clean_columns(pd.read_csv(path))
symbols = _symbol_frame(raw)
side = _text(_series_or_default(raw, ["side", "action", "direction"], ""))
requested = _signed_shares(
_series_or_default(raw, ["requested_shares", "target_shares", "amount", "order_amount"], 0),
side,
)
filled = _signed_shares(
_series_or_default(raw, ["filled_shares", "filled", "filled_amount", "deal_amount", "traded_shares"], 0),
side,
)
price = _numeric(_series_or_default(raw, ["fill_price", "price", "avg_cost", "avg_price"], np.nan), np.nan)
trade_value = _numeric(
_series_or_default(raw, ["trade_value", "value", "filled_value", "turnover"], np.nan),
np.nan,
)
trade_value = trade_value.fillna((filled * price).abs()).fillna(0.0)
out = pd.DataFrame({
"date": _date_series(raw),
"portfolio_name": _portfolio_series(raw, portfolio_name),
"symbol_id": symbols["symbol_id"],
"jq_symbol": symbols["jq_symbol"],
"order_id": _text(_series_or_default(raw, ["order_id", "id"], "")),
"side": side,
"requested_shares": requested.astype(float),
"filled_shares": filled.astype(float),
"fill_price": price.astype(float),
"trade_value": trade_value.astype(float),
"trade_cost": _numeric(_series_or_default(raw, ["trade_cost", "cost", "commission", "fee"], 0.0), 0.0),
"blocked": _numeric(_series_or_default(raw, ["blocked", "is_blocked"], 0), 0).astype("int64"),
"raw_status": _text(_series_or_default(raw, ["raw_status", "status", "order_status"], "")),
})
return out[JOINQUANT_FILL_COLUMNS]
def normalize_positions_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
"""Read a JoinQuant positions CSV and return ``JOINQUANT_POSITION_COLUMNS``."""
raw = _clean_columns(pd.read_csv(path))
symbols = _symbol_frame(raw)
out = pd.DataFrame({
"date": _date_series(raw),
"portfolio_name": _portfolio_series(raw, portfolio_name),
"symbol_id": symbols["symbol_id"],
"jq_symbol": symbols["jq_symbol"],
"position_shares": _numeric(
_series_or_default(raw, ["position_shares", "shares", "amount", "quantity", "total_amount"], 0),
0,
),
"position_value": _numeric(
_series_or_default(raw, ["position_value", "market_value", "value"], 0.0),
0.0,
),
"cash": _numeric(_series_or_default(raw, ["cash", "available_cash"], np.nan), np.nan),
"total_value": _numeric(
_series_or_default(raw, ["total_value", "portfolio_value", "total_asset"], np.nan),
np.nan,
),
})
return out[JOINQUANT_POSITION_COLUMNS]
def normalize_pnl_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
"""Read a JoinQuant daily PnL CSV and return ``JOINQUANT_PNL_COLUMNS``."""
raw = _clean_columns(pd.read_csv(path))
total_value = _numeric(
_series_or_default(raw, ["total_value", "portfolio_value", "total_asset"], np.nan),
np.nan,
)
pnl = _numeric(_series_or_default(raw, ["pnl", "daily_pnl", "profit", "returns_value"], np.nan), np.nan)
if pnl.isna().all() and total_value.notna().any():
pnl = total_value.diff().fillna(0.0)
out = pd.DataFrame({
"date": _date_series(raw),
"portfolio_name": _portfolio_series(raw, portfolio_name),
"gross_exposure": _numeric(
_series_or_default(raw, ["gross_exposure", "gross", "positions_value", "market_value"], np.nan),
np.nan,
),
"net_exposure": _numeric(
_series_or_default(raw, ["net_exposure", "net"], np.nan),
np.nan,
),
"cash": _numeric(_series_or_default(raw, ["cash", "available_cash"], np.nan), np.nan),
"total_value": total_value,
"pnl": pnl.fillna(0.0),
"cost": _numeric(_series_or_default(raw, ["cost", "trade_cost", "commission", "fee"], 0.0), 0.0),
"turnover": _numeric(_series_or_default(raw, ["turnover"], 0.0), 0.0),
})
return out[JOINQUANT_PNL_COLUMNS]
def ingest_joinquant_outputs(
*,
portfolio_name: str,
fills_csv: str | Path,
positions_csv: str | Path,
pnl_csv: str | Path,
out_dir: str | Path = "plugins_output/joinquant/ingested",
) -> dict[str, Path]:
"""Normalize JoinQuant CSV exports and write parquet outputs."""
out_root = Path(out_dir) / portfolio_name
out_root.mkdir(parents=True, exist_ok=True)
fills = normalize_fills_csv(fills_csv, portfolio_name)
positions = normalize_positions_csv(positions_csv, portfolio_name)
pnl = normalize_pnl_csv(pnl_csv, portfolio_name)
paths = {
"fills": out_root / "fills.pq",
"positions": out_root / "positions.pq",
"pnl": out_root / "pnl.pq",
}
fills.to_parquet(paths["fills"], index=False)
positions.to_parquet(paths["positions"], index=False)
pnl.to_parquet(paths["pnl"], index=False)
return paths
+607
View File
@@ -0,0 +1,607 @@
"""Reconcile internal simulator output against normalized JoinQuant output."""
from __future__ import annotations
from pathlib import Path
import numpy as np
import pandas as pd
from plugins.joinquant.schema import (
JOINQUANT_FILL_COLUMNS,
JOINQUANT_PNL_COLUMNS,
JOINQUANT_POSITION_COLUMNS,
JOINQUANT_TARGET_COLUMNS,
RECONCILE_COLUMNS,
)
from plugins.joinquant.symbols import to_joinquant_symbol
def _date_text(value: object) -> str:
if pd.isna(value):
return ""
return pd.Timestamp(value).strftime("%Y-%m-%d")
def _read_parquet(path: str | Path | None) -> pd.DataFrame:
if path is None:
return pd.DataFrame()
return pd.read_parquet(path)
def _normalize_common(df: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
out = df.copy()
if "date" in out.columns:
out["date"] = out["date"].map(_date_text)
else:
out["date"] = ""
if "portfolio_name" not in out.columns:
out["portfolio_name"] = portfolio_name
out["portfolio_name"] = out["portfolio_name"].fillna(portfolio_name).astype(str)
if "symbol_id" in out.columns:
out["symbol_id"] = out["symbol_id"].fillna("").astype(str)
if "jq_symbol" not in out.columns and "symbol_id" in out.columns:
out["jq_symbol"] = out["symbol_id"].map(
lambda s: to_joinquant_symbol(s) if s else ""
)
elif "jq_symbol" in out.columns:
out["jq_symbol"] = out["jq_symbol"].fillna("").astype(str)
return out
def _numeric(df: pd.DataFrame, column: str, default: float = 0.0) -> pd.Series:
if column not in df.columns:
return pd.Series([default] * len(df), index=df.index, dtype=float)
return pd.to_numeric(df[column], errors="coerce").replace([np.inf, -np.inf], np.nan)
def _weighted_price(group: pd.DataFrame, price_col: str, shares_col: str) -> float:
prices = pd.to_numeric(group[price_col], errors="coerce")
shares = pd.to_numeric(group[shares_col], errors="coerce").abs()
valid = prices.notna() & shares.notna() & (shares > 0)
if not valid.any():
return np.nan
return float(np.average(prices[valid], weights=shares[valid]))
def _load_targets(targets_dir: str | Path, portfolio_name: str) -> pd.DataFrame:
root = Path(targets_dir)
if not root.exists():
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
files_by_stem: dict[str, Path] = {}
for path in sorted(root.glob("*.csv")):
files_by_stem[path.stem] = path
for path in sorted(root.glob("*.parquet")):
files_by_stem[path.stem] = path
frames: list[pd.DataFrame] = []
for path in files_by_stem.values():
if path.suffix == ".parquet":
frame = pd.read_parquet(path)
else:
frame = pd.read_csv(path)
frames.append(frame)
if not frames:
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
targets = pd.concat(frames, ignore_index=True)
targets = _normalize_common(targets, portfolio_name)
targets = targets[targets["portfolio_name"].astype(str) == portfolio_name]
if "target_shares" not in targets.columns:
targets["target_shares"] = 0
return targets.reindex(columns=JOINQUANT_TARGET_COLUMNS)
def _aggregate_targets(targets: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if targets.empty:
return pd.DataFrame(columns=["date", "portfolio_name", "symbol_id", "jq_symbol", "target_shares"])
targets = _normalize_common(targets, portfolio_name)
targets["target_shares"] = _numeric(targets, "target_shares", 0.0)
grouped = (
targets.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
.agg(jq_symbol=("jq_symbol", "last"), target_shares=("target_shares", "last"))
)
return grouped
def _aggregate_our_fills(fills: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if fills.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "our_filled_shares",
"our_position_shares", "our_cost", "our_trade_price", "our_blocked",
"our_target_shares",
])
fills = _normalize_common(fills, portfolio_name)
fills["traded_shares"] = _numeric(fills, "traded_shares", 0.0)
fills["realized_shares"] = _numeric(fills, "realized_shares", np.nan)
fills["trade_cost"] = _numeric(fills, "trade_cost", 0.0).fillna(0.0)
fills["target_shares"] = _numeric(fills, "target_shares", np.nan)
fills["blocked"] = _numeric(fills, "blocked", 0.0).fillna(0.0)
price_col = next(
(col for col in ["trade_price", "fill_price", "execution_price", "price"] if col in fills.columns),
None,
)
rows: list[dict[str, object]] = []
for key, group in fills.groupby(["date", "portfolio_name", "symbol_id"], sort=False):
row = {
"date": key[0],
"portfolio_name": key[1],
"symbol_id": key[2],
"our_filled_shares": float(group["traded_shares"].sum()),
"our_position_shares": float(group["realized_shares"].dropna().iloc[-1])
if group["realized_shares"].notna().any() else np.nan,
"our_cost": float(group["trade_cost"].sum()),
"our_trade_price": _weighted_price(group, price_col, "traded_shares")
if price_col else np.nan,
"our_blocked": int(group["blocked"].max()),
"our_target_shares": float(group["target_shares"].dropna().iloc[-1])
if group["target_shares"].notna().any() else np.nan,
}
rows.append(row)
return pd.DataFrame(rows)
def _aggregate_our_positions(positions: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if positions.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "jq_symbol",
"our_position_fallback", "our_position_price",
])
positions = _normalize_common(positions, portfolio_name)
positions["position_shares"] = _numeric(positions, "position_shares", np.nan)
positions["price"] = _numeric(positions, "price", np.nan)
return (
positions.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
.agg(
jq_symbol=("jq_symbol", "last"),
our_position_fallback=("position_shares", "last"),
our_position_price=("price", "last"),
)
)
def _aggregate_jq_fills(fills: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if fills.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "jq_filled_shares",
"jq_trade_price", "jq_cost", "jq_blocked", "jq_requested_shares",
"raw_status",
])
fills = _normalize_common(fills, portfolio_name)
for col in JOINQUANT_FILL_COLUMNS:
if col not in fills.columns:
fills[col] = np.nan
fills["filled_shares"] = _numeric(fills, "filled_shares", 0.0).fillna(0.0)
fills["requested_shares"] = _numeric(fills, "requested_shares", np.nan)
fills["fill_price"] = _numeric(fills, "fill_price", np.nan)
fills["trade_cost"] = _numeric(fills, "trade_cost", 0.0).fillna(0.0)
fills["blocked"] = _numeric(fills, "blocked", 0.0).fillna(0.0)
fills["raw_status"] = fills["raw_status"].fillna("").astype(str)
rows: list[dict[str, object]] = []
for key, group in fills.groupby(["date", "portfolio_name", "symbol_id"], sort=False):
row = {
"date": key[0],
"portfolio_name": key[1],
"symbol_id": key[2],
"jq_filled_shares": float(group["filled_shares"].sum()),
"jq_trade_price": _weighted_price(group, "fill_price", "filled_shares"),
"jq_cost": float(group["trade_cost"].sum()),
"jq_blocked": int(group["blocked"].max()),
"jq_requested_shares": float(group["requested_shares"].dropna().iloc[-1])
if group["requested_shares"].notna().any() else np.nan,
"raw_status": ";".join([s for s in group["raw_status"].astype(str) if s]),
}
rows.append(row)
return pd.DataFrame(rows)
def _aggregate_jq_positions(positions: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if positions.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "jq_symbol", "jq_position_shares",
])
positions = _normalize_common(positions, portfolio_name)
for col in JOINQUANT_POSITION_COLUMNS:
if col not in positions.columns:
positions[col] = np.nan
positions["position_shares"] = _numeric(positions, "position_shares", np.nan)
return (
positions.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
.agg(jq_symbol=("jq_symbol", "last"), jq_position_shares=("position_shares", "last"))
)
def _portfolio_frame(df: pd.DataFrame, portfolio_name: str, prefix: str) -> pd.DataFrame:
if df.empty:
return pd.DataFrame(columns=["date", "portfolio_name"])
out = _normalize_common(df, portfolio_name)
for col in JOINQUANT_PNL_COLUMNS:
if col not in out.columns:
out[col] = np.nan
keep = ["date", "portfolio_name", "gross_exposure", "net_exposure", "cash", "total_value", "pnl", "cost", "turnover"]
out = out[keep].copy()
for col in keep[2:]:
out[col] = pd.to_numeric(out[col], errors="coerce")
out = out.groupby(["date", "portfolio_name"], as_index=False).last()
return out.rename(columns={col: f"{prefix}_{col}" for col in keep[2:]})
def _infer_booksize(targets: pd.DataFrame, our_pnl: pd.DataFrame, jq_pnl: pd.DataFrame) -> float:
candidates: list[float] = []
if "target_value" in targets.columns:
gross = (
pd.to_numeric(targets["target_value"], errors="coerce")
.abs()
.replace([np.inf, -np.inf], np.nan)
.dropna()
.sum()
)
if gross > 0:
candidates.append(float(gross))
for df in (our_pnl, jq_pnl):
if "gross_exposure" in df.columns:
val = pd.to_numeric(df["gross_exposure"], errors="coerce").max()
if pd.notna(val) and val > 0:
candidates.append(float(val))
return max(candidates) if candidates else 1.0
def _status_reason(raw_status: object) -> str | None:
text = str(raw_status or "").lower()
if "suspend" in text or "halt" in text:
return "SUSPENSION"
if "limit_up" in text or "limit up" in text or "up_limit" in text:
return "LIMIT_UP_BLOCK"
if "limit_down" in text or "limit down" in text or "down_limit" in text:
return "LIMIT_DOWN_BLOCK"
if "volume" in text or "liquid" in text:
return "VOLUME_OR_LIQUIDITY"
if "cash" in text or "margin" in text:
return "CASH_CONSTRAINT"
return None
def _classify_symbol_row(
row: pd.Series,
*,
share_tol: float,
price_rel_tol: float,
value_tol: float,
pnl_tol: float,
) -> str:
target = row.get("target_shares", np.nan)
filled_diff = abs(row.get("filled_share_diff", 0.0))
position_diff = abs(row.get("position_share_diff", 0.0))
our_present = bool(row.get("_our_present", False))
jq_present = bool(row.get("_jq_present", False))
if pd.notna(target) and target < 0 and (filled_diff > share_tol or position_diff > share_tol or not jq_present):
return "SHORT_NOT_SUPPORTED"
if not jq_present and (our_present or pd.notna(target)):
return "MISSING_IN_JOINQUANT"
if not our_present and jq_present:
return "MISSING_IN_OUR_SYSTEM"
status_reason = _status_reason(row.get("raw_status", ""))
if (filled_diff > share_tol or position_diff > share_tol) and status_reason:
return status_reason
if filled_diff > share_tol or position_diff > share_tol:
return "UNKNOWN"
our_price = row.get("our_trade_price", np.nan)
jq_price = row.get("jq_trade_price", np.nan)
if pd.notna(our_price) and pd.notna(jq_price):
denom = max(abs(float(our_price)), abs(float(jq_price)), 1.0)
if abs(float(our_price) - float(jq_price)) > price_rel_tol * denom:
return "PRICE_MISMATCH"
if abs(row.get("cost_diff", 0.0)) > value_tol:
return "COST_MODEL"
if abs(row.get("pnl_diff", 0.0)) > pnl_tol:
return "UNKNOWN"
return "MATCH"
def _classify_portfolio_row(row: pd.Series, value_tol: float, pnl_tol: float) -> str:
our_present = bool(row.get("_our_present", False))
jq_present = bool(row.get("_jq_present", False))
if not jq_present and our_present:
return "MISSING_IN_JOINQUANT"
if not our_present and jq_present:
return "MISSING_IN_OUR_SYSTEM"
if abs(row.get("cost_diff", 0.0)) > value_tol:
return "COST_MODEL"
if abs(row.get("pnl_diff", 0.0)) > pnl_tol:
return "UNKNOWN"
return "MATCH"
def _build_symbol_reconcile(
*,
portfolio_name: str,
targets: pd.DataFrame,
our_fills: pd.DataFrame,
our_positions: pd.DataFrame,
our_pnl: pd.DataFrame,
jq_fills: pd.DataFrame,
jq_positions: pd.DataFrame,
jq_pnl: pd.DataFrame,
share_tol: float,
price_rel_tol: float,
value_tol: float,
pnl_tol: float,
) -> pd.DataFrame:
target_agg = _aggregate_targets(targets, portfolio_name)
our_fill_agg = _aggregate_our_fills(our_fills, portfolio_name)
our_pos_agg = _aggregate_our_positions(our_positions, portfolio_name)
jq_fill_agg = _aggregate_jq_fills(jq_fills, portfolio_name)
jq_pos_agg = _aggregate_jq_positions(jq_positions, portfolio_name)
key_cols = ["date", "portfolio_name", "symbol_id"]
keys = []
for frame in [target_agg, our_fill_agg, our_pos_agg, jq_fill_agg, jq_pos_agg]:
if not frame.empty:
keys.append(frame[key_cols])
if not keys:
return pd.DataFrame(columns=RECONCILE_COLUMNS)
base = pd.concat(keys, ignore_index=True).drop_duplicates()
result = base.merge(target_agg, on=key_cols, how="left")
result = result.merge(our_fill_agg, on=key_cols, how="left")
result = result.merge(our_pos_agg, on=key_cols, how="left", suffixes=("", "_ourpos"))
result = result.merge(jq_fill_agg, on=key_cols, how="left")
result = result.merge(jq_pos_agg, on=key_cols, how="left", suffixes=("", "_jqpos"))
jq_symbol_cols = [col for col in result.columns if col.startswith("jq_symbol")]
jq_symbol_values = result[jq_symbol_cols].copy() if jq_symbol_cols else pd.DataFrame(index=result.index)
result["jq_symbol"] = ""
for col in jq_symbol_values.columns:
values = jq_symbol_values[col].fillna("").astype(str)
result["jq_symbol"] = result["jq_symbol"].mask(
result["jq_symbol"].eq("") & values.ne(""),
values,
)
result["jq_symbol"] = result.apply(
lambda row: row["jq_symbol"] or to_joinquant_symbol(row["symbol_id"]),
axis=1,
)
result["_our_present"] = (
result[["our_filled_shares", "our_position_shares", "our_position_fallback"]]
.notna()
.any(axis=1)
)
result["_jq_present"] = (
result[["jq_filled_shares", "jq_position_shares"]].notna().any(axis=1)
)
target_shares = pd.to_numeric(result["target_shares"], errors="coerce")
our_target = pd.to_numeric(result["our_target_shares"], errors="coerce")
jq_target = pd.to_numeric(result["jq_requested_shares"], errors="coerce")
result["target_shares"] = target_shares.where(target_shares.notna(), our_target)
result["target_shares"] = result["target_shares"].where(
result["target_shares"].notna(),
jq_target,
)
our_position = pd.to_numeric(result["our_position_shares"], errors="coerce")
our_position_fallback = pd.to_numeric(result["our_position_fallback"], errors="coerce")
result["our_position_shares"] = our_position.where(
our_position.notna(),
our_position_fallback,
)
for col in [
"target_shares", "our_filled_shares", "jq_filled_shares",
"our_position_shares", "jq_position_shares", "our_cost", "jq_cost",
]:
result[col] = pd.to_numeric(result[col], errors="coerce").fillna(0.0)
result["filled_share_diff"] = result["our_filled_shares"] - result["jq_filled_shares"]
result["position_share_diff"] = result["our_position_shares"] - result["jq_position_shares"]
result["trade_price_diff"] = np.where(
result["our_trade_price"].notna() & result["jq_trade_price"].notna(),
result["our_trade_price"] - result["jq_trade_price"],
np.nan,
)
result["cost_diff"] = result["our_cost"] - result["jq_cost"]
our_daily = _portfolio_frame(our_pnl, portfolio_name, "our")
jq_daily = _portfolio_frame(jq_pnl, portfolio_name, "jq")
pnl_daily = our_daily.merge(jq_daily, on=["date", "portfolio_name"], how="outer")
if not pnl_daily.empty:
pnl_daily["our_pnl"] = pd.to_numeric(pnl_daily.get("our_pnl"), errors="coerce").fillna(0.0)
pnl_daily["jq_pnl"] = pd.to_numeric(pnl_daily.get("jq_pnl"), errors="coerce").fillna(0.0)
pnl_daily["pnl_diff"] = pnl_daily["our_pnl"] - pnl_daily["jq_pnl"]
result = result.merge(
pnl_daily[["date", "portfolio_name", "our_pnl", "jq_pnl", "pnl_diff"]],
on=["date", "portfolio_name"],
how="left",
)
else:
result["our_pnl"] = 0.0
result["jq_pnl"] = 0.0
result["pnl_diff"] = 0.0
for col in ["our_pnl", "jq_pnl", "pnl_diff"]:
result[col] = pd.to_numeric(result[col], errors="coerce").fillna(0.0)
result["raw_status"] = result.get("raw_status", "").fillna("")
result["diff_reason"] = result.apply(
_classify_symbol_row,
axis=1,
share_tol=share_tol,
price_rel_tol=price_rel_tol,
value_tol=value_tol,
pnl_tol=pnl_tol,
)
return result[RECONCILE_COLUMNS].sort_values(
["date", "portfolio_name", "symbol_id"]
).reset_index(drop=True)
def _build_portfolio_summary(
*,
portfolio_name: str,
our_pnl: pd.DataFrame,
jq_pnl: pd.DataFrame,
value_tol: float,
pnl_tol: float,
) -> pd.DataFrame:
our = _portfolio_frame(our_pnl, portfolio_name, "our")
jq = _portfolio_frame(jq_pnl, portfolio_name, "jq")
if our.empty and jq.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "diff_reason",
"our_pnl", "jq_pnl", "pnl_diff",
])
summary = our.merge(jq, on=["date", "portfolio_name"], how="outer")
summary["_our_present"] = summary.filter(regex=r"^our_").notna().any(axis=1)
summary["_jq_present"] = summary.filter(regex=r"^jq_").notna().any(axis=1)
metrics = ["gross_exposure", "net_exposure", "cash", "total_value", "pnl", "cost", "turnover"]
for metric in metrics:
our_col = f"our_{metric}"
jq_col = f"jq_{metric}"
if our_col not in summary.columns:
summary[our_col] = np.nan
if jq_col not in summary.columns:
summary[jq_col] = np.nan
summary[f"{metric}_diff"] = (
pd.to_numeric(summary[our_col], errors="coerce").fillna(0.0)
- pd.to_numeric(summary[jq_col], errors="coerce").fillna(0.0)
)
summary["our_cumulative_pnl"] = (
pd.to_numeric(summary["our_pnl"], errors="coerce").fillna(0.0).cumsum()
)
summary["jq_cumulative_pnl"] = (
pd.to_numeric(summary["jq_pnl"], errors="coerce").fillna(0.0).cumsum()
)
summary["cumulative_pnl_diff"] = summary["our_cumulative_pnl"] - summary["jq_cumulative_pnl"]
summary["diff_reason"] = summary.apply(
_classify_portfolio_row,
axis=1,
value_tol=value_tol,
pnl_tol=pnl_tol,
)
summary = summary.drop(columns=["_our_present", "_jq_present"])
return summary.sort_values(["date", "portfolio_name"]).reset_index(drop=True)
def _write_summary_md(
path: Path,
*,
portfolio_name: str,
symbol_report: pd.DataFrame,
portfolio_summary: pd.DataFrame,
) -> None:
symbol_counts = (
symbol_report["diff_reason"].value_counts().sort_index()
if not symbol_report.empty else pd.Series(dtype=int)
)
portfolio_counts = (
portfolio_summary["diff_reason"].value_counts().sort_index()
if not portfolio_summary.empty else pd.Series(dtype=int)
)
lines = [
"# JoinQuant Reconciliation Summary",
"",
f"Portfolio: `{portfolio_name}`",
"",
"## Per-symbol Difference Counts",
"",
]
if symbol_counts.empty:
lines.append("No per-symbol rows were produced.")
else:
for reason, count in symbol_counts.items():
lines.append(f"- {reason}: {int(count)}")
lines.extend(["", "## Daily Portfolio Difference Counts", ""])
if portfolio_counts.empty:
lines.append("No daily portfolio rows were produced.")
else:
for reason, count in portfolio_counts.items():
lines.append(f"- {reason}: {int(count)}")
if not portfolio_summary.empty:
lines.extend(["", "## Daily Portfolio Preview", ""])
preview_cols = [
"date", "diff_reason", "our_pnl", "jq_pnl", "pnl_diff",
"our_cost", "jq_cost", "cost_diff",
]
preview_cols = [col for col in preview_cols if col in portfolio_summary.columns]
lines.append(",".join(preview_cols))
for row in portfolio_summary[preview_cols].head(20).itertuples(index=False):
lines.append(",".join(str(value) for value in row))
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
def reconcile_joinquant(
*,
portfolio_name: str,
targets_dir: str | Path,
our_fills_path: str | Path,
our_positions_path: str | Path,
our_pnl_path: str | Path,
jq_fills_path: str | Path,
jq_positions_path: str | Path,
jq_pnl_path: str | Path,
out_dir: str | Path = "plugins_output/joinquant/reconcile",
share_tolerance: float = 0.0,
price_rel_tolerance: float = 1e-4,
pnl_tolerance: float = 1.0,
booksize: float | None = None,
) -> dict[str, Path]:
"""Reconcile JoinQuant output against internal simulator output."""
targets = _load_targets(targets_dir, portfolio_name)
our_fills = _read_parquet(our_fills_path)
our_positions = _read_parquet(our_positions_path)
our_pnl = _read_parquet(our_pnl_path)
jq_fills = _read_parquet(jq_fills_path)
jq_positions = _read_parquet(jq_positions_path)
jq_pnl = _read_parquet(jq_pnl_path)
inferred_booksize = booksize or _infer_booksize(targets, our_pnl, jq_pnl)
value_tol = max(1.0, 1e-6 * float(inferred_booksize))
symbol_report = _build_symbol_reconcile(
portfolio_name=portfolio_name,
targets=targets,
our_fills=our_fills,
our_positions=our_positions,
our_pnl=our_pnl,
jq_fills=jq_fills,
jq_positions=jq_positions,
jq_pnl=jq_pnl,
share_tol=share_tolerance,
price_rel_tol=price_rel_tolerance,
value_tol=value_tol,
pnl_tol=pnl_tolerance,
)
portfolio_summary = _build_portfolio_summary(
portfolio_name=portfolio_name,
our_pnl=our_pnl,
jq_pnl=jq_pnl,
value_tol=value_tol,
pnl_tol=pnl_tolerance,
)
root = Path(out_dir) / portfolio_name
root.mkdir(parents=True, exist_ok=True)
paths = {
"daily_reconcile": root / "daily_reconcile.pq",
"summary_csv": root / "summary.csv",
"summary_md": root / "summary.md",
}
symbol_report.to_parquet(paths["daily_reconcile"], index=False)
portfolio_summary.to_csv(paths["summary_csv"], index=False)
_write_summary_md(
paths["summary_md"],
portfolio_name=portfolio_name,
symbol_report=symbol_report,
portfolio_summary=portfolio_summary,
)
return paths
+99
View File
@@ -0,0 +1,99 @@
"""Column contracts for the JoinQuant comparison plugin."""
from typing import Final
JOINQUANT_TARGET_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"target_shares",
"target_value",
"target_weight",
"export_mode",
"snapshot_id",
]
JOINQUANT_FILL_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"order_id",
"side",
"requested_shares",
"filled_shares",
"fill_price",
"trade_value",
"trade_cost",
"blocked",
"raw_status",
]
JOINQUANT_POSITION_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"position_shares",
"position_value",
"cash",
"total_value",
]
JOINQUANT_PNL_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"gross_exposure",
"net_exposure",
"cash",
"total_value",
"pnl",
"cost",
"turnover",
]
RECONCILE_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"target_shares",
"our_filled_shares",
"jq_filled_shares",
"filled_share_diff",
"our_position_shares",
"jq_position_shares",
"position_share_diff",
"our_trade_price",
"jq_trade_price",
"trade_price_diff",
"our_cost",
"jq_cost",
"cost_diff",
"our_pnl",
"jq_pnl",
"pnl_diff",
"diff_reason",
]
DIFF_REASONS: Final[list[str]] = [
"MATCH",
"SYMBOL_MAPPING",
"PRICE_MISMATCH",
"LOT_ROUNDING",
"SUSPENSION",
"LIMIT_UP_BLOCK",
"LIMIT_DOWN_BLOCK",
"VOLUME_OR_LIQUIDITY",
"COST_MODEL",
"CASH_CONSTRAINT",
"SHORT_NOT_SUPPORTED",
"CORPORATE_ACTION",
"JOINQUANT_INTERNAL_ROUNDING",
"MISSING_IN_OUR_SYSTEM",
"MISSING_IN_JOINQUANT",
"UNKNOWN",
]
+191
View File
@@ -0,0 +1,191 @@
"""End-to-end local smoke preparation for JoinQuant comparison."""
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
import pandas as pd
from pipeline.common.schema import POSITION_COLUMNS
from pipeline.data.downloader import download_universe
from pipeline.portfolio.constraints import get_constraint
from pipeline.portfolio.simulator import ReferenceSimulator
from plugins.joinquant.export_targets import export_targets
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
def build_fixed_share_positions(
data: pd.DataFrame,
*,
trade_symbol: str,
portfolio_name: str,
shares: int,
booksize: float,
max_signal_dates: int | None = None,
) -> pd.DataFrame:
"""Create a deterministic long-only fixed-share position book.
The final available data date is excluded because the internal simulator
executes each signal date at the next available open.
"""
data = data.copy()
data["date"] = pd.to_datetime(data["date"]).dt.normalize()
symbol_data = (
data[data["symbol_id"].astype(str) == trade_symbol]
.sort_values("date")
.reset_index(drop=True)
)
if len(symbol_data) < 2:
raise ValueError(f"Need at least two daily bars for {trade_symbol}")
signal_data = symbol_data.iloc[:-1].copy()
if max_signal_dates is not None and max_signal_dates > 0:
signal_data = signal_data.tail(max_signal_dates)
if signal_data.empty:
raise ValueError("No signal dates available after excluding final data date")
rows: list[dict[str, object]] = []
for row in signal_data.itertuples(index=False):
price = float(row.close)
target_value = float(shares * price)
rows.append({
"symbol_id": trade_symbol,
"date": pd.Timestamp(row.date),
"portfolio_name": portfolio_name,
"target_weight": target_value / float(booksize),
"target_value": target_value,
"target_shares": float(shares),
"position_shares": int(shares),
"position_value": target_value,
"price": price,
})
return pd.DataFrame(rows, columns=POSITION_COLUMNS)
def prepare_smoke_test(
*,
out_dir: str | Path,
universe: str = "sh600000,sz000001,sh600519,sz002594,sz300750",
trade_symbol: str = "sh600000",
start_date: str = "2024-01-02",
end_date: str = "2024-01-12",
portfolio_name: str = "jq_smoke_one_stock_long",
shares: int = 1000,
booksize: float = 1_000_000.0,
max_signal_dates: int = 3,
cost_bps: float = 5.0,
slippage_bps: float = 5.0,
volume_frac: float = 0.02,
force: bool = False,
) -> dict[str, object]:
"""Run the local side of a tiny real-data JoinQuant smoke test."""
root = Path(out_dir)
root.mkdir(parents=True, exist_ok=True)
stats = download_universe(
universe=universe,
start_date=start_date,
end_date=end_date,
output_dir=root / "daily_bars",
max_symbols=0,
chunk_size=100,
adjust="qfq",
)
data_path = Path(stats["dataset_path"])
data = pd.read_parquet(data_path)
positions = build_fixed_share_positions(
data,
trade_symbol=trade_symbol,
portfolio_name=portfolio_name,
shares=shares,
booksize=booksize,
max_signal_dates=max_signal_dates,
)
portfolio_dir = root / "portfolio"
portfolio_dir.mkdir(parents=True, exist_ok=True)
positions_path = portfolio_dir / f"{portfolio_name}.pq"
positions.to_parquet(positions_path, index=False)
constraints = [
get_constraint("suspension"),
get_constraint("price_limit"),
get_constraint("volume_cap", max_frac=volume_frac),
]
sim = ReferenceSimulator(
constraints=constraints,
cost_bps=cost_bps,
slippage_bps=slippage_bps,
)
fills, pnl = sim.run(positions, data)
execution_dir = root / "execution"
fills_dir = execution_dir / "fills"
pnl_dir = execution_dir / "pnl"
fills_dir.mkdir(parents=True, exist_ok=True)
pnl_dir.mkdir(parents=True, exist_ok=True)
fills_path = fills_dir / f"{portfolio_name}.pq"
pnl_path = pnl_dir / f"{portfolio_name}.pq"
fills.to_parquet(fills_path, index=False)
pnl.to_parquet(pnl_path, index=False)
target_root = root / "plugins_output" / "joinquant" / "targets_aligned"
snapshots = export_targets(
positions_path=positions_path,
portfolio_name=portfolio_name,
mode="target_shares",
out_dir=target_root,
execution_calendar_path=data_path,
force=force,
)
wrapper_path = root / "plugins_output" / "joinquant" / f"wrapper_strategy_{portfolio_name}.py"
write_wrapper_strategy(
portfolio_name=portfolio_name,
mode="target_shares",
out_path=wrapper_path,
)
export_dir = root / "joinquant_exports"
export_dir.mkdir(parents=True, exist_ok=True)
manifest = {
"created_at": datetime.now(timezone.utc).isoformat(),
"portfolio_name": portfolio_name,
"universe": universe,
"trade_symbol": trade_symbol,
"start_date": start_date,
"end_date": end_date,
"shares": shares,
"booksize": booksize,
"data_path": str(data_path),
"positions_path": str(positions_path),
"fills_path": str(fills_path),
"pnl_path": str(pnl_path),
"targets_dir": str(target_root / portfolio_name),
"wrapper_path": str(wrapper_path),
"joinquant_export_dir": str(export_dir),
"expected_joinquant_csvs": {
"fills": str(export_dir / "jq_fills.csv"),
"positions": str(export_dir / "jq_positions.csv"),
"pnl": str(export_dir / "jq_pnl.csv"),
},
"target_snapshots": snapshots,
"local_summary": {
"n_data_rows": int(len(data)),
"n_position_rows": int(len(positions)),
"n_fill_rows": int(len(fills)),
"n_pnl_rows": int(len(pnl)),
"total_pnl": float(pnl["pnl"].sum()) if len(pnl) else 0.0,
"total_cost": float(pnl["cost"].sum()) if len(pnl) else 0.0,
"blocked_trades": int(fills["blocked"].sum()) if len(fills) else 0,
},
}
manifest_path = root / "joinquant_smoke_manifest.json"
manifest["manifest_path"] = str(manifest_path)
manifest_path.write_text(
json.dumps(manifest, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
return manifest
+94
View File
@@ -0,0 +1,94 @@
"""Symbol conversion between internal A-share ids and JoinQuant ids."""
from __future__ import annotations
import re
_INTERNAL_RE = re.compile(r"^(?P<prefix>sh|sz)(?P<code>\d{6})$", re.IGNORECASE)
_JOINQUANT_RE = re.compile(
r"^(?P<code>\d{6})\.(?P<exchange>XSHG|XSHE)$", re.IGNORECASE
)
_BARE_RE = re.compile(r"^\d{6}$")
def _validate_internal(prefix: str, code: str) -> None:
if prefix == "sh" and code.startswith("6"):
return
if prefix == "sz" and code.startswith(("0", "3")):
return
raise ValueError(f"Unsupported A-share symbol: {prefix}{code}")
def _validate_joinquant(code: str, exchange: str) -> None:
if exchange == "XSHG" and code.startswith("6"):
return
if exchange == "XSHE" and code.startswith(("0", "3")):
return
raise ValueError(f"Unsupported JoinQuant A-share symbol: {code}.{exchange}")
def to_joinquant_symbol(symbol_id: str) -> str:
"""Convert an internal symbol like ``sh600000`` to ``600000.XSHG``.
Args:
symbol_id: Internal A-share id with ``sh`` or ``sz`` prefix.
Returns:
JoinQuant security id.
Raises:
ValueError: If the symbol is malformed or outside supported A-share
Shanghai/Shenzhen equity prefixes.
"""
text = str(symbol_id).strip().lower()
match = _INTERNAL_RE.match(text)
if not match:
raise ValueError(f"Invalid internal symbol: {symbol_id!r}")
prefix = match.group("prefix").lower()
code = match.group("code")
_validate_internal(prefix, code)
exchange = "XSHG" if prefix == "sh" else "XSHE"
return f"{code}.{exchange}"
def from_joinquant_symbol(jq_symbol: str) -> str:
"""Convert a JoinQuant symbol like ``600000.XSHG`` to ``sh600000``."""
text = str(jq_symbol).strip().upper()
match = _JOINQUANT_RE.match(text)
if not match:
raise ValueError(f"Invalid JoinQuant symbol: {jq_symbol!r}")
code = match.group("code")
exchange = match.group("exchange").upper()
_validate_joinquant(code, exchange)
prefix = "sh" if exchange == "XSHG" else "sz"
return f"{prefix}{code}"
def normalize_symbol_pair(value: object) -> tuple[str, str]:
"""Return ``(symbol_id, jq_symbol)`` from any supported symbol spelling."""
text = str(value).strip()
if not text or text.lower() == "nan":
raise ValueError("Missing symbol")
if _INTERNAL_RE.match(text):
symbol_id = text.lower()
return symbol_id, to_joinquant_symbol(symbol_id)
if _JOINQUANT_RE.match(text):
jq_symbol = text.upper()
return from_joinquant_symbol(jq_symbol), jq_symbol
if _BARE_RE.match(text):
if text.startswith("6"):
symbol_id = f"sh{text}"
elif text.startswith(("0", "3")):
symbol_id = f"sz{text}"
else:
raise ValueError(f"Unsupported bare A-share code: {text}")
return symbol_id, to_joinquant_symbol(symbol_id)
raise ValueError(f"Unsupported symbol: {value!r}")
+271
View File
@@ -0,0 +1,271 @@
"""Generate a standalone JoinQuant wrapper strategy.
The module also defines default JoinQuant strategy hooks by executing the same
template with ``run1`` / ``target_shares`` defaults. That means this file can be
copied directly into JoinQuant for a quick smoke test, while the CLI can still
write a configured standalone file for a real run.
"""
from __future__ import annotations
import json
from pathlib import Path
from string import Template
from typing import Literal
WrapperMode = Literal["target_shares", "target_value"]
_WRAPPER_TEMPLATE = Template(
r'''# Standalone JoinQuant target wrapper generated by chinese-equity-quant.
# Copy this file and the exported daily CSV target files into JoinQuant.
#
# The file loader is isolated in _read_target_file(). The default implementation
# uses JoinQuant's read_file API for uploaded files. If your JoinQuant runtime
# allows HTTP or another storage backend, replace only that function.
import csv
import io
import json
PORTFOLIO_NAME = "${portfolio_name}"
TARGET_MODE = "${mode}"
ALLOW_SHORT = ${allow_short}
TARGET_FILE_PREFIX = "" # Optional uploaded-file prefix, for example "run1/"
_EMBEDDED_TARGETS = ${embedded_targets}
def initialize(context):
set_benchmark("000300.XSHG")
set_option("use_real_price", True)
g.portfolio_name = PORTFOLIO_NAME
g.target_mode = TARGET_MODE
g.targets_by_date = {}
run_daily(load_targets, time="before_open")
run_daily(rebalance_at_open, time="open")
run_daily(record_after_close, time="after_close")
def _today_text(context):
return context.current_dt.strftime("%Y-%m-%d")
def _today_file_name(context):
return context.current_dt.strftime("%Y%m%d") + ".csv"
def _read_target_file(file_name):
${embedded_target_read}
data = read_file(TARGET_FILE_PREFIX + file_name)
if isinstance(data, bytes):
data = data.decode("utf-8")
return data
def _load_target_rows(context):
file_name = _today_file_name(context)
text = _read_target_file(file_name)
rows = list(csv.DictReader(io.StringIO(text)))
clean_rows = []
for row in rows:
if row.get("portfolio_name") and row["portfolio_name"] != PORTFOLIO_NAME:
continue
jq_symbol = row.get("jq_symbol") or row.get("security") or row.get("symbol")
if not jq_symbol:
log.warn("Skipping target row with no jq_symbol: %s" % row)
continue
if TARGET_MODE == "target_shares":
target = int(float(row.get("target_shares") or 0))
elif TARGET_MODE == "target_value":
target = float(row.get("target_value") or 0.0)
else:
raise ValueError("Unsupported TARGET_MODE: %s" % TARGET_MODE)
if not ALLOW_SHORT and target < 0:
log.warn(
"SHORT_NOT_SUPPORTED clipping %s target from %s to 0" %
(jq_symbol, target)
)
target = 0
clean_rows.append({"jq_symbol": jq_symbol, "target": target, "raw": row})
return clean_rows
def load_targets(context):
date_text = _today_text(context)
try:
rows = _load_target_rows(context)
except Exception as exc:
log.error("Failed to load JoinQuant target file for %s: %s" % (date_text, exc))
rows = []
g.targets_by_date[date_text] = rows
log.info("JOINQUANT_TARGET_LOAD|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"target_mode": TARGET_MODE,
"n_targets": len(rows),
}, sort_keys=True))
def rebalance_at_open(context):
date_text = _today_text(context)
rows = g.targets_by_date.get(date_text, [])
target_symbols = set()
for row in rows:
security = row["jq_symbol"]
target_symbols.add(security)
if TARGET_MODE == "target_shares":
order_target(security, int(row["target"]))
else:
order_target_value(security, float(row["target"]))
log.info("JOINQUANT_ORDER_SUBMIT|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
"target_mode": TARGET_MODE,
"target": row["target"],
}, sort_keys=True))
for security in list(context.portfolio.positions.keys()):
if security not in target_symbols:
order_target(security, 0)
log.info("JOINQUANT_ORDER_CLOSE|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
}, sort_keys=True))
def _position_records(context):
records = []
cash = float(context.portfolio.available_cash)
total_value = float(context.portfolio.total_value)
for security, position in context.portfolio.positions.items():
records.append({
"date": _today_text(context),
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
"position_shares": int(position.total_amount),
"position_value": float(position.value),
"cash": cash,
"total_value": total_value,
})
return records
def _trade_records(context):
records = []
try:
trades = get_trades()
except Exception:
trades = {}
for trade_id, trade in trades.items():
amount = int(getattr(trade, "amount", 0))
price = float(getattr(trade, "price", 0.0))
security = getattr(trade, "security", "")
side = "buy" if amount >= 0 else "sell"
records.append({
"date": _today_text(context),
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
"order_id": str(getattr(trade, "order_id", trade_id)),
"side": side,
"filled_shares": amount,
"fill_price": price,
"trade_value": abs(amount * price),
"trade_cost": float(getattr(trade, "commission", 0.0)),
"raw_status": "filled",
})
return records
def record_after_close(context):
date_text = _today_text(context)
for record in _trade_records(context):
log.info("JOINQUANT_FILL|%s" % json.dumps(record, sort_keys=True))
for record in _position_records(context):
log.info("JOINQUANT_POSITION|%s" % json.dumps(record, sort_keys=True))
log.info("JOINQUANT_PNL|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"cash": float(context.portfolio.available_cash),
"total_value": float(context.portfolio.total_value),
}, sort_keys=True))
'''
)
# Make this module itself usable as a JoinQuant strategy with defaults.
exec(_WRAPPER_TEMPLATE.substitute(
portfolio_name="run1",
mode="target_shares",
allow_short="False",
embedded_targets="{}",
embedded_target_read="",
))
def render_wrapper_strategy(
*,
portfolio_name: str,
mode: WrapperMode = "target_shares",
allow_short: bool = False,
embedded_targets: dict[str, str] | None = None,
) -> str:
"""Render the standalone JoinQuant wrapper strategy source."""
if mode not in {"target_shares", "target_value"}:
raise ValueError("mode must be 'target_shares' or 'target_value'")
embedded_targets = embedded_targets or {}
embedded_target_read = ""
if embedded_targets:
embedded_target_read = (
" if file_name in _EMBEDDED_TARGETS:\n"
" return _EMBEDDED_TARGETS[file_name]\n"
)
return _WRAPPER_TEMPLATE.substitute(
portfolio_name=portfolio_name,
mode=mode,
allow_short="True" if allow_short else "False",
embedded_targets=json.dumps(embedded_targets, ensure_ascii=False, indent=4),
embedded_target_read=embedded_target_read,
)
def _load_embedded_targets(targets_dir: str | Path | None) -> dict[str, str]:
if targets_dir is None:
return {}
root = Path(targets_dir)
targets = {
path.name: path.read_text(encoding="utf-8")
for path in sorted(root.glob("*.csv"))
}
if not targets:
raise ValueError(f"No CSV target files found under {root}")
return targets
def write_wrapper_strategy(
*,
portfolio_name: str,
mode: WrapperMode = "target_shares",
out_path: str | Path,
allow_short: bool = False,
embedded_targets_dir: str | Path | None = None,
) -> Path:
"""Write a configured standalone JoinQuant wrapper strategy."""
path = Path(out_path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
render_wrapper_strategy(
portfolio_name=portfolio_name,
mode=mode,
allow_short=allow_short,
embedded_targets=_load_embedded_targets(embedded_targets_dir),
),
encoding="utf-8",
)
return path
+32 -1
View File
@@ -4,7 +4,6 @@ version = "0.1.0"
description = "A modular Chinese A-share quant research framework (daily frequency)."
requires-python = ">=3.10"
dependencies = [
"backtrader>=1.9.76.123",
"akshare>=1.14.0",
"baostock>=0.8.8",
"pandas>=2.0.0",
@@ -13,10 +12,42 @@ dependencies = [
"pyarrow>=14.0.0",
]
[project.optional-dependencies]
backtrader = [
"backtrader>=1.9.76.123",
]
joinquant-browser = [
"playwright>=1.61.0",
]
[dependency-groups]
dev = [
"coverage>=7.14.1",
"pytest>=7.0.0",
]
[tool.uv]
package = false
[tool.pytest.ini_options]
markers = [
"network: tests that call live external data providers and are skipped unless explicitly enabled",
]
[tool.coverage.run]
branch = true
relative_files = true
source = [
".",
]
[tool.coverage.report]
fail_under = 95
show_missing = true
skip_covered = false
omit = [
"tests/*",
"docs/*",
"scripts/*",
".venv/*",
]
File diff suppressed because it is too large Load Diff
+40
View File
@@ -0,0 +1,40 @@
"""Run pytest under coverage in this environment.
The plain ``coverage run -m pytest`` path reloads NumPy under the current VS Code
Python startup environment, which breaks pandas/numpy reductions. Import NumPy
before starting coverage so the measured test run uses one stable NumPy module.
"""
from __future__ import annotations
import sys
import numpy # noqa: F401
import coverage
import pytest
def main(argv: list[str]) -> int:
pytest_args = argv or ["tests/", "-v"]
cov = coverage.Coverage(config_file=True)
cov.erase()
cov.start()
test_status = pytest.main(pytest_args)
cov.stop()
cov.save()
if test_status != 0:
return int(test_status)
total = cov.report()
print(f"\nCoverage total: {total:.2f}%")
fail_under = float(cov.config.fail_under)
if total < fail_under:
print(f"Coverage failure: total {total:.2f}% is below {fail_under:.2f}%")
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# End-to-end run of the outlier-robust reversal_rank alpha on the full
# all-universe dataset and on a per-date liquid subset. Records per-phase
# wall-clock time to reports/reversal_rank_timings.json.
set -euo pipefail
cd "$(dirname "$0")/.."
DATA=data/daily_bars/all
BOOK=10000000
TIMINGS=reports/reversal_rank_timings.json
mkdir -p reports
echo "{" > "$TIMINGS"
run() { # run <json_key> <cmd...>
local key="$1"; shift
local t0 t1
t0=$(date +%s.%N)
"$@"
t1=$(date +%s.%N)
printf ' "%s": %.2f,\n' "$key" "$(echo "$t1 - $t0" | bc)" >> "$TIMINGS"
echo ">>> $key took $(echo "$t1 - $t0" | bc)s"
}
# ---- full all-universe, robust rank weighting ----
run full_alpha_compute uv run python cli.py alpha compute --data-path "$DATA" \
--alpha-name reversal_rank_all --alpha-type reversal_rank --lookback 5 --output-dir alphas
run full_alpha_eval uv run python cli.py alpha eval \
--alpha-path alphas/reversal_rank_all.pq --data-path "$DATA"
run full_combo uv run python cli.py combo combine \
--alpha-paths alphas/reversal_rank_all.pq --combo-name reversal_rank_all_combo \
--method equal_weight --output-dir combos
run full_portfolio_build uv run python cli.py portfolio build \
--weights-path combos/reversal_rank_all_combo.pq --data-path "$DATA" \
--booksize "$BOOK" --portfolio-name reversal_rank_all_10m --output-dir portfolio
run full_portfolio_eval uv run python cli.py portfolio eval \
--positions-path portfolio/reversal_rank_all_10m.pq --data-path "$DATA"
run full_portfolio_simulate uv run python cli.py portfolio simulate \
--positions-path portfolio/reversal_rank_all_10m.pq --data-path "$DATA" \
--constraint suspension --constraint price_limit --constraint volume_cap \
--cost-bps 5 --slippage-bps 5 --output-dir portfolio
# ---- liquid subset (per-date investable universe), robust rank weighting ----
run liq_alpha_compute uv run python cli.py alpha compute --data-path "$DATA" \
--alpha-name reversal_rank_liq --alpha-type reversal_rank --lookback 5 \
--liquid-universe --universe-top-n 1000 --output-dir alphas
run liq_alpha_eval uv run python cli.py alpha eval \
--alpha-path alphas/reversal_rank_liq.pq --data-path "$DATA"
run liq_combo uv run python cli.py combo combine \
--alpha-paths alphas/reversal_rank_liq.pq --combo-name reversal_rank_liq_combo \
--method equal_weight --output-dir combos
run liq_portfolio_build uv run python cli.py portfolio build \
--weights-path combos/reversal_rank_liq_combo.pq --data-path "$DATA" \
--booksize "$BOOK" --portfolio-name reversal_rank_liq_10m --output-dir portfolio
run liq_portfolio_eval uv run python cli.py portfolio eval \
--positions-path portfolio/reversal_rank_liq_10m.pq --data-path "$DATA"
run liq_portfolio_simulate uv run python cli.py portfolio simulate \
--positions-path portfolio/reversal_rank_liq_10m.pq --data-path "$DATA" \
--constraint suspension --constraint price_limit --constraint volume_cap \
--cost-bps 5 --slippage-bps 5 --output-dir portfolio
printf ' "_done": true\n}\n' >> "$TIMINGS"
echo "Wrote $TIMINGS"
Binary file not shown.
+261
View File
@@ -0,0 +1,261 @@
"""Shared deterministic test data for offline workflow tests."""
from __future__ import annotations
import numpy as np
import pandas as pd
from pipeline.common.schema import (
ALPHA_COLUMNS,
COMBO_COLUMNS,
DATA_COLUMNS,
MINUTE_BAR_COLUMNS,
)
GENERATED_SYMBOLS: tuple[str, ...] = (
"sh600000",
"sz000001",
"sh600519",
"sz300750",
)
GENERATED_SYMBOL_NAMES: dict[str, str] = {
"sh600000": "PF Bank",
"sz000001": "Ping An Bank",
"sh600519": "Kweichow Moutai",
"sz300750": "CATL",
}
def generated_sessions(n_sessions: int = 12) -> pd.DatetimeIndex:
"""Return a fixed business-day calendar used by generated fixtures."""
return pd.bdate_range("2024-01-02", periods=n_sessions)
def make_generated_daily_bars(
n_sessions: int = 12,
include_missing: bool = True,
) -> pd.DataFrame:
"""Build daily bars with explicit edge cases and no randomness.
The panel covers four A-share symbols and includes a suspended row, an ST
flag, a zero-volume row, a missing symbol-date, and limit-style open/close
moves. Values are deterministic so tests can assert exact identities.
"""
dates = generated_sessions(n_sessions)
base_close = {
"sh600000": 10.00,
"sz000001": 15.00,
"sh600519": 1200.00,
"sz300750": 180.00,
}
returns = {
"sh600000": [0.000, 0.012, -0.006, 0.018, 0.100, -0.014, 0.006, 0.000, 0.008, -0.011, 0.004, 0.009],
"sz000001": [0.000, -0.008, 0.011, -0.004, 0.006, 0.000, -0.012, 0.009, 0.005, -0.007, 0.010, -0.003],
"sh600519": [0.000, 0.006, 0.004, -0.010, 0.012, -0.006, 0.005, 0.003, -0.009, 0.007, -0.004, 0.006],
"sz300750": [0.000, -0.010, 0.014, 0.006, -0.008, 0.011, -0.004, 0.009, -0.200, 0.012, -0.006, 0.008],
}
base_volume = {
"sh600000": 1_200_000.0,
"sz000001": 900_000.0,
"sh600519": 80_000.0,
"sz300750": 240_000.0,
}
rows: list[dict[str, object]] = []
for sym in GENERATED_SYMBOLS:
closes = [base_close[sym]]
pattern = returns[sym]
for step in range(1, n_sessions):
ret = pattern[step % len(pattern)]
closes.append(closes[-1] * (1.0 + ret))
closes_arr = np.asarray(closes, dtype=float)
precloses = np.concatenate([[closes_arr[0]], closes_arr[:-1]])
for i, date in enumerate(dates):
preclose = float(precloses[i])
close = float(closes_arr[i])
open_price = preclose * (1.0 + 0.25 * (close / preclose - 1.0))
high = max(open_price, close) * 1.01
low = min(open_price, close) * 0.99
volume = base_volume[sym] + 10_000.0 * i
tradestatus = 1
is_st = 0
if sym == "sh600000" and i == 4:
open_price = preclose * 1.10
close = open_price
high = open_price
low = open_price
if sym == "sh600000" and i == 7:
volume = 0.0
if sym == "sz000001" and i == 5:
open_price = preclose
close = preclose
high = preclose
low = preclose
volume = 0.0
tradestatus = 0
if sym == "sh600519" and i == 7:
is_st = 1
if sym == "sz300750" and i == 8:
open_price = preclose * 0.80
close = open_price
high = open_price
low = open_price
amount = volume * ((open_price + close) / 2.0)
vwap = amount / volume if volume > 0 else np.nan
pct_chg = (close / preclose - 1.0) * 100.0 if preclose else 0.0
rows.append({
"symbol_id": sym,
"symbol_name": GENERATED_SYMBOL_NAMES[sym],
"date": date,
"open": open_price,
"high": high,
"low": low,
"close": close,
"preclose": preclose,
"volume": volume,
"amount": amount,
"vwap": vwap,
"turn": volume / 1_000_000.0,
"pctChg": pct_chg,
"tradestatus": tradestatus,
"isST": is_st,
"peTTM": 8.0 + i,
"pbMRQ": 1.0 + 0.05 * i,
"psTTM": 2.0 + 0.03 * i,
"pcfNcfTTM": 5.0 + 0.1 * i,
})
result = pd.DataFrame(rows)
if include_missing and n_sessions > 6:
missing_mask = (
(result["symbol_id"] == "sz300750")
& (result["date"] == dates[6])
)
result = result.loc[~missing_mask].copy()
result = result[DATA_COLUMNS]
return result.sort_values(["date", "symbol_id"]).reset_index(drop=True)
def make_generated_minute_bars(
daily: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Expand generated daily bars into a tiny deterministic intraday panel."""
daily = make_generated_daily_bars() if daily is None else daily.copy()
rows: list[dict[str, object]] = []
bar_times = ("09:35:00", "10:30:00", "14:55:00")
for daily_row in daily.sort_values(["date", "symbol_id"]).itertuples(index=False):
if int(getattr(daily_row, "tradestatus", 1)) == 0:
continue
volume = float(daily_row.volume)
volume_slices = [0.25 * volume, 0.35 * volume, 0.40 * volume]
prices = np.linspace(float(daily_row.open), float(daily_row.close), len(bar_times))
for j, time_text in enumerate(bar_times):
dt = pd.Timestamp(daily_row.date) + pd.Timedelta(time_text)
open_price = prices[j - 1] if j else float(daily_row.open)
close_price = float(prices[j])
high = max(open_price, close_price) * 1.002
low = min(open_price, close_price) * 0.998
minute_volume = float(volume_slices[j])
amount = minute_volume * ((open_price + close_price) / 2.0)
rows.append({
"symbol_id": daily_row.symbol_id,
"symbol_name": daily_row.symbol_name,
"datetime": dt,
"date": pd.Timestamp(daily_row.date).normalize(),
"time": time_text,
"frequency": "5m",
"open": open_price,
"high": high,
"low": low,
"close": close_price,
"volume": minute_volume,
"amount": amount,
"vwap": amount / minute_volume if minute_volume > 0 else np.nan,
"adjustflag": "3",
})
return pd.DataFrame(rows, columns=MINUTE_BAR_COLUMNS)
def make_generated_derived_features(
daily: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Return numeric daily derived values, including NaN and infinity cells."""
daily = make_generated_daily_bars() if daily is None else daily.copy()
keys = (
daily[["symbol_id", "date"]]
.drop_duplicates()
.sort_values(["date", "symbol_id"])
.reset_index(drop=True)
)
date_rank = keys["date"].rank(method="dense").astype(float)
symbol_rank = keys["symbol_id"].map({
"sh600000": 1.0,
"sz000001": 2.0,
"sh600519": 3.0,
"sz300750": 4.0,
})
out = keys.copy()
out["toy_feature"] = symbol_rank + date_rank / 100.0
out["finite_feature"] = symbol_rank * date_rank
out["nan_feature"] = out["toy_feature"]
out["inf_feature"] = out["toy_feature"]
if len(out) >= 2:
out.loc[out.index[0], "nan_feature"] = np.nan
out.loc[out.index[1], "inf_feature"] = np.inf
return out
def make_generated_alpha_weights(
alpha_name: str = "alpha_a",
*,
scale: float = 1.0,
offset: float = 0.0,
zero_date_index: int | None = None,
n_sessions: int = 10,
) -> pd.DataFrame:
"""Create a deterministic long alpha grid with optional zero-gross date."""
dates = generated_sessions(n_sessions)
even = np.array([1.20, -0.80, 0.40, -0.80], dtype=float)
odd = np.array([-0.60, 1.10, -0.90, 0.40], dtype=float)
rows: list[dict[str, object]] = []
for i, date in enumerate(dates):
vector = even.copy() if i % 2 == 0 else odd.copy()
vector = vector + offset
vector = vector - vector.mean()
if zero_date_index is not None and i == zero_date_index:
vector = np.zeros_like(vector)
for sym, weight in zip(GENERATED_SYMBOLS, scale * vector):
rows.append({
"symbol_id": sym,
"date": date,
"alpha_name": alpha_name,
"weight": float(weight),
})
result = pd.DataFrame(rows, columns=ALPHA_COLUMNS)
return result.sort_values(["symbol_id", "date"]).reset_index(drop=True)
def make_generated_combo_weights(
combo_name: str = "combo",
*,
zero_date_index: int | None = 2,
n_sessions: int = 10,
) -> pd.DataFrame:
"""Create deterministic combo weights for portfolio construction tests."""
alpha = make_generated_alpha_weights(
"combo_source",
zero_date_index=zero_date_index,
n_sessions=n_sessions,
)
combo = alpha.rename(columns={"alpha_name": "combo_name"}).copy()
combo["combo_name"] = combo_name
return combo[COMBO_COLUMNS].sort_values(["symbol_id", "date"]).reset_index(drop=True)
+407 -4
View File
@@ -6,7 +6,11 @@ import pandas as pd
import pytest
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.compute import compute_alpha, evaluate_alpha
from pipeline.alpha.compute import (
compute_alpha,
evaluate_alpha,
investable_universe_mask,
)
from pipeline.alpha.registry import (
available_alphas,
get_alpha,
@@ -51,7 +55,7 @@ def test_reversal_sign_matches_negative_trailing_return():
data = _make_data()
alpha = compute_alpha(data, "rev5", "reversal", lookback=5)
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
raw = -close.pct_change(5)
raw = -close.pct_change(5, fill_method=None)
last = raw.index[-1]
expected_top = raw.loc[last].idxmax()
got = alpha[alpha["date"] == last].set_index("symbol_id")["weight"].idxmax()
@@ -74,6 +78,83 @@ def test_evaluate_alpha_keys():
assert key in metrics
def test_evaluate_alpha_uses_next_open_to_next_open_returns():
dates = pd.date_range("2024-01-01", periods=5)
data = pd.concat([
pd.DataFrame({
"symbol_id": "sh600000",
"symbol_name": "sh600000",
"date": dates,
"open": [100.0, 100.0, 100.0, 100.0, 200.0],
"high": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
"low": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
"close": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
"volume": 1_000.0,
"amount": 1_000.0,
}),
pd.DataFrame({
"symbol_id": "sz000001",
"symbol_name": "sz000001",
"date": dates,
"open": [100.0, 100.0, 100.0, 200.0, 200.0],
"high": [100.0, 10.0, 10.0, 10.0, 10.0],
"low": [100.0, 10.0, 10.0, 10.0, 10.0],
"close": [100.0, 10.0, 10.0, 10.0, 10.0],
"volume": 1_000.0,
"amount": 1_000.0,
}),
], ignore_index=True)
alpha = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[1], dates[1], dates[2], dates[2]],
"alpha_name": ["toy"] * 4,
"weight": [-1.0, 1.0, 1.0, -1.0],
})
metrics = evaluate_alpha(alpha, data)
assert metrics["n_dates"] == 2
assert np.isclose(metrics["cumulative_return"], 1.25)
def test_evaluate_alpha_excludes_signal_without_forward_return():
dates = pd.date_range("2024-01-01", periods=3)
data = pd.concat([
pd.DataFrame({
"symbol_id": "sh600000",
"symbol_name": "sh600000",
"date": dates,
"open": [100.0, 100.0, 200.0],
"high": [100.0, 100.0, 200.0],
"low": [100.0, 100.0, 200.0],
"close": [100.0, 100.0, 200.0],
"volume": 1_000.0,
"amount": 1_000.0,
}),
pd.DataFrame({
"symbol_id": "sz000001",
"symbol_name": "sz000001",
"date": dates,
"open": [100.0, 100.0, 100.0],
"high": [100.0, 100.0, 100.0],
"low": [100.0, 100.0, 100.0],
"close": [100.0, 100.0, 100.0],
"volume": 1_000.0,
"amount": 1_000.0,
}),
], ignore_index=True)
alpha = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[0], dates[0], dates[1], dates[1]],
"alpha_name": ["toy"] * 4,
"weight": [1.0, -1.0, -1.0, 1.0],
})
metrics = evaluate_alpha(alpha, data)
assert metrics["n_dates"] == 1
def test_equal_weight_is_mean_of_alphas():
data = _make_data()
a = compute_alpha(data, "rev", "reversal", lookback=5)
@@ -95,10 +176,34 @@ def test_combine_alphas_schema(tmp_path):
assert (combo["combo_name"] == "eq").all()
def test_combine_single_alpha_is_identity(tmp_path):
data = _make_data()
a = compute_alpha(data, "rev", "reversal", lookback=5)
a_path = tmp_path / "a.pq"
a.to_parquet(a_path, index=False)
combo = combine_alphas([str(a_path)], "rev_combo", method="equal_weight")
expected = a[["symbol_id", "date", "weight"]].reset_index(drop=True)
got = combo[["symbol_id", "date", "weight"]].reset_index(drop=True)
pd.testing.assert_frame_equal(got, expected)
assert list(combo.columns) == COMBO_COLUMNS
assert (combo["combo_name"] == "rev_combo").all()
def test_combine_alphas_rejects_unknown_method(tmp_path):
data = _make_data()
alpha_path = tmp_path / "alpha.pq"
compute_alpha(data, "rev", "reversal", lookback=5).to_parquet(alpha_path, index=False)
with pytest.raises(ValueError, match="Unknown combo method"):
combine_alphas([str(alpha_path)], "bad_combo", method="does_not_exist")
# --- registry / factory -----------------------------------------------------
def test_builtins_are_registered():
assert {"reversal", "reversal_vol", "momentum"} <= set(available_alphas())
assert {"reversal", "reversal_vol", "momentum", "reversal_rank"} <= set(available_alphas())
def test_get_alpha_filters_unaccepted_params():
@@ -109,6 +214,22 @@ def test_get_alpha_filters_unaccepted_params():
assert not hasattr(alpha, "vol_window")
def test_get_alpha_forwards_kwargs_to_flexible_alpha():
@register_alpha
class _FlexibleAlpha(BaseAlpha):
name = "_flexible_alpha_kwargs"
def __init__(self, **kwargs):
self.kwargs = kwargs
def signal(self, close):
return close
alpha = get_alpha("_flexible_alpha_kwargs", decay=0.5, label="demo")
assert alpha.kwargs == {"decay": 0.5, "label": "demo"}
def test_get_alpha_unknown_raises():
with pytest.raises(KeyError):
get_alpha("does_not_exist")
@@ -131,6 +252,31 @@ def test_register_rejects_non_basealpha():
register_alpha(object) # type: ignore[arg-type]
def test_register_rejects_empty_alpha_name():
with pytest.raises(ValueError, match="non-empty"):
@register_alpha
class NoNameAlpha(BaseAlpha):
def signal(self, close):
return close
def test_load_alpha_module_error_paths(tmp_path, monkeypatch):
missing_path = tmp_path / "missing_alpha.py"
with pytest.raises(FileNotFoundError):
load_alpha_module(str(missing_path))
bad_path = tmp_path / "bad_alpha.py"
bad_path.write_text("x = 1\n")
monkeypatch.setattr(
"pipeline.alpha.registry.importlib.util.spec_from_file_location",
lambda *args, **kwargs: None,
)
with pytest.raises(ImportError, match="Cannot load alpha module"):
load_alpha_module(str(bad_path))
load_alpha_module("math")
# --- base class --------------------------------------------------------------
def test_to_weights_are_per_date_zscore():
@@ -146,6 +292,20 @@ def test_to_weights_are_per_date_zscore():
assert (weights.mean(axis=1).abs() < 1e-9).all()
def test_base_alpha_default_signal_and_repr():
alpha = BaseAlpha()
alpha.example = 3
with pytest.raises(NotImplementedError, match="signal"):
alpha.signal(pd.DataFrame({"x": [1.0]}))
with pytest.raises(NotImplementedError, match="signal"):
alpha.signal_from_data(
pd.DataFrame({"symbol_id": ["sh600000"]}),
pd.DataFrame({"sh600000": [1.0]}),
)
assert repr(alpha) == "BaseAlpha(example=3)"
# --- external plugin loading -------------------------------------------------
def test_load_external_alpha_module(tmp_path):
@@ -163,7 +323,7 @@ def test_load_external_alpha_module(tmp_path):
self.span = span
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
return -close.pct_change(self.span)
return -close.pct_change(self.span, fill_method=None)
'''))
load_alpha_module(str(module_path))
@@ -178,3 +338,246 @@ def test_load_external_alpha_module(tmp_path):
assert list(result.columns) == ALPHA_COLUMNS
assert (result["alpha_name"] == "ext").all()
# --- rank reversal + investable universe filter ------------------------------
def _make_rich_data(n_days: int = 70, symbols=("sh600000", "sz000001", "sh600519", "sz300750")):
"""Long-format data with the columns the universe filter needs."""
dates = pd.date_range("2024-01-01", periods=n_days)
rng = np.random.default_rng(1)
frames = []
for i, sym in enumerate(symbols):
close = 100.0 + i * 5 + np.cumsum(rng.standard_normal(n_days))
frames.append(pd.DataFrame({
"symbol_id": sym,
"symbol_name": sym,
"date": dates,
"open": close, "high": close, "low": close, "close": close,
"volume": 1_000.0,
"amount": (1_000.0 + i * 5_000.0) * close, # higher i = more liquid
"isST": 0,
"tradestatus": 1,
}))
return pd.concat(frames, ignore_index=True)
def test_reversal_rank_registered_and_bounded():
data = _make_data(n_days=30)
alpha = compute_alpha(data, "rr", "reversal_rank", lookback=5)
assert list(alpha.columns) == ALPHA_COLUMNS
# Rank-demeaned weights are per-date zero-mean and bounded by the
# cross-section size, never blowing up the way a z-score outlier can.
per_date_mean = alpha.groupby("date")["weight"].mean().abs()
assert (per_date_mean < 1e-9).all()
assert alpha["weight"].abs().max() <= len(data["symbol_id"].unique())
def test_investable_universe_mask_excludes_st_and_suspended():
data = _make_rich_data()
# Flag one name ST throughout, and suspend another on the last date.
data.loc[data["symbol_id"] == "sh600000", "isST"] = 1
last = data["date"].max()
data.loc[(data["symbol_id"] == "sz000001") & (data["date"] == last), "tradestatus"] = 0
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
mask = investable_universe_mask(data, close, top_n=10, min_history=5)
assert not mask["sh600000"].any() # ST excluded on every date
assert not bool(mask.loc[last, "sz000001"]) # suspended on the last date
assert bool(mask.loc[last, "sh600519"]) # a normal name stays investable
def test_compute_alpha_universe_filter_zeros_excluded_names():
data = _make_rich_data()
data.loc[data["symbol_id"] == "sh600000", "isST"] = 1
alpha = compute_alpha(
data, "rr_liq", "reversal_rank", lookback=5,
universe={"top_n": 10, "min_history": 5},
)
# The ST name is never held; an investable name is.
st_w = alpha.loc[alpha["symbol_id"] == "sh600000", "weight"]
assert (st_w.fillna(0.0) == 0.0).all()
assert alpha.loc[alpha["symbol_id"] == "sz300750", "weight"].abs().sum() > 0.0
def test_universe_filter_does_not_corrupt_signal_history():
# Masking happens on the signal, not the price history, so weights on
# investable names match the unfiltered weights restricted to that set.
data = _make_rich_data()
universe = {"top_n": 2, "min_history": 5} # keep only the 2 most liquid names
filtered = compute_alpha(data, "f", "reversal_rank", lookback=5, universe=universe)
held = set(filtered.loc[filtered["weight"] != 0.0, "symbol_id"].unique())
# The two most liquid names (highest amount) are sh600519, sz300750.
assert held == {"sh600519", "sz300750"}
# --- feature-aware alpha integration ----------------------------------------
def test_compute_alpha_without_feature_path_matches_empty_feature_paths():
data = _make_data()
base = compute_alpha(data, "rev5", "reversal", lookback=5)
with_empty_features = compute_alpha(
data,
"rev5",
"reversal",
lookback=5,
feature_paths=[],
)
pd.testing.assert_frame_equal(base, with_empty_features)
def test_feature_aware_alpha_reads_joined_feature_column(tmp_path):
module_path = tmp_path / "feature_aware_alpha.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class FeatureAwareAlpha(BaseAlpha):
name = "feature_aware_test_alpha"
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
signal = data.pivot_table(
index="date",
columns="symbol_id",
values="toy_feature",
aggfunc="first",
)
return signal.reindex(index=close.index, columns=close.columns)
'''))
data = _make_data()
feature = data[["symbol_id", "date"]].copy()
feature["toy_feature"] = feature["symbol_id"].map({
"sh600000": 1.0,
"sz000001": 2.0,
"sh600519": 3.0,
})
feature_path = tmp_path / "toy_feature.pq"
feature.to_parquet(feature_path, index=False)
load_alpha_module(str(module_path))
result = compute_alpha(
data,
"feature_run",
"feature_aware_test_alpha",
feature_paths=[str(feature_path)],
)
assert list(result.columns) == ALPHA_COLUMNS
assert (result["alpha_name"] == "feature_run").all()
last = result[result["date"] == result["date"].max()]
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
def test_feature_paths_join_multiple_files_and_normalize_dates(tmp_path):
module_path = tmp_path / "multi_feature_alpha.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class MultiFeatureAlpha(BaseAlpha):
name = "multi_feature_test_alpha"
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
data = data.copy()
data["combined_feature"] = data["toy_a"] + data["toy_b"]
signal = data.pivot_table(
index="date",
columns="symbol_id",
values="combined_feature",
aggfunc="first",
)
return signal.reindex(index=close.index, columns=close.columns)
'''))
data = _make_data(n_days=8)
symbol_score = {"sh600000": 1.0, "sz000001": 2.0, "sh600519": 3.0}
feature_a = data[["symbol_id", "date"]].copy()
feature_a["date"] = feature_a["date"] + pd.Timedelta(hours=15)
feature_a["toy_a"] = feature_a["symbol_id"].map(symbol_score)
feature_b = data[["symbol_id", "date"]].copy()
feature_b["date"] = feature_b["date"].dt.strftime("%Y-%m-%d 09:30:00")
feature_b["toy_b"] = feature_b["symbol_id"].map(symbol_score) * 10.0
feature_a_path = tmp_path / "toy_a.pq"
feature_b_path = tmp_path / "toy_b.pq"
feature_a.to_parquet(feature_a_path, index=False)
feature_b.to_parquet(feature_b_path, index=False)
load_alpha_module(str(module_path))
result = compute_alpha(
data,
"multi_feature_run",
"multi_feature_test_alpha",
feature_paths=[str(feature_a_path), str(feature_b_path)],
)
assert list(result.columns) == ALPHA_COLUMNS
assert (result["alpha_name"] == "multi_feature_run").all()
last = result[result["date"] == result["date"].max()]
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
def test_compute_alpha_rejects_duplicate_feature_frame_columns():
data = _make_data()
duplicate_columns = pd.DataFrame(
[["sh600000", pd.Timestamp("2024-01-01"), 1.0, 2.0]],
columns=["symbol_id", "date", "toy_feature", "toy_feature"],
)
with pytest.raises(ValueError, match="duplicate columns"):
compute_alpha(
data,
"bad_features",
"reversal",
feature_frames=[duplicate_columns],
)
def test_compute_alpha_rejects_feature_path_collision_with_daily_data(tmp_path):
data = _make_data()
close_collision = data[["symbol_id", "date"]].copy()
close_collision["close"] = 1.0
close_collision_path = tmp_path / "close_collision.pq"
close_collision.to_parquet(close_collision_path, index=False)
with pytest.raises(ValueError, match="conflict"):
compute_alpha(
data,
"close_collision",
"reversal",
feature_paths=[str(close_collision_path)],
)
def test_evaluate_alpha_empty_when_signal_dates_not_on_market_calendar():
data = _make_data(n_days=3)
alpha = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2030-01-01")],
"alpha_name": ["future"],
"weight": [1.0],
})
metrics = evaluate_alpha(alpha, data)
assert metrics["n_dates"] == 0
assert metrics["cumulative_return"] == 0.0
+771
View File
@@ -0,0 +1,771 @@
"""CLI handoff tests for the offline daily workflow."""
from __future__ import annotations
import textwrap
from pathlib import Path
import click
import pandas as pd
from click.testing import CliRunner
from cli import cli
import pipeline.derived.cli as derived_cli
import pipeline.features.cli as features_cli
from tests.helpers import (
make_generated_daily_bars,
make_generated_derived_features,
make_generated_minute_bars,
)
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
def _invoke_ok(runner: CliRunner, args: list[str]):
result = runner.invoke(cli, args)
assert result.exit_code == 0, result.output
return result
def _invoke_error(runner: CliRunner, args: list[str]):
result = runner.invoke(cli, args)
assert result.exit_code != 0, result.output
return result
def test_cli_daily_workflow_handoffs_stay_in_tmp_path(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars()
minute_bars = make_generated_minute_bars(daily_bars)
derived_features = make_generated_derived_features(daily_bars)
daily_path = tmp_path / "daily_bars.pq"
minute_path = tmp_path / "minute_bars.pq"
derived_input_path = tmp_path / "derived_input.pq"
daily_bars.to_parquet(daily_path, index=False)
minute_bars.to_parquet(minute_path, index=False)
derived_features.to_parquet(derived_input_path, index=False)
ingest_dir = tmp_path / "derived_ingested"
ingest_result = _invoke_ok(runner, [
"derived", "ingest",
"--input-path", str(derived_input_path),
"--derived-name", "toy_features",
"--output-dir", str(ingest_dir),
])
ingested_feature_path = ingest_dir / "toy_features.pq"
assert "Saved derived data:" in ingest_result.output
assert ingested_feature_path.exists()
validate_result = _invoke_ok(runner, [
"derived", "validate",
"--input-path", str(ingested_feature_path),
])
assert "Valid derived data:" in validate_result.output
assert "rows" in validate_result.output
computed_derived_dir = tmp_path / "derived_computed"
derived_compute_result = _invoke_ok(runner, [
"derived", "compute",
"--daily-path", str(daily_path),
"--minute-path", str(minute_path),
"--derived-type", "minute_daily_summary",
"--derived-name", "minute_summary",
"--output-dir", str(computed_derived_dir),
])
minute_summary_path = computed_derived_dir / "minute_summary.pq"
assert "Loaded daily data:" in derived_compute_result.output
assert "Loaded minute bars:" in derived_compute_result.output
assert "Saved derived data:" in derived_compute_result.output
assert minute_summary_path.exists()
assert "minute_vwap" in pd.read_parquet(minute_summary_path).columns
alpha_module_path = tmp_path / "cli_feature_alpha.py"
alpha_module_path.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class CliFeatureAlpha(BaseAlpha):
name = "cli_feature_alpha_workflow"
def __init__(self, **kwargs):
self.kwargs = kwargs
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
signal = data.pivot_table(
index="date",
columns="symbol_id",
values="minute_intraday_return",
aggfunc="first",
)
fallback = close.pct_change(1, fill_method=None)
feature_signal = signal.reindex(index=close.index, columns=close.columns)
toy_signal = data.pivot_table(
index="date",
columns="symbol_id",
values="toy_feature",
aggfunc="first",
)
toy_signal = toy_signal.reindex(index=close.index, columns=close.columns)
return feature_signal.fillna(fallback) + toy_signal / 1000.0
"""))
alpha_dir = tmp_path / "alphas"
alpha_result = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--feature-path", str(minute_summary_path),
"--feature-path", str(ingested_feature_path),
"--alpha-module", str(alpha_module_path),
"--alpha-type", "cli_feature_alpha_workflow",
"--alpha-name", "cli_feature_alpha",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "cli_feature_alpha.pq"
assert "Loaded data:" in alpha_result.output
assert "Saved alpha:" in alpha_result.output
assert "Weight stats" in alpha_result.output
assert alpha_path.exists()
assert not pd.read_parquet(alpha_path).empty
alpha_report_dir = tmp_path / "alpha_reports"
alpha_eval_result = _invoke_ok(runner, [
"alpha", "eval",
"--alpha-path", str(alpha_path),
"--data-path", str(daily_path),
"--report-dir", str(alpha_report_dir),
])
alpha_report_path = alpha_report_dir / "cli_feature_alpha_eval.json"
assert "ALPHA EVALUATION" in alpha_eval_result.output
assert "Report saved:" in alpha_eval_result.output
assert alpha_report_path.exists()
combo_dir = tmp_path / "combos"
combo_result = _invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", f"{alpha_path},{alpha_path}",
"--combo-name", "cli_combo",
"--method", "equal_weight",
"--output-dir", str(combo_dir),
])
combo_path = combo_dir / "cli_combo.pq"
assert "Saved combo:" in combo_result.output
assert "Weight stats" in combo_result.output
assert combo_path.exists()
portfolio_dir = tmp_path / "portfolio"
build_result = _invoke_ok(runner, [
"portfolio", "build",
"--weights-path", str(combo_path),
"--data-path", str(daily_path),
"--booksize", "2000000",
"--portfolio-name", "cli_portfolio",
"--output-dir", str(portfolio_dir),
])
positions_path = portfolio_dir / "cli_portfolio.pq"
assert "Saved positions:" in build_result.output
assert "Gross exposure" in build_result.output
assert positions_path.exists()
execution_dir = tmp_path / "execution"
simulate_result = _invoke_ok(runner, [
"portfolio", "simulate",
"--positions-path", str(positions_path),
"--data-path", str(daily_path),
"--constraint", "suspension",
"--constraint", "price_limit",
"--constraint", "volume_cap",
"--cost-bps", "5",
"--slippage-bps", "5",
"--volume-frac", "0.02",
"--output-dir", str(execution_dir),
])
fills_path = execution_dir / "fills" / "cli_portfolio.pq"
pnl_path = execution_dir / "pnl" / "cli_portfolio.pq"
assert "Saved fills:" in simulate_result.output
assert "Saved pnl:" in simulate_result.output
assert "Total PnL:" in simulate_result.output
assert fills_path.exists()
assert pnl_path.exists()
eval_result = _invoke_ok(runner, [
"portfolio", "eval",
"--positions-path", str(positions_path),
"--data-path", str(daily_path),
])
assert "Research-portfolio metrics:" in eval_result.output
assert "cumulative_return" in eval_result.output
assert "fitness" in eval_result.output
pqcat_result = _invoke_ok(runner, [
"pqcat",
str(positions_path),
"--info",
])
assert "shape:" in pqcat_result.output
assert "dtypes:" in pqcat_result.output
assert "position_shares" in pqcat_result.output
alphaview_result = _invoke_ok(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(alpha_path),
"--symbol", "sh600000",
"--start-date", "2024-01-02",
"--end-date", "2024-01-12",
"--columns", "close,volume",
])
assert "symbol: sh600000" in alphaview_result.output
assert "cli_feature_alpha" in alphaview_result.output
def test_cli_pipeline_accepts_partitioned_daily_dataset(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(include_missing=False)
dataset_dir = tmp_path / "daily_dataset"
dataset_frame = daily_bars.copy()
dataset_frame["month"] = dataset_frame["date"].dt.strftime("%Y-%m")
dataset_frame.to_parquet(dataset_dir, partition_cols=["month"], index=False)
alpha_dir = tmp_path / "alphas"
alpha_result = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(dataset_dir),
"--alpha-type", "reversal",
"--alpha-name", "dataset_reversal",
"--lookback", "3",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "dataset_reversal.pq"
assert "Loaded data:" in alpha_result.output
assert alpha_path.exists()
combo_dir = tmp_path / "combos"
_invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", str(alpha_path),
"--combo-name", "dataset_combo",
"--output-dir", str(combo_dir),
])
combo_path = combo_dir / "dataset_combo.pq"
assert combo_path.exists()
portfolio_dir = tmp_path / "portfolio"
_invoke_ok(runner, [
"portfolio", "build",
"--weights-path", str(combo_path),
"--data-path", str(dataset_dir),
"--booksize", "1000000",
"--portfolio-name", "dataset_portfolio",
"--output-dir", str(portfolio_dir),
])
positions_path = portfolio_dir / "dataset_portfolio.pq"
assert positions_path.exists()
execution_dir = tmp_path / "execution"
simulate_result = _invoke_ok(runner, [
"portfolio", "simulate",
"--positions-path", str(positions_path),
"--data-path", str(dataset_dir),
"--constraint", "suspension",
"--output-dir", str(execution_dir),
])
assert "Saved fills:" in simulate_result.output
assert (execution_dir / "fills" / "dataset_portfolio.pq").exists()
assert (execution_dir / "pnl" / "dataset_portfolio.pq").exists()
def test_cli_liquid_universe_masks_to_top_liquid_names(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=75, include_missing=False)
daily_path = tmp_path / "daily_bars_75d.pq"
daily_bars.to_parquet(daily_path, index=False)
alpha_dir = tmp_path / "alphas"
result = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "reversal_rank",
"--alpha-name", "liquid_rank",
"--lookback", "3",
"--liquid-universe",
"--universe-top-n", "2",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "liquid_rank.pq"
alpha = pd.read_parquet(alpha_path)
nonzero = alpha[alpha["weight"] != 0.0]
assert "Saved alpha:" in result.output
assert alpha_path.exists()
assert not nonzero.empty
assert nonzero.groupby("date")["symbol_id"].nunique().max() <= 2
def test_cli_real_fixture_round_trips_through_portfolio(tmp_path):
runner = CliRunner()
alpha_dir = tmp_path / "alphas"
_invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(FIXTURE_PATH),
"--alpha-type", "reversal_vol",
"--alpha-name", "real_cli_reversal_vol",
"--lookback", "3",
"--vol-window", "3",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "real_cli_reversal_vol.pq"
assert alpha_path.exists()
assert not pd.read_parquet(alpha_path).empty
combo_dir = tmp_path / "combos"
_invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", str(alpha_path),
"--combo-name", "real_cli_combo",
"--output-dir", str(combo_dir),
])
combo_path = combo_dir / "real_cli_combo.pq"
assert combo_path.exists()
portfolio_dir = tmp_path / "portfolio"
_invoke_ok(runner, [
"portfolio", "build",
"--weights-path", str(combo_path),
"--data-path", str(FIXTURE_PATH),
"--booksize", "1000000",
"--portfolio-name", "real_cli_portfolio",
"--output-dir", str(portfolio_dir),
])
positions_path = portfolio_dir / "real_cli_portfolio.pq"
positions = pd.read_parquet(positions_path)
assert not positions.empty
eval_result = _invoke_ok(runner, [
"portfolio", "eval",
"--positions-path", str(positions_path),
"--data-path", str(FIXTURE_PATH),
])
assert "Research-portfolio metrics:" in eval_result.output
def test_cli_error_paths_are_clear_for_bad_user_inputs(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars()
daily_path = tmp_path / "daily_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
unknown_alpha = _invoke_error(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "does_not_exist",
"--alpha-name", "bad",
"--output-dir", str(tmp_path / "alphas"),
])
assert "Unknown alpha-type" in unknown_alpha.output
malformed_param = _invoke_error(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "reversal",
"--alpha-name", "bad_param",
"--param", "not-an-assignment",
"--output-dir", str(tmp_path / "alphas"),
])
assert "--param must be name=value" in malformed_param.output
unknown_derived = _invoke_error(runner, [
"derived", "compute",
"--daily-path", str(daily_path),
"--derived-type", "does_not_exist",
"--derived-name", "bad",
"--output-dir", str(tmp_path / "derived"),
])
assert "Unknown derived-type" in unknown_derived.output
bad_constraint_positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"portfolio_name": ["bad_constraint"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [100.0],
"position_shares": [100],
"position_value": [1000.0],
"price": [10.0],
})
positions_path = tmp_path / "positions.pq"
bad_constraint_positions.to_parquet(positions_path, index=False)
unknown_constraint = _invoke_error(runner, [
"portfolio", "simulate",
"--positions-path", str(positions_path),
"--data-path", str(daily_path),
"--constraint", "not_a_constraint",
"--output-dir", str(tmp_path / "execution"),
])
assert isinstance(unknown_constraint.exception, KeyError)
assert "not_a_constraint" in str(unknown_constraint.exception)
pqcat_missing_column = _invoke_error(runner, [
"pqcat",
str(daily_path),
"--columns", "close,not_a_column",
])
assert "Columns not found: not_a_column" in pqcat_missing_column.output
alphaview_missing_symbol = _invoke_error(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(positions_path),
"--symbol", "sh999999",
])
assert "Symbol 'sh999999' not found" in alphaview_missing_symbol.output
alphaview_missing_column = _invoke_error(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(positions_path),
"--symbol", "sh600000",
"--columns", "close,missing_bar_col",
])
assert "Bar columns not found: missing_bar_col" in alphaview_missing_column.output
alphaview_alpha = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [daily_bars["date"].min()],
"alpha_name": ["toy_alpha"],
"weight": [1.0],
})
alphaview_alpha_path = tmp_path / "alphaview_alpha.pq"
alphaview_alpha.to_parquet(alphaview_alpha_path, index=False)
alphaview_empty_range = _invoke_error(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(alphaview_alpha_path),
"--symbol", "sh600000",
"--start-date", "2030-01-01",
])
assert "No rows in the requested date range" in alphaview_empty_range.output
empty_combo_paths = _invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", " , ",
"--combo-name", "empty",
"--output-dir", str(tmp_path / "combos"),
])
assert "requires at least 1 path" in empty_combo_paths.output
def test_cli_parser_helpers_cover_string_coercion_and_bad_params():
assert derived_cli._parse_params(("n=7", "scale=2.5", "label=demo")) == {
"n": 7,
"scale": 2.5,
"label": "demo",
}
assert features_cli._parse_params(("n=7", "scale=2.5", "label=demo")) == {
"n": 7,
"scale": 2.5,
"label": "demo",
}
for module in (derived_cli, features_cli):
try:
module._parse_params(("not-an-assignment",))
except click.BadParameter as exc:
assert "--param must be name=value" in str(exc)
else:
raise AssertionError("expected BadParameter")
def test_cli_list_and_legacy_feature_paths(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
minute_bars = make_generated_minute_bars(daily_bars)
daily_path = tmp_path / "daily_bars.pq"
minute_path = tmp_path / "minute_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
minute_bars.to_parquet(minute_path, index=False)
derived_list = _invoke_ok(runner, ["derived", "list"])
feature_list = _invoke_ok(runner, ["feature", "list"])
assert "minute_daily_summary" in derived_list.output
assert "minute_daily_summary" in feature_list.output
feature_dir = tmp_path / "features"
feature_compute = _invoke_ok(runner, [
"feature",
"compute",
"--minute-path",
str(minute_path),
"--daily-path",
str(daily_path),
"--feature-type",
"minute_daily_summary",
"--feature-name",
"legacy_summary",
"--output-dir",
str(feature_dir),
])
feature_path = feature_dir / "legacy_summary.pq"
assert "Loaded minute bars:" in feature_compute.output
assert "Loaded daily data:" in feature_compute.output
assert "Saved feature:" in feature_compute.output
assert feature_path.exists()
no_input = _invoke_error(runner, [
"derived",
"compute",
"--derived-type",
"minute_daily_summary",
"--derived-name",
"missing_inputs",
"--output-dir",
str(tmp_path / "derived"),
])
assert "At least one of --daily-path or --minute-path is required" in no_input.output
missing_minute = _invoke_error(runner, [
"derived",
"compute",
"--daily-path",
str(daily_path),
"--derived-type",
"minute_daily_summary",
"--derived-name",
"daily_only",
"--output-dir",
str(tmp_path / "derived"),
])
assert "minute_daily_summary requires minute input" in missing_minute.output
malformed_feature_param = _invoke_error(runner, [
"feature",
"compute",
"--minute-path",
str(minute_path),
"--feature-type",
"minute_daily_summary",
"--feature-name",
"bad_param",
"--param",
"not-an-assignment",
"--output-dir",
str(tmp_path / "features_bad"),
])
assert "--param must be name=value" in malformed_feature_param.output
unknown_feature = _invoke_error(runner, [
"feature",
"compute",
"--minute-path",
str(minute_path),
"--feature-type",
"does_not_exist",
"--feature-name",
"bad_feature",
"--output-dir",
str(tmp_path / "features_unknown"),
])
assert "Unknown feature-type" in unknown_feature.output
def test_cli_shortcuts_and_external_module_loading(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=8, include_missing=False)
minute_bars = make_generated_minute_bars(daily_bars)
daily_path = tmp_path / "daily_bars.pq"
minute_path = tmp_path / "minute_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
minute_bars.to_parquet(minute_path, index=False)
alpha_module = tmp_path / "listed_alpha.py"
alpha_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class ListedAlpha(BaseAlpha):
name = "listed_alpha_cli"
def __init__(self, scale: float = 1.0, label: str = "x"):
self.scale = scale
self.label = label
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
return close.pct_change(1, fill_method=None) * self.scale
"""))
derived_module = tmp_path / "listed_derived.py"
derived_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class ListedDerived(BaseDerivedData):
name = "listed_derived_cli"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = daily[["symbol_id", "date", "close"]].copy()
out["listed_value"] = out.pop("close") * self.scale
return out
"""))
derived_compute_module = tmp_path / "computed_derived.py"
derived_compute_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class ComputedDerived(BaseDerivedData):
name = "computed_derived_cli"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = daily[["symbol_id", "date", "close"]].copy()
out["computed_value"] = out.pop("close") * self.scale
return out
"""))
feature_module = tmp_path / "listed_feature.py"
feature_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.features.base import BaseFeature
from pipeline.features.registry import register_feature
@register_feature
class ListedFeature(BaseFeature):
name = "listed_feature_cli"
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = minute[["symbol_id", "date", "close"]].copy()
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
out = out.groupby(["symbol_id", "date"], as_index=False)["close"].mean()
return out.rename(columns={"close": "listed_feature_value"})
"""))
feature_compute_module = tmp_path / "computed_feature.py"
feature_compute_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.features.base import BaseFeature
from pipeline.features.registry import register_feature
@register_feature
class ComputedFeature(BaseFeature):
name = "computed_feature_cli"
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = minute[["symbol_id", "date", "close"]].copy()
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
out = out.groupby(["symbol_id", "date"], as_index=False)["close"].mean()
return out.rename(columns={"close": "computed_feature_value"})
"""))
alpha_list = _invoke_ok(runner, [
"alpha", "list",
"--alpha-module", str(alpha_module),
])
assert "listed_alpha_cli" in alpha_list.output
alpha_dir = tmp_path / "alphas"
reversal = _invoke_ok(runner, [
"alpha", "reversal",
"--data-path", str(daily_path),
"--output-dir", str(alpha_dir),
"--lookback", "3",
])
reversal_vol = _invoke_ok(runner, [
"alpha", "reversal-vol",
"--data-path", str(daily_path),
"--output-dir", str(alpha_dir),
"--lookback", "3",
"--vol-window", "3",
])
external_alpha = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "listed_alpha_cli",
"--alpha-name", "listed_alpha_run",
"--param", "scale=2.5",
"--param", "label=demo",
"--output-dir", str(alpha_dir),
])
assert "Saved alpha:" in reversal.output
assert "Saved alpha:" in reversal_vol.output
assert "Saved alpha:" in external_alpha.output
assert (alpha_dir / "reversal_3d.pq").exists()
assert (alpha_dir / "reversal_vol_3d_3d.pq").exists()
assert (alpha_dir / "listed_alpha_run.pq").exists()
derived_list = _invoke_ok(runner, [
"derived", "list",
"--derived-module", str(derived_module),
])
assert "listed_derived_cli" in derived_list.output
derived_dir = tmp_path / "derived_external"
derived_compute = _invoke_ok(runner, [
"derived", "compute",
"--daily-path", str(daily_path),
"--derived-module", str(derived_compute_module),
"--derived-type", "computed_derived_cli",
"--derived-name", "listed_derived_run",
"--param", "scale=3",
"--output-dir", str(derived_dir),
])
assert "Saved derived data:" in derived_compute.output
assert (derived_dir / "listed_derived_run.pq").exists()
feature_list = _invoke_ok(runner, [
"feature", "list",
"--feature-module", str(feature_module),
])
assert "listed_feature_cli" in feature_list.output
feature_dir = tmp_path / "features_external"
feature_compute = _invoke_ok(runner, [
"feature", "compute",
"--minute-path", str(minute_path),
"--feature-module", str(feature_compute_module),
"--feature-type", "computed_feature_cli",
"--feature-name", "listed_feature_run",
"--output-dir", str(feature_dir),
])
assert "Saved feature:" in feature_compute.output
assert (feature_dir / "listed_feature_run.pq").exists()
def test_cli_pqcat_row_modes(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
daily_path = tmp_path / "daily_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
head_result = _invoke_ok(runner, [
"pqcat",
str(daily_path),
"--head",
"2",
"--columns",
"symbol_id,close",
])
tail_result = _invoke_ok(runner, [
"pqcat",
str(daily_path),
"--tail",
"1",
])
assert "symbol_id" in head_result.output
assert "close" in head_result.output
assert "date" in tail_result.output
+148
View File
@@ -0,0 +1,148 @@
"""Offline coverage for data CLI and universe resolution glue."""
from __future__ import annotations
import pandas as pd
from click.testing import CliRunner
from cli import cli
import pipeline.data.cli as data_cli
import pipeline.data.downloader as pipeline_downloader
def test_resolve_universe_handles_named_file_all_and_symbol_list(tmp_path, monkeypatch):
hs300_raw = pd.DataFrame({
"updateDate": ["2024-01-12", "2024-01-12"],
"stockName": ["浦发银行", "平安银行"],
"stockCode": ["sh.600000", "sz.000001"],
})
zz500_raw = pd.DataFrame({
"name": ["东风汽车"],
"code": ["sh.600006"],
"date": ["2024-01-12"],
})
monkeypatch.setattr(pipeline_downloader, "get_hs300_stocks", lambda: hs300_raw)
monkeypatch.setattr(pipeline_downloader, "get_zz500_stocks", lambda: zz500_raw)
monkeypatch.setattr(
pipeline_downloader,
"get_all_stocks",
lambda: pd.DataFrame({
"code": ["sh600000", "sz000001", "sh600519"],
"name": ["浦发银行", "平安银行", "贵州茅台"],
}),
)
symbol_file = tmp_path / "symbols.txt"
symbol_file.write_text("sh600000\n\nsz000001\n")
hs300 = pipeline_downloader._resolve_universe("hs300")
zz500 = pipeline_downloader._resolve_universe("csi500")
all_capped = pipeline_downloader._resolve_universe("all", max_symbols=2)
from_file = pipeline_downloader._resolve_universe(str(symbol_file))
from_list = pipeline_downloader._resolve_universe("sh600000, sz000001")
assert hs300.to_dict("list") == {
"symbol_name": ["浦发银行", "平安银行"],
"symbol_id": ["sh600000", "sz000001"],
}
assert zz500.to_dict("list") == {
"symbol_name": ["东风汽车"],
"symbol_id": ["sh600006"],
}
assert all_capped["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert from_file["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert from_file["symbol_name"].tolist() == ["sh600000", "sz000001"]
assert from_list["symbol_id"].tolist() == ["sh600000", "sz000001"]
def test_data_cli_download_commands_print_summaries_without_network(monkeypatch, tmp_path):
runner = CliRunner()
daily_calls: list[dict] = []
minute_calls: list[dict] = []
def fake_daily(**kwargs):
daily_calls.append(kwargs)
return {
"dataset_path": str(tmp_path / "daily" / kwargs["universe"]),
"n_symbols": 2,
"n_requested": 3,
"n_rows": 18,
"date_min": "2024-01-02",
"date_max": "2024-01-12",
}
def fake_minute(**kwargs):
minute_calls.append(kwargs)
return {
"dataset_path": str(tmp_path / "minute" / kwargs["universe"]),
"frequency": "15m",
"n_symbols": 1,
"n_requested": 1,
"n_rows": 32,
"date_min": "2024-01-02",
"date_max": "2024-01-03",
}
monkeypatch.setattr(data_cli, "download_universe", fake_daily)
monkeypatch.setattr(data_cli, "download_minute_universe", fake_minute)
daily_result = runner.invoke(cli, [
"data",
"download",
"--universe",
"sh600000,sz000001",
"--start-date",
"2024-01-02",
"--end-date",
"2024-01-12",
"--output-dir",
str(tmp_path / "daily"),
"--symbols",
"3",
"--chunk-size",
"2",
"--adjust",
"none",
])
minute_result = runner.invoke(cli, [
"data",
"download-minute",
"--universe",
"toy",
"--start-date",
"2024-01-02",
"--end-date",
"2024-01-03",
"--output-dir",
str(tmp_path / "minute"),
"--symbols",
"1",
"--chunk-size",
"1",
"--frequency",
"15",
])
assert daily_result.exit_code == 0, daily_result.output
assert "Summary: 2/3 symbols, 18 bars" in daily_result.output
assert daily_calls == [{
"universe": "sh600000,sz000001",
"start_date": "2024-01-02",
"end_date": "2024-01-12",
"output_dir": str(tmp_path / "daily"),
"max_symbols": 3,
"chunk_size": 2,
"adjust": "none",
}]
assert minute_result.exit_code == 0, minute_result.output
assert "frequency=15m" in minute_result.output
assert minute_calls == [{
"universe": "toy",
"start_date": "2024-01-02",
"end_date": "2024-01-03",
"output_dir": str(tmp_path / "minute"),
"max_symbols": 1,
"chunk_size": 1,
"frequency": "15",
}]
+376
View File
@@ -0,0 +1,376 @@
"""Tests for daily derived-data ingestion and plugins."""
import textwrap
import numpy as np
import pandas as pd
import pytest
from click.testing import CliRunner
from cli import cli
from pipeline.alpha.compute import join_feature_frames
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.compute import compute_derived, read_derived_frame, validate_derived_frame
from pipeline.derived.registry import (
available_derived,
get_derived,
load_derived_module,
register_derived,
)
def _daily_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
"open": [10.0, 20.0, 11.0],
"close": [10.5, 20.5, 11.5],
"volume": [1000.0, 2000.0, 1200.0],
})
def _minute_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001"],
"datetime": pd.to_datetime([
"2024-01-02 09:35:00",
"2024-01-02 09:40:00",
"2024-01-02 09:35:00",
]),
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
"time": ["09:35:00", "09:40:00", "09:35:00"],
"open": [10.0, 10.5, 20.0],
"high": [11.0, 12.0, 21.0],
"low": [9.0, 10.0, 19.0],
"close": [10.5, 11.0, 20.5],
"volume": [100.0, 300.0, 200.0],
"amount": [1000.0, 3300.0, 4100.0],
})
def test_validate_derived_frame_normalizes_and_sorts():
result = validate_derived_frame(pd.DataFrame({
"symbol_id": ["sz000001", "sh600000"],
"date": ["2024-01-02 15:00:00", "2024-01-02 09:30:00"],
"custom_value": [2.0, 1.0],
}))
assert result["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert result["date"].tolist() == [
pd.Timestamp("2024-01-02"),
pd.Timestamp("2024-01-02"),
]
def test_validate_derived_frame_rejects_missing_keys():
with pytest.raises(ValueError, match="missing required"):
validate_derived_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
def test_validate_derived_frame_rejects_duplicate_normalized_keys():
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
"x": [1.0, 2.0],
}))
def test_validate_derived_frame_rejects_duplicate_columns():
bad = pd.DataFrame(
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
columns=["symbol_id", "date", "dup", "dup"],
)
with pytest.raises(ValueError, match="duplicate columns"):
validate_derived_frame(bad)
def test_validate_derived_frame_rejects_non_numeric_values():
with pytest.raises(ValueError, match="numeric"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"bad": ["not numeric"],
}))
def test_validate_derived_frame_rejects_missing_value_columns():
with pytest.raises(ValueError, match="at least one value column"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
}))
def test_read_derived_frame_rejects_empty_csv(tmp_path):
empty_csv = tmp_path / "empty.csv"
empty_csv.write_text("")
with pytest.raises(ValueError, match="CSV input is empty"):
read_derived_frame(empty_csv)
def test_compute_derived_rejects_missing_inputs():
with pytest.raises(ValueError, match="requires --daily-path or --minute-path"):
compute_derived("minute_daily_summary")
def test_derived_ingest_cli_accepts_csv_and_parquet(tmp_path):
runner = CliRunner()
source = pd.DataFrame({
"symbol_id": ["sz000001", "sh600000"],
"date": ["2024-01-02", "2024-01-02"],
"custom_value": [2.0, 1.0],
})
csv_path = tmp_path / "custom.csv"
parquet_path = tmp_path / "custom.pq"
out_dir = tmp_path / "derived"
source.to_csv(csv_path, index=False)
source.to_parquet(parquet_path, index=False)
csv_result = runner.invoke(cli, [
"derived", "ingest",
"--input-path", str(csv_path),
"--derived-name", "csv_custom",
"--output-dir", str(out_dir),
])
assert csv_result.exit_code == 0, csv_result.output
parquet_result = runner.invoke(cli, [
"derived", "ingest",
"--input-path", str(parquet_path),
"--derived-name", "parquet_custom",
"--output-dir", str(out_dir),
])
assert parquet_result.exit_code == 0, parquet_result.output
written = pd.read_parquet(out_dir / "csv_custom.pq")
assert written["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert (out_dir / "parquet_custom.pq").exists()
def test_derived_validate_cli_rejects_duplicate_csv_columns(tmp_path):
runner = CliRunner()
csv_path = tmp_path / "bad.csv"
csv_path.write_text("symbol_id,date,x,x\nsh600000,2024-01-02,1.0,2.0\n")
result = runner.invoke(cli, [
"derived", "validate",
"--input-path", str(csv_path),
])
assert result.exit_code != 0
assert "duplicate columns" in result.output
def test_derived_ingest_cli_wraps_validation_errors(tmp_path):
runner = CliRunner()
csv_path = tmp_path / "bad_ingest.csv"
csv_path.write_text("symbol_id,date\nsh600000,2024-01-02\n")
result = runner.invoke(cli, [
"derived",
"ingest",
"--input-path",
str(csv_path),
"--derived-name",
"bad",
"--output-dir",
str(tmp_path / "derived"),
])
assert result.exit_code != 0
assert "at least one value column" in result.output
def test_external_derived_plugin_loads_filters_params_and_uses_inputs(tmp_path):
module_path = tmp_path / "external_derived.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class FlexibleDerived(BaseDerivedData):
name = "flexible_derived_test"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, daily=None, minute=None) -> pd.DataFrame:
result = None
if daily is not None:
result = daily[["symbol_id", "date", "close"]].copy()
result["daily_scaled_close"] = result.pop("close") * self.scale
if minute is not None:
minute_out = (
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
.sum()
.rename(columns={"volume": "minute_volume_sum"})
)
minute_out["minute_volume_sum"] *= self.scale
result = minute_out if result is None else result.merge(
minute_out, on=["symbol_id", "date"], how="left"
)
return result
'''))
load_derived_module(str(module_path))
assert "flexible_derived_test" in available_derived()
instance = get_derived("flexible_derived_test", scale=2.0, ignored=99)
assert instance.scale == 2.0
assert not hasattr(instance, "ignored")
daily_result = compute_derived(
"flexible_derived_test",
daily=_daily_bars(),
scale=2.0,
ignored=99,
)
assert "daily_scaled_close" in daily_result.columns
assert np.isclose(daily_result["daily_scaled_close"].iloc[0], 21.0)
minute_result = compute_derived(
"flexible_derived_test",
minute=_minute_bars(),
scale=2.0,
)
assert "minute_volume_sum" in minute_result.columns
assert np.isclose(
minute_result.loc[minute_result["symbol_id"] == "sh600000", "minute_volume_sum"].iloc[0],
800.0,
)
both_result = compute_derived(
"flexible_derived_test",
daily=_daily_bars(),
minute=_minute_bars(),
scale=1.0,
)
assert {"daily_scaled_close", "minute_volume_sum"}.issubset(both_result.columns)
def test_derived_registry_rejects_bad_plugins_and_load_failures(tmp_path, monkeypatch):
with pytest.raises(TypeError):
register_derived(object) # type: ignore[arg-type]
with pytest.raises(ValueError, match="non-empty"):
@register_derived
class NoNameDerived(BaseDerivedData):
def compute(self, daily=None, minute=None):
return pd.DataFrame()
@register_derived
class _CoverageDerived(BaseDerivedData):
name = "_coverage_derived_registry"
def compute(self, daily=None, minute=None):
return pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"x": [1.0],
})
with pytest.raises(ValueError, match="already registered"):
@register_derived
class _CoverageDerivedDuplicate(BaseDerivedData):
name = "_coverage_derived_registry"
def compute(self, daily=None, minute=None):
return pd.DataFrame()
with pytest.raises(KeyError, match="Unknown derived data"):
get_derived("does_not_exist")
missing_path = tmp_path / "missing_derived.py"
with pytest.raises(FileNotFoundError):
load_derived_module(str(missing_path))
bad_path = tmp_path / "bad_derived.py"
bad_path.write_text("x = 1\n")
monkeypatch.setattr(
"pipeline.derived.registry.importlib.util.spec_from_file_location",
lambda *args, **kwargs: None,
)
with pytest.raises(ImportError, match="Cannot load derived data module"):
load_derived_module(str(bad_path))
load_derived_module("math")
instance = _CoverageDerived()
instance.scale = 2
assert repr(instance) == "_CoverageDerived(scale=2)"
def test_derived_compute_cli_writes_builtin_minute_summary(tmp_path):
runner = CliRunner()
minute_path = tmp_path / "minute.pq"
out_dir = tmp_path / "derived"
_minute_bars().to_parquet(minute_path, index=False)
result = runner.invoke(cli, [
"derived", "compute",
"--minute-path", str(minute_path),
"--derived-type", "minute_daily_summary",
"--derived-name", "minute_summary",
"--output-dir", str(out_dir),
])
assert result.exit_code == 0, result.output
written = pd.read_parquet(out_dir / "minute_summary.pq")
assert "minute_vwap" in written.columns
def test_minute_summary_uses_time_sort_and_daily_without_close():
minute = _minute_bars().drop(columns=["datetime"]).sample(frac=1.0, random_state=0)
daily = _daily_bars()[["symbol_id", "date", "open"]]
result = compute_derived(
"minute_daily_summary",
daily=daily,
minute=minute,
)
by_symbol = result.set_index(["symbol_id", "date"])
assert np.isclose(
by_symbol.loc[("sh600000", pd.Timestamp("2024-01-02")), "minute_intraday_return"],
0.10,
)
assert "minute_vwap_deviation" in result.columns
def test_alpha_feature_join_rejects_derived_column_collisions():
data = _daily_bars()
derived_a = data[["symbol_id", "date"]].copy()
derived_a["custom_value"] = 1.0
derived_b = data[["symbol_id", "date"]].copy()
derived_b["custom_value"] = 2.0
with pytest.raises(ValueError, match="conflict"):
join_feature_frames(data, [derived_a, derived_b])
close_collision = data[["symbol_id", "date"]].copy()
close_collision["close"] = 1.0
with pytest.raises(ValueError, match="conflict"):
join_feature_frames(data, [close_collision])
def test_legacy_feature_cli_delegates_to_derived_registry(tmp_path):
runner = CliRunner()
minute_path = tmp_path / "minute.pq"
out_dir = tmp_path / "features"
_minute_bars().to_parquet(minute_path, index=False)
list_result = runner.invoke(cli, ["feature", "list"])
assert list_result.exit_code == 0, list_result.output
assert "minute_daily_summary" in list_result.output
compute_result = runner.invoke(cli, [
"feature", "compute",
"--minute-path", str(minute_path),
"--feature-type", "minute_daily_summary",
"--feature-name", "minute_summary",
"--output-dir", str(out_dir),
])
assert compute_result.exit_code == 0, compute_result.output
assert (out_dir / "minute_summary.pq").exists()
+10
View File
@@ -1,6 +1,16 @@
import os
import pytest
from data.downloader import download_daily
pytestmark = [
pytest.mark.network,
pytest.mark.skipif(
os.environ.get("CEQ_RUN_LIVE_DOWNLOADER") != "1",
reason="set CEQ_RUN_LIVE_DOWNLOADER=1 to run live baostock/akshare smoke tests",
),
]
def test_download_single_stock():
"""Smoke test: download data for 浦发银行 for a short window."""
+631
View File
@@ -0,0 +1,631 @@
"""Offline downloader contract tests with mocked data providers."""
from __future__ import annotations
import numpy as np
import pandas as pd
import pytest
import data.downloader as downloader
import pipeline.data.downloader as pipeline_downloader
from data.downloader import download_daily, download_daily_batch
from pipeline.common.schema import DATA_COLUMNS
from pipeline.data.downloader import download_universe
class _FakeResult:
def __init__(self, rows, error_code="0", error_msg=""):
self.rows = rows
self.error_code = error_code
self.error_msg = error_msg
self._idx = -1
def next(self):
self._idx += 1
return self._idx < len(self.rows)
def get_row_data(self):
return self.rows[self._idx]
def _daily_batch_row(
date: str = "2024-01-02",
open_: str = "10",
high: str = "11",
low: str = "9",
close: str = "10.5",
preclose: str = "10",
volume: str = "1000",
amount: str = "10500",
) -> list[str]:
return [
date,
open_,
high,
low,
close,
preclose,
volume,
amount,
"1.23",
"5.0",
"1",
"0",
"8.0",
"1.1",
"2.2",
"3.3",
]
def test_download_daily_uses_baostock_before_akshare_in_auto(monkeypatch):
calls: list[str] = []
expected = pd.DataFrame({
"symbol": ["sh600000"],
"date": ["2024-01-02"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
})
def fake_baostock(symbol, start, end, adjust):
calls.append("baostock")
return expected
def fake_akshare(symbol, start, end, adjust):
calls.append("akshare")
raise AssertionError("akshare should not be called after baostock succeeds")
monkeypatch.setattr(downloader, "_download_baostock", fake_baostock)
monkeypatch.setattr(downloader, "_download_akshare", fake_akshare)
result = download_daily("sh600000", "2024-01-02", "2024-01-02", source="auto")
assert calls == ["baostock"]
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
assert result["close"].tolist() == [10.5]
def test_download_daily_falls_back_to_akshare_when_baostock_empty(monkeypatch):
calls: list[str] = []
fallback = pd.DataFrame({
"symbol": ["sz000001"],
"date": ["2024-01-02"],
"open": [20.0],
"high": [21.0],
"low": [19.0],
"close": [20.5],
"volume": [2000.0],
"amount": [41000.0],
})
monkeypatch.setattr(
downloader,
"_download_baostock",
lambda symbol, start, end, adjust: calls.append("baostock") or None,
)
monkeypatch.setattr(
downloader,
"_download_akshare",
lambda symbol, start, end, adjust: calls.append("akshare") or fallback,
)
result = download_daily("sz000001", "2024-01-02", "2024-01-02", source="auto")
assert calls == ["baostock", "akshare"]
assert result["symbol"].tolist() == ["sz000001"]
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
def test_download_daily_raises_when_requested_source_has_no_data(monkeypatch):
monkeypatch.setattr(downloader, "_download_baostock", lambda *args: None)
with pytest.raises(RuntimeError, match="Failed to download data for sh600000"):
download_daily(
"sh600000",
"2024-01-02",
"2024-01-02",
source="baostock",
)
def test_download_daily_akshare_source_skips_baostock(monkeypatch):
calls: list[str] = []
fallback = pd.DataFrame({
"symbol": ["sh600000"],
"date": ["2024-01-02"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
})
monkeypatch.setattr(
downloader,
"_download_baostock",
lambda *args: calls.append("baostock") or None,
)
monkeypatch.setattr(
downloader,
"_download_akshare",
lambda *args: calls.append("akshare") or fallback,
)
result = download_daily(
"sh600000",
"2024-01-02",
"2024-01-02",
source="akshare",
)
assert calls == ["akshare"]
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
def test_akshare_daily_downloader_maps_columns_and_failures(monkeypatch):
calls: list[dict] = []
raw = pd.DataFrame({
"日期": ["2024-01-02"],
"开盘": [10.0],
"最高": [11.0],
"最低": [9.0],
"收盘": [10.5],
"成交量": [1000.0],
"成交额": [10500.0],
"换手率": [1.2],
})
def fake_hist(**kwargs):
calls.append(kwargs)
return raw.copy()
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", fake_hist)
result = downloader._download_akshare(
"sh600000",
"20240102",
"20240102",
adjust="",
)
assert calls == [{
"symbol": "600000",
"period": "daily",
"start_date": "20240102",
"end_date": "20240102",
"adjust": "",
}]
assert result is not None
assert result.columns.tolist() == [
"symbol", "date", "open", "high", "low", "close", "volume", "amount",
]
assert result["symbol"].tolist() == ["sh600000"]
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", lambda **kwargs: pd.DataFrame())
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
monkeypatch.setattr(
downloader.ak,
"stock_zh_a_hist",
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("provider down")),
)
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
def test_baostock_daily_downloader_maps_errors_and_logout_failures(monkeypatch):
query_calls: list[dict] = []
row = ["2024-01-02", "10", "11", "9", "10.5", "1000", "10500"]
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(
downloader.bs,
"logout",
lambda: (_ for _ in ()).throw(RuntimeError("logout failed")),
)
def fake_query(**kwargs):
query_calls.append(kwargs)
return _FakeResult([row])
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
result = downloader._download_baostock(
"sz000001",
"2024-01-02",
"2024-01-02",
adjust="none",
)
assert query_calls[0]["code"] == "sz.000001"
assert query_calls[0]["adjustflag"] == "3"
assert result is not None
assert result["symbol"].tolist() == ["sz000001"]
assert pd.api.types.is_numeric_dtype(result["close"])
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([], error_code="1", error_msg="bad symbol"),
)
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([]),
)
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("query failed")),
)
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
def test_download_daily_batch_maps_rich_schema_and_vwap(monkeypatch):
query_calls: list[dict] = []
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
def fake_query(**kwargs):
query_calls.append(kwargs)
rows = [
_daily_batch_row(volume="1000", amount="10500"),
_daily_batch_row(date="2024-01-03", volume="0", amount="0"),
]
return _FakeResult(rows)
monkeypatch.setattr(downloader.bs, "login", fake_login)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
[(symbol, frame)] = list(
download_daily_batch(
["sh600000"],
"2024-01-02",
"2024-01-03",
adjust="hfq",
)
)
assert symbol == "sh600000"
assert query_calls[0]["code"] == "sh.600000"
assert query_calls[0]["adjustflag"] == "1"
assert login_count == 1
assert logout_count == 1
assert frame is not None
assert frame.columns.tolist() == [
"symbol", "date", "open", "high", "low", "close", "preclose",
"volume", "amount", "vwap", "turn", "pctChg", "tradestatus", "isST",
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
]
assert np.isclose(frame["vwap"].iloc[0], 10.5)
assert pd.isna(frame["vwap"].iloc[1])
assert pd.api.types.is_datetime64_any_dtype(frame["date"])
assert pd.api.types.is_numeric_dtype(frame["tradestatus"])
def test_download_daily_batch_periodic_relogin_and_none_result(monkeypatch):
responses = [
_FakeResult([], error_code="1", error_msg="bad symbol"),
_FakeResult([_daily_batch_row(date="2024-01-03")]),
]
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
monkeypatch.setattr(downloader.bs, "login", fake_login)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
results = list(
download_daily_batch(
["sh600000", "sz000001"],
"2024-01-02",
"2024-01-03",
relogin_every=1,
)
)
assert results[0] == ("sh600000", None)
assert results[1][0] == "sz000001"
assert results[1][1] is not None
assert login_count == 2
assert logout_count == 2
def test_download_daily_batch_empty_rows_yields_none(monkeypatch):
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([]),
)
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_daily_batch_generic_exception_yields_none(monkeypatch):
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("query failed")),
)
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_daily_batch_relogs_and_retries_session_loss(monkeypatch):
responses = [
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
_FakeResult([_daily_batch_row()]),
]
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
monkeypatch.setattr(downloader.bs, "login", fake_login)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
[(symbol, frame)] = list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02"))
assert symbol == "sh600000"
assert frame is not None
assert len(frame) == 1
assert login_count == 2
assert logout_count == 2
def test_download_daily_batch_second_session_loss_and_logout_failure(monkeypatch):
responses = [
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
]
logout_count = 0
def fake_logout():
nonlocal logout_count
logout_count += 1
raise RuntimeError("logout failed")
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
assert logout_count == 2
def test_download_daily_batch_uses_akshare_fallback_when_enabled(monkeypatch):
fallback = pd.DataFrame({
"symbol": ["sh600000"],
"date": ["2024-01-02"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
})
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([], error_code="1", error_msg="no data"),
)
monkeypatch.setattr(
downloader,
"_download_akshare",
lambda symbol, start, end, adjust: fallback.copy(),
)
[(symbol, frame)] = list(
download_daily_batch(
["sh600000"],
"2024-01-02",
"2024-01-02",
akshare_fallback=True,
)
)
assert symbol == "sh600000"
assert frame is not None
assert frame["date"].tolist() == [pd.Timestamp("2024-01-02")]
assert frame["close"].tolist() == [10.5]
def test_download_universe_writes_daily_partitions_from_mock_batch(tmp_path, monkeypatch):
batch_frame = pd.DataFrame({
"symbol": ["sh600000", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-02-01"]),
"open": [10.0, 11.0],
"high": [11.0, 12.0],
"low": [9.0, 10.0],
"close": [10.5, 11.5],
"preclose": [10.0, 10.5],
"volume": [1000.0, 1200.0],
"amount": [10500.0, 13800.0],
"vwap": [10.5, 11.5],
"turn": [1.0, 1.1],
"pctChg": [5.0, 9.5],
"tradestatus": [1, 1],
"isST": [0, 0],
"peTTM": [8.0, 8.1],
"pbMRQ": [1.1, 1.2],
"psTTM": [2.1, 2.2],
"pcfNcfTTM": [3.1, 3.2],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000", "sz000001"],
"symbol_name": ["PF Bank", "Ping An Bank"],
}),
)
def fake_batch(symbols, start, end, adjust="qfq"):
assert symbols == ["sh600000", "sz000001"]
assert adjust == "qfq"
yield "sh600000", batch_frame
yield "sz000001", None
monkeypatch.setattr(pipeline_downloader, "download_daily_batch", fake_batch)
stale_file = tmp_path / "toy" / "month=2024-01" / "stale.pq"
stale_file.parent.mkdir(parents=True)
batch_frame.iloc[[0]][DATA_COLUMNS[2:]].to_parquet(stale_file, index=False)
stats = download_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-02-01",
output_dir=str(tmp_path),
chunk_size=1,
)
dataset_path = tmp_path / "toy"
written = pd.read_parquet(dataset_path).sort_values(["date", "symbol_id"]).reset_index(drop=True)
assert stats == {
"dataset_path": str(dataset_path),
"n_symbols": 1,
"n_requested": 2,
"n_rows": 2,
"date_min": "2024-01-02",
"date_max": "2024-02-01",
}
assert (dataset_path / "month=2024-01").exists()
assert (dataset_path / "month=2024-02").exists()
assert not stale_file.exists()
assert written[DATA_COLUMNS].columns.tolist() == DATA_COLUMNS
assert written["symbol_name"].tolist() == ["PF Bank", "PF Bank"]
def test_download_universe_raises_when_all_daily_symbols_empty(tmp_path, monkeypatch):
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000"],
"symbol_name": ["PF Bank"],
}),
)
monkeypatch.setattr(
pipeline_downloader,
"download_daily_batch",
lambda symbols, start, end, adjust="qfq": iter([("sh600000", None)]),
)
with pytest.raises(RuntimeError, match="No data downloaded"):
download_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
)
def test_download_universe_progress_branch_at_100_symbols(tmp_path, monkeypatch):
symbols = [f"sh6{i:05d}" for i in range(100)]
batch_frame = pd.DataFrame({
"symbol": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"preclose": [10.0],
"volume": [1000.0],
"amount": [10500.0],
"vwap": [10.5],
"turn": [1.0],
"pctChg": [5.0],
"tradestatus": [1],
"isST": [0],
"peTTM": [8.0],
"pbMRQ": [1.1],
"psTTM": [2.1],
"pcfNcfTTM": [3.1],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": symbols,
"symbol_name": symbols,
}),
)
def fake_batch(requested_symbols, start, end, adjust="qfq"):
assert requested_symbols == symbols
for symbol in requested_symbols:
yield symbol, batch_frame.copy()
monkeypatch.setattr(pipeline_downloader, "download_daily_batch", fake_batch)
stats = download_universe(
universe="toy100",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
chunk_size=200,
)
assert stats["n_symbols"] == 100
assert stats["n_rows"] == 100
+188
View File
@@ -0,0 +1,188 @@
"""Tests for minute-derived daily feature plugins."""
import textwrap
import numpy as np
import pandas as pd
import pytest
from pipeline.features.compute import compute_feature, validate_feature_frame
from pipeline.features.compute import read_feature_frames
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
from pipeline.features.registry import (
available_features,
get_feature,
load_feature_module,
)
from pipeline.derived.compute import compute_derived
def _minute_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001"],
"symbol_name": ["PF Bank", "PF Bank", "Ping An"],
"datetime": pd.to_datetime([
"2024-01-02 09:35:00",
"2024-01-02 09:40:00",
"2024-01-02 09:35:00",
]),
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
"time": ["09:35:00", "09:40:00", "09:35:00"],
"frequency": ["5m", "5m", "5m"],
"open": [10.0, 10.5, 20.0],
"high": [11.0, 12.0, 21.0],
"low": [9.0, 10.0, 19.0],
"close": [10.5, 11.0, 20.5],
"volume": [100.0, 300.0, 200.0],
"amount": [1000.0, 3300.0, 4100.0],
"vwap": [10.0, 11.0, 20.5],
"adjustflag": ["3", "3", "3"],
})
def test_built_in_minute_daily_summary():
daily = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
"close": [11.0, 20.5, 12.0],
})
result = compute_feature(
minute=_minute_bars(),
daily=daily,
feature_type="minute_daily_summary",
)
assert "minute_daily_summary" in available_features()
row = result[
(result["symbol_id"] == "sh600000")
& (result["date"] == pd.Timestamp("2024-01-02"))
].iloc[0]
assert row["minute_bar_count"] == 2
assert np.isclose(row["minute_intraday_return"], 11.0 / 10.0 - 1.0)
assert np.isclose(row["minute_intraday_range"], 12.0 / 9.0 - 1.0)
assert np.isclose(row["minute_vwap"], 4300.0 / 400.0)
assert np.isclose(row["minute_vwap_deviation"], (4300.0 / 400.0) / 11.0 - 1.0)
missing = result[
(result["symbol_id"] == "sh600000")
& (result["date"] == pd.Timestamp("2024-01-03"))
].iloc[0]
assert pd.isna(missing["minute_vwap"])
def test_minute_daily_summary_feature_preserves_legacy_positional_compute():
direct = MinuteDailySummaryFeature().compute(_minute_bars())
via_registry = get_feature("minute_daily_summary").compute(_minute_bars())
assert "minute_vwap" in direct.columns
pd.testing.assert_frame_equal(direct, via_registry)
def test_legacy_feature_compute_matches_canonical_derived_compute():
daily = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
"close": [11.0, 20.5, 12.0],
})
legacy_feature = compute_feature(
minute=_minute_bars(),
daily=daily,
feature_type="minute_daily_summary",
)
canonical_derived = compute_derived(
"minute_daily_summary",
daily=daily,
minute=_minute_bars(),
)
pd.testing.assert_frame_equal(legacy_feature, canonical_derived)
def test_read_feature_frames_delegates_to_derived_validation(tmp_path):
feature = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": ["2024-01-02 15:00:00"],
"toy_feature": [1.5],
})
feature_path = tmp_path / "feature.pq"
feature.to_parquet(feature_path, index=False)
[result] = read_feature_frames([feature_path])
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
assert result["toy_feature"].tolist() == [1.5]
def test_load_external_feature_module_and_filter_params(tmp_path):
module_path = tmp_path / "external_feature.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.features.base import BaseFeature
from pipeline.features.registry import register_feature
@register_feature
class ExternalVolumeFeature(BaseFeature):
name = "external_volume_feature"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, minute: pd.DataFrame, daily=None) -> pd.DataFrame:
out = (
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
.sum()
.rename(columns={"volume": "scaled_volume"})
)
out["scaled_volume"] *= self.scale
return out
'''))
load_feature_module(str(module_path))
assert "external_volume_feature" in available_features()
instance = get_feature("external_volume_feature", scale=2.0, ignored=99)
assert instance.scale == 2.0
assert not hasattr(instance, "ignored")
result = compute_feature(
minute=_minute_bars(),
feature_type="external_volume_feature",
scale=2.0,
ignored=99,
)
row = result[result["symbol_id"] == "sh600000"].iloc[0]
assert np.isclose(row["scaled_volume"], 800.0)
def test_validate_feature_frame_rejects_missing_keys():
with pytest.raises(ValueError, match="missing required"):
validate_feature_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
def test_validate_feature_frame_rejects_duplicate_keys_after_date_normalization():
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
validate_feature_frame(pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": ["2024-01-02", pd.Timestamp("2024-01-02")],
"x": [1.0, 2.0],
}))
def test_validate_feature_frame_rejects_duplicate_columns():
bad = pd.DataFrame(
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
columns=["symbol_id", "date", "dup", "dup"],
)
with pytest.raises(ValueError, match="duplicate columns"):
validate_feature_frame(bad)
def test_validate_feature_frame_rejects_non_numeric_feature_columns():
with pytest.raises(ValueError, match="numeric"):
validate_feature_frame(pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"bad": ["not numeric"],
}))
+650
View File
@@ -0,0 +1,650 @@
"""Tests for the JoinQuant comparison plugin (network-free)."""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
import pandas as pd
import pytest
from click.testing import CliRunner
from cli import cli
from pipeline.common.schema import FILL_COLUMNS, PNL_COLUMNS, POSITION_COLUMNS
from plugins.joinquant.browser import (
default_browser_config,
load_env_file,
resolve_template,
write_browser_config_template,
)
from plugins.joinquant.export_targets import export_targets
from plugins.joinquant.ingest import (
ingest_joinquant_outputs,
normalize_fills_csv,
)
from plugins.joinquant.reconcile import reconcile_joinquant
from plugins.joinquant.schema import (
JOINQUANT_FILL_COLUMNS,
JOINQUANT_PNL_COLUMNS,
JOINQUANT_POSITION_COLUMNS,
JOINQUANT_TARGET_COLUMNS,
RECONCILE_COLUMNS,
)
from plugins.joinquant.smoke import build_fixed_share_positions
from plugins.joinquant.symbols import from_joinquant_symbol, to_joinquant_symbol
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
def _positions(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
target_value = float(shares * price)
weight = target_value / 1_000_000.0
return pd.DataFrame([{
"symbol_id": symbol,
"date": pd.Timestamp(date),
"portfolio_name": portfolio_name,
"target_weight": weight,
"target_value": target_value,
"target_shares": float(shares) + 0.25,
"position_shares": shares,
"position_value": target_value,
"price": price,
}], columns=POSITION_COLUMNS)
def _our_fills(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
cost: float = 5.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
fills = pd.DataFrame([{
"symbol_id": symbol,
"date": pd.Timestamp(date),
"portfolio_name": portfolio_name,
"prev_shares": 0,
"target_shares": shares,
"traded_shares": shares,
"realized_shares": shares,
"blocked": 0,
"trade_cost": cost,
"trade_price": price,
}])
return fills
def _our_pnl(
*,
date: str = "2026-07-01",
pnl: float = 100.0,
cost: float = 5.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": pd.Timestamp(date),
"portfolio_name": portfolio_name,
"gross_exposure": 10_000.0,
"net_exposure": 10_000.0,
"pnl": pnl,
"cost": cost,
"turnover": 1.0,
"n_positions": 1,
}], columns=PNL_COLUMNS)
def _jq_fills(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
cost: float = 5.0,
portfolio_name: str = "run1",
raw_status: str = "filled",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": date,
"portfolio_name": portfolio_name,
"symbol_id": symbol,
"jq_symbol": to_joinquant_symbol(symbol),
"order_id": "ord-1",
"side": "buy" if shares >= 0 else "sell",
"requested_shares": shares,
"filled_shares": shares,
"fill_price": price,
"trade_value": abs(shares * price),
"trade_cost": cost,
"blocked": 0,
"raw_status": raw_status,
}], columns=JOINQUANT_FILL_COLUMNS)
def _jq_positions(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": date,
"portfolio_name": portfolio_name,
"symbol_id": symbol,
"jq_symbol": to_joinquant_symbol(symbol),
"position_shares": shares,
"position_value": shares * price,
"cash": 990_000.0,
"total_value": 1_000_000.0,
}], columns=JOINQUANT_POSITION_COLUMNS)
def _jq_pnl(
*,
date: str = "2026-07-01",
pnl: float = 100.0,
cost: float = 5.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": date,
"portfolio_name": portfolio_name,
"gross_exposure": 10_000.0,
"net_exposure": 10_000.0,
"cash": 990_000.0,
"total_value": 1_000_000.0,
"pnl": pnl,
"cost": cost,
"turnover": 1.0,
}], columns=JOINQUANT_PNL_COLUMNS)
def _write_parquets(tmp_path: Path, frames: dict[str, pd.DataFrame]) -> dict[str, Path]:
paths = {}
for name, frame in frames.items():
path = tmp_path / f"{name}.pq"
frame.to_parquet(path, index=False)
paths[name] = path
return paths
def _export_targets_for(tmp_path: Path, positions: pd.DataFrame) -> tuple[Path, Path]:
positions_path = tmp_path / "positions.pq"
positions.to_parquet(positions_path, index=False)
targets_root = tmp_path / "targets"
export_targets(
positions_path,
portfolio_name="run1",
out_dir=targets_root,
mode="target_shares",
)
return positions_path, targets_root / "run1"
@pytest.mark.parametrize(
("internal", "joinquant"),
[
("sh600000", "600000.XSHG"),
("sh688001", "688001.XSHG"),
("sz000001", "000001.XSHE"),
("sz001001", "001001.XSHE"),
("sz002594", "002594.XSHE"),
("sz300001", "300001.XSHE"),
],
)
def test_symbol_mapping_both_directions(internal, joinquant):
assert to_joinquant_symbol(internal) == joinquant
assert from_joinquant_symbol(joinquant) == internal
@pytest.mark.parametrize("bad", ["600000", "bj830000", "sh000001", "sz600000", "abc"])
def test_symbol_mapping_rejects_invalid_symbols(bad):
with pytest.raises(ValueError):
to_joinquant_symbol(bad)
@pytest.mark.parametrize("bad", ["600000", "600000.XSHE", "000001.XSHG", "abc.XSHG"])
def test_reverse_symbol_mapping_rejects_invalid_symbols(bad):
with pytest.raises(ValueError):
from_joinquant_symbol(bad)
def test_export_targets_schema_snapshot_hash_and_no_overwrite(tmp_path):
positions_path = tmp_path / "positions.pq"
_positions().to_parquet(positions_path, index=False)
snapshots = export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets",
mode="target_shares",
)
csv_path = tmp_path / "targets" / "run1" / "20260701.csv"
parquet_path = tmp_path / "targets" / "run1" / "20260701.parquet"
snapshot_path = tmp_path / "snapshots" / "run1" / "20260701.json"
assert csv_path.exists()
assert parquet_path.exists()
assert snapshot_path.exists()
target = pd.read_csv(csv_path)
assert list(target.columns) == JOINQUANT_TARGET_COLUMNS
assert int(target.loc[0, "target_shares"]) == 1000
assert float(target.loc[0, "target_value"]) == 10_000.0
assert target.loc[0, "export_mode"] == "target_shares"
snapshot = json.loads(snapshot_path.read_text())
actual_hash = hashlib.sha256(csv_path.read_bytes()).hexdigest()
assert snapshots[0]["file_sha256"] == actual_hash
assert snapshot["file_sha256"] == actual_hash
assert snapshot["n_symbols"] == 1
with pytest.raises(FileExistsError):
export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets",
mode="target_shares",
)
def test_export_targets_target_value_mode_from_position_columns(tmp_path):
positions_path = tmp_path / "positions.pq"
_positions(shares=250, price=20.0).to_parquet(positions_path, index=False)
export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets_value",
mode="target_value",
)
target = pd.read_parquet(tmp_path / "targets_value" / "run1" / "20260701.parquet")
assert list(target.columns) == JOINQUANT_TARGET_COLUMNS
assert target.loc[0, "export_mode"] == "target_value"
assert target.loc[0, "target_value"] == 5_000.0
assert target.loc[0, "target_shares"] == 250
def test_export_targets_can_shift_to_next_execution_session(tmp_path):
positions_path = tmp_path / "positions.pq"
_positions(date="2024-01-09").to_parquet(positions_path, index=False)
calendar_path = tmp_path / "daily.pq"
pd.DataFrame({
"date": pd.to_datetime(["2024-01-09", "2024-01-10", "2024-01-11"]),
"symbol_id": ["sh600000", "sh600000", "sh600000"],
}).to_parquet(calendar_path, index=False)
snapshots = export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets_shifted",
mode="target_shares",
start_date="2024-01-10",
end_date="2024-01-10",
execution_calendar_path=calendar_path,
)
assert len(snapshots) == 1
assert snapshots[0]["date"] == "2024-01-10"
assert (tmp_path / "targets_shifted" / "run1" / "20240110.csv").exists()
target = pd.read_csv(tmp_path / "targets_shifted" / "run1" / "20240110.csv")
assert target.loc[0, "date"] == "2024-01-10"
def test_ingest_permissive_csv_column_mapping_and_output_schemas(tmp_path):
fills_csv = tmp_path / "jq_fills.csv"
positions_csv = tmp_path / "jq_positions.csv"
pnl_csv = tmp_path / "jq_pnl.csv"
pd.DataFrame([{
"Trade Date": "2026-07-01 09:31:00",
"Security": "600000.XSHG",
"Direction": "buy",
"Order Amount": 1000,
"Filled Amount": 1000,
"Price": 10.0,
"Status": "filled",
}]).to_csv(fills_csv, index=False)
pd.DataFrame([{
"Date": "2026-07-01",
"Security": "600000.XSHG",
"Shares": 1000,
"Market Value": 10_000.0,
"Cash": 990_000.0,
"Portfolio Value": 1_000_000.0,
}]).to_csv(positions_csv, index=False)
pd.DataFrame([{
"Date": "2026-07-01",
"Portfolio Value": 1_000_000.0,
"Daily PnL": 100.0,
"Turnover": 1.0,
}]).to_csv(pnl_csv, index=False)
fills = normalize_fills_csv(fills_csv, "run1")
assert list(fills.columns) == JOINQUANT_FILL_COLUMNS
assert fills.loc[0, "symbol_id"] == "sh600000"
assert fills.loc[0, "jq_symbol"] == "600000.XSHG"
assert fills.loc[0, "trade_cost"] == 0.0
assert fills.loc[0, "blocked"] == 0
paths = ingest_joinquant_outputs(
portfolio_name="run1",
fills_csv=fills_csv,
positions_csv=positions_csv,
pnl_csv=pnl_csv,
out_dir=tmp_path / "ingested",
)
assert list(pd.read_parquet(paths["fills"]).columns) == JOINQUANT_FILL_COLUMNS
assert list(pd.read_parquet(paths["positions"]).columns) == JOINQUANT_POSITION_COLUMNS
assert list(pd.read_parquet(paths["pnl"]).columns) == JOINQUANT_PNL_COLUMNS
def _run_reconcile_case(
tmp_path: Path,
*,
positions: pd.DataFrame | None = None,
our_fills: pd.DataFrame | None = None,
jq_fills: pd.DataFrame | None = None,
jq_positions: pd.DataFrame | None = None,
our_pnl: pd.DataFrame | None = None,
jq_pnl: pd.DataFrame | None = None,
) -> pd.DataFrame:
positions = _positions() if positions is None else positions
_, targets_dir = _export_targets_for(tmp_path, positions)
paths = _write_parquets(tmp_path, {
"our_fills": _our_fills() if our_fills is None else our_fills,
"our_positions": positions,
"our_pnl": _our_pnl() if our_pnl is None else our_pnl,
"jq_fills": _jq_fills() if jq_fills is None else jq_fills,
"jq_positions": _jq_positions() if jq_positions is None else jq_positions,
"jq_pnl": _jq_pnl() if jq_pnl is None else jq_pnl,
})
out_paths = reconcile_joinquant(
portfolio_name="run1",
targets_dir=targets_dir,
our_fills_path=paths["our_fills"],
our_positions_path=paths["our_positions"],
our_pnl_path=paths["our_pnl"],
jq_fills_path=paths["jq_fills"],
jq_positions_path=paths["jq_positions"],
jq_pnl_path=paths["jq_pnl"],
out_dir=tmp_path / "reconcile",
)
report = pd.read_parquet(out_paths["daily_reconcile"])
assert list(report.columns) == RECONCILE_COLUMNS
assert out_paths["summary_md"].exists()
assert out_paths["summary_csv"].exists()
return report
def test_reconcile_exact_match(tmp_path):
report = _run_reconcile_case(tmp_path)
assert report.loc[0, "diff_reason"] == "MATCH"
assert report.loc[0, "filled_share_diff"] == 0
assert report.loc[0, "position_share_diff"] == 0
def test_reconcile_price_mismatch(tmp_path):
report = _run_reconcile_case(tmp_path, jq_fills=_jq_fills(price=10.5))
assert report.loc[0, "diff_reason"] == "PRICE_MISMATCH"
def test_reconcile_cost_mismatch(tmp_path):
report = _run_reconcile_case(
tmp_path,
jq_fills=_jq_fills(cost=8.0),
jq_pnl=_jq_pnl(cost=8.0),
)
assert report.loc[0, "diff_reason"] == "COST_MODEL"
def test_reconcile_missing_symbol_in_joinquant(tmp_path):
empty_jq_fills = pd.DataFrame(columns=JOINQUANT_FILL_COLUMNS)
empty_jq_positions = pd.DataFrame(columns=JOINQUANT_POSITION_COLUMNS)
report = _run_reconcile_case(
tmp_path,
jq_fills=empty_jq_fills,
jq_positions=empty_jq_positions,
)
assert report.loc[0, "diff_reason"] == "MISSING_IN_JOINQUANT"
def test_reconcile_short_target_with_long_only_joinquant_output(tmp_path):
positions = _positions(shares=-100, price=10.0)
our_fills = _our_fills(shares=-100, price=10.0)
jq_fills = _jq_fills(shares=0, price=10.0, cost=0.0, raw_status="short clipped")
jq_positions = _jq_positions(shares=0, price=10.0)
report = _run_reconcile_case(
tmp_path,
positions=positions,
our_fills=our_fills,
jq_fills=jq_fills,
jq_positions=jq_positions,
)
assert report.loc[0, "diff_reason"] == "SHORT_NOT_SUPPORTED"
def test_joinquant_cli_smoke_export_ingest_reconcile_and_wrapper(tmp_path):
runner = CliRunner()
positions_path = tmp_path / "positions.pq"
_positions().to_parquet(positions_path, index=False)
result = runner.invoke(cli, [
"joinquant", "export-targets",
"--positions-path", str(positions_path),
"--portfolio-name", "run1",
"--mode", "target_shares",
"--out-dir", str(tmp_path / "targets"),
])
assert result.exit_code == 0, result.output
assert "Exported JoinQuant targets" in result.output
fills_csv = tmp_path / "jq_fills.csv"
positions_csv = tmp_path / "jq_positions.csv"
pnl_csv = tmp_path / "jq_pnl.csv"
_jq_fills().to_csv(fills_csv, index=False)
_jq_positions().to_csv(positions_csv, index=False)
_jq_pnl().to_csv(pnl_csv, index=False)
result = runner.invoke(cli, [
"joinquant", "ingest",
"--portfolio-name", "run1",
"--fills-csv", str(fills_csv),
"--positions-csv", str(positions_csv),
"--pnl-csv", str(pnl_csv),
"--out-dir", str(tmp_path / "ingested"),
])
assert result.exit_code == 0, result.output
assert "Saved JoinQuant fills" in result.output
paths = _write_parquets(tmp_path, {
"our_fills": _our_fills(),
"our_pnl": _our_pnl(),
})
result = runner.invoke(cli, [
"joinquant", "reconcile",
"--portfolio-name", "run1",
"--targets-dir", str(tmp_path / "targets" / "run1"),
"--our-fills-path", str(paths["our_fills"]),
"--our-positions-path", str(positions_path),
"--our-pnl-path", str(paths["our_pnl"]),
"--jq-fills-path", str(tmp_path / "ingested" / "run1" / "fills.pq"),
"--jq-positions-path", str(tmp_path / "ingested" / "run1" / "positions.pq"),
"--jq-pnl-path", str(tmp_path / "ingested" / "run1" / "pnl.pq"),
"--out-dir", str(tmp_path / "reconcile"),
])
assert result.exit_code == 0, result.output
assert "Saved reconciliation parquet" in result.output
wrapper_path = tmp_path / "wrapper_strategy_run1.py"
result = runner.invoke(cli, [
"joinquant", "write-wrapper",
"--portfolio-name", "run1",
"--mode", "target_shares",
"--out-path", str(wrapper_path),
])
assert result.exit_code == 0, result.output
assert "Saved JoinQuant wrapper strategy" in result.output
text = wrapper_path.read_text()
assert 'PORTFOLIO_NAME = "run1"' in text
assert 'TARGET_MODE = "target_shares"' in text
assert "ALLOW_SHORT = False" in text
def test_wrapper_strategy_generation_smoke(tmp_path):
path = write_wrapper_strategy(
portfolio_name="run2",
mode="target_value",
out_path=tmp_path / "wrapper.py",
)
text = path.read_text()
assert 'PORTFOLIO_NAME = "run2"' in text
assert 'TARGET_MODE = "target_value"' in text
assert "order_target_value" in text
def test_wrapper_strategy_can_embed_target_csvs(tmp_path):
targets_dir = tmp_path / "targets"
targets_dir.mkdir()
(targets_dir / "20260701.csv").write_text(
"date,portfolio_name,symbol_id,jq_symbol,target_shares,target_value,export_mode\n"
"2026-07-01,run1,sh600000,600000.XSHG,1000,10000,target_shares\n",
encoding="utf-8",
)
path = write_wrapper_strategy(
portfolio_name="run1",
mode="target_shares",
out_path=tmp_path / "wrapper_embedded.py",
embedded_targets_dir=targets_dir,
)
text = path.read_text()
assert '"20260701.csv"' in text
assert "600000.XSHG" in text
assert "if file_name in _EMBEDDED_TARGETS" in text
def test_build_fixed_share_positions_excludes_final_executionless_date():
data = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sh600000"],
"date": pd.to_datetime(["2024-01-09", "2024-01-10", "2024-01-11"]),
"close": [10.0, 10.5, 11.0],
})
positions = build_fixed_share_positions(
data,
trade_symbol="sh600000",
portfolio_name="run1",
shares=1000,
booksize=1_000_000.0,
)
assert list(positions.columns) == POSITION_COLUMNS
assert positions["date"].dt.strftime("%Y-%m-%d").tolist() == [
"2024-01-09",
"2024-01-10",
]
assert positions["position_shares"].tolist() == [1000, 1000]
assert positions["target_value"].tolist() == [10_000.0, 10_500.0]
def test_browser_config_template_and_placeholder_resolution(tmp_path):
config_path = write_browser_config_template(
tmp_path / "browser_config.json",
strategy_url="https://www.joinquant.com/example",
)
config = json.loads(config_path.read_text())
assert config["strategy_url"] == "https://www.joinquant.com/example"
assert config["actions"][0]["type"] == "goto"
context = {
"wrapper_path": "/tmp/wrapper.py",
"target_csvs": ["/tmp/20240110.csv", "/tmp/20240111.csv"],
"expected_joinquant_csvs": {"fills": "/tmp/jq_fills.csv"},
}
assert resolve_template("{wrapper_path}", context) == "/tmp/wrapper.py"
assert resolve_template("{target_csvs}", context) == [
"/tmp/20240110.csv",
"/tmp/20240111.csv",
]
assert resolve_template("save:{expected_joinquant_csvs.fills}", context) == "save:/tmp/jq_fills.csv"
script = "(arg) => { return new Event('change', {bubbles: true}); }"
assert resolve_template(script, context) == script
def test_load_env_file_handles_quotes_without_shell_sourcing(tmp_path):
env_path = tmp_path / "joinquant.env"
env_path.write_text(
"JOINQUANT_USERNAME=alice\n"
"JOINQUANT_PASSWORD=\"secret\"\n"
"JOINQUANT_STRATEGY_URL='https://example.test/path?x=1&y=2'\n"
)
env = load_env_file(env_path)
assert env["JOINQUANT_USERNAME"] == "alice"
assert env["JOINQUANT_PASSWORD"] == "secret"
assert env["JOINQUANT_STRATEGY_URL"] == "https://example.test/path?x=1&y=2"
def test_sim_trade_browser_config_template(tmp_path):
config_path = write_browser_config_template(
tmp_path / "sim_config.json",
strategy_url="https://www.joinquant.com/sim",
flow="sim-trade",
)
config = json.loads(config_path.read_text())
assert config["flow"] == "sim-trade"
selectors = " ".join(
action.get("selector", "") for action in config["actions"]
)
assert "模拟盘" in selectors
assert "模拟交易" in selectors
assert any(
action["type"] == "screenshot"
and action["path"] == "{run_artifact_dir}/sim_trade_final.png"
for action in config["actions"]
)
def test_joinquant_cli_browser_config_smoke(tmp_path):
runner = CliRunner()
config_path = tmp_path / "browser_config.json"
result = runner.invoke(cli, [
"joinquant",
"write-browser-config",
"--out-path",
str(config_path),
"--strategy-url",
"https://www.joinquant.com/example",
])
assert result.exit_code == 0, result.output
assert config_path.exists()
assert default_browser_config()["actions"]
sim_config_path = tmp_path / "sim_browser_config.json"
result = runner.invoke(cli, [
"joinquant",
"write-browser-config",
"--out-path",
str(sim_config_path),
"--strategy-url",
"https://www.joinquant.com/sim",
"--flow",
"sim-trade",
])
assert result.exit_code == 0, result.output
assert json.loads(sim_config_path.read_text())["flow"] == "sim-trade"
+369
View File
@@ -0,0 +1,369 @@
"""Tests for raw Baostock minute bar download plumbing."""
from pathlib import Path
import numpy as np
import pandas as pd
import pytest
import data.downloader as low_level_downloader
import pipeline.data.downloader as pipeline_downloader
from data.downloader import download_minute_batch
from pipeline.common.schema import MINUTE_BAR_COLUMNS
from pipeline.data.downloader import download_minute_universe
class _FakeResult:
def __init__(self, rows, error_code="0", error_msg=""):
self.rows = rows
self.error_code = error_code
self.error_msg = error_msg
self._idx = -1
def next(self):
self._idx += 1
return self._idx < len(self.rows)
def get_row_data(self):
return self.rows[self._idx]
def test_download_minute_batch_maps_and_parses_baostock_rows(monkeypatch):
rows = [
[
"2024-01-02",
"20240102093500000",
"sh.600000",
"10",
"11",
"9",
"10.5",
"1000",
"10500",
"3",
],
[
"2024-01-02",
"20240102094000000",
"sh.600000",
"10.5",
"12",
"10",
"11",
"2000",
"22000",
"3",
],
]
calls = []
def fake_query(**kwargs):
calls.append(kwargs)
return _FakeResult(rows)
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
fake_query,
)
[(symbol, df)] = list(
download_minute_batch(
["sh600000"],
"2024-01-02",
"2024-01-02",
frequency=5,
)
)
assert symbol == "sh600000"
assert calls[0]["code"] == "sh.600000"
assert calls[0]["frequency"] == "5"
assert calls[0]["adjustflag"] == "3"
assert df is not None
assert df["datetime"].iloc[0] == pd.Timestamp("2024-01-02 09:35:00")
assert df["time"].tolist() == ["09:35:00", "09:40:00"]
assert (df["frequency"] == "5m").all()
assert np.isclose(df["open"].iloc[0], 10.0)
assert np.isclose(df["vwap"].iloc[0], 10.5)
assert pd.api.types.is_numeric_dtype(df["volume"])
def test_minute_frequency_and_timestamp_parsing_edge_cases():
frequency, label = low_level_downloader._normalize_minute_frequency("15m")
assert (frequency, label) == ("15", "15m")
with pytest.raises(ValueError, match="Unsupported minute frequency"):
low_level_downloader._normalize_minute_frequency("1m")
parsed = low_level_downloader._parse_minute_datetime(
pd.Series(["2024-01-02", "2024-01-02"]),
pd.Series(["0935", "09:40:00"]),
)
assert parsed.tolist() == [
pd.Timestamp("2024-01-02 09:35:00"),
pd.Timestamp("2024-01-02 09:40:00"),
]
def test_download_minute_batch_empty_result_yields_none(monkeypatch):
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([]),
)
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_minute_batch_non_login_error_and_periodic_relogin(monkeypatch):
responses = [
_FakeResult([], error_code="1", error_msg="bad symbol"),
_FakeResult([]),
]
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
monkeypatch.setattr(low_level_downloader.bs, "login", fake_login)
monkeypatch.setattr(low_level_downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(
download_minute_batch(
["sh600000", "sz000001"],
"2024-01-02",
"2024-01-02",
relogin_every=1,
)
) == [("sh600000", None), ("sz000001", None)]
assert login_count == 2
assert logout_count == 2
def test_download_minute_batch_second_session_loss_yields_none(monkeypatch):
responses = [
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
]
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_minute_batch_ignores_relogin_and_final_logout_failures(monkeypatch):
responses = [
_FakeResult([], error_code="1", error_msg="bad symbol"),
_FakeResult([]),
]
logout_count = 0
def fake_logout():
nonlocal logout_count
logout_count += 1
raise RuntimeError("logout failed")
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(
download_minute_batch(
["sh600000", "sz000001"],
"2024-01-02",
"2024-01-02",
relogin_every=1,
)
) == [("sh600000", None), ("sz000001", None)]
assert logout_count == 2
def test_download_minute_batch_rejects_unparsed_timestamps(monkeypatch):
bad_rows = [[
"2024-01-02",
"not-a-time",
"sh.600000",
"10",
"11",
"9",
"10.5",
"1000",
"10500",
"3",
]]
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult(bad_rows),
)
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_minute_universe_writes_frequency_month_partitions(tmp_path, monkeypatch):
minute = pd.DataFrame({
"symbol": ["sh600000", "sh600000"],
"datetime": pd.to_datetime(["2024-01-02 09:35:00", "2024-01-02 09:40:00"]),
"date": pd.to_datetime(["2024-01-02", "2024-01-02"]),
"time": ["09:35:00", "09:40:00"],
"frequency": ["5m", "5m"],
"open": [10.0, 10.5],
"high": [11.0, 12.0],
"low": [9.0, 10.0],
"close": [10.5, 11.0],
"volume": [1000.0, 2000.0],
"amount": [10500.0, 22000.0],
"vwap": [10.5, 11.0],
"adjustflag": ["3", "3"],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000"],
"symbol_name": ["PF Bank"],
}),
)
def fake_batch(symbols, start, end, frequency=5):
assert symbols == ["sh600000"]
assert frequency == "5"
yield "sh600000", minute
monkeypatch.setattr(pipeline_downloader, "download_minute_batch", fake_batch)
preserved = tmp_path / "toy" / "frequency=15m" / "month=2024-01" / "old.pq"
preserved.parent.mkdir(parents=True)
preserved_minute = minute.copy()
preserved_minute["frequency"] = "15m"
preserved_minute["symbol_id"] = "sh600000"
preserved_minute["symbol_name"] = "PF Bank"
preserved_minute[MINUTE_BAR_COLUMNS].to_parquet(preserved, index=False)
stale = tmp_path / "toy" / "frequency=5m" / "month=2024-01" / "stale.pq"
stale.parent.mkdir(parents=True)
preserved_minute.assign(frequency="5m")[MINUTE_BAR_COLUMNS].to_parquet(stale, index=False)
stats = download_minute_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
chunk_size=1,
frequency="5",
)
dataset_path = Path(stats["dataset_path"])
assert (dataset_path / "frequency=5m" / "month=2024-01").is_dir()
assert preserved.exists()
assert not stale.exists()
out = pd.read_parquet(dataset_path / "frequency=5m")
assert (set(MINUTE_BAR_COLUMNS) - {"frequency"}) <= set(out.columns)
assert set(out["symbol_id"]) == {"sh600000"}
assert set(out["symbol_name"]) == {"PF Bank"}
assert stats["n_rows"] == 2
def test_download_minute_universe_raises_when_all_symbols_empty(tmp_path, monkeypatch):
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000"],
"symbol_name": ["PF Bank"],
}),
)
monkeypatch.setattr(
pipeline_downloader,
"download_minute_batch",
lambda symbols, start, end, frequency=5: iter([("sh600000", None)]),
)
with pytest.raises(RuntimeError, match="No minute data"):
download_minute_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
)
def test_download_minute_universe_progress_branch_at_100_symbols(tmp_path, monkeypatch):
symbols = [f"sh6{i:05d}" for i in range(100)]
minute = pd.DataFrame({
"symbol": ["sh600000"],
"datetime": [pd.Timestamp("2024-01-02 09:35:00")],
"date": [pd.Timestamp("2024-01-02")],
"time": ["09:35:00"],
"frequency": ["5m"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
"vwap": [10.5],
"adjustflag": ["3"],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": symbols,
"symbol_name": symbols,
}),
)
def fake_batch(requested_symbols, start, end, frequency=5):
assert requested_symbols == symbols
for symbol in requested_symbols:
yield symbol, minute.copy()
monkeypatch.setattr(pipeline_downloader, "download_minute_batch", fake_batch)
stats = download_minute_universe(
universe="toy100",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
chunk_size=200,
)
assert stats["n_symbols"] == 100
assert stats["n_rows"] == 100
+305
View File
@@ -0,0 +1,305 @@
"""End-to-end correctness invariants for the reversal_5d pipeline (no network).
Each test maps 1:1 to one of the ten review checks. Naming convention: the
*execution date* ``d`` is the market session on which a target is actually
filled at the open; the *signal date* ``t`` is the session whose close formed
that target. The documented convention is ``d = next(t)`` (see
``docs/portfolio_trading_cost_model.md``), so ``close[d-1] == close[t]``.
"""
import numpy as np
import pandas as pd
from pipeline.alpha.compute import compute_alpha, investable_universe_mask
from pipeline.portfolio.construct import construct_positions
from pipeline.portfolio.discretize import repair_exposure, round_to_valid_lot
from pipeline.portfolio.market_rules import MarketRule
from pipeline.portfolio.costs import SimpleProportionalCostModel
from pipeline.portfolio.constraints import (
SuspensionConstraint,
VolumeCapConstraint,
)
from pipeline.portfolio.simulator import ReferenceSimulator
_SYMBOLS = ("sh600000", "sz000001", "sh688981", "sz300750")
def _panel(n_days=12, symbols=_SYMBOLS, start="2024-01-01", seed=0,
distinct_open=True):
"""Contiguous long-format DATA frame with all columns the pipeline needs.
Open and close differ (so overnight vs intraday PnL terms are separable),
and the calendar is gap-free so each session is the next session's ``d-1``.
"""
dates = pd.date_range(start, periods=n_days)
rng = np.random.default_rng(seed)
frames = []
for i, sym in enumerate(symbols):
close = np.abs(50.0 + i * 10 + np.cumsum(rng.standard_normal(n_days))) + 5.0
open_ = close * (0.99 + 0.02 * rng.random(n_days)) if distinct_open else close.copy()
preclose = np.concatenate([[close[0]], close[:-1]])
frames.append(pd.DataFrame({
"symbol_id": sym,
"symbol_name": sym,
"date": dates,
"open": open_,
"high": np.maximum(open_, close),
"low": np.minimum(open_, close),
"close": close,
"preclose": preclose,
"volume": 1_000_000.0,
"amount": 1_000_000.0 * close,
"tradestatus": 1,
"isST": 0,
}))
return pd.concat(frames, ignore_index=True)
# --- 1. reversal signal uses only close[d-1] and earlier ---------------------
def test_reversal_signal_does_not_peek_at_future_closes():
data = _panel(n_days=12)
base = compute_alpha(data, "rev", "reversal_rank", lookback=5)
# Perturb every close strictly AFTER an interior signal date t; the weight
# dated t (executed at d = t+1) must be unchanged — it may use close[t]
# (== close[d-1]) and earlier only.
t = sorted(data["date"].unique())[6]
future = data.copy()
mask = future["date"] > t
future.loc[mask, ["open", "high", "low", "close"]] *= 1.5
perturbed = compute_alpha(future, "rev", "reversal_rank", lookback=5)
b = base[base["date"] <= t].set_index(["symbol_id", "date"])["weight"]
p = perturbed[perturbed["date"] <= t].set_index(["symbol_id", "date"])["weight"]
pd.testing.assert_series_equal(b.sort_index(), p.sort_index())
# --- 2. the executed (fill/PnL) date is the open-execution date d = next(t) --
def test_fill_date_is_next_session_open_execution_date():
data = _panel(n_days=8)
weights = compute_alpha(data, "c", "reversal_rank", lookback=3)
weights = weights.rename(columns={"alpha_name": "combo_name"})
weights["combo_name"] = "c"
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(pos, data)
sessions = sorted(data["date"].unique())
nxt = {s: sessions[i + 1] for i, s in enumerate(sessions[:-1])}
# Every executed date equals the session AFTER some position (signal) date.
pos_dates = set(pos["date"].unique())
exec_dates = set(pnl["date"].unique())
assert exec_dates == {nxt[t] for t in pos_dates if t in nxt}
# Execution price is the open of the execution date, not the signal close.
opn = data.pivot_table(index="date", columns="symbol_id", values="open",
aggfunc="first").sort_index()
d = sorted(exec_dates)[1]
row = fills[(fills["date"] == d) & (fills["traded_shares"] != 0)].iloc[0]
sym = row["symbol_id"]
expected_cost = abs(row["traded_shares"]) * opn.loc[d, sym] * (5 + 5) / 1e4
assert np.isclose(row["trade_cost"], expected_cost)
# --- 3. PnL identity: overnight(old book) + intraday(new book) - cost --------
def test_daily_pnl_matches_overnight_plus_intraday_minus_cost():
data = _panel(n_days=8)
weights = compute_alpha(data, "c", "reversal_rank", lookback=3)
weights = weights.rename(columns={"alpha_name": "combo_name"})
weights["combo_name"] = "c"
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(pos, data)
opn = data.pivot_table(index="date", columns="symbol_id", values="open", aggfunc="first").sort_index()
cls = data.pivot_table(index="date", columns="symbol_id", values="close", aggfunc="first").sort_index()
sessions = list(cls.index)
prev_close_of = {sessions[i]: sessions[i - 1] for i in range(1, len(sessions))}
for d in sorted(pnl["date"].unique()):
day = fills[fills["date"] == d]
prev = day.set_index("symbol_id")["prev_shares"]
realized = day.set_index("symbol_id")["realized_shares"]
cost = day["trade_cost"].sum()
intraday = float((realized * (cls.loc[d] - opn.loc[d]).reindex(realized.index)).sum())
# Overnight gap on the OLD book is taken from the previous *executed*
# date's close. With a gap-free calendar and daily execution that is the
# immediately preceding session; the first executed date has no prior
# book so the term is naturally zero (prev_shares == 0 there).
pc = prev_close_of.get(d)
if pc is not None and (prev != 0).any():
overnight = float((prev * (opn.loc[d] - cls.loc[pc]).reindex(prev.index)).sum())
else:
overnight = 0.0
expected = overnight + intraday - cost
got = float(pnl[pnl["date"] == d]["pnl"].iloc[0])
assert np.isclose(got, expected, rtol=1e-6, atol=1e-3), (d, got, expected)
# --- 4. realized shares (not target shares) are threaded into the next day ---
def test_realized_not_target_threaded_forward():
data = _panel(n_days=6)
weights = compute_alpha(data, "c", "reversal_rank", lookback=2)
weights = weights.rename(columns={"alpha_name": "combo_name"})
weights["combo_name"] = "c"
pos = construct_positions(weights, data, booksize=1e8, portfolio_name="run1")
# A tight volume cap forces partial fills, so realized != target on most
# names — exactly the case where threading target vs realized diverges.
fills, _ = ReferenceSimulator(
constraints=[VolumeCapConstraint(max_frac=1e-6)]
).run(pos, data)
wide_prev = fills.pivot_table(index="date", columns="symbol_id", values="prev_shares", aggfunc="first")
wide_real = fills.pivot_table(index="date", columns="symbol_id", values="realized_shares", aggfunc="first")
exec_dates = list(wide_prev.index)
assert len(exec_dates) >= 2
# Today's prev_shares == yesterday's realized_shares for every name.
for a, b in zip(exec_dates[:-1], exec_dates[1:]):
prev_today = wide_prev.loc[b].dropna()
real_yest = wide_real.loc[a].reindex(prev_today.index).fillna(0.0)
pd.testing.assert_series_equal(
prev_today.astype(float), real_yest.astype(float), check_names=False
)
# And realized actually diverged from target (cap bit), so the test is real.
assert (fills["realized_shares"] != fills["target_shares"]).any()
# --- 5. blocked trades create zero traded_shares and zero trade_cost ---------
def test_blocked_trade_has_zero_shares_and_zero_cost():
data = _panel(n_days=6)
# Suspend one name on every session so any attempt to trade it is blocked.
data.loc[data["symbol_id"] == "sz000001", "tradestatus"] = 0
weights = compute_alpha(data, "c", "reversal_rank", lookback=2)
weights = weights.rename(columns={"alpha_name": "combo_name"})
weights["combo_name"] = "c"
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
fills, _ = ReferenceSimulator(
constraints=[SuspensionConstraint()], cost_bps=5, slippage_bps=5
).run(pos, data)
blocked = fills[fills["blocked"] == 1]
assert (blocked["traded_shares"] == 0).all()
assert (blocked["trade_cost"] == 0.0).all()
# The suspended name never trades and never accrues cost.
susp = fills[fills["symbol_id"] == "sz000001"]
assert (susp["traded_shares"] == 0).all()
assert (susp["trade_cost"] == 0.0).all()
# --- 6. liquid universe uses only information known before open[d] -----------
def test_investable_universe_mask_is_causal():
data = _panel(n_days=14)
close = data.pivot_table(index="date", columns="symbol_id", values="close", aggfunc="first").sort_index()
full = investable_universe_mask(data, close, top_n=10, min_history=3)
t = sorted(data["date"].unique())[8]
# Recompute the mask from data truncated at the signal date t: the mask row
# for t must be identical, proving it never reads dates > t (i.e. nothing
# from open[d=t+1] onward).
trunc = data[data["date"] <= t]
close_t = close.loc[:t]
mask_t = investable_universe_mask(trunc, close_t, top_n=10, min_history=3)
pd.testing.assert_series_equal(
full.loc[t].sort_index(), mask_t.loc[t].sort_index(), check_names=False
)
# --- 7. cost bps is one-way per-trade (a round trip is charged twice) --------
def test_cost_bps_is_one_way_per_trade():
model = SimpleProportionalCostModel(cost_bps=5, slippage_bps=5)
price = np.array([20.0])
buy = model.compute(np.array([1000]), price, np.array([1]), date=None)
sell = model.compute(np.array([-1000]), price, np.array([-1]), date=None)
one_way = 1000 * 20 * (5 + 5) / 1e4
assert np.isclose(buy[0], one_way) # charged once on the buy leg
assert np.isclose(sell[0], one_way) # charged again on the sell leg
# A full round trip (enter then exit) therefore costs ~2x the one-way rate.
assert np.isclose(buy[0] + sell[0], 2 * one_way)
# --- 8. execution & PnL use raw tradable prices on the same scale as shares --
def test_position_value_is_shares_times_raw_price():
data = _panel(n_days=10)
weights = compute_alpha(data, "c", "reversal_rank", lookback=3)
weights = weights.rename(columns={"alpha_name": "combo_name"})
weights["combo_name"] = "c"
pos = construct_positions(weights, data, booksize=1e6, portfolio_name="run1")
finite = pos["price"] > 0
# The stored value is exactly integer shares × the raw construction price —
# no adjusted-price factor is mixed into the share→value accounting.
expected = pos.loc[finite, "position_shares"] * pos.loc[finite, "price"]
pd.testing.assert_series_equal(
pos.loc[finite, "position_value"].astype(float),
expected.astype(float),
check_names=False,
)
# --- 9. alpha is scale-free (adjusted prices ok); accounting uses raw units --
def test_alpha_weights_invariant_to_per_symbol_price_scaling():
data = _panel(n_days=12)
base = compute_alpha(data, "rev", "reversal_rank", lookback=5)
# A qfq/hfq adjustment is (per symbol) a multiplicative rescaling of the
# price series; pct_change is scale-free, so the alpha weights must not move.
scaled = data.copy()
factor = {"sh600000": 2.0, "sz000001": 0.5, "sh688981": 3.0, "sz300750": 1.25}
for sym, f in factor.items():
m = scaled["symbol_id"] == sym
scaled.loc[m, ["open", "high", "low", "close"]] *= f
scaled_alpha = compute_alpha(scaled, "rev", "reversal_rank", lookback=5)
b = base.set_index(["symbol_id", "date"])["weight"].sort_index()
s = scaled_alpha.set_index(["symbol_id", "date"])["weight"].sort_index()
pd.testing.assert_series_equal(b, s)
# --- 10. repaired book stays on valid A-share lot lattices -------------------
def _on_lattice(q, min_open, increment):
q = np.abs(np.asarray(q, dtype=np.int64))
on = (q == 0) | ((q >= min_open) & ((q - min_open) % increment == 0))
return bool(on.all())
def test_repair_output_stays_on_lot_lattice():
# Pre-2023 main board has a 100-share increment (the strongest lattice
# constraint); STAR uses min 200 / increment 1.
symbols = np.array(["sh600000", "sz000001", "sh688981", "sz300750"], dtype=object)
rule = MarketRule()
on = "2022-06-01" # pre 2023-08-10 → main-board increment is 100
min_open, increment, odd_full, _ = rule.get_rules_vectorized(
symbols, on, np.zeros(len(symbols), dtype=bool)
)
assert min_open[0] == 100 and increment[0] == 100 # main board pre-2023
price = np.array([12.3, 8.7, 45.0, 230.0])
prev = np.zeros(len(symbols), dtype=np.int64)
q_target = np.array([3251.0, -7777.0, 640.0, -415.0])
q_round = round_to_valid_lot(q_target, prev, min_open, increment, odd_full)
assert _on_lattice(q_round, min_open, increment)
repaired = repair_exposure(
q_round, q_target, price, increment, min_open, prev, odd_full,
booksize=float(np.abs(q_target * price).sum()),
)
assert _on_lattice(repaired, min_open, increment)
# Repair never flips a name's sign relative to the rounded book.
nz = q_round != 0
assert np.all(np.sign(repaired[nz]) * np.sign(q_round[nz]) >= 0)
+488 -1
View File
@@ -13,15 +13,21 @@ from pipeline.portfolio.market_rules import (
Board,
LimitStatus,
MarketRule,
_to_date,
compute_limit_status,
detect_board,
)
from pipeline.portfolio.research import evaluate_portfolio
from pipeline.portfolio.constraints import (
TradeConstraint,
available_constraints,
get_constraint,
PriceLimitConstraint,
register_constraint,
SuspensionConstraint,
VolumeCapConstraint,
)
from pipeline.portfolio.costs import SimpleProportionalCostModel
from pipeline.portfolio.simulator import (
MarketSlice,
ReferenceSimulator,
@@ -64,7 +70,7 @@ def _make_data(n_days: int = 40, symbols=_SYMBOLS, start="2024-01-01",
def _make_weights(data: pd.DataFrame, name="combo") -> pd.DataFrame:
"""Demeaned per-date signed weights so the cross-section is dollar-neutral."""
close = data.pivot_table(index="date", columns="symbol_id", values="close").sort_index()
raw = -close.pct_change(5)
raw = -close.pct_change(5, fill_method=None)
demeaned = raw.sub(raw.mean(axis=1), axis=0)
long = demeaned.reset_index().melt(id_vars="date", var_name="symbol_id",
value_name="weight").dropna()
@@ -81,6 +87,7 @@ def test_detect_board():
assert detect_board("sh688981") == Board.STAR
assert detect_board("sz300750") == Board.CHINEXT
assert detect_board("bj830000") == Board.UNKNOWN
assert detect_board("sh") == Board.UNKNOWN
# --- MarketRule date transitions ---------------------------------------------
@@ -119,6 +126,26 @@ def test_get_rules_vectorized():
assert list(limit) == [0.10, 0.20, 0.20]
def test_market_rule_date_coercion_unknown_board_and_st_vector_override():
rules = MarketRule()
assert _to_date(dt.datetime(2024, 1, 2, 15, 0)) == dt.date(2024, 1, 2)
assert _to_date(pd.Timestamp("2024-01-03 09:30")) == dt.date(2024, 1, 3)
assert _to_date("2024-01-04 10:00:00") == dt.date(2024, 1, 4)
unknown_rule = rules.get_rule("xx999999", "2024-01-02")
assert unknown_rule.minimum_open_size == 100
assert unknown_rule.share_increment == 100
assert unknown_rule.price_limit_pct == 0.10
_, _, _, limit = rules.get_rules_vectorized(
np.array(["sh600000", "sh600000"], dtype=object),
"2024-01-02",
np.array([0, 1]),
)
assert limit.tolist() == [0.10, 0.05]
def test_compute_limit_status():
price = np.array([110.0, 90.0, 100.0])
preclose = np.array([100.0, 100.0, 100.0])
@@ -129,6 +156,20 @@ def test_compute_limit_status():
assert status[2] == LimitStatus.NORMAL.value
def test_compute_limit_status_treats_bad_preclose_as_normal():
status = compute_limit_status(
price=np.array([np.nan, 110.0, 90.0]),
preclose=np.array([100.0, np.nan, 0.0]),
limit_pct=np.array([0.10, 0.10, 0.10]),
)
assert status.tolist() == [
LimitStatus.NORMAL.value,
LimitStatus.NORMAL.value,
LimitStatus.NORMAL.value,
]
# --- continuous targets ------------------------------------------------------
def test_continuous_targets_normalization():
@@ -294,6 +335,70 @@ def test_repair_scales_to_4000_names():
assert abs(gross - B) <= 0.03 * B
def test_repair_handles_empty_input():
result = repair_exposure(
np.array([], dtype=np.int64),
np.array([], dtype=float),
np.array([], dtype=float),
np.array([], dtype=np.int64),
np.array([], dtype=np.int64),
np.array([], dtype=np.int64),
)
assert result.dtype == np.int64
assert result.tolist() == []
def test_repair_respects_max_iters_and_zero_increment_noop():
q_round = np.array([100, -100], dtype=np.int64)
q_target = np.array([300.0, -300.0])
price = np.array([10.0, 10.0])
min_open = np.array([100, 100])
prev = np.zeros(2, dtype=np.int64)
capped = repair_exposure(
q_round,
q_target,
price,
increment=np.array([1, 1]),
min_open=min_open,
prev_shares=prev,
booksize=10_000.0,
gross_tol=0.0,
max_iters=0,
)
zero_increment = repair_exposure(
q_round,
q_target,
price,
increment=np.array([0, 0]),
min_open=min_open,
prev_shares=prev,
booksize=0.0,
net_tol=0.0,
gross_tol=0.0,
)
assert capped.tolist() == [100, -100]
assert zero_increment.tolist() == [100, -100]
def test_repair_gross_growth_obeys_net_band():
pos = repair_exposure(
q_round=np.array([100], dtype=np.int64),
q_target=np.array([300.0]),
price=np.array([10.0]),
increment=np.array([1]),
min_open=np.array([100]),
prev_shares=np.array([0], dtype=np.int64),
booksize=2_000.0,
net_tol=0.5,
gross_tol=0.0,
)
assert pos.tolist() == [100]
# --- construct_positions -----------------------------------------------------
def test_construct_positions_schema():
@@ -305,6 +410,47 @@ def test_construct_positions_schema():
assert pos["position_shares"].dtype == np.int64
def test_construct_positions_empty_weights_returns_schema():
data = _make_data(n_days=3)
empty_weights = pd.DataFrame(columns=["symbol_id", "date", "combo_name", "weight"])
pos = construct_positions(
empty_weights,
data,
booksize=1e6,
portfolio_name="empty",
)
assert list(pos.columns) == POSITION_COLUMNS
assert pos.empty
def test_construct_positions_ignores_absent_or_bad_prices():
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
data = pd.DataFrame([
{"symbol_id": "sh600000", "date": dates[0], "close": np.nan, "isST": 0},
{"symbol_id": "sz000001", "date": dates[0], "close": 20.0, "isST": 0},
{"symbol_id": "sh600000", "date": dates[1], "close": 10.0, "isST": 0},
{"symbol_id": "sz000001", "date": dates[1], "close": 20.0, "isST": 0},
])
weights = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[0], dates[0], dates[1], dates[1]],
"combo_name": ["combo"] * 4,
"weight": [1.0, -1.0, 1.0, -1.0],
})
pos = construct_positions(weights, data, booksize=10000.0, portfolio_name="bad_price")
bad_price_rows = pos[
(pos["date"] == dates[0])
& (pos["symbol_id"] == "sh600000")
]
assert bad_price_rows.empty or (bad_price_rows["target_weight"] == 0.0).all()
assert np.isfinite(pos["target_value"]).all()
assert np.isfinite(pos["position_value"]).all()
def test_construct_positions_threads_state_and_closes_absent():
data = _make_data()
weights = _make_weights(data)
@@ -320,6 +466,34 @@ def test_construct_positions_threads_state_and_closes_absent():
assert final.empty or (final["position_shares"] == 0).all()
def test_construct_positions_closes_absent_short_position():
dates = pd.to_datetime(["2024-01-02", "2024-01-03"])
data = pd.DataFrame([
{"symbol_id": sym, "date": d, "close": price, "isST": 0}
for d in dates
for sym, price in (("sh600000", 10.0), ("sz000001", 20.0))
])
weights = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sz000001"],
"date": [dates[0], dates[0], dates[1]],
"combo_name": ["combo", "combo", "combo"],
"weight": [-1.0, 1.0, 1.0],
})
pos = construct_positions(weights, data, booksize=20000.0, portfolio_name="absent_short")
first_day_short = pos[
(pos["date"] == dates[0])
& (pos["symbol_id"] == "sh600000")
]
final_day_short = pos[
(pos["date"] == dates[1])
& (pos["symbol_id"] == "sh600000")
]
assert (first_day_short["position_shares"] < 0).all()
assert final_day_short.empty or (final_day_short["position_shares"] == 0).all()
def test_construct_positions_carries_book_on_zero_gross(caplog):
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
symbols = ["sh600000", "sz000001"]
@@ -391,8 +565,117 @@ def test_volume_cap_uses_traded_value():
assert low[0] == -10000.0
def test_constraints_compose_repeatably_regardless_of_order():
n = 1
sl = _slice(
n,
tradestatus=np.array([0.0]),
limit_status=np.array([LimitStatus.UP_LIMIT.value], dtype=np.int8),
amount=np.array([1_000.0]),
price=np.array([10.0]),
)
ctx = TradeContext(np.zeros(n, np.int64), np.array([500]), sl, 1e6)
first_order = ReferenceSimulator(
constraints=[
SuspensionConstraint(),
PriceLimitConstraint(),
VolumeCapConstraint(max_frac=0.1),
],
cost_bps=10,
).fill(ctx)
reversed_order = ReferenceSimulator(
constraints=[
VolumeCapConstraint(max_frac=0.1),
PriceLimitConstraint(),
SuspensionConstraint(),
],
cost_bps=10,
).fill(ctx)
assert first_order.traded_shares.tolist() == [0]
assert first_order.realized_shares.tolist() == [0]
assert first_order.blocked.tolist() == [1]
assert np.array_equal(first_order.traded_shares, reversed_order.traded_shares)
assert np.array_equal(first_order.realized_shares, reversed_order.realized_shares)
assert np.array_equal(first_order.blocked, reversed_order.blocked)
assert np.array_equal(first_order.cost, reversed_order.cost)
def test_constraint_registry_and_default_adjust_targets():
class _NoopConstraint(TradeConstraint):
name = "_coverage_noop_constraint"
def delta_bounds(self, ctx):
return np.zeros(1), np.ones(1)
registered = register_constraint(_NoopConstraint)
assert registered is _NoopConstraint
assert "_coverage_noop_constraint" in available_constraints()
assert isinstance(get_constraint("_coverage_noop_constraint"), _NoopConstraint)
assert get_constraint("_coverage_noop_constraint").adjust_targets(object()) is None
with np.testing.assert_raises(KeyError):
get_constraint("_missing_constraint")
with np.testing.assert_raises(TypeError):
register_constraint(object) # type: ignore[arg-type]
with np.testing.assert_raises(ValueError):
class _NoNameConstraint(TradeConstraint):
def delta_bounds(self, ctx):
return np.zeros(1), np.ones(1)
register_constraint(_NoNameConstraint)
with np.testing.assert_raises(ValueError):
class _DuplicateConstraint(TradeConstraint):
name = "_coverage_noop_constraint"
def delta_bounds(self, ctx):
return np.zeros(1), np.ones(1)
register_constraint(_DuplicateConstraint)
# --- ReferenceSimulator ------------------------------------------------------
def test_simulator_applies_constraint_target_adjustment():
class _HalveTarget(TradeConstraint):
name = "_halve_target"
def adjust_targets(self, ctx):
return ctx.target_shares // 2
def delta_bounds(self, ctx):
return np.full(len(ctx.target_shares), -np.inf), np.full(len(ctx.target_shares), np.inf)
sl = _slice(1, price=np.array([10.0]))
ctx = TradeContext(np.array([0], np.int64), np.array([100], np.int64), sl, 1e6)
result = ReferenceSimulator(constraints=[_HalveTarget()]).fill(ctx)
assert result.traded_shares.tolist() == [50]
assert result.realized_shares.tolist() == [50]
def test_simulator_empty_positions_uses_default_booksize():
data = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"open": [10.0],
"close": [10.0],
"preclose": [10.0],
"amount": [1e9],
"tradestatus": [1],
"isST": [0],
})
positions = pd.DataFrame(columns=POSITION_COLUMNS)
fills, pnl = ReferenceSimulator().run(positions, data)
assert list(fills.columns) == FILL_COLUMNS
assert list(pnl.columns) == PNL_COLUMNS
assert fills.empty
assert pnl.empty
def test_simulator_next_open_and_blocked_buy_holds_prev():
data = _make_data(n_days=15)
weights = _make_weights(data)
@@ -445,6 +728,7 @@ def test_simulator_blocked_buy_when_suspended():
assert res.traded_shares[0] == 0
assert res.realized_shares[0] == 0
assert res.blocked[0] == 1
assert res.cost[0] == 0.0
def test_simulator_cost_is_positive_when_trading():
@@ -458,6 +742,154 @@ def test_simulator_cost_is_positive_when_trading():
assert np.isclose(res.cost[0], 1000 * 20 * 15 / 1e4)
def test_simulator_cost_only_on_nonzero_realized_trades():
n = 2
sim = ReferenceSimulator(constraints=[], cost_bps=10)
sl = _slice(n, price=np.array([10.0, 20.0]))
ctx = TradeContext(np.array([100, 100], np.int64),
np.array([100, 150], np.int64), sl, 1e6)
res = sim.fill(ctx)
assert res.traded_shares.tolist() == [0, 50]
assert res.cost[0] == 0.0
assert np.isclose(res.cost[1], 50 * 20 * 10 / 1e4)
def test_simulator_short_to_long_flip_trades_full_delta():
dates = pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"])
positions = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": [dates[0], dates[1]],
"portfolio_name": ["flip", "flip"],
"target_weight": [-1.0, 1.0],
"target_value": [-1000.0, 1000.0],
"target_shares": [-100.0, 100.0],
"position_shares": [-100, 100],
"position_value": [-1000.0, 1000.0],
"price": [10.0, 10.0],
})
data = pd.DataFrame({
"symbol_id": ["sh600000"] * 3,
"date": dates,
"open": [10.0, 10.0, 10.0],
"close": [10.0, 10.0, 10.0],
"preclose": [10.0, 10.0, 10.0],
"amount": [1e9, 1e9, 1e9],
"tradestatus": [1, 1, 1],
"isST": [0, 0, 0],
})
fills, _ = ReferenceSimulator().run(positions, data)
by_date = fills.set_index("date")
assert by_date.loc[dates[1], "traded_shares"] == -100
assert by_date.loc[dates[1], "realized_shares"] == -100
assert by_date.loc[dates[2], "prev_shares"] == -100
assert by_date.loc[dates[2], "traded_shares"] == 200
assert by_date.loc[dates[2], "realized_shares"] == 100
def test_simulator_volume_cap_partially_fills_sell():
sl = _slice(1, amount=np.array([10_000.0]), price=np.array([10.0]))
ctx = TradeContext(
np.array([1000], np.int64),
np.array([0], np.int64),
sl,
1_000_000.0,
)
result = ReferenceSimulator(
constraints=[VolumeCapConstraint(max_frac=0.10)]
).fill(ctx)
assert result.traded_shares.tolist() == [-100]
assert result.realized_shares.tolist() == [900]
assert result.blocked.tolist() == [1]
def test_simulator_missing_next_open_has_zero_cost_and_turnover():
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [dates[0]],
"portfolio_name": ["missing_open"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [100.0],
"position_shares": [100],
"position_value": [1000.0],
"price": [10.0],
})
data = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": dates,
"open": [10.0, np.nan],
"close": [10.0, 10.0],
"preclose": [10.0, 10.0],
"amount": [1e9, 1e9],
"tradestatus": [1, 1],
"isST": [0, 0],
})
fills, pnl = ReferenceSimulator(cost_bps=10, slippage_bps=5).run(positions, data)
assert fills["traded_shares"].iloc[0] == 100
assert fills["trade_cost"].iloc[0] == 0.0
assert pnl["cost"].iloc[0] == 0.0
assert pnl["turnover"].iloc[0] == 0.0
assert pnl["gross_exposure"].iloc[0] == 1000.0
def test_simple_cost_model_adds_cost_and_slippage_without_price_adjustment():
model = SimpleProportionalCostModel(cost_bps=10, slippage_bps=5)
cost = model.compute(
traded_shares=np.array([1000, -1000]),
execution_price=np.array([20.0, 20.0]),
side=np.array([1, -1]),
date=dt.date(2024, 1, 2),
)
assert np.allclose(cost, np.array([30.0, 30.0]))
def test_daily_pnl_cost_matches_fill_trade_cost_sum():
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
positions = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001"],
"date": [dates[0], dates[0]],
"portfolio_name": ["run1", "run1"],
"target_weight": [0.5, -0.5],
"target_value": [1000.0, -1000.0],
"target_shares": [100.0, -50.0],
"position_shares": [100, -50],
"position_value": [1000.0, -1000.0],
"price": [10.0, 20.0],
})
data = pd.DataFrame([
{
"symbol_id": sym,
"date": d,
"open": price,
"close": price,
"preclose": price,
"amount": 1e9,
"tradestatus": 1,
"isST": 0,
}
for d in dates
for sym, price in (("sh600000", 10.0), ("sz000001", 20.0))
])
fills, pnl = ReferenceSimulator(cost_bps=10, slippage_bps=5).run(positions, data)
total_fill_cost = fills["trade_cost"].sum()
assert np.isclose(total_fill_cost, 3.0)
assert np.isclose(pnl["cost"].iloc[0], total_fill_cost)
assert np.isclose(pnl["pnl"].iloc[0], -total_fill_cost)
# --- evaluate_portfolio ------------------------------------------------------
def test_evaluate_portfolio_keys_no_ic():
@@ -470,3 +902,58 @@ def test_evaluate_portfolio_keys_no_ic():
assert key in metrics
assert "ic" not in metrics
assert "rank_ic" not in metrics
def test_evaluate_portfolio_excludes_signal_without_forward_return():
dates = pd.date_range("2024-01-01", periods=3)
data = pd.DataFrame([
{"symbol_id": sym, "date": d, "open": price, "close": price}
for d, prices in zip(dates, [(100.0, 100.0), (100.0, 100.0), (200.0, 100.0)])
for sym, price in zip(("sh600000", "sz000001"), prices)
])
positions = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[0], dates[0], dates[1], dates[1]],
"portfolio_name": ["run1"] * 4,
"target_weight": [0.5, -0.5, -0.5, 0.5],
"target_value": [500.0, -500.0, -500.0, 500.0],
"target_shares": [5.0, -5.0, -2.5, 5.0],
"position_shares": [5, -5, -2, 5],
"position_value": [500.0, -500.0, -400.0, 500.0],
"price": [100.0, 100.0, 200.0, 100.0],
})
metrics = evaluate_portfolio(positions, data)
assert metrics["n_dates"] == 1
def test_evaluate_portfolio_empty_and_single_return_paths():
empty_metrics = evaluate_portfolio(
pd.DataFrame(columns=POSITION_COLUMNS),
pd.DataFrame(columns=["symbol_id", "date", "open"]),
)
assert empty_metrics["n_dates"] == 0
assert empty_metrics["cumulative_return"] == 0.0
dates = pd.date_range("2024-01-01", periods=3)
data = pd.DataFrame([
{"symbol_id": "sh600000", "date": d, "open": price}
for d, price in zip(dates, [100.0, 100.0, 110.0])
])
positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [dates[0]],
"portfolio_name": ["single"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [10.0],
"position_shares": [10],
"position_value": [1000.0],
"price": [100.0],
})
single_metrics = evaluate_portfolio(positions, data)
assert single_metrics["n_dates"] == 1
assert single_metrics["cumulative_return"] == 0.0
+98
View File
@@ -0,0 +1,98 @@
"""Malformed parquet/input tests for phase boundary contracts."""
from __future__ import annotations
import pandas as pd
import pytest
from pipeline.alpha.compute import compute_alpha
from pipeline.combo.combine import combine_alphas
from pipeline.derived.compute import validate_derived_frame
from pipeline.portfolio.construct import construct_positions
from pipeline.portfolio.simulator import ReferenceSimulator
from tests.helpers import make_generated_daily_bars
def test_alpha_compute_rejects_daily_data_without_close():
daily = make_generated_daily_bars().drop(columns=["close"])
with pytest.raises(KeyError, match="close"):
compute_alpha(daily, "bad", "reversal", lookback=3)
def test_alpha_feature_path_rejects_duplicate_symbol_dates(tmp_path):
daily = make_generated_daily_bars()
feature = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
"toy_feature": [1.0, 2.0],
})
feature_path = tmp_path / "duplicate_feature_keys.pq"
feature.to_parquet(feature_path, index=False)
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
compute_alpha(
daily,
"bad_features",
"reversal",
lookback=3,
feature_paths=[str(feature_path)],
)
def test_derived_validation_rejects_bool_value_columns():
derived = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"is_good": [True],
})
with pytest.raises(ValueError, match="numeric"):
validate_derived_frame(derived)
def test_combo_combine_rejects_missing_weight_column(tmp_path):
bad_alpha = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"alpha_name": ["bad"],
})
bad_alpha_path = tmp_path / "bad_alpha.pq"
bad_alpha.to_parquet(bad_alpha_path, index=False)
with pytest.raises(KeyError, match="weight"):
combine_alphas([str(bad_alpha_path)], "bad_combo")
def test_portfolio_build_rejects_weights_without_symbol_id():
daily = make_generated_daily_bars()
bad_weights = pd.DataFrame({
"date": [pd.Timestamp("2024-01-02")],
"combo_name": ["bad"],
"weight": [1.0],
})
with pytest.raises(KeyError, match="symbol_id"):
construct_positions(
bad_weights,
daily,
booksize=1_000_000.0,
portfolio_name="bad_portfolio",
)
def test_portfolio_simulate_rejects_positions_without_position_shares():
daily = make_generated_daily_bars()
bad_positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"portfolio_name": ["bad"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [100.0],
"position_value": [1000.0],
"price": [10.0],
})
with pytest.raises(KeyError, match="position_shares"):
ReferenceSimulator().run(bad_positions, daily)
+92
View File
@@ -0,0 +1,92 @@
"""Offline tests for baostock-backed universe helpers."""
from __future__ import annotations
import pandas as pd
import data.universe as universe
class _FakeResult:
def __init__(self, rows, fields=None):
self.rows = rows
self.fields = fields or ["code", "name", "date"]
self._idx = -1
def next(self):
self._idx += 1
return self._idx < len(self.rows)
def get_row_data(self):
return self.rows[self._idx]
def test_index_constituent_helpers_normalize_dotted_codes(monkeypatch):
calls: list[str] = []
monkeypatch.setattr(universe.bs, "login", lambda: calls.append("login"))
monkeypatch.setattr(universe.bs, "logout", lambda: calls.append("logout"))
monkeypatch.setattr(
universe.bs,
"query_hs300_stocks",
lambda: _FakeResult([
["sh.600000", "浦发银行", "2024-01-12"],
["sz.000001", "平安银行", "2024-01-12"],
]),
)
monkeypatch.setattr(
universe.bs,
"query_zz500_stocks",
lambda: _FakeResult([
["sh.600006", "东风汽车", "2024-01-12"],
]),
)
hs300 = universe.get_hs300_stocks()
zz500 = universe.get_zz500_stocks()
assert calls == ["login", "logout", "login", "logout"]
assert hs300["code"].tolist() == ["sh600000", "sz000001"]
assert zz500["code"].tolist() == ["sh600006"]
assert hs300["name"].tolist() == ["浦发银行", "平安银行"]
def test_get_all_stocks_walks_back_and_filters_to_listed_a_shares(monkeypatch):
fields = ["code", "tradeStatus", "code_name"]
responses = [
_FakeResult([], fields=fields),
_FakeResult(
[
["sh.600000", "1", "浦发银行"],
["sh.688001", "1", "华兴源创"],
["sz.000001", "1", "平安银行"],
["sz.300750", "1", "宁德时代"],
["sz.399001", "1", "深证成指"],
["sz.200001", "1", "深物业B"],
["bj.430047", "1", "北交所样本"],
],
fields=fields,
),
]
query_days: list[str] = []
monkeypatch.setattr(universe.bs, "login", lambda: None)
monkeypatch.setattr(universe.bs, "logout", lambda: None)
def fake_query_all_stock(day):
query_days.append(day)
return responses.pop(0)
monkeypatch.setattr(universe.bs, "query_all_stock", fake_query_all_stock)
result = universe.get_all_stocks("2024-01-07")
assert query_days == ["2024-01-07", "2024-01-06"]
assert result.columns.tolist() == ["code", "name"]
assert result["code"].tolist() == [
"sh600000",
"sh688001",
"sz000001",
"sz300750",
]
assert result["name"].tolist() == ["浦发银行", "华兴源创", "平安银行", "宁德时代"]
+508
View File
@@ -0,0 +1,508 @@
"""Verbose offline checks for the daily research workflow."""
from __future__ import annotations
from pathlib import Path
import numpy as np
import pandas as pd
from pipeline.alpha.compute import compute_alpha
from pipeline.combo.combine import combine_alphas
from pipeline.common.schema import (
ALPHA_COLUMNS,
COMBO_COLUMNS,
FILL_COLUMNS,
PNL_COLUMNS,
POSITION_COLUMNS,
)
from pipeline.portfolio.constraints import (
PriceLimitConstraint,
SuspensionConstraint,
VolumeCapConstraint,
)
from pipeline.portfolio.construct import construct_positions
from pipeline.portfolio.research import evaluate_portfolio
from pipeline.portfolio.simulator import ReferenceSimulator
from tests.helpers import (
GENERATED_SYMBOLS,
generated_sessions,
make_generated_alpha_weights,
make_generated_combo_weights,
make_generated_daily_bars,
)
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
def _assert_sorted_by_symbol_date(frame: pd.DataFrame) -> None:
expected = frame.sort_values(["symbol_id", "date"]).reset_index(drop=True)
pd.testing.assert_frame_equal(frame.reset_index(drop=True), expected)
def _assert_metric_dict_is_finite(metrics: dict[str, float]) -> None:
for key in (
"cumulative_return",
"sharpe_annual",
"turnover_annual",
"max_drawdown",
"hit_rate",
"n_dates",
):
assert key in metrics
assert np.isfinite(metrics[key])
assert "ic" not in metrics
assert "rank_ic" not in metrics
assert "ir" not in metrics
def test_tiny_workflow_golden_outputs_are_stable(tmp_path):
dates = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"])
daily_bars = pd.DataFrame([
{
"symbol_id": "sh600000",
"symbol_name": "A",
"date": dates[0],
"open": 10.0,
"high": 10.0,
"low": 10.0,
"close": 10.0,
"preclose": 10.0,
"volume": 1_000_000.0,
"amount": 10_000_000.0,
"vwap": 10.0,
"turn": 1.0,
"pctChg": 0.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sz000001",
"symbol_name": "B",
"date": dates[0],
"open": 20.0,
"high": 20.0,
"low": 20.0,
"close": 20.0,
"preclose": 20.0,
"volume": 1_000_000.0,
"amount": 20_000_000.0,
"vwap": 20.0,
"turn": 1.0,
"pctChg": 0.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sh600000",
"symbol_name": "A",
"date": dates[1],
"open": 10.0,
"high": 12.0,
"low": 10.0,
"close": 12.0,
"preclose": 10.0,
"volume": 1_000_000.0,
"amount": 10_000_000.0,
"vwap": 10.0,
"turn": 1.0,
"pctChg": 20.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sz000001",
"symbol_name": "B",
"date": dates[1],
"open": 20.0,
"high": 20.0,
"low": 18.0,
"close": 18.0,
"preclose": 20.0,
"volume": 1_000_000.0,
"amount": 20_000_000.0,
"vwap": 20.0,
"turn": 1.0,
"pctChg": -10.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sh600000",
"symbol_name": "A",
"date": dates[2],
"open": 12.0,
"high": 13.0,
"low": 12.0,
"close": 13.0,
"preclose": 12.0,
"volume": 1_000_000.0,
"amount": 12_000_000.0,
"vwap": 12.0,
"turn": 1.0,
"pctChg": 8.33,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sz000001",
"symbol_name": "B",
"date": dates[2],
"open": 18.0,
"high": 21.0,
"low": 18.0,
"close": 21.0,
"preclose": 18.0,
"volume": 1_000_000.0,
"amount": 18_000_000.0,
"vwap": 18.0,
"turn": 1.0,
"pctChg": 16.67,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
])
alpha = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[0], dates[0], dates[1], dates[1]],
"alpha_name": ["gold_alpha"] * 4,
"weight": [1.0, -1.0, -1.0, 1.0],
})
alpha_path = tmp_path / "gold_alpha.pq"
alpha.to_parquet(alpha_path, index=False)
combo = combine_alphas([str(alpha_path)], "gold_combo")
positions = construct_positions(combo, daily_bars, booksize=20_000.0, portfolio_name="gold_port")
fills, pnl = ReferenceSimulator().run(positions, daily_bars)
expected_combo = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
"date": [dates[0], dates[1], dates[0], dates[1]],
"combo_name": ["gold_combo"] * 4,
"weight": [1.0, -1.0, -1.0, 1.0],
})
expected_positions = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
"date": [dates[0], dates[1], dates[0], dates[1]],
"portfolio_name": ["gold_port"] * 4,
"target_weight": [0.5, -0.5, -0.5, 0.5],
"target_value": [10000.0, -10000.0, -10000.0, 10000.0],
"target_shares": [1000.0, -10000.0 / 12.0, -500.0, 10000.0 / 18.0],
"position_shares": [1000, -833, -500, 556],
"position_value": [10000.0, -9996.0, -10000.0, 10008.0],
"price": [10.0, 12.0, 20.0, 18.0],
})
expected_fills = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[1], dates[1], dates[2], dates[2]],
"portfolio_name": ["gold_port"] * 4,
"prev_shares": [0, 0, 1000, -500],
"target_shares": [1000, -500, -833, 556],
"traded_shares": [1000, -500, -1833, 1056],
"realized_shares": [1000, -500, -833, 556],
"blocked": [0, 0, 0, 0],
"trade_cost": [0.0, 0.0, 0.0, 0.0],
})
expected_pnl = pd.DataFrame({
"date": [dates[1], dates[2]],
"portfolio_name": ["gold_port", "gold_port"],
"gross_exposure": [21000.0, 22505.0],
"net_exposure": [3000.0, 847.0],
"pnl": [3000.0, 835.0],
"cost": [0.0, 0.0],
"turnover": [1.0, 2.0502],
"n_positions": [2, 2],
})
pd.testing.assert_frame_equal(combo, expected_combo)
pd.testing.assert_frame_equal(positions, expected_positions)
pd.testing.assert_frame_equal(fills, expected_fills)
pd.testing.assert_frame_equal(pnl, expected_pnl)
def test_generated_alpha_combo_portfolio_execution_workflow(tmp_path):
daily_bars = make_generated_daily_bars()
computed_alpha = compute_alpha(
data=daily_bars,
alpha_name="generated_reversal_3d",
alpha_type="reversal",
lookback=3,
)
assert list(computed_alpha.columns) == ALPHA_COLUMNS
assert not computed_alpha.empty
assert set(computed_alpha["symbol_id"]).issubset(set(GENERATED_SYMBOLS))
assert computed_alpha["date"].min() > daily_bars["date"].min()
assert computed_alpha["weight"].notna().all()
assert computed_alpha["weight"].abs().sum() > 0.0
assert {"ic", "rank_ic", "ir"}.isdisjoint(computed_alpha.columns)
_assert_sorted_by_symbol_date(computed_alpha)
alpha_a = make_generated_alpha_weights("alpha_a", zero_date_index=2)
alpha_b = make_generated_alpha_weights(
"alpha_b",
scale=0.5,
offset=0.25,
zero_date_index=2,
)
alpha_a_path = tmp_path / "alpha_a.pq"
alpha_b_path = tmp_path / "alpha_b.pq"
alpha_a.to_parquet(alpha_a_path, index=False)
alpha_b.to_parquet(alpha_b_path, index=False)
identity_combo = combine_alphas([str(alpha_a_path)], "identity_combo")
assert list(identity_combo.columns) == COMBO_COLUMNS
assert (identity_combo["combo_name"] == "identity_combo").all()
pd.testing.assert_frame_equal(
identity_combo[["symbol_id", "date", "weight"]],
alpha_a[["symbol_id", "date", "weight"]],
)
equal_combo = combine_alphas([str(alpha_a_path), str(alpha_b_path)], "equal_combo")
expected_equal_weights = (
pd.concat([alpha_a, alpha_b], ignore_index=True)
.groupby(["symbol_id", "date"], as_index=False)["weight"]
.mean()
.sort_values(["symbol_id", "date"])
.reset_index(drop=True)
)
pd.testing.assert_frame_equal(
equal_combo[["symbol_id", "date", "weight"]],
expected_equal_weights,
)
portfolio_weights = make_generated_combo_weights("workflow_combo", zero_date_index=2)
positions = construct_positions(
weights_df=portfolio_weights,
data_df=daily_bars,
booksize=2_000_000.0,
portfolio_name="workflow_portfolio",
)
assert list(positions.columns) == POSITION_COLUMNS
assert not positions.empty
assert (positions["portfolio_name"] == "workflow_portfolio").all()
assert pd.api.types.is_integer_dtype(positions["position_shares"])
assert np.allclose(
positions["position_value"],
positions["position_shares"].astype(float) * positions["price"].fillna(0.0),
)
target_gross_by_date = positions.groupby("date")["target_weight"].apply(lambda s: s.abs().sum())
nonzero_target_dates = target_gross_by_date[target_gross_by_date > 0.0]
assert np.allclose(nonzero_target_dates, 1.0)
nonzero_share_counts = positions.loc[positions["position_shares"] != 0, "position_shares"].abs()
assert (nonzero_share_counts >= 100).all()
zero_gross_date = generated_sessions(10)[2]
previous_date = generated_sessions(10)[1]
zero_gross_positions = positions[positions["date"] == zero_gross_date].set_index("symbol_id")
previous_positions = positions[positions["date"] == previous_date].set_index("symbol_id")
common_symbols = zero_gross_positions.index.intersection(previous_positions.index)
assert not common_symbols.empty
assert (zero_gross_positions.loc[common_symbols, "target_weight"] == 0.0).all()
pd.testing.assert_series_equal(
zero_gross_positions.loc[common_symbols, "position_shares"],
previous_positions.loc[common_symbols, "position_shares"],
check_names=False,
)
simulator = ReferenceSimulator(
constraints=[
SuspensionConstraint(),
PriceLimitConstraint(),
VolumeCapConstraint(max_frac=0.02),
],
cost_bps=5,
slippage_bps=5,
)
fills, pnl = simulator.run(positions, daily_bars)
assert list(fills.columns) == FILL_COLUMNS
assert list(pnl.columns) == PNL_COLUMNS
assert not fills.empty
assert not pnl.empty
assert (fills["realized_shares"] == fills["prev_shares"] + fills["traded_shares"]).all()
assert fills["blocked"].sum() > 0
fill_prices = fills.merge(
daily_bars[["symbol_id", "date", "open"]],
on=["symbol_id", "date"],
how="left",
validate="many_to_one",
)
expected_trade_cost = (
fill_prices["traded_shares"].abs()
* fill_prices["open"].fillna(0.0)
* 10
/ 10_000
)
assert np.allclose(fill_prices["trade_cost"], expected_trade_cost)
cost_by_date = fills.groupby("date")["trade_cost"].sum()
assert np.allclose(
pnl.set_index("date")["cost"],
cost_by_date.reindex(pnl["date"], fill_value=0.0),
)
booksize_used_by_simulator = positions.groupby("date")["target_value"].apply(lambda s: s.abs().sum()).max()
traded_value_by_date = (
fill_prices.assign(traded_value=fill_prices["traded_shares"].abs() * fill_prices["open"])
.groupby("date")["traded_value"]
.sum()
)
assert np.allclose(
pnl.set_index("date")["turnover"],
traded_value_by_date.reindex(pnl["date"], fill_value=0.0) / booksize_used_by_simulator,
)
metrics = evaluate_portfolio(positions, daily_bars)
_assert_metric_dict_is_finite(metrics)
def test_generated_workflow_outputs_keep_parquet_schema_contracts(tmp_path):
daily_bars = make_generated_daily_bars(n_sessions=10, include_missing=False)
alpha = compute_alpha(
data=daily_bars,
alpha_name="schema_reversal",
alpha_type="reversal",
lookback=3,
)
alpha_path = tmp_path / "schema_reversal.pq"
alpha.to_parquet(alpha_path, index=False)
combo = combine_alphas([str(alpha_path)], "schema_combo")
positions = construct_positions(
weights_df=combo,
data_df=daily_bars,
booksize=1_000_000.0,
portfolio_name="schema_portfolio",
)
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, daily_bars)
assert list(alpha.columns) == ALPHA_COLUMNS
assert pd.api.types.is_object_dtype(alpha["symbol_id"])
assert pd.api.types.is_datetime64_any_dtype(alpha["date"])
assert pd.api.types.is_object_dtype(alpha["alpha_name"])
assert pd.api.types.is_float_dtype(alpha["weight"])
assert not alpha.isna().any().any()
assert np.isfinite(alpha["weight"]).all()
assert list(combo.columns) == COMBO_COLUMNS
assert pd.api.types.is_object_dtype(combo["symbol_id"])
assert pd.api.types.is_datetime64_any_dtype(combo["date"])
assert pd.api.types.is_object_dtype(combo["combo_name"])
assert pd.api.types.is_float_dtype(combo["weight"])
assert not combo.isna().any().any()
assert np.isfinite(combo["weight"]).all()
assert list(positions.columns) == POSITION_COLUMNS
assert pd.api.types.is_integer_dtype(positions["position_shares"])
assert pd.api.types.is_datetime64_any_dtype(positions["date"])
assert not positions.isna().any().any()
position_numeric_columns = [
"target_weight",
"target_value",
"target_shares",
"position_value",
"price",
]
assert np.isfinite(positions[position_numeric_columns]).all().all()
assert list(fills.columns) == FILL_COLUMNS
assert pd.api.types.is_integer_dtype(fills["prev_shares"])
assert pd.api.types.is_integer_dtype(fills["target_shares"])
assert pd.api.types.is_integer_dtype(fills["traded_shares"])
assert pd.api.types.is_integer_dtype(fills["realized_shares"])
assert pd.api.types.is_integer_dtype(fills["blocked"])
assert not fills.isna().any().any()
assert np.isfinite(fills["trade_cost"]).all()
assert list(pnl.columns) == PNL_COLUMNS
assert pd.api.types.is_integer_dtype(pnl["n_positions"])
assert not pnl.isna().any().any()
pnl_numeric_columns = [
"gross_exposure",
"net_exposure",
"pnl",
"cost",
"turnover",
]
assert np.isfinite(pnl[pnl_numeric_columns]).all().all()
def test_frozen_real_fixture_runs_high_level_workflow(tmp_path):
real_daily_bars = pd.read_parquet(FIXTURE_PATH)
assert real_daily_bars.shape == (36, 19)
assert set(real_daily_bars["symbol_id"]) == set(GENERATED_SYMBOLS)
assert real_daily_bars["date"].min() == pd.Timestamp("2024-01-02")
assert real_daily_bars["date"].max() == pd.Timestamp("2024-01-12")
assert real_daily_bars.groupby("date")["symbol_id"].nunique().eq(4).all()
reversal_alpha = compute_alpha(
data=real_daily_bars,
alpha_name="real_reversal_3d",
alpha_type="reversal",
lookback=3,
)
reversal_vol_alpha = compute_alpha(
data=real_daily_bars,
alpha_name="real_reversal_vol_3d",
alpha_type="reversal_vol",
lookback=3,
vol_window=3,
)
reversal_path = tmp_path / "real_reversal.pq"
reversal_vol_path = tmp_path / "real_reversal_vol.pq"
reversal_alpha.to_parquet(reversal_path, index=False)
reversal_vol_alpha.to_parquet(reversal_vol_path, index=False)
combo = combine_alphas([str(reversal_path), str(reversal_vol_path)], "real_equal_combo")
positions = construct_positions(
weights_df=combo,
data_df=real_daily_bars,
booksize=1_000_000.0,
portfolio_name="real_fixture_portfolio",
)
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, real_daily_bars)
metrics = evaluate_portfolio(positions, real_daily_bars)
assert not reversal_alpha.empty
assert not reversal_vol_alpha.empty
assert not combo.empty
assert not positions.empty
assert not fills.empty
assert not pnl.empty
assert np.isfinite(combo["weight"]).all()
assert np.isfinite(positions["target_weight"]).all()
assert np.isfinite(pnl[["gross_exposure", "net_exposure", "pnl", "cost", "turnover"]]).all().all()
_assert_metric_dict_is_finite(metrics)
Generated
+246 -3
View File
@@ -284,7 +284,6 @@ version = "0.1.0"
source = { virtual = "." }
dependencies = [
{ name = "akshare" },
{ name = "backtrader" },
{ name = "baostock" },
{ name = "click" },
{ name = "matplotlib" },
@@ -293,24 +292,38 @@ dependencies = [
{ name = "pyarrow" },
]
[package.optional-dependencies]
backtrader = [
{ name = "backtrader" },
]
joinquant-browser = [
{ name = "playwright" },
]
[package.dev-dependencies]
dev = [
{ name = "coverage" },
{ name = "pytest" },
]
[package.metadata]
requires-dist = [
{ name = "akshare", specifier = ">=1.14.0" },
{ name = "backtrader", specifier = ">=1.9.76.123" },
{ name = "backtrader", marker = "extra == 'backtrader'", specifier = ">=1.9.76.123" },
{ name = "baostock", specifier = ">=0.8.8" },
{ name = "click", specifier = ">=8.0.0" },
{ name = "matplotlib", specifier = ">=3.7.0" },
{ name = "pandas", specifier = ">=2.0.0" },
{ name = "playwright", marker = "extra == 'joinquant-browser'", specifier = ">=1.61.0" },
{ name = "pyarrow", specifier = ">=14.0.0" },
]
provides-extras = ["backtrader", "joinquant-browser"]
[package.metadata.requires-dev]
dev = [{ name = "pytest", specifier = ">=7.0.0" }]
dev = [
{ name = "coverage", specifier = ">=7.14.1" },
{ name = "pytest", specifier = ">=7.0.0" },
]
[[package]]
name = "click"
@@ -493,6 +506,119 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" },
]
[[package]]
name = "coverage"
version = "7.14.1"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/54/fd/0ab2772530e946e1be1abd0bc09e647ec9b02e88f0867857601fefca8953/coverage-7.14.1.tar.gz", hash = "sha256:30c08f7d90415aa98b3c990385dea2939b0da55f38515e5b369b83655f8523be", size = 920132, upload-time = "2026-05-26T20:41:36.783Z" }
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/92/69/0d2ef01ff4b8fcecd4cba920d11e92fa4f96ae412441d3b56a90a258e69b/coverage-7.14.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:3e3680291c4a1d0dadfa84a2c459576a4af5133abb617905714339a0c73138cf", size = 219722, upload-time = "2026-05-26T20:38:14.002Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f8/ae/9afdeaa31b9d9ce98124b6abf8bb49119bf71aecae04f8567c189d91299f/coverage-7.14.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a5274669f37f2343635a347b91a60777621341ab3378e9c6ac9335eee704bddf", size = 220240, upload-time = "2026-05-26T20:38:17.424Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/69/c998589871df7ea7dba865cc5ee32b5a3e1d47ba6c68ef91104c7c46fa5e/coverage-7.14.1-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cfe5a5fec635799ef33428f1e5e61bafa45a92a96190ba731561ba558ccc214d", size = 246981, upload-time = "2026-05-26T20:38:19.266Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/10/1c7d04c13040dac531d21b712bbe08f902e6dd9b58f5d77875c4d030f8f2/coverage-7.14.1-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:62a9f70b52e0b5a95cfef4a5c5641b06983cadc5e538a3feeb5c00211f523ac2", size = 248812, upload-time = "2026-05-26T20:38:20.75Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c1/65/2a38a4607ef27cadcfbcee034dba5830ae2569f90144a0f4c7dbf47d30b0/coverage-7.14.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3c18ebc343e15be53049b3a2dce38fe82d58f37e20ab9094b3a39c0aa4f6bb47", size = 250675, upload-time = "2026-05-26T20:38:22.159Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/a2/a446ed9752a4a59b79e0fb6cbb319f6facb2183045c0725462625e66f87e/coverage-7.14.1-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b84ffdf877644e7096aa936991efeed873f7f3df57b9cd001312b7668ab08550", size = 252590, upload-time = "2026-05-26T20:38:23.63Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9e/fd/e81fbd7ba752365546e9842b1cbdaad3d6919d2a522c590aef16a281ec5e/coverage-7.14.1-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e854312c4103f2ad4c0dc023b69b77ebfd2c89db5f86c4c94dc2353f9a92167e", size = 247691, upload-time = "2026-05-26T20:38:25.057Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/53/35/f3c26fdaae9ea937d154ca4d372e5ea0a4167ff70d36c6074ac2eacb2f83/coverage-7.14.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:c643734307300234fafa36bf2a040a7235f8f177ea1fd6ec1423aea6fb7b929f", size = 248716, upload-time = "2026-05-26T20:38:26.406Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2e/14/940b6c49551fd343e8507ee2b0ba7af5d0aa04ed5bf768285cb7c72a9884/coverage-7.14.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:84ac9499e48700399a5dd0ea7085b5091961fec52c68d66b4ec0d3cf7f4441b1", size = 246721, upload-time = "2026-05-26T20:38:28.282Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/aa/2c/40fc0634186c28292a662dff578866b3913983d6c375a3c2a74020938719/coverage-7.14.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:7f02d09f70776579b926d889a4c9c235070a1f47c40458aeaca563fae5acfdb5", size = 250533, upload-time = "2026-05-26T20:38:29.753Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/e3/2c26bf1e811f9df991ff2a9bdddebdd13ee0665d564df7d05979f9146297/coverage-7.14.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:ce66d8e46da2bb5ee313a745cbd2e391d319176c1f7a9451bfcd3a2fb920859b", size = 246990, upload-time = "2026-05-26T20:38:31.516Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a8/b0/060260ef56bd92363ebdce0c7095ce422b06e69aae71828efeca473ab1ca/coverage-7.14.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:c912c259304cfb5ee584481cfb7ce1ff932b4d61e6c9140b8f19cb7b5ed82332", size = 247593, upload-time = "2026-05-26T20:38:33.065Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/63/f3/501502046efeb0d6d94b5ca54941d95f1184183dd6bdb7f283985783bb4a/coverage-7.14.1-cp310-cp310-win32.whl", hash = "sha256:1238cb94638e610e972c60dac68e813f868dc7d6e982535270558443058d9d59", size = 222330, upload-time = "2026-05-26T20:38:35.36Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/5d/1bf99f2c558f128faf7906817ccbdb576ba815d3b41ce2ac1719b70a3663/coverage-7.14.1-cp310-cp310-win_amd64.whl", hash = "sha256:fc459e5d73be2d6332fcfe8dbf3d8994671fe33c700f4565988ecfa511547253", size = 223261, upload-time = "2026-05-26T20:38:37.196Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/d7/477ad149490e6cb849f28abea1dabb9c823cea72e7500c81b4240ce619c0/coverage-7.14.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:478b5bcd63c2e1357c5c7e16c070690df7b07f676b1c114d7b93e533c664309f", size = 219848, upload-time = "2026-05-26T20:38:38.715Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/91/82/a5eb47257c50601bb7b9a9d2857c67b7a3a85ad74180eb2c98bb1fbe0ce5/coverage-7.14.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a24a81f9715ee42ef59a316cc11611c98fe23920f7c81861315c9f3ff4a230f4", size = 220354, upload-time = "2026-05-26T20:38:40.232Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/8b/78419b5391a5cb706b6544390507e469d83ffc9a8248b02c4011aceb9365/coverage-7.14.1-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:196a13319ad88d6d8ef5ab489ec4f44ddde2143c0c7d5b27786f6c3ffd56a7e1", size = 250771, upload-time = "2026-05-26T20:38:41.782Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/77/63/e77aaacd491182210d639636b7a8bba23ffffa9b82aa3762da9431855fa9/coverage-7.14.1-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3d452fd08b5c72c5167c93e6867b5c08500bd40f2a21e1e854a500550b6cc36f", size = 252683, upload-time = "2026-05-26T20:38:43.305Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/65/1c/a022e3cfbec2ac241640003cb3a817e161d9c7f5aa9b49173756cdc03204/coverage-7.14.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:23bf7fa51ac02e07fc7c96849b82946da47ae862dc8f86d183b2a4864fc38129", size = 254791, upload-time = "2026-05-26T20:38:45.361Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/61/d6/967e408aca4c1ceb88cb0cc677169110ae7f5995fb5eaf5fb1f5a1bb8f5d/coverage-7.14.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bcaa50684dcaadfa599ac48f81103c756d791cfd85c97203d2217c593d48b860", size = 256748, upload-time = "2026-05-26T20:38:46.91Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/be/869188f7fe28638078ec479331ace6dc5f7b40b7153eb616f47ab79404d8/coverage-7.14.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4ea1c034f95c9b056e856b794630b17f9fa3d57e4800ff1e503d3be0f9c9078c", size = 250907, upload-time = "2026-05-26T20:38:48.493Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/07/aa/adb7d3b4278d690e68703abcd76ab1b948242e3668d921711551b78f9ddb/coverage-7.14.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c7e057326434e441306226fbeb5d1aaf14a2637efe97ba668306635835f32ad7", size = 252483, upload-time = "2026-05-26T20:38:50.074Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/61/331c74103c62dcb0c4b9b3a0de9a61aca016208b0a90f109592a9f9ecc28/coverage-7.14.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:59baf88468dbc8d63b1887afd92bda52e40bb1561696e5819670601403810cec", size = 250545, upload-time = "2026-05-26T20:38:51.613Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/b6/c5dae3c104d89be04828f61810e6b3473825482e4c288cc4ed04553e08ae/coverage-7.14.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:d34d75f892b3ab73ba11cab5442cce7b3e168fd64162b16f0e1e0d09c508edef", size = 254310, upload-time = "2026-05-26T20:38:53.503Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ad/a1/2b9d5863e3b83c01ad8199e3c597802fbb3a9dc90b058885804c20296d31/coverage-7.14.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:3a56abc20a472baf0304c455721bc601477440d28ecfde8a03dde79ede07e0df", size = 250266, upload-time = "2026-05-26T20:38:55.414Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/5e/0e511fbdb269359be26fe678a1c3fa1f2aa2a01573cc3f54268c8d6d4797/coverage-7.14.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:6a3cb83d1552c0cd1b4906655b6a33fd4a8473229633a901c6b73bf86914dee9", size = 251174, upload-time = "2026-05-26T20:38:57.141Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/85/10/e55307b622b3dd9671cb321824502dc10f93e72f2802b9946159a8edadeb/coverage-7.14.1-cp311-cp311-win32.whl", hash = "sha256:10274a1fbeb8ec5d72966e17bb198a3104257aca4ac09d98667c5f8aca8c8548", size = 222354, upload-time = "2026-05-26T20:38:58.727Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/cf/107421693cfb71e4f1ca5bf70443f64d4161878068d07a3e51c7ad21d17b/coverage-7.14.1-cp311-cp311-win_amd64.whl", hash = "sha256:87ebdf787d4888e3f3f2d523eadc6e18c6d18c6d0eb173801a189641627fb37e", size = 223290, upload-time = "2026-05-26T20:39:00.413Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/1d/3e3644585eb29e9dafefb19555078529a4d7cce12bd21929664eea989277/coverage-7.14.1-cp311-cp311-win_arm64.whl", hash = "sha256:dd34767fa19848d35659ffc0a75314f58c7af3f1cd87ec521e8292a1238398a3", size = 221953, upload-time = "2026-05-26T20:39:02.159Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/b7/bdbb725ba02c5b42825b200c940f38b7a54fcad24627b7192f78f8110d76/coverage-7.14.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a06c76364a9360e33d6d23769aefdf7f66f38e2ffb60ceb1baaa4989d83b695c", size = 220022, upload-time = "2026-05-26T20:39:03.702Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/72/81/fdc0898a55c6219223291ec1a1fe89966ef212ce82276aa0899df84b5de0/coverage-7.14.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fad54e871165f6ec2f536063ac74c3104508a12963e64072ba44bd822de52b0c", size = 220379, upload-time = "2026-05-26T20:39:05.381Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/72/de048c4a25e13bce59ac6a339351c10bdf2515e07459afcdaf04dc3143a2/coverage-7.14.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:84b535f00655ecafe1d929d1fb00ed5d6fa3051ea643ab2c161a3887b86f294b", size = 251888, upload-time = "2026-05-26T20:39:07.367Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/28/30/300c343f68beb9d4cbb64ec81e58c5b6b80b56927f72d2b38654ac26e013/coverage-7.14.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6b6b0853b895fe0e98cbfc580d1ec3393d9302b4b1e96a77b3f5c91fdab899e6", size = 254624, upload-time = "2026-05-26T20:39:09.037Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b1/ed/7b25642496e8170b6bac14adce00537c6e5fa2d586159401a4de3e8b49e6/coverage-7.14.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:442cc9c952b2df400cda54bb04ab87330cf2cd08a8692cbbea36773531eb6f37", size = 255739, upload-time = "2026-05-26T20:39:10.889Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/a2/abd210b8c4e29c24e4624916db97bb519097a91034aaeb767f937e7da794/coverage-7.14.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8270544c361ed405a27a060dbc9ed2c124b084d96dfdc2d9a2510482aef981ad", size = 257998, upload-time = "2026-05-26T20:39:12.722Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/24/7c50beed3792fe62f6ce0545c6686ce83379719e2c0276179333d97eae92/coverage-7.14.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:48b283b1dd6372e8de2a7a9a4c4d5dc06f4d4fd209b876f3c88a7a205a0c8f84", size = 252296, upload-time = "2026-05-26T20:39:14.259Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/15/05/0f874628ebcbfc77ead559ff210281ef06a97db08481832e7dd39274a135/coverage-7.14.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5b0c99ba93a07d56f6df340bb79be53202a082b2fdb81bfe6190b741a3470d54", size = 253658, upload-time = "2026-05-26T20:39:15.923Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/99/6f/ca6ad067364b337ef997802115e7ecad2abd2248b05471464b0dea02b4d4/coverage-7.14.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:e471bc5769ff073b058cfadb0d736b56ce067c8560eabeb0da88462df98c23e7", size = 251803, upload-time = "2026-05-26T20:39:17.537Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c0/30/b9b4d377cd9f40baf228068f5a81faf8450c6228503011bd499708483a50/coverage-7.14.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f497a1ea81d4cd7c10ddcaa685135b9aabd291af3d55775a9ddf3cb7a364cdd9", size = 255873, upload-time = "2026-05-26T20:39:19.414Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3c/21/7c721a9e5e6bb88547d30a787aefb97512d3f54c1324c7488d9b3743f7f9/coverage-7.14.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2222be86d0b54f5dd5a38f45f17f315f737245e857bf0bdedc70734f84a13c02", size = 251372, upload-time = "2026-05-26T20:39:21.169Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9d/8c/f8ae5a2200130e1503cd7661a6cd3b2b7bacef98277fbf3571fb13f8b766/coverage-7.14.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:85e85586565842f6932abebd4c18bcb1074223dc0b3576e7d173ca710622813a", size = 253245, upload-time = "2026-05-26T20:39:23.097Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/34/62/70a9024672a5f6910517d9628c52c9afbdd3cf8f46426af52bb148a56fff/coverage-7.14.1-cp312-cp312-win32.whl", hash = "sha256:4a28fd227808366b196a75476dced2eb35b351d6766ba9c858dc93319e87f4f1", size = 222567, upload-time = "2026-05-26T20:39:24.868Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/81/8b7cd386839b039ebe1855733b9f9449a8dec5d79564018234f185a7fa70/coverage-7.14.1-cp312-cp312-win_amd64.whl", hash = "sha256:54acdb6674a4661768d7bf7db32dfb9f46ab1d764f8aba6df75ce1a6a088724e", size = 223372, upload-time = "2026-05-26T20:39:26.603Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ae/ba/b44d472022f620d289d95fa830143235c0c36461c6f2437ea8d51e5481ed/coverage-7.14.1-cp312-cp312-win_arm64.whl", hash = "sha256:99cd41ff91afd94896fea3bc002706b6ae4ce95727d06e4a0f39c0a8d8bd8b1a", size = 221989, upload-time = "2026-05-26T20:39:28.242Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/9e/5f6d56327c62b185225d145191c607e07515294a0aa6338e58805cd4a5ac/coverage-7.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:be9f2c802dcfce3f71298303aa5dad0dce440a76c52f2f60dacd8656dab78793", size = 220044, upload-time = "2026-05-26T20:39:29.902Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/75/92/e82aca356744cbbc0f77a0b623e38918c1872361963413a3bab5d0340393/coverage-7.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6223a72fd0e4c7156353ec0f08a5f93623e1d3034d0e2683b9bb8ea674131b1d", size = 220412, upload-time = "2026-05-26T20:39:31.561Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/c9/385bde0bf7ed0f4bf3a7ee5367060a86b5d218718cfd6fb943c0f836b34f/coverage-7.14.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:7279d2110a28cebc738b6459ecda2771735a4c18465fbbd36b3288fe5ed92247", size = 251412, upload-time = "2026-05-26T20:39:33.337Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/8c/23faf6a2343a0d17f960a4bd56c43bc7eb4cf312f774dd6ceebd82c7d8fc/coverage-7.14.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9eeb3fcbc13ba40dfbdb22d01d196a28e9cef9ed4c29b60061a1e0e823a9929d", size = 254008, upload-time = "2026-05-26T20:39:35.009Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/42/06/36f4aa9ca8a815e6036156e80706a67828bb97bd826948244f6996dda957/coverage-7.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f0cfc27c539f07cf5c0a4cfe211d0b6cae039f8f40526dbaa71944e64b50a7b", size = 255241, upload-time = "2026-05-26T20:39:36.71Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/79/95266316352f90f6b1c6736bb413302edfde2453fb32422d3911642691b3/coverage-7.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:221c70f316241a78e77e607c227cefc8808d4e08f28d99c04f35694690e940be", size = 257373, upload-time = "2026-05-26T20:39:38.412Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e3/9c/58316d1f66c488b5fca8a0eb3e98348807813efa8a0d0833b9021be27488/coverage-7.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:da028256b04ec30e5e0114b6f76172938c313991f0a2d3d894271315cf5d5e43", size = 251635, upload-time = "2026-05-26T20:39:40.268Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ef/5a/ca2398a568e16fed7bb713e84ba3603a7164fb65779abe645c565ec890d5/coverage-7.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:76a085d7005236a767e3426148b2c407e53ad61695c562f8a81da2d373324901", size = 253373, upload-time = "2026-05-26T20:39:42.145Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6e/2c/0396562c32deaebe7be51d865b3a41e9a87d7561acafe1a28f53b07e019a/coverage-7.14.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b553d04b5e778a8e56d57eb134aff42a92718ecba45e79c4764ecfa40efd92ff", size = 251341, upload-time = "2026-05-26T20:39:43.907Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fd/8f/a94f9221184c9cae1ee115820e3798e48b6b17777a9f19e46fb9a0c8dc74/coverage-7.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:46f714d2fb8ae2f4f29f23ada7f1e79b759fff5a70f94a1dac23af204c3ec9e4", size = 255497, upload-time = "2026-05-26T20:39:46.166Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/69/505d70e47db1eaebcd002c39759707621ef184cd6b1ae084d9f41293f323/coverage-7.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:1896f5e19ff3f0431c7ce2172adc54890fd97f86b59ced8ca1649145d9ffe35d", size = 251159, upload-time = "2026-05-26T20:39:48.03Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e0/aa/58681c383aa33a9d2ed40a02d7a22fbf780d1fa4d575396365777828198c/coverage-7.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:62fd185ef9df3c33d1c8178c5af105f762afbad96038de9a4ae100aa6297ca33", size = 252934, upload-time = "2026-05-26T20:39:49.872Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/eb/fd/11c928cd6bdffc7074bb5965c173d9ebf517fb00205e1da524b98d29ef92/coverage-7.14.1-cp313-cp313-win32.whl", hash = "sha256:ab4af6352741a604c431c6072fce5bee33bf0f20dc7a56618d6bf6bb89e9810c", size = 222584, upload-time = "2026-05-26T20:39:51.68Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6f/92/fb416fc26d340dcba19518c418d6048e913186e17243982c5e435e41fa7a/coverage-7.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:7af486dabe8954d03b087f0021540897afe084f04e16ff5579e08cc46f871416", size = 223394, upload-time = "2026-05-26T20:39:53.472Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/73/c6/02d56e3867972f77d5036de924643f26c056e848f00452cafb4dbc3c29b4/coverage-7.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:2224f89ffd0c5605ccce1ed7a584da162bc7c55f601ab1c946bc9de31a486b42", size = 222015, upload-time = "2026-05-26T20:39:55.374Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/4d/9e/fcc77914050df73f7662fa1f00902774c79c075a8388ab334074574bf77e/coverage-7.14.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:de286598cc65d2b489411174b1faec2f5a7775fb3201fd925db2a76b4030f37d", size = 220733, upload-time = "2026-05-26T20:39:57.189Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f7/67/2963cbdaf5cbadec44efa3a1e39eaa1f02df4079585f05387607a221e126/coverage-7.14.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:042c46ded7c288aeb07cf14a28b6c1e10b78fcba40171c3fa1e939377eeef0b5", size = 221086, upload-time = "2026-05-26T20:39:59.019Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c8/c5/8701645574e11881f2f47d8930f98bc48b5d43b25eb5b4430dfc4a2f9f48/coverage-7.14.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:f4ddbe407477f04c45115d1a4e5bc480f753553b534d338d4c3358b1cdd0ea52", size = 262381, upload-time = "2026-05-26T20:40:00.822Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7c/28/7a64d73598263e0c5abd5084211a8474488d31b3c552ff531c719dfcff62/coverage-7.14.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d13e6725992e2d2fd7d81d4f5241952d13740121dfd501da09201be39b2c003a", size = 264458, upload-time = "2026-05-26T20:40:02.506Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fa/d8/4969179db9f7eb4df218e69540adf829d1c835f59452513d065d15446802/coverage-7.14.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f747dc8edcfe740130f28f32f3995e955494285717e86ee25af51db2219df08a", size = 266884, upload-time = "2026-05-26T20:40:04.421Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a6/78/a45d5794dbc9bafd97afc96a4377c86c7820d78b6cf51b89bc1d4e919275/coverage-7.14.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ced2f09ef276fd58611a1ef502164ad266d2b75174e5a40cabbdb4033f9f6cf2", size = 268022, upload-time = "2026-05-26T20:40:06.298Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/21/cb/4f5e354e9e3e67af96bd4e57113e6db6b22298c7168b13eec408a549903d/coverage-7.14.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b84800013769a78ccb9ef4659402e26d06867e337b61ec365f77ad008adea80e", size = 261631, upload-time = "2026-05-26T20:40:08.226Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ec/49/eced49af4cb996d5d8b7e94e736175c513e4facd3398507b89892b4326d8/coverage-7.14.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:ea8cd6ca0ee9f616aaef3afc6882e32c2cbf18b00d96313ffd76af650574034d", size = 264443, upload-time = "2026-05-26T20:40:10.137Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f1/d8/5603a88a7c5913a6b54f6cb1a8c46f7b39cbb30f27cd3f492908da09b2d7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:aa5e304a873fabddc11e484e9b6b738bd38bd7bed17b09aa84eecf5332e8b8bb", size = 262069, upload-time = "2026-05-26T20:40:11.999Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f0/59/2ae3cb79da554a06c8619d6c88ea19dd1e4aed4b834b6a83bb1fa243bdc5/coverage-7.14.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:5a1c5215be81035e629d5bc756650634d0bf31991038db7a0eccb90f025ce16d", size = 265780, upload-time = "2026-05-26T20:40:13.858Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/af/5f/b130c1dc999031f2648bd25317fbce505ad8d5562079b4ed81e736a84967/coverage-7.14.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:79058c47dae6788504b5effb319961bcd72d7240551464b91d474bc0ed186d69", size = 260970, upload-time = "2026-05-26T20:40:16.142Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/87/d1/ec13ccddeb48ec963bdfa72a11224bac2584bd045ba13beca82f8113e9c7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:370c5afae3fa0658e11694a32b24c2778f6bc2d17718121f94ee185e69f26b54", size = 263157, upload-time = "2026-05-26T20:40:18.382Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cf/c2/cd91ead503045161092d3845f7bb95ea2f25131ce96d3e314dd835d91b9c/coverage-7.14.1-cp313-cp313t-win32.whl", hash = "sha256:3758dd0a7f1fa57365ef2e781df0f0731d38b6e3772259d13dae4bd8a958d4b1", size = 223259, upload-time = "2026-05-26T20:40:20.381Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/9f/1e28d97e6bd2c76b07f38b7c02870f1371255ff6717f54eca578fcbbdd0e/coverage-7.14.1-cp313-cp313t-win_amd64.whl", hash = "sha256:6ff665fb023a77386fe11685190cee1f60a7d635994a30d9b0a061533d470fce", size = 224320, upload-time = "2026-05-26T20:40:22.316Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a9/e0/d936e908f0e1efa55e52b91e01b52f1055cef5e1ab2718493390ed8e2fb8/coverage-7.14.1-cp313-cp313t-win_arm64.whl", hash = "sha256:17a5a241e5997621a956a7f402a7433ef4221e5152809b785bec79e2323799f1", size = 222577, upload-time = "2026-05-26T20:40:24.894Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d6/34/fc2f101b151af3799a101f0550b0454aa008afdc0add677394ec4aa8ea10/coverage-7.14.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d5ed429d0b8edaac649e889b4ffcedb6c80b06629a3f93050e3dddfb99235bee", size = 220091, upload-time = "2026-05-26T20:40:27.249Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/a7/1ebae2ab5b961b5c79bb09fe7b3ac99edb190d8be4a8c510b2cf66f46468/coverage-7.14.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8011224a62280e50dab346960c03cf47aca1a1e09e608c0fb33fd6e0cc8e9500", size = 220421, upload-time = "2026-05-26T20:40:30.084Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5e/90/92aca9cf0acc95123c96cd1eb1f08917897a7f5dee01e15738922971ec31/coverage-7.14.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:12c42ec1e14f553c4f817e989365982e646e27211f10a0f717855b94a79c8906", size = 251466, upload-time = "2026-05-26T20:40:32.542Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/26/2b/78048cbe3b999f6cbf9cc0d90abba6a88a3e0863a8c1c6cbc762f3f8802f/coverage-7.14.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:06144cd511cf2624873a035c5069cf297144f6e77a73ee3d7a55b605ec5efb42", size = 253973, upload-time = "2026-05-26T20:40:34.473Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/21/c2e33b29d1cfde484a19d437afc343c6cd30b08d78cbbf9f5aff14e57b2b/coverage-7.14.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a311d8e1da24be5c1ccf85cbfb06315dbaa1703d5a1eab3f6432c72b837917c8", size = 255318, upload-time = "2026-05-26T20:40:38.154Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/ee/aad2f108d63b769121005302f16bf66db8625c88ceaba466942e09a2607e/coverage-7.14.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c79cead5b5bc584d9c71451cb984d0e3a84e0c0937379c8efcbf27c8d661b851", size = 257633, upload-time = "2026-05-26T20:40:40.164Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c2/f8/11a2c29b4fd76d9849f81d0bb812ec0017a9396df3217214e38934a8c837/coverage-7.14.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:dcbf65f1f66a26cdd88c35cf68fb4729c5d1cd2e88added72420541dfb212034", size = 251488, upload-time = "2026-05-26T20:40:42.631Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/b8/9a5820de4b8ac2b71d85e3b5fb49108d7469c665f0e2ad0dd7569023e305/coverage-7.14.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fd86572566fb40189a8260446158235159bc7a82dfbc87a3b39cf4fb57fcec1c", size = 253329, upload-time = "2026-05-26T20:40:45.208Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6b/ff/f33e4823667e27548e8fd8df44217515303f9808d0ff29817db56f87d990/coverage-7.14.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:7771b601718fdde84832c3a434ca9bbf4ae9adbc49d84198b4110700c3c77c36", size = 251291, upload-time = "2026-05-26T20:40:47.502Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/68/9b/489db0ebb209054766b90a9014a45f6d26eb724c02ec21311c3733b5a644/coverage-7.14.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:39b21e212c55af06fa375e3dbf90a8a8e38792f3a910c580066d23563830ddd5", size = 255564, upload-time = "2026-05-26T20:40:49.372Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/b5/16bc2d4c2409b23c7737edb68c83bc89e345f378050549fe1d75ac7d34d5/coverage-7.14.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:f2302660e32562a532b442480121aef8aa61a5bdb20b30bf0adab29f10a5a4b4", size = 251107, upload-time = "2026-05-26T20:40:51.677Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/0c/2629997469a00cd069d588a41c9dc887610f2775ae89d250c4791e65272a/coverage-7.14.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:03a6f93c1ec3b7f2e77b5dbcc5573a2c21f12529a5c6bbe0f16f72303cc2fa4d", size = 252764, upload-time = "2026-05-26T20:40:54.267Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d2/ee/f78d63c8f079e0d7211c7e2401fa17e311514534ba61bae03e4b287ce4ab/coverage-7.14.1-cp314-cp314-win32.whl", hash = "sha256:8a3ce026d73290f42f08dafecbd82c193a74df280461fbf97300fec51fd133ee", size = 222837, upload-time = "2026-05-26T20:40:56.496Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/dc/b9/be539854f93a70dfbeec69117f33ec70dc42ff0b65b5b07ab8d40d04228e/coverage-7.14.1-cp314-cp314-win_amd64.whl", hash = "sha256:114c95ef29302423b87d159075805f4ab973254a2638a5d7d046c94887cc87d7", size = 223650, upload-time = "2026-05-26T20:40:58.351Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fe/9e/24e2842fef40f35ac82ba3a7719c8023d011bf3bf652d0675316a9d088a1/coverage-7.14.1-cp314-cp314-win_arm64.whl", hash = "sha256:a07891c3f4805442b31b71e84ba3cf29ed1aa9a428284e06deeb4b23e5b46343", size = 222218, upload-time = "2026-05-26T20:41:00.321Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0a/1d/ac0a9df5fe31c1e8bdd658074905fc12844a05c1a7e3fdb8417e97c31e23/coverage-7.14.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:1101a5ebb083aecb625ebb6209d4105b58f647b093cb2dc8122d7b33f743cfe1", size = 220822, upload-time = "2026-05-26T20:41:02.281Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/32/cf/f964fd9aff20323f9f1a726c97135f8a76bcd87b92dad141a456a43f3c64/coverage-7.14.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:851b9e1e4e8a4608e77c79714b2e77c0970d2ed7202a05e92ae407817481887b", size = 221084, upload-time = "2026-05-26T20:41:04.593Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d8/5e/7e5ef2aba844de2b80d678619fcf0841b42e3f37f16411226f3fe4c1016f/coverage-7.14.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d5b89cdfb2ee051b71e8c3c70bd81a9eff81100f736a269136fe1a68efe00474", size = 262454, upload-time = "2026-05-26T20:41:06.641Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/64/62/75809bded87015cc4935524218a2a8ed8dd1a8498bfed30a2f4f7a4b4d34/coverage-7.14.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0177614a0370f227888b4e436a7c55686d6a9f90eb1ade2b624ba685a1686e86", size = 264578, upload-time = "2026-05-26T20:41:08.556Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f3/42/d33392dc14633525012d2d504fa1a33b05538bf535f5c1d64675e5754b78/coverage-7.14.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d69af5dea2de76fc485a83032a630523f985198b7e25be901ec60181587b01e", size = 266981, upload-time = "2026-05-26T20:41:10.824Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2a/49/0157c4428c2aca7f1e09d5565930586fd5ae36f1655f08b0daa7cf1fcae1/coverage-7.14.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:35ab22d91de736e8966b980dc355cbcdd2c6dbbcfe275f9a2991bc8a91b3df65", size = 268112, upload-time = "2026-05-26T20:41:12.966Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/96/26/86b9ce71f4092b1ed325ce1421698081df1286b833400b6836912834d6e0/coverage-7.14.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:357d4e32935c36588aaba057d734fa32428c360c9fc2e4442afbf1b646beee6e", size = 261558, upload-time = "2026-05-26T20:41:15Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/20/4c/c311210c5472cf5401d8422b0d7812cdd520f24417673afabda6c323faca/coverage-7.14.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:51bd64741cc6fa065abd300ede1afe5a5291ece9c31da8b24884deda48bcc3f8", size = 264447, upload-time = "2026-05-26T20:41:17.369Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/71/59513f8710ed3e6b0ac0a050a5b7e977bb9c9e880354863b5d00d8809256/coverage-7.14.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:9132cd363a68a4c3daa7c8704a654b1e39d3360f6f5b8ddd470608a945236c07", size = 262048, upload-time = "2026-05-26T20:41:19.309Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/84/8d/bceed32dc494f5bbf50f775cd2e78ca814953942b5ea28d3c1c3ac316f14/coverage-7.14.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:07c6290b1697b862c0478eab545eec949a0d0e4d6d03497f446d706da3b4f2de", size = 265781, upload-time = "2026-05-26T20:41:21.559Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e7/c5/9348fe40dbfd4991aaf78df2c6c3098bfb2cc834d1fd362a64b4efef855a/coverage-7.14.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:5ea0c297e27133853b4d8a3eb799bff5a2dbd9f2f41537a240d337ac9b4df890", size = 260896, upload-time = "2026-05-26T20:41:23.428Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/92/1ea0f03929da7cf87206b1fa24f4c8e9c158be0455481af29ec0a1f3503f/coverage-7.14.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:01b7733daad0237daa01ef80fe2dfceffc911e6a17fa7b55d14aa8214eaaaecd", size = 263214, upload-time = "2026-05-26T20:41:25.419Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/a9/b2493c054c0e01a643266742ab45e15744e60743f9260cd930c7142b1124/coverage-7.14.1-cp314-cp314t-win32.whl", hash = "sha256:6adc5a36984624a70bf11d7184e20fa0a49aa7c47ffab43804106a1a695ea22e", size = 223624, upload-time = "2026-05-26T20:41:27.795Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/bd/3e1e6a57fccd2d7c83fcdf338e93ba98eb85c6e877dd34731ac585375490/coverage-7.14.1-cp314-cp314t-win_amd64.whl", hash = "sha256:ddf799247318f34dbcd2efa8c95a8d0642674e926bb1774cf9b63dfd2a389d1c", size = 224728, upload-time = "2026-05-26T20:41:30.098Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bb/d7/31066cf1d2f0c6c797fce911bcfa01dd35642dc6da992a950256097c5860/coverage-7.14.1-cp314-cp314t-win_arm64.whl", hash = "sha256:145986fe66647eb489f18d9a997567a3fd358584c4b5a808769113abc07466af", size = 222752, upload-time = "2026-05-26T20:41:32.123Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/3c/1a983b9a745d7f83d53f057bcc5bf79ba6a2bbc08266b3f0c7d6fe630c9b/coverage-7.14.1-py3-none-any.whl", hash = "sha256:a252f21c27e38347e60111a3266b03827422a7d5525951aceee313aa68bab1d2", size = 211815, upload-time = "2026-05-26T20:41:34.078Z" },
]
[[package]]
name = "curl-cffi"
version = "0.15.0"
@@ -622,6 +748,92 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" },
]
[[package]]
name = "greenlet"
version = "3.5.3"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e2/f1/fbbfef6af0bad0548f09bc28948ea3c275b4edb19e17fc5ca9900a6a634d/greenlet-3.5.3.tar.gz", hash = "sha256:a61efc018fd3eb317eeca31aba90ee9e7f26f22884a79b6c6ec715bf71bb62f1", size = 200270, upload-time = "2026-06-26T19:28:24.832Z" }
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/a1/1f7f0c555f5858fd2906fe9f7b0a3554fddb85cb70df7a6aaec41dc292c2/greenlet-3.5.3-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:c180d22d325fb613956b443c3c6f4406eb70e6defc70d3974da2a7b59e06f48c", size = 285838, upload-time = "2026-06-26T18:21:05.167Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0a/29/be9f43ed61677a5759b38c8a9389248133c8c731bbfc0574ecdff66c99fc/greenlet-3.5.3-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:483d08c11181c83a6ce1a7a61df0f624a208ec40817a3bb2302714592eee4f04", size = 602342, upload-time = "2026-06-26T19:07:06.908Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b9/42/ba41c97ec36aa4b3ec25e5aa691d79561254805fad7f2f826dd6770587e2/greenlet-3.5.3-cp310-cp310-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1dae6e0091eae084317e411f047f0b7cb241c6db570f7c45fd6b900a274914ce", size = 615541, upload-time = "2026-06-26T19:10:04.909Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2e/8c/231ca675b0df779816950ca66b40b1fa14dbff4a0ed9814a9a29ec399140/greenlet-3.5.3-cp310-cp310-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0f6ff50ff8dbd51fae9b37f4101648b04ea0df19b3f50ab2beb5061e7716a5c8", size = 622473, upload-time = "2026-06-26T19:24:12.786Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/c7/28747042e1df8a9cd120a1ebe15529fc4be3b486e13e8d551ff307a82412/greenlet-3.5.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9bcd2d72ccd70a1ec68ba6ef93e7fbb4420ef9997dabc7010d893bd4015e0bec", size = 615675, upload-time = "2026-06-26T18:32:14.444Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/81/fe/dd97c483a3ff82849196ccd07851600edd3ac9de74669ca8a6022ada9ea1/greenlet-3.5.3-cp310-cp310-manylinux_2_39_riscv64.whl", hash = "sha256:37bf9c538f5ae6e63d643f88dec37c0c83bdf0e2ebc62961dedcf458822f7b71", size = 418421, upload-time = "2026-06-26T19:25:34.503Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cc/a8/b85525a6c8fba9f009a5f7c8df1545de8fb0f0bf3e0179194ef4e500317f/greenlet-3.5.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:73f152c895e09907e0dbe24f6c2db37beb085cd63db91c3825a0fcd0064124a8", size = 1575057, upload-time = "2026-06-26T19:09:00.264Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/03/79/fb76edb218fe6735ab0edeba176c7ab80df9618f7c02ce4208979f3ae7db/greenlet-3.5.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8bdb43e1a1d1873721acab2be99c5befd4d2044ddfd52e4d610801019880a702", size = 1641692, upload-time = "2026-06-26T18:31:41.454Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6b/79/86fe3ee50ed55d9b3907eecd3208b5c3fe8a79515519aae98b4753c3fa1d/greenlet-3.5.3-cp310-cp310-win_amd64.whl", hash = "sha256:0909f9355a9f24845d3299f3112e266a06afb68302041989fd26bd68894933db", size = 238742, upload-time = "2026-06-26T18:20:40.758Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/58/5404031044f55afad7aad1aff8be3f22b1bed03e237cfeabbc7e5c8cfde0/greenlet-3.5.3-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:aca9b4ce85b152b5524ef7d88170efdff80dc0032aa8b75f9aaf7f3479ea95b4", size = 287424, upload-time = "2026-06-26T18:20:31.469Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/bf/1c65e9b94a54d547068fa5b5a8a06f221f3316b48908e08668d29c77cb50/greenlet-3.5.3-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0f71be4920368fe1fabeeaa53d1e3548337e2b223d9565f8ad5e392a75ba23fc", size = 606523, upload-time = "2026-06-26T19:07:08.859Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/c7/b66baacc95775ad511287acb0137b95574a9ce5491902372b7564799d790/greenlet-3.5.3-cp311-cp311-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4d77e67f65f98449e3fb83f795b5d0a8437aead2f874ca89c96576caf4be3af6", size = 618315, upload-time = "2026-06-26T19:10:06.055Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b0/a0/68afd1ebad40db87dac0a28ffa120726b98bf9c7c40c481b0f63c105d298/greenlet-3.5.3-cp311-cp311-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e18619ba655ac05d78d80fc83cac4ba892bd6927b99e3b8237aee861aaacc8bb", size = 626155, upload-time = "2026-06-26T19:24:14.44Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/78/2b/28ed29463522fdbe4c15b1f63922041626a7478316b34ab4adda3f0a4aba/greenlet-3.5.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8540f1e6205bd13ca0ce685581037219ca54a1b41a0a15d228c6c9b8ad5903d7", size = 617381, upload-time = "2026-06-26T18:32:16.077Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/07/7f/e327d912239ec4b3b49999e3967389bcf1ee8722b9ee9194d2752ecd558a/greenlet-3.5.3-cp311-cp311-manylinux_2_39_riscv64.whl", hash = "sha256:d27c0c653a60d9535f690226474a5cc1036a8b0d7b57504d1c4f89c44a07a80c", size = 421083, upload-time = "2026-06-26T19:25:35.804Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2a/7b/ad04e9d1337fc04965dc9fc616b6a72cb65a24b800a014c011ec812f5489/greenlet-3.5.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7ef56fe650f50575bf843acde967b9c567687f3c22340941a899b7bc56e956a8", size = 1577771, upload-time = "2026-06-26T19:09:01.537Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d8/33/6c87ab7ba663f70ca21f3022aad1ffe56d3f3e0521e836c2415e13abcc3c/greenlet-3.5.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5121af01cf911e70056c00d4b46d5e9b5d1415550038573d744138bacb59e6b8", size = 1644048, upload-time = "2026-06-26T18:31:42.996Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/1c/35/f0d8ee998b422cf8693b270f098e55d8d4ec8006b061b333f54f177d28d9/greenlet-3.5.3-cp311-cp311-win_amd64.whl", hash = "sha256:0f41e4a05a3c0cb31b17023eff28dd111e1d16bf7d7d00406cd7df23f31398a7", size = 239137, upload-time = "2026-06-26T18:23:21.664Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/96/b9820295576ef18c9edc404f10e260ae7215ceaf3781a54b720ed2627862/greenlet-3.5.3-cp311-cp311-win_arm64.whl", hash = "sha256:ec6f1af59f6b5f3fc9678e2ea062d8377d22ac644f7844cb7a292910cf12ff44", size = 237630, upload-time = "2026-06-26T18:24:00.281Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5d/6e/4c37d51a2b7f82d2ff11bb6b5f7d766d9a011726624af255e843727627a3/greenlet-3.5.3-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:719757059f5a53fd0dde23f78cffeafcdd97b21c850ddb7ca684a3c1a1f122e2", size = 288685, upload-time = "2026-06-26T18:22:08.977Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/73/815dd90131c1b71ebdf53dbc7c276cafec2a1173b97559f97aba72724a87/greenlet-3.5.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:efa9f765dd09f9d0cdac651ffdf631ee59ec5dc6ee7a73e0c012ba9c52fbdf5b", size = 604761, upload-time = "2026-06-26T19:07:10.114Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9f/57/079cfe76bcef36b153b25607ee91c6fcb58f17f8b23c86bbbeabe0c88d72/greenlet-3.5.3-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7faba15ac005376e02a0384504e0243be3370ce010296a44a820feb342b505ab", size = 617044, upload-time = "2026-06-26T19:10:07.25Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/fb/d97dc261209c80744b7c8132693a30d70ec6e7315e632cb0a10b3fec94dd/greenlet-3.5.3-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5795cd1101371140551c645f2d408b8d3c01a5a29cf8a9bce6e759c983682d23", size = 622351, upload-time = "2026-06-26T19:24:16.32Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/37/87/b4d095775a3fb1bcafbb483fc206b27ebb785724c83051447737085dc54e/greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:87142215824be6ac05e2e8e2786eec307ccbc27c36723c3881959df654af6861", size = 614244, upload-time = "2026-06-26T18:32:17.594Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/ac/e5fee13cbbd0e8de312d9a146584b8a51891c68847330ef9dc8b5109d23f/greenlet-3.5.3-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:af4923b3096e26a36d7e9cf24ab88083a20f97d191e3b97f253731ce9b41b28c", size = 425395, upload-time = "2026-06-26T19:25:37.144Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/70/7559b609683650fa2b95b8ab84b4ab0b26556a635d19675e12aa832d826d/greenlet-3.5.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:215275b1b49320987352e6c1b054acca0064f965a2c66992bed9a6f7d913f149", size = 1574210, upload-time = "2026-06-26T19:09:03.077Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ae/73/be55392074c60fc37655ca40fa6022457bfbf6718e9e342a7b0b41f96dd2/greenlet-3.5.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6b1b0eed82364b0e32c4ea0f221452d33e6bb17ae094d9f72aed9851812747ea", size = 1638627, upload-time = "2026-06-26T18:31:44.748Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/14/40/c57489acf8e37d74e2913d4eff63aa0dba17acccc4bdeef874dde2dbbec9/greenlet-3.5.3-cp312-cp312-win_amd64.whl", hash = "sha256:cde8adafa2365676f74a979744629589999093bc86e2484214f58e61df08902c", size = 239882, upload-time = "2026-06-26T18:23:27.518Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/fd/6fea0e3d6600f785069481ee637e09378dd4118acdfd38ad88ae2db31c98/greenlet-3.5.3-cp312-cp312-win_arm64.whl", hash = "sha256:c4e7b79d83805475f0102008843f6eb45fd3bb0b2e88c774adab5fbaab27117d", size = 238211, upload-time = "2026-06-26T18:22:37.671Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9b/ff/a620267401db30a50cc8450ee90730e2d4a85658c055c0e760d4ed47fb13/greenlet-3.5.3-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:c8d87c2134d871df96ecdea9cec7cbaab286dadab0f56476e57aaf9e8ac11550", size = 287609, upload-time = "2026-06-26T18:21:14.724Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d6/fa/5401ac78021c826a25b6dde0c705e0a8f29b617509f9185a31dac15fbe1b/greenlet-3.5.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a2d185dd1621757e70c3861cceffd5317ab4e7ed7eb09c82994828468527ade5", size = 607435, upload-time = "2026-06-26T19:07:11.412Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e9/76/1dc144a2e56e65d36405078ed774224375ea520a1870a6e46e08bb4ac7bf/greenlet-3.5.3-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1c514a468149bf8fbbab874188a3535cd8a48a3e353eb53a3d424296f8dbacd3", size = 619787, upload-time = "2026-06-26T19:10:08.396Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/57/61/2f5b1adf256d039f5dab8005de8d3d7ad2b0070a3219c0e036b3fbfeb440/greenlet-3.5.3-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9ad04dd75458c6300b047c61b8639092433d205a25a14e310d6582a480efcca1", size = 625580, upload-time = "2026-06-26T19:24:18.344Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bf/87/c298cee62df1de4ad7fec32abda73526cff347fd143a6ed4ac369246668a/greenlet-3.5.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:915f887cf2682b66419b879423a2e072634aa7b7dce6f3ada4957cfced3f1e9a", size = 616786, upload-time = "2026-06-26T18:32:19.128Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3e/d9/ab7fc9e543e44d6879b0a6ef9a4b2188940fd180cc65d6f646883ddf7201/greenlet-3.5.3-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:afaabdd554cd7ae9bbb3ca070b0d7fdfd207dbf1d16865f7233837709d354bda", size = 427933, upload-time = "2026-06-26T19:25:38.219Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9e/2e/e6f009885ed0705ccf33fe0583c117cfd03cde77e31a596dd5785a30762b/greenlet-3.5.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:766cfd421c13e450feb340cd472a3ed9957d438727b7b4593ad7c76c5d2b0deb", size = 1574316, upload-time = "2026-06-26T19:09:04.273Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ef/fe/43fd110b01e40da0adb7c90ac7ea744bef2d43dca00de5095fd2351c2a68/greenlet-3.5.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2ecda9ec22edf38fa389369eaed8c3d37c05f3c54e69f69438dbb2cc1de1458b", size = 1638614, upload-time = "2026-06-26T18:31:46.297Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0f/7c/062447147a61f8b4337b156fe70d32a165fcf2f89d7ca6255e572806705c/greenlet-3.5.3-cp313-cp313-win_amd64.whl", hash = "sha256:c82304750f057167ff60d188df1d0cc1764ce9567eadf03e6a7443bcedd0b30b", size = 239850, upload-time = "2026-06-26T18:21:54.613Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c7/7e/220a7f5824a64a60443fc03b39dfac4ea63a7fb6d481efa27eafa928e7f4/greenlet-3.5.3-cp313-cp313-win_arm64.whl", hash = "sha256:dc133a1569ee667b2a6ef56ce551084aeefd87a5acbc4736d336d1e2edc6cfc4", size = 238141, upload-time = "2026-06-26T18:22:48.507Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c3/93/43e116ee114b28737ba7e12952a0d4e2f55944d0f84e42bc91ba7192a3c9/greenlet-3.5.3-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:fd2e02fa07485778536a036222d616ab957b1d533f36b3ed98ce725d9c9d3117", size = 288202, upload-time = "2026-06-26T18:23:49.604Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/82/2f/146d218299046a43d1f029fd544b3d110d0f175a09c715c7e8da4a4a345d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df0a0628d1597eb0897b62f55d1343f772405fd25f3b2a796c76874b0c2e22e8", size = 654096, upload-time = "2026-06-26T19:07:12.71Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/cc/04738cafb3f45fa991ea44f9de94c47dcec964f5a972300988a6751f49d9/greenlet-3.5.3-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ebd933a6adabc298bab47731a130fe6bfb888bd934eee37810f151159544540d", size = 666304, upload-time = "2026-06-26T19:10:09.503Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/86/a9/73fa62893d5b84b4205544e6b673c654cc43aa5b9899bac00f04d64af73d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8d19fe6c39ebff9259f07bcc685d3290f8fa4ea2278e51dd0008e4d6b0f2d814", size = 670657, upload-time = "2026-06-26T19:24:19.967Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ce/aa/4e0dad5e605c270c784ab911c43da6adb136ccd4d81180f763ca429a723d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4b9d501b40e80b70e32323c799dd9b420a5577a9601469d362ae1ffb690f3a7c", size = 663635, upload-time = "2026-06-26T18:32:20.802Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/29/7e/2ffce64929fb3cab7b65d5a0b20aaf9764e227681d731b041077fc9a525a/greenlet-3.5.3-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:962c5df2db8cb446da51edf1ca5296c389d93b99c9d8aa2ee4c7d0d8f1218260", size = 473497, upload-time = "2026-06-26T19:25:39.421Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d1/50/13efdbea246fe3d3b735e191fec08fb50809f53cd2383ebe123d0809e44b/greenlet-3.5.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a1fad1d11e7d6aab184107baa8e4ece11ccba3ec9599cd7efa5ff4d70d43256a", size = 1621252, upload-time = "2026-06-26T19:09:05.647Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f7/22/c0a336ae4a1410fd5f5121098e5bfbf1865f64c5ef80b4b5412886c4a332/greenlet-3.5.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:fad5aec764399f1b5cc347ad250a59660f20c8f8888ea6bae1f93b769cce1154", size = 1684824, upload-time = "2026-06-26T18:31:47.738Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/94/91aec0030bea75c4b3244251d0de60a1f3432d1ecb53ab6c437fb5c3ba61/greenlet-3.5.3-cp314-cp314-win_amd64.whl", hash = "sha256:7669aa24cf2a1041d6f7899575b494a3ab4cf68bfcc8609b1dc0be7272db835e", size = 240754, upload-time = "2026-06-26T18:22:15.669Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e5/06/68d0983e79e02138f64b4d303c500c27ddb48e5e77f3debb80888a921eae/greenlet-3.5.3-cp314-cp314-win_arm64.whl", hash = "sha256:5b4807c4082c9d1b6d9eed56fcd041863e37f2228106eef24c30ca096e238605", size = 239549, upload-time = "2026-06-26T18:22:42.996Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/91/95/3e161213d7f1d378d15aa9e792093e9bfe01844680d04b7fd6e0107c9098/greenlet-3.5.3-cp314-cp314t-macosx_11_0_universal2.whl", hash = "sha256:271a8ea7c1024e8a0d7dd2be66dd66dda8a07193f41a17b9e924f7600f5b62be", size = 296389, upload-time = "2026-06-26T18:22:20.657Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/00/92/715c44721abe2b4d1ae9abde4179411868a5bff312479f54e105d372f131/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:19131729ae0ddc3c2e1ef85e650169b5e37ee32e400f215f78b94d7b0d567310", size = 653382, upload-time = "2026-06-26T19:07:14.209Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/83/37a10372a1090a6624cca8e74c12df1a36c2dc36429ed0255b7fb1aeee23/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1540dd8e5fc2a5aec40fbb98ef8e149fa47c89a4b4a1cf2575a14d3d1869d7a8", size = 659401, upload-time = "2026-06-26T19:10:10.876Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cb/73/8faec206b851c22b1733545fda900829a1f3f5b1c78ae7e0fb3dba57d9f4/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b897d97759425953f69a9c0fac67f8fe333ec0ce7377ef186fb2b0c3ad5e354d", size = 659582, upload-time = "2026-06-26T19:24:21.357Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/db/e2/d1509cad4207da559cc42986ecdd8fc67ad0d1bba2bf03023c467fd5e0f3/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e81fa194a1d20967877bdf9c7794db2bc99063e5be36aee710c08f04c5bb087f", size = 656969, upload-time = "2026-06-26T18:32:22.272Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/55/50c19e49f8045834ada71ef12f8ad048eba8517c6aa41161bed676328fae/greenlet-3.5.3-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:3236754d423955ea08e9bb5f6c04a7895f9e22c290b66aa7653fcb922d839eb0", size = 491037, upload-time = "2026-06-26T19:25:40.672Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/86/7d/eaf70de20aadca3a5884aec58362861c64ce45e7b277f47ed026926a3b89/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:55cf4d777485d43110e47133cbba6d74a8885a87ec1227ef0267f9ee80c5aa21", size = 1617822, upload-time = "2026-06-26T19:09:06.893Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/f9/414d38fc400ae4350d4185eaad1827676f7cf5287b9136e0ed1cbbe20a7f/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:12a248ba75f6a9a236375f52296c498c89ff1d8badf32deb9eca7abd5853f7da", size = 1677983, upload-time = "2026-06-26T18:31:49.396Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e4/15/7edb977e08f9bff702fe42d6c902702786ff6b9694058b4e6a2a6ac90e57/greenlet-3.5.3-cp314-cp314t-win_amd64.whl", hash = "sha256:efc6bd60ea02e085862c74a3ef64b147ffc6f1a5ea7d9f26e7a939943f68c1e3", size = 243626, upload-time = "2026-06-26T18:24:41.485Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/8a/93928dce91e6b3598b5e779e8d1fd6576a504640c58e78627077f6a7a91a/greenlet-3.5.3-cp315-cp315-macosx_11_0_universal2.whl", hash = "sha256:ea03f2f04367845d6b58eeed276e1e56e51f0b97d8ad5a88a7d20a91dc9056cc", size = 288860, upload-time = "2026-06-26T18:22:48.07Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/4f/ca/69db42d447a1378043e2c8f19c09cbbd1263371505053c496b49066d3d16/greenlet-3.5.3-cp315-cp315-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:78dbef602fda6d97d957eb7937f70c9ce9e9527330347f8f6b6f9e554a9e7a47", size = 659747, upload-time = "2026-06-26T19:07:15.565Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a8/0b/af7ac2ef8dd41e3da1a40dda6305c23b9a03e13ba975ec916357b50f8575/greenlet-3.5.3-cp315-cp315-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6f73857adb8fee13fa56c172bd11262f888c0c648f9fea113e777bb2c7904a81", size = 670419, upload-time = "2026-06-26T19:10:12.293Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/25/aa/952cf28c2ff949a8c971134fb43854dd7eaa737218723aaef758f8c9aead/greenlet-3.5.3-cp315-cp315-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:cefa9cef4b371f9844c6053db71f1138bc6807bab1578b0dae5149c1f1141357", size = 674261, upload-time = "2026-06-26T19:24:22.79Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/1e/1d51640cacbfc455dbe9f9a9f594c49e4e244f63b9971a2f4764e46cc53d/greenlet-3.5.3-cp315-cp315-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:232fec92e823addaf02d9472cf7381e24a1d046a6ced1103c5caa4c21b9dfc1d", size = 668787, upload-time = "2026-06-26T18:32:24.298Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/dc/f2/b00d6f5e63e531a93562b2ec1a4c320fbee91f580fc42e6417af69d706e5/greenlet-3.5.3-cp315-cp315-manylinux_2_39_riscv64.whl", hash = "sha256:6219b6d04dbf6ba6084d77dc609e8473060dc55f759cbf626d512122781fa128", size = 480322, upload-time = "2026-06-26T19:25:41.852Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/21/66/4030d5b0b5894500023f003bb054d9bb354dfbd1e186c3a296759172f5f5/greenlet-3.5.3-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:2421c3564da9429d5586d46ca31ebb26516b5498a802cf65c041a8e8a8980d34", size = 1626305, upload-time = "2026-06-26T19:09:08.281Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0e/50/5221371c7550108dfa3c378debc41d032aa9c78e89abb01d8011cfc93289/greenlet-3.5.3-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e0f0d160f0b2e558e6c75f7930967183255dc9735e5f5b8cae58ee09c9576d8b", size = 1688631, upload-time = "2026-06-26T18:31:51.278Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/68/5d/00d469daae3c65d2bf620b10eee82eb022127d483c6bc8c69fae6f3fbf17/greenlet-3.5.3-cp315-cp315-win_amd64.whl", hash = "sha256:dd99329bbc15ca78dcc583dba05d0b1b0bae01ab6c2174989f5aaee3e41ac930", size = 241027, upload-time = "2026-06-26T18:22:38.203Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e7/e8/883785b44c5780ed71e83d3e4437e710470be17a2e181e8b601e2da0dc4a/greenlet-3.5.3-cp315-cp315-win_arm64.whl", hash = "sha256:499fef2acede88c1864a57bb586b4bf533c81e1b82df7ab93451cdb47dfec227", size = 240085, upload-time = "2026-06-26T18:23:54.217Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/1c/da/4f4a8450962fad137c1c8981a3f1b8919d06c829993d4d476f9c525d5173/greenlet-3.5.3-cp315-cp315t-macosx_11_0_universal2.whl", hash = "sha256:176bc16a721fa5fc294d70b87b4dfa5fbdd251b3da5d5372735ecef9bd7d6d0c", size = 297221, upload-time = "2026-06-26T18:23:27.176Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/57/66/b3bfae3e220a9b63ea539a0eea681800c69ab1aada757eae8789f183e7ce/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:629b614d2b786e89c50440e246f33eea78f58a962d0bdbbcc809e6d13605903f", size = 657221, upload-time = "2026-06-26T19:07:16.973Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7b/81/b6d4d73a709684fc77e7fa034d7c2fe82cffa9fc920fadcaa659c2626213/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2b2e857ae16f5f72142edf75f9f176fe7526ba19a2841df1420516f83831c9f2", size = 663226, upload-time = "2026-06-26T19:10:13.723Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e9/39/0e0938a75115b939d42733a2a12e1d349653c9531fe6fe563e8a681f04e6/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:16d192579ed281051396dddd7f7754dac6259e6b1fb26378c87b66622f8e3f91", size = 663706, upload-time = "2026-06-26T19:24:24.312Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/07/e210b02b589f16e74ff48b730690e4a34ffe984219fce4f3c1a0e7ec8545/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e515757e2e36bcbf1fad09a46e1557e8b1ae1797d4b44d09da7deed88ad28608", size = 660802, upload-time = "2026-06-26T18:32:26.081Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5b/41/35d1c678cdb3c3b9e6bee691728e563cfb294202b23c7a4c3c2ccc343589/greenlet-3.5.3-cp315-cp315t-manylinux_2_39_riscv64.whl", hash = "sha256:4399eb8d041f20b68d943918bc55502a93d6fdc0a37c14da7881c04139acee9d", size = 498803, upload-time = "2026-06-26T19:25:43.063Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/eb/2e/5303eb3fa06bca089060f479707182a93e360683bc252acf846c3090d34e/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:b363d46ed1ea431825fdb01471bb024fc08399bad1572a616e853c7684415adb", size = 1622157, upload-time = "2026-06-26T19:09:09.527Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/54/70/50de47a488f14df260b50ae34fb5d56016e308b098eab02c878b5223c26a/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:e44da2f5bbdaabaf7d80b73dbb430c7035771e9f244e3c8b769715c9d8fa0a16", size = 1681159, upload-time = "2026-06-26T18:31:52.986Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a7/13/1055e1dda7882073eda533e2b96c62e55bbd2db7fda6d5ece992febc7071/greenlet-3.5.3-cp315-cp315t-win_amd64.whl", hash = "sha256:8ff8bed3e3baa20a3ea261ce00526f1898ad4801d4886fd2220580ee0ad8fadf", size = 244007, upload-time = "2026-06-26T18:22:04.353Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/0d/ca7d15afbdc397e3401134c9e1800d51d12b829661786187a4ad08fe484f/greenlet-3.5.3-cp315-cp315t-win_arm64.whl", hash = "sha256:b7068bd09f761f3f5b4d214c2bed063186b2a86148c740b3873e3f56d79bac31", size = 242586, upload-time = "2026-06-26T18:23:37.93Z" },
]
[[package]]
name = "html5lib"
version = "1.1"
@@ -1412,6 +1624,25 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bc/60/5382c03e1970de634027cee8e1b7d39776b778b81812aaf45b694dfe9e28/pillow-12.2.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:bfa9c230d2fe991bed5318a5f119bd6780cda2915cca595393649fc118ab895e", size = 7080946, upload-time = "2026-04-01T14:46:11.734Z" },
]
[[package]]
name = "playwright"
version = "1.61.0"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
dependencies = [
{ name = "greenlet" },
{ name = "pyee" },
]
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/44/ee/31e4e0db36588b817a10b299a0285082545fde7d36543c2abe498bb3d61a/playwright-1.61.0-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:ff138c3a604f69911e9d42fd036e55c2a171e5616edf04c1e7f60a2a285540b0", size = 43421877, upload-time = "2026-06-29T10:32:48.428Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/42/35/71395dd3ecc798965be4a3ef8c443217d4abca168e7cb34536304f9489e6/playwright-1.61.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:009588c2a7e499bc5a8b425b61fa65490968bbda9cd69e0cf2cff10f8304659a", size = 42205016, upload-time = "2026-06-29T10:32:52.104Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f4/44/323164cf5cd1647bdefce76ffce27651aadb959d089b48f53ea40918276e/playwright-1.61.0-py3-none-macosx_11_0_universal2.whl", hash = "sha256:9f7de4536088d12037c13a52b7ea34b59270b78926bb56935070597ffac6b1af", size = 43421884, upload-time = "2026-06-29T10:32:55.773Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ab/f8/a35bf179e4ba2522c1893635094a64e407572547bd61528820fc0abc87fe/playwright-1.61.0-py3-none-manylinux1_x86_64.whl", hash = "sha256:54f3b39f6eab832e33458c1dd7da0b5682aedab3b09ae731b5c59fa12fd2024e", size = 47421381, upload-time = "2026-06-29T10:32:59.903Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b7/eb/e3f922348ec17c315f98c463f72faa1181a1c3de0bfe31a8d2edf6561723/playwright-1.61.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:93454322ade8c11d5d6c211bfd91bdfb9ffb4810e3e026371bcbc4bec1b7ee4c", size = 47120545, upload-time = "2026-06-29T10:33:03.574Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c2/a6/5be4e52b40a9c0c8a073e7c5b0785c05cf5a9ea8f8a7b5b260e32d970342/playwright-1.61.0-py3-none-win32.whl", hash = "sha256:372d55a6f1248fa1dd47599686980cb8fb5bbe6fcda59eab793eb657c11d8a9b", size = 37844841, upload-time = "2026-06-29T10:33:07.361Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6c/fd/2b78036e5fbe9d5f5645bbe08a1eac7160c51243c0093963edbcf67c35d9/playwright-1.61.0-py3-none-win_amd64.whl", hash = "sha256:35c6cc4589a5d00964a59d7b3e59641e0aac0c02f15479a7af77d20f6bc79597", size = 37844846, upload-time = "2026-06-29T10:33:10.637Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/0d/1b0f3c4ee4eb0514bc805b5c2f9a223e5b6de4f11a926f5235d51d0fc81b/playwright-1.61.0-py3-none-win_arm64.whl", hash = "sha256:e9fcbffcf557a8620fdedd92491eb59a32d18e23d6f3b4f6214b952be324fe51", size = 33955127, upload-time = "2026-06-29T10:33:14.008Z" },
]
[[package]]
name = "pluggy"
version = "1.6.0"
@@ -1496,6 +1727,18 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
]
[[package]]
name = "pyee"
version = "13.0.1"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
dependencies = [
{ name = "typing-extensions" },
]
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8b/04/e7c1fe4dc78a6fdbfd6c337b1c3732ff543b8a397683ab38378447baa331/pyee-13.0.1.tar.gz", hash = "sha256:0b931f7c14535667ed4c7e0d531716368715e860b988770fc7eb8578d1f67fc8", size = 31655, upload-time = "2026-02-14T21:12:28.044Z" }
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/c4/b4d4827c93ef43c01f599ef31453ccc1c132b353284fc6c87d535c233129/pyee-13.0.1-py3-none-any.whl", hash = "sha256:af2f8fede4171ef667dfded53f96e2ed0d6e6bd7ee3bb46437f77e3b57689228", size = 15659, upload-time = "2026-02-14T21:12:26.263Z" },
]
[[package]]
name = "pygments"
version = "2.20.0"