Compare commits

..

13 Commits

Author SHA1 Message Date
Yuxuan Yan c91415aff8 Document JoinQuant cost model findings 2026-07-06 15:35:41 +08:00
Yuxuan Yan efa9d24c73 Harden JoinQuant browser smoke automation 2026-07-04 19:27:48 +08:00
Yuxuan Yan 5fada75d23 Add JoinQuant env browser login helper 2026-07-04 18:37:34 +08:00
Yuxuan Yan d05931f41a Add JoinQuant simulated trading automation 2026-07-04 18:12:01 +08:00
Yuxuan Yan 39a93259e1 Add JoinQuant browser backtest automation 2026-07-04 18:08:31 +08:00
Yuxuan Yan 5388359dc8 Automate local JoinQuant smoke prep 2026-07-04 17:55:19 +08:00
Yuxuan Yan c200508f9e Align JoinQuant targets to execution dates 2026-07-04 17:50:55 +08:00
Yuxuan Yan f25db279bf Add JoinQuant comparison plugin 2026-07-04 17:43:09 +08:00
Yuxuan Yan 528620b271 Raise coverage threshold to 95% and expand test coverage
- pyproject.toml: fail_under 80 → 95
- test_alpha: +79 lines
- test_cli_workflow: +226 lines
- test_derived: +121 lines
- test_downloader_contracts: +169 lines
- test_features: +16 lines
- test_minute_downloader: +81 lines
- test_portfolio: +208 lines
2026-06-16 21:10:30 +08:00
Yuxuan Yan b5c8c0b8da Improve offline coverage for data boundaries 2026-06-16 17:42:20 +08:00
Yuxuan Yan 31baa18ce5 Add offline workflow and coverage tests 2026-06-16 17:37:16 +08:00
Yuxuan Yan 8d908477e2 Add daily derived data pipeline 2026-06-16 15:55:30 +08:00
Yuxuan Yan 83a006bbe4 Add minute bar feature pipeline 2026-06-16 13:57:17 +08:00
58 changed files with 9822 additions and 21 deletions
+1
View File
@@ -1,6 +1,7 @@
__pycache__/ __pycache__/
*.py[cod] *.py[cod]
.pytest_cache/ .pytest_cache/
.coverage
*.egg-info/ *.egg-info/
.venv/ .venv/
venv/ venv/
+78
View File
@@ -120,6 +120,49 @@ partitions already written. Pass the **dataset directory** (`{output_dir}/{unive
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code). set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
### `derived` — daily custom/derived data
Derived data is daily-only v1 research data keyed by `symbol_id,date`, with one
or more numeric value columns. It can come from user CSV/parquet files or Python
plugins, and is written as a single parquet file at `derived/{name}.pq`.
The validator normalizes `date` to the trading day, requires unique
`symbol_id,date` keys, rejects duplicate columns, and rejects non-numeric value
columns. Alpha computation consumes derived data through the existing
`--feature-path` flag.
```bash
# Validate a user file without writing output.
uv run python cli.py derived validate --input-path vendor_factor.csv
# Ingest CSV/parquet into the canonical derived/ layout.
uv run python cli.py derived ingest \
--input-path vendor_factor.csv \
--derived-name vendor_factor
# List built-in and external derived-data plugin types.
uv run python cli.py derived list
uv run python cli.py derived list --derived-module path/to/my_derived.py
# Compute a derived file from daily and/or minute inputs.
uv run python cli.py derived compute \
--minute-path data/minute_bars/sh600000 \
--daily-path data/daily_bars/sh600000 \
--derived-type minute_daily_summary \
--derived-name minute_summary
# Join derived columns into a feature-aware alpha.
uv run python cli.py alpha compute \
--data-path data/daily_bars/sh600000 \
--feature-path derived/minute_summary.pq \
--alpha-type my_feature_aware_alpha \
--alpha-name my_run
```
For compatibility, `feature list` and `feature compute` remain available and
delegate to the same derived-data registry. Existing `features/*.pq` files are
still valid `--feature-path` inputs when they satisfy the daily numeric contract.
### `alpha list` — show registered alpha types ### `alpha list` — show registered alpha types
```bash ```bash
@@ -137,6 +180,7 @@ uv run python cli.py alpha list --alpha-module path/to/my_alpha.py # include a
| `--output-dir` | `alphas` | Output directory | | `--output-dir` | `alphas` | Output directory |
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) | | `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) | | `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
| `--feature-path` | — | Daily derived/feature parquet file or dataset to left-join on `symbol_id,date`; repeatable |
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file | | `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
| `--param` | — | Extra constructor param as `name=value`; repeatable | | `--param` | — | Extra constructor param as `name=value`; repeatable |
@@ -370,6 +414,8 @@ between phases (data is stored long/tidy):
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change, OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are `tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
baostock valuation ratios.) baostock valuation ratios.)
- **derived** (`DERIVED_KEY_COLUMNS` + values): required keys `symbol_id, date`;
value columns are user/plugin-defined and must be numeric in v1.
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight` - **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight` - **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price` - **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
@@ -387,8 +433,11 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
- `cli.py` — entry point wiring the file-based phases together - `cli.py` — entry point wiring the file-based phases together
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq` - `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
- `pipeline/derived/` — daily derived-data ingestion, validation, plugin registry,
and built-in derived computations → `derived/*.pq`
- `pipeline/alpha/``base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader), - `pipeline/alpha/``base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`) `library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
- `pipeline/features/` — compatibility wrappers for the derived-data registry
- `pipeline/combo/` — alpha combination → `combos/*.pq` - `pipeline/combo/` — alpha combination → `combos/*.pq`
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints, - `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
reference next-open simulator, and research metrics reference next-open simulator, and research metrics
@@ -410,6 +459,9 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
- [x] **Reference execution simulation** — next-open fills over constructed - [x] **Reference execution simulation** — next-open fills over constructed
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost, `position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
and slippage controls. and slippage controls.
- [x] **Derived/custom daily data ("Level 2")** — ingest user CSV/parquet files
or compute plugin outputs as validated numeric daily datasets under
`derived/{name}.pq`; alpha joins continue through `--feature-path`.
- [ ] **Optional Backtrader adapter** — Backtrader is available as the - [ ] **Optional Backtrader adapter** — Backtrader is available as the
`backtrader` extra for possible future event-driven/broker-style experiments, `backtrader` extra for possible future event-driven/broker-style experiments,
but it is not part of the current canonical portfolio workflow. but it is not part of the current canonical portfolio workflow.
@@ -419,3 +471,29 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
and intraday VWAP. These need a tick / L1L2 quote feed (typically a paid or and intraday VWAP. These need a tick / L1L2 quote feed (typically a paid or
brokerage data tier); the free daily sources here only expose daily bars, so brokerage data tier); the free daily sources here only expose daily bars, so
this is a separate data phase rather than extra columns on the daily schema. this is a separate data phase rather than extra columns on the daily schema.
### Additional TODOs
The following items are intended extensions beyond the current daily
alpha-to-portfolio pipeline:
- **Long-only portfolio mode** — add a construction option that converts
alpha/combo weights into a long-only book while preserving existing lot,
price, suspension, and volume-cap handling.
- **Index-short hedging mode** — support portfolios that hold long A-share
names while shorting an index or index proxy for market exposure control.
- **Expanded universe presets** — add explicit universe aliases for CSI 300,
CSI 500, CSI 1000, and CSI 1800, while keeping file-based and comma-separated
custom universes available.
- **Categorical derived data** — extend the numeric-only derived-data v1 contract
to support categorical inputs such as industry classifications. In this
project, "Level 2" means customized second-level research data produced by
users or plugins; it does not necessarily mean exchange order-book/L2 quote
feeds.
- **Minute bar data** — continue extending the raw minute-bar and feature
workflow. The initial Baostock 5-minute download and daily feature plugin path
exist; intraday execution and replacing canonical daily bars remain out of
scope unless explicitly added later.
- **Industry data** — add industry classification inputs for filtering,
grouping, exposure reporting, neutralization, or industry-aware portfolio
construction.
+8
View File
@@ -3,7 +3,9 @@
Phases: Phases:
data — Download daily bars to parquet data — Download daily bars to parquet
derived — Ingest or compute daily derived data
alpha — Compute alpha weights from data alpha — Compute alpha weights from data
feature — Compute daily features from minute bars
combo — Combine alphas into a single weight combo — Combine alphas into a single weight
portfolio — Build tradable positions and simulate execution portfolio — Build tradable positions and simulate execution
""" """
@@ -13,9 +15,12 @@ import logging
import click import click
from pipeline.data.cli import data from pipeline.data.cli import data
from pipeline.derived.cli import derived
from pipeline.alpha.cli import alpha from pipeline.alpha.cli import alpha
from pipeline.features.cli import feature
from pipeline.combo.cli import combo from pipeline.combo.cli import combo
from pipeline.portfolio.cli import portfolio from pipeline.portfolio.cli import portfolio
from plugins.joinquant.cli import joinquant
from tools.pqcat import pqcat from tools.pqcat import pqcat
from tools.alphaview import alphaview from tools.alphaview import alphaview
@@ -39,9 +44,12 @@ def cli(log_level):
cli.add_command(data) cli.add_command(data)
cli.add_command(derived)
cli.add_command(alpha) cli.add_command(alpha)
cli.add_command(feature)
cli.add_command(combo) cli.add_command(combo)
cli.add_command(portfolio) cli.add_command(portfolio)
cli.add_command(joinquant)
cli.add_command(pqcat) cli.add_command(pqcat)
cli.add_command(alphaview) cli.add_command(alphaview)
+158
View File
@@ -31,11 +31,68 @@ _BATCH_COLUMNS = [
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM", "peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
] ]
# Raw Baostock minute bars. The ``time`` field is usually compact
# YYYYMMDDHHMMSSmmm; parsing below also tolerates HH:MM:SS strings in tests.
_MINUTE_FIELDS = "date,time,code,open,high,low,close,volume,amount,adjustflag"
_MINUTE_NUMERIC = ["open", "high", "low", "close", "volume", "amount"]
_MINUTE_COLUMNS = [
"symbol", "datetime", "date", "time", "frequency",
"open", "high", "low", "close", "volume", "amount", "vwap", "adjustflag",
]
_MINUTE_FREQUENCIES = {"5", "15", "30", "60"}
class _SessionLost(Exception): class _SessionLost(Exception):
"""baostock reported the session was dropped (``用户未登录``).""" """baostock reported the session was dropped (``用户未登录``)."""
def _normalize_minute_frequency(frequency: str | int) -> tuple[str, str]:
"""Return Baostock frequency and partition label for a minute interval."""
raw = str(frequency).strip().lower()
if raw.endswith("m"):
raw = raw[:-1]
if raw not in _MINUTE_FREQUENCIES:
raise ValueError(
f"Unsupported minute frequency '{frequency}'. "
f"Expected one of {sorted(_MINUTE_FREQUENCIES)} minutes."
)
return raw, f"{raw}m"
def _parse_minute_datetime(date: pd.Series, time: pd.Series) -> pd.Series:
"""Parse Baostock minute timestamps into pandas datetimes."""
date_dt = pd.to_datetime(date, errors="coerce")
date_compact = date_dt.dt.strftime("%Y%m%d")
time_text = time.astype(str).str.strip()
time_digits = time_text.str.replace(r"\D", "", regex=True)
full_digits = time_digits.str.slice(0, 14)
from_full = pd.to_datetime(full_digits, format="%Y%m%d%H%M%S", errors="coerce")
from_short = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
short_time = time_digits.str.len().between(1, 6)
if short_time.any():
short_digits = (
time_digits.loc[short_time]
.str.pad(6, side="right", fillchar="0")
.str.slice(0, 6)
)
from_short.loc[short_time] = pd.to_datetime(
date_compact.loc[short_time] + short_digits,
format="%Y%m%d%H%M%S",
errors="coerce",
)
from_text = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
text_time = time_text.str.contains(":", regex=False)
if text_time.any():
from_text.loc[text_time] = pd.to_datetime(
date.astype(str).loc[text_time] + " " + time_text.loc[text_time],
errors="coerce",
)
return from_full.fillna(from_short).fillna(from_text)
def _download_akshare(symbol: str, start: str, end: str, adjust: str = "qfq") -> Optional[pd.DataFrame]: def _download_akshare(symbol: str, start: str, end: str, adjust: str = "qfq") -> Optional[pd.DataFrame]:
"""Download daily bars from akshare. Returns DataFrame with OHLCV columns.""" """Download daily bars from akshare. Returns DataFrame with OHLCV columns."""
try: try:
@@ -239,3 +296,104 @@ def download_daily_batch(
except Exception: except Exception:
pass pass
def download_minute_batch(
symbols: Iterable[str],
start: str,
end: str,
frequency: str | int = 5,
relogin_every: int = 200,
) -> Iterator[Tuple[str, Optional[pd.DataFrame]]]:
"""Download raw Baostock minute bars for many symbols.
Minute bars are intentionally unadjusted (`adjustflag='3'`) because the
output is raw intraday market data for downstream feature aggregation, not a
tradable daily price series.
Args:
symbols: Internal-form symbols (``sh600000`` / ``sz000001``).
start, end: ``YYYY-MM-DD`` bounds.
frequency: Baostock minute frequency. ``5``/``"5"``/``"5m"`` all mean
5-minute bars.
relogin_every: Proactively refresh the baostock session every N symbols.
Yields:
``(symbol, df)`` where ``df`` has raw minute bars or ``None`` when no
data is available.
"""
query_frequency, frequency_label = _normalize_minute_frequency(frequency)
adjustflag = _BAOSTOCK_ADJUST["none"]
def _relogin() -> None:
try:
bs.logout()
except Exception:
pass
bs.login()
def _fetch(symbol: str) -> Optional[pd.DataFrame]:
"""One Baostock minute query; returns df, None, or raises _SessionLost."""
code = f"{symbol[:2]}.{symbol[2:]}"
rs = bs.query_history_k_data_plus(
code=code,
fields=_MINUTE_FIELDS,
start_date=start,
end_date=end,
frequency=query_frequency,
adjustflag=adjustflag,
)
if rs.error_code != "0":
if "未登录" in (rs.error_msg or ""):
raise _SessionLost(rs.error_msg)
logger.warning("baostock minute error for %s: %s", symbol, rs.error_msg)
return None
rows = []
while rs.next():
rows.append(rs.get_row_data())
if not rows:
return None
df = pd.DataFrame(rows, columns=_MINUTE_FIELDS.split(","))
df[_MINUTE_NUMERIC] = df[_MINUTE_NUMERIC].apply(pd.to_numeric, errors="coerce")
df["datetime"] = _parse_minute_datetime(df["date"], df["time"])
bad_timestamps = df["datetime"].isna()
if bad_timestamps.any():
raise ValueError(
f"Could not parse {int(bad_timestamps.sum())} minute timestamp(s)"
)
df["date"] = pd.to_datetime(df["date"], errors="coerce")
df["time"] = df["datetime"].dt.strftime("%H:%M:%S")
df["frequency"] = frequency_label
df["vwap"] = (df["amount"] / df["volume"]).where(df["volume"] > 0)
df["symbol"] = symbol
return df[_MINUTE_COLUMNS].sort_values("datetime").reset_index(drop=True)
bs.login()
try:
for i, symbol in enumerate(symbols):
if i and relogin_every and i % relogin_every == 0:
_relogin()
df: Optional[pd.DataFrame] = None
for attempt in (1, 2):
try:
df = _fetch(symbol)
break
except _SessionLost:
if attempt == 1:
_relogin()
continue
logger.warning("baostock minute session lost for %s after relogin", symbol)
except Exception as e:
logger.warning("baostock minute download failed for %s: %s", symbol, e)
break
if df is not None and not df.empty:
yield symbol, df
else:
yield symbol, None
finally:
try:
bs.logout()
except Exception:
pass
+352
View File
@@ -0,0 +1,352 @@
# JoinQuant Comparison Plugin
## Why a Plugin
JoinQuant is an external execution and simulation reference. Keeping this code
under `plugins/joinquant/` prevents vendor-specific assumptions from entering
`pipeline/portfolio/`, where the internal reference simulator remains the
canonical implementation.
## What It Validates
The comparison is for system correctness:
- date alignment
- internal to JoinQuant symbol mapping
- target position generation
- once-per-day open execution timing
- lot rounding and filled shares
- position carry
- trading cost
- PnL accounting
- blocked trades from suspension, limit-up, and limit-down conditions
## What It Does Not Validate
It does not validate alpha quality, IC, IR, forecast skill, or whether the
strategy is economically useful. Differences can be expected when JoinQuant
uses different fee, slippage, cash, corporate-action, or internal rounding
rules.
## Historical Backtest Workflow
```bash
# 1. Build internal portfolio targets.
uv run python cli.py portfolio build ...
# 2. Export JoinQuant-compatible frozen targets.
uv run python cli.py joinquant export-targets \
--positions-path portfolio/run1.pq \
--portfolio-name run1 \
--mode target_shares \
--execution-calendar-path data/daily_bars/<universe> \
--out-dir plugins_output/joinquant/targets
# 3. Generate and copy the wrapper strategy and target files into JoinQuant.
uv run python cli.py joinquant write-wrapper \
--portfolio-name run1 \
--mode target_shares \
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
# 4. Run the JoinQuant backtest or simulated trading job.
# 5. Export JoinQuant fills, positions, and daily PnL to CSV.
# 6. Ingest JoinQuant output.
uv run python cli.py joinquant ingest \
--portfolio-name run1 \
--fills-csv path/to/jq_fills.csv \
--positions-csv path/to/jq_positions.csv \
--pnl-csv path/to/jq_pnl.csv
# 7. Reconcile.
uv run python cli.py joinquant reconcile \
--portfolio-name run1 \
--targets-dir plugins_output/joinquant/targets/run1 \
--our-fills-path fills/run1.pq \
--our-positions-path portfolio/run1.pq \
--our-pnl-path pnl/run1.pq \
--jq-fills-path plugins_output/joinquant/ingested/run1/fills.pq \
--jq-positions-path plugins_output/joinquant/ingested/run1/positions.pq \
--jq-pnl-path plugins_output/joinquant/ingested/run1/pnl.pq
```
## Forward-Testing Workflow
After the T-1 close and after the data update:
```bash
uv run python cli.py portfolio build ...
uv run python cli.py joinquant export-targets \
--positions-path portfolio/run1.pq \
--portfolio-name run1 \
--mode target_shares \
--execution-calendar-path data/daily_bars/<universe> \
--start-date T \
--end-date T
```
Before the T open, upload or expose the frozen target file to JoinQuant. During
the T open, the JoinQuant wrapper reads that file and submits orders, while the
internal simulator should run against the same frozen target. After T close or
after JoinQuant results are available, ingest the JoinQuant CSV files and run
`joinquant reconcile`.
Forward target files must be frozen before execution. Do not regenerate a
target file after observing open or close data for the same trading date. The
exporter writes a snapshot JSON with a SHA-256 hash for this reason and refuses
to overwrite existing target/snapshot files unless `--force` is passed.
For comparisons against the internal simulator, pass
`--execution-calendar-path` to `joinquant export-targets`. The positions file is
dated by construction/signal date, while the simulator executes at the next
available open. The calendar option shifts exported target files to that next
trading date, so JoinQuant reads the same target on the same execution session.
For JoinQuant 模拟盘, the browser automation has two operational phases:
```bash
# Before T open: upload frozen target(s), save strategy, and start/restart 模拟盘.
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--strategy-url "https://www.joinquant.com/<your 模拟盘 page>" \
--flow sim-trade
uv run python cli.py joinquant run-browser-sim \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
--headed
# After T close: download/export JoinQuant fills, positions, and pnl, then
# ingest/reconcile. The same run-browser-sim command can do this if the config
# includes download actions, otherwise use the ingest/reconcile commands.
```
The default simulated-trading template includes selectors for saving the
strategy and clicking simulated-trading controls such as `模拟盘`, `模拟交易`,
`启动`, and `重启`. These selectors are intentionally configurable because the
JoinQuant web UI can differ by account and page version.
## Target-Shares Mode
`target_shares` is the default and preferred correctness mode. The exported
`target_shares` field comes from the internal `position_shares` column produced
by `portfolio build`, because the internal simulator executes that discretized
integer book. The generated wrapper calls:
```python
order_target(jq_symbol, target_shares)
```
This mode makes filled shares, position carry, and blocked trades easiest to
compare.
## Target-Value Mode
`target_value` mode exports `target_value` and `target_weight` from the
portfolio file. The generated wrapper calls:
```python
order_target_value(jq_symbol, target_value)
```
This can be useful for portfolio-level comparisons, but JoinQuant may apply its
own rounding, cash, and lot rules. Differences are often classified as
`JOINQUANT_INTERNAL_ROUNDING`, `LOT_ROUNDING`, or `CASH_CONSTRAINT` depending
on the observed output.
## Symbol Mapping
Internal symbols are converted as follows:
```text
sh600000 -> 600000.XSHG
sh688001 -> 688001.XSHG
sz000001 -> 000001.XSHE
sz300001 -> 300001.XSHE
```
Reverse mapping is also supported. Invalid exchanges or unsupported A-share
prefixes raise `ValueError` instead of silently guessing.
## Wrapper Strategy Usage
Generate a configured wrapper:
```bash
uv run python cli.py joinquant write-wrapper \
--portfolio-name run1 \
--mode target_shares \
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
```
Copy the generated file and daily CSV target files into JoinQuant. The default
loader uses JoinQuant `read_file`, which works for uploaded files. If your
JoinQuant runtime allows HTTP or another storage backend, replace only
`_read_target_file()` in the generated strategy.
The wrapper is long-only by default:
```python
ALLOW_SHORT = False
```
Negative targets are clipped to zero and logged. Use `--allow-short` only if
the target JoinQuant account supports the required shorting mechanics.
## Ingesting JoinQuant Outputs
The ingest command accepts permissive CSV column names and writes strict plugin
schemas:
```text
plugins_output/joinquant/ingested/{portfolio_name}/fills.pq
plugins_output/joinquant/ingested/{portfolio_name}/positions.pq
plugins_output/joinquant/ingested/{portfolio_name}/pnl.pq
```
Missing cost fields default to zero. Missing blocked status defaults to zero.
Symbols and dates are normalized.
## Reading Reconciliation Reports
The reconcile command writes:
```text
plugins_output/joinquant/reconcile/{portfolio_name}/daily_reconcile.pq
plugins_output/joinquant/reconcile/{portfolio_name}/summary.csv
plugins_output/joinquant/reconcile/{portfolio_name}/summary.md
```
`daily_reconcile.pq` is per-symbol and includes target shares, internal filled
shares, JoinQuant filled shares, realized positions, trade prices, costs, PnL,
and a `diff_reason`. `summary.csv` is the daily portfolio-level view for gross
exposure, net exposure, cash, total value, PnL, cumulative PnL, turnover, and
cost.
Difference reasons include:
```text
MATCH SYMBOL_MAPPING PRICE_MISMATCH LOT_ROUNDING SUSPENSION LIMIT_UP_BLOCK
LIMIT_DOWN_BLOCK VOLUME_OR_LIQUIDITY COST_MODEL CASH_CONSTRAINT
SHORT_NOT_SUPPORTED CORPORATE_ACTION JOINQUANT_INTERNAL_ROUNDING
MISSING_IN_OUR_SYSTEM MISSING_IN_JOINQUANT UNKNOWN
```
Default tolerances are exact share matching, `1e-4` relative trade-price
tolerance, and value tolerance `max(1 yuan, 1e-6 * booksize)`. PnL tolerance is
configurable with `--pnl-tolerance`.
## Minimal Example
Create a 5-stock equal-weight or fixed-share test portfolio:
```text
sh600000, sz000001, sh600519, sz002594, sz300750
```
Build positions for a small date range, export `target_shares`, upload the CSV
files and wrapper to JoinQuant, run the JoinQuant backtest, export fills,
positions, and PnL, then run ingest and reconcile. Start with one or two days
before expanding the sample.
For the first one-stock long-only smoke test, the local side can be prepared in
one command:
```bash
uv run python cli.py joinquant prepare-smoke \
--out-dir /tmp/chinese-equity-quant-realdata
```
The command downloads a tiny public daily-bar sample, builds a fixed-share
`sh600000` long-only position file, simulates it internally, exports aligned
JoinQuant target files, writes a configured wrapper strategy, and creates
`joinquant_smoke_manifest.json` with all output paths.
## Browser Backtest Automation
JoinQuant's public `jqdatasdk` is a data SDK. It supports authenticated data
calls such as `auth(username, password)` and `get_price(...)`, but cloud
strategy upload, backtest execution, and result export are web-application
workflows. The plugin therefore automates those remote steps through Playwright
with a saved browser session.
Install the optional browser runner in the uv environment:
```bash
uv sync --extra joinquant-browser
uv run playwright install chromium
```
Save a reusable login state. This opens a browser; log in normally, including
any CAPTCHA or 2FA, then press Enter in the terminal to save state:
```bash
uv run python cli.py joinquant browser-login \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
```
Create a selector/action config for a historical backtest:
```bash
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--strategy-url "https://www.joinquant.com/<your strategy page>" \
--flow backtest
```
If selectors need tuning, capture the logged-in page:
```bash
uv run python cli.py joinquant browser-snapshot \
--url "https://www.joinquant.com/<your strategy page>" \
--out-dir /tmp/chinese-equity-quant-realdata/browser_snapshot
```
Then run the remote backtest automation:
```bash
uv run python cli.py joinquant run-browser-backtest \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
--headed
```
The config is declarative: actions can navigate, paste the generated wrapper,
upload all target CSV files, fill dates, click run, wait for completion,
download result CSVs, and take screenshots. When the configured downloads
produce fills, positions, and PnL CSVs, the runner automatically calls
`joinquant ingest` and `joinquant reconcile`.
For forward testing / 模拟盘, create the config with `--flow sim-trade` and run:
```bash
uv run python cli.py joinquant run-browser-sim \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json \
--headed
```
Do not store raw JoinQuant passwords in this repository. The browser state file
is created with `0600` permissions and should live under `~/.config`, outside
the repo.
## Recommended First Sanity Checks
1. One liquid stock with a fixed target share count.
2. A 10-stock equal-weight long-only portfolio.
3. A forced suspension, limit-up, and limit-down sample.
4. A short target in long-only mode to confirm `SHORT_NOT_SUPPORTED`.
5. A 5-day reversal portfolio after the mechanical checks pass.
## Known Limitations
- JoinQuant internal execution details may differ from the reference simulator.
- External file loading depends on the JoinQuant environment.
- Short selling may not be supported.
- Fee, tax, slippage, and minimum-fee models may differ.
- Corporate actions may need special handling and should not be hidden.
- The internal simulator does not currently emit execution price in
`FILL_COLUMNS`; price reconciliation uses explicit price columns if supplied.
+167
View File
@@ -0,0 +1,167 @@
# JoinQuant Cost Model Findings
Generated: 2026-07-06
This report summarizes the JoinQuant trading-cost behavior observed from the
browser-automated real-data backtests and compares it with the current internal
simulator model. The JoinQuant cost formula below is inferred from rendered
transaction tables and strategy logs for this account. Treat it as an observed
platform default, not as a guaranteed external contract.
## Runs Used
### Longer Comparison Run
- Portfolio: `jq_long_one_stock_long`
- Window: `2024-01-11` to `2024-02-29`
- Booksize: `1,000,000 CNY`
- Instrument: `600000.XSHG`
- Target: buy and hold `1,000` shares
- JoinQuant rendered result: completed
- Local total PnL: `546.22 CNY`
- JoinQuant total PnL from positions tab: `575.00 CNY`
- Difference: `28.78 CNY`, or `0.002878` percentage points on the book
The first JoinQuant transaction was:
| Date | Side | Shares | Price | Turnover | Fee |
|---|---:|---:|---:|---:|---:|
| `2024-01-11` | Buy | `1,000` | `6.57` | `6,570.00` | `5.00` |
That trade hit a minimum fee. The local simulator charged `6.1415 CNY` because
the smoke runner used a flat `10 bps` cash cost on adjusted-price turnover.
### Cost Probe Run
- Portfolio: `jq_cost_probe_buy_sell`
- Window: `2024-01-11` to `2024-01-12`
- Booksize: `1,000,000 CNY`
- Instrument: `600000.XSHG`
- Targets:
- `2024-01-11`: buy `100,000` shares
- `2024-01-12`: sell to `0` shares
Observed JoinQuant transactions:
| Date | Side | Shares | Price | Turnover | Fee | Implied Fee |
|---|---:|---:|---:|---:|---:|---:|
| `2024-01-11` | Buy | `100,000` | `6.57` | `657,000.00` | `197.10` | `3.0000 bps` |
| `2024-01-12` | Sell | `-100,000` | `6.51` | `651,000.00` | `846.30` | `13.0000 bps` |
The transaction-table numbers match this formula exactly:
```text
buy fee = max(5 CNY, turnover * 0.0003)
sell fee = max(5 CNY, turnover * 0.0003) + turnover * 0.001
```
In basis points:
- Buy commission: `3 bps`
- Sell commission: `3 bps`
- Sell stamp tax: `10 bps`
- Minimum commission: `5 CNY`
No separate transfer fee was visible in this probe. If a separate transfer fee
was present as an additional charge, the observed fees would not match the
formula above exactly. It may still be folded into JoinQuant's displayed
commission field, so this finding should be read as "not separately observable"
rather than "impossible".
No slippage was visible. The wrapper submitted open-time market orders with
`run_daily(..., time="open")`, and JoinQuant filled them at the displayed open
prices.
## Evidence From Strategy Logs
The JoinQuant order log for the cost probe showed:
```text
2024-01-11 ... trade price: 6.57, amount:100000, commission: 197.1
2024-01-12 ... trade price: 6.51, amount:100000, commission: 846.3
```
The normal transaction tab showed the same fee values. However, the generated
wrapper's `JOINQUANT_FILL` log records had `trade_cost: 0.0`, even for those
same fills. That means `get_trades()` did not expose the usable commission
value through the field the wrapper currently reads.
For reconciliation, use the transaction table or JoinQuant order logs for fee
details. Do not rely on the wrapper's current `JOINQUANT_FILL.trade_cost`.
## Difference From The Internal Simulator
The current internal simulator cost model is
`SimpleProportionalCostModel` in `pipeline/portfolio/costs.py`:
```text
trade_cost = abs(traded_shares * execution_price)
* (cost_bps + slippage_bps) / 10000
```
The smoke runner used:
- `cost_bps = 5`
- `slippage_bps = 5`
- combined one-way cash cost: `10 bps`
Important differences:
- The internal simulator uses the same rate for buys and sells.
- It has no minimum commission.
- It has no sell-only stamp tax.
- Slippage is modeled as an extra cash cost.
- JoinQuant did not show slippage in the observed open-time fills.
- The local smoke download used `adjust="qfq"`, while the JoinQuant wrapper set
`set_option("use_real_price", True)`. That price-scale mismatch also affects
PnL and cost comparisons.
## Practical Implications
For a buy-only smoke test, JoinQuant may charge less than the local model when
the trade is large enough for `3 bps` to apply, but it may charge more on small
orders because of the `5 CNY` minimum.
For any test with sells, JoinQuant's default sell fee is materially higher than
the current local flat model because of the inferred `10 bps` stamp tax.
The earlier 30-day buy-and-hold discrepancy was small because only one buy was
executed. A rebalancing strategy with many sells will show a larger cost-model
difference unless the local simulator is configured to match JoinQuant.
## Recommended Follow-Ups
1. Add a JoinQuant-style cost model to the internal simulator:
```text
commission = max(min_commission, turnover * commission_bps / 10000)
stamp_tax = turnover * sell_stamp_tax_bps / 10000 for sells only
trade_cost = commission + stamp_tax
```
2. Add a CLI option or preset for `portfolio simulate`, for example
`--cost-model joinquant-stock`.
3. Update JoinQuant reconciliation to parse fee values from the transaction
table or order logs when CSV exports are unavailable.
4. Run a second local-vs-JoinQuant comparison with:
- raw or real-price local bars, not adjusted-price bars
- JoinQuant-style costs
- slippage disabled locally
That test should isolate remaining differences to data alignment, price source,
rounding, and JoinQuant internal execution behavior.
## Local Artifacts
The temporary artifacts from the investigation are:
- `/tmp/chinese-equity-quant-jq-long/comparison_report.md`
- `/tmp/chinese-equity-quant-jq-long/parsed_joinquant/daily_pnl_compare_from_positions_tab.csv`
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_analysis_report.md`
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_analysis_summary.json`
- `/tmp/chinese-equity-quant-jq-cost-probe/jq_cost_probe_transactions_parsed.csv`
- `/tmp/chinese-equity-quant-jq-cost-probe/detail_tabs/transactions.txt`
+70
View File
@@ -0,0 +1,70 @@
# Minute Bar Data Notes
The minute-bar path downloads raw Baostock intraday bars and stores them as a
Hive-partitioned dataset:
```bash
uv run python cli.py data download-minute \
--universe sh600000 \
--start-date 2024-01-02 --end-date 2024-01-05 \
--frequency 5
```
The default layout is:
```text
data/minute_bars/{universe}/frequency=5m/month=YYYY-MM/*.pq
```
Derived-data plugins can aggregate those bars to daily `symbol_id,date` numeric
files, for example:
```bash
uv run python cli.py derived compute \
--minute-path data/minute_bars/sh600000 \
--daily-path data/daily_bars/sh600000 \
--derived-type minute_daily_summary \
--derived-name minute_summary
```
The legacy `feature compute` command delegates to the same derived-data
registry and remains available for existing scripts.
## Daily vs Minute Reconciliation
Baostock's daily raw bars and 5-minute raw bars are close, but they should not
be treated as perfectly reconstructible from each other.
When checking consistency, compare daily raw bars (`data download --adjust none`)
against minute bars on the same raw price scale. The minute aggregation should
use:
- `open`: first minute open
- `high`: max minute high
- `low`: min minute low
- `close`: last minute close
- `volume`: sum minute volume
- `amount`: sum minute amount
- `vwap`: `sum(amount) / sum(volume)`
In a sanity check for `sh600000` from `2024-01-02` through `2024-01-05`, Baostock
returned 4 daily rows and 192 5-minute bars, exactly 48 bars per day. Open, low,
and close matched daily exactly on all 4 days. High matched on 3 of 4 days; on
`2024-01-04`, the daily high was `6.67` while the max 5-minute high was `6.66`.
Minute-summed volume and amount were higher than daily by roughly `0.16%` to
`1.23%`. VWAP remained very close, with max relative difference around
`0.0043%`.
This appears to be a source-level Baostock reconciliation caveat, not a parser
or ordering issue: the minute bars covered the regular `09:35:00` through
`15:00:00` range and sorted correctly by timestamp.
Practical guidance:
- Use tolerance-based daily-vs-minute checks; do not require exact equality for
high, volume, or amount.
- Expect open/close alignment to be a stronger sanity check than exact volume
reconstruction.
- Use minute-derived values as separate daily features, not as replacements for
the canonical daily bar dataset unless a strategy explicitly wants that
source convention.
+19 -5
View File
@@ -5,7 +5,7 @@ position weights. Subclasses implement :meth:`signal` — the raw, unnormalized
score. The base class turns a signal into cross-sectionally z-scored weights score. The base class turns a signal into cross-sectionally z-scored weights
via :meth:`to_weights` (override it for a different normalization). via :meth:`to_weights` (override it for a different normalization).
""" """
from abc import ABC, abstractmethod from abc import ABC
import numpy as np import numpy as np
import pandas as pd import pandas as pd
@@ -15,15 +15,14 @@ class BaseAlpha(ABC):
"""A position-weight alpha over a cross-section of stocks. """A position-weight alpha over a cross-section of stocks.
Concrete subclasses must set a unique class-level :attr:`name` (the registry Concrete subclasses must set a unique class-level :attr:`name` (the registry
key) and implement :meth:`signal`. Construct subclasses with their own typed key) and implement either :meth:`signal` or :meth:`signal_from_data`.
parameters (e.g. ``lookback``); the factory passes only the parameters a Construct subclasses with their own typed parameters (e.g. ``lookback``);
given ``__init__`` accepts. the factory passes only the parameters a given ``__init__`` accepts.
""" """
#: Unique registry key. Every concrete alpha must set this to a non-empty str. #: Unique registry key. Every concrete alpha must set this to a non-empty str.
name: str = "" name: str = ""
@abstractmethod
def signal(self, close: pd.DataFrame) -> pd.DataFrame: def signal(self, close: pd.DataFrame) -> pd.DataFrame:
"""Compute the raw signal. """Compute the raw signal.
@@ -34,6 +33,21 @@ class BaseAlpha(ABC):
A wide DataFrame aligned to ``close`` where higher values indicate a A wide DataFrame aligned to ``close`` where higher values indicate a
stronger long. Use NaN where the signal is undefined. stronger long. Use NaN where the signal is undefined.
""" """
raise NotImplementedError(
f"{type(self).__name__} must implement signal() or signal_from_data()"
)
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
"""Compute the raw signal from long daily data plus wide closes.
Feature-aware alphas can override this to pivot joined feature columns
from ``data``. The default preserves the existing close-only alpha API.
"""
return self.signal(close)
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame: def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
"""Cross-sectionally z-score a signal into signed position weights. """Cross-sectionally z-score a signal into signed position weights.
+10 -4
View File
@@ -56,6 +56,10 @@ def list_(alpha_modules):
@click.option("--output-dir", default="alphas", help="Directory to save alpha parquet") @click.option("--output-dir", default="alphas", help="Directory to save alpha parquet")
@click.option("--lookback", default=5, type=int, help="Lookback days") @click.option("--lookback", default=5, type=int, help="Lookback days")
@click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)") @click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)")
@click.option(
"--feature-path", "feature_paths", multiple=True,
help="Daily derived/feature parquet file or dataset to left-join on symbol_id,date (repeatable)",
)
@click.option( @click.option(
"--alpha-module", "alpha_modules", multiple=True, "--alpha-module", "alpha_modules", multiple=True,
help="External module(s) to import so their alphas register (dotted path or .py file)", help="External module(s) to import so their alphas register (dotted path or .py file)",
@@ -74,7 +78,7 @@ def list_(alpha_modules):
help="Most-liquid names kept per date when --liquid-universe is set", help="Most-liquid names kept per date when --liquid-universe is set",
) )
def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window, def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
alpha_modules, extra_params, liquid_universe, universe_top_n): feature_paths, alpha_modules, extra_params, liquid_universe, universe_top_n):
"""Compute one alpha from raw data and save as parquet.""" """Compute one alpha from raw data and save as parquet."""
for spec in alpha_modules: for spec in alpha_modules:
load_alpha_module(spec) load_alpha_module(spec)
@@ -100,6 +104,7 @@ def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
alpha_name=alpha_name, alpha_name=alpha_name,
alpha_type=alpha_type, alpha_type=alpha_type,
universe=universe, universe=universe,
feature_paths=feature_paths,
**params, **params,
) )
@@ -155,7 +160,8 @@ def reversal_vol(data_path, output_dir, lookback, vol_window):
@alpha.command("eval") @alpha.command("eval")
@click.option("--alpha-path", required=True, help="Path to alpha parquet file") @click.option("--alpha-path", required=True, help="Path to alpha parquet file")
@click.option("--data-path", required=True, help="Path to data parquet (for price data)") @click.option("--data-path", required=True, help="Path to data parquet (for price data)")
def eval_(alpha_path, data_path): @click.option("--report-dir", default="reports", help="Directory to save JSON report")
def eval_(alpha_path, data_path, report_dir):
"""Evaluate an alpha's performance (return, Sharpe, turnover). """Evaluate an alpha's performance (return, Sharpe, turnover).
Alphas are interpreted as position WEIGHTS, not return predictors. Alphas are interpreted as position WEIGHTS, not return predictors.
@@ -178,9 +184,9 @@ def eval_(alpha_path, data_path):
click.echo("=" * 50) click.echo("=" * 50)
# Also dump JSON # Also dump JSON
os.makedirs("reports", exist_ok=True) os.makedirs(report_dir, exist_ok=True)
alpha_name = alpha_df["alpha_name"].iloc[0] alpha_name = alpha_df["alpha_name"].iloc[0]
json_path = f"reports/{alpha_name}_eval.json" json_path = os.path.join(report_dir, f"{alpha_name}_eval.json")
with open(json_path, "w") as f: with open(json_path, "w") as f:
json.dump(metrics, f, indent=2) json.dump(metrics, f, indent=2)
click.echo(f"\nReport saved: {json_path}") click.echo(f"\nReport saved: {json_path}")
+55 -2
View File
@@ -7,12 +7,19 @@ through :mod:`pipeline.alpha.registry`.
""" """
import logging import logging
from pathlib import Path
from typing import Iterable
import numpy as np import numpy as np
import pandas as pd import pandas as pd
from pipeline.alpha.registry import get_alpha from pipeline.alpha.registry import get_alpha
from pipeline.common.schema import ALPHA_COLUMNS from pipeline.common.schema import ALPHA_COLUMNS
from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
read_derived_frames,
validate_derived_frame,
)
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -33,6 +40,38 @@ def _pivot_open(df: pd.DataFrame) -> pd.DataFrame:
return pivot.sort_index() return pivot.sort_index()
def join_feature_frames(
data: pd.DataFrame,
feature_frames: Iterable[pd.DataFrame],
) -> pd.DataFrame:
"""Left-join validated daily derived/feature frames onto long daily data."""
out = data.copy()
out["date"] = pd.to_datetime(out["date"])
existing = set(out.columns)
joined_cols: list[str] = []
for frame in feature_frames:
features = validate_derived_frame(frame)
feature_cols = [col for col in features.columns if col not in DERIVED_KEY_COLUMNS]
overlap = sorted(existing.intersection(feature_cols))
if overlap:
raise ValueError(
f"Feature columns conflict with existing daily data columns: {overlap}"
)
out = out.merge(
features,
on=DERIVED_KEY_COLUMNS,
how="left",
validate="many_to_one",
)
existing.update(feature_cols)
joined_cols.extend(feature_cols)
if joined_cols:
logger.info("Joined feature columns into daily data: %s", joined_cols)
return out
def _forward_open_to_open_returns(open_: pd.DataFrame) -> pd.DataFrame: def _forward_open_to_open_returns(open_: pd.DataFrame) -> pd.DataFrame:
"""Return earned by a close-formed signal after next-open execution. """Return earned by a close-formed signal after next-open execution.
@@ -105,6 +144,8 @@ def compute_alpha(
alpha_name: str, alpha_name: str,
alpha_type: str, alpha_type: str,
universe: dict | None = None, universe: dict | None = None,
feature_paths: Iterable[str | Path] | None = None,
feature_frames: Iterable[pd.DataFrame] | None = None,
**params, **params,
) -> pd.DataFrame: ) -> pd.DataFrame:
"""Compute alpha weights from raw data. """Compute alpha weights from raw data.
@@ -118,6 +159,10 @@ def compute_alpha(
:func:`investable_universe_mask`) *before* it is turned into :func:`investable_universe_mask`) *before* it is turned into
weights, so unheld names get weight 0. Keys are forwarded as keyword weights, so unheld names get weight 0. Keys are forwarded as keyword
arguments to :func:`investable_universe_mask`. arguments to :func:`investable_universe_mask`.
feature_paths: Optional parquet files/datasets keyed by ``symbol_id``
and ``date``. Their numeric feature columns are left-joined onto
``data`` before alpha logic runs.
feature_frames: Optional in-memory feature frames with the same schema.
**params: Constructor parameters for the alpha (e.g. ``lookback``, **params: Constructor parameters for the alpha (e.g. ``lookback``,
``vol_window``). Only the params the alpha's ``__init__`` accepts are ``vol_window``). Only the params the alpha's ``__init__`` accepts are
used; extras are ignored. used; extras are ignored.
@@ -128,12 +173,20 @@ def compute_alpha(
Raises: Raises:
KeyError: If ``alpha_type`` is not registered. KeyError: If ``alpha_type`` is not registered.
""" """
feature_inputs: list[pd.DataFrame] = []
if feature_paths:
feature_inputs.extend(read_derived_frames(feature_paths))
if feature_frames:
feature_inputs.extend(feature_frames)
if feature_inputs:
data = join_feature_frames(data, feature_inputs)
alpha = get_alpha(alpha_type, **params) alpha = get_alpha(alpha_type, **params)
close = _pivot_close(data) close = _pivot_close(data)
signal = alpha.signal_from_data(data, close)
if universe is None: if universe is None:
weights = alpha.weights(close) weights = alpha.to_weights(signal)
else: else:
signal = alpha.signal(close)
mask = investable_universe_mask(data, signal, **universe) mask = investable_universe_mask(data, signal, **universe)
weights = alpha.to_weights(signal.where(mask)) weights = alpha.to_weights(signal.where(mask))
+25
View File
@@ -26,6 +26,31 @@ DATA_COLUMNS: Final[list[str]] = [
"pcfNcfTTM", # float64: P/CF (net cash flow, TTM) "pcfNcfTTM", # float64: P/CF (net cash flow, TTM)
] ]
# Required columns for raw intraday minute bar parquet files.
MINUTE_BAR_COLUMNS: Final[list[str]] = [
"symbol_id", # str: internal code like 'sh600000'
"symbol_name", # str: stock name like '浦发银行'
"datetime", # datetime64: intraday bar timestamp
"date", # date component, aligned with daily DATA_COLUMNS date
"time", # str: HH:MM:SS bar time
"frequency", # str: e.g. '5m'
"open", # float64
"high", # float64
"low", # float64
"close", # float64
"volume", # float64 (shares)
"amount", # float64 (turnover in yuan, raw/unadjusted)
"vwap", # float64: amount / volume
"adjustflag", # str: baostock adjustment flag; '3' for raw/unadjusted
]
# Required key columns for daily derived-data parquet files. Value columns are
# user/plugin-defined and must be numeric.
DERIVED_KEY_COLUMNS: Final[list[str]] = [
"symbol_id", # str
"date", # date: normalized daily timestamp
]
# Required columns for alpha parquet files. # Required columns for alpha parquet files.
# Alphas are position WEIGHTS: positive=long, negative=short. # Alphas are position WEIGHTS: positive=long, negative=short.
ALPHA_COLUMNS: Final[list[str]] = [ ALPHA_COLUMNS: Final[list[str]] = [
+35 -1
View File
@@ -3,7 +3,7 @@
import click import click
from datetime import date from datetime import date
from pipeline.data.downloader import download_universe from pipeline.data.downloader import download_minute_universe, download_universe
@click.group(name="data") @click.group(name="data")
@@ -42,3 +42,37 @@ def download(universe, start_date, end_date, output_dir, symbols, chunk_size, ad
f"{stats['n_rows']:,} bars, {stats['date_min']}{stats['date_max']}" f"{stats['n_rows']:,} bars, {stats['date_min']}{stats['date_max']}"
) )
click.echo(f"Dataset: {stats['dataset_path']}") click.echo(f"Dataset: {stats['dataset_path']}")
@data.command("download-minute")
@click.option(
"--universe", default="csi500",
help="Which universe: hs300, csi500, all (~5000 A-shares), file path, or comma-separated symbols",
)
@click.option("--start-date", default="2017-01-01", help="Start date YYYY-MM-DD")
@click.option("--end-date", default=str(date.today()), help="End date YYYY-MM-DD")
@click.option("--output-dir", default="data/minute_bars", help="Root for the partitioned dataset")
@click.option("--symbols", default=0, type=int, help="Max symbols (0=all)")
@click.option("--chunk-size", default=100, type=int, help="Symbols per durability flush")
@click.option("--frequency", default="5", help="Minute frequency: 5, 15, 30, or 60")
def download_minute(universe, start_date, end_date, output_dir, symbols, chunk_size, frequency):
"""Download raw Baostock minute bars into a partitioned parquet dataset.
Writes ``{output_dir}/{universe}/frequency=5m/month=YYYY-MM/*.pq`` for the
default 5-minute frequency.
"""
stats = download_minute_universe(
universe=universe,
start_date=start_date,
end_date=end_date,
output_dir=output_dir,
max_symbols=symbols,
chunk_size=chunk_size,
frequency=frequency,
)
click.echo(
f"\nSummary: {stats['n_symbols']}/{stats['n_requested']} symbols, "
f"{stats['n_rows']:,} bars, {stats['date_min']}{stats['date_max']}, "
f"frequency={stats['frequency']}"
)
click.echo(f"Dataset: {stats['dataset_path']}")
+126 -2
View File
@@ -11,9 +11,13 @@ import pyarrow.dataset as pads
# Reuse existing downloader and universe modules # Reuse existing downloader and universe modules
sys.path.insert(0, str(Path(__file__).resolve().parents[2])) sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from data.downloader import download_daily_batch from data.downloader import (
_normalize_minute_frequency,
download_daily_batch,
download_minute_batch,
)
from data.universe import get_all_stocks, get_hs300_stocks, get_zz500_stocks from data.universe import get_all_stocks, get_hs300_stocks, get_zz500_stocks
from pipeline.common.schema import DATA_COLUMNS from pipeline.common.schema import DATA_COLUMNS, MINUTE_BAR_COLUMNS
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -89,6 +93,25 @@ def _write_month_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: s
) )
def _write_minute_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: str) -> None:
"""Append rows to a Hive-partitioned minute dataset.
Layout: ``frequency=5m/month=YYYY-MM/*.pq``.
"""
out = df.copy()
out["month"] = pd.to_datetime(out["date"]).dt.strftime("%Y-%m")
table = pa.Table.from_pandas(out, preserve_index=False)
pads.write_dataset(
table,
str(base_dir),
format="parquet",
partitioning=["frequency", "month"],
partitioning_flavor="hive",
basename_template=f"{basename_prefix}-{{i}}.pq",
existing_data_behavior="overwrite_or_ignore",
)
def download_universe( def download_universe(
universe: str = "csi500", universe: str = "csi500",
start_date: str = "2017-01-01", start_date: str = "2017-01-01",
@@ -177,3 +200,104 @@ def download_universe(
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()), "date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()), "date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
} }
def download_minute_universe(
universe: str = "csi500",
start_date: str = "2017-01-01",
end_date: str = "2026-12-31",
output_dir: str = "data/minute_bars",
max_symbols: int = 0,
chunk_size: int = 100,
frequency: str | int = 5,
) -> dict:
"""Download raw minute bars into a frequency/month-partitioned dataset.
Args:
universe: ``hs300``, ``csi500``, ``all``/``full``, a file path, or a
comma-separated symbol list.
start_date, end_date: ``YYYY-MM-DD`` bounds.
output_dir: Root under which ``{universe}/frequency=5m/month=YYYY-MM``
is written.
max_symbols: Cap on symbols (0 = all).
chunk_size: Symbols per durability flush.
frequency: Minute interval. ``5``/``"5"``/``"5m"`` are 5-minute bars.
Returns:
Stats dict with dataset path, row count, symbol count, date range, and
frequency label.
"""
_, frequency_label = _normalize_minute_frequency(frequency)
constituents = _resolve_universe(universe, max_symbols)
symbols = constituents["symbol_id"].tolist()
names = dict(zip(constituents["symbol_id"], constituents["symbol_name"]))
n_requested = len(symbols)
logger.info(
"Minute universe %s: %d symbols, %s%s, frequency=%s",
universe,
n_requested,
start_date,
end_date,
frequency,
)
base_dir = Path(output_dir) / universe
target_frequency_dir = base_dir / f"frequency={frequency_label}"
if target_frequency_dir.exists():
shutil.rmtree(target_frequency_dir)
base_dir.mkdir(parents=True, exist_ok=True)
buffer: list[pd.DataFrame] = []
chunk_idx = 0
succeeded = 0
n_rows = 0
date_min = None
date_max = None
def flush() -> None:
nonlocal buffer, chunk_idx, n_rows, date_min, date_max
if not buffer:
return
chunk = pd.concat(buffer, ignore_index=True)
_write_minute_partitions(chunk, base_dir, basename_prefix=f"chunk{chunk_idx:04d}")
n_rows += len(chunk)
cmin, cmax = chunk["date"].min(), chunk["date"].max()
date_min = cmin if date_min is None else min(date_min, cmin)
date_max = cmax if date_max is None else max(date_max, cmax)
logger.info(
"Flushed minute chunk %d: %d rows (%d symbols done)",
chunk_idx,
len(chunk),
succeeded,
)
buffer = []
chunk_idx += 1
for i, (symbol, df) in enumerate(
download_minute_batch(symbols, start_date, end_date, frequency=frequency), start=1
):
if df is None:
logger.warning(" %s: no minute data", symbol)
else:
df["symbol_id"] = symbol
df["symbol_name"] = names.get(symbol, symbol)
buffer.append(df[MINUTE_BAR_COLUMNS])
succeeded += 1
if len(buffer) >= chunk_size:
flush()
if i % 100 == 0:
logger.info("Minute progress: %d/%d symbols", i, n_requested)
flush()
if succeeded == 0:
raise RuntimeError("No minute data downloaded for any symbol")
return {
"dataset_path": str(base_dir),
"frequency": frequency_label,
"n_symbols": succeeded,
"n_requested": n_requested,
"n_rows": n_rows,
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
}
+2
View File
@@ -0,0 +1,2 @@
"""Daily derived-data plugin package."""
+38
View File
@@ -0,0 +1,38 @@
"""Base class for daily derived-data plugins."""
from abc import ABC, abstractmethod
import pandas as pd
class BaseDerivedData(ABC):
"""Compute daily, symbol-keyed numeric derived data.
Derived-data plugins may use daily bars, minute bars, or both as inputs, but
they must always return daily rows keyed by ``symbol_id,date``.
"""
#: Unique registry key. Every concrete derived-data plugin must set this.
name: str = ""
@abstractmethod
def compute(
self,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Compute daily derived data.
Args:
daily: Optional daily market data.
minute: Optional raw minute bars.
Returns:
DataFrame with ``symbol_id``, ``date``, and one or more numeric
derived-data columns.
"""
def __repr__(self) -> str:
params = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
return f"{type(self).__name__}({params})"
+145
View File
@@ -0,0 +1,145 @@
"""CLI for daily derived-data ingestion and computation."""
import click
import pandas as pd
from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
compute_derived,
read_derived_frame,
write_derived_frame,
)
from pipeline.derived.registry import (
available_derived,
load_derived_module,
)
@click.group(name="derived")
def derived():
"""Ingest, compute, and validate daily derived data."""
def _coerce(value: str):
"""Best-effort coercion of a CLI string to int, then float, else str."""
for cast in (int, float):
try:
return cast(value)
except ValueError:
continue
return value
def _parse_params(pairs: tuple[str, ...]) -> dict:
"""Parse repeated ``name=value`` options into a params dict."""
params: dict = {}
for pair in pairs:
if "=" not in pair:
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
key, value = pair.split("=", 1)
params[key.strip()] = _coerce(value.strip())
return params
def _read_optional_parquet(path: str | None) -> pd.DataFrame | None:
return None if path is None else pd.read_parquet(path)
def _summarize(result: pd.DataFrame) -> str:
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
return f"{len(result):,} rows, {len(value_cols)} columns"
@derived.command("list")
@click.option(
"--derived-module", "derived_modules", multiple=True,
help="External module(s) to import first (dotted path or .py file)",
)
def list_(derived_modules):
"""List the registered derived-data plugin types."""
for spec in derived_modules:
load_derived_module(spec)
for name in available_derived():
click.echo(name)
@derived.command("validate")
@click.option("--input-path", required=True, help="CSV/parquet file or parquet dataset to validate")
def validate(input_path):
"""Validate a daily derived-data file without writing output."""
try:
result = read_derived_frame(input_path)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Valid derived data: {input_path} ({_summarize(result)})")
@derived.command("ingest")
@click.option("--input-path", required=True, help="CSV/parquet file to ingest")
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
def ingest(input_path, derived_name, output_dir):
"""Ingest a user-provided daily derived-data CSV/parquet file."""
try:
result = read_derived_frame(input_path)
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
@derived.command("compute")
@click.option("--daily-path", default=None, help="Optional daily data parquet/dataset")
@click.option("--minute-path", default=None, help="Optional minute parquet/dataset")
@click.option("--derived-type", required=True, help="Registry key of the derived-data plugin")
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
@click.option(
"--derived-module", "derived_modules", multiple=True,
help="External module(s) to import so their derived-data plugins register",
)
@click.option(
"--param", "extra_params", multiple=True,
help="Extra derived-data constructor param as name=value (repeatable)",
)
def compute(
daily_path,
minute_path,
derived_type,
derived_name,
output_dir,
derived_modules,
extra_params,
):
"""Compute one daily derived-data file from daily and/or minute inputs."""
for spec in derived_modules:
load_derived_module(spec)
options = available_derived()
if derived_type not in options:
raise click.BadParameter(
f"Unknown derived-type '{derived_type}'. Available: {options}. "
f"Use --derived-module to register an external derived-data plugin.",
param_hint="--derived-type",
)
if daily_path is None and minute_path is None:
raise click.UsageError("At least one of --daily-path or --minute-path is required")
daily = _read_optional_parquet(daily_path)
if daily_path:
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
minute = _read_optional_parquet(minute_path)
if minute_path:
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
try:
result = compute_derived(
derived_type=derived_type,
daily=daily,
minute=minute,
**_parse_params(extra_params),
)
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
+115
View File
@@ -0,0 +1,115 @@
"""Derived-data computation and validation."""
import csv
import logging
from pathlib import Path
from typing import Iterable
import pandas as pd
from pandas.api.types import is_bool_dtype, is_numeric_dtype
from pipeline.common.schema import DERIVED_KEY_COLUMNS
from pipeline.derived.registry import get_derived
logger = logging.getLogger(__name__)
def validate_derived_frame(derived: pd.DataFrame) -> pd.DataFrame:
"""Validate and normalize a daily derived-data frame.
A valid derived frame is keyed by unique ``symbol_id,date`` rows and has at
least one numeric value column beyond those keys. Dates are normalized to
daily timestamps before duplicate-key checks.
"""
duplicated = derived.columns[derived.columns.duplicated()].tolist()
if duplicated:
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
missing = [col for col in DERIVED_KEY_COLUMNS if col not in derived.columns]
if missing:
raise ValueError(f"Derived data missing required columns: {missing}")
out = derived.copy()
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
if out.duplicated(DERIVED_KEY_COLUMNS).any():
raise ValueError("Derived data has duplicate symbol_id,date rows")
value_cols = [col for col in out.columns if col not in DERIVED_KEY_COLUMNS]
if not value_cols:
raise ValueError("Derived data must include at least one value column")
non_numeric = [
col
for col in value_cols
if is_bool_dtype(out[col]) or not is_numeric_dtype(out[col])
]
if non_numeric:
raise ValueError(f"Derived data value columns must be numeric: {non_numeric}")
out = out[DERIVED_KEY_COLUMNS + value_cols].copy()
return out.sort_values(DERIVED_KEY_COLUMNS).reset_index(drop=True)
def compute_derived(
derived_type: str,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
**params,
) -> pd.DataFrame:
"""Compute one registered derived-data plugin."""
if daily is None and minute is None:
raise ValueError("Derived data computation requires --daily-path or --minute-path")
derived = get_derived(derived_type, **params)
result = validate_derived_frame(derived.compute(daily=daily, minute=minute))
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
logger.info(
"Derived data '%s' (%r): %d symbols × %d dates, columns=%s",
derived_type,
derived,
result["symbol_id"].nunique(),
result["date"].nunique(),
value_cols,
)
return result
def read_derived_frame(path: str | Path) -> pd.DataFrame:
"""Read and validate one derived CSV/parquet file or parquet dataset."""
path = Path(path)
if path.suffix.lower() == ".csv":
return validate_derived_frame(_read_csv_with_duplicate_header_check(path))
return validate_derived_frame(pd.read_parquet(path))
def read_derived_frames(derived_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
"""Read and validate derived-data files."""
return [read_derived_frame(path) for path in derived_paths]
def write_derived_frame(
derived: pd.DataFrame,
derived_name: str,
output_dir: str | Path = "derived",
) -> Path:
"""Validate and write derived data to ``{output_dir}/{derived_name}.pq``."""
result = validate_derived_frame(derived)
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
out_path = output_dir / f"{derived_name}.pq"
result.to_parquet(out_path, index=False)
return out_path
def _read_csv_with_duplicate_header_check(path: Path) -> pd.DataFrame:
with path.open(newline="") as fh:
reader = csv.reader(fh)
try:
header = next(reader)
except StopIteration as exc:
raise ValueError("CSV input is empty") from exc
duplicated = sorted({col for col in header if header.count(col) > 1})
if duplicated:
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
return pd.read_csv(path)
+4
View File
@@ -0,0 +1,4 @@
"""Built-in derived-data library."""
from pipeline.derived.library import minute_daily_summary # noqa: F401
@@ -0,0 +1,88 @@
"""Daily summary data derived from raw minute bars."""
import numpy as np
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class MinuteDailySummaryDerived(BaseDerivedData):
"""Aggregate intraday bars into daily summary columns."""
name = "minute_daily_summary"
def compute(
self,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
) -> pd.DataFrame:
if minute is None:
raise ValueError("minute_daily_summary requires minute input")
minute = minute.copy()
minute["date"] = pd.to_datetime(minute["date"]).dt.normalize()
sort_cols = ["symbol_id", "date"]
if "datetime" in minute.columns:
minute["datetime"] = pd.to_datetime(minute["datetime"])
sort_cols.append("datetime")
elif "time" in minute.columns:
sort_cols.append("time")
minute = minute.sort_values(sort_cols)
grouped = minute.groupby(["symbol_id", "date"], sort=True)
summary = grouped.agg(
minute_bar_count=("close", "count"),
first_open=("open", "first"),
last_close=("close", "last"),
high=("high", "max"),
low=("low", "min"),
volume_sum=("volume", "sum"),
amount_sum=("amount", "sum"),
)
summary["minute_intraday_return"] = (
summary["last_close"] / summary["first_open"] - 1.0
)
summary["minute_intraday_range"] = summary["high"] / summary["low"] - 1.0
summary["minute_vwap"] = (
summary["amount_sum"] / summary["volume_sum"].where(summary["volume_sum"] > 0)
)
summary = summary.reset_index()
if daily is not None:
daily_keys = daily[["symbol_id", "date"]].copy()
daily_keys["date"] = pd.to_datetime(daily_keys["date"]).dt.normalize()
daily_keys = daily_keys.drop_duplicates(["symbol_id", "date"])
result = daily_keys.merge(summary, on=["symbol_id", "date"], how="left")
if "close" in daily.columns:
daily_close = daily[["symbol_id", "date", "close"]].copy()
daily_close["date"] = pd.to_datetime(daily_close["date"]).dt.normalize()
daily_close = daily_close.drop_duplicates(["symbol_id", "date"])
result = result.merge(
daily_close.rename(columns={"close": "daily_close"}),
on=["symbol_id", "date"],
how="left",
)
reference_close = result["daily_close"].fillna(result["last_close"])
else:
reference_close = result["last_close"]
else:
result = summary
reference_close = result["last_close"]
result["minute_vwap_deviation"] = (
result["minute_vwap"] / reference_close.replace(0.0, np.nan) - 1.0
)
return result[
[
"symbol_id",
"date",
"minute_bar_count",
"minute_intraday_return",
"minute_intraday_range",
"minute_vwap",
"minute_vwap_deviation",
]
]
+80
View File
@@ -0,0 +1,80 @@
"""Registry and factory for daily derived-data plugins."""
import importlib
import importlib.util
import inspect
from pathlib import Path
from typing import Optional, Type
from pipeline.derived.base import BaseDerivedData
_REGISTRY: dict[str, Type[BaseDerivedData]] = {}
_builtins_loaded = False
def register_derived(cls: Type[BaseDerivedData]) -> Type[BaseDerivedData]:
"""Class decorator that registers derived data under ``BaseDerivedData.name``."""
if not (isinstance(cls, type) and issubclass(cls, BaseDerivedData)):
raise TypeError(f"{cls!r} is not a BaseDerivedData subclass")
key = getattr(cls, "name", "")
if not key:
raise ValueError(f"{cls.__name__} must set a non-empty class attribute `name`")
existing = _REGISTRY.get(key)
if existing is not None and existing is not cls:
raise ValueError(
f"Derived data name '{key}' already registered by {existing.__name__}"
)
_REGISTRY[key] = cls
return cls
def available_derived() -> list[str]:
"""Sorted names of all registered derived-data plugins."""
_ensure_builtins()
return sorted(_REGISTRY)
def get_derived(name: str, **params) -> BaseDerivedData:
"""Instantiate a registered derived-data plugin by name.
Only parameters accepted by the plugin class's ``__init__`` are forwarded.
"""
_ensure_builtins()
if name not in _REGISTRY:
raise KeyError(f"Unknown derived data '{name}'. Available: {sorted(_REGISTRY)}")
cls = _REGISTRY[name]
accepted = _accepted_params(cls)
kwargs = params if accepted is None else {k: v for k, v in params.items() if k in accepted}
return cls(**kwargs)
def load_derived_module(spec: str) -> None:
"""Import an external module so its ``@register_derived`` classes register."""
looks_like_file = spec.endswith(".py") or Path(spec).expanduser().exists()
if looks_like_file:
path = Path(spec).expanduser().resolve()
if not path.exists():
raise FileNotFoundError(f"Derived data module not found: {path}")
module_spec = importlib.util.spec_from_file_location(path.stem, path)
if module_spec is None or module_spec.loader is None:
raise ImportError(f"Cannot load derived data module from {path}")
module = importlib.util.module_from_spec(module_spec)
module_spec.loader.exec_module(module)
else:
importlib.import_module(spec)
def _accepted_params(cls: Type[BaseDerivedData]) -> Optional[set[str]]:
"""Param names ``cls.__init__`` accepts, or None if it takes ``**kwargs``."""
sig = inspect.signature(cls.__init__)
if any(p.kind is p.VAR_KEYWORD for p in sig.parameters.values()):
return None
return {name for name in sig.parameters if name != "self"}
def _ensure_builtins() -> None:
global _builtins_loaded
if not _builtins_loaded:
import pipeline.derived.library # noqa: F401
_builtins_loaded = True
+1
View File
@@ -0,0 +1 @@
"""Daily feature plugin package."""
+8
View File
@@ -0,0 +1,8 @@
"""Compatibility alias for daily feature plugins.
The canonical plugin API is ``pipeline.derived``. ``BaseFeature`` remains as an
alias so existing external feature modules continue to register unchanged.
"""
from pipeline.derived.base import BaseDerivedData as BaseFeature
+108
View File
@@ -0,0 +1,108 @@
"""CLI for daily feature computation."""
import os
import click
import pandas as pd
from pipeline.features.compute import compute_feature
from pipeline.features.registry import available_features, load_feature_module
@click.group(name="feature")
def feature():
"""Compute daily feature parquet files from minute bars."""
def _coerce(value: str):
"""Best-effort coercion of a CLI string to int, then float, else str."""
for cast in (int, float):
try:
return cast(value)
except ValueError:
continue
return value
def _parse_params(pairs: tuple[str, ...]) -> dict:
"""Parse repeated ``name=value`` options into a params dict."""
params: dict = {}
for pair in pairs:
if "=" not in pair:
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
key, value = pair.split("=", 1)
params[key.strip()] = _coerce(value.strip())
return params
@feature.command("list")
@click.option(
"--feature-module", "feature_modules", multiple=True,
help="External module(s) to import first (dotted path or .py file)",
)
def list_(feature_modules):
"""List the registered feature types."""
for spec in feature_modules:
load_feature_module(spec)
for name in available_features():
click.echo(name)
@feature.command("compute")
@click.option("--minute-path", required=True, help="Path to minute parquet dataset/file")
@click.option("--daily-path", default=None, help="Optional daily data parquet for alignment")
@click.option("--feature-type", required=True, help="Registry key of the feature class")
@click.option("--feature-name", required=True, help="Name for this feature run/output file")
@click.option("--output-dir", default="features", help="Directory to save feature parquet")
@click.option(
"--feature-module", "feature_modules", multiple=True,
help="External module(s) to import so their features register (dotted path or .py file)",
)
@click.option(
"--param", "extra_params", multiple=True,
help="Extra feature constructor param as name=value (repeatable)",
)
def compute(
minute_path,
daily_path,
feature_type,
feature_name,
output_dir,
feature_modules,
extra_params,
):
"""Compute one daily feature file from raw minute bars."""
for spec in feature_modules:
load_feature_module(spec)
options = available_features()
if feature_type not in options:
raise click.BadParameter(
f"Unknown feature-type '{feature_type}'. Available: {options}. "
f"Use --feature-module to register an external feature.",
param_hint="--feature-type",
)
minute = pd.read_parquet(minute_path)
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
daily = None
if daily_path:
daily = pd.read_parquet(daily_path)
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
result = compute_feature(
minute=minute,
daily=daily,
feature_type=feature_type,
**_parse_params(extra_params),
)
os.makedirs(output_dir, exist_ok=True)
out_path = f"{output_dir}/{feature_name}.pq"
result.to_parquet(out_path, index=False)
feature_cols = [col for col in result.columns if col not in ("symbol_id", "date")]
click.echo(
f"Saved feature: {out_path} ({len(result):,} rows, "
f"{len(feature_cols)} columns)"
)
+41
View File
@@ -0,0 +1,41 @@
"""Compatibility wrappers for daily feature computation and validation."""
from pathlib import Path
from typing import Iterable
import pandas as pd
from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
compute_derived,
read_derived_frames,
validate_derived_frame,
)
FEATURE_KEY_COLUMNS = DERIVED_KEY_COLUMNS
def validate_feature_frame(features: pd.DataFrame) -> pd.DataFrame:
"""Validate and normalize a legacy daily feature frame."""
return validate_derived_frame(features)
def compute_feature(
minute: pd.DataFrame,
feature_type: str,
daily: pd.DataFrame | None = None,
**params,
) -> pd.DataFrame:
"""Compute one registered feature through the derived-data registry."""
return compute_derived(
derived_type=feature_type,
daily=daily,
minute=minute,
**params,
)
def read_feature_frames(feature_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
"""Read and validate feature/derived-data parquet files."""
return read_derived_frames(feature_paths)
+3
View File
@@ -0,0 +1,3 @@
"""Built-in feature library."""
from pipeline.features.library import minute_daily_summary # noqa: F401
@@ -0,0 +1,16 @@
"""Compatibility wrapper for the built-in minute daily summary plugin."""
import pandas as pd
from pipeline.derived.library.minute_daily_summary import MinuteDailySummaryDerived
class MinuteDailySummaryFeature(MinuteDailySummaryDerived):
"""Legacy minute-first wrapper around the derived-data implementation."""
def compute(
self,
minute: pd.DataFrame,
daily: pd.DataFrame | None = None,
) -> pd.DataFrame:
return super().compute(daily=daily, minute=minute)
+36
View File
@@ -0,0 +1,36 @@
"""Compatibility registry wrappers for daily feature plugins."""
from typing import Type
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import (
available_derived,
get_derived,
load_derived_module,
register_derived,
)
from pipeline.features.base import BaseFeature
def register_feature(cls: Type[BaseFeature]) -> Type[BaseFeature]:
"""Register a legacy feature plugin in the derived-data registry."""
return register_derived(cls)
def available_features() -> list[str]:
"""Sorted names of all registered feature/derived-data plugins."""
return available_derived()
def get_feature(name: str, **params) -> BaseDerivedData:
"""Instantiate a registered feature/derived-data plugin by name."""
if name == "minute_daily_summary":
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
return MinuteDailySummaryFeature(**params)
return get_derived(name, **params)
def load_feature_module(spec: str) -> None:
"""Import an external module so its ``@register_feature`` classes register."""
load_derived_module(spec)
+6 -5
View File
@@ -166,13 +166,14 @@ class ReferenceSimulator(ExecutionSimulator):
st = wide(data_df, "isST") if "isST" in data_df.columns else opn * 0.0 st = wide(data_df, "isST") if "isST" in data_df.columns else opn * 0.0
symbols = sorted(set(tgt.columns) | set(opn.columns)) symbols = sorted(set(tgt.columns) | set(opn.columns))
data_index = close.index
tgt = tgt.reindex(columns=symbols) tgt = tgt.reindex(columns=symbols)
opn = opn.reindex(columns=symbols) opn = opn.reindex(index=data_index, columns=symbols)
close = close.reindex(columns=symbols) close = close.reindex(columns=symbols)
preclose = preclose.reindex(columns=symbols) preclose = preclose.reindex(index=data_index, columns=symbols)
amount = amount.reindex(columns=symbols) amount = amount.reindex(index=data_index, columns=symbols)
tstat = tstat.reindex(columns=symbols) tstat = tstat.reindex(index=data_index, columns=symbols)
st = st.reindex(columns=symbols) st = st.reindex(index=data_index, columns=symbols)
sym_arr = np.asarray(symbols, dtype=object) sym_arr = np.asarray(symbols, dtype=object)
n = len(symbols) n = len(symbols)
+2
View File
@@ -0,0 +1,2 @@
"""Optional plugin packages for the research pipeline."""
+103
View File
@@ -0,0 +1,103 @@
# JoinQuant Comparison Plugin
This plugin exports frozen targets from the internal A-share research pipeline,
drives a standalone JoinQuant wrapper strategy, ingests JoinQuant output files,
and reconciles them against the internal reference simulator.
The plugin validates system mechanics, not alpha quality:
- date alignment
- symbol mapping
- target position generation
- open execution timing
- lot rounding and filled shares
- position carry
- trading cost and PnL accounting
- blocked trades from suspension and price limits
## Commands
```bash
uv run python cli.py joinquant prepare-smoke \
--out-dir /tmp/chinese-equity-quant-realdata
uv sync --extra joinquant-browser
uv run playwright install chromium
uv run python cli.py joinquant browser-login \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--strategy-url "https://www.joinquant.com/..." \
--flow backtest
uv run python cli.py joinquant run-browser-backtest \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_browser_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
uv run python cli.py joinquant write-browser-config \
--out-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--strategy-url "https://www.joinquant.com/<模拟盘 page>" \
--flow sim-trade
uv run python cli.py joinquant run-browser-sim \
--manifest-path /tmp/chinese-equity-quant-realdata/joinquant_smoke_manifest.json \
--config-path /tmp/chinese-equity-quant-realdata/joinquant_sim_config.json \
--storage-state ~/.config/chinese-equity-quant/joinquant_storage_state.json
uv run python cli.py joinquant export-targets \
--positions-path portfolio/run1.pq \
--portfolio-name run1 \
--mode target_shares \
--execution-calendar-path data/daily_bars/csi500 \
--start-date 2026-07-01 \
--end-date 2026-07-31 \
--out-dir plugins_output/joinquant/targets
uv run python cli.py joinquant write-wrapper \
--portfolio-name run1 \
--mode target_shares \
--out-path plugins_output/joinquant/wrapper_strategy_run1.py
uv run python cli.py joinquant ingest \
--portfolio-name run1 \
--fills-csv path/to/jq_fills.csv \
--positions-csv path/to/jq_positions.csv \
--pnl-csv path/to/jq_pnl.csv \
--out-dir plugins_output/joinquant/ingested
uv run python cli.py joinquant reconcile \
--portfolio-name run1 \
--targets-dir plugins_output/joinquant/targets/run1 \
--our-fills-path fills/run1.pq \
--our-positions-path portfolio/run1.pq \
--our-pnl-path pnl/run1.pq \
--jq-fills-path plugins_output/joinquant/ingested/run1/fills.pq \
--jq-positions-path plugins_output/joinquant/ingested/run1/positions.pq \
--jq-pnl-path plugins_output/joinquant/ingested/run1/pnl.pq \
--out-dir plugins_output/joinquant/reconcile
```
`target_shares` is the default and uses the built integer `position_shares`
from `portfolio build`, matching what the internal simulator executes.
For strict simulator-vs-JoinQuant comparison, pass `--execution-calendar-path`
so position dates are shifted to the next session open, matching the internal
simulator's next-open convention.
`prepare-smoke` automates the local side of the first sanity check: tiny real
data download, one-stock long-only position file, internal simulation, aligned
target export, wrapper generation, and a manifest with expected JoinQuant CSV
export paths.
`run-browser-backtest` automates the remote JoinQuant web run through
Playwright. It reuses a saved browser login state, executes the configured UI
actions, downloads JoinQuant CSVs when configured, and runs ingest/reconcile
automatically once all three CSVs are present.
`run-browser-sim` is the forward-test / 模拟盘 equivalent. Use a `--flow
sim-trade` config to upload the frozen next-session target file, save the
strategy, and start or restart the JoinQuant simulated-trading job. After close,
run it again with download actions or use `ingest` / `reconcile` directly on
exported CSVs.
+10
View File
@@ -0,0 +1,10 @@
"""JoinQuant comparison plugin.
This package keeps JoinQuant-specific export, ingest, and reconciliation code
outside the core portfolio modules.
"""
from plugins.joinquant.symbols import from_joinquant_symbol, to_joinquant_symbol
__all__ = ["from_joinquant_symbol", "to_joinquant_symbol"]
+710
View File
@@ -0,0 +1,710 @@
"""Browser automation for JoinQuant cloud backtest and simulated-trading runs.
JoinQuant's public ``jqdatasdk`` is a data API; cloud strategy upload/run/export
is exposed through the web application. This module keeps that automation
optional and configurable so UI selector drift does not affect the core test
suite.
"""
from __future__ import annotations
import json
import re
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from plugins.joinquant.ingest import ingest_joinquant_outputs
from plugins.joinquant.reconcile import reconcile_joinquant
DEFAULT_LOGIN_URL = "https://www.joinquant.com/user/login/index"
def _require_playwright():
try:
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError
from playwright.sync_api import sync_playwright
except ImportError as exc:
raise RuntimeError(
"Playwright is required for JoinQuant browser automation. Install it "
"in this uv environment, then install Chromium:\n"
" uv sync --extra joinquant-browser\n"
" uv run playwright install chromium"
) from exc
return sync_playwright, PlaywrightTimeoutError
def _backtest_actions() -> list[dict[str, Any]]:
return [
{"type": "goto", "url": "{strategy_url}"},
{
"type": "click",
"selector": "text=不再提示",
"timeout_ms": 5000,
"force": True,
"optional": True,
"description": "Dismiss JoinQuant first-run editor guide.",
},
{
"type": "click",
"selector": "text=确定",
"timeout_ms": 5000,
"force": True,
"optional": True,
"description": "Close JoinQuant first-run editor guide.",
},
{
"type": "click",
"selector": "text=跳过",
"timeout_ms": 5000,
"force": True,
"optional": True,
"description": "Skip JoinQuant first-run editor guide.",
},
{
"type": "evaluate",
"script": (
"() => {"
"document.querySelectorAll("
"'.introjs-overlay,.introjs-helperLayer,.introjs-tooltipReferenceLayer,"
".introjs-tooltip,.introjs-disableInteraction'"
").forEach((node) => node.remove());"
"document.body.classList.remove('introjs-noscroll');"
"}"
),
"optional": True,
"description": "Remove any remaining JoinQuant intro overlay.",
},
{
"type": "set_ace_editor_file",
"selector": ".ace_editor",
"path": "{wrapper_path}",
"description": "Set generated wrapper strategy in the Ace code editor.",
},
{
"type": "set_input_files",
"selector": "input[type=file]",
"paths": "{target_csvs}",
"description": "Upload all aligned daily target CSV files.",
"timeout_ms": 5000,
"optional": True,
},
{
"type": "evaluate",
"script": (
"(arg) => {"
"const setValue = (selector, value) => {"
"const el = document.querySelector(selector);"
"if (!el) return false;"
"el.value = value;"
"el.setAttribute('value', value);"
"el.dispatchEvent(new Event('input', {bubbles: true}));"
"el.dispatchEvent(new Event('change', {bubbles: true}));"
"return true;"
"};"
"return {"
"start: setValue('#startTime', arg.start),"
"end: setValue('#endTime', arg.end),"
"capital: setValue('#daily_backtest_capital_base_box', arg.capital)"
"};"
"}"
),
"arg": {
"start": "{backtest_start_date}",
"end": "{backtest_end_date}",
"capital": "{booksize}",
},
"description": "Set JoinQuant backtest dates and starting capital.",
"optional": True,
},
{
"type": "click",
"selector": "#algo-save-button",
"timeout_ms": 5000,
"description": "Save the JoinQuant strategy code.",
"optional": True,
},
{
"type": "click",
"selector": ".bootstrap-dialog .btn-primary, .modal.in button:has-text(\"确定\")",
"timeout_ms": 5000,
"force": True,
"description": "Confirm any JoinQuant save dialog.",
"optional": True,
},
{"type": "wait_for_timeout", "timeout_ms": 2000},
{
"type": "click",
"selector": "#daily-new-backtest-button",
"description": "Start the JoinQuant backtest.",
},
{
"type": "wait_for_selector",
"selector": "text=/回测完成|运行完成|Backtest complete|Finished/i",
"timeout_ms": 600_000,
"optional": True,
},
{
"type": "download",
"selector": "text=/导出成交|下载成交|fills|trades/i",
"save_as": "{expected_joinquant_csvs.fills}",
"timeout_ms": 15000,
"optional": True,
},
{
"type": "download",
"selector": "text=/导出持仓|下载持仓|positions/i",
"save_as": "{expected_joinquant_csvs.positions}",
"timeout_ms": 15000,
"optional": True,
},
{
"type": "download",
"selector": "text=/导出收益|下载收益|pnl|收益/i",
"save_as": "{expected_joinquant_csvs.pnl}",
"timeout_ms": 15000,
"optional": True,
},
{"type": "screenshot", "path": "{run_artifact_dir}/final.png"},
]
def _sim_trade_actions() -> list[dict[str, Any]]:
return [
{"type": "goto", "url": "{strategy_url}"},
{
"type": "paste_text_file",
"selector": "textarea, .ace_text-input, .cm-content, .CodeMirror textarea",
"path": "{wrapper_path}",
"description": "Paste generated wrapper strategy into the simulated-trading strategy editor.",
},
{
"type": "set_input_files",
"selector": "input[type=file]",
"paths": "{target_csvs}",
"description": "Upload frozen target CSV files for 模拟盘.",
"optional": True,
},
{
"type": "click",
"selector": "text=/保存|Save/i",
"description": "Save the strategy code and uploaded files.",
"optional": True,
},
{
"type": "wait_for_selector",
"selector": "text=/保存成功|已保存|Saved/i",
"timeout_ms": 120_000,
"optional": True,
},
{
"type": "click",
"selector": "text=/模拟盘|模拟交易|启动模拟|运行模拟|启动|重启|Run Sim|Start|Restart/i",
"description": "Start or restart the JoinQuant simulated-trading job.",
},
{
"type": "wait_for_selector",
"selector": "text=/运行中|已启动|模拟交易运行|Started|Running/i",
"timeout_ms": 180_000,
"optional": True,
},
{"type": "screenshot", "path": "{run_artifact_dir}/sim_trade_final.png"},
]
def default_browser_config(strategy_url: str = "", *, flow: str = "backtest") -> dict[str, Any]:
"""Return a selector/action template for JoinQuant browser automation."""
if flow not in {"backtest", "sim-trade", "sim_trade"}:
raise ValueError("flow must be 'backtest' or 'sim-trade'")
normalized_flow = "sim-trade" if flow == "sim_trade" else flow
actions = _sim_trade_actions() if normalized_flow == "sim-trade" else _backtest_actions()
return {
"flow": normalized_flow,
"strategy_url": strategy_url,
"login_url": DEFAULT_LOGIN_URL,
"headless": False,
"timeout_ms": 120_000,
"notes": [
"Fill strategy_url and selectors after inspecting your JoinQuant strategy page.",
"Use `joinquant browser-snapshot` to capture HTML/screenshots for selector tuning.",
"The action list is declarative and runs in order.",
],
"actions": actions,
}
def write_browser_config_template(
path: str | Path,
*,
strategy_url: str = "",
flow: str = "backtest",
) -> Path:
"""Write a JSON config template for browser automation."""
out_path = Path(path)
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text(
json.dumps(
default_browser_config(strategy_url, flow=flow),
indent=2,
ensure_ascii=False,
) + "\n",
encoding="utf-8",
)
return out_path
def load_json(path: str | Path) -> dict[str, Any]:
"""Load a JSON object from disk."""
data = json.loads(Path(path).read_text(encoding="utf-8"))
if not isinstance(data, dict):
raise ValueError(f"Expected JSON object in {path}")
return data
def load_env_file(path: str | Path) -> dict[str, str]:
"""Load simple KEY=VALUE env files without requiring shell-safe quoting."""
values: dict[str, str] = {}
for raw in Path(path).expanduser().read_text(encoding="utf-8").splitlines():
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
clean = value.strip()
if len(clean) >= 2 and clean[0] == clean[-1] and clean[0] in {"'", '"'}:
clean = clean[1:-1]
else:
clean = clean.strip("'\"")
values[key.strip()] = clean
return values
def _get_dotted(data: dict[str, Any], dotted: str) -> Any:
current: Any = data
for part in dotted.split("."):
if isinstance(current, dict) and part in current:
current = current[part]
else:
raise KeyError(dotted)
return current
def _manifest_context(manifest: dict[str, Any], artifact_dir: Path, config: dict[str, Any]) -> dict[str, Any]:
targets_dir = Path(str(manifest["targets_dir"]))
target_csvs = [str(path) for path in sorted(targets_dir.glob("*.csv"))]
if not target_csvs:
raise ValueError(f"No target CSV files found under {targets_dir}")
target_dates = [
datetime.strptime(Path(path).stem, "%Y%m%d").strftime("%Y-%m-%d")
for path in target_csvs
]
context = dict(manifest)
context.update({
"strategy_url": config.get("strategy_url", ""),
"target_csvs": target_csvs,
"target_csvs_csv": ",".join(target_csvs),
"backtest_start_date": config.get("backtest_start_date") or min(target_dates),
"backtest_end_date": config.get("backtest_end_date") or max(target_dates),
"run_artifact_dir": str(artifact_dir),
})
return context
_TOKEN_RE = re.compile(r"^\{([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)\}$")
_PARTIAL_TOKEN_RE = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*)\}")
def resolve_template(value: Any, context: dict[str, Any]) -> Any:
"""Resolve ``{tokens}`` in config values against the manifest context."""
if isinstance(value, str):
whole = _TOKEN_RE.match(value)
if whole:
return _get_dotted(context, whole.group(1))
def replace(match: re.Match[str]) -> str:
resolved = _get_dotted(context, match.group(1))
return str(resolved)
return _PARTIAL_TOKEN_RE.sub(replace, value)
if isinstance(value, list):
return [resolve_template(item, context) for item in value]
if isinstance(value, dict):
return {key: resolve_template(val, context) for key, val in value.items()}
return value
def save_login_state(
*,
storage_state: str | Path,
login_url: str = DEFAULT_LOGIN_URL,
headless: bool = False,
wait_seconds: int = 0,
) -> Path:
"""Open a browser for manual login and save the authenticated state."""
if headless and wait_seconds <= 0:
raise ValueError("headless browser-login requires --wait-seconds > 0")
sync_playwright, _ = _require_playwright()
state_path = Path(storage_state).expanduser()
state_path.parent.mkdir(parents=True, exist_ok=True)
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=headless)
context = browser.new_context()
page = context.new_page()
page.goto(login_url, wait_until="domcontentloaded")
if wait_seconds > 0:
page.wait_for_timeout(wait_seconds * 1000)
else:
input("Log in to JoinQuant in the opened browser, then press Enter here to save state...")
context.storage_state(path=str(state_path))
browser.close()
state_path.chmod(0o600)
return state_path
def save_login_state_from_env(
*,
env_path: str | Path,
storage_state: str | Path,
login_url: str = DEFAULT_LOGIN_URL,
headless: bool = True,
out_dir: str | Path | None = None,
timeout_ms: int = 120_000,
) -> dict[str, Any]:
"""Try to log in with env-file credentials and save browser state.
This handles the normal password-login form. If JoinQuant presents CAPTCHA,
slide verification, SMS verification, or 2FA, the report will mark the run
as not logged in and save a screenshot for manual diagnosis.
"""
env = load_env_file(env_path)
username = env.get("JOINQUANT_USERNAME")
password = env.get("JOINQUANT_PASSWORD")
if not username or not password:
raise ValueError("JOINQUANT_USERNAME and JOINQUANT_PASSWORD are required")
sync_playwright, _ = _require_playwright()
state_path = Path(storage_state).expanduser()
state_path.parent.mkdir(parents=True, exist_ok=True)
artifact_dir = Path(out_dir or state_path.parent / "joinquant_login_artifacts")
artifact_dir.mkdir(parents=True, exist_ok=True)
screenshot_path = artifact_dir / "login_after_submit.png"
html_path = artifact_dir / "login_after_submit.html"
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=headless)
context = browser.new_context()
page = context.new_page()
page.set_default_timeout(timeout_ms)
page.goto(login_url, wait_until="networkidle", timeout=timeout_ms)
page.locator('input[name="username"], input.pwd-phone').first.fill(username)
page.locator('input[name="pwd"], input.jq-login__password').first.fill(password)
checkbox = page.locator("#agreementBox, input.agreement-box").first
if checkbox.count():
checkbox.check(force=True)
page.locator("button.btnPwdSubmit, button.login-submit").first.click()
page.wait_for_timeout(5000)
html = page.content()
html_path.write_text(html, encoding="utf-8")
page.screenshot(path=str(screenshot_path), full_page=True)
login_inputs = page.locator('input[name="username"], input[name="pwd"]').count()
current_url = page.url
logged_in = login_inputs == 0 and "/user/login" not in current_url
if logged_in:
context.storage_state(path=str(state_path))
state_path.chmod(0o600)
browser.close()
report = {
"created_at": datetime.now(timezone.utc).isoformat(),
"logged_in": bool(logged_in),
"storage_state": str(state_path) if logged_in else "",
"artifact_dir": str(artifact_dir),
"screenshot": str(screenshot_path),
"html": str(html_path),
"current_url": current_url,
"notes": (
"Logged in and saved browser state."
if logged_in
else "Login did not complete. CAPTCHA/SMS/2FA or invalid credentials may be required."
),
}
report_path = artifact_dir / "login_report.json"
report["report_path"] = str(report_path)
report_path.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
return report
def browser_snapshot(
*,
url: str,
storage_state: str | Path,
out_dir: str | Path,
headless: bool = True,
timeout_ms: int = 60_000,
) -> dict[str, Path]:
"""Save a logged-in page screenshot and HTML for selector discovery."""
sync_playwright, _ = _require_playwright()
root = Path(out_dir)
root.mkdir(parents=True, exist_ok=True)
html_path = root / "page.html"
screenshot_path = root / "page.png"
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=headless)
context = browser.new_context(storage_state=str(Path(storage_state).expanduser()))
page = context.new_page()
page.set_default_timeout(timeout_ms)
page.goto(url, wait_until="networkidle")
html_path.write_text(page.content(), encoding="utf-8")
page.screenshot(path=str(screenshot_path), full_page=True)
browser.close()
return {"html": html_path, "screenshot": screenshot_path}
def _locator(page: Any, selector: str):
return page.locator(selector).first
def _action_fail(action: dict[str, Any], exc: Exception) -> None:
if action.get("optional", False):
return
raise exc
def _run_action(page: Any, action: dict[str, Any], context: dict[str, Any], timeout_default: int) -> dict[str, Any]:
resolved = resolve_template(action, context)
kind = resolved["type"]
timeout_ms = int(resolved.get("timeout_ms", timeout_default))
record: dict[str, Any] = {"type": kind, "status": "ok"}
try:
if kind == "goto":
page.goto(str(resolved["url"]), wait_until=resolved.get("wait_until", "domcontentloaded"), timeout=timeout_ms)
elif kind == "click":
_locator(page, str(resolved["selector"])).click(
timeout=timeout_ms,
force=bool(resolved.get("force", False)),
)
elif kind == "fill":
_locator(page, str(resolved["selector"])).fill(str(resolved.get("text", "")), timeout=timeout_ms)
elif kind == "press":
_locator(page, str(resolved["selector"])).press(str(resolved["key"]), timeout=timeout_ms)
elif kind == "set_input_files":
paths = resolved.get("paths", [])
if isinstance(paths, str):
paths = [path for path in paths.split(",") if path]
_locator(page, str(resolved["selector"])).set_input_files(paths, timeout=timeout_ms)
record["n_files"] = len(paths)
elif kind == "paste_text_file":
text = Path(str(resolved["path"])).read_text(encoding="utf-8")
loc = _locator(page, str(resolved["selector"]))
loc.click(timeout=timeout_ms)
page.keyboard.press("Control+A")
page.keyboard.insert_text(text)
record["n_chars"] = len(text)
elif kind == "set_ace_editor_file":
text = Path(str(resolved["path"])).read_text(encoding="utf-8")
selector = str(resolved.get("selector", ".ace_editor"))
page.wait_for_function(
"(selector) => Boolean(window.ace && document.querySelector(selector))",
arg=selector,
timeout=timeout_ms,
)
page.evaluate(
"""(arg) => {
const node = document.querySelector(arg.selector);
if (!node || !window.ace) {
throw new Error(`Ace editor not found for ${arg.selector}`);
}
const editor = window.ace.edit(node);
editor.setValue(arg.text, -1);
editor.clearSelection();
editor.focus();
return editor.getValue().length;
}""",
{"selector": selector, "text": text},
)
record["n_chars"] = len(text)
elif kind == "evaluate":
page.evaluate(str(resolved["script"]), resolved.get("arg"))
elif kind == "wait_for_selector":
page.wait_for_selector(str(resolved["selector"]), timeout=timeout_ms)
elif kind == "wait_for_timeout":
page.wait_for_timeout(int(resolved.get("timeout_ms", 1000)))
elif kind == "download":
save_as = Path(str(resolved["save_as"]))
save_as.parent.mkdir(parents=True, exist_ok=True)
with page.expect_download(timeout=timeout_ms) as download_info:
_locator(page, str(resolved["selector"])).click(timeout=timeout_ms)
download = download_info.value
download.save_as(str(save_as))
record["path"] = str(save_as)
elif kind == "screenshot":
path = Path(str(resolved["path"]))
path.parent.mkdir(parents=True, exist_ok=True)
page.screenshot(path=str(path), full_page=bool(resolved.get("full_page", True)))
record["path"] = str(path)
else:
raise ValueError(f"Unsupported browser action type: {kind}")
except Exception as exc: # pragma: no cover - exercised only with Playwright.
record["status"] = "skipped" if resolved.get("optional", False) else "failed"
record["error"] = str(exc)
_action_fail(resolved, exc)
return record
def run_browser_flow(
*,
manifest_path: str | Path,
config_path: str | Path,
storage_state: str | Path,
out_dir: str | Path | None = None,
headless: bool | None = None,
auto_reconcile: bool = True,
flow_name: str | None = None,
) -> dict[str, Any]:
"""Run configured browser automation for a JoinQuant web workflow."""
sync_playwright, _ = _require_playwright()
manifest = load_json(manifest_path)
config = load_json(config_path)
flow = flow_name or str(config.get("flow") or "browser")
artifact_dir = Path(out_dir or Path(str(manifest["joinquant_export_dir"])).parent / f"browser_{flow}")
artifact_dir.mkdir(parents=True, exist_ok=True)
context_vars = _manifest_context(manifest, artifact_dir, config)
timeout_ms = int(config.get("timeout_ms", 120_000))
actions = config.get("actions") or []
if not actions:
raise ValueError("Browser config contains no actions")
records: list[dict[str, Any]] = []
failure: Exception | None = None
failure_artifacts: dict[str, str] = {}
with sync_playwright() as pw:
browser = pw.chromium.launch(
headless=bool(config.get("headless", False) if headless is None else headless)
)
context = browser.new_context(
storage_state=str(Path(storage_state).expanduser()),
accept_downloads=True,
)
page = context.new_page()
page.set_default_timeout(timeout_ms)
try:
for action in actions:
records.append(_run_action(page, action, context_vars, timeout_ms))
except Exception as exc: # pragma: no cover - requires live browser UI.
failure = exc
screenshot_path = artifact_dir / "failure.png"
html_path = artifact_dir / "failure.html"
try:
page.screenshot(path=str(screenshot_path), full_page=True)
html_path.write_text(page.content(), encoding="utf-8")
failure_artifacts = {
"screenshot": str(screenshot_path),
"html": str(html_path),
}
except Exception as artifact_exc:
failure_artifacts = {"artifact_error": str(artifact_exc)}
records.append({
"type": "browser_flow",
"status": "failed",
"error": str(exc),
**failure_artifacts,
})
browser.close()
expected = manifest.get("expected_joinquant_csvs", {})
downloaded = {
key: str(path)
for key, value in expected.items()
if (path := Path(str(value))).exists()
}
reconcile_paths: dict[str, str] = {}
if auto_reconcile and {"fills", "positions", "pnl"}.issubset(downloaded):
ingested = ingest_joinquant_outputs(
portfolio_name=str(manifest["portfolio_name"]),
fills_csv=downloaded["fills"],
positions_csv=downloaded["positions"],
pnl_csv=downloaded["pnl"],
out_dir=Path(str(manifest["joinquant_export_dir"])).parent / "ingested",
)
reconciled = reconcile_joinquant(
portfolio_name=str(manifest["portfolio_name"]),
targets_dir=str(manifest["targets_dir"]),
our_fills_path=str(manifest["fills_path"]),
our_positions_path=str(manifest["positions_path"]),
our_pnl_path=str(manifest["pnl_path"]),
jq_fills_path=str(ingested["fills"]),
jq_positions_path=str(ingested["positions"]),
jq_pnl_path=str(ingested["pnl"]),
out_dir=Path(str(manifest["joinquant_export_dir"])).parent / "reconcile",
)
reconcile_paths = {key: str(value) for key, value in reconciled.items()}
report = {
"created_at": datetime.now(timezone.utc).isoformat(),
"manifest_path": str(manifest_path),
"config_path": str(config_path),
"flow": flow,
"status": "failed" if failure else "ok",
"storage_state": str(storage_state),
"artifact_dir": str(artifact_dir),
"actions": records,
"downloaded": downloaded,
"reconcile_paths": reconcile_paths,
}
report_path = artifact_dir / "browser_run_report.json"
report["report_path"] = str(report_path)
report_path.write_text(
json.dumps(report, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
if failure is not None:
raise RuntimeError(
f"Browser flow failed; report saved to {report_path}: {failure}"
) from failure
return report
def run_browser_backtest(
*,
manifest_path: str | Path,
config_path: str | Path,
storage_state: str | Path,
out_dir: str | Path | None = None,
headless: bool | None = None,
auto_reconcile: bool = True,
) -> dict[str, Any]:
"""Run configured browser automation for a JoinQuant backtest."""
return run_browser_flow(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=headless,
auto_reconcile=auto_reconcile,
flow_name="backtest",
)
def run_browser_sim_trade(
*,
manifest_path: str | Path,
config_path: str | Path,
storage_state: str | Path,
out_dir: str | Path | None = None,
headless: bool | None = None,
auto_reconcile: bool = True,
) -> dict[str, Any]:
"""Run configured browser automation for JoinQuant 模拟盘."""
return run_browser_flow(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=headless,
auto_reconcile=auto_reconcile,
flow_name="sim-trade",
)
+426
View File
@@ -0,0 +1,426 @@
"""CLI commands for the JoinQuant comparison plugin."""
from __future__ import annotations
import click
from plugins.joinquant.browser import (
browser_snapshot,
load_env_file,
run_browser_backtest,
run_browser_sim_trade,
save_login_state,
save_login_state_from_env,
write_browser_config_template,
)
from plugins.joinquant.export_targets import export_targets
from plugins.joinquant.ingest import ingest_joinquant_outputs
from plugins.joinquant.reconcile import reconcile_joinquant
from plugins.joinquant.smoke import prepare_smoke_test
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
@click.group(name="joinquant")
def joinquant():
"""Compare internal portfolio simulation with JoinQuant output."""
@joinquant.command("export-targets")
@click.option("--positions-path", required=True, help="Portfolio positions parquet from `portfolio build`")
@click.option("--portfolio-name", required=True, help="Portfolio run to export")
@click.option(
"--mode",
"mode",
type=click.Choice(["target_shares", "target_value"]),
default="target_shares",
show_default=True,
help="JoinQuant target order mode",
)
@click.option("--start-date", default=None, help="Inclusive YYYY-MM-DD start date")
@click.option("--end-date", default=None, help="Inclusive YYYY-MM-DD end date")
@click.option(
"--execution-calendar-path",
default=None,
help="Daily data parquet/dataset used to shift position dates to next execution session",
)
@click.option("--out-dir", default="plugins_output/joinquant/targets", show_default=True)
@click.option("--force", is_flag=True, help="Overwrite frozen target/snapshot files")
def export_targets_cmd(
positions_path,
portfolio_name,
mode,
start_date,
end_date,
execution_calendar_path,
out_dir,
force,
):
"""Export frozen daily target files for JoinQuant."""
snapshots = export_targets(
positions_path=positions_path,
portfolio_name=portfolio_name,
mode=mode,
start_date=start_date,
end_date=end_date,
execution_calendar_path=execution_calendar_path,
out_dir=out_dir,
force=force,
)
click.echo(f"Exported JoinQuant targets: {len(snapshots)} day(s)")
for snapshot in snapshots:
click.echo(
f" {snapshot['date']}: {snapshot['n_symbols']} symbols, "
f"sha256={str(snapshot['file_sha256'])[:12]}"
)
@joinquant.command("ingest")
@click.option("--portfolio-name", required=True, help="Portfolio run name")
@click.option("--fills-csv", required=True, help="JoinQuant fills CSV")
@click.option("--positions-csv", required=True, help="JoinQuant positions CSV")
@click.option("--pnl-csv", required=True, help="JoinQuant daily PnL CSV")
@click.option("--out-dir", default="plugins_output/joinquant/ingested", show_default=True)
def ingest_cmd(portfolio_name, fills_csv, positions_csv, pnl_csv, out_dir):
"""Normalize JoinQuant CSV exports to parquet."""
paths = ingest_joinquant_outputs(
portfolio_name=portfolio_name,
fills_csv=fills_csv,
positions_csv=positions_csv,
pnl_csv=pnl_csv,
out_dir=out_dir,
)
click.echo(f"Saved JoinQuant fills: {paths['fills']}")
click.echo(f"Saved JoinQuant positions: {paths['positions']}")
click.echo(f"Saved JoinQuant pnl: {paths['pnl']}")
@joinquant.command("reconcile")
@click.option("--portfolio-name", required=True, help="Portfolio run name")
@click.option("--targets-dir", required=True, help="Directory containing exported daily target files")
@click.option("--our-fills-path", required=True, help="Internal simulator fills parquet")
@click.option("--our-positions-path", required=True, help="Internal portfolio positions parquet")
@click.option("--our-pnl-path", required=True, help="Internal simulator PnL parquet")
@click.option("--jq-fills-path", required=True, help="Normalized JoinQuant fills parquet")
@click.option("--jq-positions-path", required=True, help="Normalized JoinQuant positions parquet")
@click.option("--jq-pnl-path", required=True, help="Normalized JoinQuant PnL parquet")
@click.option("--out-dir", default="plugins_output/joinquant/reconcile", show_default=True)
@click.option("--share-tolerance", default=0.0, show_default=True, type=float)
@click.option("--price-rel-tolerance", default=1e-4, show_default=True, type=float)
@click.option("--pnl-tolerance", default=1.0, show_default=True, type=float)
@click.option("--booksize", default=None, type=float, help="Booksize for value tolerance inference")
def reconcile_cmd(
portfolio_name,
targets_dir,
our_fills_path,
our_positions_path,
our_pnl_path,
jq_fills_path,
jq_positions_path,
jq_pnl_path,
out_dir,
share_tolerance,
price_rel_tolerance,
pnl_tolerance,
booksize,
):
"""Write per-symbol and daily JoinQuant reconciliation reports."""
paths = reconcile_joinquant(
portfolio_name=portfolio_name,
targets_dir=targets_dir,
our_fills_path=our_fills_path,
our_positions_path=our_positions_path,
our_pnl_path=our_pnl_path,
jq_fills_path=jq_fills_path,
jq_positions_path=jq_positions_path,
jq_pnl_path=jq_pnl_path,
out_dir=out_dir,
share_tolerance=share_tolerance,
price_rel_tolerance=price_rel_tolerance,
pnl_tolerance=pnl_tolerance,
booksize=booksize,
)
click.echo(f"Saved reconciliation parquet: {paths['daily_reconcile']}")
click.echo(f"Saved reconciliation summary: {paths['summary_md']}")
click.echo(f"Saved reconciliation CSV: {paths['summary_csv']}")
@joinquant.command("write-wrapper")
@click.option("--portfolio-name", required=True, help="Portfolio run name")
@click.option(
"--mode",
"mode",
type=click.Choice(["target_shares", "target_value"]),
default="target_shares",
show_default=True,
)
@click.option("--out-path", required=True, help="Path for generated standalone strategy")
@click.option("--allow-short", is_flag=True, help="Do not clip negative targets in the generated wrapper")
@click.option(
"--embed-targets-dir",
default=None,
help="Embed CSV target files from this directory into the strategy source",
)
def write_wrapper_cmd(portfolio_name, mode, out_path, allow_short, embed_targets_dir):
"""Generate a standalone JoinQuant wrapper strategy."""
path = write_wrapper_strategy(
portfolio_name=portfolio_name,
mode=mode,
out_path=out_path,
allow_short=allow_short,
embedded_targets_dir=embed_targets_dir,
)
click.echo(f"Saved JoinQuant wrapper strategy: {path}")
@joinquant.command("prepare-smoke")
@click.option("--out-dir", required=True, help="Root directory for generated smoke-test artifacts")
@click.option(
"--universe",
default="sh600000,sz000001,sh600519,sz002594,sz300750",
show_default=True,
help="Universe for the tiny real-data download",
)
@click.option("--trade-symbol", default="sh600000", show_default=True)
@click.option("--start-date", default="2024-01-02", show_default=True)
@click.option("--end-date", default="2024-01-12", show_default=True)
@click.option("--portfolio-name", default="jq_smoke_one_stock_long", show_default=True)
@click.option("--shares", default=1000, show_default=True, type=int)
@click.option("--booksize", default=1_000_000.0, show_default=True, type=float)
@click.option("--max-signal-dates", default=3, show_default=True, type=int)
@click.option("--cost-bps", default=5.0, show_default=True, type=float)
@click.option("--slippage-bps", default=5.0, show_default=True, type=float)
@click.option("--volume-frac", default=0.02, show_default=True, type=float)
@click.option("--force", is_flag=True, help="Overwrite existing frozen target files")
def prepare_smoke_cmd(
out_dir,
universe,
trade_symbol,
start_date,
end_date,
portfolio_name,
shares,
booksize,
max_signal_dates,
cost_bps,
slippage_bps,
volume_frac,
force,
):
"""Prepare a one-command local real-data JoinQuant smoke test."""
manifest = prepare_smoke_test(
out_dir=out_dir,
universe=universe,
trade_symbol=trade_symbol,
start_date=start_date,
end_date=end_date,
portfolio_name=portfolio_name,
shares=shares,
booksize=booksize,
max_signal_dates=max_signal_dates,
cost_bps=cost_bps,
slippage_bps=slippage_bps,
volume_frac=volume_frac,
force=force,
)
click.echo(f"Prepared JoinQuant smoke manifest: {manifest['manifest_path']}")
click.echo(f"Wrapper: {manifest['wrapper_path']}")
click.echo(f"Targets: {manifest['targets_dir']}")
click.echo(f"Expected JoinQuant exports: {manifest['joinquant_export_dir']}")
summary = manifest["local_summary"]
click.echo(
f"Local simulator: {summary['n_pnl_rows']} days, "
f"PnL={summary['total_pnl']:,.2f}, cost={summary['total_cost']:,.2f}, "
f"blocked={summary['blocked_trades']}"
)
@joinquant.command("browser-login")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
help="Path where the authenticated browser state is stored",
)
@click.option(
"--login-url",
default="https://www.joinquant.com/user/login/index",
show_default=True,
)
@click.option("--headless", is_flag=True, help="Use a headless browser")
@click.option(
"--wait-seconds",
default=0,
show_default=True,
type=int,
help="Seconds to wait before saving state; 0 prompts for Enter after login",
)
def browser_login_cmd(storage_state, login_url, headless, wait_seconds):
"""Open JoinQuant login and save reusable browser session state."""
path = save_login_state(
storage_state=storage_state,
login_url=login_url,
headless=headless,
wait_seconds=wait_seconds,
)
click.echo(f"Saved JoinQuant browser state: {path}")
@joinquant.command("browser-login-env")
@click.option(
"--env-path",
default="~/.config/chinese-equity-quant/joinquant.env",
show_default=True,
help="Env file with JOINQUANT_USERNAME and JOINQUANT_PASSWORD",
)
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
help="Path where the authenticated browser state is stored",
)
@click.option(
"--login-url",
default="https://www.joinquant.com/user/login/index",
show_default=True,
)
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--out-dir", default=None, help="Login artifact directory")
@click.option("--timeout-ms", default=120_000, show_default=True, type=int)
def browser_login_env_cmd(env_path, storage_state, login_url, headed, out_dir, timeout_ms):
"""Try credential-based JoinQuant login from an env file."""
env = load_env_file(env_path)
missing = [
key for key in ["JOINQUANT_USERNAME", "JOINQUANT_PASSWORD"]
if not env.get(key)
]
if missing:
raise click.ClickException(f"Missing required env keys: {', '.join(missing)}")
report = save_login_state_from_env(
env_path=env_path,
storage_state=storage_state,
login_url=login_url,
headless=not headed,
out_dir=out_dir,
timeout_ms=timeout_ms,
)
click.echo(f"Saved login report: {report['report_path']}")
if report["logged_in"]:
click.echo(f"Saved JoinQuant browser state: {report['storage_state']}")
else:
raise click.ClickException(
"JoinQuant login did not complete. Check the saved screenshot/HTML; "
"CAPTCHA/SMS/2FA may require `joinquant browser-login` once."
)
@joinquant.command("write-browser-config")
@click.option("--out-path", required=True, help="Path for JSON browser automation config")
@click.option("--strategy-url", default="", help="JoinQuant strategy edit/backtest/模拟盘 URL")
@click.option(
"--flow",
type=click.Choice(["backtest", "sim-trade"]),
default="backtest",
show_default=True,
help="Browser automation template to write",
)
def write_browser_config_cmd(out_path, strategy_url, flow):
"""Write a browser automation selector/action config template."""
path = write_browser_config_template(out_path, strategy_url=strategy_url, flow=flow)
click.echo(f"Saved JoinQuant browser config template: {path}")
@joinquant.command("browser-snapshot")
@click.option("--url", required=True, help="Logged-in JoinQuant page URL to inspect")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
)
@click.option("--out-dir", required=True, help="Directory for page.html and page.png")
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--timeout-ms", default=60_000, show_default=True, type=int)
def browser_snapshot_cmd(url, storage_state, out_dir, headed, timeout_ms):
"""Save a page screenshot and HTML for selector discovery."""
paths = browser_snapshot(
url=url,
storage_state=storage_state,
out_dir=out_dir,
headless=not headed,
timeout_ms=timeout_ms,
)
click.echo(f"Saved HTML: {paths['html']}")
click.echo(f"Saved screenshot: {paths['screenshot']}")
@joinquant.command("run-browser-backtest")
@click.option("--manifest-path", required=True, help="Manifest from `joinquant prepare-smoke`")
@click.option("--config-path", required=True, help="JSON browser automation config")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
)
@click.option("--out-dir", default=None, help="Browser-run artifact directory")
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--no-auto-reconcile", is_flag=True, help="Skip ingest/reconcile after downloads")
def run_browser_backtest_cmd(
manifest_path,
config_path,
storage_state,
out_dir,
headed,
no_auto_reconcile,
):
"""Run JoinQuant browser automation from manifest and config."""
report = run_browser_backtest(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=not headed,
auto_reconcile=not no_auto_reconcile,
)
click.echo(f"Saved browser run report: {report['report_path']}")
if report["downloaded"]:
click.echo("Downloaded JoinQuant CSVs:")
for key, path in report["downloaded"].items():
click.echo(f" {key}: {path}")
if report["reconcile_paths"]:
click.echo(f"Saved reconciliation summary: {report['reconcile_paths']['summary_md']}")
@joinquant.command("run-browser-sim")
@click.option("--manifest-path", required=True, help="Manifest from target preparation")
@click.option("--config-path", required=True, help="JSON simulated-trading browser config")
@click.option(
"--storage-state",
default="~/.config/chinese-equity-quant/joinquant_storage_state.json",
show_default=True,
)
@click.option("--out-dir", default=None, help="Browser-run artifact directory")
@click.option("--headed", is_flag=True, help="Run with a visible browser")
@click.option("--no-auto-reconcile", is_flag=True, help="Skip ingest/reconcile after downloads")
def run_browser_sim_cmd(
manifest_path,
config_path,
storage_state,
out_dir,
headed,
no_auto_reconcile,
):
"""Run JoinQuant 模拟盘 browser automation from manifest and config."""
report = run_browser_sim_trade(
manifest_path=manifest_path,
config_path=config_path,
storage_state=storage_state,
out_dir=out_dir,
headless=not headed,
auto_reconcile=not no_auto_reconcile,
)
click.echo(f"Saved browser sim-trade report: {report['report_path']}")
if report["downloaded"]:
click.echo("Downloaded JoinQuant CSVs:")
for key, path in report["downloaded"].items():
click.echo(f" {key}: {path}")
if report["reconcile_paths"]:
click.echo(f"Saved reconciliation summary: {report['reconcile_paths']['summary_md']}")
+248
View File
@@ -0,0 +1,248 @@
"""Export portfolio positions as frozen JoinQuant target files."""
from __future__ import annotations
import hashlib
import json
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Iterable, Literal
import pandas as pd
from pipeline.common.schema import POSITION_COLUMNS
from plugins.joinquant.schema import JOINQUANT_TARGET_COLUMNS
from plugins.joinquant.symbols import to_joinquant_symbol
ExportMode = Literal["target_shares", "target_value"]
def _date_text(value: object) -> str:
return pd.Timestamp(value).strftime("%Y-%m-%d")
def _date_file_stem(date_text: str) -> str:
return pd.Timestamp(date_text).strftime("%Y%m%d")
def _snapshot_root_for(targets_root: Path) -> Path:
if targets_root.name == "targets":
return targets_root.parent / "snapshots"
return targets_root / "snapshots"
def _sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as fh:
for chunk in iter(lambda: fh.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def _check_position_columns(df: pd.DataFrame) -> None:
missing = [col for col in POSITION_COLUMNS if col not in df.columns]
if missing:
raise ValueError(f"Positions input missing required columns: {missing}")
def _filter_dates(
df: pd.DataFrame,
start_date: str | None,
end_date: str | None,
*,
date_column: str = "date",
) -> pd.DataFrame:
out = df.copy()
out[date_column] = pd.to_datetime(out[date_column]).dt.normalize()
if start_date:
out = out[out[date_column] >= pd.Timestamp(start_date).normalize()]
if end_date:
out = out[out[date_column] <= pd.Timestamp(end_date).normalize()]
return out
def _read_execution_calendar(path: str | Path) -> pd.DatetimeIndex:
data = pd.read_parquet(path)
if "date" not in data.columns:
raise ValueError("execution calendar parquet must contain a 'date' column")
dates = pd.to_datetime(data["date"], errors="coerce").dropna().dt.normalize()
return pd.DatetimeIndex(sorted(dates.unique()))
def _apply_execution_calendar(df: pd.DataFrame, calendar_path: str | Path) -> pd.DataFrame:
calendar = _read_execution_calendar(calendar_path)
if calendar.empty:
raise ValueError("execution calendar contains no dates")
source_dates = pd.to_datetime(df["date"]).dt.normalize()
positions = calendar.searchsorted(source_dates, side="right")
out = df.copy()
out["source_date"] = source_dates
out["export_date"] = pd.NaT
valid = positions < len(calendar)
out.loc[valid, "export_date"] = calendar.take(positions[valid])
return out[out["export_date"].notna()].copy()
def build_target_frame(
positions: pd.DataFrame,
*,
portfolio_name: str | None = None,
mode: ExportMode = "target_shares",
start_date: str | None = None,
end_date: str | None = None,
snapshot_ids: dict[str, str] | None = None,
execution_calendar_path: str | Path | None = None,
) -> pd.DataFrame:
"""Build normalized JoinQuant target rows from portfolio positions.
``target_shares`` is populated from ``position_shares`` because the core
simulator executes the discretized book, not continuous research shares.
"""
if mode not in {"target_shares", "target_value"}:
raise ValueError("mode must be 'target_shares' or 'target_value'")
_check_position_columns(positions)
df = positions.copy()
df["date"] = pd.to_datetime(df["date"]).dt.normalize()
if execution_calendar_path is not None:
df = _apply_execution_calendar(df, execution_calendar_path)
df = df.drop(columns=["date"]).rename(columns={"export_date": "date"})
df["source_date"] = pd.to_datetime(df["source_date"]).dt.strftime("%Y-%m-%d")
df = _filter_dates(df, start_date, end_date)
else:
df = _filter_dates(df, start_date, end_date)
if portfolio_name is not None:
df = df[df["portfolio_name"].astype(str) == portfolio_name]
if df.empty:
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
out = pd.DataFrame({
"date": df["date"].map(_date_text),
"portfolio_name": df["portfolio_name"].astype(str),
"symbol_id": df["symbol_id"].astype(str),
"jq_symbol": df["symbol_id"].map(to_joinquant_symbol),
"target_shares": pd.to_numeric(df["position_shares"], errors="coerce").fillna(0).astype("int64"),
"target_value": pd.to_numeric(df["target_value"], errors="coerce").fillna(0.0),
"target_weight": pd.to_numeric(df["target_weight"], errors="coerce").fillna(0.0),
"export_mode": mode,
"snapshot_id": "",
})
if snapshot_ids:
out["snapshot_id"] = out["date"].map(snapshot_ids).fillna("")
return out[JOINQUANT_TARGET_COLUMNS].sort_values(
["date", "portfolio_name", "symbol_id"]
).reset_index(drop=True)
def export_targets(
positions_path: str | Path,
*,
portfolio_name: str,
mode: ExportMode = "target_shares",
out_dir: str | Path = "plugins_output/joinquant/targets",
start_date: str | None = None,
end_date: str | None = None,
execution_calendar_path: str | Path | None = None,
force: bool = False,
) -> list[dict[str, object]]:
"""Export one daily CSV/parquet target file plus a snapshot JSON per date.
Args:
positions_path: Parquet file produced by ``portfolio build``.
portfolio_name: Portfolio run to export.
mode: ``target_shares`` or ``target_value``.
out_dir: Target root. Files are written to ``out_dir/portfolio_name``.
If the root is named ``targets``, snapshots are written to the
sibling ``snapshots`` directory.
start_date: Optional inclusive start date.
end_date: Optional inclusive end date.
execution_calendar_path: Optional daily-bar parquet dataset used to
shift each position date to the next available execution session.
This matches the internal simulator's next-open convention.
force: If false, existing target or snapshot files are treated as
frozen and cause ``FileExistsError``.
Returns:
Snapshot metadata dictionaries, one per exported date.
"""
positions_path = Path(positions_path)
targets_root = Path(out_dir)
snapshot_root = _snapshot_root_for(targets_root)
targets_portfolio_dir = targets_root / portfolio_name
snapshots_portfolio_dir = snapshot_root / portfolio_name
targets_portfolio_dir.mkdir(parents=True, exist_ok=True)
snapshots_portfolio_dir.mkdir(parents=True, exist_ok=True)
positions = pd.read_parquet(positions_path)
filtered = positions.copy()
filtered["date"] = pd.to_datetime(filtered["date"]).dt.normalize()
if execution_calendar_path is not None:
filtered = _apply_execution_calendar(filtered, execution_calendar_path)
filtered = filtered.drop(columns=["date"]).rename(columns={"export_date": "date"})
filtered["source_date"] = pd.to_datetime(filtered["source_date"]).dt.strftime("%Y-%m-%d")
filtered = _filter_dates(filtered, start_date, end_date)
else:
filtered = _filter_dates(filtered, start_date, end_date)
filtered = filtered[filtered["portfolio_name"].astype(str) == portfolio_name]
if filtered.empty:
return []
date_texts = sorted(filtered["date"].map(_date_text).unique())
snapshot_ids = {
date_text: f"jq-{portfolio_name}-{date_text}-{uuid.uuid4().hex[:12]}"
for date_text in date_texts
}
targets = build_target_frame(
filtered,
portfolio_name=portfolio_name,
mode=mode,
snapshot_ids=snapshot_ids,
execution_calendar_path=None,
)
snapshots: list[dict[str, object]] = []
for date_text, daily in targets.groupby("date", sort=True):
stem = _date_file_stem(date_text)
csv_path = targets_portfolio_dir / f"{stem}.csv"
parquet_path = targets_portfolio_dir / f"{stem}.parquet"
snapshot_path = snapshots_portfolio_dir / f"{stem}.json"
existing: Iterable[Path] = (csv_path, parquet_path, snapshot_path)
if not force:
conflicts = [str(path) for path in existing if path.exists()]
if conflicts:
raise FileExistsError(
"Frozen JoinQuant target already exists; use --force to overwrite: "
+ ", ".join(conflicts)
)
daily = daily[JOINQUANT_TARGET_COLUMNS].reset_index(drop=True)
daily.to_csv(csv_path, index=False)
daily.to_parquet(parquet_path, index=False)
file_hash = _sha256_file(csv_path)
snapshot = {
"snapshot_id": snapshot_ids[date_text],
"portfolio_name": portfolio_name,
"date": date_text,
"export_mode": mode,
"source_positions_path": str(positions_path),
"execution_calendar_path": str(execution_calendar_path) if execution_calendar_path else None,
"created_at": datetime.now(timezone.utc).isoformat(),
"n_symbols": int(len(daily)),
"file_sha256": file_hash,
"notes": "Frozen JoinQuant target file.",
"target_csv_path": str(csv_path),
"target_parquet_path": str(parquet_path),
}
snapshot_path.write_text(
json.dumps(snapshot, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
snapshots.append(snapshot)
return snapshots
+225
View File
@@ -0,0 +1,225 @@
"""Normalize JoinQuant CSV exports into plugin parquet schemas."""
from __future__ import annotations
import re
from pathlib import Path
import numpy as np
import pandas as pd
from plugins.joinquant.schema import (
JOINQUANT_FILL_COLUMNS,
JOINQUANT_PNL_COLUMNS,
JOINQUANT_POSITION_COLUMNS,
)
from plugins.joinquant.symbols import normalize_symbol_pair, to_joinquant_symbol
def _clean_name(name: object) -> str:
text = str(name).strip().lower()
text = re.sub(r"[\s\-.()/]+", "_", text)
return re.sub(r"[^0-9a-z_]+", "", text).strip("_")
def _clean_columns(df: pd.DataFrame) -> pd.DataFrame:
out = df.copy()
out.columns = [_clean_name(col) for col in out.columns]
return out
def _pick(df: pd.DataFrame, candidates: list[str]) -> str | None:
for candidate in candidates:
clean = _clean_name(candidate)
if clean in df.columns:
return clean
return None
def _series_or_default(df: pd.DataFrame, candidates: list[str], default: object) -> pd.Series:
col = _pick(df, candidates)
if col is None:
return pd.Series([default] * len(df), index=df.index)
return df[col]
def _date_series(df: pd.DataFrame) -> pd.Series:
values = _series_or_default(
df,
["date", "trade_date", "datetime", "time", "created_at"],
pd.NaT,
)
parsed = pd.to_datetime(values, errors="coerce").dt.normalize()
return parsed.dt.strftime("%Y-%m-%d").fillna("")
def _numeric(values: pd.Series, default: float = 0.0) -> pd.Series:
return pd.to_numeric(values, errors="coerce").replace([np.inf, -np.inf], np.nan).fillna(default)
def _text(values: pd.Series, default: str = "") -> pd.Series:
return values.fillna(default).astype(str)
def _portfolio_series(df: pd.DataFrame, portfolio_name: str) -> pd.Series:
return _text(_series_or_default(df, ["portfolio_name", "portfolio", "strategy"], portfolio_name), portfolio_name)
def _symbol_frame(df: pd.DataFrame) -> pd.DataFrame:
internal_col = _pick(df, ["symbol_id", "internal_symbol"])
jq_col = _pick(df, ["jq_symbol", "security", "stock", "symbol", "code", "order_book_id"])
symbol_ids: list[str] = []
jq_symbols: list[str] = []
for idx in df.index:
internal = df.at[idx, internal_col] if internal_col else None
jq_value = df.at[idx, jq_col] if jq_col else None
value = internal if internal is not None and str(internal).strip() else jq_value
if value is None or not str(value).strip():
symbol_ids.append("")
jq_symbols.append("")
continue
symbol_id, jq_symbol = normalize_symbol_pair(value)
if jq_value is not None and str(jq_value).strip():
try:
_, jq_symbol = normalize_symbol_pair(jq_value)
except ValueError:
jq_symbol = to_joinquant_symbol(symbol_id)
symbol_ids.append(symbol_id)
jq_symbols.append(jq_symbol)
return pd.DataFrame({"symbol_id": symbol_ids, "jq_symbol": jq_symbols}, index=df.index)
def _signed_shares(shares: pd.Series, side: pd.Series) -> pd.Series:
signed = _numeric(shares, 0.0)
side_text = side.fillna("").astype(str).str.lower()
sell = side_text.str.contains("sell|short|close|reduce|-", regex=True)
buy = side_text.str.contains("buy|long|open|add|\\+", regex=True)
signed = signed.abs()
signed = signed.mask(sell, -signed)
signed = signed.mask(~(sell | buy), _numeric(shares, 0.0))
return signed
def normalize_fills_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
"""Read a JoinQuant fills CSV and return ``JOINQUANT_FILL_COLUMNS``."""
raw = _clean_columns(pd.read_csv(path))
symbols = _symbol_frame(raw)
side = _text(_series_or_default(raw, ["side", "action", "direction"], ""))
requested = _signed_shares(
_series_or_default(raw, ["requested_shares", "target_shares", "amount", "order_amount"], 0),
side,
)
filled = _signed_shares(
_series_or_default(raw, ["filled_shares", "filled", "filled_amount", "deal_amount", "traded_shares"], 0),
side,
)
price = _numeric(_series_or_default(raw, ["fill_price", "price", "avg_cost", "avg_price"], np.nan), np.nan)
trade_value = _numeric(
_series_or_default(raw, ["trade_value", "value", "filled_value", "turnover"], np.nan),
np.nan,
)
trade_value = trade_value.fillna((filled * price).abs()).fillna(0.0)
out = pd.DataFrame({
"date": _date_series(raw),
"portfolio_name": _portfolio_series(raw, portfolio_name),
"symbol_id": symbols["symbol_id"],
"jq_symbol": symbols["jq_symbol"],
"order_id": _text(_series_or_default(raw, ["order_id", "id"], "")),
"side": side,
"requested_shares": requested.astype(float),
"filled_shares": filled.astype(float),
"fill_price": price.astype(float),
"trade_value": trade_value.astype(float),
"trade_cost": _numeric(_series_or_default(raw, ["trade_cost", "cost", "commission", "fee"], 0.0), 0.0),
"blocked": _numeric(_series_or_default(raw, ["blocked", "is_blocked"], 0), 0).astype("int64"),
"raw_status": _text(_series_or_default(raw, ["raw_status", "status", "order_status"], "")),
})
return out[JOINQUANT_FILL_COLUMNS]
def normalize_positions_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
"""Read a JoinQuant positions CSV and return ``JOINQUANT_POSITION_COLUMNS``."""
raw = _clean_columns(pd.read_csv(path))
symbols = _symbol_frame(raw)
out = pd.DataFrame({
"date": _date_series(raw),
"portfolio_name": _portfolio_series(raw, portfolio_name),
"symbol_id": symbols["symbol_id"],
"jq_symbol": symbols["jq_symbol"],
"position_shares": _numeric(
_series_or_default(raw, ["position_shares", "shares", "amount", "quantity", "total_amount"], 0),
0,
),
"position_value": _numeric(
_series_or_default(raw, ["position_value", "market_value", "value"], 0.0),
0.0,
),
"cash": _numeric(_series_or_default(raw, ["cash", "available_cash"], np.nan), np.nan),
"total_value": _numeric(
_series_or_default(raw, ["total_value", "portfolio_value", "total_asset"], np.nan),
np.nan,
),
})
return out[JOINQUANT_POSITION_COLUMNS]
def normalize_pnl_csv(path: str | Path, portfolio_name: str) -> pd.DataFrame:
"""Read a JoinQuant daily PnL CSV and return ``JOINQUANT_PNL_COLUMNS``."""
raw = _clean_columns(pd.read_csv(path))
total_value = _numeric(
_series_or_default(raw, ["total_value", "portfolio_value", "total_asset"], np.nan),
np.nan,
)
pnl = _numeric(_series_or_default(raw, ["pnl", "daily_pnl", "profit", "returns_value"], np.nan), np.nan)
if pnl.isna().all() and total_value.notna().any():
pnl = total_value.diff().fillna(0.0)
out = pd.DataFrame({
"date": _date_series(raw),
"portfolio_name": _portfolio_series(raw, portfolio_name),
"gross_exposure": _numeric(
_series_or_default(raw, ["gross_exposure", "gross", "positions_value", "market_value"], np.nan),
np.nan,
),
"net_exposure": _numeric(
_series_or_default(raw, ["net_exposure", "net"], np.nan),
np.nan,
),
"cash": _numeric(_series_or_default(raw, ["cash", "available_cash"], np.nan), np.nan),
"total_value": total_value,
"pnl": pnl.fillna(0.0),
"cost": _numeric(_series_or_default(raw, ["cost", "trade_cost", "commission", "fee"], 0.0), 0.0),
"turnover": _numeric(_series_or_default(raw, ["turnover"], 0.0), 0.0),
})
return out[JOINQUANT_PNL_COLUMNS]
def ingest_joinquant_outputs(
*,
portfolio_name: str,
fills_csv: str | Path,
positions_csv: str | Path,
pnl_csv: str | Path,
out_dir: str | Path = "plugins_output/joinquant/ingested",
) -> dict[str, Path]:
"""Normalize JoinQuant CSV exports and write parquet outputs."""
out_root = Path(out_dir) / portfolio_name
out_root.mkdir(parents=True, exist_ok=True)
fills = normalize_fills_csv(fills_csv, portfolio_name)
positions = normalize_positions_csv(positions_csv, portfolio_name)
pnl = normalize_pnl_csv(pnl_csv, portfolio_name)
paths = {
"fills": out_root / "fills.pq",
"positions": out_root / "positions.pq",
"pnl": out_root / "pnl.pq",
}
fills.to_parquet(paths["fills"], index=False)
positions.to_parquet(paths["positions"], index=False)
pnl.to_parquet(paths["pnl"], index=False)
return paths
+607
View File
@@ -0,0 +1,607 @@
"""Reconcile internal simulator output against normalized JoinQuant output."""
from __future__ import annotations
from pathlib import Path
import numpy as np
import pandas as pd
from plugins.joinquant.schema import (
JOINQUANT_FILL_COLUMNS,
JOINQUANT_PNL_COLUMNS,
JOINQUANT_POSITION_COLUMNS,
JOINQUANT_TARGET_COLUMNS,
RECONCILE_COLUMNS,
)
from plugins.joinquant.symbols import to_joinquant_symbol
def _date_text(value: object) -> str:
if pd.isna(value):
return ""
return pd.Timestamp(value).strftime("%Y-%m-%d")
def _read_parquet(path: str | Path | None) -> pd.DataFrame:
if path is None:
return pd.DataFrame()
return pd.read_parquet(path)
def _normalize_common(df: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
out = df.copy()
if "date" in out.columns:
out["date"] = out["date"].map(_date_text)
else:
out["date"] = ""
if "portfolio_name" not in out.columns:
out["portfolio_name"] = portfolio_name
out["portfolio_name"] = out["portfolio_name"].fillna(portfolio_name).astype(str)
if "symbol_id" in out.columns:
out["symbol_id"] = out["symbol_id"].fillna("").astype(str)
if "jq_symbol" not in out.columns and "symbol_id" in out.columns:
out["jq_symbol"] = out["symbol_id"].map(
lambda s: to_joinquant_symbol(s) if s else ""
)
elif "jq_symbol" in out.columns:
out["jq_symbol"] = out["jq_symbol"].fillna("").astype(str)
return out
def _numeric(df: pd.DataFrame, column: str, default: float = 0.0) -> pd.Series:
if column not in df.columns:
return pd.Series([default] * len(df), index=df.index, dtype=float)
return pd.to_numeric(df[column], errors="coerce").replace([np.inf, -np.inf], np.nan)
def _weighted_price(group: pd.DataFrame, price_col: str, shares_col: str) -> float:
prices = pd.to_numeric(group[price_col], errors="coerce")
shares = pd.to_numeric(group[shares_col], errors="coerce").abs()
valid = prices.notna() & shares.notna() & (shares > 0)
if not valid.any():
return np.nan
return float(np.average(prices[valid], weights=shares[valid]))
def _load_targets(targets_dir: str | Path, portfolio_name: str) -> pd.DataFrame:
root = Path(targets_dir)
if not root.exists():
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
files_by_stem: dict[str, Path] = {}
for path in sorted(root.glob("*.csv")):
files_by_stem[path.stem] = path
for path in sorted(root.glob("*.parquet")):
files_by_stem[path.stem] = path
frames: list[pd.DataFrame] = []
for path in files_by_stem.values():
if path.suffix == ".parquet":
frame = pd.read_parquet(path)
else:
frame = pd.read_csv(path)
frames.append(frame)
if not frames:
return pd.DataFrame(columns=JOINQUANT_TARGET_COLUMNS)
targets = pd.concat(frames, ignore_index=True)
targets = _normalize_common(targets, portfolio_name)
targets = targets[targets["portfolio_name"].astype(str) == portfolio_name]
if "target_shares" not in targets.columns:
targets["target_shares"] = 0
return targets.reindex(columns=JOINQUANT_TARGET_COLUMNS)
def _aggregate_targets(targets: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if targets.empty:
return pd.DataFrame(columns=["date", "portfolio_name", "symbol_id", "jq_symbol", "target_shares"])
targets = _normalize_common(targets, portfolio_name)
targets["target_shares"] = _numeric(targets, "target_shares", 0.0)
grouped = (
targets.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
.agg(jq_symbol=("jq_symbol", "last"), target_shares=("target_shares", "last"))
)
return grouped
def _aggregate_our_fills(fills: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if fills.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "our_filled_shares",
"our_position_shares", "our_cost", "our_trade_price", "our_blocked",
"our_target_shares",
])
fills = _normalize_common(fills, portfolio_name)
fills["traded_shares"] = _numeric(fills, "traded_shares", 0.0)
fills["realized_shares"] = _numeric(fills, "realized_shares", np.nan)
fills["trade_cost"] = _numeric(fills, "trade_cost", 0.0).fillna(0.0)
fills["target_shares"] = _numeric(fills, "target_shares", np.nan)
fills["blocked"] = _numeric(fills, "blocked", 0.0).fillna(0.0)
price_col = next(
(col for col in ["trade_price", "fill_price", "execution_price", "price"] if col in fills.columns),
None,
)
rows: list[dict[str, object]] = []
for key, group in fills.groupby(["date", "portfolio_name", "symbol_id"], sort=False):
row = {
"date": key[0],
"portfolio_name": key[1],
"symbol_id": key[2],
"our_filled_shares": float(group["traded_shares"].sum()),
"our_position_shares": float(group["realized_shares"].dropna().iloc[-1])
if group["realized_shares"].notna().any() else np.nan,
"our_cost": float(group["trade_cost"].sum()),
"our_trade_price": _weighted_price(group, price_col, "traded_shares")
if price_col else np.nan,
"our_blocked": int(group["blocked"].max()),
"our_target_shares": float(group["target_shares"].dropna().iloc[-1])
if group["target_shares"].notna().any() else np.nan,
}
rows.append(row)
return pd.DataFrame(rows)
def _aggregate_our_positions(positions: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if positions.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "jq_symbol",
"our_position_fallback", "our_position_price",
])
positions = _normalize_common(positions, portfolio_name)
positions["position_shares"] = _numeric(positions, "position_shares", np.nan)
positions["price"] = _numeric(positions, "price", np.nan)
return (
positions.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
.agg(
jq_symbol=("jq_symbol", "last"),
our_position_fallback=("position_shares", "last"),
our_position_price=("price", "last"),
)
)
def _aggregate_jq_fills(fills: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if fills.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "jq_filled_shares",
"jq_trade_price", "jq_cost", "jq_blocked", "jq_requested_shares",
"raw_status",
])
fills = _normalize_common(fills, portfolio_name)
for col in JOINQUANT_FILL_COLUMNS:
if col not in fills.columns:
fills[col] = np.nan
fills["filled_shares"] = _numeric(fills, "filled_shares", 0.0).fillna(0.0)
fills["requested_shares"] = _numeric(fills, "requested_shares", np.nan)
fills["fill_price"] = _numeric(fills, "fill_price", np.nan)
fills["trade_cost"] = _numeric(fills, "trade_cost", 0.0).fillna(0.0)
fills["blocked"] = _numeric(fills, "blocked", 0.0).fillna(0.0)
fills["raw_status"] = fills["raw_status"].fillna("").astype(str)
rows: list[dict[str, object]] = []
for key, group in fills.groupby(["date", "portfolio_name", "symbol_id"], sort=False):
row = {
"date": key[0],
"portfolio_name": key[1],
"symbol_id": key[2],
"jq_filled_shares": float(group["filled_shares"].sum()),
"jq_trade_price": _weighted_price(group, "fill_price", "filled_shares"),
"jq_cost": float(group["trade_cost"].sum()),
"jq_blocked": int(group["blocked"].max()),
"jq_requested_shares": float(group["requested_shares"].dropna().iloc[-1])
if group["requested_shares"].notna().any() else np.nan,
"raw_status": ";".join([s for s in group["raw_status"].astype(str) if s]),
}
rows.append(row)
return pd.DataFrame(rows)
def _aggregate_jq_positions(positions: pd.DataFrame, portfolio_name: str) -> pd.DataFrame:
if positions.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "symbol_id", "jq_symbol", "jq_position_shares",
])
positions = _normalize_common(positions, portfolio_name)
for col in JOINQUANT_POSITION_COLUMNS:
if col not in positions.columns:
positions[col] = np.nan
positions["position_shares"] = _numeric(positions, "position_shares", np.nan)
return (
positions.groupby(["date", "portfolio_name", "symbol_id"], as_index=False)
.agg(jq_symbol=("jq_symbol", "last"), jq_position_shares=("position_shares", "last"))
)
def _portfolio_frame(df: pd.DataFrame, portfolio_name: str, prefix: str) -> pd.DataFrame:
if df.empty:
return pd.DataFrame(columns=["date", "portfolio_name"])
out = _normalize_common(df, portfolio_name)
for col in JOINQUANT_PNL_COLUMNS:
if col not in out.columns:
out[col] = np.nan
keep = ["date", "portfolio_name", "gross_exposure", "net_exposure", "cash", "total_value", "pnl", "cost", "turnover"]
out = out[keep].copy()
for col in keep[2:]:
out[col] = pd.to_numeric(out[col], errors="coerce")
out = out.groupby(["date", "portfolio_name"], as_index=False).last()
return out.rename(columns={col: f"{prefix}_{col}" for col in keep[2:]})
def _infer_booksize(targets: pd.DataFrame, our_pnl: pd.DataFrame, jq_pnl: pd.DataFrame) -> float:
candidates: list[float] = []
if "target_value" in targets.columns:
gross = (
pd.to_numeric(targets["target_value"], errors="coerce")
.abs()
.replace([np.inf, -np.inf], np.nan)
.dropna()
.sum()
)
if gross > 0:
candidates.append(float(gross))
for df in (our_pnl, jq_pnl):
if "gross_exposure" in df.columns:
val = pd.to_numeric(df["gross_exposure"], errors="coerce").max()
if pd.notna(val) and val > 0:
candidates.append(float(val))
return max(candidates) if candidates else 1.0
def _status_reason(raw_status: object) -> str | None:
text = str(raw_status or "").lower()
if "suspend" in text or "halt" in text:
return "SUSPENSION"
if "limit_up" in text or "limit up" in text or "up_limit" in text:
return "LIMIT_UP_BLOCK"
if "limit_down" in text or "limit down" in text or "down_limit" in text:
return "LIMIT_DOWN_BLOCK"
if "volume" in text or "liquid" in text:
return "VOLUME_OR_LIQUIDITY"
if "cash" in text or "margin" in text:
return "CASH_CONSTRAINT"
return None
def _classify_symbol_row(
row: pd.Series,
*,
share_tol: float,
price_rel_tol: float,
value_tol: float,
pnl_tol: float,
) -> str:
target = row.get("target_shares", np.nan)
filled_diff = abs(row.get("filled_share_diff", 0.0))
position_diff = abs(row.get("position_share_diff", 0.0))
our_present = bool(row.get("_our_present", False))
jq_present = bool(row.get("_jq_present", False))
if pd.notna(target) and target < 0 and (filled_diff > share_tol or position_diff > share_tol or not jq_present):
return "SHORT_NOT_SUPPORTED"
if not jq_present and (our_present or pd.notna(target)):
return "MISSING_IN_JOINQUANT"
if not our_present and jq_present:
return "MISSING_IN_OUR_SYSTEM"
status_reason = _status_reason(row.get("raw_status", ""))
if (filled_diff > share_tol or position_diff > share_tol) and status_reason:
return status_reason
if filled_diff > share_tol or position_diff > share_tol:
return "UNKNOWN"
our_price = row.get("our_trade_price", np.nan)
jq_price = row.get("jq_trade_price", np.nan)
if pd.notna(our_price) and pd.notna(jq_price):
denom = max(abs(float(our_price)), abs(float(jq_price)), 1.0)
if abs(float(our_price) - float(jq_price)) > price_rel_tol * denom:
return "PRICE_MISMATCH"
if abs(row.get("cost_diff", 0.0)) > value_tol:
return "COST_MODEL"
if abs(row.get("pnl_diff", 0.0)) > pnl_tol:
return "UNKNOWN"
return "MATCH"
def _classify_portfolio_row(row: pd.Series, value_tol: float, pnl_tol: float) -> str:
our_present = bool(row.get("_our_present", False))
jq_present = bool(row.get("_jq_present", False))
if not jq_present and our_present:
return "MISSING_IN_JOINQUANT"
if not our_present and jq_present:
return "MISSING_IN_OUR_SYSTEM"
if abs(row.get("cost_diff", 0.0)) > value_tol:
return "COST_MODEL"
if abs(row.get("pnl_diff", 0.0)) > pnl_tol:
return "UNKNOWN"
return "MATCH"
def _build_symbol_reconcile(
*,
portfolio_name: str,
targets: pd.DataFrame,
our_fills: pd.DataFrame,
our_positions: pd.DataFrame,
our_pnl: pd.DataFrame,
jq_fills: pd.DataFrame,
jq_positions: pd.DataFrame,
jq_pnl: pd.DataFrame,
share_tol: float,
price_rel_tol: float,
value_tol: float,
pnl_tol: float,
) -> pd.DataFrame:
target_agg = _aggregate_targets(targets, portfolio_name)
our_fill_agg = _aggregate_our_fills(our_fills, portfolio_name)
our_pos_agg = _aggregate_our_positions(our_positions, portfolio_name)
jq_fill_agg = _aggregate_jq_fills(jq_fills, portfolio_name)
jq_pos_agg = _aggregate_jq_positions(jq_positions, portfolio_name)
key_cols = ["date", "portfolio_name", "symbol_id"]
keys = []
for frame in [target_agg, our_fill_agg, our_pos_agg, jq_fill_agg, jq_pos_agg]:
if not frame.empty:
keys.append(frame[key_cols])
if not keys:
return pd.DataFrame(columns=RECONCILE_COLUMNS)
base = pd.concat(keys, ignore_index=True).drop_duplicates()
result = base.merge(target_agg, on=key_cols, how="left")
result = result.merge(our_fill_agg, on=key_cols, how="left")
result = result.merge(our_pos_agg, on=key_cols, how="left", suffixes=("", "_ourpos"))
result = result.merge(jq_fill_agg, on=key_cols, how="left")
result = result.merge(jq_pos_agg, on=key_cols, how="left", suffixes=("", "_jqpos"))
jq_symbol_cols = [col for col in result.columns if col.startswith("jq_symbol")]
jq_symbol_values = result[jq_symbol_cols].copy() if jq_symbol_cols else pd.DataFrame(index=result.index)
result["jq_symbol"] = ""
for col in jq_symbol_values.columns:
values = jq_symbol_values[col].fillna("").astype(str)
result["jq_symbol"] = result["jq_symbol"].mask(
result["jq_symbol"].eq("") & values.ne(""),
values,
)
result["jq_symbol"] = result.apply(
lambda row: row["jq_symbol"] or to_joinquant_symbol(row["symbol_id"]),
axis=1,
)
result["_our_present"] = (
result[["our_filled_shares", "our_position_shares", "our_position_fallback"]]
.notna()
.any(axis=1)
)
result["_jq_present"] = (
result[["jq_filled_shares", "jq_position_shares"]].notna().any(axis=1)
)
target_shares = pd.to_numeric(result["target_shares"], errors="coerce")
our_target = pd.to_numeric(result["our_target_shares"], errors="coerce")
jq_target = pd.to_numeric(result["jq_requested_shares"], errors="coerce")
result["target_shares"] = target_shares.where(target_shares.notna(), our_target)
result["target_shares"] = result["target_shares"].where(
result["target_shares"].notna(),
jq_target,
)
our_position = pd.to_numeric(result["our_position_shares"], errors="coerce")
our_position_fallback = pd.to_numeric(result["our_position_fallback"], errors="coerce")
result["our_position_shares"] = our_position.where(
our_position.notna(),
our_position_fallback,
)
for col in [
"target_shares", "our_filled_shares", "jq_filled_shares",
"our_position_shares", "jq_position_shares", "our_cost", "jq_cost",
]:
result[col] = pd.to_numeric(result[col], errors="coerce").fillna(0.0)
result["filled_share_diff"] = result["our_filled_shares"] - result["jq_filled_shares"]
result["position_share_diff"] = result["our_position_shares"] - result["jq_position_shares"]
result["trade_price_diff"] = np.where(
result["our_trade_price"].notna() & result["jq_trade_price"].notna(),
result["our_trade_price"] - result["jq_trade_price"],
np.nan,
)
result["cost_diff"] = result["our_cost"] - result["jq_cost"]
our_daily = _portfolio_frame(our_pnl, portfolio_name, "our")
jq_daily = _portfolio_frame(jq_pnl, portfolio_name, "jq")
pnl_daily = our_daily.merge(jq_daily, on=["date", "portfolio_name"], how="outer")
if not pnl_daily.empty:
pnl_daily["our_pnl"] = pd.to_numeric(pnl_daily.get("our_pnl"), errors="coerce").fillna(0.0)
pnl_daily["jq_pnl"] = pd.to_numeric(pnl_daily.get("jq_pnl"), errors="coerce").fillna(0.0)
pnl_daily["pnl_diff"] = pnl_daily["our_pnl"] - pnl_daily["jq_pnl"]
result = result.merge(
pnl_daily[["date", "portfolio_name", "our_pnl", "jq_pnl", "pnl_diff"]],
on=["date", "portfolio_name"],
how="left",
)
else:
result["our_pnl"] = 0.0
result["jq_pnl"] = 0.0
result["pnl_diff"] = 0.0
for col in ["our_pnl", "jq_pnl", "pnl_diff"]:
result[col] = pd.to_numeric(result[col], errors="coerce").fillna(0.0)
result["raw_status"] = result.get("raw_status", "").fillna("")
result["diff_reason"] = result.apply(
_classify_symbol_row,
axis=1,
share_tol=share_tol,
price_rel_tol=price_rel_tol,
value_tol=value_tol,
pnl_tol=pnl_tol,
)
return result[RECONCILE_COLUMNS].sort_values(
["date", "portfolio_name", "symbol_id"]
).reset_index(drop=True)
def _build_portfolio_summary(
*,
portfolio_name: str,
our_pnl: pd.DataFrame,
jq_pnl: pd.DataFrame,
value_tol: float,
pnl_tol: float,
) -> pd.DataFrame:
our = _portfolio_frame(our_pnl, portfolio_name, "our")
jq = _portfolio_frame(jq_pnl, portfolio_name, "jq")
if our.empty and jq.empty:
return pd.DataFrame(columns=[
"date", "portfolio_name", "diff_reason",
"our_pnl", "jq_pnl", "pnl_diff",
])
summary = our.merge(jq, on=["date", "portfolio_name"], how="outer")
summary["_our_present"] = summary.filter(regex=r"^our_").notna().any(axis=1)
summary["_jq_present"] = summary.filter(regex=r"^jq_").notna().any(axis=1)
metrics = ["gross_exposure", "net_exposure", "cash", "total_value", "pnl", "cost", "turnover"]
for metric in metrics:
our_col = f"our_{metric}"
jq_col = f"jq_{metric}"
if our_col not in summary.columns:
summary[our_col] = np.nan
if jq_col not in summary.columns:
summary[jq_col] = np.nan
summary[f"{metric}_diff"] = (
pd.to_numeric(summary[our_col], errors="coerce").fillna(0.0)
- pd.to_numeric(summary[jq_col], errors="coerce").fillna(0.0)
)
summary["our_cumulative_pnl"] = (
pd.to_numeric(summary["our_pnl"], errors="coerce").fillna(0.0).cumsum()
)
summary["jq_cumulative_pnl"] = (
pd.to_numeric(summary["jq_pnl"], errors="coerce").fillna(0.0).cumsum()
)
summary["cumulative_pnl_diff"] = summary["our_cumulative_pnl"] - summary["jq_cumulative_pnl"]
summary["diff_reason"] = summary.apply(
_classify_portfolio_row,
axis=1,
value_tol=value_tol,
pnl_tol=pnl_tol,
)
summary = summary.drop(columns=["_our_present", "_jq_present"])
return summary.sort_values(["date", "portfolio_name"]).reset_index(drop=True)
def _write_summary_md(
path: Path,
*,
portfolio_name: str,
symbol_report: pd.DataFrame,
portfolio_summary: pd.DataFrame,
) -> None:
symbol_counts = (
symbol_report["diff_reason"].value_counts().sort_index()
if not symbol_report.empty else pd.Series(dtype=int)
)
portfolio_counts = (
portfolio_summary["diff_reason"].value_counts().sort_index()
if not portfolio_summary.empty else pd.Series(dtype=int)
)
lines = [
"# JoinQuant Reconciliation Summary",
"",
f"Portfolio: `{portfolio_name}`",
"",
"## Per-symbol Difference Counts",
"",
]
if symbol_counts.empty:
lines.append("No per-symbol rows were produced.")
else:
for reason, count in symbol_counts.items():
lines.append(f"- {reason}: {int(count)}")
lines.extend(["", "## Daily Portfolio Difference Counts", ""])
if portfolio_counts.empty:
lines.append("No daily portfolio rows were produced.")
else:
for reason, count in portfolio_counts.items():
lines.append(f"- {reason}: {int(count)}")
if not portfolio_summary.empty:
lines.extend(["", "## Daily Portfolio Preview", ""])
preview_cols = [
"date", "diff_reason", "our_pnl", "jq_pnl", "pnl_diff",
"our_cost", "jq_cost", "cost_diff",
]
preview_cols = [col for col in preview_cols if col in portfolio_summary.columns]
lines.append(",".join(preview_cols))
for row in portfolio_summary[preview_cols].head(20).itertuples(index=False):
lines.append(",".join(str(value) for value in row))
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
def reconcile_joinquant(
*,
portfolio_name: str,
targets_dir: str | Path,
our_fills_path: str | Path,
our_positions_path: str | Path,
our_pnl_path: str | Path,
jq_fills_path: str | Path,
jq_positions_path: str | Path,
jq_pnl_path: str | Path,
out_dir: str | Path = "plugins_output/joinquant/reconcile",
share_tolerance: float = 0.0,
price_rel_tolerance: float = 1e-4,
pnl_tolerance: float = 1.0,
booksize: float | None = None,
) -> dict[str, Path]:
"""Reconcile JoinQuant output against internal simulator output."""
targets = _load_targets(targets_dir, portfolio_name)
our_fills = _read_parquet(our_fills_path)
our_positions = _read_parquet(our_positions_path)
our_pnl = _read_parquet(our_pnl_path)
jq_fills = _read_parquet(jq_fills_path)
jq_positions = _read_parquet(jq_positions_path)
jq_pnl = _read_parquet(jq_pnl_path)
inferred_booksize = booksize or _infer_booksize(targets, our_pnl, jq_pnl)
value_tol = max(1.0, 1e-6 * float(inferred_booksize))
symbol_report = _build_symbol_reconcile(
portfolio_name=portfolio_name,
targets=targets,
our_fills=our_fills,
our_positions=our_positions,
our_pnl=our_pnl,
jq_fills=jq_fills,
jq_positions=jq_positions,
jq_pnl=jq_pnl,
share_tol=share_tolerance,
price_rel_tol=price_rel_tolerance,
value_tol=value_tol,
pnl_tol=pnl_tolerance,
)
portfolio_summary = _build_portfolio_summary(
portfolio_name=portfolio_name,
our_pnl=our_pnl,
jq_pnl=jq_pnl,
value_tol=value_tol,
pnl_tol=pnl_tolerance,
)
root = Path(out_dir) / portfolio_name
root.mkdir(parents=True, exist_ok=True)
paths = {
"daily_reconcile": root / "daily_reconcile.pq",
"summary_csv": root / "summary.csv",
"summary_md": root / "summary.md",
}
symbol_report.to_parquet(paths["daily_reconcile"], index=False)
portfolio_summary.to_csv(paths["summary_csv"], index=False)
_write_summary_md(
paths["summary_md"],
portfolio_name=portfolio_name,
symbol_report=symbol_report,
portfolio_summary=portfolio_summary,
)
return paths
+99
View File
@@ -0,0 +1,99 @@
"""Column contracts for the JoinQuant comparison plugin."""
from typing import Final
JOINQUANT_TARGET_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"target_shares",
"target_value",
"target_weight",
"export_mode",
"snapshot_id",
]
JOINQUANT_FILL_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"order_id",
"side",
"requested_shares",
"filled_shares",
"fill_price",
"trade_value",
"trade_cost",
"blocked",
"raw_status",
]
JOINQUANT_POSITION_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"position_shares",
"position_value",
"cash",
"total_value",
]
JOINQUANT_PNL_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"gross_exposure",
"net_exposure",
"cash",
"total_value",
"pnl",
"cost",
"turnover",
]
RECONCILE_COLUMNS: Final[list[str]] = [
"date",
"portfolio_name",
"symbol_id",
"jq_symbol",
"target_shares",
"our_filled_shares",
"jq_filled_shares",
"filled_share_diff",
"our_position_shares",
"jq_position_shares",
"position_share_diff",
"our_trade_price",
"jq_trade_price",
"trade_price_diff",
"our_cost",
"jq_cost",
"cost_diff",
"our_pnl",
"jq_pnl",
"pnl_diff",
"diff_reason",
]
DIFF_REASONS: Final[list[str]] = [
"MATCH",
"SYMBOL_MAPPING",
"PRICE_MISMATCH",
"LOT_ROUNDING",
"SUSPENSION",
"LIMIT_UP_BLOCK",
"LIMIT_DOWN_BLOCK",
"VOLUME_OR_LIQUIDITY",
"COST_MODEL",
"CASH_CONSTRAINT",
"SHORT_NOT_SUPPORTED",
"CORPORATE_ACTION",
"JOINQUANT_INTERNAL_ROUNDING",
"MISSING_IN_OUR_SYSTEM",
"MISSING_IN_JOINQUANT",
"UNKNOWN",
]
+191
View File
@@ -0,0 +1,191 @@
"""End-to-end local smoke preparation for JoinQuant comparison."""
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
import pandas as pd
from pipeline.common.schema import POSITION_COLUMNS
from pipeline.data.downloader import download_universe
from pipeline.portfolio.constraints import get_constraint
from pipeline.portfolio.simulator import ReferenceSimulator
from plugins.joinquant.export_targets import export_targets
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
def build_fixed_share_positions(
data: pd.DataFrame,
*,
trade_symbol: str,
portfolio_name: str,
shares: int,
booksize: float,
max_signal_dates: int | None = None,
) -> pd.DataFrame:
"""Create a deterministic long-only fixed-share position book.
The final available data date is excluded because the internal simulator
executes each signal date at the next available open.
"""
data = data.copy()
data["date"] = pd.to_datetime(data["date"]).dt.normalize()
symbol_data = (
data[data["symbol_id"].astype(str) == trade_symbol]
.sort_values("date")
.reset_index(drop=True)
)
if len(symbol_data) < 2:
raise ValueError(f"Need at least two daily bars for {trade_symbol}")
signal_data = symbol_data.iloc[:-1].copy()
if max_signal_dates is not None and max_signal_dates > 0:
signal_data = signal_data.tail(max_signal_dates)
if signal_data.empty:
raise ValueError("No signal dates available after excluding final data date")
rows: list[dict[str, object]] = []
for row in signal_data.itertuples(index=False):
price = float(row.close)
target_value = float(shares * price)
rows.append({
"symbol_id": trade_symbol,
"date": pd.Timestamp(row.date),
"portfolio_name": portfolio_name,
"target_weight": target_value / float(booksize),
"target_value": target_value,
"target_shares": float(shares),
"position_shares": int(shares),
"position_value": target_value,
"price": price,
})
return pd.DataFrame(rows, columns=POSITION_COLUMNS)
def prepare_smoke_test(
*,
out_dir: str | Path,
universe: str = "sh600000,sz000001,sh600519,sz002594,sz300750",
trade_symbol: str = "sh600000",
start_date: str = "2024-01-02",
end_date: str = "2024-01-12",
portfolio_name: str = "jq_smoke_one_stock_long",
shares: int = 1000,
booksize: float = 1_000_000.0,
max_signal_dates: int = 3,
cost_bps: float = 5.0,
slippage_bps: float = 5.0,
volume_frac: float = 0.02,
force: bool = False,
) -> dict[str, object]:
"""Run the local side of a tiny real-data JoinQuant smoke test."""
root = Path(out_dir)
root.mkdir(parents=True, exist_ok=True)
stats = download_universe(
universe=universe,
start_date=start_date,
end_date=end_date,
output_dir=root / "daily_bars",
max_symbols=0,
chunk_size=100,
adjust="qfq",
)
data_path = Path(stats["dataset_path"])
data = pd.read_parquet(data_path)
positions = build_fixed_share_positions(
data,
trade_symbol=trade_symbol,
portfolio_name=portfolio_name,
shares=shares,
booksize=booksize,
max_signal_dates=max_signal_dates,
)
portfolio_dir = root / "portfolio"
portfolio_dir.mkdir(parents=True, exist_ok=True)
positions_path = portfolio_dir / f"{portfolio_name}.pq"
positions.to_parquet(positions_path, index=False)
constraints = [
get_constraint("suspension"),
get_constraint("price_limit"),
get_constraint("volume_cap", max_frac=volume_frac),
]
sim = ReferenceSimulator(
constraints=constraints,
cost_bps=cost_bps,
slippage_bps=slippage_bps,
)
fills, pnl = sim.run(positions, data)
execution_dir = root / "execution"
fills_dir = execution_dir / "fills"
pnl_dir = execution_dir / "pnl"
fills_dir.mkdir(parents=True, exist_ok=True)
pnl_dir.mkdir(parents=True, exist_ok=True)
fills_path = fills_dir / f"{portfolio_name}.pq"
pnl_path = pnl_dir / f"{portfolio_name}.pq"
fills.to_parquet(fills_path, index=False)
pnl.to_parquet(pnl_path, index=False)
target_root = root / "plugins_output" / "joinquant" / "targets_aligned"
snapshots = export_targets(
positions_path=positions_path,
portfolio_name=portfolio_name,
mode="target_shares",
out_dir=target_root,
execution_calendar_path=data_path,
force=force,
)
wrapper_path = root / "plugins_output" / "joinquant" / f"wrapper_strategy_{portfolio_name}.py"
write_wrapper_strategy(
portfolio_name=portfolio_name,
mode="target_shares",
out_path=wrapper_path,
)
export_dir = root / "joinquant_exports"
export_dir.mkdir(parents=True, exist_ok=True)
manifest = {
"created_at": datetime.now(timezone.utc).isoformat(),
"portfolio_name": portfolio_name,
"universe": universe,
"trade_symbol": trade_symbol,
"start_date": start_date,
"end_date": end_date,
"shares": shares,
"booksize": booksize,
"data_path": str(data_path),
"positions_path": str(positions_path),
"fills_path": str(fills_path),
"pnl_path": str(pnl_path),
"targets_dir": str(target_root / portfolio_name),
"wrapper_path": str(wrapper_path),
"joinquant_export_dir": str(export_dir),
"expected_joinquant_csvs": {
"fills": str(export_dir / "jq_fills.csv"),
"positions": str(export_dir / "jq_positions.csv"),
"pnl": str(export_dir / "jq_pnl.csv"),
},
"target_snapshots": snapshots,
"local_summary": {
"n_data_rows": int(len(data)),
"n_position_rows": int(len(positions)),
"n_fill_rows": int(len(fills)),
"n_pnl_rows": int(len(pnl)),
"total_pnl": float(pnl["pnl"].sum()) if len(pnl) else 0.0,
"total_cost": float(pnl["cost"].sum()) if len(pnl) else 0.0,
"blocked_trades": int(fills["blocked"].sum()) if len(fills) else 0,
},
}
manifest_path = root / "joinquant_smoke_manifest.json"
manifest["manifest_path"] = str(manifest_path)
manifest_path.write_text(
json.dumps(manifest, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
return manifest
+94
View File
@@ -0,0 +1,94 @@
"""Symbol conversion between internal A-share ids and JoinQuant ids."""
from __future__ import annotations
import re
_INTERNAL_RE = re.compile(r"^(?P<prefix>sh|sz)(?P<code>\d{6})$", re.IGNORECASE)
_JOINQUANT_RE = re.compile(
r"^(?P<code>\d{6})\.(?P<exchange>XSHG|XSHE)$", re.IGNORECASE
)
_BARE_RE = re.compile(r"^\d{6}$")
def _validate_internal(prefix: str, code: str) -> None:
if prefix == "sh" and code.startswith("6"):
return
if prefix == "sz" and code.startswith(("0", "3")):
return
raise ValueError(f"Unsupported A-share symbol: {prefix}{code}")
def _validate_joinquant(code: str, exchange: str) -> None:
if exchange == "XSHG" and code.startswith("6"):
return
if exchange == "XSHE" and code.startswith(("0", "3")):
return
raise ValueError(f"Unsupported JoinQuant A-share symbol: {code}.{exchange}")
def to_joinquant_symbol(symbol_id: str) -> str:
"""Convert an internal symbol like ``sh600000`` to ``600000.XSHG``.
Args:
symbol_id: Internal A-share id with ``sh`` or ``sz`` prefix.
Returns:
JoinQuant security id.
Raises:
ValueError: If the symbol is malformed or outside supported A-share
Shanghai/Shenzhen equity prefixes.
"""
text = str(symbol_id).strip().lower()
match = _INTERNAL_RE.match(text)
if not match:
raise ValueError(f"Invalid internal symbol: {symbol_id!r}")
prefix = match.group("prefix").lower()
code = match.group("code")
_validate_internal(prefix, code)
exchange = "XSHG" if prefix == "sh" else "XSHE"
return f"{code}.{exchange}"
def from_joinquant_symbol(jq_symbol: str) -> str:
"""Convert a JoinQuant symbol like ``600000.XSHG`` to ``sh600000``."""
text = str(jq_symbol).strip().upper()
match = _JOINQUANT_RE.match(text)
if not match:
raise ValueError(f"Invalid JoinQuant symbol: {jq_symbol!r}")
code = match.group("code")
exchange = match.group("exchange").upper()
_validate_joinquant(code, exchange)
prefix = "sh" if exchange == "XSHG" else "sz"
return f"{prefix}{code}"
def normalize_symbol_pair(value: object) -> tuple[str, str]:
"""Return ``(symbol_id, jq_symbol)`` from any supported symbol spelling."""
text = str(value).strip()
if not text or text.lower() == "nan":
raise ValueError("Missing symbol")
if _INTERNAL_RE.match(text):
symbol_id = text.lower()
return symbol_id, to_joinquant_symbol(symbol_id)
if _JOINQUANT_RE.match(text):
jq_symbol = text.upper()
return from_joinquant_symbol(jq_symbol), jq_symbol
if _BARE_RE.match(text):
if text.startswith("6"):
symbol_id = f"sh{text}"
elif text.startswith(("0", "3")):
symbol_id = f"sz{text}"
else:
raise ValueError(f"Unsupported bare A-share code: {text}")
return symbol_id, to_joinquant_symbol(symbol_id)
raise ValueError(f"Unsupported symbol: {value!r}")
+271
View File
@@ -0,0 +1,271 @@
"""Generate a standalone JoinQuant wrapper strategy.
The module also defines default JoinQuant strategy hooks by executing the same
template with ``run1`` / ``target_shares`` defaults. That means this file can be
copied directly into JoinQuant for a quick smoke test, while the CLI can still
write a configured standalone file for a real run.
"""
from __future__ import annotations
import json
from pathlib import Path
from string import Template
from typing import Literal
WrapperMode = Literal["target_shares", "target_value"]
_WRAPPER_TEMPLATE = Template(
r'''# Standalone JoinQuant target wrapper generated by chinese-equity-quant.
# Copy this file and the exported daily CSV target files into JoinQuant.
#
# The file loader is isolated in _read_target_file(). The default implementation
# uses JoinQuant's read_file API for uploaded files. If your JoinQuant runtime
# allows HTTP or another storage backend, replace only that function.
import csv
import io
import json
PORTFOLIO_NAME = "${portfolio_name}"
TARGET_MODE = "${mode}"
ALLOW_SHORT = ${allow_short}
TARGET_FILE_PREFIX = "" # Optional uploaded-file prefix, for example "run1/"
_EMBEDDED_TARGETS = ${embedded_targets}
def initialize(context):
set_benchmark("000300.XSHG")
set_option("use_real_price", True)
g.portfolio_name = PORTFOLIO_NAME
g.target_mode = TARGET_MODE
g.targets_by_date = {}
run_daily(load_targets, time="before_open")
run_daily(rebalance_at_open, time="open")
run_daily(record_after_close, time="after_close")
def _today_text(context):
return context.current_dt.strftime("%Y-%m-%d")
def _today_file_name(context):
return context.current_dt.strftime("%Y%m%d") + ".csv"
def _read_target_file(file_name):
${embedded_target_read}
data = read_file(TARGET_FILE_PREFIX + file_name)
if isinstance(data, bytes):
data = data.decode("utf-8")
return data
def _load_target_rows(context):
file_name = _today_file_name(context)
text = _read_target_file(file_name)
rows = list(csv.DictReader(io.StringIO(text)))
clean_rows = []
for row in rows:
if row.get("portfolio_name") and row["portfolio_name"] != PORTFOLIO_NAME:
continue
jq_symbol = row.get("jq_symbol") or row.get("security") or row.get("symbol")
if not jq_symbol:
log.warn("Skipping target row with no jq_symbol: %s" % row)
continue
if TARGET_MODE == "target_shares":
target = int(float(row.get("target_shares") or 0))
elif TARGET_MODE == "target_value":
target = float(row.get("target_value") or 0.0)
else:
raise ValueError("Unsupported TARGET_MODE: %s" % TARGET_MODE)
if not ALLOW_SHORT and target < 0:
log.warn(
"SHORT_NOT_SUPPORTED clipping %s target from %s to 0" %
(jq_symbol, target)
)
target = 0
clean_rows.append({"jq_symbol": jq_symbol, "target": target, "raw": row})
return clean_rows
def load_targets(context):
date_text = _today_text(context)
try:
rows = _load_target_rows(context)
except Exception as exc:
log.error("Failed to load JoinQuant target file for %s: %s" % (date_text, exc))
rows = []
g.targets_by_date[date_text] = rows
log.info("JOINQUANT_TARGET_LOAD|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"target_mode": TARGET_MODE,
"n_targets": len(rows),
}, sort_keys=True))
def rebalance_at_open(context):
date_text = _today_text(context)
rows = g.targets_by_date.get(date_text, [])
target_symbols = set()
for row in rows:
security = row["jq_symbol"]
target_symbols.add(security)
if TARGET_MODE == "target_shares":
order_target(security, int(row["target"]))
else:
order_target_value(security, float(row["target"]))
log.info("JOINQUANT_ORDER_SUBMIT|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
"target_mode": TARGET_MODE,
"target": row["target"],
}, sort_keys=True))
for security in list(context.portfolio.positions.keys()):
if security not in target_symbols:
order_target(security, 0)
log.info("JOINQUANT_ORDER_CLOSE|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
}, sort_keys=True))
def _position_records(context):
records = []
cash = float(context.portfolio.available_cash)
total_value = float(context.portfolio.total_value)
for security, position in context.portfolio.positions.items():
records.append({
"date": _today_text(context),
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
"position_shares": int(position.total_amount),
"position_value": float(position.value),
"cash": cash,
"total_value": total_value,
})
return records
def _trade_records(context):
records = []
try:
trades = get_trades()
except Exception:
trades = {}
for trade_id, trade in trades.items():
amount = int(getattr(trade, "amount", 0))
price = float(getattr(trade, "price", 0.0))
security = getattr(trade, "security", "")
side = "buy" if amount >= 0 else "sell"
records.append({
"date": _today_text(context),
"portfolio_name": PORTFOLIO_NAME,
"jq_symbol": security,
"order_id": str(getattr(trade, "order_id", trade_id)),
"side": side,
"filled_shares": amount,
"fill_price": price,
"trade_value": abs(amount * price),
"trade_cost": float(getattr(trade, "commission", 0.0)),
"raw_status": "filled",
})
return records
def record_after_close(context):
date_text = _today_text(context)
for record in _trade_records(context):
log.info("JOINQUANT_FILL|%s" % json.dumps(record, sort_keys=True))
for record in _position_records(context):
log.info("JOINQUANT_POSITION|%s" % json.dumps(record, sort_keys=True))
log.info("JOINQUANT_PNL|%s" % json.dumps({
"date": date_text,
"portfolio_name": PORTFOLIO_NAME,
"cash": float(context.portfolio.available_cash),
"total_value": float(context.portfolio.total_value),
}, sort_keys=True))
'''
)
# Make this module itself usable as a JoinQuant strategy with defaults.
exec(_WRAPPER_TEMPLATE.substitute(
portfolio_name="run1",
mode="target_shares",
allow_short="False",
embedded_targets="{}",
embedded_target_read="",
))
def render_wrapper_strategy(
*,
portfolio_name: str,
mode: WrapperMode = "target_shares",
allow_short: bool = False,
embedded_targets: dict[str, str] | None = None,
) -> str:
"""Render the standalone JoinQuant wrapper strategy source."""
if mode not in {"target_shares", "target_value"}:
raise ValueError("mode must be 'target_shares' or 'target_value'")
embedded_targets = embedded_targets or {}
embedded_target_read = ""
if embedded_targets:
embedded_target_read = (
" if file_name in _EMBEDDED_TARGETS:\n"
" return _EMBEDDED_TARGETS[file_name]\n"
)
return _WRAPPER_TEMPLATE.substitute(
portfolio_name=portfolio_name,
mode=mode,
allow_short="True" if allow_short else "False",
embedded_targets=json.dumps(embedded_targets, ensure_ascii=False, indent=4),
embedded_target_read=embedded_target_read,
)
def _load_embedded_targets(targets_dir: str | Path | None) -> dict[str, str]:
if targets_dir is None:
return {}
root = Path(targets_dir)
targets = {
path.name: path.read_text(encoding="utf-8")
for path in sorted(root.glob("*.csv"))
}
if not targets:
raise ValueError(f"No CSV target files found under {root}")
return targets
def write_wrapper_strategy(
*,
portfolio_name: str,
mode: WrapperMode = "target_shares",
out_path: str | Path,
allow_short: bool = False,
embedded_targets_dir: str | Path | None = None,
) -> Path:
"""Write a configured standalone JoinQuant wrapper strategy."""
path = Path(out_path)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
render_wrapper_strategy(
portfolio_name=portfolio_name,
mode=mode,
allow_short=allow_short,
embedded_targets=_load_embedded_targets(embedded_targets_dir),
),
encoding="utf-8",
)
return path
+27
View File
@@ -16,11 +16,38 @@ dependencies = [
backtrader = [ backtrader = [
"backtrader>=1.9.76.123", "backtrader>=1.9.76.123",
] ]
joinquant-browser = [
"playwright>=1.61.0",
]
[dependency-groups] [dependency-groups]
dev = [ dev = [
"coverage>=7.14.1",
"pytest>=7.0.0", "pytest>=7.0.0",
] ]
[tool.uv] [tool.uv]
package = false package = false
[tool.pytest.ini_options]
markers = [
"network: tests that call live external data providers and are skipped unless explicitly enabled",
]
[tool.coverage.run]
branch = true
relative_files = true
source = [
".",
]
[tool.coverage.report]
fail_under = 95
show_missing = true
skip_covered = false
omit = [
"tests/*",
"docs/*",
"scripts/*",
".venv/*",
]
+40
View File
@@ -0,0 +1,40 @@
"""Run pytest under coverage in this environment.
The plain ``coverage run -m pytest`` path reloads NumPy under the current VS Code
Python startup environment, which breaks pandas/numpy reductions. Import NumPy
before starting coverage so the measured test run uses one stable NumPy module.
"""
from __future__ import annotations
import sys
import numpy # noqa: F401
import coverage
import pytest
def main(argv: list[str]) -> int:
pytest_args = argv or ["tests/", "-v"]
cov = coverage.Coverage(config_file=True)
cov.erase()
cov.start()
test_status = pytest.main(pytest_args)
cov.stop()
cov.save()
if test_status != 0:
return int(test_status)
total = cov.report()
print(f"\nCoverage total: {total:.2f}%")
fail_under = float(cov.config.fail_under)
if total < fail_under:
print(f"Coverage failure: total {total:.2f}% is below {fail_under:.2f}%")
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))
Binary file not shown.
+261
View File
@@ -0,0 +1,261 @@
"""Shared deterministic test data for offline workflow tests."""
from __future__ import annotations
import numpy as np
import pandas as pd
from pipeline.common.schema import (
ALPHA_COLUMNS,
COMBO_COLUMNS,
DATA_COLUMNS,
MINUTE_BAR_COLUMNS,
)
GENERATED_SYMBOLS: tuple[str, ...] = (
"sh600000",
"sz000001",
"sh600519",
"sz300750",
)
GENERATED_SYMBOL_NAMES: dict[str, str] = {
"sh600000": "PF Bank",
"sz000001": "Ping An Bank",
"sh600519": "Kweichow Moutai",
"sz300750": "CATL",
}
def generated_sessions(n_sessions: int = 12) -> pd.DatetimeIndex:
"""Return a fixed business-day calendar used by generated fixtures."""
return pd.bdate_range("2024-01-02", periods=n_sessions)
def make_generated_daily_bars(
n_sessions: int = 12,
include_missing: bool = True,
) -> pd.DataFrame:
"""Build daily bars with explicit edge cases and no randomness.
The panel covers four A-share symbols and includes a suspended row, an ST
flag, a zero-volume row, a missing symbol-date, and limit-style open/close
moves. Values are deterministic so tests can assert exact identities.
"""
dates = generated_sessions(n_sessions)
base_close = {
"sh600000": 10.00,
"sz000001": 15.00,
"sh600519": 1200.00,
"sz300750": 180.00,
}
returns = {
"sh600000": [0.000, 0.012, -0.006, 0.018, 0.100, -0.014, 0.006, 0.000, 0.008, -0.011, 0.004, 0.009],
"sz000001": [0.000, -0.008, 0.011, -0.004, 0.006, 0.000, -0.012, 0.009, 0.005, -0.007, 0.010, -0.003],
"sh600519": [0.000, 0.006, 0.004, -0.010, 0.012, -0.006, 0.005, 0.003, -0.009, 0.007, -0.004, 0.006],
"sz300750": [0.000, -0.010, 0.014, 0.006, -0.008, 0.011, -0.004, 0.009, -0.200, 0.012, -0.006, 0.008],
}
base_volume = {
"sh600000": 1_200_000.0,
"sz000001": 900_000.0,
"sh600519": 80_000.0,
"sz300750": 240_000.0,
}
rows: list[dict[str, object]] = []
for sym in GENERATED_SYMBOLS:
closes = [base_close[sym]]
pattern = returns[sym]
for step in range(1, n_sessions):
ret = pattern[step % len(pattern)]
closes.append(closes[-1] * (1.0 + ret))
closes_arr = np.asarray(closes, dtype=float)
precloses = np.concatenate([[closes_arr[0]], closes_arr[:-1]])
for i, date in enumerate(dates):
preclose = float(precloses[i])
close = float(closes_arr[i])
open_price = preclose * (1.0 + 0.25 * (close / preclose - 1.0))
high = max(open_price, close) * 1.01
low = min(open_price, close) * 0.99
volume = base_volume[sym] + 10_000.0 * i
tradestatus = 1
is_st = 0
if sym == "sh600000" and i == 4:
open_price = preclose * 1.10
close = open_price
high = open_price
low = open_price
if sym == "sh600000" and i == 7:
volume = 0.0
if sym == "sz000001" and i == 5:
open_price = preclose
close = preclose
high = preclose
low = preclose
volume = 0.0
tradestatus = 0
if sym == "sh600519" and i == 7:
is_st = 1
if sym == "sz300750" and i == 8:
open_price = preclose * 0.80
close = open_price
high = open_price
low = open_price
amount = volume * ((open_price + close) / 2.0)
vwap = amount / volume if volume > 0 else np.nan
pct_chg = (close / preclose - 1.0) * 100.0 if preclose else 0.0
rows.append({
"symbol_id": sym,
"symbol_name": GENERATED_SYMBOL_NAMES[sym],
"date": date,
"open": open_price,
"high": high,
"low": low,
"close": close,
"preclose": preclose,
"volume": volume,
"amount": amount,
"vwap": vwap,
"turn": volume / 1_000_000.0,
"pctChg": pct_chg,
"tradestatus": tradestatus,
"isST": is_st,
"peTTM": 8.0 + i,
"pbMRQ": 1.0 + 0.05 * i,
"psTTM": 2.0 + 0.03 * i,
"pcfNcfTTM": 5.0 + 0.1 * i,
})
result = pd.DataFrame(rows)
if include_missing and n_sessions > 6:
missing_mask = (
(result["symbol_id"] == "sz300750")
& (result["date"] == dates[6])
)
result = result.loc[~missing_mask].copy()
result = result[DATA_COLUMNS]
return result.sort_values(["date", "symbol_id"]).reset_index(drop=True)
def make_generated_minute_bars(
daily: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Expand generated daily bars into a tiny deterministic intraday panel."""
daily = make_generated_daily_bars() if daily is None else daily.copy()
rows: list[dict[str, object]] = []
bar_times = ("09:35:00", "10:30:00", "14:55:00")
for daily_row in daily.sort_values(["date", "symbol_id"]).itertuples(index=False):
if int(getattr(daily_row, "tradestatus", 1)) == 0:
continue
volume = float(daily_row.volume)
volume_slices = [0.25 * volume, 0.35 * volume, 0.40 * volume]
prices = np.linspace(float(daily_row.open), float(daily_row.close), len(bar_times))
for j, time_text in enumerate(bar_times):
dt = pd.Timestamp(daily_row.date) + pd.Timedelta(time_text)
open_price = prices[j - 1] if j else float(daily_row.open)
close_price = float(prices[j])
high = max(open_price, close_price) * 1.002
low = min(open_price, close_price) * 0.998
minute_volume = float(volume_slices[j])
amount = minute_volume * ((open_price + close_price) / 2.0)
rows.append({
"symbol_id": daily_row.symbol_id,
"symbol_name": daily_row.symbol_name,
"datetime": dt,
"date": pd.Timestamp(daily_row.date).normalize(),
"time": time_text,
"frequency": "5m",
"open": open_price,
"high": high,
"low": low,
"close": close_price,
"volume": minute_volume,
"amount": amount,
"vwap": amount / minute_volume if minute_volume > 0 else np.nan,
"adjustflag": "3",
})
return pd.DataFrame(rows, columns=MINUTE_BAR_COLUMNS)
def make_generated_derived_features(
daily: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Return numeric daily derived values, including NaN and infinity cells."""
daily = make_generated_daily_bars() if daily is None else daily.copy()
keys = (
daily[["symbol_id", "date"]]
.drop_duplicates()
.sort_values(["date", "symbol_id"])
.reset_index(drop=True)
)
date_rank = keys["date"].rank(method="dense").astype(float)
symbol_rank = keys["symbol_id"].map({
"sh600000": 1.0,
"sz000001": 2.0,
"sh600519": 3.0,
"sz300750": 4.0,
})
out = keys.copy()
out["toy_feature"] = symbol_rank + date_rank / 100.0
out["finite_feature"] = symbol_rank * date_rank
out["nan_feature"] = out["toy_feature"]
out["inf_feature"] = out["toy_feature"]
if len(out) >= 2:
out.loc[out.index[0], "nan_feature"] = np.nan
out.loc[out.index[1], "inf_feature"] = np.inf
return out
def make_generated_alpha_weights(
alpha_name: str = "alpha_a",
*,
scale: float = 1.0,
offset: float = 0.0,
zero_date_index: int | None = None,
n_sessions: int = 10,
) -> pd.DataFrame:
"""Create a deterministic long alpha grid with optional zero-gross date."""
dates = generated_sessions(n_sessions)
even = np.array([1.20, -0.80, 0.40, -0.80], dtype=float)
odd = np.array([-0.60, 1.10, -0.90, 0.40], dtype=float)
rows: list[dict[str, object]] = []
for i, date in enumerate(dates):
vector = even.copy() if i % 2 == 0 else odd.copy()
vector = vector + offset
vector = vector - vector.mean()
if zero_date_index is not None and i == zero_date_index:
vector = np.zeros_like(vector)
for sym, weight in zip(GENERATED_SYMBOLS, scale * vector):
rows.append({
"symbol_id": sym,
"date": date,
"alpha_name": alpha_name,
"weight": float(weight),
})
result = pd.DataFrame(rows, columns=ALPHA_COLUMNS)
return result.sort_values(["symbol_id", "date"]).reset_index(drop=True)
def make_generated_combo_weights(
combo_name: str = "combo",
*,
zero_date_index: int | None = 2,
n_sessions: int = 10,
) -> pd.DataFrame:
"""Create deterministic combo weights for portfolio construction tests."""
alpha = make_generated_alpha_weights(
"combo_source",
zero_date_index=zero_date_index,
n_sessions=n_sessions,
)
combo = alpha.rename(columns={"alpha_name": "combo_name"}).copy()
combo["combo_name"] = combo_name
return combo[COMBO_COLUMNS].sort_values(["symbol_id", "date"]).reset_index(drop=True)
+234
View File
@@ -191,6 +191,15 @@ def test_combine_single_alpha_is_identity(tmp_path):
assert (combo["combo_name"] == "rev_combo").all() assert (combo["combo_name"] == "rev_combo").all()
def test_combine_alphas_rejects_unknown_method(tmp_path):
data = _make_data()
alpha_path = tmp_path / "alpha.pq"
compute_alpha(data, "rev", "reversal", lookback=5).to_parquet(alpha_path, index=False)
with pytest.raises(ValueError, match="Unknown combo method"):
combine_alphas([str(alpha_path)], "bad_combo", method="does_not_exist")
# --- registry / factory ----------------------------------------------------- # --- registry / factory -----------------------------------------------------
def test_builtins_are_registered(): def test_builtins_are_registered():
@@ -205,6 +214,22 @@ def test_get_alpha_filters_unaccepted_params():
assert not hasattr(alpha, "vol_window") assert not hasattr(alpha, "vol_window")
def test_get_alpha_forwards_kwargs_to_flexible_alpha():
@register_alpha
class _FlexibleAlpha(BaseAlpha):
name = "_flexible_alpha_kwargs"
def __init__(self, **kwargs):
self.kwargs = kwargs
def signal(self, close):
return close
alpha = get_alpha("_flexible_alpha_kwargs", decay=0.5, label="demo")
assert alpha.kwargs == {"decay": 0.5, "label": "demo"}
def test_get_alpha_unknown_raises(): def test_get_alpha_unknown_raises():
with pytest.raises(KeyError): with pytest.raises(KeyError):
get_alpha("does_not_exist") get_alpha("does_not_exist")
@@ -227,6 +252,31 @@ def test_register_rejects_non_basealpha():
register_alpha(object) # type: ignore[arg-type] register_alpha(object) # type: ignore[arg-type]
def test_register_rejects_empty_alpha_name():
with pytest.raises(ValueError, match="non-empty"):
@register_alpha
class NoNameAlpha(BaseAlpha):
def signal(self, close):
return close
def test_load_alpha_module_error_paths(tmp_path, monkeypatch):
missing_path = tmp_path / "missing_alpha.py"
with pytest.raises(FileNotFoundError):
load_alpha_module(str(missing_path))
bad_path = tmp_path / "bad_alpha.py"
bad_path.write_text("x = 1\n")
monkeypatch.setattr(
"pipeline.alpha.registry.importlib.util.spec_from_file_location",
lambda *args, **kwargs: None,
)
with pytest.raises(ImportError, match="Cannot load alpha module"):
load_alpha_module(str(bad_path))
load_alpha_module("math")
# --- base class -------------------------------------------------------------- # --- base class --------------------------------------------------------------
def test_to_weights_are_per_date_zscore(): def test_to_weights_are_per_date_zscore():
@@ -242,6 +292,20 @@ def test_to_weights_are_per_date_zscore():
assert (weights.mean(axis=1).abs() < 1e-9).all() assert (weights.mean(axis=1).abs() < 1e-9).all()
def test_base_alpha_default_signal_and_repr():
alpha = BaseAlpha()
alpha.example = 3
with pytest.raises(NotImplementedError, match="signal"):
alpha.signal(pd.DataFrame({"x": [1.0]}))
with pytest.raises(NotImplementedError, match="signal"):
alpha.signal_from_data(
pd.DataFrame({"symbol_id": ["sh600000"]}),
pd.DataFrame({"sh600000": [1.0]}),
)
assert repr(alpha) == "BaseAlpha(example=3)"
# --- external plugin loading ------------------------------------------------- # --- external plugin loading -------------------------------------------------
def test_load_external_alpha_module(tmp_path): def test_load_external_alpha_module(tmp_path):
@@ -347,3 +411,173 @@ def test_universe_filter_does_not_corrupt_signal_history():
held = set(filtered.loc[filtered["weight"] != 0.0, "symbol_id"].unique()) held = set(filtered.loc[filtered["weight"] != 0.0, "symbol_id"].unique())
# The two most liquid names (highest amount) are sh600519, sz300750. # The two most liquid names (highest amount) are sh600519, sz300750.
assert held == {"sh600519", "sz300750"} assert held == {"sh600519", "sz300750"}
# --- feature-aware alpha integration ----------------------------------------
def test_compute_alpha_without_feature_path_matches_empty_feature_paths():
data = _make_data()
base = compute_alpha(data, "rev5", "reversal", lookback=5)
with_empty_features = compute_alpha(
data,
"rev5",
"reversal",
lookback=5,
feature_paths=[],
)
pd.testing.assert_frame_equal(base, with_empty_features)
def test_feature_aware_alpha_reads_joined_feature_column(tmp_path):
module_path = tmp_path / "feature_aware_alpha.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class FeatureAwareAlpha(BaseAlpha):
name = "feature_aware_test_alpha"
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
signal = data.pivot_table(
index="date",
columns="symbol_id",
values="toy_feature",
aggfunc="first",
)
return signal.reindex(index=close.index, columns=close.columns)
'''))
data = _make_data()
feature = data[["symbol_id", "date"]].copy()
feature["toy_feature"] = feature["symbol_id"].map({
"sh600000": 1.0,
"sz000001": 2.0,
"sh600519": 3.0,
})
feature_path = tmp_path / "toy_feature.pq"
feature.to_parquet(feature_path, index=False)
load_alpha_module(str(module_path))
result = compute_alpha(
data,
"feature_run",
"feature_aware_test_alpha",
feature_paths=[str(feature_path)],
)
assert list(result.columns) == ALPHA_COLUMNS
assert (result["alpha_name"] == "feature_run").all()
last = result[result["date"] == result["date"].max()]
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
def test_feature_paths_join_multiple_files_and_normalize_dates(tmp_path):
module_path = tmp_path / "multi_feature_alpha.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class MultiFeatureAlpha(BaseAlpha):
name = "multi_feature_test_alpha"
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
data = data.copy()
data["combined_feature"] = data["toy_a"] + data["toy_b"]
signal = data.pivot_table(
index="date",
columns="symbol_id",
values="combined_feature",
aggfunc="first",
)
return signal.reindex(index=close.index, columns=close.columns)
'''))
data = _make_data(n_days=8)
symbol_score = {"sh600000": 1.0, "sz000001": 2.0, "sh600519": 3.0}
feature_a = data[["symbol_id", "date"]].copy()
feature_a["date"] = feature_a["date"] + pd.Timedelta(hours=15)
feature_a["toy_a"] = feature_a["symbol_id"].map(symbol_score)
feature_b = data[["symbol_id", "date"]].copy()
feature_b["date"] = feature_b["date"].dt.strftime("%Y-%m-%d 09:30:00")
feature_b["toy_b"] = feature_b["symbol_id"].map(symbol_score) * 10.0
feature_a_path = tmp_path / "toy_a.pq"
feature_b_path = tmp_path / "toy_b.pq"
feature_a.to_parquet(feature_a_path, index=False)
feature_b.to_parquet(feature_b_path, index=False)
load_alpha_module(str(module_path))
result = compute_alpha(
data,
"multi_feature_run",
"multi_feature_test_alpha",
feature_paths=[str(feature_a_path), str(feature_b_path)],
)
assert list(result.columns) == ALPHA_COLUMNS
assert (result["alpha_name"] == "multi_feature_run").all()
last = result[result["date"] == result["date"].max()]
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
def test_compute_alpha_rejects_duplicate_feature_frame_columns():
data = _make_data()
duplicate_columns = pd.DataFrame(
[["sh600000", pd.Timestamp("2024-01-01"), 1.0, 2.0]],
columns=["symbol_id", "date", "toy_feature", "toy_feature"],
)
with pytest.raises(ValueError, match="duplicate columns"):
compute_alpha(
data,
"bad_features",
"reversal",
feature_frames=[duplicate_columns],
)
def test_compute_alpha_rejects_feature_path_collision_with_daily_data(tmp_path):
data = _make_data()
close_collision = data[["symbol_id", "date"]].copy()
close_collision["close"] = 1.0
close_collision_path = tmp_path / "close_collision.pq"
close_collision.to_parquet(close_collision_path, index=False)
with pytest.raises(ValueError, match="conflict"):
compute_alpha(
data,
"close_collision",
"reversal",
feature_paths=[str(close_collision_path)],
)
def test_evaluate_alpha_empty_when_signal_dates_not_on_market_calendar():
data = _make_data(n_days=3)
alpha = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2030-01-01")],
"alpha_name": ["future"],
"weight": [1.0],
})
metrics = evaluate_alpha(alpha, data)
assert metrics["n_dates"] == 0
assert metrics["cumulative_return"] == 0.0
+771
View File
@@ -0,0 +1,771 @@
"""CLI handoff tests for the offline daily workflow."""
from __future__ import annotations
import textwrap
from pathlib import Path
import click
import pandas as pd
from click.testing import CliRunner
from cli import cli
import pipeline.derived.cli as derived_cli
import pipeline.features.cli as features_cli
from tests.helpers import (
make_generated_daily_bars,
make_generated_derived_features,
make_generated_minute_bars,
)
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
def _invoke_ok(runner: CliRunner, args: list[str]):
result = runner.invoke(cli, args)
assert result.exit_code == 0, result.output
return result
def _invoke_error(runner: CliRunner, args: list[str]):
result = runner.invoke(cli, args)
assert result.exit_code != 0, result.output
return result
def test_cli_daily_workflow_handoffs_stay_in_tmp_path(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars()
minute_bars = make_generated_minute_bars(daily_bars)
derived_features = make_generated_derived_features(daily_bars)
daily_path = tmp_path / "daily_bars.pq"
minute_path = tmp_path / "minute_bars.pq"
derived_input_path = tmp_path / "derived_input.pq"
daily_bars.to_parquet(daily_path, index=False)
minute_bars.to_parquet(minute_path, index=False)
derived_features.to_parquet(derived_input_path, index=False)
ingest_dir = tmp_path / "derived_ingested"
ingest_result = _invoke_ok(runner, [
"derived", "ingest",
"--input-path", str(derived_input_path),
"--derived-name", "toy_features",
"--output-dir", str(ingest_dir),
])
ingested_feature_path = ingest_dir / "toy_features.pq"
assert "Saved derived data:" in ingest_result.output
assert ingested_feature_path.exists()
validate_result = _invoke_ok(runner, [
"derived", "validate",
"--input-path", str(ingested_feature_path),
])
assert "Valid derived data:" in validate_result.output
assert "rows" in validate_result.output
computed_derived_dir = tmp_path / "derived_computed"
derived_compute_result = _invoke_ok(runner, [
"derived", "compute",
"--daily-path", str(daily_path),
"--minute-path", str(minute_path),
"--derived-type", "minute_daily_summary",
"--derived-name", "minute_summary",
"--output-dir", str(computed_derived_dir),
])
minute_summary_path = computed_derived_dir / "minute_summary.pq"
assert "Loaded daily data:" in derived_compute_result.output
assert "Loaded minute bars:" in derived_compute_result.output
assert "Saved derived data:" in derived_compute_result.output
assert minute_summary_path.exists()
assert "minute_vwap" in pd.read_parquet(minute_summary_path).columns
alpha_module_path = tmp_path / "cli_feature_alpha.py"
alpha_module_path.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class CliFeatureAlpha(BaseAlpha):
name = "cli_feature_alpha_workflow"
def __init__(self, **kwargs):
self.kwargs = kwargs
def signal_from_data(
self,
data: pd.DataFrame,
close: pd.DataFrame,
) -> pd.DataFrame:
signal = data.pivot_table(
index="date",
columns="symbol_id",
values="minute_intraday_return",
aggfunc="first",
)
fallback = close.pct_change(1, fill_method=None)
feature_signal = signal.reindex(index=close.index, columns=close.columns)
toy_signal = data.pivot_table(
index="date",
columns="symbol_id",
values="toy_feature",
aggfunc="first",
)
toy_signal = toy_signal.reindex(index=close.index, columns=close.columns)
return feature_signal.fillna(fallback) + toy_signal / 1000.0
"""))
alpha_dir = tmp_path / "alphas"
alpha_result = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--feature-path", str(minute_summary_path),
"--feature-path", str(ingested_feature_path),
"--alpha-module", str(alpha_module_path),
"--alpha-type", "cli_feature_alpha_workflow",
"--alpha-name", "cli_feature_alpha",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "cli_feature_alpha.pq"
assert "Loaded data:" in alpha_result.output
assert "Saved alpha:" in alpha_result.output
assert "Weight stats" in alpha_result.output
assert alpha_path.exists()
assert not pd.read_parquet(alpha_path).empty
alpha_report_dir = tmp_path / "alpha_reports"
alpha_eval_result = _invoke_ok(runner, [
"alpha", "eval",
"--alpha-path", str(alpha_path),
"--data-path", str(daily_path),
"--report-dir", str(alpha_report_dir),
])
alpha_report_path = alpha_report_dir / "cli_feature_alpha_eval.json"
assert "ALPHA EVALUATION" in alpha_eval_result.output
assert "Report saved:" in alpha_eval_result.output
assert alpha_report_path.exists()
combo_dir = tmp_path / "combos"
combo_result = _invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", f"{alpha_path},{alpha_path}",
"--combo-name", "cli_combo",
"--method", "equal_weight",
"--output-dir", str(combo_dir),
])
combo_path = combo_dir / "cli_combo.pq"
assert "Saved combo:" in combo_result.output
assert "Weight stats" in combo_result.output
assert combo_path.exists()
portfolio_dir = tmp_path / "portfolio"
build_result = _invoke_ok(runner, [
"portfolio", "build",
"--weights-path", str(combo_path),
"--data-path", str(daily_path),
"--booksize", "2000000",
"--portfolio-name", "cli_portfolio",
"--output-dir", str(portfolio_dir),
])
positions_path = portfolio_dir / "cli_portfolio.pq"
assert "Saved positions:" in build_result.output
assert "Gross exposure" in build_result.output
assert positions_path.exists()
execution_dir = tmp_path / "execution"
simulate_result = _invoke_ok(runner, [
"portfolio", "simulate",
"--positions-path", str(positions_path),
"--data-path", str(daily_path),
"--constraint", "suspension",
"--constraint", "price_limit",
"--constraint", "volume_cap",
"--cost-bps", "5",
"--slippage-bps", "5",
"--volume-frac", "0.02",
"--output-dir", str(execution_dir),
])
fills_path = execution_dir / "fills" / "cli_portfolio.pq"
pnl_path = execution_dir / "pnl" / "cli_portfolio.pq"
assert "Saved fills:" in simulate_result.output
assert "Saved pnl:" in simulate_result.output
assert "Total PnL:" in simulate_result.output
assert fills_path.exists()
assert pnl_path.exists()
eval_result = _invoke_ok(runner, [
"portfolio", "eval",
"--positions-path", str(positions_path),
"--data-path", str(daily_path),
])
assert "Research-portfolio metrics:" in eval_result.output
assert "cumulative_return" in eval_result.output
assert "fitness" in eval_result.output
pqcat_result = _invoke_ok(runner, [
"pqcat",
str(positions_path),
"--info",
])
assert "shape:" in pqcat_result.output
assert "dtypes:" in pqcat_result.output
assert "position_shares" in pqcat_result.output
alphaview_result = _invoke_ok(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(alpha_path),
"--symbol", "sh600000",
"--start-date", "2024-01-02",
"--end-date", "2024-01-12",
"--columns", "close,volume",
])
assert "symbol: sh600000" in alphaview_result.output
assert "cli_feature_alpha" in alphaview_result.output
def test_cli_pipeline_accepts_partitioned_daily_dataset(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(include_missing=False)
dataset_dir = tmp_path / "daily_dataset"
dataset_frame = daily_bars.copy()
dataset_frame["month"] = dataset_frame["date"].dt.strftime("%Y-%m")
dataset_frame.to_parquet(dataset_dir, partition_cols=["month"], index=False)
alpha_dir = tmp_path / "alphas"
alpha_result = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(dataset_dir),
"--alpha-type", "reversal",
"--alpha-name", "dataset_reversal",
"--lookback", "3",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "dataset_reversal.pq"
assert "Loaded data:" in alpha_result.output
assert alpha_path.exists()
combo_dir = tmp_path / "combos"
_invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", str(alpha_path),
"--combo-name", "dataset_combo",
"--output-dir", str(combo_dir),
])
combo_path = combo_dir / "dataset_combo.pq"
assert combo_path.exists()
portfolio_dir = tmp_path / "portfolio"
_invoke_ok(runner, [
"portfolio", "build",
"--weights-path", str(combo_path),
"--data-path", str(dataset_dir),
"--booksize", "1000000",
"--portfolio-name", "dataset_portfolio",
"--output-dir", str(portfolio_dir),
])
positions_path = portfolio_dir / "dataset_portfolio.pq"
assert positions_path.exists()
execution_dir = tmp_path / "execution"
simulate_result = _invoke_ok(runner, [
"portfolio", "simulate",
"--positions-path", str(positions_path),
"--data-path", str(dataset_dir),
"--constraint", "suspension",
"--output-dir", str(execution_dir),
])
assert "Saved fills:" in simulate_result.output
assert (execution_dir / "fills" / "dataset_portfolio.pq").exists()
assert (execution_dir / "pnl" / "dataset_portfolio.pq").exists()
def test_cli_liquid_universe_masks_to_top_liquid_names(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=75, include_missing=False)
daily_path = tmp_path / "daily_bars_75d.pq"
daily_bars.to_parquet(daily_path, index=False)
alpha_dir = tmp_path / "alphas"
result = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "reversal_rank",
"--alpha-name", "liquid_rank",
"--lookback", "3",
"--liquid-universe",
"--universe-top-n", "2",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "liquid_rank.pq"
alpha = pd.read_parquet(alpha_path)
nonzero = alpha[alpha["weight"] != 0.0]
assert "Saved alpha:" in result.output
assert alpha_path.exists()
assert not nonzero.empty
assert nonzero.groupby("date")["symbol_id"].nunique().max() <= 2
def test_cli_real_fixture_round_trips_through_portfolio(tmp_path):
runner = CliRunner()
alpha_dir = tmp_path / "alphas"
_invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(FIXTURE_PATH),
"--alpha-type", "reversal_vol",
"--alpha-name", "real_cli_reversal_vol",
"--lookback", "3",
"--vol-window", "3",
"--output-dir", str(alpha_dir),
])
alpha_path = alpha_dir / "real_cli_reversal_vol.pq"
assert alpha_path.exists()
assert not pd.read_parquet(alpha_path).empty
combo_dir = tmp_path / "combos"
_invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", str(alpha_path),
"--combo-name", "real_cli_combo",
"--output-dir", str(combo_dir),
])
combo_path = combo_dir / "real_cli_combo.pq"
assert combo_path.exists()
portfolio_dir = tmp_path / "portfolio"
_invoke_ok(runner, [
"portfolio", "build",
"--weights-path", str(combo_path),
"--data-path", str(FIXTURE_PATH),
"--booksize", "1000000",
"--portfolio-name", "real_cli_portfolio",
"--output-dir", str(portfolio_dir),
])
positions_path = portfolio_dir / "real_cli_portfolio.pq"
positions = pd.read_parquet(positions_path)
assert not positions.empty
eval_result = _invoke_ok(runner, [
"portfolio", "eval",
"--positions-path", str(positions_path),
"--data-path", str(FIXTURE_PATH),
])
assert "Research-portfolio metrics:" in eval_result.output
def test_cli_error_paths_are_clear_for_bad_user_inputs(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars()
daily_path = tmp_path / "daily_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
unknown_alpha = _invoke_error(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "does_not_exist",
"--alpha-name", "bad",
"--output-dir", str(tmp_path / "alphas"),
])
assert "Unknown alpha-type" in unknown_alpha.output
malformed_param = _invoke_error(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "reversal",
"--alpha-name", "bad_param",
"--param", "not-an-assignment",
"--output-dir", str(tmp_path / "alphas"),
])
assert "--param must be name=value" in malformed_param.output
unknown_derived = _invoke_error(runner, [
"derived", "compute",
"--daily-path", str(daily_path),
"--derived-type", "does_not_exist",
"--derived-name", "bad",
"--output-dir", str(tmp_path / "derived"),
])
assert "Unknown derived-type" in unknown_derived.output
bad_constraint_positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"portfolio_name": ["bad_constraint"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [100.0],
"position_shares": [100],
"position_value": [1000.0],
"price": [10.0],
})
positions_path = tmp_path / "positions.pq"
bad_constraint_positions.to_parquet(positions_path, index=False)
unknown_constraint = _invoke_error(runner, [
"portfolio", "simulate",
"--positions-path", str(positions_path),
"--data-path", str(daily_path),
"--constraint", "not_a_constraint",
"--output-dir", str(tmp_path / "execution"),
])
assert isinstance(unknown_constraint.exception, KeyError)
assert "not_a_constraint" in str(unknown_constraint.exception)
pqcat_missing_column = _invoke_error(runner, [
"pqcat",
str(daily_path),
"--columns", "close,not_a_column",
])
assert "Columns not found: not_a_column" in pqcat_missing_column.output
alphaview_missing_symbol = _invoke_error(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(positions_path),
"--symbol", "sh999999",
])
assert "Symbol 'sh999999' not found" in alphaview_missing_symbol.output
alphaview_missing_column = _invoke_error(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(positions_path),
"--symbol", "sh600000",
"--columns", "close,missing_bar_col",
])
assert "Bar columns not found: missing_bar_col" in alphaview_missing_column.output
alphaview_alpha = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [daily_bars["date"].min()],
"alpha_name": ["toy_alpha"],
"weight": [1.0],
})
alphaview_alpha_path = tmp_path / "alphaview_alpha.pq"
alphaview_alpha.to_parquet(alphaview_alpha_path, index=False)
alphaview_empty_range = _invoke_error(runner, [
"alphaview",
"--data-path", str(daily_path),
"--alpha-path", str(alphaview_alpha_path),
"--symbol", "sh600000",
"--start-date", "2030-01-01",
])
assert "No rows in the requested date range" in alphaview_empty_range.output
empty_combo_paths = _invoke_ok(runner, [
"combo", "combine",
"--alpha-paths", " , ",
"--combo-name", "empty",
"--output-dir", str(tmp_path / "combos"),
])
assert "requires at least 1 path" in empty_combo_paths.output
def test_cli_parser_helpers_cover_string_coercion_and_bad_params():
assert derived_cli._parse_params(("n=7", "scale=2.5", "label=demo")) == {
"n": 7,
"scale": 2.5,
"label": "demo",
}
assert features_cli._parse_params(("n=7", "scale=2.5", "label=demo")) == {
"n": 7,
"scale": 2.5,
"label": "demo",
}
for module in (derived_cli, features_cli):
try:
module._parse_params(("not-an-assignment",))
except click.BadParameter as exc:
assert "--param must be name=value" in str(exc)
else:
raise AssertionError("expected BadParameter")
def test_cli_list_and_legacy_feature_paths(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
minute_bars = make_generated_minute_bars(daily_bars)
daily_path = tmp_path / "daily_bars.pq"
minute_path = tmp_path / "minute_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
minute_bars.to_parquet(minute_path, index=False)
derived_list = _invoke_ok(runner, ["derived", "list"])
feature_list = _invoke_ok(runner, ["feature", "list"])
assert "minute_daily_summary" in derived_list.output
assert "minute_daily_summary" in feature_list.output
feature_dir = tmp_path / "features"
feature_compute = _invoke_ok(runner, [
"feature",
"compute",
"--minute-path",
str(minute_path),
"--daily-path",
str(daily_path),
"--feature-type",
"minute_daily_summary",
"--feature-name",
"legacy_summary",
"--output-dir",
str(feature_dir),
])
feature_path = feature_dir / "legacy_summary.pq"
assert "Loaded minute bars:" in feature_compute.output
assert "Loaded daily data:" in feature_compute.output
assert "Saved feature:" in feature_compute.output
assert feature_path.exists()
no_input = _invoke_error(runner, [
"derived",
"compute",
"--derived-type",
"minute_daily_summary",
"--derived-name",
"missing_inputs",
"--output-dir",
str(tmp_path / "derived"),
])
assert "At least one of --daily-path or --minute-path is required" in no_input.output
missing_minute = _invoke_error(runner, [
"derived",
"compute",
"--daily-path",
str(daily_path),
"--derived-type",
"minute_daily_summary",
"--derived-name",
"daily_only",
"--output-dir",
str(tmp_path / "derived"),
])
assert "minute_daily_summary requires minute input" in missing_minute.output
malformed_feature_param = _invoke_error(runner, [
"feature",
"compute",
"--minute-path",
str(minute_path),
"--feature-type",
"minute_daily_summary",
"--feature-name",
"bad_param",
"--param",
"not-an-assignment",
"--output-dir",
str(tmp_path / "features_bad"),
])
assert "--param must be name=value" in malformed_feature_param.output
unknown_feature = _invoke_error(runner, [
"feature",
"compute",
"--minute-path",
str(minute_path),
"--feature-type",
"does_not_exist",
"--feature-name",
"bad_feature",
"--output-dir",
str(tmp_path / "features_unknown"),
])
assert "Unknown feature-type" in unknown_feature.output
def test_cli_shortcuts_and_external_module_loading(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=8, include_missing=False)
minute_bars = make_generated_minute_bars(daily_bars)
daily_path = tmp_path / "daily_bars.pq"
minute_path = tmp_path / "minute_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
minute_bars.to_parquet(minute_path, index=False)
alpha_module = tmp_path / "listed_alpha.py"
alpha_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.alpha.base import BaseAlpha
from pipeline.alpha.registry import register_alpha
@register_alpha
class ListedAlpha(BaseAlpha):
name = "listed_alpha_cli"
def __init__(self, scale: float = 1.0, label: str = "x"):
self.scale = scale
self.label = label
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
return close.pct_change(1, fill_method=None) * self.scale
"""))
derived_module = tmp_path / "listed_derived.py"
derived_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class ListedDerived(BaseDerivedData):
name = "listed_derived_cli"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = daily[["symbol_id", "date", "close"]].copy()
out["listed_value"] = out.pop("close") * self.scale
return out
"""))
derived_compute_module = tmp_path / "computed_derived.py"
derived_compute_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class ComputedDerived(BaseDerivedData):
name = "computed_derived_cli"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = daily[["symbol_id", "date", "close"]].copy()
out["computed_value"] = out.pop("close") * self.scale
return out
"""))
feature_module = tmp_path / "listed_feature.py"
feature_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.features.base import BaseFeature
from pipeline.features.registry import register_feature
@register_feature
class ListedFeature(BaseFeature):
name = "listed_feature_cli"
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = minute[["symbol_id", "date", "close"]].copy()
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
out = out.groupby(["symbol_id", "date"], as_index=False)["close"].mean()
return out.rename(columns={"close": "listed_feature_value"})
"""))
feature_compute_module = tmp_path / "computed_feature.py"
feature_compute_module.write_text(textwrap.dedent("""
import pandas as pd
from pipeline.features.base import BaseFeature
from pipeline.features.registry import register_feature
@register_feature
class ComputedFeature(BaseFeature):
name = "computed_feature_cli"
def compute(self, daily=None, minute=None) -> pd.DataFrame:
out = minute[["symbol_id", "date", "close"]].copy()
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
out = out.groupby(["symbol_id", "date"], as_index=False)["close"].mean()
return out.rename(columns={"close": "computed_feature_value"})
"""))
alpha_list = _invoke_ok(runner, [
"alpha", "list",
"--alpha-module", str(alpha_module),
])
assert "listed_alpha_cli" in alpha_list.output
alpha_dir = tmp_path / "alphas"
reversal = _invoke_ok(runner, [
"alpha", "reversal",
"--data-path", str(daily_path),
"--output-dir", str(alpha_dir),
"--lookback", "3",
])
reversal_vol = _invoke_ok(runner, [
"alpha", "reversal-vol",
"--data-path", str(daily_path),
"--output-dir", str(alpha_dir),
"--lookback", "3",
"--vol-window", "3",
])
external_alpha = _invoke_ok(runner, [
"alpha", "compute",
"--data-path", str(daily_path),
"--alpha-type", "listed_alpha_cli",
"--alpha-name", "listed_alpha_run",
"--param", "scale=2.5",
"--param", "label=demo",
"--output-dir", str(alpha_dir),
])
assert "Saved alpha:" in reversal.output
assert "Saved alpha:" in reversal_vol.output
assert "Saved alpha:" in external_alpha.output
assert (alpha_dir / "reversal_3d.pq").exists()
assert (alpha_dir / "reversal_vol_3d_3d.pq").exists()
assert (alpha_dir / "listed_alpha_run.pq").exists()
derived_list = _invoke_ok(runner, [
"derived", "list",
"--derived-module", str(derived_module),
])
assert "listed_derived_cli" in derived_list.output
derived_dir = tmp_path / "derived_external"
derived_compute = _invoke_ok(runner, [
"derived", "compute",
"--daily-path", str(daily_path),
"--derived-module", str(derived_compute_module),
"--derived-type", "computed_derived_cli",
"--derived-name", "listed_derived_run",
"--param", "scale=3",
"--output-dir", str(derived_dir),
])
assert "Saved derived data:" in derived_compute.output
assert (derived_dir / "listed_derived_run.pq").exists()
feature_list = _invoke_ok(runner, [
"feature", "list",
"--feature-module", str(feature_module),
])
assert "listed_feature_cli" in feature_list.output
feature_dir = tmp_path / "features_external"
feature_compute = _invoke_ok(runner, [
"feature", "compute",
"--minute-path", str(minute_path),
"--feature-module", str(feature_compute_module),
"--feature-type", "computed_feature_cli",
"--feature-name", "listed_feature_run",
"--output-dir", str(feature_dir),
])
assert "Saved feature:" in feature_compute.output
assert (feature_dir / "listed_feature_run.pq").exists()
def test_cli_pqcat_row_modes(tmp_path):
runner = CliRunner()
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
daily_path = tmp_path / "daily_bars.pq"
daily_bars.to_parquet(daily_path, index=False)
head_result = _invoke_ok(runner, [
"pqcat",
str(daily_path),
"--head",
"2",
"--columns",
"symbol_id,close",
])
tail_result = _invoke_ok(runner, [
"pqcat",
str(daily_path),
"--tail",
"1",
])
assert "symbol_id" in head_result.output
assert "close" in head_result.output
assert "date" in tail_result.output
+148
View File
@@ -0,0 +1,148 @@
"""Offline coverage for data CLI and universe resolution glue."""
from __future__ import annotations
import pandas as pd
from click.testing import CliRunner
from cli import cli
import pipeline.data.cli as data_cli
import pipeline.data.downloader as pipeline_downloader
def test_resolve_universe_handles_named_file_all_and_symbol_list(tmp_path, monkeypatch):
hs300_raw = pd.DataFrame({
"updateDate": ["2024-01-12", "2024-01-12"],
"stockName": ["浦发银行", "平安银行"],
"stockCode": ["sh.600000", "sz.000001"],
})
zz500_raw = pd.DataFrame({
"name": ["东风汽车"],
"code": ["sh.600006"],
"date": ["2024-01-12"],
})
monkeypatch.setattr(pipeline_downloader, "get_hs300_stocks", lambda: hs300_raw)
monkeypatch.setattr(pipeline_downloader, "get_zz500_stocks", lambda: zz500_raw)
monkeypatch.setattr(
pipeline_downloader,
"get_all_stocks",
lambda: pd.DataFrame({
"code": ["sh600000", "sz000001", "sh600519"],
"name": ["浦发银行", "平安银行", "贵州茅台"],
}),
)
symbol_file = tmp_path / "symbols.txt"
symbol_file.write_text("sh600000\n\nsz000001\n")
hs300 = pipeline_downloader._resolve_universe("hs300")
zz500 = pipeline_downloader._resolve_universe("csi500")
all_capped = pipeline_downloader._resolve_universe("all", max_symbols=2)
from_file = pipeline_downloader._resolve_universe(str(symbol_file))
from_list = pipeline_downloader._resolve_universe("sh600000, sz000001")
assert hs300.to_dict("list") == {
"symbol_name": ["浦发银行", "平安银行"],
"symbol_id": ["sh600000", "sz000001"],
}
assert zz500.to_dict("list") == {
"symbol_name": ["东风汽车"],
"symbol_id": ["sh600006"],
}
assert all_capped["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert from_file["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert from_file["symbol_name"].tolist() == ["sh600000", "sz000001"]
assert from_list["symbol_id"].tolist() == ["sh600000", "sz000001"]
def test_data_cli_download_commands_print_summaries_without_network(monkeypatch, tmp_path):
runner = CliRunner()
daily_calls: list[dict] = []
minute_calls: list[dict] = []
def fake_daily(**kwargs):
daily_calls.append(kwargs)
return {
"dataset_path": str(tmp_path / "daily" / kwargs["universe"]),
"n_symbols": 2,
"n_requested": 3,
"n_rows": 18,
"date_min": "2024-01-02",
"date_max": "2024-01-12",
}
def fake_minute(**kwargs):
minute_calls.append(kwargs)
return {
"dataset_path": str(tmp_path / "minute" / kwargs["universe"]),
"frequency": "15m",
"n_symbols": 1,
"n_requested": 1,
"n_rows": 32,
"date_min": "2024-01-02",
"date_max": "2024-01-03",
}
monkeypatch.setattr(data_cli, "download_universe", fake_daily)
monkeypatch.setattr(data_cli, "download_minute_universe", fake_minute)
daily_result = runner.invoke(cli, [
"data",
"download",
"--universe",
"sh600000,sz000001",
"--start-date",
"2024-01-02",
"--end-date",
"2024-01-12",
"--output-dir",
str(tmp_path / "daily"),
"--symbols",
"3",
"--chunk-size",
"2",
"--adjust",
"none",
])
minute_result = runner.invoke(cli, [
"data",
"download-minute",
"--universe",
"toy",
"--start-date",
"2024-01-02",
"--end-date",
"2024-01-03",
"--output-dir",
str(tmp_path / "minute"),
"--symbols",
"1",
"--chunk-size",
"1",
"--frequency",
"15",
])
assert daily_result.exit_code == 0, daily_result.output
assert "Summary: 2/3 symbols, 18 bars" in daily_result.output
assert daily_calls == [{
"universe": "sh600000,sz000001",
"start_date": "2024-01-02",
"end_date": "2024-01-12",
"output_dir": str(tmp_path / "daily"),
"max_symbols": 3,
"chunk_size": 2,
"adjust": "none",
}]
assert minute_result.exit_code == 0, minute_result.output
assert "frequency=15m" in minute_result.output
assert minute_calls == [{
"universe": "toy",
"start_date": "2024-01-02",
"end_date": "2024-01-03",
"output_dir": str(tmp_path / "minute"),
"max_symbols": 1,
"chunk_size": 1,
"frequency": "15",
}]
+376
View File
@@ -0,0 +1,376 @@
"""Tests for daily derived-data ingestion and plugins."""
import textwrap
import numpy as np
import pandas as pd
import pytest
from click.testing import CliRunner
from cli import cli
from pipeline.alpha.compute import join_feature_frames
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.compute import compute_derived, read_derived_frame, validate_derived_frame
from pipeline.derived.registry import (
available_derived,
get_derived,
load_derived_module,
register_derived,
)
def _daily_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
"open": [10.0, 20.0, 11.0],
"close": [10.5, 20.5, 11.5],
"volume": [1000.0, 2000.0, 1200.0],
})
def _minute_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001"],
"datetime": pd.to_datetime([
"2024-01-02 09:35:00",
"2024-01-02 09:40:00",
"2024-01-02 09:35:00",
]),
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
"time": ["09:35:00", "09:40:00", "09:35:00"],
"open": [10.0, 10.5, 20.0],
"high": [11.0, 12.0, 21.0],
"low": [9.0, 10.0, 19.0],
"close": [10.5, 11.0, 20.5],
"volume": [100.0, 300.0, 200.0],
"amount": [1000.0, 3300.0, 4100.0],
})
def test_validate_derived_frame_normalizes_and_sorts():
result = validate_derived_frame(pd.DataFrame({
"symbol_id": ["sz000001", "sh600000"],
"date": ["2024-01-02 15:00:00", "2024-01-02 09:30:00"],
"custom_value": [2.0, 1.0],
}))
assert result["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert result["date"].tolist() == [
pd.Timestamp("2024-01-02"),
pd.Timestamp("2024-01-02"),
]
def test_validate_derived_frame_rejects_missing_keys():
with pytest.raises(ValueError, match="missing required"):
validate_derived_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
def test_validate_derived_frame_rejects_duplicate_normalized_keys():
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
"x": [1.0, 2.0],
}))
def test_validate_derived_frame_rejects_duplicate_columns():
bad = pd.DataFrame(
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
columns=["symbol_id", "date", "dup", "dup"],
)
with pytest.raises(ValueError, match="duplicate columns"):
validate_derived_frame(bad)
def test_validate_derived_frame_rejects_non_numeric_values():
with pytest.raises(ValueError, match="numeric"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"bad": ["not numeric"],
}))
def test_validate_derived_frame_rejects_missing_value_columns():
with pytest.raises(ValueError, match="at least one value column"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
}))
def test_read_derived_frame_rejects_empty_csv(tmp_path):
empty_csv = tmp_path / "empty.csv"
empty_csv.write_text("")
with pytest.raises(ValueError, match="CSV input is empty"):
read_derived_frame(empty_csv)
def test_compute_derived_rejects_missing_inputs():
with pytest.raises(ValueError, match="requires --daily-path or --minute-path"):
compute_derived("minute_daily_summary")
def test_derived_ingest_cli_accepts_csv_and_parquet(tmp_path):
runner = CliRunner()
source = pd.DataFrame({
"symbol_id": ["sz000001", "sh600000"],
"date": ["2024-01-02", "2024-01-02"],
"custom_value": [2.0, 1.0],
})
csv_path = tmp_path / "custom.csv"
parquet_path = tmp_path / "custom.pq"
out_dir = tmp_path / "derived"
source.to_csv(csv_path, index=False)
source.to_parquet(parquet_path, index=False)
csv_result = runner.invoke(cli, [
"derived", "ingest",
"--input-path", str(csv_path),
"--derived-name", "csv_custom",
"--output-dir", str(out_dir),
])
assert csv_result.exit_code == 0, csv_result.output
parquet_result = runner.invoke(cli, [
"derived", "ingest",
"--input-path", str(parquet_path),
"--derived-name", "parquet_custom",
"--output-dir", str(out_dir),
])
assert parquet_result.exit_code == 0, parquet_result.output
written = pd.read_parquet(out_dir / "csv_custom.pq")
assert written["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert (out_dir / "parquet_custom.pq").exists()
def test_derived_validate_cli_rejects_duplicate_csv_columns(tmp_path):
runner = CliRunner()
csv_path = tmp_path / "bad.csv"
csv_path.write_text("symbol_id,date,x,x\nsh600000,2024-01-02,1.0,2.0\n")
result = runner.invoke(cli, [
"derived", "validate",
"--input-path", str(csv_path),
])
assert result.exit_code != 0
assert "duplicate columns" in result.output
def test_derived_ingest_cli_wraps_validation_errors(tmp_path):
runner = CliRunner()
csv_path = tmp_path / "bad_ingest.csv"
csv_path.write_text("symbol_id,date\nsh600000,2024-01-02\n")
result = runner.invoke(cli, [
"derived",
"ingest",
"--input-path",
str(csv_path),
"--derived-name",
"bad",
"--output-dir",
str(tmp_path / "derived"),
])
assert result.exit_code != 0
assert "at least one value column" in result.output
def test_external_derived_plugin_loads_filters_params_and_uses_inputs(tmp_path):
module_path = tmp_path / "external_derived.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class FlexibleDerived(BaseDerivedData):
name = "flexible_derived_test"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, daily=None, minute=None) -> pd.DataFrame:
result = None
if daily is not None:
result = daily[["symbol_id", "date", "close"]].copy()
result["daily_scaled_close"] = result.pop("close") * self.scale
if minute is not None:
minute_out = (
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
.sum()
.rename(columns={"volume": "minute_volume_sum"})
)
minute_out["minute_volume_sum"] *= self.scale
result = minute_out if result is None else result.merge(
minute_out, on=["symbol_id", "date"], how="left"
)
return result
'''))
load_derived_module(str(module_path))
assert "flexible_derived_test" in available_derived()
instance = get_derived("flexible_derived_test", scale=2.0, ignored=99)
assert instance.scale == 2.0
assert not hasattr(instance, "ignored")
daily_result = compute_derived(
"flexible_derived_test",
daily=_daily_bars(),
scale=2.0,
ignored=99,
)
assert "daily_scaled_close" in daily_result.columns
assert np.isclose(daily_result["daily_scaled_close"].iloc[0], 21.0)
minute_result = compute_derived(
"flexible_derived_test",
minute=_minute_bars(),
scale=2.0,
)
assert "minute_volume_sum" in minute_result.columns
assert np.isclose(
minute_result.loc[minute_result["symbol_id"] == "sh600000", "minute_volume_sum"].iloc[0],
800.0,
)
both_result = compute_derived(
"flexible_derived_test",
daily=_daily_bars(),
minute=_minute_bars(),
scale=1.0,
)
assert {"daily_scaled_close", "minute_volume_sum"}.issubset(both_result.columns)
def test_derived_registry_rejects_bad_plugins_and_load_failures(tmp_path, monkeypatch):
with pytest.raises(TypeError):
register_derived(object) # type: ignore[arg-type]
with pytest.raises(ValueError, match="non-empty"):
@register_derived
class NoNameDerived(BaseDerivedData):
def compute(self, daily=None, minute=None):
return pd.DataFrame()
@register_derived
class _CoverageDerived(BaseDerivedData):
name = "_coverage_derived_registry"
def compute(self, daily=None, minute=None):
return pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"x": [1.0],
})
with pytest.raises(ValueError, match="already registered"):
@register_derived
class _CoverageDerivedDuplicate(BaseDerivedData):
name = "_coverage_derived_registry"
def compute(self, daily=None, minute=None):
return pd.DataFrame()
with pytest.raises(KeyError, match="Unknown derived data"):
get_derived("does_not_exist")
missing_path = tmp_path / "missing_derived.py"
with pytest.raises(FileNotFoundError):
load_derived_module(str(missing_path))
bad_path = tmp_path / "bad_derived.py"
bad_path.write_text("x = 1\n")
monkeypatch.setattr(
"pipeline.derived.registry.importlib.util.spec_from_file_location",
lambda *args, **kwargs: None,
)
with pytest.raises(ImportError, match="Cannot load derived data module"):
load_derived_module(str(bad_path))
load_derived_module("math")
instance = _CoverageDerived()
instance.scale = 2
assert repr(instance) == "_CoverageDerived(scale=2)"
def test_derived_compute_cli_writes_builtin_minute_summary(tmp_path):
runner = CliRunner()
minute_path = tmp_path / "minute.pq"
out_dir = tmp_path / "derived"
_minute_bars().to_parquet(minute_path, index=False)
result = runner.invoke(cli, [
"derived", "compute",
"--minute-path", str(minute_path),
"--derived-type", "minute_daily_summary",
"--derived-name", "minute_summary",
"--output-dir", str(out_dir),
])
assert result.exit_code == 0, result.output
written = pd.read_parquet(out_dir / "minute_summary.pq")
assert "minute_vwap" in written.columns
def test_minute_summary_uses_time_sort_and_daily_without_close():
minute = _minute_bars().drop(columns=["datetime"]).sample(frac=1.0, random_state=0)
daily = _daily_bars()[["symbol_id", "date", "open"]]
result = compute_derived(
"minute_daily_summary",
daily=daily,
minute=minute,
)
by_symbol = result.set_index(["symbol_id", "date"])
assert np.isclose(
by_symbol.loc[("sh600000", pd.Timestamp("2024-01-02")), "minute_intraday_return"],
0.10,
)
assert "minute_vwap_deviation" in result.columns
def test_alpha_feature_join_rejects_derived_column_collisions():
data = _daily_bars()
derived_a = data[["symbol_id", "date"]].copy()
derived_a["custom_value"] = 1.0
derived_b = data[["symbol_id", "date"]].copy()
derived_b["custom_value"] = 2.0
with pytest.raises(ValueError, match="conflict"):
join_feature_frames(data, [derived_a, derived_b])
close_collision = data[["symbol_id", "date"]].copy()
close_collision["close"] = 1.0
with pytest.raises(ValueError, match="conflict"):
join_feature_frames(data, [close_collision])
def test_legacy_feature_cli_delegates_to_derived_registry(tmp_path):
runner = CliRunner()
minute_path = tmp_path / "minute.pq"
out_dir = tmp_path / "features"
_minute_bars().to_parquet(minute_path, index=False)
list_result = runner.invoke(cli, ["feature", "list"])
assert list_result.exit_code == 0, list_result.output
assert "minute_daily_summary" in list_result.output
compute_result = runner.invoke(cli, [
"feature", "compute",
"--minute-path", str(minute_path),
"--feature-type", "minute_daily_summary",
"--feature-name", "minute_summary",
"--output-dir", str(out_dir),
])
assert compute_result.exit_code == 0, compute_result.output
assert (out_dir / "minute_summary.pq").exists()
+10
View File
@@ -1,6 +1,16 @@
import os
import pytest import pytest
from data.downloader import download_daily from data.downloader import download_daily
pytestmark = [
pytest.mark.network,
pytest.mark.skipif(
os.environ.get("CEQ_RUN_LIVE_DOWNLOADER") != "1",
reason="set CEQ_RUN_LIVE_DOWNLOADER=1 to run live baostock/akshare smoke tests",
),
]
def test_download_single_stock(): def test_download_single_stock():
"""Smoke test: download data for 浦发银行 for a short window.""" """Smoke test: download data for 浦发银行 for a short window."""
+631
View File
@@ -0,0 +1,631 @@
"""Offline downloader contract tests with mocked data providers."""
from __future__ import annotations
import numpy as np
import pandas as pd
import pytest
import data.downloader as downloader
import pipeline.data.downloader as pipeline_downloader
from data.downloader import download_daily, download_daily_batch
from pipeline.common.schema import DATA_COLUMNS
from pipeline.data.downloader import download_universe
class _FakeResult:
def __init__(self, rows, error_code="0", error_msg=""):
self.rows = rows
self.error_code = error_code
self.error_msg = error_msg
self._idx = -1
def next(self):
self._idx += 1
return self._idx < len(self.rows)
def get_row_data(self):
return self.rows[self._idx]
def _daily_batch_row(
date: str = "2024-01-02",
open_: str = "10",
high: str = "11",
low: str = "9",
close: str = "10.5",
preclose: str = "10",
volume: str = "1000",
amount: str = "10500",
) -> list[str]:
return [
date,
open_,
high,
low,
close,
preclose,
volume,
amount,
"1.23",
"5.0",
"1",
"0",
"8.0",
"1.1",
"2.2",
"3.3",
]
def test_download_daily_uses_baostock_before_akshare_in_auto(monkeypatch):
calls: list[str] = []
expected = pd.DataFrame({
"symbol": ["sh600000"],
"date": ["2024-01-02"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
})
def fake_baostock(symbol, start, end, adjust):
calls.append("baostock")
return expected
def fake_akshare(symbol, start, end, adjust):
calls.append("akshare")
raise AssertionError("akshare should not be called after baostock succeeds")
monkeypatch.setattr(downloader, "_download_baostock", fake_baostock)
monkeypatch.setattr(downloader, "_download_akshare", fake_akshare)
result = download_daily("sh600000", "2024-01-02", "2024-01-02", source="auto")
assert calls == ["baostock"]
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
assert result["close"].tolist() == [10.5]
def test_download_daily_falls_back_to_akshare_when_baostock_empty(monkeypatch):
calls: list[str] = []
fallback = pd.DataFrame({
"symbol": ["sz000001"],
"date": ["2024-01-02"],
"open": [20.0],
"high": [21.0],
"low": [19.0],
"close": [20.5],
"volume": [2000.0],
"amount": [41000.0],
})
monkeypatch.setattr(
downloader,
"_download_baostock",
lambda symbol, start, end, adjust: calls.append("baostock") or None,
)
monkeypatch.setattr(
downloader,
"_download_akshare",
lambda symbol, start, end, adjust: calls.append("akshare") or fallback,
)
result = download_daily("sz000001", "2024-01-02", "2024-01-02", source="auto")
assert calls == ["baostock", "akshare"]
assert result["symbol"].tolist() == ["sz000001"]
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
def test_download_daily_raises_when_requested_source_has_no_data(monkeypatch):
monkeypatch.setattr(downloader, "_download_baostock", lambda *args: None)
with pytest.raises(RuntimeError, match="Failed to download data for sh600000"):
download_daily(
"sh600000",
"2024-01-02",
"2024-01-02",
source="baostock",
)
def test_download_daily_akshare_source_skips_baostock(monkeypatch):
calls: list[str] = []
fallback = pd.DataFrame({
"symbol": ["sh600000"],
"date": ["2024-01-02"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
})
monkeypatch.setattr(
downloader,
"_download_baostock",
lambda *args: calls.append("baostock") or None,
)
monkeypatch.setattr(
downloader,
"_download_akshare",
lambda *args: calls.append("akshare") or fallback,
)
result = download_daily(
"sh600000",
"2024-01-02",
"2024-01-02",
source="akshare",
)
assert calls == ["akshare"]
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
def test_akshare_daily_downloader_maps_columns_and_failures(monkeypatch):
calls: list[dict] = []
raw = pd.DataFrame({
"日期": ["2024-01-02"],
"开盘": [10.0],
"最高": [11.0],
"最低": [9.0],
"收盘": [10.5],
"成交量": [1000.0],
"成交额": [10500.0],
"换手率": [1.2],
})
def fake_hist(**kwargs):
calls.append(kwargs)
return raw.copy()
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", fake_hist)
result = downloader._download_akshare(
"sh600000",
"20240102",
"20240102",
adjust="",
)
assert calls == [{
"symbol": "600000",
"period": "daily",
"start_date": "20240102",
"end_date": "20240102",
"adjust": "",
}]
assert result is not None
assert result.columns.tolist() == [
"symbol", "date", "open", "high", "low", "close", "volume", "amount",
]
assert result["symbol"].tolist() == ["sh600000"]
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", lambda **kwargs: pd.DataFrame())
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
monkeypatch.setattr(
downloader.ak,
"stock_zh_a_hist",
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("provider down")),
)
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
def test_baostock_daily_downloader_maps_errors_and_logout_failures(monkeypatch):
query_calls: list[dict] = []
row = ["2024-01-02", "10", "11", "9", "10.5", "1000", "10500"]
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(
downloader.bs,
"logout",
lambda: (_ for _ in ()).throw(RuntimeError("logout failed")),
)
def fake_query(**kwargs):
query_calls.append(kwargs)
return _FakeResult([row])
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
result = downloader._download_baostock(
"sz000001",
"2024-01-02",
"2024-01-02",
adjust="none",
)
assert query_calls[0]["code"] == "sz.000001"
assert query_calls[0]["adjustflag"] == "3"
assert result is not None
assert result["symbol"].tolist() == ["sz000001"]
assert pd.api.types.is_numeric_dtype(result["close"])
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([], error_code="1", error_msg="bad symbol"),
)
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([]),
)
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("query failed")),
)
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
def test_download_daily_batch_maps_rich_schema_and_vwap(monkeypatch):
query_calls: list[dict] = []
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
def fake_query(**kwargs):
query_calls.append(kwargs)
rows = [
_daily_batch_row(volume="1000", amount="10500"),
_daily_batch_row(date="2024-01-03", volume="0", amount="0"),
]
return _FakeResult(rows)
monkeypatch.setattr(downloader.bs, "login", fake_login)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
[(symbol, frame)] = list(
download_daily_batch(
["sh600000"],
"2024-01-02",
"2024-01-03",
adjust="hfq",
)
)
assert symbol == "sh600000"
assert query_calls[0]["code"] == "sh.600000"
assert query_calls[0]["adjustflag"] == "1"
assert login_count == 1
assert logout_count == 1
assert frame is not None
assert frame.columns.tolist() == [
"symbol", "date", "open", "high", "low", "close", "preclose",
"volume", "amount", "vwap", "turn", "pctChg", "tradestatus", "isST",
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
]
assert np.isclose(frame["vwap"].iloc[0], 10.5)
assert pd.isna(frame["vwap"].iloc[1])
assert pd.api.types.is_datetime64_any_dtype(frame["date"])
assert pd.api.types.is_numeric_dtype(frame["tradestatus"])
def test_download_daily_batch_periodic_relogin_and_none_result(monkeypatch):
responses = [
_FakeResult([], error_code="1", error_msg="bad symbol"),
_FakeResult([_daily_batch_row(date="2024-01-03")]),
]
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
monkeypatch.setattr(downloader.bs, "login", fake_login)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
results = list(
download_daily_batch(
["sh600000", "sz000001"],
"2024-01-02",
"2024-01-03",
relogin_every=1,
)
)
assert results[0] == ("sh600000", None)
assert results[1][0] == "sz000001"
assert results[1][1] is not None
assert login_count == 2
assert logout_count == 2
def test_download_daily_batch_empty_rows_yields_none(monkeypatch):
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([]),
)
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_daily_batch_generic_exception_yields_none(monkeypatch):
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("query failed")),
)
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_daily_batch_relogs_and_retries_session_loss(monkeypatch):
responses = [
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
_FakeResult([_daily_batch_row()]),
]
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
monkeypatch.setattr(downloader.bs, "login", fake_login)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
[(symbol, frame)] = list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02"))
assert symbol == "sh600000"
assert frame is not None
assert len(frame) == 1
assert login_count == 2
assert logout_count == 2
def test_download_daily_batch_second_session_loss_and_logout_failure(monkeypatch):
responses = [
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
]
logout_count = 0
def fake_logout():
nonlocal logout_count
logout_count += 1
raise RuntimeError("logout failed")
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
assert logout_count == 2
def test_download_daily_batch_uses_akshare_fallback_when_enabled(monkeypatch):
fallback = pd.DataFrame({
"symbol": ["sh600000"],
"date": ["2024-01-02"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
})
monkeypatch.setattr(downloader.bs, "login", lambda: None)
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([], error_code="1", error_msg="no data"),
)
monkeypatch.setattr(
downloader,
"_download_akshare",
lambda symbol, start, end, adjust: fallback.copy(),
)
[(symbol, frame)] = list(
download_daily_batch(
["sh600000"],
"2024-01-02",
"2024-01-02",
akshare_fallback=True,
)
)
assert symbol == "sh600000"
assert frame is not None
assert frame["date"].tolist() == [pd.Timestamp("2024-01-02")]
assert frame["close"].tolist() == [10.5]
def test_download_universe_writes_daily_partitions_from_mock_batch(tmp_path, monkeypatch):
batch_frame = pd.DataFrame({
"symbol": ["sh600000", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-02-01"]),
"open": [10.0, 11.0],
"high": [11.0, 12.0],
"low": [9.0, 10.0],
"close": [10.5, 11.5],
"preclose": [10.0, 10.5],
"volume": [1000.0, 1200.0],
"amount": [10500.0, 13800.0],
"vwap": [10.5, 11.5],
"turn": [1.0, 1.1],
"pctChg": [5.0, 9.5],
"tradestatus": [1, 1],
"isST": [0, 0],
"peTTM": [8.0, 8.1],
"pbMRQ": [1.1, 1.2],
"psTTM": [2.1, 2.2],
"pcfNcfTTM": [3.1, 3.2],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000", "sz000001"],
"symbol_name": ["PF Bank", "Ping An Bank"],
}),
)
def fake_batch(symbols, start, end, adjust="qfq"):
assert symbols == ["sh600000", "sz000001"]
assert adjust == "qfq"
yield "sh600000", batch_frame
yield "sz000001", None
monkeypatch.setattr(pipeline_downloader, "download_daily_batch", fake_batch)
stale_file = tmp_path / "toy" / "month=2024-01" / "stale.pq"
stale_file.parent.mkdir(parents=True)
batch_frame.iloc[[0]][DATA_COLUMNS[2:]].to_parquet(stale_file, index=False)
stats = download_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-02-01",
output_dir=str(tmp_path),
chunk_size=1,
)
dataset_path = tmp_path / "toy"
written = pd.read_parquet(dataset_path).sort_values(["date", "symbol_id"]).reset_index(drop=True)
assert stats == {
"dataset_path": str(dataset_path),
"n_symbols": 1,
"n_requested": 2,
"n_rows": 2,
"date_min": "2024-01-02",
"date_max": "2024-02-01",
}
assert (dataset_path / "month=2024-01").exists()
assert (dataset_path / "month=2024-02").exists()
assert not stale_file.exists()
assert written[DATA_COLUMNS].columns.tolist() == DATA_COLUMNS
assert written["symbol_name"].tolist() == ["PF Bank", "PF Bank"]
def test_download_universe_raises_when_all_daily_symbols_empty(tmp_path, monkeypatch):
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000"],
"symbol_name": ["PF Bank"],
}),
)
monkeypatch.setattr(
pipeline_downloader,
"download_daily_batch",
lambda symbols, start, end, adjust="qfq": iter([("sh600000", None)]),
)
with pytest.raises(RuntimeError, match="No data downloaded"):
download_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
)
def test_download_universe_progress_branch_at_100_symbols(tmp_path, monkeypatch):
symbols = [f"sh6{i:05d}" for i in range(100)]
batch_frame = pd.DataFrame({
"symbol": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"preclose": [10.0],
"volume": [1000.0],
"amount": [10500.0],
"vwap": [10.5],
"turn": [1.0],
"pctChg": [5.0],
"tradestatus": [1],
"isST": [0],
"peTTM": [8.0],
"pbMRQ": [1.1],
"psTTM": [2.1],
"pcfNcfTTM": [3.1],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": symbols,
"symbol_name": symbols,
}),
)
def fake_batch(requested_symbols, start, end, adjust="qfq"):
assert requested_symbols == symbols
for symbol in requested_symbols:
yield symbol, batch_frame.copy()
monkeypatch.setattr(pipeline_downloader, "download_daily_batch", fake_batch)
stats = download_universe(
universe="toy100",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
chunk_size=200,
)
assert stats["n_symbols"] == 100
assert stats["n_rows"] == 100
+188
View File
@@ -0,0 +1,188 @@
"""Tests for minute-derived daily feature plugins."""
import textwrap
import numpy as np
import pandas as pd
import pytest
from pipeline.features.compute import compute_feature, validate_feature_frame
from pipeline.features.compute import read_feature_frames
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
from pipeline.features.registry import (
available_features,
get_feature,
load_feature_module,
)
from pipeline.derived.compute import compute_derived
def _minute_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001"],
"symbol_name": ["PF Bank", "PF Bank", "Ping An"],
"datetime": pd.to_datetime([
"2024-01-02 09:35:00",
"2024-01-02 09:40:00",
"2024-01-02 09:35:00",
]),
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
"time": ["09:35:00", "09:40:00", "09:35:00"],
"frequency": ["5m", "5m", "5m"],
"open": [10.0, 10.5, 20.0],
"high": [11.0, 12.0, 21.0],
"low": [9.0, 10.0, 19.0],
"close": [10.5, 11.0, 20.5],
"volume": [100.0, 300.0, 200.0],
"amount": [1000.0, 3300.0, 4100.0],
"vwap": [10.0, 11.0, 20.5],
"adjustflag": ["3", "3", "3"],
})
def test_built_in_minute_daily_summary():
daily = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
"close": [11.0, 20.5, 12.0],
})
result = compute_feature(
minute=_minute_bars(),
daily=daily,
feature_type="minute_daily_summary",
)
assert "minute_daily_summary" in available_features()
row = result[
(result["symbol_id"] == "sh600000")
& (result["date"] == pd.Timestamp("2024-01-02"))
].iloc[0]
assert row["minute_bar_count"] == 2
assert np.isclose(row["minute_intraday_return"], 11.0 / 10.0 - 1.0)
assert np.isclose(row["minute_intraday_range"], 12.0 / 9.0 - 1.0)
assert np.isclose(row["minute_vwap"], 4300.0 / 400.0)
assert np.isclose(row["minute_vwap_deviation"], (4300.0 / 400.0) / 11.0 - 1.0)
missing = result[
(result["symbol_id"] == "sh600000")
& (result["date"] == pd.Timestamp("2024-01-03"))
].iloc[0]
assert pd.isna(missing["minute_vwap"])
def test_minute_daily_summary_feature_preserves_legacy_positional_compute():
direct = MinuteDailySummaryFeature().compute(_minute_bars())
via_registry = get_feature("minute_daily_summary").compute(_minute_bars())
assert "minute_vwap" in direct.columns
pd.testing.assert_frame_equal(direct, via_registry)
def test_legacy_feature_compute_matches_canonical_derived_compute():
daily = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
"close": [11.0, 20.5, 12.0],
})
legacy_feature = compute_feature(
minute=_minute_bars(),
daily=daily,
feature_type="minute_daily_summary",
)
canonical_derived = compute_derived(
"minute_daily_summary",
daily=daily,
minute=_minute_bars(),
)
pd.testing.assert_frame_equal(legacy_feature, canonical_derived)
def test_read_feature_frames_delegates_to_derived_validation(tmp_path):
feature = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": ["2024-01-02 15:00:00"],
"toy_feature": [1.5],
})
feature_path = tmp_path / "feature.pq"
feature.to_parquet(feature_path, index=False)
[result] = read_feature_frames([feature_path])
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
assert result["toy_feature"].tolist() == [1.5]
def test_load_external_feature_module_and_filter_params(tmp_path):
module_path = tmp_path / "external_feature.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.features.base import BaseFeature
from pipeline.features.registry import register_feature
@register_feature
class ExternalVolumeFeature(BaseFeature):
name = "external_volume_feature"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, minute: pd.DataFrame, daily=None) -> pd.DataFrame:
out = (
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
.sum()
.rename(columns={"volume": "scaled_volume"})
)
out["scaled_volume"] *= self.scale
return out
'''))
load_feature_module(str(module_path))
assert "external_volume_feature" in available_features()
instance = get_feature("external_volume_feature", scale=2.0, ignored=99)
assert instance.scale == 2.0
assert not hasattr(instance, "ignored")
result = compute_feature(
minute=_minute_bars(),
feature_type="external_volume_feature",
scale=2.0,
ignored=99,
)
row = result[result["symbol_id"] == "sh600000"].iloc[0]
assert np.isclose(row["scaled_volume"], 800.0)
def test_validate_feature_frame_rejects_missing_keys():
with pytest.raises(ValueError, match="missing required"):
validate_feature_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
def test_validate_feature_frame_rejects_duplicate_keys_after_date_normalization():
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
validate_feature_frame(pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": ["2024-01-02", pd.Timestamp("2024-01-02")],
"x": [1.0, 2.0],
}))
def test_validate_feature_frame_rejects_duplicate_columns():
bad = pd.DataFrame(
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
columns=["symbol_id", "date", "dup", "dup"],
)
with pytest.raises(ValueError, match="duplicate columns"):
validate_feature_frame(bad)
def test_validate_feature_frame_rejects_non_numeric_feature_columns():
with pytest.raises(ValueError, match="numeric"):
validate_feature_frame(pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"bad": ["not numeric"],
}))
+650
View File
@@ -0,0 +1,650 @@
"""Tests for the JoinQuant comparison plugin (network-free)."""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
import pandas as pd
import pytest
from click.testing import CliRunner
from cli import cli
from pipeline.common.schema import FILL_COLUMNS, PNL_COLUMNS, POSITION_COLUMNS
from plugins.joinquant.browser import (
default_browser_config,
load_env_file,
resolve_template,
write_browser_config_template,
)
from plugins.joinquant.export_targets import export_targets
from plugins.joinquant.ingest import (
ingest_joinquant_outputs,
normalize_fills_csv,
)
from plugins.joinquant.reconcile import reconcile_joinquant
from plugins.joinquant.schema import (
JOINQUANT_FILL_COLUMNS,
JOINQUANT_PNL_COLUMNS,
JOINQUANT_POSITION_COLUMNS,
JOINQUANT_TARGET_COLUMNS,
RECONCILE_COLUMNS,
)
from plugins.joinquant.smoke import build_fixed_share_positions
from plugins.joinquant.symbols import from_joinquant_symbol, to_joinquant_symbol
from plugins.joinquant.wrapper_strategy import write_wrapper_strategy
def _positions(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
target_value = float(shares * price)
weight = target_value / 1_000_000.0
return pd.DataFrame([{
"symbol_id": symbol,
"date": pd.Timestamp(date),
"portfolio_name": portfolio_name,
"target_weight": weight,
"target_value": target_value,
"target_shares": float(shares) + 0.25,
"position_shares": shares,
"position_value": target_value,
"price": price,
}], columns=POSITION_COLUMNS)
def _our_fills(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
cost: float = 5.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
fills = pd.DataFrame([{
"symbol_id": symbol,
"date": pd.Timestamp(date),
"portfolio_name": portfolio_name,
"prev_shares": 0,
"target_shares": shares,
"traded_shares": shares,
"realized_shares": shares,
"blocked": 0,
"trade_cost": cost,
"trade_price": price,
}])
return fills
def _our_pnl(
*,
date: str = "2026-07-01",
pnl: float = 100.0,
cost: float = 5.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": pd.Timestamp(date),
"portfolio_name": portfolio_name,
"gross_exposure": 10_000.0,
"net_exposure": 10_000.0,
"pnl": pnl,
"cost": cost,
"turnover": 1.0,
"n_positions": 1,
}], columns=PNL_COLUMNS)
def _jq_fills(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
cost: float = 5.0,
portfolio_name: str = "run1",
raw_status: str = "filled",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": date,
"portfolio_name": portfolio_name,
"symbol_id": symbol,
"jq_symbol": to_joinquant_symbol(symbol),
"order_id": "ord-1",
"side": "buy" if shares >= 0 else "sell",
"requested_shares": shares,
"filled_shares": shares,
"fill_price": price,
"trade_value": abs(shares * price),
"trade_cost": cost,
"blocked": 0,
"raw_status": raw_status,
}], columns=JOINQUANT_FILL_COLUMNS)
def _jq_positions(
*,
symbol: str = "sh600000",
date: str = "2026-07-01",
shares: int = 1000,
price: float = 10.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": date,
"portfolio_name": portfolio_name,
"symbol_id": symbol,
"jq_symbol": to_joinquant_symbol(symbol),
"position_shares": shares,
"position_value": shares * price,
"cash": 990_000.0,
"total_value": 1_000_000.0,
}], columns=JOINQUANT_POSITION_COLUMNS)
def _jq_pnl(
*,
date: str = "2026-07-01",
pnl: float = 100.0,
cost: float = 5.0,
portfolio_name: str = "run1",
) -> pd.DataFrame:
return pd.DataFrame([{
"date": date,
"portfolio_name": portfolio_name,
"gross_exposure": 10_000.0,
"net_exposure": 10_000.0,
"cash": 990_000.0,
"total_value": 1_000_000.0,
"pnl": pnl,
"cost": cost,
"turnover": 1.0,
}], columns=JOINQUANT_PNL_COLUMNS)
def _write_parquets(tmp_path: Path, frames: dict[str, pd.DataFrame]) -> dict[str, Path]:
paths = {}
for name, frame in frames.items():
path = tmp_path / f"{name}.pq"
frame.to_parquet(path, index=False)
paths[name] = path
return paths
def _export_targets_for(tmp_path: Path, positions: pd.DataFrame) -> tuple[Path, Path]:
positions_path = tmp_path / "positions.pq"
positions.to_parquet(positions_path, index=False)
targets_root = tmp_path / "targets"
export_targets(
positions_path,
portfolio_name="run1",
out_dir=targets_root,
mode="target_shares",
)
return positions_path, targets_root / "run1"
@pytest.mark.parametrize(
("internal", "joinquant"),
[
("sh600000", "600000.XSHG"),
("sh688001", "688001.XSHG"),
("sz000001", "000001.XSHE"),
("sz001001", "001001.XSHE"),
("sz002594", "002594.XSHE"),
("sz300001", "300001.XSHE"),
],
)
def test_symbol_mapping_both_directions(internal, joinquant):
assert to_joinquant_symbol(internal) == joinquant
assert from_joinquant_symbol(joinquant) == internal
@pytest.mark.parametrize("bad", ["600000", "bj830000", "sh000001", "sz600000", "abc"])
def test_symbol_mapping_rejects_invalid_symbols(bad):
with pytest.raises(ValueError):
to_joinquant_symbol(bad)
@pytest.mark.parametrize("bad", ["600000", "600000.XSHE", "000001.XSHG", "abc.XSHG"])
def test_reverse_symbol_mapping_rejects_invalid_symbols(bad):
with pytest.raises(ValueError):
from_joinquant_symbol(bad)
def test_export_targets_schema_snapshot_hash_and_no_overwrite(tmp_path):
positions_path = tmp_path / "positions.pq"
_positions().to_parquet(positions_path, index=False)
snapshots = export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets",
mode="target_shares",
)
csv_path = tmp_path / "targets" / "run1" / "20260701.csv"
parquet_path = tmp_path / "targets" / "run1" / "20260701.parquet"
snapshot_path = tmp_path / "snapshots" / "run1" / "20260701.json"
assert csv_path.exists()
assert parquet_path.exists()
assert snapshot_path.exists()
target = pd.read_csv(csv_path)
assert list(target.columns) == JOINQUANT_TARGET_COLUMNS
assert int(target.loc[0, "target_shares"]) == 1000
assert float(target.loc[0, "target_value"]) == 10_000.0
assert target.loc[0, "export_mode"] == "target_shares"
snapshot = json.loads(snapshot_path.read_text())
actual_hash = hashlib.sha256(csv_path.read_bytes()).hexdigest()
assert snapshots[0]["file_sha256"] == actual_hash
assert snapshot["file_sha256"] == actual_hash
assert snapshot["n_symbols"] == 1
with pytest.raises(FileExistsError):
export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets",
mode="target_shares",
)
def test_export_targets_target_value_mode_from_position_columns(tmp_path):
positions_path = tmp_path / "positions.pq"
_positions(shares=250, price=20.0).to_parquet(positions_path, index=False)
export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets_value",
mode="target_value",
)
target = pd.read_parquet(tmp_path / "targets_value" / "run1" / "20260701.parquet")
assert list(target.columns) == JOINQUANT_TARGET_COLUMNS
assert target.loc[0, "export_mode"] == "target_value"
assert target.loc[0, "target_value"] == 5_000.0
assert target.loc[0, "target_shares"] == 250
def test_export_targets_can_shift_to_next_execution_session(tmp_path):
positions_path = tmp_path / "positions.pq"
_positions(date="2024-01-09").to_parquet(positions_path, index=False)
calendar_path = tmp_path / "daily.pq"
pd.DataFrame({
"date": pd.to_datetime(["2024-01-09", "2024-01-10", "2024-01-11"]),
"symbol_id": ["sh600000", "sh600000", "sh600000"],
}).to_parquet(calendar_path, index=False)
snapshots = export_targets(
positions_path,
portfolio_name="run1",
out_dir=tmp_path / "targets_shifted",
mode="target_shares",
start_date="2024-01-10",
end_date="2024-01-10",
execution_calendar_path=calendar_path,
)
assert len(snapshots) == 1
assert snapshots[0]["date"] == "2024-01-10"
assert (tmp_path / "targets_shifted" / "run1" / "20240110.csv").exists()
target = pd.read_csv(tmp_path / "targets_shifted" / "run1" / "20240110.csv")
assert target.loc[0, "date"] == "2024-01-10"
def test_ingest_permissive_csv_column_mapping_and_output_schemas(tmp_path):
fills_csv = tmp_path / "jq_fills.csv"
positions_csv = tmp_path / "jq_positions.csv"
pnl_csv = tmp_path / "jq_pnl.csv"
pd.DataFrame([{
"Trade Date": "2026-07-01 09:31:00",
"Security": "600000.XSHG",
"Direction": "buy",
"Order Amount": 1000,
"Filled Amount": 1000,
"Price": 10.0,
"Status": "filled",
}]).to_csv(fills_csv, index=False)
pd.DataFrame([{
"Date": "2026-07-01",
"Security": "600000.XSHG",
"Shares": 1000,
"Market Value": 10_000.0,
"Cash": 990_000.0,
"Portfolio Value": 1_000_000.0,
}]).to_csv(positions_csv, index=False)
pd.DataFrame([{
"Date": "2026-07-01",
"Portfolio Value": 1_000_000.0,
"Daily PnL": 100.0,
"Turnover": 1.0,
}]).to_csv(pnl_csv, index=False)
fills = normalize_fills_csv(fills_csv, "run1")
assert list(fills.columns) == JOINQUANT_FILL_COLUMNS
assert fills.loc[0, "symbol_id"] == "sh600000"
assert fills.loc[0, "jq_symbol"] == "600000.XSHG"
assert fills.loc[0, "trade_cost"] == 0.0
assert fills.loc[0, "blocked"] == 0
paths = ingest_joinquant_outputs(
portfolio_name="run1",
fills_csv=fills_csv,
positions_csv=positions_csv,
pnl_csv=pnl_csv,
out_dir=tmp_path / "ingested",
)
assert list(pd.read_parquet(paths["fills"]).columns) == JOINQUANT_FILL_COLUMNS
assert list(pd.read_parquet(paths["positions"]).columns) == JOINQUANT_POSITION_COLUMNS
assert list(pd.read_parquet(paths["pnl"]).columns) == JOINQUANT_PNL_COLUMNS
def _run_reconcile_case(
tmp_path: Path,
*,
positions: pd.DataFrame | None = None,
our_fills: pd.DataFrame | None = None,
jq_fills: pd.DataFrame | None = None,
jq_positions: pd.DataFrame | None = None,
our_pnl: pd.DataFrame | None = None,
jq_pnl: pd.DataFrame | None = None,
) -> pd.DataFrame:
positions = _positions() if positions is None else positions
_, targets_dir = _export_targets_for(tmp_path, positions)
paths = _write_parquets(tmp_path, {
"our_fills": _our_fills() if our_fills is None else our_fills,
"our_positions": positions,
"our_pnl": _our_pnl() if our_pnl is None else our_pnl,
"jq_fills": _jq_fills() if jq_fills is None else jq_fills,
"jq_positions": _jq_positions() if jq_positions is None else jq_positions,
"jq_pnl": _jq_pnl() if jq_pnl is None else jq_pnl,
})
out_paths = reconcile_joinquant(
portfolio_name="run1",
targets_dir=targets_dir,
our_fills_path=paths["our_fills"],
our_positions_path=paths["our_positions"],
our_pnl_path=paths["our_pnl"],
jq_fills_path=paths["jq_fills"],
jq_positions_path=paths["jq_positions"],
jq_pnl_path=paths["jq_pnl"],
out_dir=tmp_path / "reconcile",
)
report = pd.read_parquet(out_paths["daily_reconcile"])
assert list(report.columns) == RECONCILE_COLUMNS
assert out_paths["summary_md"].exists()
assert out_paths["summary_csv"].exists()
return report
def test_reconcile_exact_match(tmp_path):
report = _run_reconcile_case(tmp_path)
assert report.loc[0, "diff_reason"] == "MATCH"
assert report.loc[0, "filled_share_diff"] == 0
assert report.loc[0, "position_share_diff"] == 0
def test_reconcile_price_mismatch(tmp_path):
report = _run_reconcile_case(tmp_path, jq_fills=_jq_fills(price=10.5))
assert report.loc[0, "diff_reason"] == "PRICE_MISMATCH"
def test_reconcile_cost_mismatch(tmp_path):
report = _run_reconcile_case(
tmp_path,
jq_fills=_jq_fills(cost=8.0),
jq_pnl=_jq_pnl(cost=8.0),
)
assert report.loc[0, "diff_reason"] == "COST_MODEL"
def test_reconcile_missing_symbol_in_joinquant(tmp_path):
empty_jq_fills = pd.DataFrame(columns=JOINQUANT_FILL_COLUMNS)
empty_jq_positions = pd.DataFrame(columns=JOINQUANT_POSITION_COLUMNS)
report = _run_reconcile_case(
tmp_path,
jq_fills=empty_jq_fills,
jq_positions=empty_jq_positions,
)
assert report.loc[0, "diff_reason"] == "MISSING_IN_JOINQUANT"
def test_reconcile_short_target_with_long_only_joinquant_output(tmp_path):
positions = _positions(shares=-100, price=10.0)
our_fills = _our_fills(shares=-100, price=10.0)
jq_fills = _jq_fills(shares=0, price=10.0, cost=0.0, raw_status="short clipped")
jq_positions = _jq_positions(shares=0, price=10.0)
report = _run_reconcile_case(
tmp_path,
positions=positions,
our_fills=our_fills,
jq_fills=jq_fills,
jq_positions=jq_positions,
)
assert report.loc[0, "diff_reason"] == "SHORT_NOT_SUPPORTED"
def test_joinquant_cli_smoke_export_ingest_reconcile_and_wrapper(tmp_path):
runner = CliRunner()
positions_path = tmp_path / "positions.pq"
_positions().to_parquet(positions_path, index=False)
result = runner.invoke(cli, [
"joinquant", "export-targets",
"--positions-path", str(positions_path),
"--portfolio-name", "run1",
"--mode", "target_shares",
"--out-dir", str(tmp_path / "targets"),
])
assert result.exit_code == 0, result.output
assert "Exported JoinQuant targets" in result.output
fills_csv = tmp_path / "jq_fills.csv"
positions_csv = tmp_path / "jq_positions.csv"
pnl_csv = tmp_path / "jq_pnl.csv"
_jq_fills().to_csv(fills_csv, index=False)
_jq_positions().to_csv(positions_csv, index=False)
_jq_pnl().to_csv(pnl_csv, index=False)
result = runner.invoke(cli, [
"joinquant", "ingest",
"--portfolio-name", "run1",
"--fills-csv", str(fills_csv),
"--positions-csv", str(positions_csv),
"--pnl-csv", str(pnl_csv),
"--out-dir", str(tmp_path / "ingested"),
])
assert result.exit_code == 0, result.output
assert "Saved JoinQuant fills" in result.output
paths = _write_parquets(tmp_path, {
"our_fills": _our_fills(),
"our_pnl": _our_pnl(),
})
result = runner.invoke(cli, [
"joinquant", "reconcile",
"--portfolio-name", "run1",
"--targets-dir", str(tmp_path / "targets" / "run1"),
"--our-fills-path", str(paths["our_fills"]),
"--our-positions-path", str(positions_path),
"--our-pnl-path", str(paths["our_pnl"]),
"--jq-fills-path", str(tmp_path / "ingested" / "run1" / "fills.pq"),
"--jq-positions-path", str(tmp_path / "ingested" / "run1" / "positions.pq"),
"--jq-pnl-path", str(tmp_path / "ingested" / "run1" / "pnl.pq"),
"--out-dir", str(tmp_path / "reconcile"),
])
assert result.exit_code == 0, result.output
assert "Saved reconciliation parquet" in result.output
wrapper_path = tmp_path / "wrapper_strategy_run1.py"
result = runner.invoke(cli, [
"joinquant", "write-wrapper",
"--portfolio-name", "run1",
"--mode", "target_shares",
"--out-path", str(wrapper_path),
])
assert result.exit_code == 0, result.output
assert "Saved JoinQuant wrapper strategy" in result.output
text = wrapper_path.read_text()
assert 'PORTFOLIO_NAME = "run1"' in text
assert 'TARGET_MODE = "target_shares"' in text
assert "ALLOW_SHORT = False" in text
def test_wrapper_strategy_generation_smoke(tmp_path):
path = write_wrapper_strategy(
portfolio_name="run2",
mode="target_value",
out_path=tmp_path / "wrapper.py",
)
text = path.read_text()
assert 'PORTFOLIO_NAME = "run2"' in text
assert 'TARGET_MODE = "target_value"' in text
assert "order_target_value" in text
def test_wrapper_strategy_can_embed_target_csvs(tmp_path):
targets_dir = tmp_path / "targets"
targets_dir.mkdir()
(targets_dir / "20260701.csv").write_text(
"date,portfolio_name,symbol_id,jq_symbol,target_shares,target_value,export_mode\n"
"2026-07-01,run1,sh600000,600000.XSHG,1000,10000,target_shares\n",
encoding="utf-8",
)
path = write_wrapper_strategy(
portfolio_name="run1",
mode="target_shares",
out_path=tmp_path / "wrapper_embedded.py",
embedded_targets_dir=targets_dir,
)
text = path.read_text()
assert '"20260701.csv"' in text
assert "600000.XSHG" in text
assert "if file_name in _EMBEDDED_TARGETS" in text
def test_build_fixed_share_positions_excludes_final_executionless_date():
data = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sh600000"],
"date": pd.to_datetime(["2024-01-09", "2024-01-10", "2024-01-11"]),
"close": [10.0, 10.5, 11.0],
})
positions = build_fixed_share_positions(
data,
trade_symbol="sh600000",
portfolio_name="run1",
shares=1000,
booksize=1_000_000.0,
)
assert list(positions.columns) == POSITION_COLUMNS
assert positions["date"].dt.strftime("%Y-%m-%d").tolist() == [
"2024-01-09",
"2024-01-10",
]
assert positions["position_shares"].tolist() == [1000, 1000]
assert positions["target_value"].tolist() == [10_000.0, 10_500.0]
def test_browser_config_template_and_placeholder_resolution(tmp_path):
config_path = write_browser_config_template(
tmp_path / "browser_config.json",
strategy_url="https://www.joinquant.com/example",
)
config = json.loads(config_path.read_text())
assert config["strategy_url"] == "https://www.joinquant.com/example"
assert config["actions"][0]["type"] == "goto"
context = {
"wrapper_path": "/tmp/wrapper.py",
"target_csvs": ["/tmp/20240110.csv", "/tmp/20240111.csv"],
"expected_joinquant_csvs": {"fills": "/tmp/jq_fills.csv"},
}
assert resolve_template("{wrapper_path}", context) == "/tmp/wrapper.py"
assert resolve_template("{target_csvs}", context) == [
"/tmp/20240110.csv",
"/tmp/20240111.csv",
]
assert resolve_template("save:{expected_joinquant_csvs.fills}", context) == "save:/tmp/jq_fills.csv"
script = "(arg) => { return new Event('change', {bubbles: true}); }"
assert resolve_template(script, context) == script
def test_load_env_file_handles_quotes_without_shell_sourcing(tmp_path):
env_path = tmp_path / "joinquant.env"
env_path.write_text(
"JOINQUANT_USERNAME=alice\n"
"JOINQUANT_PASSWORD=\"secret\"\n"
"JOINQUANT_STRATEGY_URL='https://example.test/path?x=1&y=2'\n"
)
env = load_env_file(env_path)
assert env["JOINQUANT_USERNAME"] == "alice"
assert env["JOINQUANT_PASSWORD"] == "secret"
assert env["JOINQUANT_STRATEGY_URL"] == "https://example.test/path?x=1&y=2"
def test_sim_trade_browser_config_template(tmp_path):
config_path = write_browser_config_template(
tmp_path / "sim_config.json",
strategy_url="https://www.joinquant.com/sim",
flow="sim-trade",
)
config = json.loads(config_path.read_text())
assert config["flow"] == "sim-trade"
selectors = " ".join(
action.get("selector", "") for action in config["actions"]
)
assert "模拟盘" in selectors
assert "模拟交易" in selectors
assert any(
action["type"] == "screenshot"
and action["path"] == "{run_artifact_dir}/sim_trade_final.png"
for action in config["actions"]
)
def test_joinquant_cli_browser_config_smoke(tmp_path):
runner = CliRunner()
config_path = tmp_path / "browser_config.json"
result = runner.invoke(cli, [
"joinquant",
"write-browser-config",
"--out-path",
str(config_path),
"--strategy-url",
"https://www.joinquant.com/example",
])
assert result.exit_code == 0, result.output
assert config_path.exists()
assert default_browser_config()["actions"]
sim_config_path = tmp_path / "sim_browser_config.json"
result = runner.invoke(cli, [
"joinquant",
"write-browser-config",
"--out-path",
str(sim_config_path),
"--strategy-url",
"https://www.joinquant.com/sim",
"--flow",
"sim-trade",
])
assert result.exit_code == 0, result.output
assert json.loads(sim_config_path.read_text())["flow"] == "sim-trade"
+369
View File
@@ -0,0 +1,369 @@
"""Tests for raw Baostock minute bar download plumbing."""
from pathlib import Path
import numpy as np
import pandas as pd
import pytest
import data.downloader as low_level_downloader
import pipeline.data.downloader as pipeline_downloader
from data.downloader import download_minute_batch
from pipeline.common.schema import MINUTE_BAR_COLUMNS
from pipeline.data.downloader import download_minute_universe
class _FakeResult:
def __init__(self, rows, error_code="0", error_msg=""):
self.rows = rows
self.error_code = error_code
self.error_msg = error_msg
self._idx = -1
def next(self):
self._idx += 1
return self._idx < len(self.rows)
def get_row_data(self):
return self.rows[self._idx]
def test_download_minute_batch_maps_and_parses_baostock_rows(monkeypatch):
rows = [
[
"2024-01-02",
"20240102093500000",
"sh.600000",
"10",
"11",
"9",
"10.5",
"1000",
"10500",
"3",
],
[
"2024-01-02",
"20240102094000000",
"sh.600000",
"10.5",
"12",
"10",
"11",
"2000",
"22000",
"3",
],
]
calls = []
def fake_query(**kwargs):
calls.append(kwargs)
return _FakeResult(rows)
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
fake_query,
)
[(symbol, df)] = list(
download_minute_batch(
["sh600000"],
"2024-01-02",
"2024-01-02",
frequency=5,
)
)
assert symbol == "sh600000"
assert calls[0]["code"] == "sh.600000"
assert calls[0]["frequency"] == "5"
assert calls[0]["adjustflag"] == "3"
assert df is not None
assert df["datetime"].iloc[0] == pd.Timestamp("2024-01-02 09:35:00")
assert df["time"].tolist() == ["09:35:00", "09:40:00"]
assert (df["frequency"] == "5m").all()
assert np.isclose(df["open"].iloc[0], 10.0)
assert np.isclose(df["vwap"].iloc[0], 10.5)
assert pd.api.types.is_numeric_dtype(df["volume"])
def test_minute_frequency_and_timestamp_parsing_edge_cases():
frequency, label = low_level_downloader._normalize_minute_frequency("15m")
assert (frequency, label) == ("15", "15m")
with pytest.raises(ValueError, match="Unsupported minute frequency"):
low_level_downloader._normalize_minute_frequency("1m")
parsed = low_level_downloader._parse_minute_datetime(
pd.Series(["2024-01-02", "2024-01-02"]),
pd.Series(["0935", "09:40:00"]),
)
assert parsed.tolist() == [
pd.Timestamp("2024-01-02 09:35:00"),
pd.Timestamp("2024-01-02 09:40:00"),
]
def test_download_minute_batch_empty_result_yields_none(monkeypatch):
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult([]),
)
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_minute_batch_non_login_error_and_periodic_relogin(monkeypatch):
responses = [
_FakeResult([], error_code="1", error_msg="bad symbol"),
_FakeResult([]),
]
login_count = 0
logout_count = 0
def fake_login():
nonlocal login_count
login_count += 1
def fake_logout():
nonlocal logout_count
logout_count += 1
monkeypatch.setattr(low_level_downloader.bs, "login", fake_login)
monkeypatch.setattr(low_level_downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(
download_minute_batch(
["sh600000", "sz000001"],
"2024-01-02",
"2024-01-02",
relogin_every=1,
)
) == [("sh600000", None), ("sz000001", None)]
assert login_count == 2
assert logout_count == 2
def test_download_minute_batch_second_session_loss_yields_none(monkeypatch):
responses = [
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
]
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_minute_batch_ignores_relogin_and_final_logout_failures(monkeypatch):
responses = [
_FakeResult([], error_code="1", error_msg="bad symbol"),
_FakeResult([]),
]
logout_count = 0
def fake_logout():
nonlocal logout_count
logout_count += 1
raise RuntimeError("logout failed")
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", fake_logout)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: responses.pop(0),
)
assert list(
download_minute_batch(
["sh600000", "sz000001"],
"2024-01-02",
"2024-01-02",
relogin_every=1,
)
) == [("sh600000", None), ("sz000001", None)]
assert logout_count == 2
def test_download_minute_batch_rejects_unparsed_timestamps(monkeypatch):
bad_rows = [[
"2024-01-02",
"not-a-time",
"sh.600000",
"10",
"11",
"9",
"10.5",
"1000",
"10500",
"3",
]]
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
monkeypatch.setattr(
low_level_downloader.bs,
"query_history_k_data_plus",
lambda **kwargs: _FakeResult(bad_rows),
)
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
("sh600000", None)
]
def test_download_minute_universe_writes_frequency_month_partitions(tmp_path, monkeypatch):
minute = pd.DataFrame({
"symbol": ["sh600000", "sh600000"],
"datetime": pd.to_datetime(["2024-01-02 09:35:00", "2024-01-02 09:40:00"]),
"date": pd.to_datetime(["2024-01-02", "2024-01-02"]),
"time": ["09:35:00", "09:40:00"],
"frequency": ["5m", "5m"],
"open": [10.0, 10.5],
"high": [11.0, 12.0],
"low": [9.0, 10.0],
"close": [10.5, 11.0],
"volume": [1000.0, 2000.0],
"amount": [10500.0, 22000.0],
"vwap": [10.5, 11.0],
"adjustflag": ["3", "3"],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000"],
"symbol_name": ["PF Bank"],
}),
)
def fake_batch(symbols, start, end, frequency=5):
assert symbols == ["sh600000"]
assert frequency == "5"
yield "sh600000", minute
monkeypatch.setattr(pipeline_downloader, "download_minute_batch", fake_batch)
preserved = tmp_path / "toy" / "frequency=15m" / "month=2024-01" / "old.pq"
preserved.parent.mkdir(parents=True)
preserved_minute = minute.copy()
preserved_minute["frequency"] = "15m"
preserved_minute["symbol_id"] = "sh600000"
preserved_minute["symbol_name"] = "PF Bank"
preserved_minute[MINUTE_BAR_COLUMNS].to_parquet(preserved, index=False)
stale = tmp_path / "toy" / "frequency=5m" / "month=2024-01" / "stale.pq"
stale.parent.mkdir(parents=True)
preserved_minute.assign(frequency="5m")[MINUTE_BAR_COLUMNS].to_parquet(stale, index=False)
stats = download_minute_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
chunk_size=1,
frequency="5",
)
dataset_path = Path(stats["dataset_path"])
assert (dataset_path / "frequency=5m" / "month=2024-01").is_dir()
assert preserved.exists()
assert not stale.exists()
out = pd.read_parquet(dataset_path / "frequency=5m")
assert (set(MINUTE_BAR_COLUMNS) - {"frequency"}) <= set(out.columns)
assert set(out["symbol_id"]) == {"sh600000"}
assert set(out["symbol_name"]) == {"PF Bank"}
assert stats["n_rows"] == 2
def test_download_minute_universe_raises_when_all_symbols_empty(tmp_path, monkeypatch):
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": ["sh600000"],
"symbol_name": ["PF Bank"],
}),
)
monkeypatch.setattr(
pipeline_downloader,
"download_minute_batch",
lambda symbols, start, end, frequency=5: iter([("sh600000", None)]),
)
with pytest.raises(RuntimeError, match="No minute data"):
download_minute_universe(
universe="toy",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
)
def test_download_minute_universe_progress_branch_at_100_symbols(tmp_path, monkeypatch):
symbols = [f"sh6{i:05d}" for i in range(100)]
minute = pd.DataFrame({
"symbol": ["sh600000"],
"datetime": [pd.Timestamp("2024-01-02 09:35:00")],
"date": [pd.Timestamp("2024-01-02")],
"time": ["09:35:00"],
"frequency": ["5m"],
"open": [10.0],
"high": [11.0],
"low": [9.0],
"close": [10.5],
"volume": [1000.0],
"amount": [10500.0],
"vwap": [10.5],
"adjustflag": ["3"],
})
monkeypatch.setattr(
pipeline_downloader,
"_resolve_universe",
lambda universe, max_symbols=0: pd.DataFrame({
"symbol_id": symbols,
"symbol_name": symbols,
}),
)
def fake_batch(requested_symbols, start, end, frequency=5):
assert requested_symbols == symbols
for symbol in requested_symbols:
yield symbol, minute.copy()
monkeypatch.setattr(pipeline_downloader, "download_minute_batch", fake_batch)
stats = download_minute_universe(
universe="toy100",
start_date="2024-01-02",
end_date="2024-01-02",
output_dir=str(tmp_path),
chunk_size=200,
)
assert stats["n_symbols"] == 100
assert stats["n_rows"] == 100
+398
View File
@@ -13,12 +13,17 @@ from pipeline.portfolio.market_rules import (
Board, Board,
LimitStatus, LimitStatus,
MarketRule, MarketRule,
_to_date,
compute_limit_status, compute_limit_status,
detect_board, detect_board,
) )
from pipeline.portfolio.research import evaluate_portfolio from pipeline.portfolio.research import evaluate_portfolio
from pipeline.portfolio.constraints import ( from pipeline.portfolio.constraints import (
TradeConstraint,
available_constraints,
get_constraint,
PriceLimitConstraint, PriceLimitConstraint,
register_constraint,
SuspensionConstraint, SuspensionConstraint,
VolumeCapConstraint, VolumeCapConstraint,
) )
@@ -82,6 +87,7 @@ def test_detect_board():
assert detect_board("sh688981") == Board.STAR assert detect_board("sh688981") == Board.STAR
assert detect_board("sz300750") == Board.CHINEXT assert detect_board("sz300750") == Board.CHINEXT
assert detect_board("bj830000") == Board.UNKNOWN assert detect_board("bj830000") == Board.UNKNOWN
assert detect_board("sh") == Board.UNKNOWN
# --- MarketRule date transitions --------------------------------------------- # --- MarketRule date transitions ---------------------------------------------
@@ -120,6 +126,26 @@ def test_get_rules_vectorized():
assert list(limit) == [0.10, 0.20, 0.20] assert list(limit) == [0.10, 0.20, 0.20]
def test_market_rule_date_coercion_unknown_board_and_st_vector_override():
rules = MarketRule()
assert _to_date(dt.datetime(2024, 1, 2, 15, 0)) == dt.date(2024, 1, 2)
assert _to_date(pd.Timestamp("2024-01-03 09:30")) == dt.date(2024, 1, 3)
assert _to_date("2024-01-04 10:00:00") == dt.date(2024, 1, 4)
unknown_rule = rules.get_rule("xx999999", "2024-01-02")
assert unknown_rule.minimum_open_size == 100
assert unknown_rule.share_increment == 100
assert unknown_rule.price_limit_pct == 0.10
_, _, _, limit = rules.get_rules_vectorized(
np.array(["sh600000", "sh600000"], dtype=object),
"2024-01-02",
np.array([0, 1]),
)
assert limit.tolist() == [0.10, 0.05]
def test_compute_limit_status(): def test_compute_limit_status():
price = np.array([110.0, 90.0, 100.0]) price = np.array([110.0, 90.0, 100.0])
preclose = np.array([100.0, 100.0, 100.0]) preclose = np.array([100.0, 100.0, 100.0])
@@ -130,6 +156,20 @@ def test_compute_limit_status():
assert status[2] == LimitStatus.NORMAL.value assert status[2] == LimitStatus.NORMAL.value
def test_compute_limit_status_treats_bad_preclose_as_normal():
status = compute_limit_status(
price=np.array([np.nan, 110.0, 90.0]),
preclose=np.array([100.0, np.nan, 0.0]),
limit_pct=np.array([0.10, 0.10, 0.10]),
)
assert status.tolist() == [
LimitStatus.NORMAL.value,
LimitStatus.NORMAL.value,
LimitStatus.NORMAL.value,
]
# --- continuous targets ------------------------------------------------------ # --- continuous targets ------------------------------------------------------
def test_continuous_targets_normalization(): def test_continuous_targets_normalization():
@@ -295,6 +335,70 @@ def test_repair_scales_to_4000_names():
assert abs(gross - B) <= 0.03 * B assert abs(gross - B) <= 0.03 * B
def test_repair_handles_empty_input():
result = repair_exposure(
np.array([], dtype=np.int64),
np.array([], dtype=float),
np.array([], dtype=float),
np.array([], dtype=np.int64),
np.array([], dtype=np.int64),
np.array([], dtype=np.int64),
)
assert result.dtype == np.int64
assert result.tolist() == []
def test_repair_respects_max_iters_and_zero_increment_noop():
q_round = np.array([100, -100], dtype=np.int64)
q_target = np.array([300.0, -300.0])
price = np.array([10.0, 10.0])
min_open = np.array([100, 100])
prev = np.zeros(2, dtype=np.int64)
capped = repair_exposure(
q_round,
q_target,
price,
increment=np.array([1, 1]),
min_open=min_open,
prev_shares=prev,
booksize=10_000.0,
gross_tol=0.0,
max_iters=0,
)
zero_increment = repair_exposure(
q_round,
q_target,
price,
increment=np.array([0, 0]),
min_open=min_open,
prev_shares=prev,
booksize=0.0,
net_tol=0.0,
gross_tol=0.0,
)
assert capped.tolist() == [100, -100]
assert zero_increment.tolist() == [100, -100]
def test_repair_gross_growth_obeys_net_band():
pos = repair_exposure(
q_round=np.array([100], dtype=np.int64),
q_target=np.array([300.0]),
price=np.array([10.0]),
increment=np.array([1]),
min_open=np.array([100]),
prev_shares=np.array([0], dtype=np.int64),
booksize=2_000.0,
net_tol=0.5,
gross_tol=0.0,
)
assert pos.tolist() == [100]
# --- construct_positions ----------------------------------------------------- # --- construct_positions -----------------------------------------------------
def test_construct_positions_schema(): def test_construct_positions_schema():
@@ -306,6 +410,47 @@ def test_construct_positions_schema():
assert pos["position_shares"].dtype == np.int64 assert pos["position_shares"].dtype == np.int64
def test_construct_positions_empty_weights_returns_schema():
data = _make_data(n_days=3)
empty_weights = pd.DataFrame(columns=["symbol_id", "date", "combo_name", "weight"])
pos = construct_positions(
empty_weights,
data,
booksize=1e6,
portfolio_name="empty",
)
assert list(pos.columns) == POSITION_COLUMNS
assert pos.empty
def test_construct_positions_ignores_absent_or_bad_prices():
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
data = pd.DataFrame([
{"symbol_id": "sh600000", "date": dates[0], "close": np.nan, "isST": 0},
{"symbol_id": "sz000001", "date": dates[0], "close": 20.0, "isST": 0},
{"symbol_id": "sh600000", "date": dates[1], "close": 10.0, "isST": 0},
{"symbol_id": "sz000001", "date": dates[1], "close": 20.0, "isST": 0},
])
weights = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[0], dates[0], dates[1], dates[1]],
"combo_name": ["combo"] * 4,
"weight": [1.0, -1.0, 1.0, -1.0],
})
pos = construct_positions(weights, data, booksize=10000.0, portfolio_name="bad_price")
bad_price_rows = pos[
(pos["date"] == dates[0])
& (pos["symbol_id"] == "sh600000")
]
assert bad_price_rows.empty or (bad_price_rows["target_weight"] == 0.0).all()
assert np.isfinite(pos["target_value"]).all()
assert np.isfinite(pos["position_value"]).all()
def test_construct_positions_threads_state_and_closes_absent(): def test_construct_positions_threads_state_and_closes_absent():
data = _make_data() data = _make_data()
weights = _make_weights(data) weights = _make_weights(data)
@@ -321,6 +466,34 @@ def test_construct_positions_threads_state_and_closes_absent():
assert final.empty or (final["position_shares"] == 0).all() assert final.empty or (final["position_shares"] == 0).all()
def test_construct_positions_closes_absent_short_position():
dates = pd.to_datetime(["2024-01-02", "2024-01-03"])
data = pd.DataFrame([
{"symbol_id": sym, "date": d, "close": price, "isST": 0}
for d in dates
for sym, price in (("sh600000", 10.0), ("sz000001", 20.0))
])
weights = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sz000001"],
"date": [dates[0], dates[0], dates[1]],
"combo_name": ["combo", "combo", "combo"],
"weight": [-1.0, 1.0, 1.0],
})
pos = construct_positions(weights, data, booksize=20000.0, portfolio_name="absent_short")
first_day_short = pos[
(pos["date"] == dates[0])
& (pos["symbol_id"] == "sh600000")
]
final_day_short = pos[
(pos["date"] == dates[1])
& (pos["symbol_id"] == "sh600000")
]
assert (first_day_short["position_shares"] < 0).all()
assert final_day_short.empty or (final_day_short["position_shares"] == 0).all()
def test_construct_positions_carries_book_on_zero_gross(caplog): def test_construct_positions_carries_book_on_zero_gross(caplog):
dates = pd.to_datetime(["2024-01-01", "2024-01-02"]) dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
symbols = ["sh600000", "sz000001"] symbols = ["sh600000", "sz000001"]
@@ -392,8 +565,117 @@ def test_volume_cap_uses_traded_value():
assert low[0] == -10000.0 assert low[0] == -10000.0
def test_constraints_compose_repeatably_regardless_of_order():
n = 1
sl = _slice(
n,
tradestatus=np.array([0.0]),
limit_status=np.array([LimitStatus.UP_LIMIT.value], dtype=np.int8),
amount=np.array([1_000.0]),
price=np.array([10.0]),
)
ctx = TradeContext(np.zeros(n, np.int64), np.array([500]), sl, 1e6)
first_order = ReferenceSimulator(
constraints=[
SuspensionConstraint(),
PriceLimitConstraint(),
VolumeCapConstraint(max_frac=0.1),
],
cost_bps=10,
).fill(ctx)
reversed_order = ReferenceSimulator(
constraints=[
VolumeCapConstraint(max_frac=0.1),
PriceLimitConstraint(),
SuspensionConstraint(),
],
cost_bps=10,
).fill(ctx)
assert first_order.traded_shares.tolist() == [0]
assert first_order.realized_shares.tolist() == [0]
assert first_order.blocked.tolist() == [1]
assert np.array_equal(first_order.traded_shares, reversed_order.traded_shares)
assert np.array_equal(first_order.realized_shares, reversed_order.realized_shares)
assert np.array_equal(first_order.blocked, reversed_order.blocked)
assert np.array_equal(first_order.cost, reversed_order.cost)
def test_constraint_registry_and_default_adjust_targets():
class _NoopConstraint(TradeConstraint):
name = "_coverage_noop_constraint"
def delta_bounds(self, ctx):
return np.zeros(1), np.ones(1)
registered = register_constraint(_NoopConstraint)
assert registered is _NoopConstraint
assert "_coverage_noop_constraint" in available_constraints()
assert isinstance(get_constraint("_coverage_noop_constraint"), _NoopConstraint)
assert get_constraint("_coverage_noop_constraint").adjust_targets(object()) is None
with np.testing.assert_raises(KeyError):
get_constraint("_missing_constraint")
with np.testing.assert_raises(TypeError):
register_constraint(object) # type: ignore[arg-type]
with np.testing.assert_raises(ValueError):
class _NoNameConstraint(TradeConstraint):
def delta_bounds(self, ctx):
return np.zeros(1), np.ones(1)
register_constraint(_NoNameConstraint)
with np.testing.assert_raises(ValueError):
class _DuplicateConstraint(TradeConstraint):
name = "_coverage_noop_constraint"
def delta_bounds(self, ctx):
return np.zeros(1), np.ones(1)
register_constraint(_DuplicateConstraint)
# --- ReferenceSimulator ------------------------------------------------------ # --- ReferenceSimulator ------------------------------------------------------
def test_simulator_applies_constraint_target_adjustment():
class _HalveTarget(TradeConstraint):
name = "_halve_target"
def adjust_targets(self, ctx):
return ctx.target_shares // 2
def delta_bounds(self, ctx):
return np.full(len(ctx.target_shares), -np.inf), np.full(len(ctx.target_shares), np.inf)
sl = _slice(1, price=np.array([10.0]))
ctx = TradeContext(np.array([0], np.int64), np.array([100], np.int64), sl, 1e6)
result = ReferenceSimulator(constraints=[_HalveTarget()]).fill(ctx)
assert result.traded_shares.tolist() == [50]
assert result.realized_shares.tolist() == [50]
def test_simulator_empty_positions_uses_default_booksize():
data = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"open": [10.0],
"close": [10.0],
"preclose": [10.0],
"amount": [1e9],
"tradestatus": [1],
"isST": [0],
})
positions = pd.DataFrame(columns=POSITION_COLUMNS)
fills, pnl = ReferenceSimulator().run(positions, data)
assert list(fills.columns) == FILL_COLUMNS
assert list(pnl.columns) == PNL_COLUMNS
assert fills.empty
assert pnl.empty
def test_simulator_next_open_and_blocked_buy_holds_prev(): def test_simulator_next_open_and_blocked_buy_holds_prev():
data = _make_data(n_days=15) data = _make_data(n_days=15)
weights = _make_weights(data) weights = _make_weights(data)
@@ -474,6 +756,91 @@ def test_simulator_cost_only_on_nonzero_realized_trades():
assert np.isclose(res.cost[1], 50 * 20 * 10 / 1e4) assert np.isclose(res.cost[1], 50 * 20 * 10 / 1e4)
def test_simulator_short_to_long_flip_trades_full_delta():
dates = pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"])
positions = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": [dates[0], dates[1]],
"portfolio_name": ["flip", "flip"],
"target_weight": [-1.0, 1.0],
"target_value": [-1000.0, 1000.0],
"target_shares": [-100.0, 100.0],
"position_shares": [-100, 100],
"position_value": [-1000.0, 1000.0],
"price": [10.0, 10.0],
})
data = pd.DataFrame({
"symbol_id": ["sh600000"] * 3,
"date": dates,
"open": [10.0, 10.0, 10.0],
"close": [10.0, 10.0, 10.0],
"preclose": [10.0, 10.0, 10.0],
"amount": [1e9, 1e9, 1e9],
"tradestatus": [1, 1, 1],
"isST": [0, 0, 0],
})
fills, _ = ReferenceSimulator().run(positions, data)
by_date = fills.set_index("date")
assert by_date.loc[dates[1], "traded_shares"] == -100
assert by_date.loc[dates[1], "realized_shares"] == -100
assert by_date.loc[dates[2], "prev_shares"] == -100
assert by_date.loc[dates[2], "traded_shares"] == 200
assert by_date.loc[dates[2], "realized_shares"] == 100
def test_simulator_volume_cap_partially_fills_sell():
sl = _slice(1, amount=np.array([10_000.0]), price=np.array([10.0]))
ctx = TradeContext(
np.array([1000], np.int64),
np.array([0], np.int64),
sl,
1_000_000.0,
)
result = ReferenceSimulator(
constraints=[VolumeCapConstraint(max_frac=0.10)]
).fill(ctx)
assert result.traded_shares.tolist() == [-100]
assert result.realized_shares.tolist() == [900]
assert result.blocked.tolist() == [1]
def test_simulator_missing_next_open_has_zero_cost_and_turnover():
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [dates[0]],
"portfolio_name": ["missing_open"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [100.0],
"position_shares": [100],
"position_value": [1000.0],
"price": [10.0],
})
data = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": dates,
"open": [10.0, np.nan],
"close": [10.0, 10.0],
"preclose": [10.0, 10.0],
"amount": [1e9, 1e9],
"tradestatus": [1, 1],
"isST": [0, 0],
})
fills, pnl = ReferenceSimulator(cost_bps=10, slippage_bps=5).run(positions, data)
assert fills["traded_shares"].iloc[0] == 100
assert fills["trade_cost"].iloc[0] == 0.0
assert pnl["cost"].iloc[0] == 0.0
assert pnl["turnover"].iloc[0] == 0.0
assert pnl["gross_exposure"].iloc[0] == 1000.0
def test_simple_cost_model_adds_cost_and_slippage_without_price_adjustment(): def test_simple_cost_model_adds_cost_and_slippage_without_price_adjustment():
model = SimpleProportionalCostModel(cost_bps=10, slippage_bps=5) model = SimpleProportionalCostModel(cost_bps=10, slippage_bps=5)
@@ -559,3 +926,34 @@ def test_evaluate_portfolio_excludes_signal_without_forward_return():
metrics = evaluate_portfolio(positions, data) metrics = evaluate_portfolio(positions, data)
assert metrics["n_dates"] == 1 assert metrics["n_dates"] == 1
def test_evaluate_portfolio_empty_and_single_return_paths():
empty_metrics = evaluate_portfolio(
pd.DataFrame(columns=POSITION_COLUMNS),
pd.DataFrame(columns=["symbol_id", "date", "open"]),
)
assert empty_metrics["n_dates"] == 0
assert empty_metrics["cumulative_return"] == 0.0
dates = pd.date_range("2024-01-01", periods=3)
data = pd.DataFrame([
{"symbol_id": "sh600000", "date": d, "open": price}
for d, price in zip(dates, [100.0, 100.0, 110.0])
])
positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [dates[0]],
"portfolio_name": ["single"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [10.0],
"position_shares": [10],
"position_value": [1000.0],
"price": [100.0],
})
single_metrics = evaluate_portfolio(positions, data)
assert single_metrics["n_dates"] == 1
assert single_metrics["cumulative_return"] == 0.0
+98
View File
@@ -0,0 +1,98 @@
"""Malformed parquet/input tests for phase boundary contracts."""
from __future__ import annotations
import pandas as pd
import pytest
from pipeline.alpha.compute import compute_alpha
from pipeline.combo.combine import combine_alphas
from pipeline.derived.compute import validate_derived_frame
from pipeline.portfolio.construct import construct_positions
from pipeline.portfolio.simulator import ReferenceSimulator
from tests.helpers import make_generated_daily_bars
def test_alpha_compute_rejects_daily_data_without_close():
daily = make_generated_daily_bars().drop(columns=["close"])
with pytest.raises(KeyError, match="close"):
compute_alpha(daily, "bad", "reversal", lookback=3)
def test_alpha_feature_path_rejects_duplicate_symbol_dates(tmp_path):
daily = make_generated_daily_bars()
feature = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
"toy_feature": [1.0, 2.0],
})
feature_path = tmp_path / "duplicate_feature_keys.pq"
feature.to_parquet(feature_path, index=False)
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
compute_alpha(
daily,
"bad_features",
"reversal",
lookback=3,
feature_paths=[str(feature_path)],
)
def test_derived_validation_rejects_bool_value_columns():
derived = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"is_good": [True],
})
with pytest.raises(ValueError, match="numeric"):
validate_derived_frame(derived)
def test_combo_combine_rejects_missing_weight_column(tmp_path):
bad_alpha = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"alpha_name": ["bad"],
})
bad_alpha_path = tmp_path / "bad_alpha.pq"
bad_alpha.to_parquet(bad_alpha_path, index=False)
with pytest.raises(KeyError, match="weight"):
combine_alphas([str(bad_alpha_path)], "bad_combo")
def test_portfolio_build_rejects_weights_without_symbol_id():
daily = make_generated_daily_bars()
bad_weights = pd.DataFrame({
"date": [pd.Timestamp("2024-01-02")],
"combo_name": ["bad"],
"weight": [1.0],
})
with pytest.raises(KeyError, match="symbol_id"):
construct_positions(
bad_weights,
daily,
booksize=1_000_000.0,
portfolio_name="bad_portfolio",
)
def test_portfolio_simulate_rejects_positions_without_position_shares():
daily = make_generated_daily_bars()
bad_positions = pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"portfolio_name": ["bad"],
"target_weight": [1.0],
"target_value": [1000.0],
"target_shares": [100.0],
"position_value": [1000.0],
"price": [10.0],
})
with pytest.raises(KeyError, match="position_shares"):
ReferenceSimulator().run(bad_positions, daily)
+92
View File
@@ -0,0 +1,92 @@
"""Offline tests for baostock-backed universe helpers."""
from __future__ import annotations
import pandas as pd
import data.universe as universe
class _FakeResult:
def __init__(self, rows, fields=None):
self.rows = rows
self.fields = fields or ["code", "name", "date"]
self._idx = -1
def next(self):
self._idx += 1
return self._idx < len(self.rows)
def get_row_data(self):
return self.rows[self._idx]
def test_index_constituent_helpers_normalize_dotted_codes(monkeypatch):
calls: list[str] = []
monkeypatch.setattr(universe.bs, "login", lambda: calls.append("login"))
monkeypatch.setattr(universe.bs, "logout", lambda: calls.append("logout"))
monkeypatch.setattr(
universe.bs,
"query_hs300_stocks",
lambda: _FakeResult([
["sh.600000", "浦发银行", "2024-01-12"],
["sz.000001", "平安银行", "2024-01-12"],
]),
)
monkeypatch.setattr(
universe.bs,
"query_zz500_stocks",
lambda: _FakeResult([
["sh.600006", "东风汽车", "2024-01-12"],
]),
)
hs300 = universe.get_hs300_stocks()
zz500 = universe.get_zz500_stocks()
assert calls == ["login", "logout", "login", "logout"]
assert hs300["code"].tolist() == ["sh600000", "sz000001"]
assert zz500["code"].tolist() == ["sh600006"]
assert hs300["name"].tolist() == ["浦发银行", "平安银行"]
def test_get_all_stocks_walks_back_and_filters_to_listed_a_shares(monkeypatch):
fields = ["code", "tradeStatus", "code_name"]
responses = [
_FakeResult([], fields=fields),
_FakeResult(
[
["sh.600000", "1", "浦发银行"],
["sh.688001", "1", "华兴源创"],
["sz.000001", "1", "平安银行"],
["sz.300750", "1", "宁德时代"],
["sz.399001", "1", "深证成指"],
["sz.200001", "1", "深物业B"],
["bj.430047", "1", "北交所样本"],
],
fields=fields,
),
]
query_days: list[str] = []
monkeypatch.setattr(universe.bs, "login", lambda: None)
monkeypatch.setattr(universe.bs, "logout", lambda: None)
def fake_query_all_stock(day):
query_days.append(day)
return responses.pop(0)
monkeypatch.setattr(universe.bs, "query_all_stock", fake_query_all_stock)
result = universe.get_all_stocks("2024-01-07")
assert query_days == ["2024-01-07", "2024-01-06"]
assert result.columns.tolist() == ["code", "name"]
assert result["code"].tolist() == [
"sh600000",
"sh688001",
"sz000001",
"sz300750",
]
assert result["name"].tolist() == ["浦发银行", "华兴源创", "平安银行", "宁德时代"]
+508
View File
@@ -0,0 +1,508 @@
"""Verbose offline checks for the daily research workflow."""
from __future__ import annotations
from pathlib import Path
import numpy as np
import pandas as pd
from pipeline.alpha.compute import compute_alpha
from pipeline.combo.combine import combine_alphas
from pipeline.common.schema import (
ALPHA_COLUMNS,
COMBO_COLUMNS,
FILL_COLUMNS,
PNL_COLUMNS,
POSITION_COLUMNS,
)
from pipeline.portfolio.constraints import (
PriceLimitConstraint,
SuspensionConstraint,
VolumeCapConstraint,
)
from pipeline.portfolio.construct import construct_positions
from pipeline.portfolio.research import evaluate_portfolio
from pipeline.portfolio.simulator import ReferenceSimulator
from tests.helpers import (
GENERATED_SYMBOLS,
generated_sessions,
make_generated_alpha_weights,
make_generated_combo_weights,
make_generated_daily_bars,
)
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
def _assert_sorted_by_symbol_date(frame: pd.DataFrame) -> None:
expected = frame.sort_values(["symbol_id", "date"]).reset_index(drop=True)
pd.testing.assert_frame_equal(frame.reset_index(drop=True), expected)
def _assert_metric_dict_is_finite(metrics: dict[str, float]) -> None:
for key in (
"cumulative_return",
"sharpe_annual",
"turnover_annual",
"max_drawdown",
"hit_rate",
"n_dates",
):
assert key in metrics
assert np.isfinite(metrics[key])
assert "ic" not in metrics
assert "rank_ic" not in metrics
assert "ir" not in metrics
def test_tiny_workflow_golden_outputs_are_stable(tmp_path):
dates = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"])
daily_bars = pd.DataFrame([
{
"symbol_id": "sh600000",
"symbol_name": "A",
"date": dates[0],
"open": 10.0,
"high": 10.0,
"low": 10.0,
"close": 10.0,
"preclose": 10.0,
"volume": 1_000_000.0,
"amount": 10_000_000.0,
"vwap": 10.0,
"turn": 1.0,
"pctChg": 0.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sz000001",
"symbol_name": "B",
"date": dates[0],
"open": 20.0,
"high": 20.0,
"low": 20.0,
"close": 20.0,
"preclose": 20.0,
"volume": 1_000_000.0,
"amount": 20_000_000.0,
"vwap": 20.0,
"turn": 1.0,
"pctChg": 0.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sh600000",
"symbol_name": "A",
"date": dates[1],
"open": 10.0,
"high": 12.0,
"low": 10.0,
"close": 12.0,
"preclose": 10.0,
"volume": 1_000_000.0,
"amount": 10_000_000.0,
"vwap": 10.0,
"turn": 1.0,
"pctChg": 20.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sz000001",
"symbol_name": "B",
"date": dates[1],
"open": 20.0,
"high": 20.0,
"low": 18.0,
"close": 18.0,
"preclose": 20.0,
"volume": 1_000_000.0,
"amount": 20_000_000.0,
"vwap": 20.0,
"turn": 1.0,
"pctChg": -10.0,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sh600000",
"symbol_name": "A",
"date": dates[2],
"open": 12.0,
"high": 13.0,
"low": 12.0,
"close": 13.0,
"preclose": 12.0,
"volume": 1_000_000.0,
"amount": 12_000_000.0,
"vwap": 12.0,
"turn": 1.0,
"pctChg": 8.33,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
{
"symbol_id": "sz000001",
"symbol_name": "B",
"date": dates[2],
"open": 18.0,
"high": 21.0,
"low": 18.0,
"close": 21.0,
"preclose": 18.0,
"volume": 1_000_000.0,
"amount": 18_000_000.0,
"vwap": 18.0,
"turn": 1.0,
"pctChg": 16.67,
"tradestatus": 1,
"isST": 0,
"peTTM": 1.0,
"pbMRQ": 1.0,
"psTTM": 1.0,
"pcfNcfTTM": 1.0,
},
])
alpha = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[0], dates[0], dates[1], dates[1]],
"alpha_name": ["gold_alpha"] * 4,
"weight": [1.0, -1.0, -1.0, 1.0],
})
alpha_path = tmp_path / "gold_alpha.pq"
alpha.to_parquet(alpha_path, index=False)
combo = combine_alphas([str(alpha_path)], "gold_combo")
positions = construct_positions(combo, daily_bars, booksize=20_000.0, portfolio_name="gold_port")
fills, pnl = ReferenceSimulator().run(positions, daily_bars)
expected_combo = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
"date": [dates[0], dates[1], dates[0], dates[1]],
"combo_name": ["gold_combo"] * 4,
"weight": [1.0, -1.0, -1.0, 1.0],
})
expected_positions = pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
"date": [dates[0], dates[1], dates[0], dates[1]],
"portfolio_name": ["gold_port"] * 4,
"target_weight": [0.5, -0.5, -0.5, 0.5],
"target_value": [10000.0, -10000.0, -10000.0, 10000.0],
"target_shares": [1000.0, -10000.0 / 12.0, -500.0, 10000.0 / 18.0],
"position_shares": [1000, -833, -500, 556],
"position_value": [10000.0, -9996.0, -10000.0, 10008.0],
"price": [10.0, 12.0, 20.0, 18.0],
})
expected_fills = pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
"date": [dates[1], dates[1], dates[2], dates[2]],
"portfolio_name": ["gold_port"] * 4,
"prev_shares": [0, 0, 1000, -500],
"target_shares": [1000, -500, -833, 556],
"traded_shares": [1000, -500, -1833, 1056],
"realized_shares": [1000, -500, -833, 556],
"blocked": [0, 0, 0, 0],
"trade_cost": [0.0, 0.0, 0.0, 0.0],
})
expected_pnl = pd.DataFrame({
"date": [dates[1], dates[2]],
"portfolio_name": ["gold_port", "gold_port"],
"gross_exposure": [21000.0, 22505.0],
"net_exposure": [3000.0, 847.0],
"pnl": [3000.0, 835.0],
"cost": [0.0, 0.0],
"turnover": [1.0, 2.0502],
"n_positions": [2, 2],
})
pd.testing.assert_frame_equal(combo, expected_combo)
pd.testing.assert_frame_equal(positions, expected_positions)
pd.testing.assert_frame_equal(fills, expected_fills)
pd.testing.assert_frame_equal(pnl, expected_pnl)
def test_generated_alpha_combo_portfolio_execution_workflow(tmp_path):
daily_bars = make_generated_daily_bars()
computed_alpha = compute_alpha(
data=daily_bars,
alpha_name="generated_reversal_3d",
alpha_type="reversal",
lookback=3,
)
assert list(computed_alpha.columns) == ALPHA_COLUMNS
assert not computed_alpha.empty
assert set(computed_alpha["symbol_id"]).issubset(set(GENERATED_SYMBOLS))
assert computed_alpha["date"].min() > daily_bars["date"].min()
assert computed_alpha["weight"].notna().all()
assert computed_alpha["weight"].abs().sum() > 0.0
assert {"ic", "rank_ic", "ir"}.isdisjoint(computed_alpha.columns)
_assert_sorted_by_symbol_date(computed_alpha)
alpha_a = make_generated_alpha_weights("alpha_a", zero_date_index=2)
alpha_b = make_generated_alpha_weights(
"alpha_b",
scale=0.5,
offset=0.25,
zero_date_index=2,
)
alpha_a_path = tmp_path / "alpha_a.pq"
alpha_b_path = tmp_path / "alpha_b.pq"
alpha_a.to_parquet(alpha_a_path, index=False)
alpha_b.to_parquet(alpha_b_path, index=False)
identity_combo = combine_alphas([str(alpha_a_path)], "identity_combo")
assert list(identity_combo.columns) == COMBO_COLUMNS
assert (identity_combo["combo_name"] == "identity_combo").all()
pd.testing.assert_frame_equal(
identity_combo[["symbol_id", "date", "weight"]],
alpha_a[["symbol_id", "date", "weight"]],
)
equal_combo = combine_alphas([str(alpha_a_path), str(alpha_b_path)], "equal_combo")
expected_equal_weights = (
pd.concat([alpha_a, alpha_b], ignore_index=True)
.groupby(["symbol_id", "date"], as_index=False)["weight"]
.mean()
.sort_values(["symbol_id", "date"])
.reset_index(drop=True)
)
pd.testing.assert_frame_equal(
equal_combo[["symbol_id", "date", "weight"]],
expected_equal_weights,
)
portfolio_weights = make_generated_combo_weights("workflow_combo", zero_date_index=2)
positions = construct_positions(
weights_df=portfolio_weights,
data_df=daily_bars,
booksize=2_000_000.0,
portfolio_name="workflow_portfolio",
)
assert list(positions.columns) == POSITION_COLUMNS
assert not positions.empty
assert (positions["portfolio_name"] == "workflow_portfolio").all()
assert pd.api.types.is_integer_dtype(positions["position_shares"])
assert np.allclose(
positions["position_value"],
positions["position_shares"].astype(float) * positions["price"].fillna(0.0),
)
target_gross_by_date = positions.groupby("date")["target_weight"].apply(lambda s: s.abs().sum())
nonzero_target_dates = target_gross_by_date[target_gross_by_date > 0.0]
assert np.allclose(nonzero_target_dates, 1.0)
nonzero_share_counts = positions.loc[positions["position_shares"] != 0, "position_shares"].abs()
assert (nonzero_share_counts >= 100).all()
zero_gross_date = generated_sessions(10)[2]
previous_date = generated_sessions(10)[1]
zero_gross_positions = positions[positions["date"] == zero_gross_date].set_index("symbol_id")
previous_positions = positions[positions["date"] == previous_date].set_index("symbol_id")
common_symbols = zero_gross_positions.index.intersection(previous_positions.index)
assert not common_symbols.empty
assert (zero_gross_positions.loc[common_symbols, "target_weight"] == 0.0).all()
pd.testing.assert_series_equal(
zero_gross_positions.loc[common_symbols, "position_shares"],
previous_positions.loc[common_symbols, "position_shares"],
check_names=False,
)
simulator = ReferenceSimulator(
constraints=[
SuspensionConstraint(),
PriceLimitConstraint(),
VolumeCapConstraint(max_frac=0.02),
],
cost_bps=5,
slippage_bps=5,
)
fills, pnl = simulator.run(positions, daily_bars)
assert list(fills.columns) == FILL_COLUMNS
assert list(pnl.columns) == PNL_COLUMNS
assert not fills.empty
assert not pnl.empty
assert (fills["realized_shares"] == fills["prev_shares"] + fills["traded_shares"]).all()
assert fills["blocked"].sum() > 0
fill_prices = fills.merge(
daily_bars[["symbol_id", "date", "open"]],
on=["symbol_id", "date"],
how="left",
validate="many_to_one",
)
expected_trade_cost = (
fill_prices["traded_shares"].abs()
* fill_prices["open"].fillna(0.0)
* 10
/ 10_000
)
assert np.allclose(fill_prices["trade_cost"], expected_trade_cost)
cost_by_date = fills.groupby("date")["trade_cost"].sum()
assert np.allclose(
pnl.set_index("date")["cost"],
cost_by_date.reindex(pnl["date"], fill_value=0.0),
)
booksize_used_by_simulator = positions.groupby("date")["target_value"].apply(lambda s: s.abs().sum()).max()
traded_value_by_date = (
fill_prices.assign(traded_value=fill_prices["traded_shares"].abs() * fill_prices["open"])
.groupby("date")["traded_value"]
.sum()
)
assert np.allclose(
pnl.set_index("date")["turnover"],
traded_value_by_date.reindex(pnl["date"], fill_value=0.0) / booksize_used_by_simulator,
)
metrics = evaluate_portfolio(positions, daily_bars)
_assert_metric_dict_is_finite(metrics)
def test_generated_workflow_outputs_keep_parquet_schema_contracts(tmp_path):
daily_bars = make_generated_daily_bars(n_sessions=10, include_missing=False)
alpha = compute_alpha(
data=daily_bars,
alpha_name="schema_reversal",
alpha_type="reversal",
lookback=3,
)
alpha_path = tmp_path / "schema_reversal.pq"
alpha.to_parquet(alpha_path, index=False)
combo = combine_alphas([str(alpha_path)], "schema_combo")
positions = construct_positions(
weights_df=combo,
data_df=daily_bars,
booksize=1_000_000.0,
portfolio_name="schema_portfolio",
)
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, daily_bars)
assert list(alpha.columns) == ALPHA_COLUMNS
assert pd.api.types.is_object_dtype(alpha["symbol_id"])
assert pd.api.types.is_datetime64_any_dtype(alpha["date"])
assert pd.api.types.is_object_dtype(alpha["alpha_name"])
assert pd.api.types.is_float_dtype(alpha["weight"])
assert not alpha.isna().any().any()
assert np.isfinite(alpha["weight"]).all()
assert list(combo.columns) == COMBO_COLUMNS
assert pd.api.types.is_object_dtype(combo["symbol_id"])
assert pd.api.types.is_datetime64_any_dtype(combo["date"])
assert pd.api.types.is_object_dtype(combo["combo_name"])
assert pd.api.types.is_float_dtype(combo["weight"])
assert not combo.isna().any().any()
assert np.isfinite(combo["weight"]).all()
assert list(positions.columns) == POSITION_COLUMNS
assert pd.api.types.is_integer_dtype(positions["position_shares"])
assert pd.api.types.is_datetime64_any_dtype(positions["date"])
assert not positions.isna().any().any()
position_numeric_columns = [
"target_weight",
"target_value",
"target_shares",
"position_value",
"price",
]
assert np.isfinite(positions[position_numeric_columns]).all().all()
assert list(fills.columns) == FILL_COLUMNS
assert pd.api.types.is_integer_dtype(fills["prev_shares"])
assert pd.api.types.is_integer_dtype(fills["target_shares"])
assert pd.api.types.is_integer_dtype(fills["traded_shares"])
assert pd.api.types.is_integer_dtype(fills["realized_shares"])
assert pd.api.types.is_integer_dtype(fills["blocked"])
assert not fills.isna().any().any()
assert np.isfinite(fills["trade_cost"]).all()
assert list(pnl.columns) == PNL_COLUMNS
assert pd.api.types.is_integer_dtype(pnl["n_positions"])
assert not pnl.isna().any().any()
pnl_numeric_columns = [
"gross_exposure",
"net_exposure",
"pnl",
"cost",
"turnover",
]
assert np.isfinite(pnl[pnl_numeric_columns]).all().all()
def test_frozen_real_fixture_runs_high_level_workflow(tmp_path):
real_daily_bars = pd.read_parquet(FIXTURE_PATH)
assert real_daily_bars.shape == (36, 19)
assert set(real_daily_bars["symbol_id"]) == set(GENERATED_SYMBOLS)
assert real_daily_bars["date"].min() == pd.Timestamp("2024-01-02")
assert real_daily_bars["date"].max() == pd.Timestamp("2024-01-12")
assert real_daily_bars.groupby("date")["symbol_id"].nunique().eq(4).all()
reversal_alpha = compute_alpha(
data=real_daily_bars,
alpha_name="real_reversal_3d",
alpha_type="reversal",
lookback=3,
)
reversal_vol_alpha = compute_alpha(
data=real_daily_bars,
alpha_name="real_reversal_vol_3d",
alpha_type="reversal_vol",
lookback=3,
vol_window=3,
)
reversal_path = tmp_path / "real_reversal.pq"
reversal_vol_path = tmp_path / "real_reversal_vol.pq"
reversal_alpha.to_parquet(reversal_path, index=False)
reversal_vol_alpha.to_parquet(reversal_vol_path, index=False)
combo = combine_alphas([str(reversal_path), str(reversal_vol_path)], "real_equal_combo")
positions = construct_positions(
weights_df=combo,
data_df=real_daily_bars,
booksize=1_000_000.0,
portfolio_name="real_fixture_portfolio",
)
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, real_daily_bars)
metrics = evaluate_portfolio(positions, real_daily_bars)
assert not reversal_alpha.empty
assert not reversal_vol_alpha.empty
assert not combo.empty
assert not positions.empty
assert not fills.empty
assert not pnl.empty
assert np.isfinite(combo["weight"]).all()
assert np.isfinite(positions["target_weight"]).all()
assert np.isfinite(pnl[["gross_exposure", "net_exposure", "pnl", "cost", "turnover"]]).all().all()
_assert_metric_dict_is_finite(metrics)
Generated
+240 -2
View File
@@ -296,9 +296,13 @@ dependencies = [
backtrader = [ backtrader = [
{ name = "backtrader" }, { name = "backtrader" },
] ]
joinquant-browser = [
{ name = "playwright" },
]
[package.dev-dependencies] [package.dev-dependencies]
dev = [ dev = [
{ name = "coverage" },
{ name = "pytest" }, { name = "pytest" },
] ]
@@ -310,12 +314,16 @@ requires-dist = [
{ name = "click", specifier = ">=8.0.0" }, { name = "click", specifier = ">=8.0.0" },
{ name = "matplotlib", specifier = ">=3.7.0" }, { name = "matplotlib", specifier = ">=3.7.0" },
{ name = "pandas", specifier = ">=2.0.0" }, { name = "pandas", specifier = ">=2.0.0" },
{ name = "playwright", marker = "extra == 'joinquant-browser'", specifier = ">=1.61.0" },
{ name = "pyarrow", specifier = ">=14.0.0" }, { name = "pyarrow", specifier = ">=14.0.0" },
] ]
provides-extras = ["backtrader"] provides-extras = ["backtrader", "joinquant-browser"]
[package.metadata.requires-dev] [package.metadata.requires-dev]
dev = [{ name = "pytest", specifier = ">=7.0.0" }] dev = [
{ name = "coverage", specifier = ">=7.14.1" },
{ name = "pytest", specifier = ">=7.0.0" },
]
[[package]] [[package]]
name = "click" name = "click"
@@ -498,6 +506,119 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" }, { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" },
] ]
[[package]]
name = "coverage"
version = "7.14.1"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/54/fd/0ab2772530e946e1be1abd0bc09e647ec9b02e88f0867857601fefca8953/coverage-7.14.1.tar.gz", hash = "sha256:30c08f7d90415aa98b3c990385dea2939b0da55f38515e5b369b83655f8523be", size = 920132, upload-time = "2026-05-26T20:41:36.783Z" }
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/92/69/0d2ef01ff4b8fcecd4cba920d11e92fa4f96ae412441d3b56a90a258e69b/coverage-7.14.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:3e3680291c4a1d0dadfa84a2c459576a4af5133abb617905714339a0c73138cf", size = 219722, upload-time = "2026-05-26T20:38:14.002Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f8/ae/9afdeaa31b9d9ce98124b6abf8bb49119bf71aecae04f8567c189d91299f/coverage-7.14.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a5274669f37f2343635a347b91a60777621341ab3378e9c6ac9335eee704bddf", size = 220240, upload-time = "2026-05-26T20:38:17.424Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/69/c998589871df7ea7dba865cc5ee32b5a3e1d47ba6c68ef91104c7c46fa5e/coverage-7.14.1-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cfe5a5fec635799ef33428f1e5e61bafa45a92a96190ba731561ba558ccc214d", size = 246981, upload-time = "2026-05-26T20:38:19.266Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/10/1c7d04c13040dac531d21b712bbe08f902e6dd9b58f5d77875c4d030f8f2/coverage-7.14.1-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:62a9f70b52e0b5a95cfef4a5c5641b06983cadc5e538a3feeb5c00211f523ac2", size = 248812, upload-time = "2026-05-26T20:38:20.75Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c1/65/2a38a4607ef27cadcfbcee034dba5830ae2569f90144a0f4c7dbf47d30b0/coverage-7.14.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3c18ebc343e15be53049b3a2dce38fe82d58f37e20ab9094b3a39c0aa4f6bb47", size = 250675, upload-time = "2026-05-26T20:38:22.159Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/a2/a446ed9752a4a59b79e0fb6cbb319f6facb2183045c0725462625e66f87e/coverage-7.14.1-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b84ffdf877644e7096aa936991efeed873f7f3df57b9cd001312b7668ab08550", size = 252590, upload-time = "2026-05-26T20:38:23.63Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9e/fd/e81fbd7ba752365546e9842b1cbdaad3d6919d2a522c590aef16a281ec5e/coverage-7.14.1-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e854312c4103f2ad4c0dc023b69b77ebfd2c89db5f86c4c94dc2353f9a92167e", size = 247691, upload-time = "2026-05-26T20:38:25.057Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/53/35/f3c26fdaae9ea937d154ca4d372e5ea0a4167ff70d36c6074ac2eacb2f83/coverage-7.14.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:c643734307300234fafa36bf2a040a7235f8f177ea1fd6ec1423aea6fb7b929f", size = 248716, upload-time = "2026-05-26T20:38:26.406Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2e/14/940b6c49551fd343e8507ee2b0ba7af5d0aa04ed5bf768285cb7c72a9884/coverage-7.14.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:84ac9499e48700399a5dd0ea7085b5091961fec52c68d66b4ec0d3cf7f4441b1", size = 246721, upload-time = "2026-05-26T20:38:28.282Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/aa/2c/40fc0634186c28292a662dff578866b3913983d6c375a3c2a74020938719/coverage-7.14.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:7f02d09f70776579b926d889a4c9c235070a1f47c40458aeaca563fae5acfdb5", size = 250533, upload-time = "2026-05-26T20:38:29.753Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/e3/2c26bf1e811f9df991ff2a9bdddebdd13ee0665d564df7d05979f9146297/coverage-7.14.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:ce66d8e46da2bb5ee313a745cbd2e391d319176c1f7a9451bfcd3a2fb920859b", size = 246990, upload-time = "2026-05-26T20:38:31.516Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a8/b0/060260ef56bd92363ebdce0c7095ce422b06e69aae71828efeca473ab1ca/coverage-7.14.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:c912c259304cfb5ee584481cfb7ce1ff932b4d61e6c9140b8f19cb7b5ed82332", size = 247593, upload-time = "2026-05-26T20:38:33.065Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/63/f3/501502046efeb0d6d94b5ca54941d95f1184183dd6bdb7f283985783bb4a/coverage-7.14.1-cp310-cp310-win32.whl", hash = "sha256:1238cb94638e610e972c60dac68e813f868dc7d6e982535270558443058d9d59", size = 222330, upload-time = "2026-05-26T20:38:35.36Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/5d/1bf99f2c558f128faf7906817ccbdb576ba815d3b41ce2ac1719b70a3663/coverage-7.14.1-cp310-cp310-win_amd64.whl", hash = "sha256:fc459e5d73be2d6332fcfe8dbf3d8994671fe33c700f4565988ecfa511547253", size = 223261, upload-time = "2026-05-26T20:38:37.196Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/d7/477ad149490e6cb849f28abea1dabb9c823cea72e7500c81b4240ce619c0/coverage-7.14.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:478b5bcd63c2e1357c5c7e16c070690df7b07f676b1c114d7b93e533c664309f", size = 219848, upload-time = "2026-05-26T20:38:38.715Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/91/82/a5eb47257c50601bb7b9a9d2857c67b7a3a85ad74180eb2c98bb1fbe0ce5/coverage-7.14.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a24a81f9715ee42ef59a316cc11611c98fe23920f7c81861315c9f3ff4a230f4", size = 220354, upload-time = "2026-05-26T20:38:40.232Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/8b/78419b5391a5cb706b6544390507e469d83ffc9a8248b02c4011aceb9365/coverage-7.14.1-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:196a13319ad88d6d8ef5ab489ec4f44ddde2143c0c7d5b27786f6c3ffd56a7e1", size = 250771, upload-time = "2026-05-26T20:38:41.782Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/77/63/e77aaacd491182210d639636b7a8bba23ffffa9b82aa3762da9431855fa9/coverage-7.14.1-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3d452fd08b5c72c5167c93e6867b5c08500bd40f2a21e1e854a500550b6cc36f", size = 252683, upload-time = "2026-05-26T20:38:43.305Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/65/1c/a022e3cfbec2ac241640003cb3a817e161d9c7f5aa9b49173756cdc03204/coverage-7.14.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:23bf7fa51ac02e07fc7c96849b82946da47ae862dc8f86d183b2a4864fc38129", size = 254791, upload-time = "2026-05-26T20:38:45.361Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/61/d6/967e408aca4c1ceb88cb0cc677169110ae7f5995fb5eaf5fb1f5a1bb8f5d/coverage-7.14.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bcaa50684dcaadfa599ac48f81103c756d791cfd85c97203d2217c593d48b860", size = 256748, upload-time = "2026-05-26T20:38:46.91Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/be/869188f7fe28638078ec479331ace6dc5f7b40b7153eb616f47ab79404d8/coverage-7.14.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4ea1c034f95c9b056e856b794630b17f9fa3d57e4800ff1e503d3be0f9c9078c", size = 250907, upload-time = "2026-05-26T20:38:48.493Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/07/aa/adb7d3b4278d690e68703abcd76ab1b948242e3668d921711551b78f9ddb/coverage-7.14.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c7e057326434e441306226fbeb5d1aaf14a2637efe97ba668306635835f32ad7", size = 252483, upload-time = "2026-05-26T20:38:50.074Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/61/331c74103c62dcb0c4b9b3a0de9a61aca016208b0a90f109592a9f9ecc28/coverage-7.14.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:59baf88468dbc8d63b1887afd92bda52e40bb1561696e5819670601403810cec", size = 250545, upload-time = "2026-05-26T20:38:51.613Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/b6/c5dae3c104d89be04828f61810e6b3473825482e4c288cc4ed04553e08ae/coverage-7.14.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:d34d75f892b3ab73ba11cab5442cce7b3e168fd64162b16f0e1e0d09c508edef", size = 254310, upload-time = "2026-05-26T20:38:53.503Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ad/a1/2b9d5863e3b83c01ad8199e3c597802fbb3a9dc90b058885804c20296d31/coverage-7.14.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:3a56abc20a472baf0304c455721bc601477440d28ecfde8a03dde79ede07e0df", size = 250266, upload-time = "2026-05-26T20:38:55.414Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/5e/0e511fbdb269359be26fe678a1c3fa1f2aa2a01573cc3f54268c8d6d4797/coverage-7.14.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:6a3cb83d1552c0cd1b4906655b6a33fd4a8473229633a901c6b73bf86914dee9", size = 251174, upload-time = "2026-05-26T20:38:57.141Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/85/10/e55307b622b3dd9671cb321824502dc10f93e72f2802b9946159a8edadeb/coverage-7.14.1-cp311-cp311-win32.whl", hash = "sha256:10274a1fbeb8ec5d72966e17bb198a3104257aca4ac09d98667c5f8aca8c8548", size = 222354, upload-time = "2026-05-26T20:38:58.727Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/cf/107421693cfb71e4f1ca5bf70443f64d4161878068d07a3e51c7ad21d17b/coverage-7.14.1-cp311-cp311-win_amd64.whl", hash = "sha256:87ebdf787d4888e3f3f2d523eadc6e18c6d18c6d0eb173801a189641627fb37e", size = 223290, upload-time = "2026-05-26T20:39:00.413Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/1d/3e3644585eb29e9dafefb19555078529a4d7cce12bd21929664eea989277/coverage-7.14.1-cp311-cp311-win_arm64.whl", hash = "sha256:dd34767fa19848d35659ffc0a75314f58c7af3f1cd87ec521e8292a1238398a3", size = 221953, upload-time = "2026-05-26T20:39:02.159Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/b7/bdbb725ba02c5b42825b200c940f38b7a54fcad24627b7192f78f8110d76/coverage-7.14.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a06c76364a9360e33d6d23769aefdf7f66f38e2ffb60ceb1baaa4989d83b695c", size = 220022, upload-time = "2026-05-26T20:39:03.702Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/72/81/fdc0898a55c6219223291ec1a1fe89966ef212ce82276aa0899df84b5de0/coverage-7.14.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fad54e871165f6ec2f536063ac74c3104508a12963e64072ba44bd822de52b0c", size = 220379, upload-time = "2026-05-26T20:39:05.381Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/72/de048c4a25e13bce59ac6a339351c10bdf2515e07459afcdaf04dc3143a2/coverage-7.14.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:84b535f00655ecafe1d929d1fb00ed5d6fa3051ea643ab2c161a3887b86f294b", size = 251888, upload-time = "2026-05-26T20:39:07.367Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/28/30/300c343f68beb9d4cbb64ec81e58c5b6b80b56927f72d2b38654ac26e013/coverage-7.14.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6b6b0853b895fe0e98cbfc580d1ec3393d9302b4b1e96a77b3f5c91fdab899e6", size = 254624, upload-time = "2026-05-26T20:39:09.037Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b1/ed/7b25642496e8170b6bac14adce00537c6e5fa2d586159401a4de3e8b49e6/coverage-7.14.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:442cc9c952b2df400cda54bb04ab87330cf2cd08a8692cbbea36773531eb6f37", size = 255739, upload-time = "2026-05-26T20:39:10.889Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/a2/abd210b8c4e29c24e4624916db97bb519097a91034aaeb767f937e7da794/coverage-7.14.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8270544c361ed405a27a060dbc9ed2c124b084d96dfdc2d9a2510482aef981ad", size = 257998, upload-time = "2026-05-26T20:39:12.722Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/24/7c50beed3792fe62f6ce0545c6686ce83379719e2c0276179333d97eae92/coverage-7.14.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:48b283b1dd6372e8de2a7a9a4c4d5dc06f4d4fd209b876f3c88a7a205a0c8f84", size = 252296, upload-time = "2026-05-26T20:39:14.259Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/15/05/0f874628ebcbfc77ead559ff210281ef06a97db08481832e7dd39274a135/coverage-7.14.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5b0c99ba93a07d56f6df340bb79be53202a082b2fdb81bfe6190b741a3470d54", size = 253658, upload-time = "2026-05-26T20:39:15.923Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/99/6f/ca6ad067364b337ef997802115e7ecad2abd2248b05471464b0dea02b4d4/coverage-7.14.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:e471bc5769ff073b058cfadb0d736b56ce067c8560eabeb0da88462df98c23e7", size = 251803, upload-time = "2026-05-26T20:39:17.537Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c0/30/b9b4d377cd9f40baf228068f5a81faf8450c6228503011bd499708483a50/coverage-7.14.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f497a1ea81d4cd7c10ddcaa685135b9aabd291af3d55775a9ddf3cb7a364cdd9", size = 255873, upload-time = "2026-05-26T20:39:19.414Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3c/21/7c721a9e5e6bb88547d30a787aefb97512d3f54c1324c7488d9b3743f7f9/coverage-7.14.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2222be86d0b54f5dd5a38f45f17f315f737245e857bf0bdedc70734f84a13c02", size = 251372, upload-time = "2026-05-26T20:39:21.169Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9d/8c/f8ae5a2200130e1503cd7661a6cd3b2b7bacef98277fbf3571fb13f8b766/coverage-7.14.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:85e85586565842f6932abebd4c18bcb1074223dc0b3576e7d173ca710622813a", size = 253245, upload-time = "2026-05-26T20:39:23.097Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/34/62/70a9024672a5f6910517d9628c52c9afbdd3cf8f46426af52bb148a56fff/coverage-7.14.1-cp312-cp312-win32.whl", hash = "sha256:4a28fd227808366b196a75476dced2eb35b351d6766ba9c858dc93319e87f4f1", size = 222567, upload-time = "2026-05-26T20:39:24.868Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/81/8b7cd386839b039ebe1855733b9f9449a8dec5d79564018234f185a7fa70/coverage-7.14.1-cp312-cp312-win_amd64.whl", hash = "sha256:54acdb6674a4661768d7bf7db32dfb9f46ab1d764f8aba6df75ce1a6a088724e", size = 223372, upload-time = "2026-05-26T20:39:26.603Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ae/ba/b44d472022f620d289d95fa830143235c0c36461c6f2437ea8d51e5481ed/coverage-7.14.1-cp312-cp312-win_arm64.whl", hash = "sha256:99cd41ff91afd94896fea3bc002706b6ae4ce95727d06e4a0f39c0a8d8bd8b1a", size = 221989, upload-time = "2026-05-26T20:39:28.242Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/9e/5f6d56327c62b185225d145191c607e07515294a0aa6338e58805cd4a5ac/coverage-7.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:be9f2c802dcfce3f71298303aa5dad0dce440a76c52f2f60dacd8656dab78793", size = 220044, upload-time = "2026-05-26T20:39:29.902Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/75/92/e82aca356744cbbc0f77a0b623e38918c1872361963413a3bab5d0340393/coverage-7.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6223a72fd0e4c7156353ec0f08a5f93623e1d3034d0e2683b9bb8ea674131b1d", size = 220412, upload-time = "2026-05-26T20:39:31.561Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/c9/385bde0bf7ed0f4bf3a7ee5367060a86b5d218718cfd6fb943c0f836b34f/coverage-7.14.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:7279d2110a28cebc738b6459ecda2771735a4c18465fbbd36b3288fe5ed92247", size = 251412, upload-time = "2026-05-26T20:39:33.337Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/8c/23faf6a2343a0d17f960a4bd56c43bc7eb4cf312f774dd6ceebd82c7d8fc/coverage-7.14.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9eeb3fcbc13ba40dfbdb22d01d196a28e9cef9ed4c29b60061a1e0e823a9929d", size = 254008, upload-time = "2026-05-26T20:39:35.009Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/42/06/36f4aa9ca8a815e6036156e80706a67828bb97bd826948244f6996dda957/coverage-7.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f0cfc27c539f07cf5c0a4cfe211d0b6cae039f8f40526dbaa71944e64b50a7b", size = 255241, upload-time = "2026-05-26T20:39:36.71Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/79/95266316352f90f6b1c6736bb413302edfde2453fb32422d3911642691b3/coverage-7.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:221c70f316241a78e77e607c227cefc8808d4e08f28d99c04f35694690e940be", size = 257373, upload-time = "2026-05-26T20:39:38.412Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e3/9c/58316d1f66c488b5fca8a0eb3e98348807813efa8a0d0833b9021be27488/coverage-7.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:da028256b04ec30e5e0114b6f76172938c313991f0a2d3d894271315cf5d5e43", size = 251635, upload-time = "2026-05-26T20:39:40.268Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ef/5a/ca2398a568e16fed7bb713e84ba3603a7164fb65779abe645c565ec890d5/coverage-7.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:76a085d7005236a767e3426148b2c407e53ad61695c562f8a81da2d373324901", size = 253373, upload-time = "2026-05-26T20:39:42.145Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6e/2c/0396562c32deaebe7be51d865b3a41e9a87d7561acafe1a28f53b07e019a/coverage-7.14.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b553d04b5e778a8e56d57eb134aff42a92718ecba45e79c4764ecfa40efd92ff", size = 251341, upload-time = "2026-05-26T20:39:43.907Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fd/8f/a94f9221184c9cae1ee115820e3798e48b6b17777a9f19e46fb9a0c8dc74/coverage-7.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:46f714d2fb8ae2f4f29f23ada7f1e79b759fff5a70f94a1dac23af204c3ec9e4", size = 255497, upload-time = "2026-05-26T20:39:46.166Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/69/505d70e47db1eaebcd002c39759707621ef184cd6b1ae084d9f41293f323/coverage-7.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:1896f5e19ff3f0431c7ce2172adc54890fd97f86b59ced8ca1649145d9ffe35d", size = 251159, upload-time = "2026-05-26T20:39:48.03Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e0/aa/58681c383aa33a9d2ed40a02d7a22fbf780d1fa4d575396365777828198c/coverage-7.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:62fd185ef9df3c33d1c8178c5af105f762afbad96038de9a4ae100aa6297ca33", size = 252934, upload-time = "2026-05-26T20:39:49.872Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/eb/fd/11c928cd6bdffc7074bb5965c173d9ebf517fb00205e1da524b98d29ef92/coverage-7.14.1-cp313-cp313-win32.whl", hash = "sha256:ab4af6352741a604c431c6072fce5bee33bf0f20dc7a56618d6bf6bb89e9810c", size = 222584, upload-time = "2026-05-26T20:39:51.68Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6f/92/fb416fc26d340dcba19518c418d6048e913186e17243982c5e435e41fa7a/coverage-7.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:7af486dabe8954d03b087f0021540897afe084f04e16ff5579e08cc46f871416", size = 223394, upload-time = "2026-05-26T20:39:53.472Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/73/c6/02d56e3867972f77d5036de924643f26c056e848f00452cafb4dbc3c29b4/coverage-7.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:2224f89ffd0c5605ccce1ed7a584da162bc7c55f601ab1c946bc9de31a486b42", size = 222015, upload-time = "2026-05-26T20:39:55.374Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/4d/9e/fcc77914050df73f7662fa1f00902774c79c075a8388ab334074574bf77e/coverage-7.14.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:de286598cc65d2b489411174b1faec2f5a7775fb3201fd925db2a76b4030f37d", size = 220733, upload-time = "2026-05-26T20:39:57.189Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f7/67/2963cbdaf5cbadec44efa3a1e39eaa1f02df4079585f05387607a221e126/coverage-7.14.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:042c46ded7c288aeb07cf14a28b6c1e10b78fcba40171c3fa1e939377eeef0b5", size = 221086, upload-time = "2026-05-26T20:39:59.019Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c8/c5/8701645574e11881f2f47d8930f98bc48b5d43b25eb5b4430dfc4a2f9f48/coverage-7.14.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:f4ddbe407477f04c45115d1a4e5bc480f753553b534d338d4c3358b1cdd0ea52", size = 262381, upload-time = "2026-05-26T20:40:00.822Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7c/28/7a64d73598263e0c5abd5084211a8474488d31b3c552ff531c719dfcff62/coverage-7.14.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d13e6725992e2d2fd7d81d4f5241952d13740121dfd501da09201be39b2c003a", size = 264458, upload-time = "2026-05-26T20:40:02.506Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fa/d8/4969179db9f7eb4df218e69540adf829d1c835f59452513d065d15446802/coverage-7.14.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f747dc8edcfe740130f28f32f3995e955494285717e86ee25af51db2219df08a", size = 266884, upload-time = "2026-05-26T20:40:04.421Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a6/78/a45d5794dbc9bafd97afc96a4377c86c7820d78b6cf51b89bc1d4e919275/coverage-7.14.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ced2f09ef276fd58611a1ef502164ad266d2b75174e5a40cabbdb4033f9f6cf2", size = 268022, upload-time = "2026-05-26T20:40:06.298Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/21/cb/4f5e354e9e3e67af96bd4e57113e6db6b22298c7168b13eec408a549903d/coverage-7.14.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b84800013769a78ccb9ef4659402e26d06867e337b61ec365f77ad008adea80e", size = 261631, upload-time = "2026-05-26T20:40:08.226Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ec/49/eced49af4cb996d5d8b7e94e736175c513e4facd3398507b89892b4326d8/coverage-7.14.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:ea8cd6ca0ee9f616aaef3afc6882e32c2cbf18b00d96313ffd76af650574034d", size = 264443, upload-time = "2026-05-26T20:40:10.137Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f1/d8/5603a88a7c5913a6b54f6cb1a8c46f7b39cbb30f27cd3f492908da09b2d7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:aa5e304a873fabddc11e484e9b6b738bd38bd7bed17b09aa84eecf5332e8b8bb", size = 262069, upload-time = "2026-05-26T20:40:11.999Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f0/59/2ae3cb79da554a06c8619d6c88ea19dd1e4aed4b834b6a83bb1fa243bdc5/coverage-7.14.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:5a1c5215be81035e629d5bc756650634d0bf31991038db7a0eccb90f025ce16d", size = 265780, upload-time = "2026-05-26T20:40:13.858Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/af/5f/b130c1dc999031f2648bd25317fbce505ad8d5562079b4ed81e736a84967/coverage-7.14.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:79058c47dae6788504b5effb319961bcd72d7240551464b91d474bc0ed186d69", size = 260970, upload-time = "2026-05-26T20:40:16.142Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/87/d1/ec13ccddeb48ec963bdfa72a11224bac2584bd045ba13beca82f8113e9c7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:370c5afae3fa0658e11694a32b24c2778f6bc2d17718121f94ee185e69f26b54", size = 263157, upload-time = "2026-05-26T20:40:18.382Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cf/c2/cd91ead503045161092d3845f7bb95ea2f25131ce96d3e314dd835d91b9c/coverage-7.14.1-cp313-cp313t-win32.whl", hash = "sha256:3758dd0a7f1fa57365ef2e781df0f0731d38b6e3772259d13dae4bd8a958d4b1", size = 223259, upload-time = "2026-05-26T20:40:20.381Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/9f/1e28d97e6bd2c76b07f38b7c02870f1371255ff6717f54eca578fcbbdd0e/coverage-7.14.1-cp313-cp313t-win_amd64.whl", hash = "sha256:6ff665fb023a77386fe11685190cee1f60a7d635994a30d9b0a061533d470fce", size = 224320, upload-time = "2026-05-26T20:40:22.316Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a9/e0/d936e908f0e1efa55e52b91e01b52f1055cef5e1ab2718493390ed8e2fb8/coverage-7.14.1-cp313-cp313t-win_arm64.whl", hash = "sha256:17a5a241e5997621a956a7f402a7433ef4221e5152809b785bec79e2323799f1", size = 222577, upload-time = "2026-05-26T20:40:24.894Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d6/34/fc2f101b151af3799a101f0550b0454aa008afdc0add677394ec4aa8ea10/coverage-7.14.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d5ed429d0b8edaac649e889b4ffcedb6c80b06629a3f93050e3dddfb99235bee", size = 220091, upload-time = "2026-05-26T20:40:27.249Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/a7/1ebae2ab5b961b5c79bb09fe7b3ac99edb190d8be4a8c510b2cf66f46468/coverage-7.14.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8011224a62280e50dab346960c03cf47aca1a1e09e608c0fb33fd6e0cc8e9500", size = 220421, upload-time = "2026-05-26T20:40:30.084Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5e/90/92aca9cf0acc95123c96cd1eb1f08917897a7f5dee01e15738922971ec31/coverage-7.14.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:12c42ec1e14f553c4f817e989365982e646e27211f10a0f717855b94a79c8906", size = 251466, upload-time = "2026-05-26T20:40:32.542Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/26/2b/78048cbe3b999f6cbf9cc0d90abba6a88a3e0863a8c1c6cbc762f3f8802f/coverage-7.14.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:06144cd511cf2624873a035c5069cf297144f6e77a73ee3d7a55b605ec5efb42", size = 253973, upload-time = "2026-05-26T20:40:34.473Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/21/c2e33b29d1cfde484a19d437afc343c6cd30b08d78cbbf9f5aff14e57b2b/coverage-7.14.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a311d8e1da24be5c1ccf85cbfb06315dbaa1703d5a1eab3f6432c72b837917c8", size = 255318, upload-time = "2026-05-26T20:40:38.154Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/ee/aad2f108d63b769121005302f16bf66db8625c88ceaba466942e09a2607e/coverage-7.14.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c79cead5b5bc584d9c71451cb984d0e3a84e0c0937379c8efcbf27c8d661b851", size = 257633, upload-time = "2026-05-26T20:40:40.164Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c2/f8/11a2c29b4fd76d9849f81d0bb812ec0017a9396df3217214e38934a8c837/coverage-7.14.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:dcbf65f1f66a26cdd88c35cf68fb4729c5d1cd2e88added72420541dfb212034", size = 251488, upload-time = "2026-05-26T20:40:42.631Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/b8/9a5820de4b8ac2b71d85e3b5fb49108d7469c665f0e2ad0dd7569023e305/coverage-7.14.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fd86572566fb40189a8260446158235159bc7a82dfbc87a3b39cf4fb57fcec1c", size = 253329, upload-time = "2026-05-26T20:40:45.208Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6b/ff/f33e4823667e27548e8fd8df44217515303f9808d0ff29817db56f87d990/coverage-7.14.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:7771b601718fdde84832c3a434ca9bbf4ae9adbc49d84198b4110700c3c77c36", size = 251291, upload-time = "2026-05-26T20:40:47.502Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/68/9b/489db0ebb209054766b90a9014a45f6d26eb724c02ec21311c3733b5a644/coverage-7.14.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:39b21e212c55af06fa375e3dbf90a8a8e38792f3a910c580066d23563830ddd5", size = 255564, upload-time = "2026-05-26T20:40:49.372Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/b5/16bc2d4c2409b23c7737edb68c83bc89e345f378050549fe1d75ac7d34d5/coverage-7.14.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:f2302660e32562a532b442480121aef8aa61a5bdb20b30bf0adab29f10a5a4b4", size = 251107, upload-time = "2026-05-26T20:40:51.677Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/0c/2629997469a00cd069d588a41c9dc887610f2775ae89d250c4791e65272a/coverage-7.14.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:03a6f93c1ec3b7f2e77b5dbcc5573a2c21f12529a5c6bbe0f16f72303cc2fa4d", size = 252764, upload-time = "2026-05-26T20:40:54.267Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d2/ee/f78d63c8f079e0d7211c7e2401fa17e311514534ba61bae03e4b287ce4ab/coverage-7.14.1-cp314-cp314-win32.whl", hash = "sha256:8a3ce026d73290f42f08dafecbd82c193a74df280461fbf97300fec51fd133ee", size = 222837, upload-time = "2026-05-26T20:40:56.496Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/dc/b9/be539854f93a70dfbeec69117f33ec70dc42ff0b65b5b07ab8d40d04228e/coverage-7.14.1-cp314-cp314-win_amd64.whl", hash = "sha256:114c95ef29302423b87d159075805f4ab973254a2638a5d7d046c94887cc87d7", size = 223650, upload-time = "2026-05-26T20:40:58.351Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fe/9e/24e2842fef40f35ac82ba3a7719c8023d011bf3bf652d0675316a9d088a1/coverage-7.14.1-cp314-cp314-win_arm64.whl", hash = "sha256:a07891c3f4805442b31b71e84ba3cf29ed1aa9a428284e06deeb4b23e5b46343", size = 222218, upload-time = "2026-05-26T20:41:00.321Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0a/1d/ac0a9df5fe31c1e8bdd658074905fc12844a05c1a7e3fdb8417e97c31e23/coverage-7.14.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:1101a5ebb083aecb625ebb6209d4105b58f647b093cb2dc8122d7b33f743cfe1", size = 220822, upload-time = "2026-05-26T20:41:02.281Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/32/cf/f964fd9aff20323f9f1a726c97135f8a76bcd87b92dad141a456a43f3c64/coverage-7.14.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:851b9e1e4e8a4608e77c79714b2e77c0970d2ed7202a05e92ae407817481887b", size = 221084, upload-time = "2026-05-26T20:41:04.593Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d8/5e/7e5ef2aba844de2b80d678619fcf0841b42e3f37f16411226f3fe4c1016f/coverage-7.14.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d5b89cdfb2ee051b71e8c3c70bd81a9eff81100f736a269136fe1a68efe00474", size = 262454, upload-time = "2026-05-26T20:41:06.641Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/64/62/75809bded87015cc4935524218a2a8ed8dd1a8498bfed30a2f4f7a4b4d34/coverage-7.14.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0177614a0370f227888b4e436a7c55686d6a9f90eb1ade2b624ba685a1686e86", size = 264578, upload-time = "2026-05-26T20:41:08.556Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f3/42/d33392dc14633525012d2d504fa1a33b05538bf535f5c1d64675e5754b78/coverage-7.14.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d69af5dea2de76fc485a83032a630523f985198b7e25be901ec60181587b01e", size = 266981, upload-time = "2026-05-26T20:41:10.824Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2a/49/0157c4428c2aca7f1e09d5565930586fd5ae36f1655f08b0daa7cf1fcae1/coverage-7.14.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:35ab22d91de736e8966b980dc355cbcdd2c6dbbcfe275f9a2991bc8a91b3df65", size = 268112, upload-time = "2026-05-26T20:41:12.966Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/96/26/86b9ce71f4092b1ed325ce1421698081df1286b833400b6836912834d6e0/coverage-7.14.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:357d4e32935c36588aaba057d734fa32428c360c9fc2e4442afbf1b646beee6e", size = 261558, upload-time = "2026-05-26T20:41:15Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/20/4c/c311210c5472cf5401d8422b0d7812cdd520f24417673afabda6c323faca/coverage-7.14.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:51bd64741cc6fa065abd300ede1afe5a5291ece9c31da8b24884deda48bcc3f8", size = 264447, upload-time = "2026-05-26T20:41:17.369Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/71/59513f8710ed3e6b0ac0a050a5b7e977bb9c9e880354863b5d00d8809256/coverage-7.14.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:9132cd363a68a4c3daa7c8704a654b1e39d3360f6f5b8ddd470608a945236c07", size = 262048, upload-time = "2026-05-26T20:41:19.309Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/84/8d/bceed32dc494f5bbf50f775cd2e78ca814953942b5ea28d3c1c3ac316f14/coverage-7.14.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:07c6290b1697b862c0478eab545eec949a0d0e4d6d03497f446d706da3b4f2de", size = 265781, upload-time = "2026-05-26T20:41:21.559Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e7/c5/9348fe40dbfd4991aaf78df2c6c3098bfb2cc834d1fd362a64b4efef855a/coverage-7.14.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:5ea0c297e27133853b4d8a3eb799bff5a2dbd9f2f41537a240d337ac9b4df890", size = 260896, upload-time = "2026-05-26T20:41:23.428Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/92/1ea0f03929da7cf87206b1fa24f4c8e9c158be0455481af29ec0a1f3503f/coverage-7.14.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:01b7733daad0237daa01ef80fe2dfceffc911e6a17fa7b55d14aa8214eaaaecd", size = 263214, upload-time = "2026-05-26T20:41:25.419Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/a9/b2493c054c0e01a643266742ab45e15744e60743f9260cd930c7142b1124/coverage-7.14.1-cp314-cp314t-win32.whl", hash = "sha256:6adc5a36984624a70bf11d7184e20fa0a49aa7c47ffab43804106a1a695ea22e", size = 223624, upload-time = "2026-05-26T20:41:27.795Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/bd/3e1e6a57fccd2d7c83fcdf338e93ba98eb85c6e877dd34731ac585375490/coverage-7.14.1-cp314-cp314t-win_amd64.whl", hash = "sha256:ddf799247318f34dbcd2efa8c95a8d0642674e926bb1774cf9b63dfd2a389d1c", size = 224728, upload-time = "2026-05-26T20:41:30.098Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bb/d7/31066cf1d2f0c6c797fce911bcfa01dd35642dc6da992a950256097c5860/coverage-7.14.1-cp314-cp314t-win_arm64.whl", hash = "sha256:145986fe66647eb489f18d9a997567a3fd358584c4b5a808769113abc07466af", size = 222752, upload-time = "2026-05-26T20:41:32.123Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/3c/1a983b9a745d7f83d53f057bcc5bf79ba6a2bbc08266b3f0c7d6fe630c9b/coverage-7.14.1-py3-none-any.whl", hash = "sha256:a252f21c27e38347e60111a3266b03827422a7d5525951aceee313aa68bab1d2", size = 211815, upload-time = "2026-05-26T20:41:34.078Z" },
]
[[package]] [[package]]
name = "curl-cffi" name = "curl-cffi"
version = "0.15.0" version = "0.15.0"
@@ -627,6 +748,92 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" }, { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" },
] ]
[[package]]
name = "greenlet"
version = "3.5.3"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e2/f1/fbbfef6af0bad0548f09bc28948ea3c275b4edb19e17fc5ca9900a6a634d/greenlet-3.5.3.tar.gz", hash = "sha256:a61efc018fd3eb317eeca31aba90ee9e7f26f22884a79b6c6ec715bf71bb62f1", size = 200270, upload-time = "2026-06-26T19:28:24.832Z" }
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/a1/1f7f0c555f5858fd2906fe9f7b0a3554fddb85cb70df7a6aaec41dc292c2/greenlet-3.5.3-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:c180d22d325fb613956b443c3c6f4406eb70e6defc70d3974da2a7b59e06f48c", size = 285838, upload-time = "2026-06-26T18:21:05.167Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0a/29/be9f43ed61677a5759b38c8a9389248133c8c731bbfc0574ecdff66c99fc/greenlet-3.5.3-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:483d08c11181c83a6ce1a7a61df0f624a208ec40817a3bb2302714592eee4f04", size = 602342, upload-time = "2026-06-26T19:07:06.908Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b9/42/ba41c97ec36aa4b3ec25e5aa691d79561254805fad7f2f826dd6770587e2/greenlet-3.5.3-cp310-cp310-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1dae6e0091eae084317e411f047f0b7cb241c6db570f7c45fd6b900a274914ce", size = 615541, upload-time = "2026-06-26T19:10:04.909Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2e/8c/231ca675b0df779816950ca66b40b1fa14dbff4a0ed9814a9a29ec399140/greenlet-3.5.3-cp310-cp310-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0f6ff50ff8dbd51fae9b37f4101648b04ea0df19b3f50ab2beb5061e7716a5c8", size = 622473, upload-time = "2026-06-26T19:24:12.786Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/c7/28747042e1df8a9cd120a1ebe15529fc4be3b486e13e8d551ff307a82412/greenlet-3.5.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9bcd2d72ccd70a1ec68ba6ef93e7fbb4420ef9997dabc7010d893bd4015e0bec", size = 615675, upload-time = "2026-06-26T18:32:14.444Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/81/fe/dd97c483a3ff82849196ccd07851600edd3ac9de74669ca8a6022ada9ea1/greenlet-3.5.3-cp310-cp310-manylinux_2_39_riscv64.whl", hash = "sha256:37bf9c538f5ae6e63d643f88dec37c0c83bdf0e2ebc62961dedcf458822f7b71", size = 418421, upload-time = "2026-06-26T19:25:34.503Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cc/a8/b85525a6c8fba9f009a5f7c8df1545de8fb0f0bf3e0179194ef4e500317f/greenlet-3.5.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:73f152c895e09907e0dbe24f6c2db37beb085cd63db91c3825a0fcd0064124a8", size = 1575057, upload-time = "2026-06-26T19:09:00.264Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/03/79/fb76edb218fe6735ab0edeba176c7ab80df9618f7c02ce4208979f3ae7db/greenlet-3.5.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8bdb43e1a1d1873721acab2be99c5befd4d2044ddfd52e4d610801019880a702", size = 1641692, upload-time = "2026-06-26T18:31:41.454Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6b/79/86fe3ee50ed55d9b3907eecd3208b5c3fe8a79515519aae98b4753c3fa1d/greenlet-3.5.3-cp310-cp310-win_amd64.whl", hash = "sha256:0909f9355a9f24845d3299f3112e266a06afb68302041989fd26bd68894933db", size = 238742, upload-time = "2026-06-26T18:20:40.758Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/58/5404031044f55afad7aad1aff8be3f22b1bed03e237cfeabbc7e5c8cfde0/greenlet-3.5.3-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:aca9b4ce85b152b5524ef7d88170efdff80dc0032aa8b75f9aaf7f3479ea95b4", size = 287424, upload-time = "2026-06-26T18:20:31.469Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/bf/1c65e9b94a54d547068fa5b5a8a06f221f3316b48908e08668d29c77cb50/greenlet-3.5.3-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0f71be4920368fe1fabeeaa53d1e3548337e2b223d9565f8ad5e392a75ba23fc", size = 606523, upload-time = "2026-06-26T19:07:08.859Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/c7/b66baacc95775ad511287acb0137b95574a9ce5491902372b7564799d790/greenlet-3.5.3-cp311-cp311-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4d77e67f65f98449e3fb83f795b5d0a8437aead2f874ca89c96576caf4be3af6", size = 618315, upload-time = "2026-06-26T19:10:06.055Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b0/a0/68afd1ebad40db87dac0a28ffa120726b98bf9c7c40c481b0f63c105d298/greenlet-3.5.3-cp311-cp311-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e18619ba655ac05d78d80fc83cac4ba892bd6927b99e3b8237aee861aaacc8bb", size = 626155, upload-time = "2026-06-26T19:24:14.44Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/78/2b/28ed29463522fdbe4c15b1f63922041626a7478316b34ab4adda3f0a4aba/greenlet-3.5.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8540f1e6205bd13ca0ce685581037219ca54a1b41a0a15d228c6c9b8ad5903d7", size = 617381, upload-time = "2026-06-26T18:32:16.077Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/07/7f/e327d912239ec4b3b49999e3967389bcf1ee8722b9ee9194d2752ecd558a/greenlet-3.5.3-cp311-cp311-manylinux_2_39_riscv64.whl", hash = "sha256:d27c0c653a60d9535f690226474a5cc1036a8b0d7b57504d1c4f89c44a07a80c", size = 421083, upload-time = "2026-06-26T19:25:35.804Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2a/7b/ad04e9d1337fc04965dc9fc616b6a72cb65a24b800a014c011ec812f5489/greenlet-3.5.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7ef56fe650f50575bf843acde967b9c567687f3c22340941a899b7bc56e956a8", size = 1577771, upload-time = "2026-06-26T19:09:01.537Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d8/33/6c87ab7ba663f70ca21f3022aad1ffe56d3f3e0521e836c2415e13abcc3c/greenlet-3.5.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5121af01cf911e70056c00d4b46d5e9b5d1415550038573d744138bacb59e6b8", size = 1644048, upload-time = "2026-06-26T18:31:42.996Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/1c/35/f0d8ee998b422cf8693b270f098e55d8d4ec8006b061b333f54f177d28d9/greenlet-3.5.3-cp311-cp311-win_amd64.whl", hash = "sha256:0f41e4a05a3c0cb31b17023eff28dd111e1d16bf7d7d00406cd7df23f31398a7", size = 239137, upload-time = "2026-06-26T18:23:21.664Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/96/b9820295576ef18c9edc404f10e260ae7215ceaf3781a54b720ed2627862/greenlet-3.5.3-cp311-cp311-win_arm64.whl", hash = "sha256:ec6f1af59f6b5f3fc9678e2ea062d8377d22ac644f7844cb7a292910cf12ff44", size = 237630, upload-time = "2026-06-26T18:24:00.281Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5d/6e/4c37d51a2b7f82d2ff11bb6b5f7d766d9a011726624af255e843727627a3/greenlet-3.5.3-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:719757059f5a53fd0dde23f78cffeafcdd97b21c850ddb7ca684a3c1a1f122e2", size = 288685, upload-time = "2026-06-26T18:22:08.977Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/73/815dd90131c1b71ebdf53dbc7c276cafec2a1173b97559f97aba72724a87/greenlet-3.5.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:efa9f765dd09f9d0cdac651ffdf631ee59ec5dc6ee7a73e0c012ba9c52fbdf5b", size = 604761, upload-time = "2026-06-26T19:07:10.114Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9f/57/079cfe76bcef36b153b25607ee91c6fcb58f17f8b23c86bbbeabe0c88d72/greenlet-3.5.3-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7faba15ac005376e02a0384504e0243be3370ce010296a44a820feb342b505ab", size = 617044, upload-time = "2026-06-26T19:10:07.25Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/fb/d97dc261209c80744b7c8132693a30d70ec6e7315e632cb0a10b3fec94dd/greenlet-3.5.3-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5795cd1101371140551c645f2d408b8d3c01a5a29cf8a9bce6e759c983682d23", size = 622351, upload-time = "2026-06-26T19:24:16.32Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/37/87/b4d095775a3fb1bcafbb483fc206b27ebb785724c83051447737085dc54e/greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:87142215824be6ac05e2e8e2786eec307ccbc27c36723c3881959df654af6861", size = 614244, upload-time = "2026-06-26T18:32:17.594Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/ac/e5fee13cbbd0e8de312d9a146584b8a51891c68847330ef9dc8b5109d23f/greenlet-3.5.3-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:af4923b3096e26a36d7e9cf24ab88083a20f97d191e3b97f253731ce9b41b28c", size = 425395, upload-time = "2026-06-26T19:25:37.144Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/70/7559b609683650fa2b95b8ab84b4ab0b26556a635d19675e12aa832d826d/greenlet-3.5.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:215275b1b49320987352e6c1b054acca0064f965a2c66992bed9a6f7d913f149", size = 1574210, upload-time = "2026-06-26T19:09:03.077Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ae/73/be55392074c60fc37655ca40fa6022457bfbf6718e9e342a7b0b41f96dd2/greenlet-3.5.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6b1b0eed82364b0e32c4ea0f221452d33e6bb17ae094d9f72aed9851812747ea", size = 1638627, upload-time = "2026-06-26T18:31:44.748Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/14/40/c57489acf8e37d74e2913d4eff63aa0dba17acccc4bdeef874dde2dbbec9/greenlet-3.5.3-cp312-cp312-win_amd64.whl", hash = "sha256:cde8adafa2365676f74a979744629589999093bc86e2484214f58e61df08902c", size = 239882, upload-time = "2026-06-26T18:23:27.518Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/fd/6fea0e3d6600f785069481ee637e09378dd4118acdfd38ad88ae2db31c98/greenlet-3.5.3-cp312-cp312-win_arm64.whl", hash = "sha256:c4e7b79d83805475f0102008843f6eb45fd3bb0b2e88c774adab5fbaab27117d", size = 238211, upload-time = "2026-06-26T18:22:37.671Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9b/ff/a620267401db30a50cc8450ee90730e2d4a85658c055c0e760d4ed47fb13/greenlet-3.5.3-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:c8d87c2134d871df96ecdea9cec7cbaab286dadab0f56476e57aaf9e8ac11550", size = 287609, upload-time = "2026-06-26T18:21:14.724Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d6/fa/5401ac78021c826a25b6dde0c705e0a8f29b617509f9185a31dac15fbe1b/greenlet-3.5.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a2d185dd1621757e70c3861cceffd5317ab4e7ed7eb09c82994828468527ade5", size = 607435, upload-time = "2026-06-26T19:07:11.412Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e9/76/1dc144a2e56e65d36405078ed774224375ea520a1870a6e46e08bb4ac7bf/greenlet-3.5.3-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1c514a468149bf8fbbab874188a3535cd8a48a3e353eb53a3d424296f8dbacd3", size = 619787, upload-time = "2026-06-26T19:10:08.396Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/57/61/2f5b1adf256d039f5dab8005de8d3d7ad2b0070a3219c0e036b3fbfeb440/greenlet-3.5.3-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9ad04dd75458c6300b047c61b8639092433d205a25a14e310d6582a480efcca1", size = 625580, upload-time = "2026-06-26T19:24:18.344Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bf/87/c298cee62df1de4ad7fec32abda73526cff347fd143a6ed4ac369246668a/greenlet-3.5.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:915f887cf2682b66419b879423a2e072634aa7b7dce6f3ada4957cfced3f1e9a", size = 616786, upload-time = "2026-06-26T18:32:19.128Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3e/d9/ab7fc9e543e44d6879b0a6ef9a4b2188940fd180cc65d6f646883ddf7201/greenlet-3.5.3-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:afaabdd554cd7ae9bbb3ca070b0d7fdfd207dbf1d16865f7233837709d354bda", size = 427933, upload-time = "2026-06-26T19:25:38.219Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9e/2e/e6f009885ed0705ccf33fe0583c117cfd03cde77e31a596dd5785a30762b/greenlet-3.5.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:766cfd421c13e450feb340cd472a3ed9957d438727b7b4593ad7c76c5d2b0deb", size = 1574316, upload-time = "2026-06-26T19:09:04.273Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ef/fe/43fd110b01e40da0adb7c90ac7ea744bef2d43dca00de5095fd2351c2a68/greenlet-3.5.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2ecda9ec22edf38fa389369eaed8c3d37c05f3c54e69f69438dbb2cc1de1458b", size = 1638614, upload-time = "2026-06-26T18:31:46.297Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0f/7c/062447147a61f8b4337b156fe70d32a165fcf2f89d7ca6255e572806705c/greenlet-3.5.3-cp313-cp313-win_amd64.whl", hash = "sha256:c82304750f057167ff60d188df1d0cc1764ce9567eadf03e6a7443bcedd0b30b", size = 239850, upload-time = "2026-06-26T18:21:54.613Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c7/7e/220a7f5824a64a60443fc03b39dfac4ea63a7fb6d481efa27eafa928e7f4/greenlet-3.5.3-cp313-cp313-win_arm64.whl", hash = "sha256:dc133a1569ee667b2a6ef56ce551084aeefd87a5acbc4736d336d1e2edc6cfc4", size = 238141, upload-time = "2026-06-26T18:22:48.507Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c3/93/43e116ee114b28737ba7e12952a0d4e2f55944d0f84e42bc91ba7192a3c9/greenlet-3.5.3-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:fd2e02fa07485778536a036222d616ab957b1d533f36b3ed98ce725d9c9d3117", size = 288202, upload-time = "2026-06-26T18:23:49.604Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/82/2f/146d218299046a43d1f029fd544b3d110d0f175a09c715c7e8da4a4a345d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df0a0628d1597eb0897b62f55d1343f772405fd25f3b2a796c76874b0c2e22e8", size = 654096, upload-time = "2026-06-26T19:07:12.71Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/cc/04738cafb3f45fa991ea44f9de94c47dcec964f5a972300988a6751f49d9/greenlet-3.5.3-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ebd933a6adabc298bab47731a130fe6bfb888bd934eee37810f151159544540d", size = 666304, upload-time = "2026-06-26T19:10:09.503Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/86/a9/73fa62893d5b84b4205544e6b673c654cc43aa5b9899bac00f04d64af73d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8d19fe6c39ebff9259f07bcc685d3290f8fa4ea2278e51dd0008e4d6b0f2d814", size = 670657, upload-time = "2026-06-26T19:24:19.967Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ce/aa/4e0dad5e605c270c784ab911c43da6adb136ccd4d81180f763ca429a723d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4b9d501b40e80b70e32323c799dd9b420a5577a9601469d362ae1ffb690f3a7c", size = 663635, upload-time = "2026-06-26T18:32:20.802Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/29/7e/2ffce64929fb3cab7b65d5a0b20aaf9764e227681d731b041077fc9a525a/greenlet-3.5.3-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:962c5df2db8cb446da51edf1ca5296c389d93b99c9d8aa2ee4c7d0d8f1218260", size = 473497, upload-time = "2026-06-26T19:25:39.421Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d1/50/13efdbea246fe3d3b735e191fec08fb50809f53cd2383ebe123d0809e44b/greenlet-3.5.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a1fad1d11e7d6aab184107baa8e4ece11ccba3ec9599cd7efa5ff4d70d43256a", size = 1621252, upload-time = "2026-06-26T19:09:05.647Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f7/22/c0a336ae4a1410fd5f5121098e5bfbf1865f64c5ef80b4b5412886c4a332/greenlet-3.5.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:fad5aec764399f1b5cc347ad250a59660f20c8f8888ea6bae1f93b769cce1154", size = 1684824, upload-time = "2026-06-26T18:31:47.738Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7a/94/91aec0030bea75c4b3244251d0de60a1f3432d1ecb53ab6c437fb5c3ba61/greenlet-3.5.3-cp314-cp314-win_amd64.whl", hash = "sha256:7669aa24cf2a1041d6f7899575b494a3ab4cf68bfcc8609b1dc0be7272db835e", size = 240754, upload-time = "2026-06-26T18:22:15.669Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e5/06/68d0983e79e02138f64b4d303c500c27ddb48e5e77f3debb80888a921eae/greenlet-3.5.3-cp314-cp314-win_arm64.whl", hash = "sha256:5b4807c4082c9d1b6d9eed56fcd041863e37f2228106eef24c30ca096e238605", size = 239549, upload-time = "2026-06-26T18:22:42.996Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/91/95/3e161213d7f1d378d15aa9e792093e9bfe01844680d04b7fd6e0107c9098/greenlet-3.5.3-cp314-cp314t-macosx_11_0_universal2.whl", hash = "sha256:271a8ea7c1024e8a0d7dd2be66dd66dda8a07193f41a17b9e924f7600f5b62be", size = 296389, upload-time = "2026-06-26T18:22:20.657Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/00/92/715c44721abe2b4d1ae9abde4179411868a5bff312479f54e105d372f131/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:19131729ae0ddc3c2e1ef85e650169b5e37ee32e400f215f78b94d7b0d567310", size = 653382, upload-time = "2026-06-26T19:07:14.209Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/83/37a10372a1090a6624cca8e74c12df1a36c2dc36429ed0255b7fb1aeee23/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1540dd8e5fc2a5aec40fbb98ef8e149fa47c89a4b4a1cf2575a14d3d1869d7a8", size = 659401, upload-time = "2026-06-26T19:10:10.876Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cb/73/8faec206b851c22b1733545fda900829a1f3f5b1c78ae7e0fb3dba57d9f4/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b897d97759425953f69a9c0fac67f8fe333ec0ce7377ef186fb2b0c3ad5e354d", size = 659582, upload-time = "2026-06-26T19:24:21.357Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/db/e2/d1509cad4207da559cc42986ecdd8fc67ad0d1bba2bf03023c467fd5e0f3/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e81fa194a1d20967877bdf9c7794db2bc99063e5be36aee710c08f04c5bb087f", size = 656969, upload-time = "2026-06-26T18:32:22.272Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/55/50c19e49f8045834ada71ef12f8ad048eba8517c6aa41161bed676328fae/greenlet-3.5.3-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:3236754d423955ea08e9bb5f6c04a7895f9e22c290b66aa7653fcb922d839eb0", size = 491037, upload-time = "2026-06-26T19:25:40.672Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/86/7d/eaf70de20aadca3a5884aec58362861c64ce45e7b277f47ed026926a3b89/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:55cf4d777485d43110e47133cbba6d74a8885a87ec1227ef0267f9ee80c5aa21", size = 1617822, upload-time = "2026-06-26T19:09:06.893Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/f9/414d38fc400ae4350d4185eaad1827676f7cf5287b9136e0ed1cbbe20a7f/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:12a248ba75f6a9a236375f52296c498c89ff1d8badf32deb9eca7abd5853f7da", size = 1677983, upload-time = "2026-06-26T18:31:49.396Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e4/15/7edb977e08f9bff702fe42d6c902702786ff6b9694058b4e6a2a6ac90e57/greenlet-3.5.3-cp314-cp314t-win_amd64.whl", hash = "sha256:efc6bd60ea02e085862c74a3ef64b147ffc6f1a5ea7d9f26e7a939943f68c1e3", size = 243626, upload-time = "2026-06-26T18:24:41.485Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2c/8a/93928dce91e6b3598b5e779e8d1fd6576a504640c58e78627077f6a7a91a/greenlet-3.5.3-cp315-cp315-macosx_11_0_universal2.whl", hash = "sha256:ea03f2f04367845d6b58eeed276e1e56e51f0b97d8ad5a88a7d20a91dc9056cc", size = 288860, upload-time = "2026-06-26T18:22:48.07Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/4f/ca/69db42d447a1378043e2c8f19c09cbbd1263371505053c496b49066d3d16/greenlet-3.5.3-cp315-cp315-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:78dbef602fda6d97d957eb7937f70c9ce9e9527330347f8f6b6f9e554a9e7a47", size = 659747, upload-time = "2026-06-26T19:07:15.565Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a8/0b/af7ac2ef8dd41e3da1a40dda6305c23b9a03e13ba975ec916357b50f8575/greenlet-3.5.3-cp315-cp315-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6f73857adb8fee13fa56c172bd11262f888c0c648f9fea113e777bb2c7904a81", size = 670419, upload-time = "2026-06-26T19:10:12.293Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/25/aa/952cf28c2ff949a8c971134fb43854dd7eaa737218723aaef758f8c9aead/greenlet-3.5.3-cp315-cp315-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:cefa9cef4b371f9844c6053db71f1138bc6807bab1578b0dae5149c1f1141357", size = 674261, upload-time = "2026-06-26T19:24:22.79Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/1e/1d51640cacbfc455dbe9f9a9f594c49e4e244f63b9971a2f4764e46cc53d/greenlet-3.5.3-cp315-cp315-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:232fec92e823addaf02d9472cf7381e24a1d046a6ced1103c5caa4c21b9dfc1d", size = 668787, upload-time = "2026-06-26T18:32:24.298Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/dc/f2/b00d6f5e63e531a93562b2ec1a4c320fbee91f580fc42e6417af69d706e5/greenlet-3.5.3-cp315-cp315-manylinux_2_39_riscv64.whl", hash = "sha256:6219b6d04dbf6ba6084d77dc609e8473060dc55f759cbf626d512122781fa128", size = 480322, upload-time = "2026-06-26T19:25:41.852Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/21/66/4030d5b0b5894500023f003bb054d9bb354dfbd1e186c3a296759172f5f5/greenlet-3.5.3-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:2421c3564da9429d5586d46ca31ebb26516b5498a802cf65c041a8e8a8980d34", size = 1626305, upload-time = "2026-06-26T19:09:08.281Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0e/50/5221371c7550108dfa3c378debc41d032aa9c78e89abb01d8011cfc93289/greenlet-3.5.3-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e0f0d160f0b2e558e6c75f7930967183255dc9735e5f5b8cae58ee09c9576d8b", size = 1688631, upload-time = "2026-06-26T18:31:51.278Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/68/5d/00d469daae3c65d2bf620b10eee82eb022127d483c6bc8c69fae6f3fbf17/greenlet-3.5.3-cp315-cp315-win_amd64.whl", hash = "sha256:dd99329bbc15ca78dcc583dba05d0b1b0bae01ab6c2174989f5aaee3e41ac930", size = 241027, upload-time = "2026-06-26T18:22:38.203Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e7/e8/883785b44c5780ed71e83d3e4437e710470be17a2e181e8b601e2da0dc4a/greenlet-3.5.3-cp315-cp315-win_arm64.whl", hash = "sha256:499fef2acede88c1864a57bb586b4bf533c81e1b82df7ab93451cdb47dfec227", size = 240085, upload-time = "2026-06-26T18:23:54.217Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/1c/da/4f4a8450962fad137c1c8981a3f1b8919d06c829993d4d476f9c525d5173/greenlet-3.5.3-cp315-cp315t-macosx_11_0_universal2.whl", hash = "sha256:176bc16a721fa5fc294d70b87b4dfa5fbdd251b3da5d5372735ecef9bd7d6d0c", size = 297221, upload-time = "2026-06-26T18:23:27.176Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/57/66/b3bfae3e220a9b63ea539a0eea681800c69ab1aada757eae8789f183e7ce/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:629b614d2b786e89c50440e246f33eea78f58a962d0bdbbcc809e6d13605903f", size = 657221, upload-time = "2026-06-26T19:07:16.973Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7b/81/b6d4d73a709684fc77e7fa034d7c2fe82cffa9fc920fadcaa659c2626213/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2b2e857ae16f5f72142edf75f9f176fe7526ba19a2841df1420516f83831c9f2", size = 663226, upload-time = "2026-06-26T19:10:13.723Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e9/39/0e0938a75115b939d42733a2a12e1d349653c9531fe6fe563e8a681f04e6/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:16d192579ed281051396dddd7f7754dac6259e6b1fb26378c87b66622f8e3f91", size = 663706, upload-time = "2026-06-26T19:24:24.312Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f5/07/e210b02b589f16e74ff48b730690e4a34ffe984219fce4f3c1a0e7ec8545/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e515757e2e36bcbf1fad09a46e1557e8b1ae1797d4b44d09da7deed88ad28608", size = 660802, upload-time = "2026-06-26T18:32:26.081Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5b/41/35d1c678cdb3c3b9e6bee691728e563cfb294202b23c7a4c3c2ccc343589/greenlet-3.5.3-cp315-cp315t-manylinux_2_39_riscv64.whl", hash = "sha256:4399eb8d041f20b68d943918bc55502a93d6fdc0a37c14da7881c04139acee9d", size = 498803, upload-time = "2026-06-26T19:25:43.063Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/eb/2e/5303eb3fa06bca089060f479707182a93e360683bc252acf846c3090d34e/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:b363d46ed1ea431825fdb01471bb024fc08399bad1572a616e853c7684415adb", size = 1622157, upload-time = "2026-06-26T19:09:09.527Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/54/70/50de47a488f14df260b50ae34fb5d56016e308b098eab02c878b5223c26a/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:e44da2f5bbdaabaf7d80b73dbb430c7035771e9f244e3c8b769715c9d8fa0a16", size = 1681159, upload-time = "2026-06-26T18:31:52.986Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a7/13/1055e1dda7882073eda533e2b96c62e55bbd2db7fda6d5ece992febc7071/greenlet-3.5.3-cp315-cp315t-win_amd64.whl", hash = "sha256:8ff8bed3e3baa20a3ea261ce00526f1898ad4801d4886fd2220580ee0ad8fadf", size = 244007, upload-time = "2026-06-26T18:22:04.353Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b4/0d/ca7d15afbdc397e3401134c9e1800d51d12b829661786187a4ad08fe484f/greenlet-3.5.3-cp315-cp315t-win_arm64.whl", hash = "sha256:b7068bd09f761f3f5b4d214c2bed063186b2a86148c740b3873e3f56d79bac31", size = 242586, upload-time = "2026-06-26T18:23:37.93Z" },
]
[[package]] [[package]]
name = "html5lib" name = "html5lib"
version = "1.1" version = "1.1"
@@ -1417,6 +1624,25 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bc/60/5382c03e1970de634027cee8e1b7d39776b778b81812aaf45b694dfe9e28/pillow-12.2.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:bfa9c230d2fe991bed5318a5f119bd6780cda2915cca595393649fc118ab895e", size = 7080946, upload-time = "2026-04-01T14:46:11.734Z" }, { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bc/60/5382c03e1970de634027cee8e1b7d39776b778b81812aaf45b694dfe9e28/pillow-12.2.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:bfa9c230d2fe991bed5318a5f119bd6780cda2915cca595393649fc118ab895e", size = 7080946, upload-time = "2026-04-01T14:46:11.734Z" },
] ]
[[package]]
name = "playwright"
version = "1.61.0"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
dependencies = [
{ name = "greenlet" },
{ name = "pyee" },
]
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/44/ee/31e4e0db36588b817a10b299a0285082545fde7d36543c2abe498bb3d61a/playwright-1.61.0-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:ff138c3a604f69911e9d42fd036e55c2a171e5616edf04c1e7f60a2a285540b0", size = 43421877, upload-time = "2026-06-29T10:32:48.428Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/42/35/71395dd3ecc798965be4a3ef8c443217d4abca168e7cb34536304f9489e6/playwright-1.61.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:009588c2a7e499bc5a8b425b61fa65490968bbda9cd69e0cf2cff10f8304659a", size = 42205016, upload-time = "2026-06-29T10:32:52.104Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f4/44/323164cf5cd1647bdefce76ffce27651aadb959d089b48f53ea40918276e/playwright-1.61.0-py3-none-macosx_11_0_universal2.whl", hash = "sha256:9f7de4536088d12037c13a52b7ea34b59270b78926bb56935070597ffac6b1af", size = 43421884, upload-time = "2026-06-29T10:32:55.773Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ab/f8/a35bf179e4ba2522c1893635094a64e407572547bd61528820fc0abc87fe/playwright-1.61.0-py3-none-manylinux1_x86_64.whl", hash = "sha256:54f3b39f6eab832e33458c1dd7da0b5682aedab3b09ae731b5c59fa12fd2024e", size = 47421381, upload-time = "2026-06-29T10:32:59.903Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b7/eb/e3f922348ec17c315f98c463f72faa1181a1c3de0bfe31a8d2edf6561723/playwright-1.61.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:93454322ade8c11d5d6c211bfd91bdfb9ffb4810e3e026371bcbc4bec1b7ee4c", size = 47120545, upload-time = "2026-06-29T10:33:03.574Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c2/a6/5be4e52b40a9c0c8a073e7c5b0785c05cf5a9ea8f8a7b5b260e32d970342/playwright-1.61.0-py3-none-win32.whl", hash = "sha256:372d55a6f1248fa1dd47599686980cb8fb5bbe6fcda59eab793eb657c11d8a9b", size = 37844841, upload-time = "2026-06-29T10:33:07.361Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6c/fd/2b78036e5fbe9d5f5645bbe08a1eac7160c51243c0093963edbcf67c35d9/playwright-1.61.0-py3-none-win_amd64.whl", hash = "sha256:35c6cc4589a5d00964a59d7b3e59641e0aac0c02f15479a7af77d20f6bc79597", size = 37844846, upload-time = "2026-06-29T10:33:10.637Z" },
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/0d/1b0f3c4ee4eb0514bc805b5c2f9a223e5b6de4f11a926f5235d51d0fc81b/playwright-1.61.0-py3-none-win_arm64.whl", hash = "sha256:e9fcbffcf557a8620fdedd92491eb59a32d18e23d6f3b4f6214b952be324fe51", size = 33955127, upload-time = "2026-06-29T10:33:14.008Z" },
]
[[package]] [[package]]
name = "pluggy" name = "pluggy"
version = "1.6.0" version = "1.6.0"
@@ -1501,6 +1727,18 @@ wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" }, { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
] ]
[[package]]
name = "pyee"
version = "13.0.1"
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
dependencies = [
{ name = "typing-extensions" },
]
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8b/04/e7c1fe4dc78a6fdbfd6c337b1c3732ff543b8a397683ab38378447baa331/pyee-13.0.1.tar.gz", hash = "sha256:0b931f7c14535667ed4c7e0d531716368715e860b988770fc7eb8578d1f67fc8", size = 31655, upload-time = "2026-02-14T21:12:28.044Z" }
wheels = [
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/c4/b4d4827c93ef43c01f599ef31453ccc1c132b353284fc6c87d535c233129/pyee-13.0.1-py3-none-any.whl", hash = "sha256:af2f8fede4171ef667dfded53f96e2ed0d6e6bd7ee3bb46437f77e3b57689228", size = 15659, upload-time = "2026-02-14T21:12:26.263Z" },
]
[[package]] [[package]]
name = "pygments" name = "pygments"
version = "2.20.0" version = "2.20.0"