Compare commits
4 Commits
17fa75495d
...
b5c8c0b8da
| Author | SHA1 | Date | |
|---|---|---|---|
| b5c8c0b8da | |||
| 31baa18ce5 | |||
| 8d908477e2 | |||
| 83a006bbe4 |
@@ -1,6 +1,7 @@
|
|||||||
__pycache__/
|
__pycache__/
|
||||||
*.py[cod]
|
*.py[cod]
|
||||||
.pytest_cache/
|
.pytest_cache/
|
||||||
|
.coverage
|
||||||
*.egg-info/
|
*.egg-info/
|
||||||
.venv/
|
.venv/
|
||||||
venv/
|
venv/
|
||||||
|
|||||||
@@ -120,6 +120,49 @@ partitions already written. Pass the **dataset directory** (`{output_dir}/{unive
|
|||||||
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
|
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
|
||||||
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
|
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
|
||||||
|
|
||||||
|
### `derived` — daily custom/derived data
|
||||||
|
|
||||||
|
Derived data is daily-only v1 research data keyed by `symbol_id,date`, with one
|
||||||
|
or more numeric value columns. It can come from user CSV/parquet files or Python
|
||||||
|
plugins, and is written as a single parquet file at `derived/{name}.pq`.
|
||||||
|
|
||||||
|
The validator normalizes `date` to the trading day, requires unique
|
||||||
|
`symbol_id,date` keys, rejects duplicate columns, and rejects non-numeric value
|
||||||
|
columns. Alpha computation consumes derived data through the existing
|
||||||
|
`--feature-path` flag.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Validate a user file without writing output.
|
||||||
|
uv run python cli.py derived validate --input-path vendor_factor.csv
|
||||||
|
|
||||||
|
# Ingest CSV/parquet into the canonical derived/ layout.
|
||||||
|
uv run python cli.py derived ingest \
|
||||||
|
--input-path vendor_factor.csv \
|
||||||
|
--derived-name vendor_factor
|
||||||
|
|
||||||
|
# List built-in and external derived-data plugin types.
|
||||||
|
uv run python cli.py derived list
|
||||||
|
uv run python cli.py derived list --derived-module path/to/my_derived.py
|
||||||
|
|
||||||
|
# Compute a derived file from daily and/or minute inputs.
|
||||||
|
uv run python cli.py derived compute \
|
||||||
|
--minute-path data/minute_bars/sh600000 \
|
||||||
|
--daily-path data/daily_bars/sh600000 \
|
||||||
|
--derived-type minute_daily_summary \
|
||||||
|
--derived-name minute_summary
|
||||||
|
|
||||||
|
# Join derived columns into a feature-aware alpha.
|
||||||
|
uv run python cli.py alpha compute \
|
||||||
|
--data-path data/daily_bars/sh600000 \
|
||||||
|
--feature-path derived/minute_summary.pq \
|
||||||
|
--alpha-type my_feature_aware_alpha \
|
||||||
|
--alpha-name my_run
|
||||||
|
```
|
||||||
|
|
||||||
|
For compatibility, `feature list` and `feature compute` remain available and
|
||||||
|
delegate to the same derived-data registry. Existing `features/*.pq` files are
|
||||||
|
still valid `--feature-path` inputs when they satisfy the daily numeric contract.
|
||||||
|
|
||||||
### `alpha list` — show registered alpha types
|
### `alpha list` — show registered alpha types
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -137,6 +180,7 @@ uv run python cli.py alpha list --alpha-module path/to/my_alpha.py # include a
|
|||||||
| `--output-dir` | `alphas` | Output directory |
|
| `--output-dir` | `alphas` | Output directory |
|
||||||
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
|
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
|
||||||
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
|
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
|
||||||
|
| `--feature-path` | — | Daily derived/feature parquet file or dataset to left-join on `symbol_id,date`; repeatable |
|
||||||
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
|
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
|
||||||
| `--param` | — | Extra constructor param as `name=value`; repeatable |
|
| `--param` | — | Extra constructor param as `name=value`; repeatable |
|
||||||
|
|
||||||
@@ -370,6 +414,8 @@ between phases (data is stored long/tidy):
|
|||||||
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
|
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
|
||||||
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
|
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
|
||||||
baostock valuation ratios.)
|
baostock valuation ratios.)
|
||||||
|
- **derived** (`DERIVED_KEY_COLUMNS` + values): required keys `symbol_id, date`;
|
||||||
|
value columns are user/plugin-defined and must be numeric in v1.
|
||||||
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
|
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
|
||||||
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
|
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
|
||||||
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
|
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
|
||||||
@@ -387,8 +433,11 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
|
|||||||
|
|
||||||
- `cli.py` — entry point wiring the file-based phases together
|
- `cli.py` — entry point wiring the file-based phases together
|
||||||
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
|
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
|
||||||
|
- `pipeline/derived/` — daily derived-data ingestion, validation, plugin registry,
|
||||||
|
and built-in derived computations → `derived/*.pq`
|
||||||
- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
|
- `pipeline/alpha/` — `base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
|
||||||
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
|
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
|
||||||
|
- `pipeline/features/` — compatibility wrappers for the derived-data registry
|
||||||
- `pipeline/combo/` — alpha combination → `combos/*.pq`
|
- `pipeline/combo/` — alpha combination → `combos/*.pq`
|
||||||
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
|
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
|
||||||
reference next-open simulator, and research metrics
|
reference next-open simulator, and research metrics
|
||||||
@@ -410,6 +459,9 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
|
|||||||
- [x] **Reference execution simulation** — next-open fills over constructed
|
- [x] **Reference execution simulation** — next-open fills over constructed
|
||||||
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
|
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
|
||||||
and slippage controls.
|
and slippage controls.
|
||||||
|
- [x] **Derived/custom daily data ("Level 2")** — ingest user CSV/parquet files
|
||||||
|
or compute plugin outputs as validated numeric daily datasets under
|
||||||
|
`derived/{name}.pq`; alpha joins continue through `--feature-path`.
|
||||||
- [ ] **Optional Backtrader adapter** — Backtrader is available as the
|
- [ ] **Optional Backtrader adapter** — Backtrader is available as the
|
||||||
`backtrader` extra for possible future event-driven/broker-style experiments,
|
`backtrader` extra for possible future event-driven/broker-style experiments,
|
||||||
but it is not part of the current canonical portfolio workflow.
|
but it is not part of the current canonical portfolio workflow.
|
||||||
@@ -419,3 +471,29 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
|
|||||||
and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
|
and intraday VWAP. These need a tick / L1–L2 quote feed (typically a paid or
|
||||||
brokerage data tier); the free daily sources here only expose daily bars, so
|
brokerage data tier); the free daily sources here only expose daily bars, so
|
||||||
this is a separate data phase rather than extra columns on the daily schema.
|
this is a separate data phase rather than extra columns on the daily schema.
|
||||||
|
|
||||||
|
### Additional TODOs
|
||||||
|
|
||||||
|
The following items are intended extensions beyond the current daily
|
||||||
|
alpha-to-portfolio pipeline:
|
||||||
|
|
||||||
|
- **Long-only portfolio mode** — add a construction option that converts
|
||||||
|
alpha/combo weights into a long-only book while preserving existing lot,
|
||||||
|
price, suspension, and volume-cap handling.
|
||||||
|
- **Index-short hedging mode** — support portfolios that hold long A-share
|
||||||
|
names while shorting an index or index proxy for market exposure control.
|
||||||
|
- **Expanded universe presets** — add explicit universe aliases for CSI 300,
|
||||||
|
CSI 500, CSI 1000, and CSI 1800, while keeping file-based and comma-separated
|
||||||
|
custom universes available.
|
||||||
|
- **Categorical derived data** — extend the numeric-only derived-data v1 contract
|
||||||
|
to support categorical inputs such as industry classifications. In this
|
||||||
|
project, "Level 2" means customized second-level research data produced by
|
||||||
|
users or plugins; it does not necessarily mean exchange order-book/L2 quote
|
||||||
|
feeds.
|
||||||
|
- **Minute bar data** — continue extending the raw minute-bar and feature
|
||||||
|
workflow. The initial Baostock 5-minute download and daily feature plugin path
|
||||||
|
exist; intraday execution and replacing canonical daily bars remain out of
|
||||||
|
scope unless explicitly added later.
|
||||||
|
- **Industry data** — add industry classification inputs for filtering,
|
||||||
|
grouping, exposure reporting, neutralization, or industry-aware portfolio
|
||||||
|
construction.
|
||||||
|
|||||||
@@ -3,7 +3,9 @@
|
|||||||
|
|
||||||
Phases:
|
Phases:
|
||||||
data — Download daily bars to parquet
|
data — Download daily bars to parquet
|
||||||
|
derived — Ingest or compute daily derived data
|
||||||
alpha — Compute alpha weights from data
|
alpha — Compute alpha weights from data
|
||||||
|
feature — Compute daily features from minute bars
|
||||||
combo — Combine alphas into a single weight
|
combo — Combine alphas into a single weight
|
||||||
portfolio — Build tradable positions and simulate execution
|
portfolio — Build tradable positions and simulate execution
|
||||||
"""
|
"""
|
||||||
@@ -13,7 +15,9 @@ import logging
|
|||||||
import click
|
import click
|
||||||
|
|
||||||
from pipeline.data.cli import data
|
from pipeline.data.cli import data
|
||||||
|
from pipeline.derived.cli import derived
|
||||||
from pipeline.alpha.cli import alpha
|
from pipeline.alpha.cli import alpha
|
||||||
|
from pipeline.features.cli import feature
|
||||||
from pipeline.combo.cli import combo
|
from pipeline.combo.cli import combo
|
||||||
from pipeline.portfolio.cli import portfolio
|
from pipeline.portfolio.cli import portfolio
|
||||||
from tools.pqcat import pqcat
|
from tools.pqcat import pqcat
|
||||||
@@ -39,7 +43,9 @@ def cli(log_level):
|
|||||||
|
|
||||||
|
|
||||||
cli.add_command(data)
|
cli.add_command(data)
|
||||||
|
cli.add_command(derived)
|
||||||
cli.add_command(alpha)
|
cli.add_command(alpha)
|
||||||
|
cli.add_command(feature)
|
||||||
cli.add_command(combo)
|
cli.add_command(combo)
|
||||||
cli.add_command(portfolio)
|
cli.add_command(portfolio)
|
||||||
cli.add_command(pqcat)
|
cli.add_command(pqcat)
|
||||||
|
|||||||
@@ -31,11 +31,68 @@ _BATCH_COLUMNS = [
|
|||||||
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
|
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
# Raw Baostock minute bars. The ``time`` field is usually compact
|
||||||
|
# YYYYMMDDHHMMSSmmm; parsing below also tolerates HH:MM:SS strings in tests.
|
||||||
|
_MINUTE_FIELDS = "date,time,code,open,high,low,close,volume,amount,adjustflag"
|
||||||
|
_MINUTE_NUMERIC = ["open", "high", "low", "close", "volume", "amount"]
|
||||||
|
_MINUTE_COLUMNS = [
|
||||||
|
"symbol", "datetime", "date", "time", "frequency",
|
||||||
|
"open", "high", "low", "close", "volume", "amount", "vwap", "adjustflag",
|
||||||
|
]
|
||||||
|
_MINUTE_FREQUENCIES = {"5", "15", "30", "60"}
|
||||||
|
|
||||||
|
|
||||||
class _SessionLost(Exception):
|
class _SessionLost(Exception):
|
||||||
"""baostock reported the session was dropped (``用户未登录``)."""
|
"""baostock reported the session was dropped (``用户未登录``)."""
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_minute_frequency(frequency: str | int) -> tuple[str, str]:
|
||||||
|
"""Return Baostock frequency and partition label for a minute interval."""
|
||||||
|
raw = str(frequency).strip().lower()
|
||||||
|
if raw.endswith("m"):
|
||||||
|
raw = raw[:-1]
|
||||||
|
if raw not in _MINUTE_FREQUENCIES:
|
||||||
|
raise ValueError(
|
||||||
|
f"Unsupported minute frequency '{frequency}'. "
|
||||||
|
f"Expected one of {sorted(_MINUTE_FREQUENCIES)} minutes."
|
||||||
|
)
|
||||||
|
return raw, f"{raw}m"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_minute_datetime(date: pd.Series, time: pd.Series) -> pd.Series:
|
||||||
|
"""Parse Baostock minute timestamps into pandas datetimes."""
|
||||||
|
date_dt = pd.to_datetime(date, errors="coerce")
|
||||||
|
date_compact = date_dt.dt.strftime("%Y%m%d")
|
||||||
|
time_text = time.astype(str).str.strip()
|
||||||
|
time_digits = time_text.str.replace(r"\D", "", regex=True)
|
||||||
|
|
||||||
|
full_digits = time_digits.str.slice(0, 14)
|
||||||
|
from_full = pd.to_datetime(full_digits, format="%Y%m%d%H%M%S", errors="coerce")
|
||||||
|
|
||||||
|
from_short = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
|
||||||
|
short_time = time_digits.str.len().between(1, 6)
|
||||||
|
if short_time.any():
|
||||||
|
short_digits = (
|
||||||
|
time_digits.loc[short_time]
|
||||||
|
.str.pad(6, side="right", fillchar="0")
|
||||||
|
.str.slice(0, 6)
|
||||||
|
)
|
||||||
|
from_short.loc[short_time] = pd.to_datetime(
|
||||||
|
date_compact.loc[short_time] + short_digits,
|
||||||
|
format="%Y%m%d%H%M%S",
|
||||||
|
errors="coerce",
|
||||||
|
)
|
||||||
|
|
||||||
|
from_text = pd.Series(pd.NaT, index=date.index, dtype="datetime64[ns]")
|
||||||
|
text_time = time_text.str.contains(":", regex=False)
|
||||||
|
if text_time.any():
|
||||||
|
from_text.loc[text_time] = pd.to_datetime(
|
||||||
|
date.astype(str).loc[text_time] + " " + time_text.loc[text_time],
|
||||||
|
errors="coerce",
|
||||||
|
)
|
||||||
|
return from_full.fillna(from_short).fillna(from_text)
|
||||||
|
|
||||||
|
|
||||||
def _download_akshare(symbol: str, start: str, end: str, adjust: str = "qfq") -> Optional[pd.DataFrame]:
|
def _download_akshare(symbol: str, start: str, end: str, adjust: str = "qfq") -> Optional[pd.DataFrame]:
|
||||||
"""Download daily bars from akshare. Returns DataFrame with OHLCV columns."""
|
"""Download daily bars from akshare. Returns DataFrame with OHLCV columns."""
|
||||||
try:
|
try:
|
||||||
@@ -239,3 +296,104 @@ def download_daily_batch(
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def download_minute_batch(
|
||||||
|
symbols: Iterable[str],
|
||||||
|
start: str,
|
||||||
|
end: str,
|
||||||
|
frequency: str | int = 5,
|
||||||
|
relogin_every: int = 200,
|
||||||
|
) -> Iterator[Tuple[str, Optional[pd.DataFrame]]]:
|
||||||
|
"""Download raw Baostock minute bars for many symbols.
|
||||||
|
|
||||||
|
Minute bars are intentionally unadjusted (`adjustflag='3'`) because the
|
||||||
|
output is raw intraday market data for downstream feature aggregation, not a
|
||||||
|
tradable daily price series.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
symbols: Internal-form symbols (``sh600000`` / ``sz000001``).
|
||||||
|
start, end: ``YYYY-MM-DD`` bounds.
|
||||||
|
frequency: Baostock minute frequency. ``5``/``"5"``/``"5m"`` all mean
|
||||||
|
5-minute bars.
|
||||||
|
relogin_every: Proactively refresh the baostock session every N symbols.
|
||||||
|
|
||||||
|
Yields:
|
||||||
|
``(symbol, df)`` where ``df`` has raw minute bars or ``None`` when no
|
||||||
|
data is available.
|
||||||
|
"""
|
||||||
|
query_frequency, frequency_label = _normalize_minute_frequency(frequency)
|
||||||
|
adjustflag = _BAOSTOCK_ADJUST["none"]
|
||||||
|
|
||||||
|
def _relogin() -> None:
|
||||||
|
try:
|
||||||
|
bs.logout()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
bs.login()
|
||||||
|
|
||||||
|
def _fetch(symbol: str) -> Optional[pd.DataFrame]:
|
||||||
|
"""One Baostock minute query; returns df, None, or raises _SessionLost."""
|
||||||
|
code = f"{symbol[:2]}.{symbol[2:]}"
|
||||||
|
rs = bs.query_history_k_data_plus(
|
||||||
|
code=code,
|
||||||
|
fields=_MINUTE_FIELDS,
|
||||||
|
start_date=start,
|
||||||
|
end_date=end,
|
||||||
|
frequency=query_frequency,
|
||||||
|
adjustflag=adjustflag,
|
||||||
|
)
|
||||||
|
if rs.error_code != "0":
|
||||||
|
if "未登录" in (rs.error_msg or ""):
|
||||||
|
raise _SessionLost(rs.error_msg)
|
||||||
|
logger.warning("baostock minute error for %s: %s", symbol, rs.error_msg)
|
||||||
|
return None
|
||||||
|
rows = []
|
||||||
|
while rs.next():
|
||||||
|
rows.append(rs.get_row_data())
|
||||||
|
if not rows:
|
||||||
|
return None
|
||||||
|
|
||||||
|
df = pd.DataFrame(rows, columns=_MINUTE_FIELDS.split(","))
|
||||||
|
df[_MINUTE_NUMERIC] = df[_MINUTE_NUMERIC].apply(pd.to_numeric, errors="coerce")
|
||||||
|
df["datetime"] = _parse_minute_datetime(df["date"], df["time"])
|
||||||
|
bad_timestamps = df["datetime"].isna()
|
||||||
|
if bad_timestamps.any():
|
||||||
|
raise ValueError(
|
||||||
|
f"Could not parse {int(bad_timestamps.sum())} minute timestamp(s)"
|
||||||
|
)
|
||||||
|
df["date"] = pd.to_datetime(df["date"], errors="coerce")
|
||||||
|
df["time"] = df["datetime"].dt.strftime("%H:%M:%S")
|
||||||
|
df["frequency"] = frequency_label
|
||||||
|
df["vwap"] = (df["amount"] / df["volume"]).where(df["volume"] > 0)
|
||||||
|
df["symbol"] = symbol
|
||||||
|
return df[_MINUTE_COLUMNS].sort_values("datetime").reset_index(drop=True)
|
||||||
|
|
||||||
|
bs.login()
|
||||||
|
try:
|
||||||
|
for i, symbol in enumerate(symbols):
|
||||||
|
if i and relogin_every and i % relogin_every == 0:
|
||||||
|
_relogin()
|
||||||
|
|
||||||
|
df: Optional[pd.DataFrame] = None
|
||||||
|
for attempt in (1, 2):
|
||||||
|
try:
|
||||||
|
df = _fetch(symbol)
|
||||||
|
break
|
||||||
|
except _SessionLost:
|
||||||
|
if attempt == 1:
|
||||||
|
_relogin()
|
||||||
|
continue
|
||||||
|
logger.warning("baostock minute session lost for %s after relogin", symbol)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("baostock minute download failed for %s: %s", symbol, e)
|
||||||
|
break
|
||||||
|
|
||||||
|
if df is not None and not df.empty:
|
||||||
|
yield symbol, df
|
||||||
|
else:
|
||||||
|
yield symbol, None
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
bs.logout()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Minute Bar Data Notes
|
||||||
|
|
||||||
|
The minute-bar path downloads raw Baostock intraday bars and stores them as a
|
||||||
|
Hive-partitioned dataset:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py data download-minute \
|
||||||
|
--universe sh600000 \
|
||||||
|
--start-date 2024-01-02 --end-date 2024-01-05 \
|
||||||
|
--frequency 5
|
||||||
|
```
|
||||||
|
|
||||||
|
The default layout is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
data/minute_bars/{universe}/frequency=5m/month=YYYY-MM/*.pq
|
||||||
|
```
|
||||||
|
|
||||||
|
Derived-data plugins can aggregate those bars to daily `symbol_id,date` numeric
|
||||||
|
files, for example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python cli.py derived compute \
|
||||||
|
--minute-path data/minute_bars/sh600000 \
|
||||||
|
--daily-path data/daily_bars/sh600000 \
|
||||||
|
--derived-type minute_daily_summary \
|
||||||
|
--derived-name minute_summary
|
||||||
|
```
|
||||||
|
|
||||||
|
The legacy `feature compute` command delegates to the same derived-data
|
||||||
|
registry and remains available for existing scripts.
|
||||||
|
|
||||||
|
## Daily vs Minute Reconciliation
|
||||||
|
|
||||||
|
Baostock's daily raw bars and 5-minute raw bars are close, but they should not
|
||||||
|
be treated as perfectly reconstructible from each other.
|
||||||
|
|
||||||
|
When checking consistency, compare daily raw bars (`data download --adjust none`)
|
||||||
|
against minute bars on the same raw price scale. The minute aggregation should
|
||||||
|
use:
|
||||||
|
|
||||||
|
- `open`: first minute open
|
||||||
|
- `high`: max minute high
|
||||||
|
- `low`: min minute low
|
||||||
|
- `close`: last minute close
|
||||||
|
- `volume`: sum minute volume
|
||||||
|
- `amount`: sum minute amount
|
||||||
|
- `vwap`: `sum(amount) / sum(volume)`
|
||||||
|
|
||||||
|
In a sanity check for `sh600000` from `2024-01-02` through `2024-01-05`, Baostock
|
||||||
|
returned 4 daily rows and 192 5-minute bars, exactly 48 bars per day. Open, low,
|
||||||
|
and close matched daily exactly on all 4 days. High matched on 3 of 4 days; on
|
||||||
|
`2024-01-04`, the daily high was `6.67` while the max 5-minute high was `6.66`.
|
||||||
|
Minute-summed volume and amount were higher than daily by roughly `0.16%` to
|
||||||
|
`1.23%`. VWAP remained very close, with max relative difference around
|
||||||
|
`0.0043%`.
|
||||||
|
|
||||||
|
This appears to be a source-level Baostock reconciliation caveat, not a parser
|
||||||
|
or ordering issue: the minute bars covered the regular `09:35:00` through
|
||||||
|
`15:00:00` range and sorted correctly by timestamp.
|
||||||
|
|
||||||
|
Practical guidance:
|
||||||
|
|
||||||
|
- Use tolerance-based daily-vs-minute checks; do not require exact equality for
|
||||||
|
high, volume, or amount.
|
||||||
|
- Expect open/close alignment to be a stronger sanity check than exact volume
|
||||||
|
reconstruction.
|
||||||
|
- Use minute-derived values as separate daily features, not as replacements for
|
||||||
|
the canonical daily bar dataset unless a strategy explicitly wants that
|
||||||
|
source convention.
|
||||||
+19
-5
@@ -5,7 +5,7 @@ position weights. Subclasses implement :meth:`signal` — the raw, unnormalized
|
|||||||
score. The base class turns a signal into cross-sectionally z-scored weights
|
score. The base class turns a signal into cross-sectionally z-scored weights
|
||||||
via :meth:`to_weights` (override it for a different normalization).
|
via :meth:`to_weights` (override it for a different normalization).
|
||||||
"""
|
"""
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
@@ -15,15 +15,14 @@ class BaseAlpha(ABC):
|
|||||||
"""A position-weight alpha over a cross-section of stocks.
|
"""A position-weight alpha over a cross-section of stocks.
|
||||||
|
|
||||||
Concrete subclasses must set a unique class-level :attr:`name` (the registry
|
Concrete subclasses must set a unique class-level :attr:`name` (the registry
|
||||||
key) and implement :meth:`signal`. Construct subclasses with their own typed
|
key) and implement either :meth:`signal` or :meth:`signal_from_data`.
|
||||||
parameters (e.g. ``lookback``); the factory passes only the parameters a
|
Construct subclasses with their own typed parameters (e.g. ``lookback``);
|
||||||
given ``__init__`` accepts.
|
the factory passes only the parameters a given ``__init__`` accepts.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
#: Unique registry key. Every concrete alpha must set this to a non-empty str.
|
#: Unique registry key. Every concrete alpha must set this to a non-empty str.
|
||||||
name: str = ""
|
name: str = ""
|
||||||
|
|
||||||
@abstractmethod
|
|
||||||
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
def signal(self, close: pd.DataFrame) -> pd.DataFrame:
|
||||||
"""Compute the raw signal.
|
"""Compute the raw signal.
|
||||||
|
|
||||||
@@ -34,6 +33,21 @@ class BaseAlpha(ABC):
|
|||||||
A wide DataFrame aligned to ``close`` where higher values indicate a
|
A wide DataFrame aligned to ``close`` where higher values indicate a
|
||||||
stronger long. Use NaN where the signal is undefined.
|
stronger long. Use NaN where the signal is undefined.
|
||||||
"""
|
"""
|
||||||
|
raise NotImplementedError(
|
||||||
|
f"{type(self).__name__} must implement signal() or signal_from_data()"
|
||||||
|
)
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute the raw signal from long daily data plus wide closes.
|
||||||
|
|
||||||
|
Feature-aware alphas can override this to pivot joined feature columns
|
||||||
|
from ``data``. The default preserves the existing close-only alpha API.
|
||||||
|
"""
|
||||||
|
return self.signal(close)
|
||||||
|
|
||||||
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
|
def to_weights(self, signal: pd.DataFrame) -> pd.DataFrame:
|
||||||
"""Cross-sectionally z-score a signal into signed position weights.
|
"""Cross-sectionally z-score a signal into signed position weights.
|
||||||
|
|||||||
+10
-4
@@ -56,6 +56,10 @@ def list_(alpha_modules):
|
|||||||
@click.option("--output-dir", default="alphas", help="Directory to save alpha parquet")
|
@click.option("--output-dir", default="alphas", help="Directory to save alpha parquet")
|
||||||
@click.option("--lookback", default=5, type=int, help="Lookback days")
|
@click.option("--lookback", default=5, type=int, help="Lookback days")
|
||||||
@click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)")
|
@click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)")
|
||||||
|
@click.option(
|
||||||
|
"--feature-path", "feature_paths", multiple=True,
|
||||||
|
help="Daily derived/feature parquet file or dataset to left-join on symbol_id,date (repeatable)",
|
||||||
|
)
|
||||||
@click.option(
|
@click.option(
|
||||||
"--alpha-module", "alpha_modules", multiple=True,
|
"--alpha-module", "alpha_modules", multiple=True,
|
||||||
help="External module(s) to import so their alphas register (dotted path or .py file)",
|
help="External module(s) to import so their alphas register (dotted path or .py file)",
|
||||||
@@ -74,7 +78,7 @@ def list_(alpha_modules):
|
|||||||
help="Most-liquid names kept per date when --liquid-universe is set",
|
help="Most-liquid names kept per date when --liquid-universe is set",
|
||||||
)
|
)
|
||||||
def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
|
def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
|
||||||
alpha_modules, extra_params, liquid_universe, universe_top_n):
|
feature_paths, alpha_modules, extra_params, liquid_universe, universe_top_n):
|
||||||
"""Compute one alpha from raw data and save as parquet."""
|
"""Compute one alpha from raw data and save as parquet."""
|
||||||
for spec in alpha_modules:
|
for spec in alpha_modules:
|
||||||
load_alpha_module(spec)
|
load_alpha_module(spec)
|
||||||
@@ -100,6 +104,7 @@ def compute(data_path, alpha_name, alpha_type, output_dir, lookback, vol_window,
|
|||||||
alpha_name=alpha_name,
|
alpha_name=alpha_name,
|
||||||
alpha_type=alpha_type,
|
alpha_type=alpha_type,
|
||||||
universe=universe,
|
universe=universe,
|
||||||
|
feature_paths=feature_paths,
|
||||||
**params,
|
**params,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -155,7 +160,8 @@ def reversal_vol(data_path, output_dir, lookback, vol_window):
|
|||||||
@alpha.command("eval")
|
@alpha.command("eval")
|
||||||
@click.option("--alpha-path", required=True, help="Path to alpha parquet file")
|
@click.option("--alpha-path", required=True, help="Path to alpha parquet file")
|
||||||
@click.option("--data-path", required=True, help="Path to data parquet (for price data)")
|
@click.option("--data-path", required=True, help="Path to data parquet (for price data)")
|
||||||
def eval_(alpha_path, data_path):
|
@click.option("--report-dir", default="reports", help="Directory to save JSON report")
|
||||||
|
def eval_(alpha_path, data_path, report_dir):
|
||||||
"""Evaluate an alpha's performance (return, Sharpe, turnover).
|
"""Evaluate an alpha's performance (return, Sharpe, turnover).
|
||||||
|
|
||||||
Alphas are interpreted as position WEIGHTS, not return predictors.
|
Alphas are interpreted as position WEIGHTS, not return predictors.
|
||||||
@@ -178,9 +184,9 @@ def eval_(alpha_path, data_path):
|
|||||||
click.echo("=" * 50)
|
click.echo("=" * 50)
|
||||||
|
|
||||||
# Also dump JSON
|
# Also dump JSON
|
||||||
os.makedirs("reports", exist_ok=True)
|
os.makedirs(report_dir, exist_ok=True)
|
||||||
alpha_name = alpha_df["alpha_name"].iloc[0]
|
alpha_name = alpha_df["alpha_name"].iloc[0]
|
||||||
json_path = f"reports/{alpha_name}_eval.json"
|
json_path = os.path.join(report_dir, f"{alpha_name}_eval.json")
|
||||||
with open(json_path, "w") as f:
|
with open(json_path, "w") as f:
|
||||||
json.dump(metrics, f, indent=2)
|
json.dump(metrics, f, indent=2)
|
||||||
click.echo(f"\nReport saved: {json_path}")
|
click.echo(f"\nReport saved: {json_path}")
|
||||||
|
|||||||
@@ -7,12 +7,19 @@ through :mod:`pipeline.alpha.registry`.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
|
|
||||||
from pipeline.alpha.registry import get_alpha
|
from pipeline.alpha.registry import get_alpha
|
||||||
from pipeline.common.schema import ALPHA_COLUMNS
|
from pipeline.common.schema import ALPHA_COLUMNS
|
||||||
|
from pipeline.derived.compute import (
|
||||||
|
DERIVED_KEY_COLUMNS,
|
||||||
|
read_derived_frames,
|
||||||
|
validate_derived_frame,
|
||||||
|
)
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -33,6 +40,38 @@ def _pivot_open(df: pd.DataFrame) -> pd.DataFrame:
|
|||||||
return pivot.sort_index()
|
return pivot.sort_index()
|
||||||
|
|
||||||
|
|
||||||
|
def join_feature_frames(
|
||||||
|
data: pd.DataFrame,
|
||||||
|
feature_frames: Iterable[pd.DataFrame],
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Left-join validated daily derived/feature frames onto long daily data."""
|
||||||
|
out = data.copy()
|
||||||
|
out["date"] = pd.to_datetime(out["date"])
|
||||||
|
existing = set(out.columns)
|
||||||
|
joined_cols: list[str] = []
|
||||||
|
|
||||||
|
for frame in feature_frames:
|
||||||
|
features = validate_derived_frame(frame)
|
||||||
|
feature_cols = [col for col in features.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
overlap = sorted(existing.intersection(feature_cols))
|
||||||
|
if overlap:
|
||||||
|
raise ValueError(
|
||||||
|
f"Feature columns conflict with existing daily data columns: {overlap}"
|
||||||
|
)
|
||||||
|
out = out.merge(
|
||||||
|
features,
|
||||||
|
on=DERIVED_KEY_COLUMNS,
|
||||||
|
how="left",
|
||||||
|
validate="many_to_one",
|
||||||
|
)
|
||||||
|
existing.update(feature_cols)
|
||||||
|
joined_cols.extend(feature_cols)
|
||||||
|
|
||||||
|
if joined_cols:
|
||||||
|
logger.info("Joined feature columns into daily data: %s", joined_cols)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _forward_open_to_open_returns(open_: pd.DataFrame) -> pd.DataFrame:
|
def _forward_open_to_open_returns(open_: pd.DataFrame) -> pd.DataFrame:
|
||||||
"""Return earned by a close-formed signal after next-open execution.
|
"""Return earned by a close-formed signal after next-open execution.
|
||||||
|
|
||||||
@@ -105,6 +144,8 @@ def compute_alpha(
|
|||||||
alpha_name: str,
|
alpha_name: str,
|
||||||
alpha_type: str,
|
alpha_type: str,
|
||||||
universe: dict | None = None,
|
universe: dict | None = None,
|
||||||
|
feature_paths: Iterable[str | Path] | None = None,
|
||||||
|
feature_frames: Iterable[pd.DataFrame] | None = None,
|
||||||
**params,
|
**params,
|
||||||
) -> pd.DataFrame:
|
) -> pd.DataFrame:
|
||||||
"""Compute alpha weights from raw data.
|
"""Compute alpha weights from raw data.
|
||||||
@@ -118,6 +159,10 @@ def compute_alpha(
|
|||||||
:func:`investable_universe_mask`) *before* it is turned into
|
:func:`investable_universe_mask`) *before* it is turned into
|
||||||
weights, so unheld names get weight 0. Keys are forwarded as keyword
|
weights, so unheld names get weight 0. Keys are forwarded as keyword
|
||||||
arguments to :func:`investable_universe_mask`.
|
arguments to :func:`investable_universe_mask`.
|
||||||
|
feature_paths: Optional parquet files/datasets keyed by ``symbol_id``
|
||||||
|
and ``date``. Their numeric feature columns are left-joined onto
|
||||||
|
``data`` before alpha logic runs.
|
||||||
|
feature_frames: Optional in-memory feature frames with the same schema.
|
||||||
**params: Constructor parameters for the alpha (e.g. ``lookback``,
|
**params: Constructor parameters for the alpha (e.g. ``lookback``,
|
||||||
``vol_window``). Only the params the alpha's ``__init__`` accepts are
|
``vol_window``). Only the params the alpha's ``__init__`` accepts are
|
||||||
used; extras are ignored.
|
used; extras are ignored.
|
||||||
@@ -128,12 +173,20 @@ def compute_alpha(
|
|||||||
Raises:
|
Raises:
|
||||||
KeyError: If ``alpha_type`` is not registered.
|
KeyError: If ``alpha_type`` is not registered.
|
||||||
"""
|
"""
|
||||||
|
feature_inputs: list[pd.DataFrame] = []
|
||||||
|
if feature_paths:
|
||||||
|
feature_inputs.extend(read_derived_frames(feature_paths))
|
||||||
|
if feature_frames:
|
||||||
|
feature_inputs.extend(feature_frames)
|
||||||
|
if feature_inputs:
|
||||||
|
data = join_feature_frames(data, feature_inputs)
|
||||||
|
|
||||||
alpha = get_alpha(alpha_type, **params)
|
alpha = get_alpha(alpha_type, **params)
|
||||||
close = _pivot_close(data)
|
close = _pivot_close(data)
|
||||||
|
signal = alpha.signal_from_data(data, close)
|
||||||
if universe is None:
|
if universe is None:
|
||||||
weights = alpha.weights(close)
|
weights = alpha.to_weights(signal)
|
||||||
else:
|
else:
|
||||||
signal = alpha.signal(close)
|
|
||||||
mask = investable_universe_mask(data, signal, **universe)
|
mask = investable_universe_mask(data, signal, **universe)
|
||||||
weights = alpha.to_weights(signal.where(mask))
|
weights = alpha.to_weights(signal.where(mask))
|
||||||
|
|
||||||
|
|||||||
@@ -26,6 +26,31 @@ DATA_COLUMNS: Final[list[str]] = [
|
|||||||
"pcfNcfTTM", # float64: P/CF (net cash flow, TTM)
|
"pcfNcfTTM", # float64: P/CF (net cash flow, TTM)
|
||||||
]
|
]
|
||||||
|
|
||||||
|
# Required columns for raw intraday minute bar parquet files.
|
||||||
|
MINUTE_BAR_COLUMNS: Final[list[str]] = [
|
||||||
|
"symbol_id", # str: internal code like 'sh600000'
|
||||||
|
"symbol_name", # str: stock name like '浦发银行'
|
||||||
|
"datetime", # datetime64: intraday bar timestamp
|
||||||
|
"date", # date component, aligned with daily DATA_COLUMNS date
|
||||||
|
"time", # str: HH:MM:SS bar time
|
||||||
|
"frequency", # str: e.g. '5m'
|
||||||
|
"open", # float64
|
||||||
|
"high", # float64
|
||||||
|
"low", # float64
|
||||||
|
"close", # float64
|
||||||
|
"volume", # float64 (shares)
|
||||||
|
"amount", # float64 (turnover in yuan, raw/unadjusted)
|
||||||
|
"vwap", # float64: amount / volume
|
||||||
|
"adjustflag", # str: baostock adjustment flag; '3' for raw/unadjusted
|
||||||
|
]
|
||||||
|
|
||||||
|
# Required key columns for daily derived-data parquet files. Value columns are
|
||||||
|
# user/plugin-defined and must be numeric.
|
||||||
|
DERIVED_KEY_COLUMNS: Final[list[str]] = [
|
||||||
|
"symbol_id", # str
|
||||||
|
"date", # date: normalized daily timestamp
|
||||||
|
]
|
||||||
|
|
||||||
# Required columns for alpha parquet files.
|
# Required columns for alpha parquet files.
|
||||||
# Alphas are position WEIGHTS: positive=long, negative=short.
|
# Alphas are position WEIGHTS: positive=long, negative=short.
|
||||||
ALPHA_COLUMNS: Final[list[str]] = [
|
ALPHA_COLUMNS: Final[list[str]] = [
|
||||||
|
|||||||
+35
-1
@@ -3,7 +3,7 @@
|
|||||||
import click
|
import click
|
||||||
from datetime import date
|
from datetime import date
|
||||||
|
|
||||||
from pipeline.data.downloader import download_universe
|
from pipeline.data.downloader import download_minute_universe, download_universe
|
||||||
|
|
||||||
|
|
||||||
@click.group(name="data")
|
@click.group(name="data")
|
||||||
@@ -42,3 +42,37 @@ def download(universe, start_date, end_date, output_dir, symbols, chunk_size, ad
|
|||||||
f"{stats['n_rows']:,} bars, {stats['date_min']} → {stats['date_max']}"
|
f"{stats['n_rows']:,} bars, {stats['date_min']} → {stats['date_max']}"
|
||||||
)
|
)
|
||||||
click.echo(f"Dataset: {stats['dataset_path']}")
|
click.echo(f"Dataset: {stats['dataset_path']}")
|
||||||
|
|
||||||
|
|
||||||
|
@data.command("download-minute")
|
||||||
|
@click.option(
|
||||||
|
"--universe", default="csi500",
|
||||||
|
help="Which universe: hs300, csi500, all (~5000 A-shares), file path, or comma-separated symbols",
|
||||||
|
)
|
||||||
|
@click.option("--start-date", default="2017-01-01", help="Start date YYYY-MM-DD")
|
||||||
|
@click.option("--end-date", default=str(date.today()), help="End date YYYY-MM-DD")
|
||||||
|
@click.option("--output-dir", default="data/minute_bars", help="Root for the partitioned dataset")
|
||||||
|
@click.option("--symbols", default=0, type=int, help="Max symbols (0=all)")
|
||||||
|
@click.option("--chunk-size", default=100, type=int, help="Symbols per durability flush")
|
||||||
|
@click.option("--frequency", default="5", help="Minute frequency: 5, 15, 30, or 60")
|
||||||
|
def download_minute(universe, start_date, end_date, output_dir, symbols, chunk_size, frequency):
|
||||||
|
"""Download raw Baostock minute bars into a partitioned parquet dataset.
|
||||||
|
|
||||||
|
Writes ``{output_dir}/{universe}/frequency=5m/month=YYYY-MM/*.pq`` for the
|
||||||
|
default 5-minute frequency.
|
||||||
|
"""
|
||||||
|
stats = download_minute_universe(
|
||||||
|
universe=universe,
|
||||||
|
start_date=start_date,
|
||||||
|
end_date=end_date,
|
||||||
|
output_dir=output_dir,
|
||||||
|
max_symbols=symbols,
|
||||||
|
chunk_size=chunk_size,
|
||||||
|
frequency=frequency,
|
||||||
|
)
|
||||||
|
click.echo(
|
||||||
|
f"\nSummary: {stats['n_symbols']}/{stats['n_requested']} symbols, "
|
||||||
|
f"{stats['n_rows']:,} bars, {stats['date_min']} → {stats['date_max']}, "
|
||||||
|
f"frequency={stats['frequency']}"
|
||||||
|
)
|
||||||
|
click.echo(f"Dataset: {stats['dataset_path']}")
|
||||||
|
|||||||
+126
-2
@@ -11,9 +11,13 @@ import pyarrow.dataset as pads
|
|||||||
|
|
||||||
# Reuse existing downloader and universe modules
|
# Reuse existing downloader and universe modules
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||||
from data.downloader import download_daily_batch
|
from data.downloader import (
|
||||||
|
_normalize_minute_frequency,
|
||||||
|
download_daily_batch,
|
||||||
|
download_minute_batch,
|
||||||
|
)
|
||||||
from data.universe import get_all_stocks, get_hs300_stocks, get_zz500_stocks
|
from data.universe import get_all_stocks, get_hs300_stocks, get_zz500_stocks
|
||||||
from pipeline.common.schema import DATA_COLUMNS
|
from pipeline.common.schema import DATA_COLUMNS, MINUTE_BAR_COLUMNS
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -89,6 +93,25 @@ def _write_month_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: s
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_minute_partitions(df: pd.DataFrame, base_dir: Path, basename_prefix: str) -> None:
|
||||||
|
"""Append rows to a Hive-partitioned minute dataset.
|
||||||
|
|
||||||
|
Layout: ``frequency=5m/month=YYYY-MM/*.pq``.
|
||||||
|
"""
|
||||||
|
out = df.copy()
|
||||||
|
out["month"] = pd.to_datetime(out["date"]).dt.strftime("%Y-%m")
|
||||||
|
table = pa.Table.from_pandas(out, preserve_index=False)
|
||||||
|
pads.write_dataset(
|
||||||
|
table,
|
||||||
|
str(base_dir),
|
||||||
|
format="parquet",
|
||||||
|
partitioning=["frequency", "month"],
|
||||||
|
partitioning_flavor="hive",
|
||||||
|
basename_template=f"{basename_prefix}-{{i}}.pq",
|
||||||
|
existing_data_behavior="overwrite_or_ignore",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def download_universe(
|
def download_universe(
|
||||||
universe: str = "csi500",
|
universe: str = "csi500",
|
||||||
start_date: str = "2017-01-01",
|
start_date: str = "2017-01-01",
|
||||||
@@ -177,3 +200,104 @@ def download_universe(
|
|||||||
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
|
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
|
||||||
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
|
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def download_minute_universe(
|
||||||
|
universe: str = "csi500",
|
||||||
|
start_date: str = "2017-01-01",
|
||||||
|
end_date: str = "2026-12-31",
|
||||||
|
output_dir: str = "data/minute_bars",
|
||||||
|
max_symbols: int = 0,
|
||||||
|
chunk_size: int = 100,
|
||||||
|
frequency: str | int = 5,
|
||||||
|
) -> dict:
|
||||||
|
"""Download raw minute bars into a frequency/month-partitioned dataset.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
universe: ``hs300``, ``csi500``, ``all``/``full``, a file path, or a
|
||||||
|
comma-separated symbol list.
|
||||||
|
start_date, end_date: ``YYYY-MM-DD`` bounds.
|
||||||
|
output_dir: Root under which ``{universe}/frequency=5m/month=YYYY-MM``
|
||||||
|
is written.
|
||||||
|
max_symbols: Cap on symbols (0 = all).
|
||||||
|
chunk_size: Symbols per durability flush.
|
||||||
|
frequency: Minute interval. ``5``/``"5"``/``"5m"`` are 5-minute bars.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Stats dict with dataset path, row count, symbol count, date range, and
|
||||||
|
frequency label.
|
||||||
|
"""
|
||||||
|
_, frequency_label = _normalize_minute_frequency(frequency)
|
||||||
|
constituents = _resolve_universe(universe, max_symbols)
|
||||||
|
symbols = constituents["symbol_id"].tolist()
|
||||||
|
names = dict(zip(constituents["symbol_id"], constituents["symbol_name"]))
|
||||||
|
n_requested = len(symbols)
|
||||||
|
logger.info(
|
||||||
|
"Minute universe %s: %d symbols, %s → %s, frequency=%s",
|
||||||
|
universe,
|
||||||
|
n_requested,
|
||||||
|
start_date,
|
||||||
|
end_date,
|
||||||
|
frequency,
|
||||||
|
)
|
||||||
|
|
||||||
|
base_dir = Path(output_dir) / universe
|
||||||
|
target_frequency_dir = base_dir / f"frequency={frequency_label}"
|
||||||
|
if target_frequency_dir.exists():
|
||||||
|
shutil.rmtree(target_frequency_dir)
|
||||||
|
base_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
buffer: list[pd.DataFrame] = []
|
||||||
|
chunk_idx = 0
|
||||||
|
succeeded = 0
|
||||||
|
n_rows = 0
|
||||||
|
date_min = None
|
||||||
|
date_max = None
|
||||||
|
|
||||||
|
def flush() -> None:
|
||||||
|
nonlocal buffer, chunk_idx, n_rows, date_min, date_max
|
||||||
|
if not buffer:
|
||||||
|
return
|
||||||
|
chunk = pd.concat(buffer, ignore_index=True)
|
||||||
|
_write_minute_partitions(chunk, base_dir, basename_prefix=f"chunk{chunk_idx:04d}")
|
||||||
|
n_rows += len(chunk)
|
||||||
|
cmin, cmax = chunk["date"].min(), chunk["date"].max()
|
||||||
|
date_min = cmin if date_min is None else min(date_min, cmin)
|
||||||
|
date_max = cmax if date_max is None else max(date_max, cmax)
|
||||||
|
logger.info(
|
||||||
|
"Flushed minute chunk %d: %d rows (%d symbols done)",
|
||||||
|
chunk_idx,
|
||||||
|
len(chunk),
|
||||||
|
succeeded,
|
||||||
|
)
|
||||||
|
buffer = []
|
||||||
|
chunk_idx += 1
|
||||||
|
|
||||||
|
for i, (symbol, df) in enumerate(
|
||||||
|
download_minute_batch(symbols, start_date, end_date, frequency=frequency), start=1
|
||||||
|
):
|
||||||
|
if df is None:
|
||||||
|
logger.warning(" %s: no minute data", symbol)
|
||||||
|
else:
|
||||||
|
df["symbol_id"] = symbol
|
||||||
|
df["symbol_name"] = names.get(symbol, symbol)
|
||||||
|
buffer.append(df[MINUTE_BAR_COLUMNS])
|
||||||
|
succeeded += 1
|
||||||
|
if len(buffer) >= chunk_size:
|
||||||
|
flush()
|
||||||
|
if i % 100 == 0:
|
||||||
|
logger.info("Minute progress: %d/%d symbols", i, n_requested)
|
||||||
|
flush()
|
||||||
|
|
||||||
|
if succeeded == 0:
|
||||||
|
raise RuntimeError("No minute data downloaded for any symbol")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"dataset_path": str(base_dir),
|
||||||
|
"frequency": frequency_label,
|
||||||
|
"n_symbols": succeeded,
|
||||||
|
"n_requested": n_requested,
|
||||||
|
"n_rows": n_rows,
|
||||||
|
"date_min": None if date_min is None else str(pd.Timestamp(date_min).date()),
|
||||||
|
"date_max": None if date_max is None else str(pd.Timestamp(date_max).date()),
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
"""Daily derived-data plugin package."""
|
||||||
|
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
"""Base class for daily derived-data plugins."""
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
|
||||||
|
class BaseDerivedData(ABC):
|
||||||
|
"""Compute daily, symbol-keyed numeric derived data.
|
||||||
|
|
||||||
|
Derived-data plugins may use daily bars, minute bars, or both as inputs, but
|
||||||
|
they must always return daily rows keyed by ``symbol_id,date``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
#: Unique registry key. Every concrete derived-data plugin must set this.
|
||||||
|
name: str = ""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
minute: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute daily derived data.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
daily: Optional daily market data.
|
||||||
|
minute: Optional raw minute bars.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
DataFrame with ``symbol_id``, ``date``, and one or more numeric
|
||||||
|
derived-data columns.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
params = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
|
||||||
|
return f"{type(self).__name__}({params})"
|
||||||
|
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
"""CLI for daily derived-data ingestion and computation."""
|
||||||
|
|
||||||
|
import click
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.compute import (
|
||||||
|
DERIVED_KEY_COLUMNS,
|
||||||
|
compute_derived,
|
||||||
|
read_derived_frame,
|
||||||
|
write_derived_frame,
|
||||||
|
)
|
||||||
|
from pipeline.derived.registry import (
|
||||||
|
available_derived,
|
||||||
|
load_derived_module,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@click.group(name="derived")
|
||||||
|
def derived():
|
||||||
|
"""Ingest, compute, and validate daily derived data."""
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce(value: str):
|
||||||
|
"""Best-effort coercion of a CLI string to int, then float, else str."""
|
||||||
|
for cast in (int, float):
|
||||||
|
try:
|
||||||
|
return cast(value)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_params(pairs: tuple[str, ...]) -> dict:
|
||||||
|
"""Parse repeated ``name=value`` options into a params dict."""
|
||||||
|
params: dict = {}
|
||||||
|
for pair in pairs:
|
||||||
|
if "=" not in pair:
|
||||||
|
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
|
||||||
|
key, value = pair.split("=", 1)
|
||||||
|
params[key.strip()] = _coerce(value.strip())
|
||||||
|
return params
|
||||||
|
|
||||||
|
|
||||||
|
def _read_optional_parquet(path: str | None) -> pd.DataFrame | None:
|
||||||
|
return None if path is None else pd.read_parquet(path)
|
||||||
|
|
||||||
|
|
||||||
|
def _summarize(result: pd.DataFrame) -> str:
|
||||||
|
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
return f"{len(result):,} rows, {len(value_cols)} columns"
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("list")
|
||||||
|
@click.option(
|
||||||
|
"--derived-module", "derived_modules", multiple=True,
|
||||||
|
help="External module(s) to import first (dotted path or .py file)",
|
||||||
|
)
|
||||||
|
def list_(derived_modules):
|
||||||
|
"""List the registered derived-data plugin types."""
|
||||||
|
for spec in derived_modules:
|
||||||
|
load_derived_module(spec)
|
||||||
|
for name in available_derived():
|
||||||
|
click.echo(name)
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("validate")
|
||||||
|
@click.option("--input-path", required=True, help="CSV/parquet file or parquet dataset to validate")
|
||||||
|
def validate(input_path):
|
||||||
|
"""Validate a daily derived-data file without writing output."""
|
||||||
|
try:
|
||||||
|
result = read_derived_frame(input_path)
|
||||||
|
except Exception as exc:
|
||||||
|
raise click.ClickException(str(exc)) from exc
|
||||||
|
click.echo(f"Valid derived data: {input_path} ({_summarize(result)})")
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("ingest")
|
||||||
|
@click.option("--input-path", required=True, help="CSV/parquet file to ingest")
|
||||||
|
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
|
||||||
|
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
|
||||||
|
def ingest(input_path, derived_name, output_dir):
|
||||||
|
"""Ingest a user-provided daily derived-data CSV/parquet file."""
|
||||||
|
try:
|
||||||
|
result = read_derived_frame(input_path)
|
||||||
|
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
|
||||||
|
except Exception as exc:
|
||||||
|
raise click.ClickException(str(exc)) from exc
|
||||||
|
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
|
||||||
|
|
||||||
|
|
||||||
|
@derived.command("compute")
|
||||||
|
@click.option("--daily-path", default=None, help="Optional daily data parquet/dataset")
|
||||||
|
@click.option("--minute-path", default=None, help="Optional minute parquet/dataset")
|
||||||
|
@click.option("--derived-type", required=True, help="Registry key of the derived-data plugin")
|
||||||
|
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
|
||||||
|
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
|
||||||
|
@click.option(
|
||||||
|
"--derived-module", "derived_modules", multiple=True,
|
||||||
|
help="External module(s) to import so their derived-data plugins register",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--param", "extra_params", multiple=True,
|
||||||
|
help="Extra derived-data constructor param as name=value (repeatable)",
|
||||||
|
)
|
||||||
|
def compute(
|
||||||
|
daily_path,
|
||||||
|
minute_path,
|
||||||
|
derived_type,
|
||||||
|
derived_name,
|
||||||
|
output_dir,
|
||||||
|
derived_modules,
|
||||||
|
extra_params,
|
||||||
|
):
|
||||||
|
"""Compute one daily derived-data file from daily and/or minute inputs."""
|
||||||
|
for spec in derived_modules:
|
||||||
|
load_derived_module(spec)
|
||||||
|
|
||||||
|
options = available_derived()
|
||||||
|
if derived_type not in options:
|
||||||
|
raise click.BadParameter(
|
||||||
|
f"Unknown derived-type '{derived_type}'. Available: {options}. "
|
||||||
|
f"Use --derived-module to register an external derived-data plugin.",
|
||||||
|
param_hint="--derived-type",
|
||||||
|
)
|
||||||
|
if daily_path is None and minute_path is None:
|
||||||
|
raise click.UsageError("At least one of --daily-path or --minute-path is required")
|
||||||
|
|
||||||
|
daily = _read_optional_parquet(daily_path)
|
||||||
|
if daily_path:
|
||||||
|
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
|
||||||
|
minute = _read_optional_parquet(minute_path)
|
||||||
|
if minute_path:
|
||||||
|
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = compute_derived(
|
||||||
|
derived_type=derived_type,
|
||||||
|
daily=daily,
|
||||||
|
minute=minute,
|
||||||
|
**_parse_params(extra_params),
|
||||||
|
)
|
||||||
|
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
|
||||||
|
except Exception as exc:
|
||||||
|
raise click.ClickException(str(exc)) from exc
|
||||||
|
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
"""Derived-data computation and validation."""
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from pandas.api.types import is_bool_dtype, is_numeric_dtype
|
||||||
|
|
||||||
|
from pipeline.common.schema import DERIVED_KEY_COLUMNS
|
||||||
|
from pipeline.derived.registry import get_derived
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_derived_frame(derived: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""Validate and normalize a daily derived-data frame.
|
||||||
|
|
||||||
|
A valid derived frame is keyed by unique ``symbol_id,date`` rows and has at
|
||||||
|
least one numeric value column beyond those keys. Dates are normalized to
|
||||||
|
daily timestamps before duplicate-key checks.
|
||||||
|
"""
|
||||||
|
duplicated = derived.columns[derived.columns.duplicated()].tolist()
|
||||||
|
if duplicated:
|
||||||
|
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
|
||||||
|
|
||||||
|
missing = [col for col in DERIVED_KEY_COLUMNS if col not in derived.columns]
|
||||||
|
if missing:
|
||||||
|
raise ValueError(f"Derived data missing required columns: {missing}")
|
||||||
|
|
||||||
|
out = derived.copy()
|
||||||
|
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
|
||||||
|
|
||||||
|
if out.duplicated(DERIVED_KEY_COLUMNS).any():
|
||||||
|
raise ValueError("Derived data has duplicate symbol_id,date rows")
|
||||||
|
|
||||||
|
value_cols = [col for col in out.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
if not value_cols:
|
||||||
|
raise ValueError("Derived data must include at least one value column")
|
||||||
|
|
||||||
|
non_numeric = [
|
||||||
|
col
|
||||||
|
for col in value_cols
|
||||||
|
if is_bool_dtype(out[col]) or not is_numeric_dtype(out[col])
|
||||||
|
]
|
||||||
|
if non_numeric:
|
||||||
|
raise ValueError(f"Derived data value columns must be numeric: {non_numeric}")
|
||||||
|
|
||||||
|
out = out[DERIVED_KEY_COLUMNS + value_cols].copy()
|
||||||
|
return out.sort_values(DERIVED_KEY_COLUMNS).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def compute_derived(
|
||||||
|
derived_type: str,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
minute: pd.DataFrame | None = None,
|
||||||
|
**params,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute one registered derived-data plugin."""
|
||||||
|
if daily is None and minute is None:
|
||||||
|
raise ValueError("Derived data computation requires --daily-path or --minute-path")
|
||||||
|
|
||||||
|
derived = get_derived(derived_type, **params)
|
||||||
|
result = validate_derived_frame(derived.compute(daily=daily, minute=minute))
|
||||||
|
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
|
||||||
|
logger.info(
|
||||||
|
"Derived data '%s' (%r): %d symbols × %d dates, columns=%s",
|
||||||
|
derived_type,
|
||||||
|
derived,
|
||||||
|
result["symbol_id"].nunique(),
|
||||||
|
result["date"].nunique(),
|
||||||
|
value_cols,
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def read_derived_frame(path: str | Path) -> pd.DataFrame:
|
||||||
|
"""Read and validate one derived CSV/parquet file or parquet dataset."""
|
||||||
|
path = Path(path)
|
||||||
|
if path.suffix.lower() == ".csv":
|
||||||
|
return validate_derived_frame(_read_csv_with_duplicate_header_check(path))
|
||||||
|
return validate_derived_frame(pd.read_parquet(path))
|
||||||
|
|
||||||
|
|
||||||
|
def read_derived_frames(derived_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
|
||||||
|
"""Read and validate derived-data files."""
|
||||||
|
return [read_derived_frame(path) for path in derived_paths]
|
||||||
|
|
||||||
|
|
||||||
|
def write_derived_frame(
|
||||||
|
derived: pd.DataFrame,
|
||||||
|
derived_name: str,
|
||||||
|
output_dir: str | Path = "derived",
|
||||||
|
) -> Path:
|
||||||
|
"""Validate and write derived data to ``{output_dir}/{derived_name}.pq``."""
|
||||||
|
result = validate_derived_frame(derived)
|
||||||
|
output_dir = Path(output_dir)
|
||||||
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
out_path = output_dir / f"{derived_name}.pq"
|
||||||
|
result.to_parquet(out_path, index=False)
|
||||||
|
return out_path
|
||||||
|
|
||||||
|
|
||||||
|
def _read_csv_with_duplicate_header_check(path: Path) -> pd.DataFrame:
|
||||||
|
with path.open(newline="") as fh:
|
||||||
|
reader = csv.reader(fh)
|
||||||
|
try:
|
||||||
|
header = next(reader)
|
||||||
|
except StopIteration as exc:
|
||||||
|
raise ValueError("CSV input is empty") from exc
|
||||||
|
duplicated = sorted({col for col in header if header.count(col) > 1})
|
||||||
|
if duplicated:
|
||||||
|
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
|
||||||
|
return pd.read_csv(path)
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
"""Built-in derived-data library."""
|
||||||
|
|
||||||
|
from pipeline.derived.library import minute_daily_summary # noqa: F401
|
||||||
|
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
"""Daily summary data derived from raw minute bars."""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import register_derived
|
||||||
|
|
||||||
|
|
||||||
|
@register_derived
|
||||||
|
class MinuteDailySummaryDerived(BaseDerivedData):
|
||||||
|
"""Aggregate intraday bars into daily summary columns."""
|
||||||
|
|
||||||
|
name = "minute_daily_summary"
|
||||||
|
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
minute: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
if minute is None:
|
||||||
|
raise ValueError("minute_daily_summary requires minute input")
|
||||||
|
|
||||||
|
minute = minute.copy()
|
||||||
|
minute["date"] = pd.to_datetime(minute["date"]).dt.normalize()
|
||||||
|
sort_cols = ["symbol_id", "date"]
|
||||||
|
if "datetime" in minute.columns:
|
||||||
|
minute["datetime"] = pd.to_datetime(minute["datetime"])
|
||||||
|
sort_cols.append("datetime")
|
||||||
|
elif "time" in minute.columns:
|
||||||
|
sort_cols.append("time")
|
||||||
|
minute = minute.sort_values(sort_cols)
|
||||||
|
|
||||||
|
grouped = minute.groupby(["symbol_id", "date"], sort=True)
|
||||||
|
summary = grouped.agg(
|
||||||
|
minute_bar_count=("close", "count"),
|
||||||
|
first_open=("open", "first"),
|
||||||
|
last_close=("close", "last"),
|
||||||
|
high=("high", "max"),
|
||||||
|
low=("low", "min"),
|
||||||
|
volume_sum=("volume", "sum"),
|
||||||
|
amount_sum=("amount", "sum"),
|
||||||
|
)
|
||||||
|
summary["minute_intraday_return"] = (
|
||||||
|
summary["last_close"] / summary["first_open"] - 1.0
|
||||||
|
)
|
||||||
|
summary["minute_intraday_range"] = summary["high"] / summary["low"] - 1.0
|
||||||
|
summary["minute_vwap"] = (
|
||||||
|
summary["amount_sum"] / summary["volume_sum"].where(summary["volume_sum"] > 0)
|
||||||
|
)
|
||||||
|
summary = summary.reset_index()
|
||||||
|
|
||||||
|
if daily is not None:
|
||||||
|
daily_keys = daily[["symbol_id", "date"]].copy()
|
||||||
|
daily_keys["date"] = pd.to_datetime(daily_keys["date"]).dt.normalize()
|
||||||
|
daily_keys = daily_keys.drop_duplicates(["symbol_id", "date"])
|
||||||
|
result = daily_keys.merge(summary, on=["symbol_id", "date"], how="left")
|
||||||
|
if "close" in daily.columns:
|
||||||
|
daily_close = daily[["symbol_id", "date", "close"]].copy()
|
||||||
|
daily_close["date"] = pd.to_datetime(daily_close["date"]).dt.normalize()
|
||||||
|
daily_close = daily_close.drop_duplicates(["symbol_id", "date"])
|
||||||
|
result = result.merge(
|
||||||
|
daily_close.rename(columns={"close": "daily_close"}),
|
||||||
|
on=["symbol_id", "date"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
reference_close = result["daily_close"].fillna(result["last_close"])
|
||||||
|
else:
|
||||||
|
reference_close = result["last_close"]
|
||||||
|
else:
|
||||||
|
result = summary
|
||||||
|
reference_close = result["last_close"]
|
||||||
|
|
||||||
|
result["minute_vwap_deviation"] = (
|
||||||
|
result["minute_vwap"] / reference_close.replace(0.0, np.nan) - 1.0
|
||||||
|
)
|
||||||
|
return result[
|
||||||
|
[
|
||||||
|
"symbol_id",
|
||||||
|
"date",
|
||||||
|
"minute_bar_count",
|
||||||
|
"minute_intraday_return",
|
||||||
|
"minute_intraday_range",
|
||||||
|
"minute_vwap",
|
||||||
|
"minute_vwap_deviation",
|
||||||
|
]
|
||||||
|
]
|
||||||
|
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
"""Registry and factory for daily derived-data plugins."""
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import importlib.util
|
||||||
|
import inspect
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional, Type
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
|
||||||
|
_REGISTRY: dict[str, Type[BaseDerivedData]] = {}
|
||||||
|
_builtins_loaded = False
|
||||||
|
|
||||||
|
|
||||||
|
def register_derived(cls: Type[BaseDerivedData]) -> Type[BaseDerivedData]:
|
||||||
|
"""Class decorator that registers derived data under ``BaseDerivedData.name``."""
|
||||||
|
if not (isinstance(cls, type) and issubclass(cls, BaseDerivedData)):
|
||||||
|
raise TypeError(f"{cls!r} is not a BaseDerivedData subclass")
|
||||||
|
key = getattr(cls, "name", "")
|
||||||
|
if not key:
|
||||||
|
raise ValueError(f"{cls.__name__} must set a non-empty class attribute `name`")
|
||||||
|
existing = _REGISTRY.get(key)
|
||||||
|
if existing is not None and existing is not cls:
|
||||||
|
raise ValueError(
|
||||||
|
f"Derived data name '{key}' already registered by {existing.__name__}"
|
||||||
|
)
|
||||||
|
_REGISTRY[key] = cls
|
||||||
|
return cls
|
||||||
|
|
||||||
|
|
||||||
|
def available_derived() -> list[str]:
|
||||||
|
"""Sorted names of all registered derived-data plugins."""
|
||||||
|
_ensure_builtins()
|
||||||
|
return sorted(_REGISTRY)
|
||||||
|
|
||||||
|
|
||||||
|
def get_derived(name: str, **params) -> BaseDerivedData:
|
||||||
|
"""Instantiate a registered derived-data plugin by name.
|
||||||
|
|
||||||
|
Only parameters accepted by the plugin class's ``__init__`` are forwarded.
|
||||||
|
"""
|
||||||
|
_ensure_builtins()
|
||||||
|
if name not in _REGISTRY:
|
||||||
|
raise KeyError(f"Unknown derived data '{name}'. Available: {sorted(_REGISTRY)}")
|
||||||
|
cls = _REGISTRY[name]
|
||||||
|
accepted = _accepted_params(cls)
|
||||||
|
kwargs = params if accepted is None else {k: v for k, v in params.items() if k in accepted}
|
||||||
|
return cls(**kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def load_derived_module(spec: str) -> None:
|
||||||
|
"""Import an external module so its ``@register_derived`` classes register."""
|
||||||
|
looks_like_file = spec.endswith(".py") or Path(spec).expanduser().exists()
|
||||||
|
if looks_like_file:
|
||||||
|
path = Path(spec).expanduser().resolve()
|
||||||
|
if not path.exists():
|
||||||
|
raise FileNotFoundError(f"Derived data module not found: {path}")
|
||||||
|
module_spec = importlib.util.spec_from_file_location(path.stem, path)
|
||||||
|
if module_spec is None or module_spec.loader is None:
|
||||||
|
raise ImportError(f"Cannot load derived data module from {path}")
|
||||||
|
module = importlib.util.module_from_spec(module_spec)
|
||||||
|
module_spec.loader.exec_module(module)
|
||||||
|
else:
|
||||||
|
importlib.import_module(spec)
|
||||||
|
|
||||||
|
|
||||||
|
def _accepted_params(cls: Type[BaseDerivedData]) -> Optional[set[str]]:
|
||||||
|
"""Param names ``cls.__init__`` accepts, or None if it takes ``**kwargs``."""
|
||||||
|
sig = inspect.signature(cls.__init__)
|
||||||
|
if any(p.kind is p.VAR_KEYWORD for p in sig.parameters.values()):
|
||||||
|
return None
|
||||||
|
return {name for name in sig.parameters if name != "self"}
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_builtins() -> None:
|
||||||
|
global _builtins_loaded
|
||||||
|
if not _builtins_loaded:
|
||||||
|
import pipeline.derived.library # noqa: F401
|
||||||
|
_builtins_loaded = True
|
||||||
|
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Daily feature plugin package."""
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
"""Compatibility alias for daily feature plugins.
|
||||||
|
|
||||||
|
The canonical plugin API is ``pipeline.derived``. ``BaseFeature`` remains as an
|
||||||
|
alias so existing external feature modules continue to register unchanged.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData as BaseFeature
|
||||||
|
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
"""CLI for daily feature computation."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
import click
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.features.compute import compute_feature
|
||||||
|
from pipeline.features.registry import available_features, load_feature_module
|
||||||
|
|
||||||
|
|
||||||
|
@click.group(name="feature")
|
||||||
|
def feature():
|
||||||
|
"""Compute daily feature parquet files from minute bars."""
|
||||||
|
|
||||||
|
|
||||||
|
def _coerce(value: str):
|
||||||
|
"""Best-effort coercion of a CLI string to int, then float, else str."""
|
||||||
|
for cast in (int, float):
|
||||||
|
try:
|
||||||
|
return cast(value)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_params(pairs: tuple[str, ...]) -> dict:
|
||||||
|
"""Parse repeated ``name=value`` options into a params dict."""
|
||||||
|
params: dict = {}
|
||||||
|
for pair in pairs:
|
||||||
|
if "=" not in pair:
|
||||||
|
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
|
||||||
|
key, value = pair.split("=", 1)
|
||||||
|
params[key.strip()] = _coerce(value.strip())
|
||||||
|
return params
|
||||||
|
|
||||||
|
|
||||||
|
@feature.command("list")
|
||||||
|
@click.option(
|
||||||
|
"--feature-module", "feature_modules", multiple=True,
|
||||||
|
help="External module(s) to import first (dotted path or .py file)",
|
||||||
|
)
|
||||||
|
def list_(feature_modules):
|
||||||
|
"""List the registered feature types."""
|
||||||
|
for spec in feature_modules:
|
||||||
|
load_feature_module(spec)
|
||||||
|
for name in available_features():
|
||||||
|
click.echo(name)
|
||||||
|
|
||||||
|
|
||||||
|
@feature.command("compute")
|
||||||
|
@click.option("--minute-path", required=True, help="Path to minute parquet dataset/file")
|
||||||
|
@click.option("--daily-path", default=None, help="Optional daily data parquet for alignment")
|
||||||
|
@click.option("--feature-type", required=True, help="Registry key of the feature class")
|
||||||
|
@click.option("--feature-name", required=True, help="Name for this feature run/output file")
|
||||||
|
@click.option("--output-dir", default="features", help="Directory to save feature parquet")
|
||||||
|
@click.option(
|
||||||
|
"--feature-module", "feature_modules", multiple=True,
|
||||||
|
help="External module(s) to import so their features register (dotted path or .py file)",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--param", "extra_params", multiple=True,
|
||||||
|
help="Extra feature constructor param as name=value (repeatable)",
|
||||||
|
)
|
||||||
|
def compute(
|
||||||
|
minute_path,
|
||||||
|
daily_path,
|
||||||
|
feature_type,
|
||||||
|
feature_name,
|
||||||
|
output_dir,
|
||||||
|
feature_modules,
|
||||||
|
extra_params,
|
||||||
|
):
|
||||||
|
"""Compute one daily feature file from raw minute bars."""
|
||||||
|
for spec in feature_modules:
|
||||||
|
load_feature_module(spec)
|
||||||
|
|
||||||
|
options = available_features()
|
||||||
|
if feature_type not in options:
|
||||||
|
raise click.BadParameter(
|
||||||
|
f"Unknown feature-type '{feature_type}'. Available: {options}. "
|
||||||
|
f"Use --feature-module to register an external feature.",
|
||||||
|
param_hint="--feature-type",
|
||||||
|
)
|
||||||
|
|
||||||
|
minute = pd.read_parquet(minute_path)
|
||||||
|
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
|
||||||
|
|
||||||
|
daily = None
|
||||||
|
if daily_path:
|
||||||
|
daily = pd.read_parquet(daily_path)
|
||||||
|
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
|
||||||
|
|
||||||
|
result = compute_feature(
|
||||||
|
minute=minute,
|
||||||
|
daily=daily,
|
||||||
|
feature_type=feature_type,
|
||||||
|
**_parse_params(extra_params),
|
||||||
|
)
|
||||||
|
|
||||||
|
os.makedirs(output_dir, exist_ok=True)
|
||||||
|
out_path = f"{output_dir}/{feature_name}.pq"
|
||||||
|
result.to_parquet(out_path, index=False)
|
||||||
|
feature_cols = [col for col in result.columns if col not in ("symbol_id", "date")]
|
||||||
|
click.echo(
|
||||||
|
f"Saved feature: {out_path} ({len(result):,} rows, "
|
||||||
|
f"{len(feature_cols)} columns)"
|
||||||
|
)
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
"""Compatibility wrappers for daily feature computation and validation."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.compute import (
|
||||||
|
DERIVED_KEY_COLUMNS,
|
||||||
|
compute_derived,
|
||||||
|
read_derived_frames,
|
||||||
|
validate_derived_frame,
|
||||||
|
)
|
||||||
|
|
||||||
|
FEATURE_KEY_COLUMNS = DERIVED_KEY_COLUMNS
|
||||||
|
|
||||||
|
|
||||||
|
def validate_feature_frame(features: pd.DataFrame) -> pd.DataFrame:
|
||||||
|
"""Validate and normalize a legacy daily feature frame."""
|
||||||
|
return validate_derived_frame(features)
|
||||||
|
|
||||||
|
|
||||||
|
def compute_feature(
|
||||||
|
minute: pd.DataFrame,
|
||||||
|
feature_type: str,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
**params,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Compute one registered feature through the derived-data registry."""
|
||||||
|
return compute_derived(
|
||||||
|
derived_type=feature_type,
|
||||||
|
daily=daily,
|
||||||
|
minute=minute,
|
||||||
|
**params,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def read_feature_frames(feature_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
|
||||||
|
"""Read and validate feature/derived-data parquet files."""
|
||||||
|
return read_derived_frames(feature_paths)
|
||||||
|
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
"""Built-in feature library."""
|
||||||
|
|
||||||
|
from pipeline.features.library import minute_daily_summary # noqa: F401
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
"""Compatibility wrapper for the built-in minute daily summary plugin."""
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.derived.library.minute_daily_summary import MinuteDailySummaryDerived
|
||||||
|
|
||||||
|
|
||||||
|
class MinuteDailySummaryFeature(MinuteDailySummaryDerived):
|
||||||
|
"""Legacy minute-first wrapper around the derived-data implementation."""
|
||||||
|
|
||||||
|
def compute(
|
||||||
|
self,
|
||||||
|
minute: pd.DataFrame,
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
return super().compute(daily=daily, minute=minute)
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""Compatibility registry wrappers for daily feature plugins."""
|
||||||
|
|
||||||
|
from typing import Type
|
||||||
|
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import (
|
||||||
|
available_derived,
|
||||||
|
get_derived,
|
||||||
|
load_derived_module,
|
||||||
|
register_derived,
|
||||||
|
)
|
||||||
|
from pipeline.features.base import BaseFeature
|
||||||
|
|
||||||
|
|
||||||
|
def register_feature(cls: Type[BaseFeature]) -> Type[BaseFeature]:
|
||||||
|
"""Register a legacy feature plugin in the derived-data registry."""
|
||||||
|
return register_derived(cls)
|
||||||
|
|
||||||
|
|
||||||
|
def available_features() -> list[str]:
|
||||||
|
"""Sorted names of all registered feature/derived-data plugins."""
|
||||||
|
return available_derived()
|
||||||
|
|
||||||
|
|
||||||
|
def get_feature(name: str, **params) -> BaseDerivedData:
|
||||||
|
"""Instantiate a registered feature/derived-data plugin by name."""
|
||||||
|
if name == "minute_daily_summary":
|
||||||
|
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
|
||||||
|
|
||||||
|
return MinuteDailySummaryFeature(**params)
|
||||||
|
return get_derived(name, **params)
|
||||||
|
|
||||||
|
|
||||||
|
def load_feature_module(spec: str) -> None:
|
||||||
|
"""Import an external module so its ``@register_feature`` classes register."""
|
||||||
|
load_derived_module(spec)
|
||||||
@@ -166,13 +166,14 @@ class ReferenceSimulator(ExecutionSimulator):
|
|||||||
st = wide(data_df, "isST") if "isST" in data_df.columns else opn * 0.0
|
st = wide(data_df, "isST") if "isST" in data_df.columns else opn * 0.0
|
||||||
|
|
||||||
symbols = sorted(set(tgt.columns) | set(opn.columns))
|
symbols = sorted(set(tgt.columns) | set(opn.columns))
|
||||||
|
data_index = close.index
|
||||||
tgt = tgt.reindex(columns=symbols)
|
tgt = tgt.reindex(columns=symbols)
|
||||||
opn = opn.reindex(columns=symbols)
|
opn = opn.reindex(index=data_index, columns=symbols)
|
||||||
close = close.reindex(columns=symbols)
|
close = close.reindex(columns=symbols)
|
||||||
preclose = preclose.reindex(columns=symbols)
|
preclose = preclose.reindex(index=data_index, columns=symbols)
|
||||||
amount = amount.reindex(columns=symbols)
|
amount = amount.reindex(index=data_index, columns=symbols)
|
||||||
tstat = tstat.reindex(columns=symbols)
|
tstat = tstat.reindex(index=data_index, columns=symbols)
|
||||||
st = st.reindex(columns=symbols)
|
st = st.reindex(index=data_index, columns=symbols)
|
||||||
|
|
||||||
sym_arr = np.asarray(symbols, dtype=object)
|
sym_arr = np.asarray(symbols, dtype=object)
|
||||||
n = len(symbols)
|
n = len(symbols)
|
||||||
|
|||||||
@@ -19,8 +19,32 @@ backtrader = [
|
|||||||
|
|
||||||
[dependency-groups]
|
[dependency-groups]
|
||||||
dev = [
|
dev = [
|
||||||
|
"coverage>=7.14.1",
|
||||||
"pytest>=7.0.0",
|
"pytest>=7.0.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.uv]
|
[tool.uv]
|
||||||
package = false
|
package = false
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
markers = [
|
||||||
|
"network: tests that call live external data providers and are skipped unless explicitly enabled",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.coverage.run]
|
||||||
|
branch = true
|
||||||
|
relative_files = true
|
||||||
|
source = [
|
||||||
|
".",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.coverage.report]
|
||||||
|
fail_under = 80
|
||||||
|
show_missing = true
|
||||||
|
skip_covered = false
|
||||||
|
omit = [
|
||||||
|
"tests/*",
|
||||||
|
"docs/*",
|
||||||
|
"scripts/*",
|
||||||
|
".venv/*",
|
||||||
|
]
|
||||||
|
|||||||
@@ -0,0 +1,40 @@
|
|||||||
|
"""Run pytest under coverage in this environment.
|
||||||
|
|
||||||
|
The plain ``coverage run -m pytest`` path reloads NumPy under the current VS Code
|
||||||
|
Python startup environment, which breaks pandas/numpy reductions. Import NumPy
|
||||||
|
before starting coverage so the measured test run uses one stable NumPy module.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import numpy # noqa: F401
|
||||||
|
import coverage
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str]) -> int:
|
||||||
|
pytest_args = argv or ["tests/", "-v"]
|
||||||
|
|
||||||
|
cov = coverage.Coverage(config_file=True)
|
||||||
|
cov.erase()
|
||||||
|
cov.start()
|
||||||
|
test_status = pytest.main(pytest_args)
|
||||||
|
cov.stop()
|
||||||
|
cov.save()
|
||||||
|
|
||||||
|
if test_status != 0:
|
||||||
|
return int(test_status)
|
||||||
|
|
||||||
|
total = cov.report()
|
||||||
|
print(f"\nCoverage total: {total:.2f}%")
|
||||||
|
fail_under = float(cov.config.fail_under)
|
||||||
|
if total < fail_under:
|
||||||
|
print(f"Coverage failure: total {total:.2f}% is below {fail_under:.2f}%")
|
||||||
|
return 2
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main(sys.argv[1:]))
|
||||||
Binary file not shown.
@@ -0,0 +1,261 @@
|
|||||||
|
"""Shared deterministic test data for offline workflow tests."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.common.schema import (
|
||||||
|
ALPHA_COLUMNS,
|
||||||
|
COMBO_COLUMNS,
|
||||||
|
DATA_COLUMNS,
|
||||||
|
MINUTE_BAR_COLUMNS,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
GENERATED_SYMBOLS: tuple[str, ...] = (
|
||||||
|
"sh600000",
|
||||||
|
"sz000001",
|
||||||
|
"sh600519",
|
||||||
|
"sz300750",
|
||||||
|
)
|
||||||
|
|
||||||
|
GENERATED_SYMBOL_NAMES: dict[str, str] = {
|
||||||
|
"sh600000": "PF Bank",
|
||||||
|
"sz000001": "Ping An Bank",
|
||||||
|
"sh600519": "Kweichow Moutai",
|
||||||
|
"sz300750": "CATL",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def generated_sessions(n_sessions: int = 12) -> pd.DatetimeIndex:
|
||||||
|
"""Return a fixed business-day calendar used by generated fixtures."""
|
||||||
|
return pd.bdate_range("2024-01-02", periods=n_sessions)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_daily_bars(
|
||||||
|
n_sessions: int = 12,
|
||||||
|
include_missing: bool = True,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Build daily bars with explicit edge cases and no randomness.
|
||||||
|
|
||||||
|
The panel covers four A-share symbols and includes a suspended row, an ST
|
||||||
|
flag, a zero-volume row, a missing symbol-date, and limit-style open/close
|
||||||
|
moves. Values are deterministic so tests can assert exact identities.
|
||||||
|
"""
|
||||||
|
dates = generated_sessions(n_sessions)
|
||||||
|
base_close = {
|
||||||
|
"sh600000": 10.00,
|
||||||
|
"sz000001": 15.00,
|
||||||
|
"sh600519": 1200.00,
|
||||||
|
"sz300750": 180.00,
|
||||||
|
}
|
||||||
|
returns = {
|
||||||
|
"sh600000": [0.000, 0.012, -0.006, 0.018, 0.100, -0.014, 0.006, 0.000, 0.008, -0.011, 0.004, 0.009],
|
||||||
|
"sz000001": [0.000, -0.008, 0.011, -0.004, 0.006, 0.000, -0.012, 0.009, 0.005, -0.007, 0.010, -0.003],
|
||||||
|
"sh600519": [0.000, 0.006, 0.004, -0.010, 0.012, -0.006, 0.005, 0.003, -0.009, 0.007, -0.004, 0.006],
|
||||||
|
"sz300750": [0.000, -0.010, 0.014, 0.006, -0.008, 0.011, -0.004, 0.009, -0.200, 0.012, -0.006, 0.008],
|
||||||
|
}
|
||||||
|
base_volume = {
|
||||||
|
"sh600000": 1_200_000.0,
|
||||||
|
"sz000001": 900_000.0,
|
||||||
|
"sh600519": 80_000.0,
|
||||||
|
"sz300750": 240_000.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
for sym in GENERATED_SYMBOLS:
|
||||||
|
closes = [base_close[sym]]
|
||||||
|
pattern = returns[sym]
|
||||||
|
for step in range(1, n_sessions):
|
||||||
|
ret = pattern[step % len(pattern)]
|
||||||
|
closes.append(closes[-1] * (1.0 + ret))
|
||||||
|
closes_arr = np.asarray(closes, dtype=float)
|
||||||
|
precloses = np.concatenate([[closes_arr[0]], closes_arr[:-1]])
|
||||||
|
|
||||||
|
for i, date in enumerate(dates):
|
||||||
|
preclose = float(precloses[i])
|
||||||
|
close = float(closes_arr[i])
|
||||||
|
open_price = preclose * (1.0 + 0.25 * (close / preclose - 1.0))
|
||||||
|
high = max(open_price, close) * 1.01
|
||||||
|
low = min(open_price, close) * 0.99
|
||||||
|
volume = base_volume[sym] + 10_000.0 * i
|
||||||
|
tradestatus = 1
|
||||||
|
is_st = 0
|
||||||
|
|
||||||
|
if sym == "sh600000" and i == 4:
|
||||||
|
open_price = preclose * 1.10
|
||||||
|
close = open_price
|
||||||
|
high = open_price
|
||||||
|
low = open_price
|
||||||
|
if sym == "sh600000" and i == 7:
|
||||||
|
volume = 0.0
|
||||||
|
if sym == "sz000001" and i == 5:
|
||||||
|
open_price = preclose
|
||||||
|
close = preclose
|
||||||
|
high = preclose
|
||||||
|
low = preclose
|
||||||
|
volume = 0.0
|
||||||
|
tradestatus = 0
|
||||||
|
if sym == "sh600519" and i == 7:
|
||||||
|
is_st = 1
|
||||||
|
if sym == "sz300750" and i == 8:
|
||||||
|
open_price = preclose * 0.80
|
||||||
|
close = open_price
|
||||||
|
high = open_price
|
||||||
|
low = open_price
|
||||||
|
|
||||||
|
amount = volume * ((open_price + close) / 2.0)
|
||||||
|
vwap = amount / volume if volume > 0 else np.nan
|
||||||
|
pct_chg = (close / preclose - 1.0) * 100.0 if preclose else 0.0
|
||||||
|
rows.append({
|
||||||
|
"symbol_id": sym,
|
||||||
|
"symbol_name": GENERATED_SYMBOL_NAMES[sym],
|
||||||
|
"date": date,
|
||||||
|
"open": open_price,
|
||||||
|
"high": high,
|
||||||
|
"low": low,
|
||||||
|
"close": close,
|
||||||
|
"preclose": preclose,
|
||||||
|
"volume": volume,
|
||||||
|
"amount": amount,
|
||||||
|
"vwap": vwap,
|
||||||
|
"turn": volume / 1_000_000.0,
|
||||||
|
"pctChg": pct_chg,
|
||||||
|
"tradestatus": tradestatus,
|
||||||
|
"isST": is_st,
|
||||||
|
"peTTM": 8.0 + i,
|
||||||
|
"pbMRQ": 1.0 + 0.05 * i,
|
||||||
|
"psTTM": 2.0 + 0.03 * i,
|
||||||
|
"pcfNcfTTM": 5.0 + 0.1 * i,
|
||||||
|
})
|
||||||
|
|
||||||
|
result = pd.DataFrame(rows)
|
||||||
|
if include_missing and n_sessions > 6:
|
||||||
|
missing_mask = (
|
||||||
|
(result["symbol_id"] == "sz300750")
|
||||||
|
& (result["date"] == dates[6])
|
||||||
|
)
|
||||||
|
result = result.loc[~missing_mask].copy()
|
||||||
|
|
||||||
|
result = result[DATA_COLUMNS]
|
||||||
|
return result.sort_values(["date", "symbol_id"]).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_minute_bars(
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Expand generated daily bars into a tiny deterministic intraday panel."""
|
||||||
|
daily = make_generated_daily_bars() if daily is None else daily.copy()
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
bar_times = ("09:35:00", "10:30:00", "14:55:00")
|
||||||
|
|
||||||
|
for daily_row in daily.sort_values(["date", "symbol_id"]).itertuples(index=False):
|
||||||
|
if int(getattr(daily_row, "tradestatus", 1)) == 0:
|
||||||
|
continue
|
||||||
|
|
||||||
|
volume = float(daily_row.volume)
|
||||||
|
volume_slices = [0.25 * volume, 0.35 * volume, 0.40 * volume]
|
||||||
|
prices = np.linspace(float(daily_row.open), float(daily_row.close), len(bar_times))
|
||||||
|
for j, time_text in enumerate(bar_times):
|
||||||
|
dt = pd.Timestamp(daily_row.date) + pd.Timedelta(time_text)
|
||||||
|
open_price = prices[j - 1] if j else float(daily_row.open)
|
||||||
|
close_price = float(prices[j])
|
||||||
|
high = max(open_price, close_price) * 1.002
|
||||||
|
low = min(open_price, close_price) * 0.998
|
||||||
|
minute_volume = float(volume_slices[j])
|
||||||
|
amount = minute_volume * ((open_price + close_price) / 2.0)
|
||||||
|
rows.append({
|
||||||
|
"symbol_id": daily_row.symbol_id,
|
||||||
|
"symbol_name": daily_row.symbol_name,
|
||||||
|
"datetime": dt,
|
||||||
|
"date": pd.Timestamp(daily_row.date).normalize(),
|
||||||
|
"time": time_text,
|
||||||
|
"frequency": "5m",
|
||||||
|
"open": open_price,
|
||||||
|
"high": high,
|
||||||
|
"low": low,
|
||||||
|
"close": close_price,
|
||||||
|
"volume": minute_volume,
|
||||||
|
"amount": amount,
|
||||||
|
"vwap": amount / minute_volume if minute_volume > 0 else np.nan,
|
||||||
|
"adjustflag": "3",
|
||||||
|
})
|
||||||
|
|
||||||
|
return pd.DataFrame(rows, columns=MINUTE_BAR_COLUMNS)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_derived_features(
|
||||||
|
daily: pd.DataFrame | None = None,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Return numeric daily derived values, including NaN and infinity cells."""
|
||||||
|
daily = make_generated_daily_bars() if daily is None else daily.copy()
|
||||||
|
keys = (
|
||||||
|
daily[["symbol_id", "date"]]
|
||||||
|
.drop_duplicates()
|
||||||
|
.sort_values(["date", "symbol_id"])
|
||||||
|
.reset_index(drop=True)
|
||||||
|
)
|
||||||
|
date_rank = keys["date"].rank(method="dense").astype(float)
|
||||||
|
symbol_rank = keys["symbol_id"].map({
|
||||||
|
"sh600000": 1.0,
|
||||||
|
"sz000001": 2.0,
|
||||||
|
"sh600519": 3.0,
|
||||||
|
"sz300750": 4.0,
|
||||||
|
})
|
||||||
|
out = keys.copy()
|
||||||
|
out["toy_feature"] = symbol_rank + date_rank / 100.0
|
||||||
|
out["finite_feature"] = symbol_rank * date_rank
|
||||||
|
out["nan_feature"] = out["toy_feature"]
|
||||||
|
out["inf_feature"] = out["toy_feature"]
|
||||||
|
if len(out) >= 2:
|
||||||
|
out.loc[out.index[0], "nan_feature"] = np.nan
|
||||||
|
out.loc[out.index[1], "inf_feature"] = np.inf
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_alpha_weights(
|
||||||
|
alpha_name: str = "alpha_a",
|
||||||
|
*,
|
||||||
|
scale: float = 1.0,
|
||||||
|
offset: float = 0.0,
|
||||||
|
zero_date_index: int | None = None,
|
||||||
|
n_sessions: int = 10,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Create a deterministic long alpha grid with optional zero-gross date."""
|
||||||
|
dates = generated_sessions(n_sessions)
|
||||||
|
even = np.array([1.20, -0.80, 0.40, -0.80], dtype=float)
|
||||||
|
odd = np.array([-0.60, 1.10, -0.90, 0.40], dtype=float)
|
||||||
|
rows: list[dict[str, object]] = []
|
||||||
|
for i, date in enumerate(dates):
|
||||||
|
vector = even.copy() if i % 2 == 0 else odd.copy()
|
||||||
|
vector = vector + offset
|
||||||
|
vector = vector - vector.mean()
|
||||||
|
if zero_date_index is not None and i == zero_date_index:
|
||||||
|
vector = np.zeros_like(vector)
|
||||||
|
for sym, weight in zip(GENERATED_SYMBOLS, scale * vector):
|
||||||
|
rows.append({
|
||||||
|
"symbol_id": sym,
|
||||||
|
"date": date,
|
||||||
|
"alpha_name": alpha_name,
|
||||||
|
"weight": float(weight),
|
||||||
|
})
|
||||||
|
result = pd.DataFrame(rows, columns=ALPHA_COLUMNS)
|
||||||
|
return result.sort_values(["symbol_id", "date"]).reset_index(drop=True)
|
||||||
|
|
||||||
|
|
||||||
|
def make_generated_combo_weights(
|
||||||
|
combo_name: str = "combo",
|
||||||
|
*,
|
||||||
|
zero_date_index: int | None = 2,
|
||||||
|
n_sessions: int = 10,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
"""Create deterministic combo weights for portfolio construction tests."""
|
||||||
|
alpha = make_generated_alpha_weights(
|
||||||
|
"combo_source",
|
||||||
|
zero_date_index=zero_date_index,
|
||||||
|
n_sessions=n_sessions,
|
||||||
|
)
|
||||||
|
combo = alpha.rename(columns={"alpha_name": "combo_name"}).copy()
|
||||||
|
combo["combo_name"] = combo_name
|
||||||
|
return combo[COMBO_COLUMNS].sort_values(["symbol_id", "date"]).reset_index(drop=True)
|
||||||
@@ -347,3 +347,158 @@ def test_universe_filter_does_not_corrupt_signal_history():
|
|||||||
held = set(filtered.loc[filtered["weight"] != 0.0, "symbol_id"].unique())
|
held = set(filtered.loc[filtered["weight"] != 0.0, "symbol_id"].unique())
|
||||||
# The two most liquid names (highest amount) are sh600519, sz300750.
|
# The two most liquid names (highest amount) are sh600519, sz300750.
|
||||||
assert held == {"sh600519", "sz300750"}
|
assert held == {"sh600519", "sz300750"}
|
||||||
|
|
||||||
|
|
||||||
|
# --- feature-aware alpha integration ----------------------------------------
|
||||||
|
|
||||||
|
def test_compute_alpha_without_feature_path_matches_empty_feature_paths():
|
||||||
|
data = _make_data()
|
||||||
|
|
||||||
|
base = compute_alpha(data, "rev5", "reversal", lookback=5)
|
||||||
|
with_empty_features = compute_alpha(
|
||||||
|
data,
|
||||||
|
"rev5",
|
||||||
|
"reversal",
|
||||||
|
lookback=5,
|
||||||
|
feature_paths=[],
|
||||||
|
)
|
||||||
|
|
||||||
|
pd.testing.assert_frame_equal(base, with_empty_features)
|
||||||
|
|
||||||
|
|
||||||
|
def test_feature_aware_alpha_reads_joined_feature_column(tmp_path):
|
||||||
|
module_path = tmp_path / "feature_aware_alpha.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class FeatureAwareAlpha(BaseAlpha):
|
||||||
|
name = "feature_aware_test_alpha"
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="toy_feature",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
return signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
'''))
|
||||||
|
|
||||||
|
data = _make_data()
|
||||||
|
feature = data[["symbol_id", "date"]].copy()
|
||||||
|
feature["toy_feature"] = feature["symbol_id"].map({
|
||||||
|
"sh600000": 1.0,
|
||||||
|
"sz000001": 2.0,
|
||||||
|
"sh600519": 3.0,
|
||||||
|
})
|
||||||
|
feature_path = tmp_path / "toy_feature.pq"
|
||||||
|
feature.to_parquet(feature_path, index=False)
|
||||||
|
|
||||||
|
load_alpha_module(str(module_path))
|
||||||
|
result = compute_alpha(
|
||||||
|
data,
|
||||||
|
"feature_run",
|
||||||
|
"feature_aware_test_alpha",
|
||||||
|
feature_paths=[str(feature_path)],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(result.columns) == ALPHA_COLUMNS
|
||||||
|
assert (result["alpha_name"] == "feature_run").all()
|
||||||
|
last = result[result["date"] == result["date"].max()]
|
||||||
|
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
|
||||||
|
|
||||||
|
|
||||||
|
def test_feature_paths_join_multiple_files_and_normalize_dates(tmp_path):
|
||||||
|
module_path = tmp_path / "multi_feature_alpha.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class MultiFeatureAlpha(BaseAlpha):
|
||||||
|
name = "multi_feature_test_alpha"
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
data = data.copy()
|
||||||
|
data["combined_feature"] = data["toy_a"] + data["toy_b"]
|
||||||
|
signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="combined_feature",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
return signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
'''))
|
||||||
|
|
||||||
|
data = _make_data(n_days=8)
|
||||||
|
symbol_score = {"sh600000": 1.0, "sz000001": 2.0, "sh600519": 3.0}
|
||||||
|
|
||||||
|
feature_a = data[["symbol_id", "date"]].copy()
|
||||||
|
feature_a["date"] = feature_a["date"] + pd.Timedelta(hours=15)
|
||||||
|
feature_a["toy_a"] = feature_a["symbol_id"].map(symbol_score)
|
||||||
|
|
||||||
|
feature_b = data[["symbol_id", "date"]].copy()
|
||||||
|
feature_b["date"] = feature_b["date"].dt.strftime("%Y-%m-%d 09:30:00")
|
||||||
|
feature_b["toy_b"] = feature_b["symbol_id"].map(symbol_score) * 10.0
|
||||||
|
|
||||||
|
feature_a_path = tmp_path / "toy_a.pq"
|
||||||
|
feature_b_path = tmp_path / "toy_b.pq"
|
||||||
|
feature_a.to_parquet(feature_a_path, index=False)
|
||||||
|
feature_b.to_parquet(feature_b_path, index=False)
|
||||||
|
|
||||||
|
load_alpha_module(str(module_path))
|
||||||
|
result = compute_alpha(
|
||||||
|
data,
|
||||||
|
"multi_feature_run",
|
||||||
|
"multi_feature_test_alpha",
|
||||||
|
feature_paths=[str(feature_a_path), str(feature_b_path)],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(result.columns) == ALPHA_COLUMNS
|
||||||
|
assert (result["alpha_name"] == "multi_feature_run").all()
|
||||||
|
last = result[result["date"] == result["date"].max()]
|
||||||
|
assert last.set_index("symbol_id")["weight"].idxmax() == "sh600519"
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_alpha_rejects_duplicate_feature_frame_columns():
|
||||||
|
data = _make_data()
|
||||||
|
duplicate_columns = pd.DataFrame(
|
||||||
|
[["sh600000", pd.Timestamp("2024-01-01"), 1.0, 2.0]],
|
||||||
|
columns=["symbol_id", "date", "toy_feature", "toy_feature"],
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="duplicate columns"):
|
||||||
|
compute_alpha(
|
||||||
|
data,
|
||||||
|
"bad_features",
|
||||||
|
"reversal",
|
||||||
|
feature_frames=[duplicate_columns],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_alpha_rejects_feature_path_collision_with_daily_data(tmp_path):
|
||||||
|
data = _make_data()
|
||||||
|
close_collision = data[["symbol_id", "date"]].copy()
|
||||||
|
close_collision["close"] = 1.0
|
||||||
|
close_collision_path = tmp_path / "close_collision.pq"
|
||||||
|
close_collision.to_parquet(close_collision_path, index=False)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="conflict"):
|
||||||
|
compute_alpha(
|
||||||
|
data,
|
||||||
|
"close_collision",
|
||||||
|
"reversal",
|
||||||
|
feature_paths=[str(close_collision_path)],
|
||||||
|
)
|
||||||
|
|||||||
@@ -0,0 +1,545 @@
|
|||||||
|
"""CLI handoff tests for the offline daily workflow."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import textwrap
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from cli import cli
|
||||||
|
from tests.helpers import (
|
||||||
|
make_generated_daily_bars,
|
||||||
|
make_generated_derived_features,
|
||||||
|
make_generated_minute_bars,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_ok(runner: CliRunner, args: list[str]):
|
||||||
|
result = runner.invoke(cli, args)
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_error(runner: CliRunner, args: list[str]):
|
||||||
|
result = runner.invoke(cli, args)
|
||||||
|
assert result.exit_code != 0, result.output
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_daily_workflow_handoffs_stay_in_tmp_path(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars()
|
||||||
|
minute_bars = make_generated_minute_bars(daily_bars)
|
||||||
|
derived_features = make_generated_derived_features(daily_bars)
|
||||||
|
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
minute_path = tmp_path / "minute_bars.pq"
|
||||||
|
derived_input_path = tmp_path / "derived_input.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
minute_bars.to_parquet(minute_path, index=False)
|
||||||
|
derived_features.to_parquet(derived_input_path, index=False)
|
||||||
|
|
||||||
|
ingest_dir = tmp_path / "derived_ingested"
|
||||||
|
ingest_result = _invoke_ok(runner, [
|
||||||
|
"derived", "ingest",
|
||||||
|
"--input-path", str(derived_input_path),
|
||||||
|
"--derived-name", "toy_features",
|
||||||
|
"--output-dir", str(ingest_dir),
|
||||||
|
])
|
||||||
|
ingested_feature_path = ingest_dir / "toy_features.pq"
|
||||||
|
assert "Saved derived data:" in ingest_result.output
|
||||||
|
assert ingested_feature_path.exists()
|
||||||
|
|
||||||
|
validate_result = _invoke_ok(runner, [
|
||||||
|
"derived", "validate",
|
||||||
|
"--input-path", str(ingested_feature_path),
|
||||||
|
])
|
||||||
|
assert "Valid derived data:" in validate_result.output
|
||||||
|
assert "rows" in validate_result.output
|
||||||
|
|
||||||
|
computed_derived_dir = tmp_path / "derived_computed"
|
||||||
|
derived_compute_result = _invoke_ok(runner, [
|
||||||
|
"derived", "compute",
|
||||||
|
"--daily-path", str(daily_path),
|
||||||
|
"--minute-path", str(minute_path),
|
||||||
|
"--derived-type", "minute_daily_summary",
|
||||||
|
"--derived-name", "minute_summary",
|
||||||
|
"--output-dir", str(computed_derived_dir),
|
||||||
|
])
|
||||||
|
minute_summary_path = computed_derived_dir / "minute_summary.pq"
|
||||||
|
assert "Loaded daily data:" in derived_compute_result.output
|
||||||
|
assert "Loaded minute bars:" in derived_compute_result.output
|
||||||
|
assert "Saved derived data:" in derived_compute_result.output
|
||||||
|
assert minute_summary_path.exists()
|
||||||
|
assert "minute_vwap" in pd.read_parquet(minute_summary_path).columns
|
||||||
|
|
||||||
|
alpha_module_path = tmp_path / "cli_feature_alpha.py"
|
||||||
|
alpha_module_path.write_text(textwrap.dedent("""
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.alpha.base import BaseAlpha
|
||||||
|
from pipeline.alpha.registry import register_alpha
|
||||||
|
|
||||||
|
@register_alpha
|
||||||
|
class CliFeatureAlpha(BaseAlpha):
|
||||||
|
name = "cli_feature_alpha_workflow"
|
||||||
|
|
||||||
|
def __init__(self, **kwargs):
|
||||||
|
self.kwargs = kwargs
|
||||||
|
|
||||||
|
def signal_from_data(
|
||||||
|
self,
|
||||||
|
data: pd.DataFrame,
|
||||||
|
close: pd.DataFrame,
|
||||||
|
) -> pd.DataFrame:
|
||||||
|
signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="minute_intraday_return",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
fallback = close.pct_change(1, fill_method=None)
|
||||||
|
feature_signal = signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
toy_signal = data.pivot_table(
|
||||||
|
index="date",
|
||||||
|
columns="symbol_id",
|
||||||
|
values="toy_feature",
|
||||||
|
aggfunc="first",
|
||||||
|
)
|
||||||
|
toy_signal = toy_signal.reindex(index=close.index, columns=close.columns)
|
||||||
|
return feature_signal.fillna(fallback) + toy_signal / 1000.0
|
||||||
|
"""))
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
alpha_result = _invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--feature-path", str(minute_summary_path),
|
||||||
|
"--feature-path", str(ingested_feature_path),
|
||||||
|
"--alpha-module", str(alpha_module_path),
|
||||||
|
"--alpha-type", "cli_feature_alpha_workflow",
|
||||||
|
"--alpha-name", "cli_feature_alpha",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
alpha_path = alpha_dir / "cli_feature_alpha.pq"
|
||||||
|
assert "Loaded data:" in alpha_result.output
|
||||||
|
assert "Saved alpha:" in alpha_result.output
|
||||||
|
assert "Weight stats" in alpha_result.output
|
||||||
|
assert alpha_path.exists()
|
||||||
|
assert not pd.read_parquet(alpha_path).empty
|
||||||
|
|
||||||
|
alpha_report_dir = tmp_path / "alpha_reports"
|
||||||
|
alpha_eval_result = _invoke_ok(runner, [
|
||||||
|
"alpha", "eval",
|
||||||
|
"--alpha-path", str(alpha_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--report-dir", str(alpha_report_dir),
|
||||||
|
])
|
||||||
|
alpha_report_path = alpha_report_dir / "cli_feature_alpha_eval.json"
|
||||||
|
assert "ALPHA EVALUATION" in alpha_eval_result.output
|
||||||
|
assert "Report saved:" in alpha_eval_result.output
|
||||||
|
assert alpha_report_path.exists()
|
||||||
|
|
||||||
|
combo_dir = tmp_path / "combos"
|
||||||
|
combo_result = _invoke_ok(runner, [
|
||||||
|
"combo", "combine",
|
||||||
|
"--alpha-paths", f"{alpha_path},{alpha_path}",
|
||||||
|
"--combo-name", "cli_combo",
|
||||||
|
"--method", "equal_weight",
|
||||||
|
"--output-dir", str(combo_dir),
|
||||||
|
])
|
||||||
|
combo_path = combo_dir / "cli_combo.pq"
|
||||||
|
assert "Saved combo:" in combo_result.output
|
||||||
|
assert "Weight stats" in combo_result.output
|
||||||
|
assert combo_path.exists()
|
||||||
|
|
||||||
|
portfolio_dir = tmp_path / "portfolio"
|
||||||
|
build_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "build",
|
||||||
|
"--weights-path", str(combo_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--booksize", "2000000",
|
||||||
|
"--portfolio-name", "cli_portfolio",
|
||||||
|
"--output-dir", str(portfolio_dir),
|
||||||
|
])
|
||||||
|
positions_path = portfolio_dir / "cli_portfolio.pq"
|
||||||
|
assert "Saved positions:" in build_result.output
|
||||||
|
assert "Gross exposure" in build_result.output
|
||||||
|
assert positions_path.exists()
|
||||||
|
|
||||||
|
execution_dir = tmp_path / "execution"
|
||||||
|
simulate_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "simulate",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--constraint", "suspension",
|
||||||
|
"--constraint", "price_limit",
|
||||||
|
"--constraint", "volume_cap",
|
||||||
|
"--cost-bps", "5",
|
||||||
|
"--slippage-bps", "5",
|
||||||
|
"--volume-frac", "0.02",
|
||||||
|
"--output-dir", str(execution_dir),
|
||||||
|
])
|
||||||
|
fills_path = execution_dir / "fills" / "cli_portfolio.pq"
|
||||||
|
pnl_path = execution_dir / "pnl" / "cli_portfolio.pq"
|
||||||
|
assert "Saved fills:" in simulate_result.output
|
||||||
|
assert "Saved pnl:" in simulate_result.output
|
||||||
|
assert "Total PnL:" in simulate_result.output
|
||||||
|
assert fills_path.exists()
|
||||||
|
assert pnl_path.exists()
|
||||||
|
|
||||||
|
eval_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "eval",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
])
|
||||||
|
assert "Research-portfolio metrics:" in eval_result.output
|
||||||
|
assert "cumulative_return" in eval_result.output
|
||||||
|
assert "fitness" in eval_result.output
|
||||||
|
|
||||||
|
pqcat_result = _invoke_ok(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(positions_path),
|
||||||
|
"--info",
|
||||||
|
])
|
||||||
|
assert "shape:" in pqcat_result.output
|
||||||
|
assert "dtypes:" in pqcat_result.output
|
||||||
|
assert "position_shares" in pqcat_result.output
|
||||||
|
|
||||||
|
alphaview_result = _invoke_ok(runner, [
|
||||||
|
"alphaview",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-path", str(alpha_path),
|
||||||
|
"--symbol", "sh600000",
|
||||||
|
"--start-date", "2024-01-02",
|
||||||
|
"--end-date", "2024-01-12",
|
||||||
|
"--columns", "close,volume",
|
||||||
|
])
|
||||||
|
assert "symbol: sh600000" in alphaview_result.output
|
||||||
|
assert "cli_feature_alpha" in alphaview_result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_pipeline_accepts_partitioned_daily_dataset(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(include_missing=False)
|
||||||
|
dataset_dir = tmp_path / "daily_dataset"
|
||||||
|
dataset_frame = daily_bars.copy()
|
||||||
|
dataset_frame["month"] = dataset_frame["date"].dt.strftime("%Y-%m")
|
||||||
|
dataset_frame.to_parquet(dataset_dir, partition_cols=["month"], index=False)
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
alpha_result = _invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(dataset_dir),
|
||||||
|
"--alpha-type", "reversal",
|
||||||
|
"--alpha-name", "dataset_reversal",
|
||||||
|
"--lookback", "3",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
alpha_path = alpha_dir / "dataset_reversal.pq"
|
||||||
|
assert "Loaded data:" in alpha_result.output
|
||||||
|
assert alpha_path.exists()
|
||||||
|
|
||||||
|
combo_dir = tmp_path / "combos"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"combo", "combine",
|
||||||
|
"--alpha-paths", str(alpha_path),
|
||||||
|
"--combo-name", "dataset_combo",
|
||||||
|
"--output-dir", str(combo_dir),
|
||||||
|
])
|
||||||
|
combo_path = combo_dir / "dataset_combo.pq"
|
||||||
|
assert combo_path.exists()
|
||||||
|
|
||||||
|
portfolio_dir = tmp_path / "portfolio"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"portfolio", "build",
|
||||||
|
"--weights-path", str(combo_path),
|
||||||
|
"--data-path", str(dataset_dir),
|
||||||
|
"--booksize", "1000000",
|
||||||
|
"--portfolio-name", "dataset_portfolio",
|
||||||
|
"--output-dir", str(portfolio_dir),
|
||||||
|
])
|
||||||
|
positions_path = portfolio_dir / "dataset_portfolio.pq"
|
||||||
|
assert positions_path.exists()
|
||||||
|
|
||||||
|
execution_dir = tmp_path / "execution"
|
||||||
|
simulate_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "simulate",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(dataset_dir),
|
||||||
|
"--constraint", "suspension",
|
||||||
|
"--output-dir", str(execution_dir),
|
||||||
|
])
|
||||||
|
assert "Saved fills:" in simulate_result.output
|
||||||
|
assert (execution_dir / "fills" / "dataset_portfolio.pq").exists()
|
||||||
|
assert (execution_dir / "pnl" / "dataset_portfolio.pq").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_liquid_universe_masks_to_top_liquid_names(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=75, include_missing=False)
|
||||||
|
daily_path = tmp_path / "daily_bars_75d.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
result = _invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-type", "reversal_rank",
|
||||||
|
"--alpha-name", "liquid_rank",
|
||||||
|
"--lookback", "3",
|
||||||
|
"--liquid-universe",
|
||||||
|
"--universe-top-n", "2",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
|
||||||
|
alpha_path = alpha_dir / "liquid_rank.pq"
|
||||||
|
alpha = pd.read_parquet(alpha_path)
|
||||||
|
nonzero = alpha[alpha["weight"] != 0.0]
|
||||||
|
assert "Saved alpha:" in result.output
|
||||||
|
assert alpha_path.exists()
|
||||||
|
assert not nonzero.empty
|
||||||
|
assert nonzero.groupby("date")["symbol_id"].nunique().max() <= 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_real_fixture_round_trips_through_portfolio(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
|
||||||
|
alpha_dir = tmp_path / "alphas"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(FIXTURE_PATH),
|
||||||
|
"--alpha-type", "reversal_vol",
|
||||||
|
"--alpha-name", "real_cli_reversal_vol",
|
||||||
|
"--lookback", "3",
|
||||||
|
"--vol-window", "3",
|
||||||
|
"--output-dir", str(alpha_dir),
|
||||||
|
])
|
||||||
|
alpha_path = alpha_dir / "real_cli_reversal_vol.pq"
|
||||||
|
assert alpha_path.exists()
|
||||||
|
assert not pd.read_parquet(alpha_path).empty
|
||||||
|
|
||||||
|
combo_dir = tmp_path / "combos"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"combo", "combine",
|
||||||
|
"--alpha-paths", str(alpha_path),
|
||||||
|
"--combo-name", "real_cli_combo",
|
||||||
|
"--output-dir", str(combo_dir),
|
||||||
|
])
|
||||||
|
combo_path = combo_dir / "real_cli_combo.pq"
|
||||||
|
assert combo_path.exists()
|
||||||
|
|
||||||
|
portfolio_dir = tmp_path / "portfolio"
|
||||||
|
_invoke_ok(runner, [
|
||||||
|
"portfolio", "build",
|
||||||
|
"--weights-path", str(combo_path),
|
||||||
|
"--data-path", str(FIXTURE_PATH),
|
||||||
|
"--booksize", "1000000",
|
||||||
|
"--portfolio-name", "real_cli_portfolio",
|
||||||
|
"--output-dir", str(portfolio_dir),
|
||||||
|
])
|
||||||
|
positions_path = portfolio_dir / "real_cli_portfolio.pq"
|
||||||
|
positions = pd.read_parquet(positions_path)
|
||||||
|
assert not positions.empty
|
||||||
|
|
||||||
|
eval_result = _invoke_ok(runner, [
|
||||||
|
"portfolio", "eval",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(FIXTURE_PATH),
|
||||||
|
])
|
||||||
|
assert "Research-portfolio metrics:" in eval_result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_error_paths_are_clear_for_bad_user_inputs(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars()
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
|
||||||
|
unknown_alpha = _invoke_error(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-type", "does_not_exist",
|
||||||
|
"--alpha-name", "bad",
|
||||||
|
"--output-dir", str(tmp_path / "alphas"),
|
||||||
|
])
|
||||||
|
assert "Unknown alpha-type" in unknown_alpha.output
|
||||||
|
|
||||||
|
malformed_param = _invoke_error(runner, [
|
||||||
|
"alpha", "compute",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-type", "reversal",
|
||||||
|
"--alpha-name", "bad_param",
|
||||||
|
"--param", "not-an-assignment",
|
||||||
|
"--output-dir", str(tmp_path / "alphas"),
|
||||||
|
])
|
||||||
|
assert "--param must be name=value" in malformed_param.output
|
||||||
|
|
||||||
|
unknown_derived = _invoke_error(runner, [
|
||||||
|
"derived", "compute",
|
||||||
|
"--daily-path", str(daily_path),
|
||||||
|
"--derived-type", "does_not_exist",
|
||||||
|
"--derived-name", "bad",
|
||||||
|
"--output-dir", str(tmp_path / "derived"),
|
||||||
|
])
|
||||||
|
assert "Unknown derived-type" in unknown_derived.output
|
||||||
|
|
||||||
|
bad_constraint_positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"portfolio_name": ["bad_constraint"],
|
||||||
|
"target_weight": [1.0],
|
||||||
|
"target_value": [1000.0],
|
||||||
|
"target_shares": [100.0],
|
||||||
|
"position_shares": [100],
|
||||||
|
"position_value": [1000.0],
|
||||||
|
"price": [10.0],
|
||||||
|
})
|
||||||
|
positions_path = tmp_path / "positions.pq"
|
||||||
|
bad_constraint_positions.to_parquet(positions_path, index=False)
|
||||||
|
unknown_constraint = _invoke_error(runner, [
|
||||||
|
"portfolio", "simulate",
|
||||||
|
"--positions-path", str(positions_path),
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--constraint", "not_a_constraint",
|
||||||
|
"--output-dir", str(tmp_path / "execution"),
|
||||||
|
])
|
||||||
|
assert isinstance(unknown_constraint.exception, KeyError)
|
||||||
|
assert "not_a_constraint" in str(unknown_constraint.exception)
|
||||||
|
|
||||||
|
pqcat_missing_column = _invoke_error(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(daily_path),
|
||||||
|
"--columns", "close,not_a_column",
|
||||||
|
])
|
||||||
|
assert "Columns not found: not_a_column" in pqcat_missing_column.output
|
||||||
|
|
||||||
|
alphaview_missing_symbol = _invoke_error(runner, [
|
||||||
|
"alphaview",
|
||||||
|
"--data-path", str(daily_path),
|
||||||
|
"--alpha-path", str(positions_path),
|
||||||
|
"--symbol", "sh999999",
|
||||||
|
])
|
||||||
|
assert "Symbol 'sh999999' not found" in alphaview_missing_symbol.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_list_and_legacy_feature_paths(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
|
||||||
|
minute_bars = make_generated_minute_bars(daily_bars)
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
minute_path = tmp_path / "minute_bars.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
minute_bars.to_parquet(minute_path, index=False)
|
||||||
|
|
||||||
|
derived_list = _invoke_ok(runner, ["derived", "list"])
|
||||||
|
feature_list = _invoke_ok(runner, ["feature", "list"])
|
||||||
|
assert "minute_daily_summary" in derived_list.output
|
||||||
|
assert "minute_daily_summary" in feature_list.output
|
||||||
|
|
||||||
|
feature_dir = tmp_path / "features"
|
||||||
|
feature_compute = _invoke_ok(runner, [
|
||||||
|
"feature",
|
||||||
|
"compute",
|
||||||
|
"--minute-path",
|
||||||
|
str(minute_path),
|
||||||
|
"--daily-path",
|
||||||
|
str(daily_path),
|
||||||
|
"--feature-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--feature-name",
|
||||||
|
"legacy_summary",
|
||||||
|
"--output-dir",
|
||||||
|
str(feature_dir),
|
||||||
|
])
|
||||||
|
feature_path = feature_dir / "legacy_summary.pq"
|
||||||
|
assert "Loaded minute bars:" in feature_compute.output
|
||||||
|
assert "Loaded daily data:" in feature_compute.output
|
||||||
|
assert "Saved feature:" in feature_compute.output
|
||||||
|
assert feature_path.exists()
|
||||||
|
|
||||||
|
no_input = _invoke_error(runner, [
|
||||||
|
"derived",
|
||||||
|
"compute",
|
||||||
|
"--derived-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--derived-name",
|
||||||
|
"missing_inputs",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "derived"),
|
||||||
|
])
|
||||||
|
assert "At least one of --daily-path or --minute-path is required" in no_input.output
|
||||||
|
|
||||||
|
missing_minute = _invoke_error(runner, [
|
||||||
|
"derived",
|
||||||
|
"compute",
|
||||||
|
"--daily-path",
|
||||||
|
str(daily_path),
|
||||||
|
"--derived-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--derived-name",
|
||||||
|
"daily_only",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "derived"),
|
||||||
|
])
|
||||||
|
assert "minute_daily_summary requires minute input" in missing_minute.output
|
||||||
|
|
||||||
|
malformed_feature_param = _invoke_error(runner, [
|
||||||
|
"feature",
|
||||||
|
"compute",
|
||||||
|
"--minute-path",
|
||||||
|
str(minute_path),
|
||||||
|
"--feature-type",
|
||||||
|
"minute_daily_summary",
|
||||||
|
"--feature-name",
|
||||||
|
"bad_param",
|
||||||
|
"--param",
|
||||||
|
"not-an-assignment",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "features_bad"),
|
||||||
|
])
|
||||||
|
assert "--param must be name=value" in malformed_feature_param.output
|
||||||
|
|
||||||
|
unknown_feature = _invoke_error(runner, [
|
||||||
|
"feature",
|
||||||
|
"compute",
|
||||||
|
"--minute-path",
|
||||||
|
str(minute_path),
|
||||||
|
"--feature-type",
|
||||||
|
"does_not_exist",
|
||||||
|
"--feature-name",
|
||||||
|
"bad_feature",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "features_unknown"),
|
||||||
|
])
|
||||||
|
assert "Unknown feature-type" in unknown_feature.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_pqcat_row_modes(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=3, include_missing=False)
|
||||||
|
daily_path = tmp_path / "daily_bars.pq"
|
||||||
|
daily_bars.to_parquet(daily_path, index=False)
|
||||||
|
|
||||||
|
head_result = _invoke_ok(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(daily_path),
|
||||||
|
"--head",
|
||||||
|
"2",
|
||||||
|
"--columns",
|
||||||
|
"symbol_id,close",
|
||||||
|
])
|
||||||
|
tail_result = _invoke_ok(runner, [
|
||||||
|
"pqcat",
|
||||||
|
str(daily_path),
|
||||||
|
"--tail",
|
||||||
|
"1",
|
||||||
|
])
|
||||||
|
|
||||||
|
assert "symbol_id" in head_result.output
|
||||||
|
assert "close" in head_result.output
|
||||||
|
assert "date" in tail_result.output
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
"""Offline coverage for data CLI and universe resolution glue."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from cli import cli
|
||||||
|
import pipeline.data.cli as data_cli
|
||||||
|
import pipeline.data.downloader as pipeline_downloader
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_universe_handles_named_file_all_and_symbol_list(tmp_path, monkeypatch):
|
||||||
|
hs300_raw = pd.DataFrame({
|
||||||
|
"updateDate": ["2024-01-12", "2024-01-12"],
|
||||||
|
"stockName": ["浦发银行", "平安银行"],
|
||||||
|
"stockCode": ["sh.600000", "sz.000001"],
|
||||||
|
})
|
||||||
|
zz500_raw = pd.DataFrame({
|
||||||
|
"name": ["东风汽车"],
|
||||||
|
"code": ["sh.600006"],
|
||||||
|
"date": ["2024-01-12"],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "get_hs300_stocks", lambda: hs300_raw)
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "get_zz500_stocks", lambda: zz500_raw)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"get_all_stocks",
|
||||||
|
lambda: pd.DataFrame({
|
||||||
|
"code": ["sh600000", "sz000001", "sh600519"],
|
||||||
|
"name": ["浦发银行", "平安银行", "贵州茅台"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
symbol_file = tmp_path / "symbols.txt"
|
||||||
|
symbol_file.write_text("sh600000\n\nsz000001\n")
|
||||||
|
|
||||||
|
hs300 = pipeline_downloader._resolve_universe("hs300")
|
||||||
|
zz500 = pipeline_downloader._resolve_universe("csi500")
|
||||||
|
all_capped = pipeline_downloader._resolve_universe("all", max_symbols=2)
|
||||||
|
from_file = pipeline_downloader._resolve_universe(str(symbol_file))
|
||||||
|
from_list = pipeline_downloader._resolve_universe("sh600000, sz000001")
|
||||||
|
|
||||||
|
assert hs300.to_dict("list") == {
|
||||||
|
"symbol_name": ["浦发银行", "平安银行"],
|
||||||
|
"symbol_id": ["sh600000", "sz000001"],
|
||||||
|
}
|
||||||
|
assert zz500.to_dict("list") == {
|
||||||
|
"symbol_name": ["东风汽车"],
|
||||||
|
"symbol_id": ["sh600006"],
|
||||||
|
}
|
||||||
|
assert all_capped["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert from_file["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert from_file["symbol_name"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert from_list["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_data_cli_download_commands_print_summaries_without_network(monkeypatch, tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
daily_calls: list[dict] = []
|
||||||
|
minute_calls: list[dict] = []
|
||||||
|
|
||||||
|
def fake_daily(**kwargs):
|
||||||
|
daily_calls.append(kwargs)
|
||||||
|
return {
|
||||||
|
"dataset_path": str(tmp_path / "daily" / kwargs["universe"]),
|
||||||
|
"n_symbols": 2,
|
||||||
|
"n_requested": 3,
|
||||||
|
"n_rows": 18,
|
||||||
|
"date_min": "2024-01-02",
|
||||||
|
"date_max": "2024-01-12",
|
||||||
|
}
|
||||||
|
|
||||||
|
def fake_minute(**kwargs):
|
||||||
|
minute_calls.append(kwargs)
|
||||||
|
return {
|
||||||
|
"dataset_path": str(tmp_path / "minute" / kwargs["universe"]),
|
||||||
|
"frequency": "15m",
|
||||||
|
"n_symbols": 1,
|
||||||
|
"n_requested": 1,
|
||||||
|
"n_rows": 32,
|
||||||
|
"date_min": "2024-01-02",
|
||||||
|
"date_max": "2024-01-03",
|
||||||
|
}
|
||||||
|
|
||||||
|
monkeypatch.setattr(data_cli, "download_universe", fake_daily)
|
||||||
|
monkeypatch.setattr(data_cli, "download_minute_universe", fake_minute)
|
||||||
|
|
||||||
|
daily_result = runner.invoke(cli, [
|
||||||
|
"data",
|
||||||
|
"download",
|
||||||
|
"--universe",
|
||||||
|
"sh600000,sz000001",
|
||||||
|
"--start-date",
|
||||||
|
"2024-01-02",
|
||||||
|
"--end-date",
|
||||||
|
"2024-01-12",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "daily"),
|
||||||
|
"--symbols",
|
||||||
|
"3",
|
||||||
|
"--chunk-size",
|
||||||
|
"2",
|
||||||
|
"--adjust",
|
||||||
|
"none",
|
||||||
|
])
|
||||||
|
minute_result = runner.invoke(cli, [
|
||||||
|
"data",
|
||||||
|
"download-minute",
|
||||||
|
"--universe",
|
||||||
|
"toy",
|
||||||
|
"--start-date",
|
||||||
|
"2024-01-02",
|
||||||
|
"--end-date",
|
||||||
|
"2024-01-03",
|
||||||
|
"--output-dir",
|
||||||
|
str(tmp_path / "minute"),
|
||||||
|
"--symbols",
|
||||||
|
"1",
|
||||||
|
"--chunk-size",
|
||||||
|
"1",
|
||||||
|
"--frequency",
|
||||||
|
"15",
|
||||||
|
])
|
||||||
|
|
||||||
|
assert daily_result.exit_code == 0, daily_result.output
|
||||||
|
assert "Summary: 2/3 symbols, 18 bars" in daily_result.output
|
||||||
|
assert daily_calls == [{
|
||||||
|
"universe": "sh600000,sz000001",
|
||||||
|
"start_date": "2024-01-02",
|
||||||
|
"end_date": "2024-01-12",
|
||||||
|
"output_dir": str(tmp_path / "daily"),
|
||||||
|
"max_symbols": 3,
|
||||||
|
"chunk_size": 2,
|
||||||
|
"adjust": "none",
|
||||||
|
}]
|
||||||
|
assert minute_result.exit_code == 0, minute_result.output
|
||||||
|
assert "frequency=15m" in minute_result.output
|
||||||
|
assert minute_calls == [{
|
||||||
|
"universe": "toy",
|
||||||
|
"start_date": "2024-01-02",
|
||||||
|
"end_date": "2024-01-03",
|
||||||
|
"output_dir": str(tmp_path / "minute"),
|
||||||
|
"max_symbols": 1,
|
||||||
|
"chunk_size": 1,
|
||||||
|
"frequency": "15",
|
||||||
|
}]
|
||||||
@@ -0,0 +1,261 @@
|
|||||||
|
"""Tests for daily derived-data ingestion and plugins."""
|
||||||
|
|
||||||
|
import textwrap
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from cli import cli
|
||||||
|
from pipeline.alpha.compute import join_feature_frames
|
||||||
|
from pipeline.derived.compute import compute_derived, validate_derived_frame
|
||||||
|
from pipeline.derived.registry import available_derived, get_derived, load_derived_module
|
||||||
|
|
||||||
|
|
||||||
|
def _daily_bars() -> pd.DataFrame:
|
||||||
|
return pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
|
||||||
|
"open": [10.0, 20.0, 11.0],
|
||||||
|
"close": [10.5, 20.5, 11.5],
|
||||||
|
"volume": [1000.0, 2000.0, 1200.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def _minute_bars() -> pd.DataFrame:
|
||||||
|
return pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001"],
|
||||||
|
"datetime": pd.to_datetime([
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
"2024-01-02 09:40:00",
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
]),
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
|
||||||
|
"time": ["09:35:00", "09:40:00", "09:35:00"],
|
||||||
|
"open": [10.0, 10.5, 20.0],
|
||||||
|
"high": [11.0, 12.0, 21.0],
|
||||||
|
"low": [9.0, 10.0, 19.0],
|
||||||
|
"close": [10.5, 11.0, 20.5],
|
||||||
|
"volume": [100.0, 300.0, 200.0],
|
||||||
|
"amount": [1000.0, 3300.0, 4100.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_normalizes_and_sorts():
|
||||||
|
result = validate_derived_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sz000001", "sh600000"],
|
||||||
|
"date": ["2024-01-02 15:00:00", "2024-01-02 09:30:00"],
|
||||||
|
"custom_value": [2.0, 1.0],
|
||||||
|
}))
|
||||||
|
|
||||||
|
assert result["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert result["date"].tolist() == [
|
||||||
|
pd.Timestamp("2024-01-02"),
|
||||||
|
pd.Timestamp("2024-01-02"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_missing_keys():
|
||||||
|
with pytest.raises(ValueError, match="missing required"):
|
||||||
|
validate_derived_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_duplicate_normalized_keys():
|
||||||
|
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
|
||||||
|
validate_derived_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
|
||||||
|
"x": [1.0, 2.0],
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_duplicate_columns():
|
||||||
|
bad = pd.DataFrame(
|
||||||
|
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
|
||||||
|
columns=["symbol_id", "date", "dup", "dup"],
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError, match="duplicate columns"):
|
||||||
|
validate_derived_frame(bad)
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_derived_frame_rejects_non_numeric_values():
|
||||||
|
with pytest.raises(ValueError, match="numeric"):
|
||||||
|
validate_derived_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"bad": ["not numeric"],
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_ingest_cli_accepts_csv_and_parquet(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
source = pd.DataFrame({
|
||||||
|
"symbol_id": ["sz000001", "sh600000"],
|
||||||
|
"date": ["2024-01-02", "2024-01-02"],
|
||||||
|
"custom_value": [2.0, 1.0],
|
||||||
|
})
|
||||||
|
csv_path = tmp_path / "custom.csv"
|
||||||
|
parquet_path = tmp_path / "custom.pq"
|
||||||
|
out_dir = tmp_path / "derived"
|
||||||
|
source.to_csv(csv_path, index=False)
|
||||||
|
source.to_parquet(parquet_path, index=False)
|
||||||
|
|
||||||
|
csv_result = runner.invoke(cli, [
|
||||||
|
"derived", "ingest",
|
||||||
|
"--input-path", str(csv_path),
|
||||||
|
"--derived-name", "csv_custom",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
assert csv_result.exit_code == 0, csv_result.output
|
||||||
|
|
||||||
|
parquet_result = runner.invoke(cli, [
|
||||||
|
"derived", "ingest",
|
||||||
|
"--input-path", str(parquet_path),
|
||||||
|
"--derived-name", "parquet_custom",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
assert parquet_result.exit_code == 0, parquet_result.output
|
||||||
|
|
||||||
|
written = pd.read_parquet(out_dir / "csv_custom.pq")
|
||||||
|
assert written["symbol_id"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert (out_dir / "parquet_custom.pq").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_validate_cli_rejects_duplicate_csv_columns(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
csv_path = tmp_path / "bad.csv"
|
||||||
|
csv_path.write_text("symbol_id,date,x,x\nsh600000,2024-01-02,1.0,2.0\n")
|
||||||
|
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"derived", "validate",
|
||||||
|
"--input-path", str(csv_path),
|
||||||
|
])
|
||||||
|
|
||||||
|
assert result.exit_code != 0
|
||||||
|
assert "duplicate columns" in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_external_derived_plugin_loads_filters_params_and_uses_inputs(tmp_path):
|
||||||
|
module_path = tmp_path / "external_derived.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.derived.base import BaseDerivedData
|
||||||
|
from pipeline.derived.registry import register_derived
|
||||||
|
|
||||||
|
@register_derived
|
||||||
|
class FlexibleDerived(BaseDerivedData):
|
||||||
|
name = "flexible_derived_test"
|
||||||
|
|
||||||
|
def __init__(self, scale: float = 1.0):
|
||||||
|
self.scale = scale
|
||||||
|
|
||||||
|
def compute(self, daily=None, minute=None) -> pd.DataFrame:
|
||||||
|
result = None
|
||||||
|
if daily is not None:
|
||||||
|
result = daily[["symbol_id", "date", "close"]].copy()
|
||||||
|
result["daily_scaled_close"] = result.pop("close") * self.scale
|
||||||
|
if minute is not None:
|
||||||
|
minute_out = (
|
||||||
|
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
|
||||||
|
.sum()
|
||||||
|
.rename(columns={"volume": "minute_volume_sum"})
|
||||||
|
)
|
||||||
|
minute_out["minute_volume_sum"] *= self.scale
|
||||||
|
result = minute_out if result is None else result.merge(
|
||||||
|
minute_out, on=["symbol_id", "date"], how="left"
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
'''))
|
||||||
|
|
||||||
|
load_derived_module(str(module_path))
|
||||||
|
assert "flexible_derived_test" in available_derived()
|
||||||
|
|
||||||
|
instance = get_derived("flexible_derived_test", scale=2.0, ignored=99)
|
||||||
|
assert instance.scale == 2.0
|
||||||
|
assert not hasattr(instance, "ignored")
|
||||||
|
|
||||||
|
daily_result = compute_derived(
|
||||||
|
"flexible_derived_test",
|
||||||
|
daily=_daily_bars(),
|
||||||
|
scale=2.0,
|
||||||
|
ignored=99,
|
||||||
|
)
|
||||||
|
assert "daily_scaled_close" in daily_result.columns
|
||||||
|
assert np.isclose(daily_result["daily_scaled_close"].iloc[0], 21.0)
|
||||||
|
|
||||||
|
minute_result = compute_derived(
|
||||||
|
"flexible_derived_test",
|
||||||
|
minute=_minute_bars(),
|
||||||
|
scale=2.0,
|
||||||
|
)
|
||||||
|
assert "minute_volume_sum" in minute_result.columns
|
||||||
|
assert np.isclose(
|
||||||
|
minute_result.loc[minute_result["symbol_id"] == "sh600000", "minute_volume_sum"].iloc[0],
|
||||||
|
800.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
both_result = compute_derived(
|
||||||
|
"flexible_derived_test",
|
||||||
|
daily=_daily_bars(),
|
||||||
|
minute=_minute_bars(),
|
||||||
|
scale=1.0,
|
||||||
|
)
|
||||||
|
assert {"daily_scaled_close", "minute_volume_sum"}.issubset(both_result.columns)
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_compute_cli_writes_builtin_minute_summary(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
minute_path = tmp_path / "minute.pq"
|
||||||
|
out_dir = tmp_path / "derived"
|
||||||
|
_minute_bars().to_parquet(minute_path, index=False)
|
||||||
|
|
||||||
|
result = runner.invoke(cli, [
|
||||||
|
"derived", "compute",
|
||||||
|
"--minute-path", str(minute_path),
|
||||||
|
"--derived-type", "minute_daily_summary",
|
||||||
|
"--derived-name", "minute_summary",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
written = pd.read_parquet(out_dir / "minute_summary.pq")
|
||||||
|
assert "minute_vwap" in written.columns
|
||||||
|
|
||||||
|
|
||||||
|
def test_alpha_feature_join_rejects_derived_column_collisions():
|
||||||
|
data = _daily_bars()
|
||||||
|
derived_a = data[["symbol_id", "date"]].copy()
|
||||||
|
derived_a["custom_value"] = 1.0
|
||||||
|
derived_b = data[["symbol_id", "date"]].copy()
|
||||||
|
derived_b["custom_value"] = 2.0
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="conflict"):
|
||||||
|
join_feature_frames(data, [derived_a, derived_b])
|
||||||
|
|
||||||
|
close_collision = data[["symbol_id", "date"]].copy()
|
||||||
|
close_collision["close"] = 1.0
|
||||||
|
with pytest.raises(ValueError, match="conflict"):
|
||||||
|
join_feature_frames(data, [close_collision])
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_feature_cli_delegates_to_derived_registry(tmp_path):
|
||||||
|
runner = CliRunner()
|
||||||
|
minute_path = tmp_path / "minute.pq"
|
||||||
|
out_dir = tmp_path / "features"
|
||||||
|
_minute_bars().to_parquet(minute_path, index=False)
|
||||||
|
|
||||||
|
list_result = runner.invoke(cli, ["feature", "list"])
|
||||||
|
assert list_result.exit_code == 0, list_result.output
|
||||||
|
assert "minute_daily_summary" in list_result.output
|
||||||
|
|
||||||
|
compute_result = runner.invoke(cli, [
|
||||||
|
"feature", "compute",
|
||||||
|
"--minute-path", str(minute_path),
|
||||||
|
"--feature-type", "minute_daily_summary",
|
||||||
|
"--feature-name", "minute_summary",
|
||||||
|
"--output-dir", str(out_dir),
|
||||||
|
])
|
||||||
|
assert compute_result.exit_code == 0, compute_result.output
|
||||||
|
assert (out_dir / "minute_summary.pq").exists()
|
||||||
|
|
||||||
@@ -1,6 +1,16 @@
|
|||||||
|
import os
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from data.downloader import download_daily
|
from data.downloader import download_daily
|
||||||
|
|
||||||
|
pytestmark = [
|
||||||
|
pytest.mark.network,
|
||||||
|
pytest.mark.skipif(
|
||||||
|
os.environ.get("CEQ_RUN_LIVE_DOWNLOADER") != "1",
|
||||||
|
reason="set CEQ_RUN_LIVE_DOWNLOADER=1 to run live baostock/akshare smoke tests",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_download_single_stock():
|
def test_download_single_stock():
|
||||||
"""Smoke test: download data for 浦发银行 for a short window."""
|
"""Smoke test: download data for 浦发银行 for a short window."""
|
||||||
|
|||||||
@@ -0,0 +1,462 @@
|
|||||||
|
"""Offline downloader contract tests with mocked data providers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import data.downloader as downloader
|
||||||
|
import pipeline.data.downloader as pipeline_downloader
|
||||||
|
from data.downloader import download_daily, download_daily_batch
|
||||||
|
from pipeline.common.schema import DATA_COLUMNS
|
||||||
|
from pipeline.data.downloader import download_universe
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResult:
|
||||||
|
def __init__(self, rows, error_code="0", error_msg=""):
|
||||||
|
self.rows = rows
|
||||||
|
self.error_code = error_code
|
||||||
|
self.error_msg = error_msg
|
||||||
|
self._idx = -1
|
||||||
|
|
||||||
|
def next(self):
|
||||||
|
self._idx += 1
|
||||||
|
return self._idx < len(self.rows)
|
||||||
|
|
||||||
|
def get_row_data(self):
|
||||||
|
return self.rows[self._idx]
|
||||||
|
|
||||||
|
|
||||||
|
def _daily_batch_row(
|
||||||
|
date: str = "2024-01-02",
|
||||||
|
open_: str = "10",
|
||||||
|
high: str = "11",
|
||||||
|
low: str = "9",
|
||||||
|
close: str = "10.5",
|
||||||
|
preclose: str = "10",
|
||||||
|
volume: str = "1000",
|
||||||
|
amount: str = "10500",
|
||||||
|
) -> list[str]:
|
||||||
|
return [
|
||||||
|
date,
|
||||||
|
open_,
|
||||||
|
high,
|
||||||
|
low,
|
||||||
|
close,
|
||||||
|
preclose,
|
||||||
|
volume,
|
||||||
|
amount,
|
||||||
|
"1.23",
|
||||||
|
"5.0",
|
||||||
|
"1",
|
||||||
|
"0",
|
||||||
|
"8.0",
|
||||||
|
"1.1",
|
||||||
|
"2.2",
|
||||||
|
"3.3",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_uses_baostock_before_akshare_in_auto(monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
expected = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000"],
|
||||||
|
"date": ["2024-01-02"],
|
||||||
|
"open": [10.0],
|
||||||
|
"high": [11.0],
|
||||||
|
"low": [9.0],
|
||||||
|
"close": [10.5],
|
||||||
|
"volume": [1000.0],
|
||||||
|
"amount": [10500.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
def fake_baostock(symbol, start, end, adjust):
|
||||||
|
calls.append("baostock")
|
||||||
|
return expected
|
||||||
|
|
||||||
|
def fake_akshare(symbol, start, end, adjust):
|
||||||
|
calls.append("akshare")
|
||||||
|
raise AssertionError("akshare should not be called after baostock succeeds")
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader, "_download_baostock", fake_baostock)
|
||||||
|
monkeypatch.setattr(downloader, "_download_akshare", fake_akshare)
|
||||||
|
|
||||||
|
result = download_daily("sh600000", "2024-01-02", "2024-01-02", source="auto")
|
||||||
|
|
||||||
|
assert calls == ["baostock"]
|
||||||
|
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
assert result["close"].tolist() == [10.5]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_falls_back_to_akshare_when_baostock_empty(monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
fallback = pd.DataFrame({
|
||||||
|
"symbol": ["sz000001"],
|
||||||
|
"date": ["2024-01-02"],
|
||||||
|
"open": [20.0],
|
||||||
|
"high": [21.0],
|
||||||
|
"low": [19.0],
|
||||||
|
"close": [20.5],
|
||||||
|
"volume": [2000.0],
|
||||||
|
"amount": [41000.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_baostock",
|
||||||
|
lambda symbol, start, end, adjust: calls.append("baostock") or None,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_akshare",
|
||||||
|
lambda symbol, start, end, adjust: calls.append("akshare") or fallback,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = download_daily("sz000001", "2024-01-02", "2024-01-02", source="auto")
|
||||||
|
|
||||||
|
assert calls == ["baostock", "akshare"]
|
||||||
|
assert result["symbol"].tolist() == ["sz000001"]
|
||||||
|
assert result["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_raises_when_requested_source_has_no_data(monkeypatch):
|
||||||
|
monkeypatch.setattr(downloader, "_download_baostock", lambda *args: None)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="Failed to download data for sh600000"):
|
||||||
|
download_daily(
|
||||||
|
"sh600000",
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
source="baostock",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_akshare_daily_downloader_maps_columns_and_failures(monkeypatch):
|
||||||
|
calls: list[dict] = []
|
||||||
|
raw = pd.DataFrame({
|
||||||
|
"日期": ["2024-01-02"],
|
||||||
|
"开盘": [10.0],
|
||||||
|
"最高": [11.0],
|
||||||
|
"最低": [9.0],
|
||||||
|
"收盘": [10.5],
|
||||||
|
"成交量": [1000.0],
|
||||||
|
"成交额": [10500.0],
|
||||||
|
"换手率": [1.2],
|
||||||
|
})
|
||||||
|
|
||||||
|
def fake_hist(**kwargs):
|
||||||
|
calls.append(kwargs)
|
||||||
|
return raw.copy()
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", fake_hist)
|
||||||
|
|
||||||
|
result = downloader._download_akshare(
|
||||||
|
"sh600000",
|
||||||
|
"20240102",
|
||||||
|
"20240102",
|
||||||
|
adjust="",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert calls == [{
|
||||||
|
"symbol": "600000",
|
||||||
|
"period": "daily",
|
||||||
|
"start_date": "20240102",
|
||||||
|
"end_date": "20240102",
|
||||||
|
"adjust": "",
|
||||||
|
}]
|
||||||
|
assert result is not None
|
||||||
|
assert result.columns.tolist() == [
|
||||||
|
"symbol", "date", "open", "high", "low", "close", "volume", "amount",
|
||||||
|
]
|
||||||
|
assert result["symbol"].tolist() == ["sh600000"]
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.ak, "stock_zh_a_hist", lambda **kwargs: pd.DataFrame())
|
||||||
|
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.ak,
|
||||||
|
"stock_zh_a_hist",
|
||||||
|
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("provider down")),
|
||||||
|
)
|
||||||
|
assert downloader._download_akshare("sh600000", "20240102", "20240102") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_baostock_daily_downloader_maps_errors_and_logout_failures(monkeypatch):
|
||||||
|
query_calls: list[dict] = []
|
||||||
|
row = ["2024-01-02", "10", "11", "9", "10.5", "1000", "10500"]
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"logout",
|
||||||
|
lambda: (_ for _ in ()).throw(RuntimeError("logout failed")),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_query(**kwargs):
|
||||||
|
query_calls.append(kwargs)
|
||||||
|
return _FakeResult([row])
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
|
||||||
|
|
||||||
|
result = downloader._download_baostock(
|
||||||
|
"sz000001",
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
adjust="none",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert query_calls[0]["code"] == "sz.000001"
|
||||||
|
assert query_calls[0]["adjustflag"] == "3"
|
||||||
|
assert result is not None
|
||||||
|
assert result["symbol"].tolist() == ["sz000001"]
|
||||||
|
assert pd.api.types.is_numeric_dtype(result["close"])
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([], error_code="1", error_msg="bad symbol"),
|
||||||
|
)
|
||||||
|
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([]),
|
||||||
|
)
|
||||||
|
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("query failed")),
|
||||||
|
)
|
||||||
|
assert downloader._download_baostock("sz000001", "2024-01-02", "2024-01-02") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_maps_rich_schema_and_vwap(monkeypatch):
|
||||||
|
query_calls: list[dict] = []
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
def fake_query(**kwargs):
|
||||||
|
query_calls.append(kwargs)
|
||||||
|
rows = [
|
||||||
|
_daily_batch_row(volume="1000", amount="10500"),
|
||||||
|
_daily_batch_row(date="2024-01-03", volume="0", amount="0"),
|
||||||
|
]
|
||||||
|
return _FakeResult(rows)
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(downloader.bs, "query_history_k_data_plus", fake_query)
|
||||||
|
|
||||||
|
[(symbol, frame)] = list(
|
||||||
|
download_daily_batch(
|
||||||
|
["sh600000"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-03",
|
||||||
|
adjust="hfq",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert query_calls[0]["code"] == "sh.600000"
|
||||||
|
assert query_calls[0]["adjustflag"] == "1"
|
||||||
|
assert login_count == 1
|
||||||
|
assert logout_count == 1
|
||||||
|
assert frame is not None
|
||||||
|
assert frame.columns.tolist() == [
|
||||||
|
"symbol", "date", "open", "high", "low", "close", "preclose",
|
||||||
|
"volume", "amount", "vwap", "turn", "pctChg", "tradestatus", "isST",
|
||||||
|
"peTTM", "pbMRQ", "psTTM", "pcfNcfTTM",
|
||||||
|
]
|
||||||
|
assert np.isclose(frame["vwap"].iloc[0], 10.5)
|
||||||
|
assert pd.isna(frame["vwap"].iloc[1])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(frame["date"])
|
||||||
|
assert pd.api.types.is_numeric_dtype(frame["tradestatus"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_periodic_relogin_and_none_result(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="1", error_msg="bad symbol"),
|
||||||
|
_FakeResult([_daily_batch_row(date="2024-01-03")]),
|
||||||
|
]
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
results = list(
|
||||||
|
download_daily_batch(
|
||||||
|
["sh600000", "sz000001"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-03",
|
||||||
|
relogin_every=1,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert results[0] == ("sh600000", None)
|
||||||
|
assert results[1][0] == "sz000001"
|
||||||
|
assert results[1][1] is not None
|
||||||
|
assert login_count == 2
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_relogs_and_retries_session_loss(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
_FakeResult([_daily_batch_row()]),
|
||||||
|
]
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
[(symbol, frame)] = list(download_daily_batch(["sh600000"], "2024-01-02", "2024-01-02"))
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert frame is not None
|
||||||
|
assert len(frame) == 1
|
||||||
|
assert login_count == 2
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_daily_batch_uses_akshare_fallback_when_enabled(monkeypatch):
|
||||||
|
fallback = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000"],
|
||||||
|
"date": ["2024-01-02"],
|
||||||
|
"open": [10.0],
|
||||||
|
"high": [11.0],
|
||||||
|
"low": [9.0],
|
||||||
|
"close": [10.5],
|
||||||
|
"volume": [1000.0],
|
||||||
|
"amount": [10500.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([], error_code="1", error_msg="no data"),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
downloader,
|
||||||
|
"_download_akshare",
|
||||||
|
lambda symbol, start, end, adjust: fallback.copy(),
|
||||||
|
)
|
||||||
|
|
||||||
|
[(symbol, frame)] = list(
|
||||||
|
download_daily_batch(
|
||||||
|
["sh600000"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
akshare_fallback=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert frame is not None
|
||||||
|
assert frame["date"].tolist() == [pd.Timestamp("2024-01-02")]
|
||||||
|
assert frame["close"].tolist() == [10.5]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_universe_writes_daily_partitions_from_mock_batch(tmp_path, monkeypatch):
|
||||||
|
batch_frame = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-02-01"]),
|
||||||
|
"open": [10.0, 11.0],
|
||||||
|
"high": [11.0, 12.0],
|
||||||
|
"low": [9.0, 10.0],
|
||||||
|
"close": [10.5, 11.5],
|
||||||
|
"preclose": [10.0, 10.5],
|
||||||
|
"volume": [1000.0, 1200.0],
|
||||||
|
"amount": [10500.0, 13800.0],
|
||||||
|
"vwap": [10.5, 11.5],
|
||||||
|
"turn": [1.0, 1.1],
|
||||||
|
"pctChg": [5.0, 9.5],
|
||||||
|
"tradestatus": [1, 1],
|
||||||
|
"isST": [0, 0],
|
||||||
|
"peTTM": [8.0, 8.1],
|
||||||
|
"pbMRQ": [1.1, 1.2],
|
||||||
|
"psTTM": [2.1, 2.2],
|
||||||
|
"pcfNcfTTM": [3.1, 3.2],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001"],
|
||||||
|
"symbol_name": ["PF Bank", "Ping An Bank"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_batch(symbols, start, end, adjust="qfq"):
|
||||||
|
assert symbols == ["sh600000", "sz000001"]
|
||||||
|
assert adjust == "qfq"
|
||||||
|
yield "sh600000", batch_frame
|
||||||
|
yield "sz000001", None
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "download_daily_batch", fake_batch)
|
||||||
|
|
||||||
|
stats = download_universe(
|
||||||
|
universe="toy",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-02-01",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
chunk_size=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
dataset_path = tmp_path / "toy"
|
||||||
|
written = pd.read_parquet(dataset_path).sort_values(["date", "symbol_id"]).reset_index(drop=True)
|
||||||
|
assert stats == {
|
||||||
|
"dataset_path": str(dataset_path),
|
||||||
|
"n_symbols": 1,
|
||||||
|
"n_requested": 2,
|
||||||
|
"n_rows": 2,
|
||||||
|
"date_min": "2024-01-02",
|
||||||
|
"date_max": "2024-02-01",
|
||||||
|
}
|
||||||
|
assert (dataset_path / "month=2024-01").exists()
|
||||||
|
assert (dataset_path / "month=2024-02").exists()
|
||||||
|
assert written[DATA_COLUMNS].columns.tolist() == DATA_COLUMNS
|
||||||
|
assert written["symbol_name"].tolist() == ["PF Bank", "PF Bank"]
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
"""Tests for minute-derived daily feature plugins."""
|
||||||
|
|
||||||
|
import textwrap
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pipeline.features.compute import compute_feature, validate_feature_frame
|
||||||
|
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
|
||||||
|
from pipeline.features.registry import (
|
||||||
|
available_features,
|
||||||
|
get_feature,
|
||||||
|
load_feature_module,
|
||||||
|
)
|
||||||
|
from pipeline.derived.compute import compute_derived
|
||||||
|
|
||||||
|
|
||||||
|
def _minute_bars() -> pd.DataFrame:
|
||||||
|
return pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001"],
|
||||||
|
"symbol_name": ["PF Bank", "PF Bank", "Ping An"],
|
||||||
|
"datetime": pd.to_datetime([
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
"2024-01-02 09:40:00",
|
||||||
|
"2024-01-02 09:35:00",
|
||||||
|
]),
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
|
||||||
|
"time": ["09:35:00", "09:40:00", "09:35:00"],
|
||||||
|
"frequency": ["5m", "5m", "5m"],
|
||||||
|
"open": [10.0, 10.5, 20.0],
|
||||||
|
"high": [11.0, 12.0, 21.0],
|
||||||
|
"low": [9.0, 10.0, 19.0],
|
||||||
|
"close": [10.5, 11.0, 20.5],
|
||||||
|
"volume": [100.0, 300.0, 200.0],
|
||||||
|
"amount": [1000.0, 3300.0, 4100.0],
|
||||||
|
"vwap": [10.0, 11.0, 20.5],
|
||||||
|
"adjustflag": ["3", "3", "3"],
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def test_built_in_minute_daily_summary():
|
||||||
|
daily = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
|
||||||
|
"close": [11.0, 20.5, 12.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
result = compute_feature(
|
||||||
|
minute=_minute_bars(),
|
||||||
|
daily=daily,
|
||||||
|
feature_type="minute_daily_summary",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert "minute_daily_summary" in available_features()
|
||||||
|
row = result[
|
||||||
|
(result["symbol_id"] == "sh600000")
|
||||||
|
& (result["date"] == pd.Timestamp("2024-01-02"))
|
||||||
|
].iloc[0]
|
||||||
|
assert row["minute_bar_count"] == 2
|
||||||
|
assert np.isclose(row["minute_intraday_return"], 11.0 / 10.0 - 1.0)
|
||||||
|
assert np.isclose(row["minute_intraday_range"], 12.0 / 9.0 - 1.0)
|
||||||
|
assert np.isclose(row["minute_vwap"], 4300.0 / 400.0)
|
||||||
|
assert np.isclose(row["minute_vwap_deviation"], (4300.0 / 400.0) / 11.0 - 1.0)
|
||||||
|
|
||||||
|
missing = result[
|
||||||
|
(result["symbol_id"] == "sh600000")
|
||||||
|
& (result["date"] == pd.Timestamp("2024-01-03"))
|
||||||
|
].iloc[0]
|
||||||
|
assert pd.isna(missing["minute_vwap"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_minute_daily_summary_feature_preserves_legacy_positional_compute():
|
||||||
|
direct = MinuteDailySummaryFeature().compute(_minute_bars())
|
||||||
|
via_registry = get_feature("minute_daily_summary").compute(_minute_bars())
|
||||||
|
|
||||||
|
assert "minute_vwap" in direct.columns
|
||||||
|
pd.testing.assert_frame_equal(direct, via_registry)
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_feature_compute_matches_canonical_derived_compute():
|
||||||
|
daily = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000"],
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
|
||||||
|
"close": [11.0, 20.5, 12.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
legacy_feature = compute_feature(
|
||||||
|
minute=_minute_bars(),
|
||||||
|
daily=daily,
|
||||||
|
feature_type="minute_daily_summary",
|
||||||
|
)
|
||||||
|
canonical_derived = compute_derived(
|
||||||
|
"minute_daily_summary",
|
||||||
|
daily=daily,
|
||||||
|
minute=_minute_bars(),
|
||||||
|
)
|
||||||
|
|
||||||
|
pd.testing.assert_frame_equal(legacy_feature, canonical_derived)
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_external_feature_module_and_filter_params(tmp_path):
|
||||||
|
module_path = tmp_path / "external_feature.py"
|
||||||
|
module_path.write_text(textwrap.dedent('''
|
||||||
|
import pandas as pd
|
||||||
|
from pipeline.features.base import BaseFeature
|
||||||
|
from pipeline.features.registry import register_feature
|
||||||
|
|
||||||
|
@register_feature
|
||||||
|
class ExternalVolumeFeature(BaseFeature):
|
||||||
|
name = "external_volume_feature"
|
||||||
|
|
||||||
|
def __init__(self, scale: float = 1.0):
|
||||||
|
self.scale = scale
|
||||||
|
|
||||||
|
def compute(self, minute: pd.DataFrame, daily=None) -> pd.DataFrame:
|
||||||
|
out = (
|
||||||
|
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
|
||||||
|
.sum()
|
||||||
|
.rename(columns={"volume": "scaled_volume"})
|
||||||
|
)
|
||||||
|
out["scaled_volume"] *= self.scale
|
||||||
|
return out
|
||||||
|
'''))
|
||||||
|
|
||||||
|
load_feature_module(str(module_path))
|
||||||
|
assert "external_volume_feature" in available_features()
|
||||||
|
|
||||||
|
instance = get_feature("external_volume_feature", scale=2.0, ignored=99)
|
||||||
|
assert instance.scale == 2.0
|
||||||
|
assert not hasattr(instance, "ignored")
|
||||||
|
|
||||||
|
result = compute_feature(
|
||||||
|
minute=_minute_bars(),
|
||||||
|
feature_type="external_volume_feature",
|
||||||
|
scale=2.0,
|
||||||
|
ignored=99,
|
||||||
|
)
|
||||||
|
row = result[result["symbol_id"] == "sh600000"].iloc[0]
|
||||||
|
assert np.isclose(row["scaled_volume"], 800.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_missing_keys():
|
||||||
|
with pytest.raises(ValueError, match="missing required"):
|
||||||
|
validate_feature_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_duplicate_keys_after_date_normalization():
|
||||||
|
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
|
||||||
|
validate_feature_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": ["2024-01-02", pd.Timestamp("2024-01-02")],
|
||||||
|
"x": [1.0, 2.0],
|
||||||
|
}))
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_duplicate_columns():
|
||||||
|
bad = pd.DataFrame(
|
||||||
|
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
|
||||||
|
columns=["symbol_id", "date", "dup", "dup"],
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError, match="duplicate columns"):
|
||||||
|
validate_feature_frame(bad)
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_feature_frame_rejects_non_numeric_feature_columns():
|
||||||
|
with pytest.raises(ValueError, match="numeric"):
|
||||||
|
validate_feature_frame(pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"bad": ["not numeric"],
|
||||||
|
}))
|
||||||
@@ -0,0 +1,288 @@
|
|||||||
|
"""Tests for raw Baostock minute bar download plumbing."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import data.downloader as low_level_downloader
|
||||||
|
import pipeline.data.downloader as pipeline_downloader
|
||||||
|
from data.downloader import download_minute_batch
|
||||||
|
from pipeline.common.schema import MINUTE_BAR_COLUMNS
|
||||||
|
from pipeline.data.downloader import download_minute_universe
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResult:
|
||||||
|
def __init__(self, rows, error_code="0", error_msg=""):
|
||||||
|
self.rows = rows
|
||||||
|
self.error_code = error_code
|
||||||
|
self.error_msg = error_msg
|
||||||
|
self._idx = -1
|
||||||
|
|
||||||
|
def next(self):
|
||||||
|
self._idx += 1
|
||||||
|
return self._idx < len(self.rows)
|
||||||
|
|
||||||
|
def get_row_data(self):
|
||||||
|
return self.rows[self._idx]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_maps_and_parses_baostock_rows(monkeypatch):
|
||||||
|
rows = [
|
||||||
|
[
|
||||||
|
"2024-01-02",
|
||||||
|
"20240102093500000",
|
||||||
|
"sh.600000",
|
||||||
|
"10",
|
||||||
|
"11",
|
||||||
|
"9",
|
||||||
|
"10.5",
|
||||||
|
"1000",
|
||||||
|
"10500",
|
||||||
|
"3",
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"2024-01-02",
|
||||||
|
"20240102094000000",
|
||||||
|
"sh.600000",
|
||||||
|
"10.5",
|
||||||
|
"12",
|
||||||
|
"10",
|
||||||
|
"11",
|
||||||
|
"2000",
|
||||||
|
"22000",
|
||||||
|
"3",
|
||||||
|
],
|
||||||
|
]
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake_query(**kwargs):
|
||||||
|
calls.append(kwargs)
|
||||||
|
return _FakeResult(rows)
|
||||||
|
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
fake_query,
|
||||||
|
)
|
||||||
|
|
||||||
|
[(symbol, df)] = list(
|
||||||
|
download_minute_batch(
|
||||||
|
["sh600000"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
frequency=5,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert symbol == "sh600000"
|
||||||
|
assert calls[0]["code"] == "sh.600000"
|
||||||
|
assert calls[0]["frequency"] == "5"
|
||||||
|
assert calls[0]["adjustflag"] == "3"
|
||||||
|
assert df is not None
|
||||||
|
assert df["datetime"].iloc[0] == pd.Timestamp("2024-01-02 09:35:00")
|
||||||
|
assert df["time"].tolist() == ["09:35:00", "09:40:00"]
|
||||||
|
assert (df["frequency"] == "5m").all()
|
||||||
|
assert np.isclose(df["open"].iloc[0], 10.0)
|
||||||
|
assert np.isclose(df["vwap"].iloc[0], 10.5)
|
||||||
|
assert pd.api.types.is_numeric_dtype(df["volume"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_minute_frequency_and_timestamp_parsing_edge_cases():
|
||||||
|
frequency, label = low_level_downloader._normalize_minute_frequency("15m")
|
||||||
|
assert (frequency, label) == ("15", "15m")
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="Unsupported minute frequency"):
|
||||||
|
low_level_downloader._normalize_minute_frequency("1m")
|
||||||
|
|
||||||
|
parsed = low_level_downloader._parse_minute_datetime(
|
||||||
|
pd.Series(["2024-01-02", "2024-01-02"]),
|
||||||
|
pd.Series(["0935", "09:40:00"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert parsed.tolist() == [
|
||||||
|
pd.Timestamp("2024-01-02 09:35:00"),
|
||||||
|
pd.Timestamp("2024-01-02 09:40:00"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_empty_result_yields_none(monkeypatch):
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult([]),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_non_login_error_and_periodic_relogin(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="1", error_msg="bad symbol"),
|
||||||
|
_FakeResult([]),
|
||||||
|
]
|
||||||
|
login_count = 0
|
||||||
|
logout_count = 0
|
||||||
|
|
||||||
|
def fake_login():
|
||||||
|
nonlocal login_count
|
||||||
|
login_count += 1
|
||||||
|
|
||||||
|
def fake_logout():
|
||||||
|
nonlocal logout_count
|
||||||
|
logout_count += 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", fake_login)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", fake_logout)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(
|
||||||
|
download_minute_batch(
|
||||||
|
["sh600000", "sz000001"],
|
||||||
|
"2024-01-02",
|
||||||
|
"2024-01-02",
|
||||||
|
relogin_every=1,
|
||||||
|
)
|
||||||
|
) == [("sh600000", None), ("sz000001", None)]
|
||||||
|
assert login_count == 2
|
||||||
|
assert logout_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_second_session_loss_yields_none(monkeypatch):
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
_FakeResult([], error_code="10002007", error_msg="用户未登录"),
|
||||||
|
]
|
||||||
|
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: responses.pop(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_batch_rejects_unparsed_timestamps(monkeypatch):
|
||||||
|
bad_rows = [[
|
||||||
|
"2024-01-02",
|
||||||
|
"not-a-time",
|
||||||
|
"sh.600000",
|
||||||
|
"10",
|
||||||
|
"11",
|
||||||
|
"9",
|
||||||
|
"10.5",
|
||||||
|
"1000",
|
||||||
|
"10500",
|
||||||
|
"3",
|
||||||
|
]]
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(low_level_downloader.bs, "logout", lambda: None)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
low_level_downloader.bs,
|
||||||
|
"query_history_k_data_plus",
|
||||||
|
lambda **kwargs: _FakeResult(bad_rows),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(download_minute_batch(["sh600000"], "2024-01-02", "2024-01-02")) == [
|
||||||
|
("sh600000", None)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_universe_writes_frequency_month_partitions(tmp_path, monkeypatch):
|
||||||
|
minute = pd.DataFrame({
|
||||||
|
"symbol": ["sh600000", "sh600000"],
|
||||||
|
"datetime": pd.to_datetime(["2024-01-02 09:35:00", "2024-01-02 09:40:00"]),
|
||||||
|
"date": pd.to_datetime(["2024-01-02", "2024-01-02"]),
|
||||||
|
"time": ["09:35:00", "09:40:00"],
|
||||||
|
"frequency": ["5m", "5m"],
|
||||||
|
"open": [10.0, 10.5],
|
||||||
|
"high": [11.0, 12.0],
|
||||||
|
"low": [9.0, 10.0],
|
||||||
|
"close": [10.5, 11.0],
|
||||||
|
"volume": [1000.0, 2000.0],
|
||||||
|
"amount": [10500.0, 22000.0],
|
||||||
|
"vwap": [10.5, 11.0],
|
||||||
|
"adjustflag": ["3", "3"],
|
||||||
|
})
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"symbol_name": ["PF Bank"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_batch(symbols, start, end, frequency=5):
|
||||||
|
assert symbols == ["sh600000"]
|
||||||
|
assert frequency == "5"
|
||||||
|
yield "sh600000", minute
|
||||||
|
|
||||||
|
monkeypatch.setattr(pipeline_downloader, "download_minute_batch", fake_batch)
|
||||||
|
|
||||||
|
preserved = tmp_path / "toy" / "frequency=15m" / "month=2024-01" / "old.pq"
|
||||||
|
preserved.parent.mkdir(parents=True)
|
||||||
|
preserved_minute = minute.copy()
|
||||||
|
preserved_minute["frequency"] = "15m"
|
||||||
|
preserved_minute["symbol_id"] = "sh600000"
|
||||||
|
preserved_minute["symbol_name"] = "PF Bank"
|
||||||
|
preserved_minute[MINUTE_BAR_COLUMNS].to_parquet(preserved, index=False)
|
||||||
|
|
||||||
|
stats = download_minute_universe(
|
||||||
|
universe="toy",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-01-02",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
chunk_size=1,
|
||||||
|
frequency="5",
|
||||||
|
)
|
||||||
|
|
||||||
|
dataset_path = Path(stats["dataset_path"])
|
||||||
|
assert (dataset_path / "frequency=5m" / "month=2024-01").is_dir()
|
||||||
|
assert preserved.exists()
|
||||||
|
out = pd.read_parquet(dataset_path / "frequency=5m")
|
||||||
|
assert (set(MINUTE_BAR_COLUMNS) - {"frequency"}) <= set(out.columns)
|
||||||
|
assert set(out["symbol_id"]) == {"sh600000"}
|
||||||
|
assert set(out["symbol_name"]) == {"PF Bank"}
|
||||||
|
assert stats["n_rows"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_download_minute_universe_raises_when_all_symbols_empty(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"_resolve_universe",
|
||||||
|
lambda universe, max_symbols=0: pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"symbol_name": ["PF Bank"],
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
pipeline_downloader,
|
||||||
|
"download_minute_batch",
|
||||||
|
lambda symbols, start, end, frequency=5: iter([("sh600000", None)]),
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="No minute data"):
|
||||||
|
download_minute_universe(
|
||||||
|
universe="toy",
|
||||||
|
start_date="2024-01-02",
|
||||||
|
end_date="2024-01-02",
|
||||||
|
output_dir=str(tmp_path),
|
||||||
|
)
|
||||||
@@ -306,6 +306,47 @@ def test_construct_positions_schema():
|
|||||||
assert pos["position_shares"].dtype == np.int64
|
assert pos["position_shares"].dtype == np.int64
|
||||||
|
|
||||||
|
|
||||||
|
def test_construct_positions_empty_weights_returns_schema():
|
||||||
|
data = _make_data(n_days=3)
|
||||||
|
empty_weights = pd.DataFrame(columns=["symbol_id", "date", "combo_name", "weight"])
|
||||||
|
|
||||||
|
pos = construct_positions(
|
||||||
|
empty_weights,
|
||||||
|
data,
|
||||||
|
booksize=1e6,
|
||||||
|
portfolio_name="empty",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(pos.columns) == POSITION_COLUMNS
|
||||||
|
assert pos.empty
|
||||||
|
|
||||||
|
|
||||||
|
def test_construct_positions_ignores_absent_or_bad_prices():
|
||||||
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
||||||
|
data = pd.DataFrame([
|
||||||
|
{"symbol_id": "sh600000", "date": dates[0], "close": np.nan, "isST": 0},
|
||||||
|
{"symbol_id": "sz000001", "date": dates[0], "close": 20.0, "isST": 0},
|
||||||
|
{"symbol_id": "sh600000", "date": dates[1], "close": 10.0, "isST": 0},
|
||||||
|
{"symbol_id": "sz000001", "date": dates[1], "close": 20.0, "isST": 0},
|
||||||
|
])
|
||||||
|
weights = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1], dates[1]],
|
||||||
|
"combo_name": ["combo"] * 4,
|
||||||
|
"weight": [1.0, -1.0, 1.0, -1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
pos = construct_positions(weights, data, booksize=10000.0, portfolio_name="bad_price")
|
||||||
|
|
||||||
|
bad_price_rows = pos[
|
||||||
|
(pos["date"] == dates[0])
|
||||||
|
& (pos["symbol_id"] == "sh600000")
|
||||||
|
]
|
||||||
|
assert bad_price_rows.empty or (bad_price_rows["target_weight"] == 0.0).all()
|
||||||
|
assert np.isfinite(pos["target_value"]).all()
|
||||||
|
assert np.isfinite(pos["position_value"]).all()
|
||||||
|
|
||||||
|
|
||||||
def test_construct_positions_threads_state_and_closes_absent():
|
def test_construct_positions_threads_state_and_closes_absent():
|
||||||
data = _make_data()
|
data = _make_data()
|
||||||
weights = _make_weights(data)
|
weights = _make_weights(data)
|
||||||
@@ -321,6 +362,34 @@ def test_construct_positions_threads_state_and_closes_absent():
|
|||||||
assert final.empty or (final["position_shares"] == 0).all()
|
assert final.empty or (final["position_shares"] == 0).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_construct_positions_closes_absent_short_position():
|
||||||
|
dates = pd.to_datetime(["2024-01-02", "2024-01-03"])
|
||||||
|
data = pd.DataFrame([
|
||||||
|
{"symbol_id": sym, "date": d, "close": price, "isST": 0}
|
||||||
|
for d in dates
|
||||||
|
for sym, price in (("sh600000", 10.0), ("sz000001", 20.0))
|
||||||
|
])
|
||||||
|
weights = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1]],
|
||||||
|
"combo_name": ["combo", "combo", "combo"],
|
||||||
|
"weight": [-1.0, 1.0, 1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
pos = construct_positions(weights, data, booksize=20000.0, portfolio_name="absent_short")
|
||||||
|
|
||||||
|
first_day_short = pos[
|
||||||
|
(pos["date"] == dates[0])
|
||||||
|
& (pos["symbol_id"] == "sh600000")
|
||||||
|
]
|
||||||
|
final_day_short = pos[
|
||||||
|
(pos["date"] == dates[1])
|
||||||
|
& (pos["symbol_id"] == "sh600000")
|
||||||
|
]
|
||||||
|
assert (first_day_short["position_shares"] < 0).all()
|
||||||
|
assert final_day_short.empty or (final_day_short["position_shares"] == 0).all()
|
||||||
|
|
||||||
|
|
||||||
def test_construct_positions_carries_book_on_zero_gross(caplog):
|
def test_construct_positions_carries_book_on_zero_gross(caplog):
|
||||||
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
||||||
symbols = ["sh600000", "sz000001"]
|
symbols = ["sh600000", "sz000001"]
|
||||||
@@ -392,6 +461,42 @@ def test_volume_cap_uses_traded_value():
|
|||||||
assert low[0] == -10000.0
|
assert low[0] == -10000.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_constraints_compose_repeatably_regardless_of_order():
|
||||||
|
n = 1
|
||||||
|
sl = _slice(
|
||||||
|
n,
|
||||||
|
tradestatus=np.array([0.0]),
|
||||||
|
limit_status=np.array([LimitStatus.UP_LIMIT.value], dtype=np.int8),
|
||||||
|
amount=np.array([1_000.0]),
|
||||||
|
price=np.array([10.0]),
|
||||||
|
)
|
||||||
|
ctx = TradeContext(np.zeros(n, np.int64), np.array([500]), sl, 1e6)
|
||||||
|
first_order = ReferenceSimulator(
|
||||||
|
constraints=[
|
||||||
|
SuspensionConstraint(),
|
||||||
|
PriceLimitConstraint(),
|
||||||
|
VolumeCapConstraint(max_frac=0.1),
|
||||||
|
],
|
||||||
|
cost_bps=10,
|
||||||
|
).fill(ctx)
|
||||||
|
reversed_order = ReferenceSimulator(
|
||||||
|
constraints=[
|
||||||
|
VolumeCapConstraint(max_frac=0.1),
|
||||||
|
PriceLimitConstraint(),
|
||||||
|
SuspensionConstraint(),
|
||||||
|
],
|
||||||
|
cost_bps=10,
|
||||||
|
).fill(ctx)
|
||||||
|
|
||||||
|
assert first_order.traded_shares.tolist() == [0]
|
||||||
|
assert first_order.realized_shares.tolist() == [0]
|
||||||
|
assert first_order.blocked.tolist() == [1]
|
||||||
|
assert np.array_equal(first_order.traded_shares, reversed_order.traded_shares)
|
||||||
|
assert np.array_equal(first_order.realized_shares, reversed_order.realized_shares)
|
||||||
|
assert np.array_equal(first_order.blocked, reversed_order.blocked)
|
||||||
|
assert np.array_equal(first_order.cost, reversed_order.cost)
|
||||||
|
|
||||||
|
|
||||||
# --- ReferenceSimulator ------------------------------------------------------
|
# --- ReferenceSimulator ------------------------------------------------------
|
||||||
|
|
||||||
def test_simulator_next_open_and_blocked_buy_holds_prev():
|
def test_simulator_next_open_and_blocked_buy_holds_prev():
|
||||||
@@ -474,6 +579,91 @@ def test_simulator_cost_only_on_nonzero_realized_trades():
|
|||||||
assert np.isclose(res.cost[1], 50 * 20 * 10 / 1e4)
|
assert np.isclose(res.cost[1], 50 * 20 * 10 / 1e4)
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_short_to_long_flip_trades_full_delta():
|
||||||
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02", "2024-01-03"])
|
||||||
|
positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": [dates[0], dates[1]],
|
||||||
|
"portfolio_name": ["flip", "flip"],
|
||||||
|
"target_weight": [-1.0, 1.0],
|
||||||
|
"target_value": [-1000.0, 1000.0],
|
||||||
|
"target_shares": [-100.0, 100.0],
|
||||||
|
"position_shares": [-100, 100],
|
||||||
|
"position_value": [-1000.0, 1000.0],
|
||||||
|
"price": [10.0, 10.0],
|
||||||
|
})
|
||||||
|
data = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"] * 3,
|
||||||
|
"date": dates,
|
||||||
|
"open": [10.0, 10.0, 10.0],
|
||||||
|
"close": [10.0, 10.0, 10.0],
|
||||||
|
"preclose": [10.0, 10.0, 10.0],
|
||||||
|
"amount": [1e9, 1e9, 1e9],
|
||||||
|
"tradestatus": [1, 1, 1],
|
||||||
|
"isST": [0, 0, 0],
|
||||||
|
})
|
||||||
|
|
||||||
|
fills, _ = ReferenceSimulator().run(positions, data)
|
||||||
|
|
||||||
|
by_date = fills.set_index("date")
|
||||||
|
assert by_date.loc[dates[1], "traded_shares"] == -100
|
||||||
|
assert by_date.loc[dates[1], "realized_shares"] == -100
|
||||||
|
assert by_date.loc[dates[2], "prev_shares"] == -100
|
||||||
|
assert by_date.loc[dates[2], "traded_shares"] == 200
|
||||||
|
assert by_date.loc[dates[2], "realized_shares"] == 100
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_volume_cap_partially_fills_sell():
|
||||||
|
sl = _slice(1, amount=np.array([10_000.0]), price=np.array([10.0]))
|
||||||
|
ctx = TradeContext(
|
||||||
|
np.array([1000], np.int64),
|
||||||
|
np.array([0], np.int64),
|
||||||
|
sl,
|
||||||
|
1_000_000.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = ReferenceSimulator(
|
||||||
|
constraints=[VolumeCapConstraint(max_frac=0.10)]
|
||||||
|
).fill(ctx)
|
||||||
|
|
||||||
|
assert result.traded_shares.tolist() == [-100]
|
||||||
|
assert result.realized_shares.tolist() == [900]
|
||||||
|
assert result.blocked.tolist() == [1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_simulator_missing_next_open_has_zero_cost_and_turnover():
|
||||||
|
dates = pd.to_datetime(["2024-01-01", "2024-01-02"])
|
||||||
|
positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [dates[0]],
|
||||||
|
"portfolio_name": ["missing_open"],
|
||||||
|
"target_weight": [1.0],
|
||||||
|
"target_value": [1000.0],
|
||||||
|
"target_shares": [100.0],
|
||||||
|
"position_shares": [100],
|
||||||
|
"position_value": [1000.0],
|
||||||
|
"price": [10.0],
|
||||||
|
})
|
||||||
|
data = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": dates,
|
||||||
|
"open": [10.0, np.nan],
|
||||||
|
"close": [10.0, 10.0],
|
||||||
|
"preclose": [10.0, 10.0],
|
||||||
|
"amount": [1e9, 1e9],
|
||||||
|
"tradestatus": [1, 1],
|
||||||
|
"isST": [0, 0],
|
||||||
|
})
|
||||||
|
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=10, slippage_bps=5).run(positions, data)
|
||||||
|
|
||||||
|
assert fills["traded_shares"].iloc[0] == 100
|
||||||
|
assert fills["trade_cost"].iloc[0] == 0.0
|
||||||
|
assert pnl["cost"].iloc[0] == 0.0
|
||||||
|
assert pnl["turnover"].iloc[0] == 0.0
|
||||||
|
assert pnl["gross_exposure"].iloc[0] == 1000.0
|
||||||
|
|
||||||
|
|
||||||
def test_simple_cost_model_adds_cost_and_slippage_without_price_adjustment():
|
def test_simple_cost_model_adds_cost_and_slippage_without_price_adjustment():
|
||||||
model = SimpleProportionalCostModel(cost_bps=10, slippage_bps=5)
|
model = SimpleProportionalCostModel(cost_bps=10, slippage_bps=5)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
"""Malformed parquet/input tests for phase boundary contracts."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pipeline.alpha.compute import compute_alpha
|
||||||
|
from pipeline.combo.combine import combine_alphas
|
||||||
|
from pipeline.derived.compute import validate_derived_frame
|
||||||
|
from pipeline.portfolio.construct import construct_positions
|
||||||
|
from pipeline.portfolio.simulator import ReferenceSimulator
|
||||||
|
from tests.helpers import make_generated_daily_bars
|
||||||
|
|
||||||
|
|
||||||
|
def test_alpha_compute_rejects_daily_data_without_close():
|
||||||
|
daily = make_generated_daily_bars().drop(columns=["close"])
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="close"):
|
||||||
|
compute_alpha(daily, "bad", "reversal", lookback=3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_alpha_feature_path_rejects_duplicate_symbol_dates(tmp_path):
|
||||||
|
daily = make_generated_daily_bars()
|
||||||
|
feature = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000"],
|
||||||
|
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
|
||||||
|
"toy_feature": [1.0, 2.0],
|
||||||
|
})
|
||||||
|
feature_path = tmp_path / "duplicate_feature_keys.pq"
|
||||||
|
feature.to_parquet(feature_path, index=False)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
|
||||||
|
compute_alpha(
|
||||||
|
daily,
|
||||||
|
"bad_features",
|
||||||
|
"reversal",
|
||||||
|
lookback=3,
|
||||||
|
feature_paths=[str(feature_path)],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_validation_rejects_bool_value_columns():
|
||||||
|
derived = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"is_good": [True],
|
||||||
|
})
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="numeric"):
|
||||||
|
validate_derived_frame(derived)
|
||||||
|
|
||||||
|
|
||||||
|
def test_combo_combine_rejects_missing_weight_column(tmp_path):
|
||||||
|
bad_alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"alpha_name": ["bad"],
|
||||||
|
})
|
||||||
|
bad_alpha_path = tmp_path / "bad_alpha.pq"
|
||||||
|
bad_alpha.to_parquet(bad_alpha_path, index=False)
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="weight"):
|
||||||
|
combine_alphas([str(bad_alpha_path)], "bad_combo")
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_build_rejects_weights_without_symbol_id():
|
||||||
|
daily = make_generated_daily_bars()
|
||||||
|
bad_weights = pd.DataFrame({
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"combo_name": ["bad"],
|
||||||
|
"weight": [1.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="symbol_id"):
|
||||||
|
construct_positions(
|
||||||
|
bad_weights,
|
||||||
|
daily,
|
||||||
|
booksize=1_000_000.0,
|
||||||
|
portfolio_name="bad_portfolio",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_portfolio_simulate_rejects_positions_without_position_shares():
|
||||||
|
daily = make_generated_daily_bars()
|
||||||
|
bad_positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000"],
|
||||||
|
"date": [pd.Timestamp("2024-01-02")],
|
||||||
|
"portfolio_name": ["bad"],
|
||||||
|
"target_weight": [1.0],
|
||||||
|
"target_value": [1000.0],
|
||||||
|
"target_shares": [100.0],
|
||||||
|
"position_value": [1000.0],
|
||||||
|
"price": [10.0],
|
||||||
|
})
|
||||||
|
|
||||||
|
with pytest.raises(KeyError, match="position_shares"):
|
||||||
|
ReferenceSimulator().run(bad_positions, daily)
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
"""Offline tests for baostock-backed universe helpers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
import data.universe as universe
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResult:
|
||||||
|
def __init__(self, rows, fields=None):
|
||||||
|
self.rows = rows
|
||||||
|
self.fields = fields or ["code", "name", "date"]
|
||||||
|
self._idx = -1
|
||||||
|
|
||||||
|
def next(self):
|
||||||
|
self._idx += 1
|
||||||
|
return self._idx < len(self.rows)
|
||||||
|
|
||||||
|
def get_row_data(self):
|
||||||
|
return self.rows[self._idx]
|
||||||
|
|
||||||
|
|
||||||
|
def test_index_constituent_helpers_normalize_dotted_codes(monkeypatch):
|
||||||
|
calls: list[str] = []
|
||||||
|
|
||||||
|
monkeypatch.setattr(universe.bs, "login", lambda: calls.append("login"))
|
||||||
|
monkeypatch.setattr(universe.bs, "logout", lambda: calls.append("logout"))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
universe.bs,
|
||||||
|
"query_hs300_stocks",
|
||||||
|
lambda: _FakeResult([
|
||||||
|
["sh.600000", "浦发银行", "2024-01-12"],
|
||||||
|
["sz.000001", "平安银行", "2024-01-12"],
|
||||||
|
]),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
universe.bs,
|
||||||
|
"query_zz500_stocks",
|
||||||
|
lambda: _FakeResult([
|
||||||
|
["sh.600006", "东风汽车", "2024-01-12"],
|
||||||
|
]),
|
||||||
|
)
|
||||||
|
|
||||||
|
hs300 = universe.get_hs300_stocks()
|
||||||
|
zz500 = universe.get_zz500_stocks()
|
||||||
|
|
||||||
|
assert calls == ["login", "logout", "login", "logout"]
|
||||||
|
assert hs300["code"].tolist() == ["sh600000", "sz000001"]
|
||||||
|
assert zz500["code"].tolist() == ["sh600006"]
|
||||||
|
assert hs300["name"].tolist() == ["浦发银行", "平安银行"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_all_stocks_walks_back_and_filters_to_listed_a_shares(monkeypatch):
|
||||||
|
fields = ["code", "tradeStatus", "code_name"]
|
||||||
|
responses = [
|
||||||
|
_FakeResult([], fields=fields),
|
||||||
|
_FakeResult(
|
||||||
|
[
|
||||||
|
["sh.600000", "1", "浦发银行"],
|
||||||
|
["sh.688001", "1", "华兴源创"],
|
||||||
|
["sz.000001", "1", "平安银行"],
|
||||||
|
["sz.300750", "1", "宁德时代"],
|
||||||
|
["sz.399001", "1", "深证成指"],
|
||||||
|
["sz.200001", "1", "深物业B"],
|
||||||
|
["bj.430047", "1", "北交所样本"],
|
||||||
|
],
|
||||||
|
fields=fields,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
query_days: list[str] = []
|
||||||
|
|
||||||
|
monkeypatch.setattr(universe.bs, "login", lambda: None)
|
||||||
|
monkeypatch.setattr(universe.bs, "logout", lambda: None)
|
||||||
|
|
||||||
|
def fake_query_all_stock(day):
|
||||||
|
query_days.append(day)
|
||||||
|
return responses.pop(0)
|
||||||
|
|
||||||
|
monkeypatch.setattr(universe.bs, "query_all_stock", fake_query_all_stock)
|
||||||
|
|
||||||
|
result = universe.get_all_stocks("2024-01-07")
|
||||||
|
|
||||||
|
assert query_days == ["2024-01-07", "2024-01-06"]
|
||||||
|
assert result.columns.tolist() == ["code", "name"]
|
||||||
|
assert result["code"].tolist() == [
|
||||||
|
"sh600000",
|
||||||
|
"sh688001",
|
||||||
|
"sz000001",
|
||||||
|
"sz300750",
|
||||||
|
]
|
||||||
|
assert result["name"].tolist() == ["浦发银行", "华兴源创", "平安银行", "宁德时代"]
|
||||||
@@ -0,0 +1,508 @@
|
|||||||
|
"""Verbose offline checks for the daily research workflow."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pipeline.alpha.compute import compute_alpha
|
||||||
|
from pipeline.combo.combine import combine_alphas
|
||||||
|
from pipeline.common.schema import (
|
||||||
|
ALPHA_COLUMNS,
|
||||||
|
COMBO_COLUMNS,
|
||||||
|
FILL_COLUMNS,
|
||||||
|
PNL_COLUMNS,
|
||||||
|
POSITION_COLUMNS,
|
||||||
|
)
|
||||||
|
from pipeline.portfolio.constraints import (
|
||||||
|
PriceLimitConstraint,
|
||||||
|
SuspensionConstraint,
|
||||||
|
VolumeCapConstraint,
|
||||||
|
)
|
||||||
|
from pipeline.portfolio.construct import construct_positions
|
||||||
|
from pipeline.portfolio.research import evaluate_portfolio
|
||||||
|
from pipeline.portfolio.simulator import ReferenceSimulator
|
||||||
|
from tests.helpers import (
|
||||||
|
GENERATED_SYMBOLS,
|
||||||
|
generated_sessions,
|
||||||
|
make_generated_alpha_weights,
|
||||||
|
make_generated_combo_weights,
|
||||||
|
make_generated_daily_bars,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
FIXTURE_PATH = Path(__file__).parent / "fixtures" / "daily_bars_real_2024_01_sample.pq"
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_sorted_by_symbol_date(frame: pd.DataFrame) -> None:
|
||||||
|
expected = frame.sort_values(["symbol_id", "date"]).reset_index(drop=True)
|
||||||
|
pd.testing.assert_frame_equal(frame.reset_index(drop=True), expected)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_metric_dict_is_finite(metrics: dict[str, float]) -> None:
|
||||||
|
for key in (
|
||||||
|
"cumulative_return",
|
||||||
|
"sharpe_annual",
|
||||||
|
"turnover_annual",
|
||||||
|
"max_drawdown",
|
||||||
|
"hit_rate",
|
||||||
|
"n_dates",
|
||||||
|
):
|
||||||
|
assert key in metrics
|
||||||
|
assert np.isfinite(metrics[key])
|
||||||
|
assert "ic" not in metrics
|
||||||
|
assert "rank_ic" not in metrics
|
||||||
|
assert "ir" not in metrics
|
||||||
|
|
||||||
|
|
||||||
|
def test_tiny_workflow_golden_outputs_are_stable(tmp_path):
|
||||||
|
dates = pd.to_datetime(["2024-01-02", "2024-01-03", "2024-01-04"])
|
||||||
|
daily_bars = pd.DataFrame([
|
||||||
|
{
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "A",
|
||||||
|
"date": dates[0],
|
||||||
|
"open": 10.0,
|
||||||
|
"high": 10.0,
|
||||||
|
"low": 10.0,
|
||||||
|
"close": 10.0,
|
||||||
|
"preclose": 10.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 10_000_000.0,
|
||||||
|
"vwap": 10.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 0.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "B",
|
||||||
|
"date": dates[0],
|
||||||
|
"open": 20.0,
|
||||||
|
"high": 20.0,
|
||||||
|
"low": 20.0,
|
||||||
|
"close": 20.0,
|
||||||
|
"preclose": 20.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 20_000_000.0,
|
||||||
|
"vwap": 20.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 0.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "A",
|
||||||
|
"date": dates[1],
|
||||||
|
"open": 10.0,
|
||||||
|
"high": 12.0,
|
||||||
|
"low": 10.0,
|
||||||
|
"close": 12.0,
|
||||||
|
"preclose": 10.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 10_000_000.0,
|
||||||
|
"vwap": 10.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 20.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "B",
|
||||||
|
"date": dates[1],
|
||||||
|
"open": 20.0,
|
||||||
|
"high": 20.0,
|
||||||
|
"low": 18.0,
|
||||||
|
"close": 18.0,
|
||||||
|
"preclose": 20.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 20_000_000.0,
|
||||||
|
"vwap": 20.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": -10.0,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sh600000",
|
||||||
|
"symbol_name": "A",
|
||||||
|
"date": dates[2],
|
||||||
|
"open": 12.0,
|
||||||
|
"high": 13.0,
|
||||||
|
"low": 12.0,
|
||||||
|
"close": 13.0,
|
||||||
|
"preclose": 12.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 12_000_000.0,
|
||||||
|
"vwap": 12.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 8.33,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"symbol_id": "sz000001",
|
||||||
|
"symbol_name": "B",
|
||||||
|
"date": dates[2],
|
||||||
|
"open": 18.0,
|
||||||
|
"high": 21.0,
|
||||||
|
"low": 18.0,
|
||||||
|
"close": 21.0,
|
||||||
|
"preclose": 18.0,
|
||||||
|
"volume": 1_000_000.0,
|
||||||
|
"amount": 18_000_000.0,
|
||||||
|
"vwap": 18.0,
|
||||||
|
"turn": 1.0,
|
||||||
|
"pctChg": 16.67,
|
||||||
|
"tradestatus": 1,
|
||||||
|
"isST": 0,
|
||||||
|
"peTTM": 1.0,
|
||||||
|
"pbMRQ": 1.0,
|
||||||
|
"psTTM": 1.0,
|
||||||
|
"pcfNcfTTM": 1.0,
|
||||||
|
},
|
||||||
|
])
|
||||||
|
alpha = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[0], dates[0], dates[1], dates[1]],
|
||||||
|
"alpha_name": ["gold_alpha"] * 4,
|
||||||
|
"weight": [1.0, -1.0, -1.0, 1.0],
|
||||||
|
})
|
||||||
|
alpha_path = tmp_path / "gold_alpha.pq"
|
||||||
|
alpha.to_parquet(alpha_path, index=False)
|
||||||
|
|
||||||
|
combo = combine_alphas([str(alpha_path)], "gold_combo")
|
||||||
|
positions = construct_positions(combo, daily_bars, booksize=20_000.0, portfolio_name="gold_port")
|
||||||
|
fills, pnl = ReferenceSimulator().run(positions, daily_bars)
|
||||||
|
|
||||||
|
expected_combo = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
|
||||||
|
"date": [dates[0], dates[1], dates[0], dates[1]],
|
||||||
|
"combo_name": ["gold_combo"] * 4,
|
||||||
|
"weight": [1.0, -1.0, -1.0, 1.0],
|
||||||
|
})
|
||||||
|
expected_positions = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sh600000", "sz000001", "sz000001"],
|
||||||
|
"date": [dates[0], dates[1], dates[0], dates[1]],
|
||||||
|
"portfolio_name": ["gold_port"] * 4,
|
||||||
|
"target_weight": [0.5, -0.5, -0.5, 0.5],
|
||||||
|
"target_value": [10000.0, -10000.0, -10000.0, 10000.0],
|
||||||
|
"target_shares": [1000.0, -10000.0 / 12.0, -500.0, 10000.0 / 18.0],
|
||||||
|
"position_shares": [1000, -833, -500, 556],
|
||||||
|
"position_value": [10000.0, -9996.0, -10000.0, 10008.0],
|
||||||
|
"price": [10.0, 12.0, 20.0, 18.0],
|
||||||
|
})
|
||||||
|
expected_fills = pd.DataFrame({
|
||||||
|
"symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
|
||||||
|
"date": [dates[1], dates[1], dates[2], dates[2]],
|
||||||
|
"portfolio_name": ["gold_port"] * 4,
|
||||||
|
"prev_shares": [0, 0, 1000, -500],
|
||||||
|
"target_shares": [1000, -500, -833, 556],
|
||||||
|
"traded_shares": [1000, -500, -1833, 1056],
|
||||||
|
"realized_shares": [1000, -500, -833, 556],
|
||||||
|
"blocked": [0, 0, 0, 0],
|
||||||
|
"trade_cost": [0.0, 0.0, 0.0, 0.0],
|
||||||
|
})
|
||||||
|
expected_pnl = pd.DataFrame({
|
||||||
|
"date": [dates[1], dates[2]],
|
||||||
|
"portfolio_name": ["gold_port", "gold_port"],
|
||||||
|
"gross_exposure": [21000.0, 22505.0],
|
||||||
|
"net_exposure": [3000.0, 847.0],
|
||||||
|
"pnl": [3000.0, 835.0],
|
||||||
|
"cost": [0.0, 0.0],
|
||||||
|
"turnover": [1.0, 2.0502],
|
||||||
|
"n_positions": [2, 2],
|
||||||
|
})
|
||||||
|
|
||||||
|
pd.testing.assert_frame_equal(combo, expected_combo)
|
||||||
|
pd.testing.assert_frame_equal(positions, expected_positions)
|
||||||
|
pd.testing.assert_frame_equal(fills, expected_fills)
|
||||||
|
pd.testing.assert_frame_equal(pnl, expected_pnl)
|
||||||
|
|
||||||
|
|
||||||
|
def test_generated_alpha_combo_portfolio_execution_workflow(tmp_path):
|
||||||
|
daily_bars = make_generated_daily_bars()
|
||||||
|
|
||||||
|
computed_alpha = compute_alpha(
|
||||||
|
data=daily_bars,
|
||||||
|
alpha_name="generated_reversal_3d",
|
||||||
|
alpha_type="reversal",
|
||||||
|
lookback=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(computed_alpha.columns) == ALPHA_COLUMNS
|
||||||
|
assert not computed_alpha.empty
|
||||||
|
assert set(computed_alpha["symbol_id"]).issubset(set(GENERATED_SYMBOLS))
|
||||||
|
assert computed_alpha["date"].min() > daily_bars["date"].min()
|
||||||
|
assert computed_alpha["weight"].notna().all()
|
||||||
|
assert computed_alpha["weight"].abs().sum() > 0.0
|
||||||
|
assert {"ic", "rank_ic", "ir"}.isdisjoint(computed_alpha.columns)
|
||||||
|
_assert_sorted_by_symbol_date(computed_alpha)
|
||||||
|
|
||||||
|
alpha_a = make_generated_alpha_weights("alpha_a", zero_date_index=2)
|
||||||
|
alpha_b = make_generated_alpha_weights(
|
||||||
|
"alpha_b",
|
||||||
|
scale=0.5,
|
||||||
|
offset=0.25,
|
||||||
|
zero_date_index=2,
|
||||||
|
)
|
||||||
|
alpha_a_path = tmp_path / "alpha_a.pq"
|
||||||
|
alpha_b_path = tmp_path / "alpha_b.pq"
|
||||||
|
alpha_a.to_parquet(alpha_a_path, index=False)
|
||||||
|
alpha_b.to_parquet(alpha_b_path, index=False)
|
||||||
|
|
||||||
|
identity_combo = combine_alphas([str(alpha_a_path)], "identity_combo")
|
||||||
|
assert list(identity_combo.columns) == COMBO_COLUMNS
|
||||||
|
assert (identity_combo["combo_name"] == "identity_combo").all()
|
||||||
|
pd.testing.assert_frame_equal(
|
||||||
|
identity_combo[["symbol_id", "date", "weight"]],
|
||||||
|
alpha_a[["symbol_id", "date", "weight"]],
|
||||||
|
)
|
||||||
|
|
||||||
|
equal_combo = combine_alphas([str(alpha_a_path), str(alpha_b_path)], "equal_combo")
|
||||||
|
expected_equal_weights = (
|
||||||
|
pd.concat([alpha_a, alpha_b], ignore_index=True)
|
||||||
|
.groupby(["symbol_id", "date"], as_index=False)["weight"]
|
||||||
|
.mean()
|
||||||
|
.sort_values(["symbol_id", "date"])
|
||||||
|
.reset_index(drop=True)
|
||||||
|
)
|
||||||
|
pd.testing.assert_frame_equal(
|
||||||
|
equal_combo[["symbol_id", "date", "weight"]],
|
||||||
|
expected_equal_weights,
|
||||||
|
)
|
||||||
|
|
||||||
|
portfolio_weights = make_generated_combo_weights("workflow_combo", zero_date_index=2)
|
||||||
|
positions = construct_positions(
|
||||||
|
weights_df=portfolio_weights,
|
||||||
|
data_df=daily_bars,
|
||||||
|
booksize=2_000_000.0,
|
||||||
|
portfolio_name="workflow_portfolio",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert list(positions.columns) == POSITION_COLUMNS
|
||||||
|
assert not positions.empty
|
||||||
|
assert (positions["portfolio_name"] == "workflow_portfolio").all()
|
||||||
|
assert pd.api.types.is_integer_dtype(positions["position_shares"])
|
||||||
|
assert np.allclose(
|
||||||
|
positions["position_value"],
|
||||||
|
positions["position_shares"].astype(float) * positions["price"].fillna(0.0),
|
||||||
|
)
|
||||||
|
|
||||||
|
target_gross_by_date = positions.groupby("date")["target_weight"].apply(lambda s: s.abs().sum())
|
||||||
|
nonzero_target_dates = target_gross_by_date[target_gross_by_date > 0.0]
|
||||||
|
assert np.allclose(nonzero_target_dates, 1.0)
|
||||||
|
|
||||||
|
nonzero_share_counts = positions.loc[positions["position_shares"] != 0, "position_shares"].abs()
|
||||||
|
assert (nonzero_share_counts >= 100).all()
|
||||||
|
|
||||||
|
zero_gross_date = generated_sessions(10)[2]
|
||||||
|
previous_date = generated_sessions(10)[1]
|
||||||
|
zero_gross_positions = positions[positions["date"] == zero_gross_date].set_index("symbol_id")
|
||||||
|
previous_positions = positions[positions["date"] == previous_date].set_index("symbol_id")
|
||||||
|
common_symbols = zero_gross_positions.index.intersection(previous_positions.index)
|
||||||
|
assert not common_symbols.empty
|
||||||
|
assert (zero_gross_positions.loc[common_symbols, "target_weight"] == 0.0).all()
|
||||||
|
pd.testing.assert_series_equal(
|
||||||
|
zero_gross_positions.loc[common_symbols, "position_shares"],
|
||||||
|
previous_positions.loc[common_symbols, "position_shares"],
|
||||||
|
check_names=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
simulator = ReferenceSimulator(
|
||||||
|
constraints=[
|
||||||
|
SuspensionConstraint(),
|
||||||
|
PriceLimitConstraint(),
|
||||||
|
VolumeCapConstraint(max_frac=0.02),
|
||||||
|
],
|
||||||
|
cost_bps=5,
|
||||||
|
slippage_bps=5,
|
||||||
|
)
|
||||||
|
fills, pnl = simulator.run(positions, daily_bars)
|
||||||
|
|
||||||
|
assert list(fills.columns) == FILL_COLUMNS
|
||||||
|
assert list(pnl.columns) == PNL_COLUMNS
|
||||||
|
assert not fills.empty
|
||||||
|
assert not pnl.empty
|
||||||
|
assert (fills["realized_shares"] == fills["prev_shares"] + fills["traded_shares"]).all()
|
||||||
|
assert fills["blocked"].sum() > 0
|
||||||
|
|
||||||
|
fill_prices = fills.merge(
|
||||||
|
daily_bars[["symbol_id", "date", "open"]],
|
||||||
|
on=["symbol_id", "date"],
|
||||||
|
how="left",
|
||||||
|
validate="many_to_one",
|
||||||
|
)
|
||||||
|
expected_trade_cost = (
|
||||||
|
fill_prices["traded_shares"].abs()
|
||||||
|
* fill_prices["open"].fillna(0.0)
|
||||||
|
* 10
|
||||||
|
/ 10_000
|
||||||
|
)
|
||||||
|
assert np.allclose(fill_prices["trade_cost"], expected_trade_cost)
|
||||||
|
|
||||||
|
cost_by_date = fills.groupby("date")["trade_cost"].sum()
|
||||||
|
assert np.allclose(
|
||||||
|
pnl.set_index("date")["cost"],
|
||||||
|
cost_by_date.reindex(pnl["date"], fill_value=0.0),
|
||||||
|
)
|
||||||
|
|
||||||
|
booksize_used_by_simulator = positions.groupby("date")["target_value"].apply(lambda s: s.abs().sum()).max()
|
||||||
|
traded_value_by_date = (
|
||||||
|
fill_prices.assign(traded_value=fill_prices["traded_shares"].abs() * fill_prices["open"])
|
||||||
|
.groupby("date")["traded_value"]
|
||||||
|
.sum()
|
||||||
|
)
|
||||||
|
assert np.allclose(
|
||||||
|
pnl.set_index("date")["turnover"],
|
||||||
|
traded_value_by_date.reindex(pnl["date"], fill_value=0.0) / booksize_used_by_simulator,
|
||||||
|
)
|
||||||
|
|
||||||
|
metrics = evaluate_portfolio(positions, daily_bars)
|
||||||
|
_assert_metric_dict_is_finite(metrics)
|
||||||
|
|
||||||
|
|
||||||
|
def test_generated_workflow_outputs_keep_parquet_schema_contracts(tmp_path):
|
||||||
|
daily_bars = make_generated_daily_bars(n_sessions=10, include_missing=False)
|
||||||
|
alpha = compute_alpha(
|
||||||
|
data=daily_bars,
|
||||||
|
alpha_name="schema_reversal",
|
||||||
|
alpha_type="reversal",
|
||||||
|
lookback=3,
|
||||||
|
)
|
||||||
|
alpha_path = tmp_path / "schema_reversal.pq"
|
||||||
|
alpha.to_parquet(alpha_path, index=False)
|
||||||
|
|
||||||
|
combo = combine_alphas([str(alpha_path)], "schema_combo")
|
||||||
|
positions = construct_positions(
|
||||||
|
weights_df=combo,
|
||||||
|
data_df=daily_bars,
|
||||||
|
booksize=1_000_000.0,
|
||||||
|
portfolio_name="schema_portfolio",
|
||||||
|
)
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, daily_bars)
|
||||||
|
|
||||||
|
assert list(alpha.columns) == ALPHA_COLUMNS
|
||||||
|
assert pd.api.types.is_object_dtype(alpha["symbol_id"])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(alpha["date"])
|
||||||
|
assert pd.api.types.is_object_dtype(alpha["alpha_name"])
|
||||||
|
assert pd.api.types.is_float_dtype(alpha["weight"])
|
||||||
|
assert not alpha.isna().any().any()
|
||||||
|
assert np.isfinite(alpha["weight"]).all()
|
||||||
|
|
||||||
|
assert list(combo.columns) == COMBO_COLUMNS
|
||||||
|
assert pd.api.types.is_object_dtype(combo["symbol_id"])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(combo["date"])
|
||||||
|
assert pd.api.types.is_object_dtype(combo["combo_name"])
|
||||||
|
assert pd.api.types.is_float_dtype(combo["weight"])
|
||||||
|
assert not combo.isna().any().any()
|
||||||
|
assert np.isfinite(combo["weight"]).all()
|
||||||
|
|
||||||
|
assert list(positions.columns) == POSITION_COLUMNS
|
||||||
|
assert pd.api.types.is_integer_dtype(positions["position_shares"])
|
||||||
|
assert pd.api.types.is_datetime64_any_dtype(positions["date"])
|
||||||
|
assert not positions.isna().any().any()
|
||||||
|
position_numeric_columns = [
|
||||||
|
"target_weight",
|
||||||
|
"target_value",
|
||||||
|
"target_shares",
|
||||||
|
"position_value",
|
||||||
|
"price",
|
||||||
|
]
|
||||||
|
assert np.isfinite(positions[position_numeric_columns]).all().all()
|
||||||
|
|
||||||
|
assert list(fills.columns) == FILL_COLUMNS
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["prev_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["target_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["traded_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["realized_shares"])
|
||||||
|
assert pd.api.types.is_integer_dtype(fills["blocked"])
|
||||||
|
assert not fills.isna().any().any()
|
||||||
|
assert np.isfinite(fills["trade_cost"]).all()
|
||||||
|
|
||||||
|
assert list(pnl.columns) == PNL_COLUMNS
|
||||||
|
assert pd.api.types.is_integer_dtype(pnl["n_positions"])
|
||||||
|
assert not pnl.isna().any().any()
|
||||||
|
pnl_numeric_columns = [
|
||||||
|
"gross_exposure",
|
||||||
|
"net_exposure",
|
||||||
|
"pnl",
|
||||||
|
"cost",
|
||||||
|
"turnover",
|
||||||
|
]
|
||||||
|
assert np.isfinite(pnl[pnl_numeric_columns]).all().all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_frozen_real_fixture_runs_high_level_workflow(tmp_path):
|
||||||
|
real_daily_bars = pd.read_parquet(FIXTURE_PATH)
|
||||||
|
|
||||||
|
assert real_daily_bars.shape == (36, 19)
|
||||||
|
assert set(real_daily_bars["symbol_id"]) == set(GENERATED_SYMBOLS)
|
||||||
|
assert real_daily_bars["date"].min() == pd.Timestamp("2024-01-02")
|
||||||
|
assert real_daily_bars["date"].max() == pd.Timestamp("2024-01-12")
|
||||||
|
assert real_daily_bars.groupby("date")["symbol_id"].nunique().eq(4).all()
|
||||||
|
|
||||||
|
reversal_alpha = compute_alpha(
|
||||||
|
data=real_daily_bars,
|
||||||
|
alpha_name="real_reversal_3d",
|
||||||
|
alpha_type="reversal",
|
||||||
|
lookback=3,
|
||||||
|
)
|
||||||
|
reversal_vol_alpha = compute_alpha(
|
||||||
|
data=real_daily_bars,
|
||||||
|
alpha_name="real_reversal_vol_3d",
|
||||||
|
alpha_type="reversal_vol",
|
||||||
|
lookback=3,
|
||||||
|
vol_window=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
reversal_path = tmp_path / "real_reversal.pq"
|
||||||
|
reversal_vol_path = tmp_path / "real_reversal_vol.pq"
|
||||||
|
reversal_alpha.to_parquet(reversal_path, index=False)
|
||||||
|
reversal_vol_alpha.to_parquet(reversal_vol_path, index=False)
|
||||||
|
|
||||||
|
combo = combine_alphas([str(reversal_path), str(reversal_vol_path)], "real_equal_combo")
|
||||||
|
positions = construct_positions(
|
||||||
|
weights_df=combo,
|
||||||
|
data_df=real_daily_bars,
|
||||||
|
booksize=1_000_000.0,
|
||||||
|
portfolio_name="real_fixture_portfolio",
|
||||||
|
)
|
||||||
|
fills, pnl = ReferenceSimulator(cost_bps=5, slippage_bps=5).run(positions, real_daily_bars)
|
||||||
|
metrics = evaluate_portfolio(positions, real_daily_bars)
|
||||||
|
|
||||||
|
assert not reversal_alpha.empty
|
||||||
|
assert not reversal_vol_alpha.empty
|
||||||
|
assert not combo.empty
|
||||||
|
assert not positions.empty
|
||||||
|
assert not fills.empty
|
||||||
|
assert not pnl.empty
|
||||||
|
assert np.isfinite(combo["weight"]).all()
|
||||||
|
assert np.isfinite(positions["target_weight"]).all()
|
||||||
|
assert np.isfinite(pnl[["gross_exposure", "net_exposure", "pnl", "cost", "turnover"]]).all().all()
|
||||||
|
_assert_metric_dict_is_finite(metrics)
|
||||||
@@ -299,6 +299,7 @@ backtrader = [
|
|||||||
|
|
||||||
[package.dev-dependencies]
|
[package.dev-dependencies]
|
||||||
dev = [
|
dev = [
|
||||||
|
{ name = "coverage" },
|
||||||
{ name = "pytest" },
|
{ name = "pytest" },
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -315,7 +316,10 @@ requires-dist = [
|
|||||||
provides-extras = ["backtrader"]
|
provides-extras = ["backtrader"]
|
||||||
|
|
||||||
[package.metadata.requires-dev]
|
[package.metadata.requires-dev]
|
||||||
dev = [{ name = "pytest", specifier = ">=7.0.0" }]
|
dev = [
|
||||||
|
{ name = "coverage", specifier = ">=7.14.1" },
|
||||||
|
{ name = "pytest", specifier = ">=7.0.0" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "click"
|
name = "click"
|
||||||
@@ -498,6 +502,119 @@ wheels = [
|
|||||||
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" },
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0c/58/bd257695f39d05594ca4ad60df5bcb7e32247f9951fd09a9b8edb82d1daa/contourpy-1.3.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:3d1a3799d62d45c18bafd41c5fa05120b96a28079f2393af559b843d1a966a77", size = 225315, upload-time = "2025-07-26T12:02:58.801Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "coverage"
|
||||||
|
version = "7.14.1"
|
||||||
|
source = { registry = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple/" }
|
||||||
|
sdist = { url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/54/fd/0ab2772530e946e1be1abd0bc09e647ec9b02e88f0867857601fefca8953/coverage-7.14.1.tar.gz", hash = "sha256:30c08f7d90415aa98b3c990385dea2939b0da55f38515e5b369b83655f8523be", size = 920132, upload-time = "2026-05-26T20:41:36.783Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/92/69/0d2ef01ff4b8fcecd4cba920d11e92fa4f96ae412441d3b56a90a258e69b/coverage-7.14.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:3e3680291c4a1d0dadfa84a2c459576a4af5133abb617905714339a0c73138cf", size = 219722, upload-time = "2026-05-26T20:38:14.002Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f8/ae/9afdeaa31b9d9ce98124b6abf8bb49119bf71aecae04f8567c189d91299f/coverage-7.14.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a5274669f37f2343635a347b91a60777621341ab3378e9c6ac9335eee704bddf", size = 220240, upload-time = "2026-05-26T20:38:17.424Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/69/c998589871df7ea7dba865cc5ee32b5a3e1d47ba6c68ef91104c7c46fa5e/coverage-7.14.1-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cfe5a5fec635799ef33428f1e5e61bafa45a92a96190ba731561ba558ccc214d", size = 246981, upload-time = "2026-05-26T20:38:19.266Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/10/1c7d04c13040dac531d21b712bbe08f902e6dd9b58f5d77875c4d030f8f2/coverage-7.14.1-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:62a9f70b52e0b5a95cfef4a5c5641b06983cadc5e538a3feeb5c00211f523ac2", size = 248812, upload-time = "2026-05-26T20:38:20.75Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c1/65/2a38a4607ef27cadcfbcee034dba5830ae2569f90144a0f4c7dbf47d30b0/coverage-7.14.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3c18ebc343e15be53049b3a2dce38fe82d58f37e20ab9094b3a39c0aa4f6bb47", size = 250675, upload-time = "2026-05-26T20:38:22.159Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/a2/a446ed9752a4a59b79e0fb6cbb319f6facb2183045c0725462625e66f87e/coverage-7.14.1-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b84ffdf877644e7096aa936991efeed873f7f3df57b9cd001312b7668ab08550", size = 252590, upload-time = "2026-05-26T20:38:23.63Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9e/fd/e81fbd7ba752365546e9842b1cbdaad3d6919d2a522c590aef16a281ec5e/coverage-7.14.1-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e854312c4103f2ad4c0dc023b69b77ebfd2c89db5f86c4c94dc2353f9a92167e", size = 247691, upload-time = "2026-05-26T20:38:25.057Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/53/35/f3c26fdaae9ea937d154ca4d372e5ea0a4167ff70d36c6074ac2eacb2f83/coverage-7.14.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:c643734307300234fafa36bf2a040a7235f8f177ea1fd6ec1423aea6fb7b929f", size = 248716, upload-time = "2026-05-26T20:38:26.406Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2e/14/940b6c49551fd343e8507ee2b0ba7af5d0aa04ed5bf768285cb7c72a9884/coverage-7.14.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:84ac9499e48700399a5dd0ea7085b5091961fec52c68d66b4ec0d3cf7f4441b1", size = 246721, upload-time = "2026-05-26T20:38:28.282Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/aa/2c/40fc0634186c28292a662dff578866b3913983d6c375a3c2a74020938719/coverage-7.14.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:7f02d09f70776579b926d889a4c9c235070a1f47c40458aeaca563fae5acfdb5", size = 250533, upload-time = "2026-05-26T20:38:29.753Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/e3/2c26bf1e811f9df991ff2a9bdddebdd13ee0665d564df7d05979f9146297/coverage-7.14.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:ce66d8e46da2bb5ee313a745cbd2e391d319176c1f7a9451bfcd3a2fb920859b", size = 246990, upload-time = "2026-05-26T20:38:31.516Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a8/b0/060260ef56bd92363ebdce0c7095ce422b06e69aae71828efeca473ab1ca/coverage-7.14.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:c912c259304cfb5ee584481cfb7ce1ff932b4d61e6c9140b8f19cb7b5ed82332", size = 247593, upload-time = "2026-05-26T20:38:33.065Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/63/f3/501502046efeb0d6d94b5ca54941d95f1184183dd6bdb7f283985783bb4a/coverage-7.14.1-cp310-cp310-win32.whl", hash = "sha256:1238cb94638e610e972c60dac68e813f868dc7d6e982535270558443058d9d59", size = 222330, upload-time = "2026-05-26T20:38:35.36Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a0/5d/1bf99f2c558f128faf7906817ccbdb576ba815d3b41ce2ac1719b70a3663/coverage-7.14.1-cp310-cp310-win_amd64.whl", hash = "sha256:fc459e5d73be2d6332fcfe8dbf3d8994671fe33c700f4565988ecfa511547253", size = 223261, upload-time = "2026-05-26T20:38:37.196Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/d7/477ad149490e6cb849f28abea1dabb9c823cea72e7500c81b4240ce619c0/coverage-7.14.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:478b5bcd63c2e1357c5c7e16c070690df7b07f676b1c114d7b93e533c664309f", size = 219848, upload-time = "2026-05-26T20:38:38.715Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/91/82/a5eb47257c50601bb7b9a9d2857c67b7a3a85ad74180eb2c98bb1fbe0ce5/coverage-7.14.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a24a81f9715ee42ef59a316cc11611c98fe23920f7c81861315c9f3ff4a230f4", size = 220354, upload-time = "2026-05-26T20:38:40.232Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/8b/78419b5391a5cb706b6544390507e469d83ffc9a8248b02c4011aceb9365/coverage-7.14.1-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:196a13319ad88d6d8ef5ab489ec4f44ddde2143c0c7d5b27786f6c3ffd56a7e1", size = 250771, upload-time = "2026-05-26T20:38:41.782Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/77/63/e77aaacd491182210d639636b7a8bba23ffffa9b82aa3762da9431855fa9/coverage-7.14.1-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3d452fd08b5c72c5167c93e6867b5c08500bd40f2a21e1e854a500550b6cc36f", size = 252683, upload-time = "2026-05-26T20:38:43.305Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/65/1c/a022e3cfbec2ac241640003cb3a817e161d9c7f5aa9b49173756cdc03204/coverage-7.14.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:23bf7fa51ac02e07fc7c96849b82946da47ae862dc8f86d183b2a4864fc38129", size = 254791, upload-time = "2026-05-26T20:38:45.361Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/61/d6/967e408aca4c1ceb88cb0cc677169110ae7f5995fb5eaf5fb1f5a1bb8f5d/coverage-7.14.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bcaa50684dcaadfa599ac48f81103c756d791cfd85c97203d2217c593d48b860", size = 256748, upload-time = "2026-05-26T20:38:46.91Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/be/869188f7fe28638078ec479331ace6dc5f7b40b7153eb616f47ab79404d8/coverage-7.14.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4ea1c034f95c9b056e856b794630b17f9fa3d57e4800ff1e503d3be0f9c9078c", size = 250907, upload-time = "2026-05-26T20:38:48.493Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/07/aa/adb7d3b4278d690e68703abcd76ab1b948242e3668d921711551b78f9ddb/coverage-7.14.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c7e057326434e441306226fbeb5d1aaf14a2637efe97ba668306635835f32ad7", size = 252483, upload-time = "2026-05-26T20:38:50.074Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/43/61/331c74103c62dcb0c4b9b3a0de9a61aca016208b0a90f109592a9f9ecc28/coverage-7.14.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:59baf88468dbc8d63b1887afd92bda52e40bb1561696e5819670601403810cec", size = 250545, upload-time = "2026-05-26T20:38:51.613Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/b6/c5dae3c104d89be04828f61810e6b3473825482e4c288cc4ed04553e08ae/coverage-7.14.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:d34d75f892b3ab73ba11cab5442cce7b3e168fd64162b16f0e1e0d09c508edef", size = 254310, upload-time = "2026-05-26T20:38:53.503Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ad/a1/2b9d5863e3b83c01ad8199e3c597802fbb3a9dc90b058885804c20296d31/coverage-7.14.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:3a56abc20a472baf0304c455721bc601477440d28ecfde8a03dde79ede07e0df", size = 250266, upload-time = "2026-05-26T20:38:55.414Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/5e/0e511fbdb269359be26fe678a1c3fa1f2aa2a01573cc3f54268c8d6d4797/coverage-7.14.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:6a3cb83d1552c0cd1b4906655b6a33fd4a8473229633a901c6b73bf86914dee9", size = 251174, upload-time = "2026-05-26T20:38:57.141Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/85/10/e55307b622b3dd9671cb321824502dc10f93e72f2802b9946159a8edadeb/coverage-7.14.1-cp311-cp311-win32.whl", hash = "sha256:10274a1fbeb8ec5d72966e17bb198a3104257aca4ac09d98667c5f8aca8c8548", size = 222354, upload-time = "2026-05-26T20:38:58.727Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/cf/107421693cfb71e4f1ca5bf70443f64d4161878068d07a3e51c7ad21d17b/coverage-7.14.1-cp311-cp311-win_amd64.whl", hash = "sha256:87ebdf787d4888e3f3f2d523eadc6e18c6d18c6d0eb173801a189641627fb37e", size = 223290, upload-time = "2026-05-26T20:39:00.413Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b8/1d/3e3644585eb29e9dafefb19555078529a4d7cce12bd21929664eea989277/coverage-7.14.1-cp311-cp311-win_arm64.whl", hash = "sha256:dd34767fa19848d35659ffc0a75314f58c7af3f1cd87ec521e8292a1238398a3", size = 221953, upload-time = "2026-05-26T20:39:02.159Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/b7/bdbb725ba02c5b42825b200c940f38b7a54fcad24627b7192f78f8110d76/coverage-7.14.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a06c76364a9360e33d6d23769aefdf7f66f38e2ffb60ceb1baaa4989d83b695c", size = 220022, upload-time = "2026-05-26T20:39:03.702Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/72/81/fdc0898a55c6219223291ec1a1fe89966ef212ce82276aa0899df84b5de0/coverage-7.14.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fad54e871165f6ec2f536063ac74c3104508a12963e64072ba44bd822de52b0c", size = 220379, upload-time = "2026-05-26T20:39:05.381Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/de/72/de048c4a25e13bce59ac6a339351c10bdf2515e07459afcdaf04dc3143a2/coverage-7.14.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:84b535f00655ecafe1d929d1fb00ed5d6fa3051ea643ab2c161a3887b86f294b", size = 251888, upload-time = "2026-05-26T20:39:07.367Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/28/30/300c343f68beb9d4cbb64ec81e58c5b6b80b56927f72d2b38654ac26e013/coverage-7.14.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6b6b0853b895fe0e98cbfc580d1ec3393d9302b4b1e96a77b3f5c91fdab899e6", size = 254624, upload-time = "2026-05-26T20:39:09.037Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/b1/ed/7b25642496e8170b6bac14adce00537c6e5fa2d586159401a4de3e8b49e6/coverage-7.14.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:442cc9c952b2df400cda54bb04ab87330cf2cd08a8692cbbea36773531eb6f37", size = 255739, upload-time = "2026-05-26T20:39:10.889Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/a2/abd210b8c4e29c24e4624916db97bb519097a91034aaeb767f937e7da794/coverage-7.14.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8270544c361ed405a27a060dbc9ed2c124b084d96dfdc2d9a2510482aef981ad", size = 257998, upload-time = "2026-05-26T20:39:12.722Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7f/24/7c50beed3792fe62f6ce0545c6686ce83379719e2c0276179333d97eae92/coverage-7.14.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:48b283b1dd6372e8de2a7a9a4c4d5dc06f4d4fd209b876f3c88a7a205a0c8f84", size = 252296, upload-time = "2026-05-26T20:39:14.259Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/15/05/0f874628ebcbfc77ead559ff210281ef06a97db08481832e7dd39274a135/coverage-7.14.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5b0c99ba93a07d56f6df340bb79be53202a082b2fdb81bfe6190b741a3470d54", size = 253658, upload-time = "2026-05-26T20:39:15.923Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/99/6f/ca6ad067364b337ef997802115e7ecad2abd2248b05471464b0dea02b4d4/coverage-7.14.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:e471bc5769ff073b058cfadb0d736b56ce067c8560eabeb0da88462df98c23e7", size = 251803, upload-time = "2026-05-26T20:39:17.537Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c0/30/b9b4d377cd9f40baf228068f5a81faf8450c6228503011bd499708483a50/coverage-7.14.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f497a1ea81d4cd7c10ddcaa685135b9aabd291af3d55775a9ddf3cb7a364cdd9", size = 255873, upload-time = "2026-05-26T20:39:19.414Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3c/21/7c721a9e5e6bb88547d30a787aefb97512d3f54c1324c7488d9b3743f7f9/coverage-7.14.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2222be86d0b54f5dd5a38f45f17f315f737245e857bf0bdedc70734f84a13c02", size = 251372, upload-time = "2026-05-26T20:39:21.169Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/9d/8c/f8ae5a2200130e1503cd7661a6cd3b2b7bacef98277fbf3571fb13f8b766/coverage-7.14.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:85e85586565842f6932abebd4c18bcb1074223dc0b3576e7d173ca710622813a", size = 253245, upload-time = "2026-05-26T20:39:23.097Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/34/62/70a9024672a5f6910517d9628c52c9afbdd3cf8f46426af52bb148a56fff/coverage-7.14.1-cp312-cp312-win32.whl", hash = "sha256:4a28fd227808366b196a75476dced2eb35b351d6766ba9c858dc93319e87f4f1", size = 222567, upload-time = "2026-05-26T20:39:24.868Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/81/8b7cd386839b039ebe1855733b9f9449a8dec5d79564018234f185a7fa70/coverage-7.14.1-cp312-cp312-win_amd64.whl", hash = "sha256:54acdb6674a4661768d7bf7db32dfb9f46ab1d764f8aba6df75ce1a6a088724e", size = 223372, upload-time = "2026-05-26T20:39:26.603Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ae/ba/b44d472022f620d289d95fa830143235c0c36461c6f2437ea8d51e5481ed/coverage-7.14.1-cp312-cp312-win_arm64.whl", hash = "sha256:99cd41ff91afd94896fea3bc002706b6ae4ce95727d06e4a0f39c0a8d8bd8b1a", size = 221989, upload-time = "2026-05-26T20:39:28.242Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/9e/5f6d56327c62b185225d145191c607e07515294a0aa6338e58805cd4a5ac/coverage-7.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:be9f2c802dcfce3f71298303aa5dad0dce440a76c52f2f60dacd8656dab78793", size = 220044, upload-time = "2026-05-26T20:39:29.902Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/75/92/e82aca356744cbbc0f77a0b623e38918c1872361963413a3bab5d0340393/coverage-7.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6223a72fd0e4c7156353ec0f08a5f93623e1d3034d0e2683b9bb8ea674131b1d", size = 220412, upload-time = "2026-05-26T20:39:31.561Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/c9/385bde0bf7ed0f4bf3a7ee5367060a86b5d218718cfd6fb943c0f836b34f/coverage-7.14.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:7279d2110a28cebc738b6459ecda2771735a4c18465fbbd36b3288fe5ed92247", size = 251412, upload-time = "2026-05-26T20:39:33.337Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/51/8c/23faf6a2343a0d17f960a4bd56c43bc7eb4cf312f774dd6ceebd82c7d8fc/coverage-7.14.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9eeb3fcbc13ba40dfbdb22d01d196a28e9cef9ed4c29b60061a1e0e823a9929d", size = 254008, upload-time = "2026-05-26T20:39:35.009Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/42/06/36f4aa9ca8a815e6036156e80706a67828bb97bd826948244f6996dda957/coverage-7.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f0cfc27c539f07cf5c0a4cfe211d0b6cae039f8f40526dbaa71944e64b50a7b", size = 255241, upload-time = "2026-05-26T20:39:36.71Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/79/95266316352f90f6b1c6736bb413302edfde2453fb32422d3911642691b3/coverage-7.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:221c70f316241a78e77e607c227cefc8808d4e08f28d99c04f35694690e940be", size = 257373, upload-time = "2026-05-26T20:39:38.412Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e3/9c/58316d1f66c488b5fca8a0eb3e98348807813efa8a0d0833b9021be27488/coverage-7.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:da028256b04ec30e5e0114b6f76172938c313991f0a2d3d894271315cf5d5e43", size = 251635, upload-time = "2026-05-26T20:39:40.268Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ef/5a/ca2398a568e16fed7bb713e84ba3603a7164fb65779abe645c565ec890d5/coverage-7.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:76a085d7005236a767e3426148b2c407e53ad61695c562f8a81da2d373324901", size = 253373, upload-time = "2026-05-26T20:39:42.145Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6e/2c/0396562c32deaebe7be51d865b3a41e9a87d7561acafe1a28f53b07e019a/coverage-7.14.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b553d04b5e778a8e56d57eb134aff42a92718ecba45e79c4764ecfa40efd92ff", size = 251341, upload-time = "2026-05-26T20:39:43.907Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fd/8f/a94f9221184c9cae1ee115820e3798e48b6b17777a9f19e46fb9a0c8dc74/coverage-7.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:46f714d2fb8ae2f4f29f23ada7f1e79b759fff5a70f94a1dac23af204c3ec9e4", size = 255497, upload-time = "2026-05-26T20:39:46.166Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/69/505d70e47db1eaebcd002c39759707621ef184cd6b1ae084d9f41293f323/coverage-7.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:1896f5e19ff3f0431c7ce2172adc54890fd97f86b59ced8ca1649145d9ffe35d", size = 251159, upload-time = "2026-05-26T20:39:48.03Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e0/aa/58681c383aa33a9d2ed40a02d7a22fbf780d1fa4d575396365777828198c/coverage-7.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:62fd185ef9df3c33d1c8178c5af105f762afbad96038de9a4ae100aa6297ca33", size = 252934, upload-time = "2026-05-26T20:39:49.872Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/eb/fd/11c928cd6bdffc7074bb5965c173d9ebf517fb00205e1da524b98d29ef92/coverage-7.14.1-cp313-cp313-win32.whl", hash = "sha256:ab4af6352741a604c431c6072fce5bee33bf0f20dc7a56618d6bf6bb89e9810c", size = 222584, upload-time = "2026-05-26T20:39:51.68Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6f/92/fb416fc26d340dcba19518c418d6048e913186e17243982c5e435e41fa7a/coverage-7.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:7af486dabe8954d03b087f0021540897afe084f04e16ff5579e08cc46f871416", size = 223394, upload-time = "2026-05-26T20:39:53.472Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/73/c6/02d56e3867972f77d5036de924643f26c056e848f00452cafb4dbc3c29b4/coverage-7.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:2224f89ffd0c5605ccce1ed7a584da162bc7c55f601ab1c946bc9de31a486b42", size = 222015, upload-time = "2026-05-26T20:39:55.374Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/4d/9e/fcc77914050df73f7662fa1f00902774c79c075a8388ab334074574bf77e/coverage-7.14.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:de286598cc65d2b489411174b1faec2f5a7775fb3201fd925db2a76b4030f37d", size = 220733, upload-time = "2026-05-26T20:39:57.189Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f7/67/2963cbdaf5cbadec44efa3a1e39eaa1f02df4079585f05387607a221e126/coverage-7.14.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:042c46ded7c288aeb07cf14a28b6c1e10b78fcba40171c3fa1e939377eeef0b5", size = 221086, upload-time = "2026-05-26T20:39:59.019Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c8/c5/8701645574e11881f2f47d8930f98bc48b5d43b25eb5b4430dfc4a2f9f48/coverage-7.14.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:f4ddbe407477f04c45115d1a4e5bc480f753553b534d338d4c3358b1cdd0ea52", size = 262381, upload-time = "2026-05-26T20:40:00.822Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7c/28/7a64d73598263e0c5abd5084211a8474488d31b3c552ff531c719dfcff62/coverage-7.14.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d13e6725992e2d2fd7d81d4f5241952d13740121dfd501da09201be39b2c003a", size = 264458, upload-time = "2026-05-26T20:40:02.506Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fa/d8/4969179db9f7eb4df218e69540adf829d1c835f59452513d065d15446802/coverage-7.14.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f747dc8edcfe740130f28f32f3995e955494285717e86ee25af51db2219df08a", size = 266884, upload-time = "2026-05-26T20:40:04.421Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a6/78/a45d5794dbc9bafd97afc96a4377c86c7820d78b6cf51b89bc1d4e919275/coverage-7.14.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ced2f09ef276fd58611a1ef502164ad266d2b75174e5a40cabbdb4033f9f6cf2", size = 268022, upload-time = "2026-05-26T20:40:06.298Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/21/cb/4f5e354e9e3e67af96bd4e57113e6db6b22298c7168b13eec408a549903d/coverage-7.14.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b84800013769a78ccb9ef4659402e26d06867e337b61ec365f77ad008adea80e", size = 261631, upload-time = "2026-05-26T20:40:08.226Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ec/49/eced49af4cb996d5d8b7e94e736175c513e4facd3398507b89892b4326d8/coverage-7.14.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:ea8cd6ca0ee9f616aaef3afc6882e32c2cbf18b00d96313ffd76af650574034d", size = 264443, upload-time = "2026-05-26T20:40:10.137Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f1/d8/5603a88a7c5913a6b54f6cb1a8c46f7b39cbb30f27cd3f492908da09b2d7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:aa5e304a873fabddc11e484e9b6b738bd38bd7bed17b09aa84eecf5332e8b8bb", size = 262069, upload-time = "2026-05-26T20:40:11.999Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f0/59/2ae3cb79da554a06c8619d6c88ea19dd1e4aed4b834b6a83bb1fa243bdc5/coverage-7.14.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:5a1c5215be81035e629d5bc756650634d0bf31991038db7a0eccb90f025ce16d", size = 265780, upload-time = "2026-05-26T20:40:13.858Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/af/5f/b130c1dc999031f2648bd25317fbce505ad8d5562079b4ed81e736a84967/coverage-7.14.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:79058c47dae6788504b5effb319961bcd72d7240551464b91d474bc0ed186d69", size = 260970, upload-time = "2026-05-26T20:40:16.142Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/87/d1/ec13ccddeb48ec963bdfa72a11224bac2584bd045ba13beca82f8113e9c7/coverage-7.14.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:370c5afae3fa0658e11694a32b24c2778f6bc2d17718121f94ee185e69f26b54", size = 263157, upload-time = "2026-05-26T20:40:18.382Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/cf/c2/cd91ead503045161092d3845f7bb95ea2f25131ce96d3e314dd835d91b9c/coverage-7.14.1-cp313-cp313t-win32.whl", hash = "sha256:3758dd0a7f1fa57365ef2e781df0f0731d38b6e3772259d13dae4bd8a958d4b1", size = 223259, upload-time = "2026-05-26T20:40:20.381Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/71/9f/1e28d97e6bd2c76b07f38b7c02870f1371255ff6717f54eca578fcbbdd0e/coverage-7.14.1-cp313-cp313t-win_amd64.whl", hash = "sha256:6ff665fb023a77386fe11685190cee1f60a7d635994a30d9b0a061533d470fce", size = 224320, upload-time = "2026-05-26T20:40:22.316Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/a9/e0/d936e908f0e1efa55e52b91e01b52f1055cef5e1ab2718493390ed8e2fb8/coverage-7.14.1-cp313-cp313t-win_arm64.whl", hash = "sha256:17a5a241e5997621a956a7f402a7433ef4221e5152809b785bec79e2323799f1", size = 222577, upload-time = "2026-05-26T20:40:24.894Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d6/34/fc2f101b151af3799a101f0550b0454aa008afdc0add677394ec4aa8ea10/coverage-7.14.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d5ed429d0b8edaac649e889b4ffcedb6c80b06629a3f93050e3dddfb99235bee", size = 220091, upload-time = "2026-05-26T20:40:27.249Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/3d/a7/1ebae2ab5b961b5c79bb09fe7b3ac99edb190d8be4a8c510b2cf66f46468/coverage-7.14.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8011224a62280e50dab346960c03cf47aca1a1e09e608c0fb33fd6e0cc8e9500", size = 220421, upload-time = "2026-05-26T20:40:30.084Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/5e/90/92aca9cf0acc95123c96cd1eb1f08917897a7f5dee01e15738922971ec31/coverage-7.14.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:12c42ec1e14f553c4f817e989365982e646e27211f10a0f717855b94a79c8906", size = 251466, upload-time = "2026-05-26T20:40:32.542Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/26/2b/78048cbe3b999f6cbf9cc0d90abba6a88a3e0863a8c1c6cbc762f3f8802f/coverage-7.14.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:06144cd511cf2624873a035c5069cf297144f6e77a73ee3d7a55b605ec5efb42", size = 253973, upload-time = "2026-05-26T20:40:34.473Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/21/c2e33b29d1cfde484a19d437afc343c6cd30b08d78cbbf9f5aff14e57b2b/coverage-7.14.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a311d8e1da24be5c1ccf85cbfb06315dbaa1703d5a1eab3f6432c72b837917c8", size = 255318, upload-time = "2026-05-26T20:40:38.154Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8e/ee/aad2f108d63b769121005302f16bf66db8625c88ceaba466942e09a2607e/coverage-7.14.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c79cead5b5bc584d9c71451cb984d0e3a84e0c0937379c8efcbf27c8d661b851", size = 257633, upload-time = "2026-05-26T20:40:40.164Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c2/f8/11a2c29b4fd76d9849f81d0bb812ec0017a9396df3217214e38934a8c837/coverage-7.14.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:dcbf65f1f66a26cdd88c35cf68fb4729c5d1cd2e88added72420541dfb212034", size = 251488, upload-time = "2026-05-26T20:40:42.631Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/c9/b8/9a5820de4b8ac2b71d85e3b5fb49108d7469c665f0e2ad0dd7569023e305/coverage-7.14.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fd86572566fb40189a8260446158235159bc7a82dfbc87a3b39cf4fb57fcec1c", size = 253329, upload-time = "2026-05-26T20:40:45.208Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/6b/ff/f33e4823667e27548e8fd8df44217515303f9808d0ff29817db56f87d990/coverage-7.14.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:7771b601718fdde84832c3a434ca9bbf4ae9adbc49d84198b4110700c3c77c36", size = 251291, upload-time = "2026-05-26T20:40:47.502Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/68/9b/489db0ebb209054766b90a9014a45f6d26eb724c02ec21311c3733b5a644/coverage-7.14.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:39b21e212c55af06fa375e3dbf90a8a8e38792f3a910c580066d23563830ddd5", size = 255564, upload-time = "2026-05-26T20:40:49.372Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/27/b5/16bc2d4c2409b23c7737edb68c83bc89e345f378050549fe1d75ac7d34d5/coverage-7.14.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:f2302660e32562a532b442480121aef8aa61a5bdb20b30bf0adab29f10a5a4b4", size = 251107, upload-time = "2026-05-26T20:40:51.677Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/7d/0c/2629997469a00cd069d588a41c9dc887610f2775ae89d250c4791e65272a/coverage-7.14.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:03a6f93c1ec3b7f2e77b5dbcc5573a2c21f12529a5c6bbe0f16f72303cc2fa4d", size = 252764, upload-time = "2026-05-26T20:40:54.267Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d2/ee/f78d63c8f079e0d7211c7e2401fa17e311514534ba61bae03e4b287ce4ab/coverage-7.14.1-cp314-cp314-win32.whl", hash = "sha256:8a3ce026d73290f42f08dafecbd82c193a74df280461fbf97300fec51fd133ee", size = 222837, upload-time = "2026-05-26T20:40:56.496Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/dc/b9/be539854f93a70dfbeec69117f33ec70dc42ff0b65b5b07ab8d40d04228e/coverage-7.14.1-cp314-cp314-win_amd64.whl", hash = "sha256:114c95ef29302423b87d159075805f4ab973254a2638a5d7d046c94887cc87d7", size = 223650, upload-time = "2026-05-26T20:40:58.351Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fe/9e/24e2842fef40f35ac82ba3a7719c8023d011bf3bf652d0675316a9d088a1/coverage-7.14.1-cp314-cp314-win_arm64.whl", hash = "sha256:a07891c3f4805442b31b71e84ba3cf29ed1aa9a428284e06deeb4b23e5b46343", size = 222218, upload-time = "2026-05-26T20:41:00.321Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/0a/1d/ac0a9df5fe31c1e8bdd658074905fc12844a05c1a7e3fdb8417e97c31e23/coverage-7.14.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:1101a5ebb083aecb625ebb6209d4105b58f647b093cb2dc8122d7b33f743cfe1", size = 220822, upload-time = "2026-05-26T20:41:02.281Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/32/cf/f964fd9aff20323f9f1a726c97135f8a76bcd87b92dad141a456a43f3c64/coverage-7.14.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:851b9e1e4e8a4608e77c79714b2e77c0970d2ed7202a05e92ae407817481887b", size = 221084, upload-time = "2026-05-26T20:41:04.593Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/d8/5e/7e5ef2aba844de2b80d678619fcf0841b42e3f37f16411226f3fe4c1016f/coverage-7.14.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d5b89cdfb2ee051b71e8c3c70bd81a9eff81100f736a269136fe1a68efe00474", size = 262454, upload-time = "2026-05-26T20:41:06.641Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/64/62/75809bded87015cc4935524218a2a8ed8dd1a8498bfed30a2f4f7a4b4d34/coverage-7.14.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0177614a0370f227888b4e436a7c55686d6a9f90eb1ade2b624ba685a1686e86", size = 264578, upload-time = "2026-05-26T20:41:08.556Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f3/42/d33392dc14633525012d2d504fa1a33b05538bf535f5c1d64675e5754b78/coverage-7.14.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d69af5dea2de76fc485a83032a630523f985198b7e25be901ec60181587b01e", size = 266981, upload-time = "2026-05-26T20:41:10.824Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/2a/49/0157c4428c2aca7f1e09d5565930586fd5ae36f1655f08b0daa7cf1fcae1/coverage-7.14.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:35ab22d91de736e8966b980dc355cbcdd2c6dbbcfe275f9a2991bc8a91b3df65", size = 268112, upload-time = "2026-05-26T20:41:12.966Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/96/26/86b9ce71f4092b1ed325ce1421698081df1286b833400b6836912834d6e0/coverage-7.14.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:357d4e32935c36588aaba057d734fa32428c360c9fc2e4442afbf1b646beee6e", size = 261558, upload-time = "2026-05-26T20:41:15Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/20/4c/c311210c5472cf5401d8422b0d7812cdd520f24417673afabda6c323faca/coverage-7.14.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:51bd64741cc6fa065abd300ede1afe5a5291ece9c31da8b24884deda48bcc3f8", size = 264447, upload-time = "2026-05-26T20:41:17.369Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fb/71/59513f8710ed3e6b0ac0a050a5b7e977bb9c9e880354863b5d00d8809256/coverage-7.14.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:9132cd363a68a4c3daa7c8704a654b1e39d3360f6f5b8ddd470608a945236c07", size = 262048, upload-time = "2026-05-26T20:41:19.309Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/84/8d/bceed32dc494f5bbf50f775cd2e78ca814953942b5ea28d3c1c3ac316f14/coverage-7.14.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:07c6290b1697b862c0478eab545eec949a0d0e4d6d03497f446d706da3b4f2de", size = 265781, upload-time = "2026-05-26T20:41:21.559Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/e7/c5/9348fe40dbfd4991aaf78df2c6c3098bfb2cc834d1fd362a64b4efef855a/coverage-7.14.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:5ea0c297e27133853b4d8a3eb799bff5a2dbd9f2f41537a240d337ac9b4df890", size = 260896, upload-time = "2026-05-26T20:41:23.428Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/ca/92/1ea0f03929da7cf87206b1fa24f4c8e9c158be0455481af29ec0a1f3503f/coverage-7.14.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:01b7733daad0237daa01ef80fe2dfceffc911e6a17fa7b55d14aa8214eaaaecd", size = 263214, upload-time = "2026-05-26T20:41:25.419Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/f6/a9/b2493c054c0e01a643266742ab45e15744e60743f9260cd930c7142b1124/coverage-7.14.1-cp314-cp314t-win32.whl", hash = "sha256:6adc5a36984624a70bf11d7184e20fa0a49aa7c47ffab43804106a1a695ea22e", size = 223624, upload-time = "2026-05-26T20:41:27.795Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/fc/bd/3e1e6a57fccd2d7c83fcdf338e93ba98eb85c6e877dd34731ac585375490/coverage-7.14.1-cp314-cp314t-win_amd64.whl", hash = "sha256:ddf799247318f34dbcd2efa8c95a8d0642674e926bb1774cf9b63dfd2a389d1c", size = 224728, upload-time = "2026-05-26T20:41:30.098Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/bb/d7/31066cf1d2f0c6c797fce911bcfa01dd35642dc6da992a950256097c5860/coverage-7.14.1-cp314-cp314t-win_arm64.whl", hash = "sha256:145986fe66647eb489f18d9a997567a3fd358584c4b5a808769113abc07466af", size = 222752, upload-time = "2026-05-26T20:41:32.123Z" },
|
||||||
|
{ url = "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/packages/8a/3c/1a983b9a745d7f83d53f057bcc5bf79ba6a2bbc08266b3f0c7d6fe630c9b/coverage-7.14.1-py3-none-any.whl", hash = "sha256:a252f21c27e38347e60111a3266b03827422a7d5525951aceee313aa68bab1d2", size = 211815, upload-time = "2026-05-26T20:41:34.078Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "curl-cffi"
|
name = "curl-cffi"
|
||||||
version = "0.15.0"
|
version = "0.15.0"
|
||||||
|
|||||||
Reference in New Issue
Block a user