Add daily derived data pipeline

This commit is contained in:
Yuxuan Yan
2026-06-16 15:55:30 +08:00
parent 83a006bbe4
commit 8d908477e2
19 changed files with 897 additions and 231 deletions
+78
View File
@@ -120,6 +120,49 @@ partitions already written. Pass the **dataset directory** (`{output_dir}/{unive
as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned as `--data-path` to later phases — `pd.read_parquet` reads the whole partitioned
set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code). set. Symbols use the internal `sh600000` / `sz000001` form (exchange prefix + code).
### `derived` — daily custom/derived data
Derived data is daily-only v1 research data keyed by `symbol_id,date`, with one
or more numeric value columns. It can come from user CSV/parquet files or Python
plugins, and is written as a single parquet file at `derived/{name}.pq`.
The validator normalizes `date` to the trading day, requires unique
`symbol_id,date` keys, rejects duplicate columns, and rejects non-numeric value
columns. Alpha computation consumes derived data through the existing
`--feature-path` flag.
```bash
# Validate a user file without writing output.
uv run python cli.py derived validate --input-path vendor_factor.csv
# Ingest CSV/parquet into the canonical derived/ layout.
uv run python cli.py derived ingest \
--input-path vendor_factor.csv \
--derived-name vendor_factor
# List built-in and external derived-data plugin types.
uv run python cli.py derived list
uv run python cli.py derived list --derived-module path/to/my_derived.py
# Compute a derived file from daily and/or minute inputs.
uv run python cli.py derived compute \
--minute-path data/minute_bars/sh600000 \
--daily-path data/daily_bars/sh600000 \
--derived-type minute_daily_summary \
--derived-name minute_summary
# Join derived columns into a feature-aware alpha.
uv run python cli.py alpha compute \
--data-path data/daily_bars/sh600000 \
--feature-path derived/minute_summary.pq \
--alpha-type my_feature_aware_alpha \
--alpha-name my_run
```
For compatibility, `feature list` and `feature compute` remain available and
delegate to the same derived-data registry. Existing `features/*.pq` files are
still valid `--feature-path` inputs when they satisfy the daily numeric contract.
### `alpha list` — show registered alpha types ### `alpha list` — show registered alpha types
```bash ```bash
@@ -137,6 +180,7 @@ uv run python cli.py alpha list --alpha-module path/to/my_alpha.py # include a
| `--output-dir` | `alphas` | Output directory | | `--output-dir` | `alphas` | Output directory |
| `--lookback` | `5` | Lookback days (passed to alphas that accept it) | | `--lookback` | `5` | Lookback days (passed to alphas that accept it) |
| `--vol-window` | `20` | Volatility window (passed to alphas that accept it) | | `--vol-window` | `20` | Volatility window (passed to alphas that accept it) |
| `--feature-path` | — | Daily derived/feature parquet file or dataset to left-join on `symbol_id,date`; repeatable |
| `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file | | `--alpha-module` | — | External module(s) to import first; repeatable. Dotted path or `.py` file |
| `--param` | — | Extra constructor param as `name=value`; repeatable | | `--param` | — | Extra constructor param as `name=value`; repeatable |
@@ -370,6 +414,8 @@ between phases (data is stored long/tidy):
OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change, OHLC scale under `qfq`/`hfq`; `turn` is turnover %, `pctChg` daily % change,
`tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are `tradestatus`/`isST` are 0/1 flags, and `peTTM`/`pbMRQ`/`psTTM`/`pcfNcfTTM` are
baostock valuation ratios.) baostock valuation ratios.)
- **derived** (`DERIVED_KEY_COLUMNS` + values): required keys `symbol_id, date`;
value columns are user/plugin-defined and must be numeric in v1.
- **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight` - **alpha** (`ALPHA_COLUMNS`): `symbol_id, date, alpha_name, weight`
- **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight` - **combo** (`COMBO_COLUMNS`): `symbol_id, date, combo_name, weight`
- **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price` - **portfolio positions** (`POSITION_COLUMNS`): `symbol_id, date, portfolio_name, target_weight, target_value, target_shares, position_shares, position_value, price`
@@ -387,8 +433,11 @@ directory yields an extra `month` (`YYYY-MM`) partition column on top of
- `cli.py` — entry point wiring the file-based phases together - `cli.py` — entry point wiring the file-based phases together
- `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq` - `pipeline/data/` — universe resolution + download → `data/daily_bars/{universe}/month=YYYY-MM/*.pq`
- `pipeline/derived/` — daily derived-data ingestion, validation, plugin registry,
and built-in derived computations → `derived/*.pq`
- `pipeline/alpha/``base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader), - `pipeline/alpha/``base.py` (`BaseAlpha`), `registry.py` (factory + plugin loader),
`library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`) `library/` (built-in alphas), `compute.py` (`compute_alpha` / `evaluate_alpha`)
- `pipeline/features/` — compatibility wrappers for the derived-data registry
- `pipeline/combo/` — alpha combination → `combos/*.pq` - `pipeline/combo/` — alpha combination → `combos/*.pq`
- `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints, - `pipeline/portfolio/` — construction, A-share lot/limit rules, constraints,
reference next-open simulator, and research metrics reference next-open simulator, and research metrics
@@ -410,6 +459,9 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
- [x] **Reference execution simulation** — next-open fills over constructed - [x] **Reference execution simulation** — next-open fills over constructed
`position_shares`, with suspension, price-limit, volume-cap, transaction-cost, `position_shares`, with suspension, price-limit, volume-cap, transaction-cost,
and slippage controls. and slippage controls.
- [x] **Derived/custom daily data ("Level 2")** — ingest user CSV/parquet files
or compute plugin outputs as validated numeric daily datasets under
`derived/{name}.pq`; alpha joins continue through `--feature-path`.
- [ ] **Optional Backtrader adapter** — Backtrader is available as the - [ ] **Optional Backtrader adapter** — Backtrader is available as the
`backtrader` extra for possible future event-driven/broker-style experiments, `backtrader` extra for possible future event-driven/broker-style experiments,
but it is not part of the current canonical portfolio workflow. but it is not part of the current canonical portfolio workflow.
@@ -419,3 +471,29 @@ constructed positions, fills/costs, P&L, and target-weight research metrics.
and intraday VWAP. These need a tick / L1L2 quote feed (typically a paid or and intraday VWAP. These need a tick / L1L2 quote feed (typically a paid or
brokerage data tier); the free daily sources here only expose daily bars, so brokerage data tier); the free daily sources here only expose daily bars, so
this is a separate data phase rather than extra columns on the daily schema. this is a separate data phase rather than extra columns on the daily schema.
### Additional TODOs
The following items are intended extensions beyond the current daily
alpha-to-portfolio pipeline:
- **Long-only portfolio mode** — add a construction option that converts
alpha/combo weights into a long-only book while preserving existing lot,
price, suspension, and volume-cap handling.
- **Index-short hedging mode** — support portfolios that hold long A-share
names while shorting an index or index proxy for market exposure control.
- **Expanded universe presets** — add explicit universe aliases for CSI 300,
CSI 500, CSI 1000, and CSI 1800, while keeping file-based and comma-separated
custom universes available.
- **Categorical derived data** — extend the numeric-only derived-data v1 contract
to support categorical inputs such as industry classifications. In this
project, "Level 2" means customized second-level research data produced by
users or plugins; it does not necessarily mean exchange order-book/L2 quote
feeds.
- **Minute bar data** — continue extending the raw minute-bar and feature
workflow. The initial Baostock 5-minute download and daily feature plugin path
exist; intraday execution and replacing canonical daily bars remain out of
scope unless explicitly added later.
- **Industry data** — add industry classification inputs for filtering,
grouping, exposure reporting, neutralization, or industry-aware portfolio
construction.
+3
View File
@@ -3,6 +3,7 @@
Phases: Phases:
data — Download daily bars to parquet data — Download daily bars to parquet
derived — Ingest or compute daily derived data
alpha — Compute alpha weights from data alpha — Compute alpha weights from data
feature — Compute daily features from minute bars feature — Compute daily features from minute bars
combo — Combine alphas into a single weight combo — Combine alphas into a single weight
@@ -14,6 +15,7 @@ import logging
import click import click
from pipeline.data.cli import data from pipeline.data.cli import data
from pipeline.derived.cli import derived
from pipeline.alpha.cli import alpha from pipeline.alpha.cli import alpha
from pipeline.features.cli import feature from pipeline.features.cli import feature
from pipeline.combo.cli import combo from pipeline.combo.cli import combo
@@ -41,6 +43,7 @@ def cli(log_level):
cli.add_command(data) cli.add_command(data)
cli.add_command(derived)
cli.add_command(alpha) cli.add_command(alpha)
cli.add_command(feature) cli.add_command(feature)
cli.add_command(combo) cli.add_command(combo)
+7 -4
View File
@@ -16,17 +16,20 @@ The default layout is:
data/minute_bars/{universe}/frequency=5m/month=YYYY-MM/*.pq data/minute_bars/{universe}/frequency=5m/month=YYYY-MM/*.pq
``` ```
Feature plugins can aggregate those bars to daily `symbol_id,date` feature Derived-data plugins can aggregate those bars to daily `symbol_id,date` numeric
files, for example: files, for example:
```bash ```bash
uv run python cli.py feature compute \ uv run python cli.py derived compute \
--minute-path data/minute_bars/sh600000 \ --minute-path data/minute_bars/sh600000 \
--daily-path data/daily_bars/sh600000 \ --daily-path data/daily_bars/sh600000 \
--feature-type minute_daily_summary \ --derived-type minute_daily_summary \
--feature-name minute_summary --derived-name minute_summary
``` ```
The legacy `feature compute` command delegates to the same derived-data
registry and remains available for existing scripts.
## Daily vs Minute Reconciliation ## Daily vs Minute Reconciliation
Baostock's daily raw bars and 5-minute raw bars are close, but they should not Baostock's daily raw bars and 5-minute raw bars are close, but they should not
+1 -1
View File
@@ -58,7 +58,7 @@ def list_(alpha_modules):
@click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)") @click.option("--vol-window", default=20, type=int, help="Volatility window (reversal_vol only)")
@click.option( @click.option(
"--feature-path", "feature_paths", multiple=True, "--feature-path", "feature_paths", multiple=True,
help="Daily feature parquet file/dataset to left-join on symbol_id,date (repeatable)", help="Daily derived/feature parquet file or dataset to left-join on symbol_id,date (repeatable)",
) )
@click.option( @click.option(
"--alpha-module", "alpha_modules", multiple=True, "--alpha-module", "alpha_modules", multiple=True,
+10 -6
View File
@@ -15,7 +15,11 @@ import pandas as pd
from pipeline.alpha.registry import get_alpha from pipeline.alpha.registry import get_alpha
from pipeline.common.schema import ALPHA_COLUMNS from pipeline.common.schema import ALPHA_COLUMNS
from pipeline.features.compute import FEATURE_KEY_COLUMNS, read_feature_frames, validate_feature_frame from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
read_derived_frames,
validate_derived_frame,
)
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -40,15 +44,15 @@ def join_feature_frames(
data: pd.DataFrame, data: pd.DataFrame,
feature_frames: Iterable[pd.DataFrame], feature_frames: Iterable[pd.DataFrame],
) -> pd.DataFrame: ) -> pd.DataFrame:
"""Left-join validated daily feature frames onto long daily data.""" """Left-join validated daily derived/feature frames onto long daily data."""
out = data.copy() out = data.copy()
out["date"] = pd.to_datetime(out["date"]) out["date"] = pd.to_datetime(out["date"])
existing = set(out.columns) existing = set(out.columns)
joined_cols: list[str] = [] joined_cols: list[str] = []
for frame in feature_frames: for frame in feature_frames:
features = validate_feature_frame(frame) features = validate_derived_frame(frame)
feature_cols = [col for col in features.columns if col not in FEATURE_KEY_COLUMNS] feature_cols = [col for col in features.columns if col not in DERIVED_KEY_COLUMNS]
overlap = sorted(existing.intersection(feature_cols)) overlap = sorted(existing.intersection(feature_cols))
if overlap: if overlap:
raise ValueError( raise ValueError(
@@ -56,7 +60,7 @@ def join_feature_frames(
) )
out = out.merge( out = out.merge(
features, features,
on=FEATURE_KEY_COLUMNS, on=DERIVED_KEY_COLUMNS,
how="left", how="left",
validate="many_to_one", validate="many_to_one",
) )
@@ -171,7 +175,7 @@ def compute_alpha(
""" """
feature_inputs: list[pd.DataFrame] = [] feature_inputs: list[pd.DataFrame] = []
if feature_paths: if feature_paths:
feature_inputs.extend(read_feature_frames(feature_paths)) feature_inputs.extend(read_derived_frames(feature_paths))
if feature_frames: if feature_frames:
feature_inputs.extend(feature_frames) feature_inputs.extend(feature_frames)
if feature_inputs: if feature_inputs:
+7
View File
@@ -44,6 +44,13 @@ MINUTE_BAR_COLUMNS: Final[list[str]] = [
"adjustflag", # str: baostock adjustment flag; '3' for raw/unadjusted "adjustflag", # str: baostock adjustment flag; '3' for raw/unadjusted
] ]
# Required key columns for daily derived-data parquet files. Value columns are
# user/plugin-defined and must be numeric.
DERIVED_KEY_COLUMNS: Final[list[str]] = [
"symbol_id", # str
"date", # date: normalized daily timestamp
]
# Required columns for alpha parquet files. # Required columns for alpha parquet files.
# Alphas are position WEIGHTS: positive=long, negative=short. # Alphas are position WEIGHTS: positive=long, negative=short.
ALPHA_COLUMNS: Final[list[str]] = [ ALPHA_COLUMNS: Final[list[str]] = [
+2
View File
@@ -0,0 +1,2 @@
"""Daily derived-data plugin package."""
+38
View File
@@ -0,0 +1,38 @@
"""Base class for daily derived-data plugins."""
from abc import ABC, abstractmethod
import pandas as pd
class BaseDerivedData(ABC):
"""Compute daily, symbol-keyed numeric derived data.
Derived-data plugins may use daily bars, minute bars, or both as inputs, but
they must always return daily rows keyed by ``symbol_id,date``.
"""
#: Unique registry key. Every concrete derived-data plugin must set this.
name: str = ""
@abstractmethod
def compute(
self,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Compute daily derived data.
Args:
daily: Optional daily market data.
minute: Optional raw minute bars.
Returns:
DataFrame with ``symbol_id``, ``date``, and one or more numeric
derived-data columns.
"""
def __repr__(self) -> str:
params = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
return f"{type(self).__name__}({params})"
+145
View File
@@ -0,0 +1,145 @@
"""CLI for daily derived-data ingestion and computation."""
import click
import pandas as pd
from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
compute_derived,
read_derived_frame,
write_derived_frame,
)
from pipeline.derived.registry import (
available_derived,
load_derived_module,
)
@click.group(name="derived")
def derived():
"""Ingest, compute, and validate daily derived data."""
def _coerce(value: str):
"""Best-effort coercion of a CLI string to int, then float, else str."""
for cast in (int, float):
try:
return cast(value)
except ValueError:
continue
return value
def _parse_params(pairs: tuple[str, ...]) -> dict:
"""Parse repeated ``name=value`` options into a params dict."""
params: dict = {}
for pair in pairs:
if "=" not in pair:
raise click.BadParameter(f"--param must be name=value, got '{pair}'")
key, value = pair.split("=", 1)
params[key.strip()] = _coerce(value.strip())
return params
def _read_optional_parquet(path: str | None) -> pd.DataFrame | None:
return None if path is None else pd.read_parquet(path)
def _summarize(result: pd.DataFrame) -> str:
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
return f"{len(result):,} rows, {len(value_cols)} columns"
@derived.command("list")
@click.option(
"--derived-module", "derived_modules", multiple=True,
help="External module(s) to import first (dotted path or .py file)",
)
def list_(derived_modules):
"""List the registered derived-data plugin types."""
for spec in derived_modules:
load_derived_module(spec)
for name in available_derived():
click.echo(name)
@derived.command("validate")
@click.option("--input-path", required=True, help="CSV/parquet file or parquet dataset to validate")
def validate(input_path):
"""Validate a daily derived-data file without writing output."""
try:
result = read_derived_frame(input_path)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Valid derived data: {input_path} ({_summarize(result)})")
@derived.command("ingest")
@click.option("--input-path", required=True, help="CSV/parquet file to ingest")
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
def ingest(input_path, derived_name, output_dir):
"""Ingest a user-provided daily derived-data CSV/parquet file."""
try:
result = read_derived_frame(input_path)
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
@derived.command("compute")
@click.option("--daily-path", default=None, help="Optional daily data parquet/dataset")
@click.option("--minute-path", default=None, help="Optional minute parquet/dataset")
@click.option("--derived-type", required=True, help="Registry key of the derived-data plugin")
@click.option("--derived-name", required=True, help="Name for this derived-data output file")
@click.option("--output-dir", default="derived", help="Directory to save derived parquet")
@click.option(
"--derived-module", "derived_modules", multiple=True,
help="External module(s) to import so their derived-data plugins register",
)
@click.option(
"--param", "extra_params", multiple=True,
help="Extra derived-data constructor param as name=value (repeatable)",
)
def compute(
daily_path,
minute_path,
derived_type,
derived_name,
output_dir,
derived_modules,
extra_params,
):
"""Compute one daily derived-data file from daily and/or minute inputs."""
for spec in derived_modules:
load_derived_module(spec)
options = available_derived()
if derived_type not in options:
raise click.BadParameter(
f"Unknown derived-type '{derived_type}'. Available: {options}. "
f"Use --derived-module to register an external derived-data plugin.",
param_hint="--derived-type",
)
if daily_path is None and minute_path is None:
raise click.UsageError("At least one of --daily-path or --minute-path is required")
daily = _read_optional_parquet(daily_path)
if daily_path:
click.echo(f"Loaded daily data: {len(daily):,} rows from {daily_path}")
minute = _read_optional_parquet(minute_path)
if minute_path:
click.echo(f"Loaded minute bars: {len(minute):,} rows from {minute_path}")
try:
result = compute_derived(
derived_type=derived_type,
daily=daily,
minute=minute,
**_parse_params(extra_params),
)
out_path = write_derived_frame(result, derived_name, output_dir=output_dir)
except Exception as exc:
raise click.ClickException(str(exc)) from exc
click.echo(f"Saved derived data: {out_path} ({_summarize(result)})")
+115
View File
@@ -0,0 +1,115 @@
"""Derived-data computation and validation."""
import csv
import logging
from pathlib import Path
from typing import Iterable
import pandas as pd
from pandas.api.types import is_bool_dtype, is_numeric_dtype
from pipeline.common.schema import DERIVED_KEY_COLUMNS
from pipeline.derived.registry import get_derived
logger = logging.getLogger(__name__)
def validate_derived_frame(derived: pd.DataFrame) -> pd.DataFrame:
"""Validate and normalize a daily derived-data frame.
A valid derived frame is keyed by unique ``symbol_id,date`` rows and has at
least one numeric value column beyond those keys. Dates are normalized to
daily timestamps before duplicate-key checks.
"""
duplicated = derived.columns[derived.columns.duplicated()].tolist()
if duplicated:
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
missing = [col for col in DERIVED_KEY_COLUMNS if col not in derived.columns]
if missing:
raise ValueError(f"Derived data missing required columns: {missing}")
out = derived.copy()
out["date"] = pd.to_datetime(out["date"]).dt.normalize()
if out.duplicated(DERIVED_KEY_COLUMNS).any():
raise ValueError("Derived data has duplicate symbol_id,date rows")
value_cols = [col for col in out.columns if col not in DERIVED_KEY_COLUMNS]
if not value_cols:
raise ValueError("Derived data must include at least one value column")
non_numeric = [
col
for col in value_cols
if is_bool_dtype(out[col]) or not is_numeric_dtype(out[col])
]
if non_numeric:
raise ValueError(f"Derived data value columns must be numeric: {non_numeric}")
out = out[DERIVED_KEY_COLUMNS + value_cols].copy()
return out.sort_values(DERIVED_KEY_COLUMNS).reset_index(drop=True)
def compute_derived(
derived_type: str,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
**params,
) -> pd.DataFrame:
"""Compute one registered derived-data plugin."""
if daily is None and minute is None:
raise ValueError("Derived data computation requires --daily-path or --minute-path")
derived = get_derived(derived_type, **params)
result = validate_derived_frame(derived.compute(daily=daily, minute=minute))
value_cols = [col for col in result.columns if col not in DERIVED_KEY_COLUMNS]
logger.info(
"Derived data '%s' (%r): %d symbols × %d dates, columns=%s",
derived_type,
derived,
result["symbol_id"].nunique(),
result["date"].nunique(),
value_cols,
)
return result
def read_derived_frame(path: str | Path) -> pd.DataFrame:
"""Read and validate one derived CSV/parquet file or parquet dataset."""
path = Path(path)
if path.suffix.lower() == ".csv":
return validate_derived_frame(_read_csv_with_duplicate_header_check(path))
return validate_derived_frame(pd.read_parquet(path))
def read_derived_frames(derived_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
"""Read and validate derived-data files."""
return [read_derived_frame(path) for path in derived_paths]
def write_derived_frame(
derived: pd.DataFrame,
derived_name: str,
output_dir: str | Path = "derived",
) -> Path:
"""Validate and write derived data to ``{output_dir}/{derived_name}.pq``."""
result = validate_derived_frame(derived)
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
out_path = output_dir / f"{derived_name}.pq"
result.to_parquet(out_path, index=False)
return out_path
def _read_csv_with_duplicate_header_check(path: Path) -> pd.DataFrame:
with path.open(newline="") as fh:
reader = csv.reader(fh)
try:
header = next(reader)
except StopIteration as exc:
raise ValueError("CSV input is empty") from exc
duplicated = sorted({col for col in header if header.count(col) > 1})
if duplicated:
raise ValueError(f"Derived data has duplicate columns: {duplicated}")
return pd.read_csv(path)
+4
View File
@@ -0,0 +1,4 @@
"""Built-in derived-data library."""
from pipeline.derived.library import minute_daily_summary # noqa: F401
@@ -0,0 +1,88 @@
"""Daily summary data derived from raw minute bars."""
import numpy as np
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class MinuteDailySummaryDerived(BaseDerivedData):
"""Aggregate intraday bars into daily summary columns."""
name = "minute_daily_summary"
def compute(
self,
daily: pd.DataFrame | None = None,
minute: pd.DataFrame | None = None,
) -> pd.DataFrame:
if minute is None:
raise ValueError("minute_daily_summary requires minute input")
minute = minute.copy()
minute["date"] = pd.to_datetime(minute["date"]).dt.normalize()
sort_cols = ["symbol_id", "date"]
if "datetime" in minute.columns:
minute["datetime"] = pd.to_datetime(minute["datetime"])
sort_cols.append("datetime")
elif "time" in minute.columns:
sort_cols.append("time")
minute = minute.sort_values(sort_cols)
grouped = minute.groupby(["symbol_id", "date"], sort=True)
summary = grouped.agg(
minute_bar_count=("close", "count"),
first_open=("open", "first"),
last_close=("close", "last"),
high=("high", "max"),
low=("low", "min"),
volume_sum=("volume", "sum"),
amount_sum=("amount", "sum"),
)
summary["minute_intraday_return"] = (
summary["last_close"] / summary["first_open"] - 1.0
)
summary["minute_intraday_range"] = summary["high"] / summary["low"] - 1.0
summary["minute_vwap"] = (
summary["amount_sum"] / summary["volume_sum"].where(summary["volume_sum"] > 0)
)
summary = summary.reset_index()
if daily is not None:
daily_keys = daily[["symbol_id", "date"]].copy()
daily_keys["date"] = pd.to_datetime(daily_keys["date"]).dt.normalize()
daily_keys = daily_keys.drop_duplicates(["symbol_id", "date"])
result = daily_keys.merge(summary, on=["symbol_id", "date"], how="left")
if "close" in daily.columns:
daily_close = daily[["symbol_id", "date", "close"]].copy()
daily_close["date"] = pd.to_datetime(daily_close["date"]).dt.normalize()
daily_close = daily_close.drop_duplicates(["symbol_id", "date"])
result = result.merge(
daily_close.rename(columns={"close": "daily_close"}),
on=["symbol_id", "date"],
how="left",
)
reference_close = result["daily_close"].fillna(result["last_close"])
else:
reference_close = result["last_close"]
else:
result = summary
reference_close = result["last_close"]
result["minute_vwap_deviation"] = (
result["minute_vwap"] / reference_close.replace(0.0, np.nan) - 1.0
)
return result[
[
"symbol_id",
"date",
"minute_bar_count",
"minute_intraday_return",
"minute_intraday_range",
"minute_vwap",
"minute_vwap_deviation",
]
]
+80
View File
@@ -0,0 +1,80 @@
"""Registry and factory for daily derived-data plugins."""
import importlib
import importlib.util
import inspect
from pathlib import Path
from typing import Optional, Type
from pipeline.derived.base import BaseDerivedData
_REGISTRY: dict[str, Type[BaseDerivedData]] = {}
_builtins_loaded = False
def register_derived(cls: Type[BaseDerivedData]) -> Type[BaseDerivedData]:
"""Class decorator that registers derived data under ``BaseDerivedData.name``."""
if not (isinstance(cls, type) and issubclass(cls, BaseDerivedData)):
raise TypeError(f"{cls!r} is not a BaseDerivedData subclass")
key = getattr(cls, "name", "")
if not key:
raise ValueError(f"{cls.__name__} must set a non-empty class attribute `name`")
existing = _REGISTRY.get(key)
if existing is not None and existing is not cls:
raise ValueError(
f"Derived data name '{key}' already registered by {existing.__name__}"
)
_REGISTRY[key] = cls
return cls
def available_derived() -> list[str]:
"""Sorted names of all registered derived-data plugins."""
_ensure_builtins()
return sorted(_REGISTRY)
def get_derived(name: str, **params) -> BaseDerivedData:
"""Instantiate a registered derived-data plugin by name.
Only parameters accepted by the plugin class's ``__init__`` are forwarded.
"""
_ensure_builtins()
if name not in _REGISTRY:
raise KeyError(f"Unknown derived data '{name}'. Available: {sorted(_REGISTRY)}")
cls = _REGISTRY[name]
accepted = _accepted_params(cls)
kwargs = params if accepted is None else {k: v for k, v in params.items() if k in accepted}
return cls(**kwargs)
def load_derived_module(spec: str) -> None:
"""Import an external module so its ``@register_derived`` classes register."""
looks_like_file = spec.endswith(".py") or Path(spec).expanduser().exists()
if looks_like_file:
path = Path(spec).expanduser().resolve()
if not path.exists():
raise FileNotFoundError(f"Derived data module not found: {path}")
module_spec = importlib.util.spec_from_file_location(path.stem, path)
if module_spec is None or module_spec.loader is None:
raise ImportError(f"Cannot load derived data module from {path}")
module = importlib.util.module_from_spec(module_spec)
module_spec.loader.exec_module(module)
else:
importlib.import_module(spec)
def _accepted_params(cls: Type[BaseDerivedData]) -> Optional[set[str]]:
"""Param names ``cls.__init__`` accepts, or None if it takes ``**kwargs``."""
sig = inspect.signature(cls.__init__)
if any(p.kind is p.VAR_KEYWORD for p in sig.parameters.values()):
return None
return {name for name in sig.parameters if name != "self"}
def _ensure_builtins() -> None:
global _builtins_loaded
if not _builtins_loaded:
import pipeline.derived.library # noqa: F401
_builtins_loaded = True
+5 -31
View File
@@ -1,34 +1,8 @@
"""Base class for daily feature plugins.""" """Compatibility alias for daily feature plugins.
from abc import ABC, abstractmethod The canonical plugin API is ``pipeline.derived``. ``BaseFeature`` remains as an
alias so existing external feature modules continue to register unchanged.
import pandas as pd
class BaseFeature(ABC):
"""Aggregate raw minute bars into daily, symbol-keyed feature columns."""
#: Unique registry key. Every concrete feature must set this to a non-empty str.
name: str = ""
@abstractmethod
def compute(
self,
minute: pd.DataFrame,
daily: pd.DataFrame | None = None,
) -> pd.DataFrame:
"""Compute daily features.
Args:
minute: Raw minute bars with ``symbol_id`` and ``date`` keys.
daily: Optional daily data frame for calendar alignment or
reference daily columns.
Returns:
DataFrame with ``symbol_id``, ``date``, and one or more numeric
feature columns.
""" """
def __repr__(self) -> str: from pipeline.derived.base import BaseDerivedData as BaseFeature
params = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
return f"{type(self).__name__}({params})"
+19 -53
View File
@@ -1,49 +1,23 @@
"""Feature computation and validation.""" """Compatibility wrappers for daily feature computation and validation."""
import logging
from pathlib import Path from pathlib import Path
from typing import Iterable from typing import Iterable
import pandas as pd import pandas as pd
from pandas.api.types import is_numeric_dtype
from pipeline.features.registry import get_feature from pipeline.derived.compute import (
DERIVED_KEY_COLUMNS,
compute_derived,
read_derived_frames,
validate_derived_frame,
)
logger = logging.getLogger(__name__) FEATURE_KEY_COLUMNS = DERIVED_KEY_COLUMNS
FEATURE_KEY_COLUMNS = ["symbol_id", "date"]
def validate_feature_frame(features: pd.DataFrame) -> pd.DataFrame: def validate_feature_frame(features: pd.DataFrame) -> pd.DataFrame:
"""Validate and normalize a daily feature frame. """Validate and normalize a legacy daily feature frame."""
return validate_derived_frame(features)
A valid feature frame is keyed by unique ``symbol_id,date`` rows and has at
least one numeric feature column beyond those keys.
"""
duplicated = features.columns[features.columns.duplicated()].tolist()
if duplicated:
raise ValueError(f"Feature output has duplicate columns: {duplicated}")
missing = [col for col in FEATURE_KEY_COLUMNS if col not in features.columns]
if missing:
raise ValueError(f"Feature output missing required columns: {missing}")
out = features.copy()
out["date"] = pd.to_datetime(out["date"])
if out.duplicated(FEATURE_KEY_COLUMNS).any():
raise ValueError("Feature output has duplicate symbol_id,date rows")
feature_cols = [col for col in out.columns if col not in FEATURE_KEY_COLUMNS]
if not feature_cols:
raise ValueError("Feature output must include at least one feature column")
non_numeric = [col for col in feature_cols if not is_numeric_dtype(out[col])]
if non_numeric:
raise ValueError(f"Feature columns must be numeric: {non_numeric}")
out = out[FEATURE_KEY_COLUMNS + feature_cols].copy()
return out.sort_values(FEATURE_KEY_COLUMNS).reset_index(drop=True)
def compute_feature( def compute_feature(
@@ -52,24 +26,16 @@ def compute_feature(
daily: pd.DataFrame | None = None, daily: pd.DataFrame | None = None,
**params, **params,
) -> pd.DataFrame: ) -> pd.DataFrame:
"""Compute one registered feature from raw minute bars.""" """Compute one registered feature through the derived-data registry."""
feature = get_feature(feature_type, **params) return compute_derived(
result = validate_feature_frame(feature.compute(minute=minute, daily=daily)) derived_type=feature_type,
feature_cols = [col for col in result.columns if col not in FEATURE_KEY_COLUMNS] daily=daily,
logger.info( minute=minute,
"Feature '%s' (%r): %d symbols × %d dates, columns=%s", **params,
feature_type,
feature,
result["symbol_id"].nunique(),
result["date"].nunique(),
feature_cols,
) )
return result
def read_feature_frames(feature_paths: Iterable[str | Path]) -> list[pd.DataFrame]: def read_feature_frames(feature_paths: Iterable[str | Path]) -> list[pd.DataFrame]:
"""Read and validate feature parquet files.""" """Read and validate feature/derived-data parquet files."""
return [ return read_derived_frames(feature_paths)
validate_feature_frame(pd.read_parquet(path))
for path in feature_paths
]
@@ -1,84 +1,16 @@
"""Daily summary features derived from raw minute bars.""" """Compatibility wrapper for the built-in minute daily summary plugin."""
import numpy as np
import pandas as pd import pandas as pd
from pipeline.features.base import BaseFeature from pipeline.derived.library.minute_daily_summary import MinuteDailySummaryDerived
from pipeline.features.registry import register_feature
@register_feature class MinuteDailySummaryFeature(MinuteDailySummaryDerived):
class MinuteDailySummaryFeature(BaseFeature): """Legacy minute-first wrapper around the derived-data implementation."""
"""Aggregate intraday bars into daily summary columns."""
name = "minute_daily_summary"
def compute( def compute(
self, self,
minute: pd.DataFrame, minute: pd.DataFrame,
daily: pd.DataFrame | None = None, daily: pd.DataFrame | None = None,
) -> pd.DataFrame: ) -> pd.DataFrame:
minute = minute.copy() return super().compute(daily=daily, minute=minute)
minute["date"] = pd.to_datetime(minute["date"])
sort_cols = ["symbol_id", "date"]
if "datetime" in minute.columns:
minute["datetime"] = pd.to_datetime(minute["datetime"])
sort_cols.append("datetime")
elif "time" in minute.columns:
sort_cols.append("time")
minute = minute.sort_values(sort_cols)
grouped = minute.groupby(["symbol_id", "date"], sort=True)
summary = grouped.agg(
minute_bar_count=("close", "count"),
first_open=("open", "first"),
last_close=("close", "last"),
high=("high", "max"),
low=("low", "min"),
volume_sum=("volume", "sum"),
amount_sum=("amount", "sum"),
)
summary["minute_intraday_return"] = (
summary["last_close"] / summary["first_open"] - 1.0
)
summary["minute_intraday_range"] = summary["high"] / summary["low"] - 1.0
summary["minute_vwap"] = (
summary["amount_sum"] / summary["volume_sum"].where(summary["volume_sum"] > 0)
)
summary = summary.reset_index()
if daily is not None:
daily_keys = daily[["symbol_id", "date"]].copy()
daily_keys["date"] = pd.to_datetime(daily_keys["date"])
daily_keys = daily_keys.drop_duplicates(["symbol_id", "date"])
result = daily_keys.merge(summary, on=["symbol_id", "date"], how="left")
if "close" in daily.columns:
daily_close = daily[["symbol_id", "date", "close"]].copy()
daily_close["date"] = pd.to_datetime(daily_close["date"])
daily_close = daily_close.drop_duplicates(["symbol_id", "date"])
result = result.merge(
daily_close.rename(columns={"close": "daily_close"}),
on=["symbol_id", "date"],
how="left",
)
reference_close = result["daily_close"].fillna(result["last_close"])
else:
reference_close = result["last_close"]
else:
result = summary
reference_close = result["last_close"]
result["minute_vwap_deviation"] = (
result["minute_vwap"] / reference_close.replace(0.0, np.nan) - 1.0
)
return result[
[
"symbol_id",
"date",
"minute_bar_count",
"minute_intraday_return",
"minute_intraday_range",
"minute_vwap",
"minute_vwap_deviation",
]
]
+20 -63
View File
@@ -1,79 +1,36 @@
"""Registry and factory for daily feature plugins.""" """Compatibility registry wrappers for daily feature plugins."""
import importlib from typing import Type
import importlib.util
import inspect
from pathlib import Path
from typing import Optional, Type
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import (
available_derived,
get_derived,
load_derived_module,
register_derived,
)
from pipeline.features.base import BaseFeature from pipeline.features.base import BaseFeature
_REGISTRY: dict[str, Type[BaseFeature]] = {}
_builtins_loaded = False
def register_feature(cls: Type[BaseFeature]) -> Type[BaseFeature]: def register_feature(cls: Type[BaseFeature]) -> Type[BaseFeature]:
"""Class decorator that registers a feature under ``BaseFeature.name``.""" """Register a legacy feature plugin in the derived-data registry."""
if not (isinstance(cls, type) and issubclass(cls, BaseFeature)): return register_derived(cls)
raise TypeError(f"{cls!r} is not a BaseFeature subclass")
key = getattr(cls, "name", "")
if not key:
raise ValueError(f"{cls.__name__} must set a non-empty class attribute `name`")
existing = _REGISTRY.get(key)
if existing is not None and existing is not cls:
raise ValueError(
f"Feature name '{key}' already registered by {existing.__name__}"
)
_REGISTRY[key] = cls
return cls
def available_features() -> list[str]: def available_features() -> list[str]:
"""Sorted names of all registered features (built-ins are loaded lazily).""" """Sorted names of all registered feature/derived-data plugins."""
_ensure_builtins() return available_derived()
return sorted(_REGISTRY)
def get_feature(name: str, **params) -> BaseFeature: def get_feature(name: str, **params) -> BaseDerivedData:
"""Instantiate a registered feature by name. """Instantiate a registered feature/derived-data plugin by name."""
if name == "minute_daily_summary":
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
Only parameters accepted by the feature class's ``__init__`` are forwarded. return MinuteDailySummaryFeature(**params)
""" return get_derived(name, **params)
_ensure_builtins()
if name not in _REGISTRY:
raise KeyError(f"Unknown feature '{name}'. Available: {sorted(_REGISTRY)}")
cls = _REGISTRY[name]
accepted = _accepted_params(cls)
kwargs = params if accepted is None else {k: v for k, v in params.items() if k in accepted}
return cls(**kwargs)
def load_feature_module(spec: str) -> None: def load_feature_module(spec: str) -> None:
"""Import an external module so its ``@register_feature`` classes register.""" """Import an external module so its ``@register_feature`` classes register."""
looks_like_file = spec.endswith(".py") or Path(spec).expanduser().exists() load_derived_module(spec)
if looks_like_file:
path = Path(spec).expanduser().resolve()
if not path.exists():
raise FileNotFoundError(f"Feature module not found: {path}")
module_spec = importlib.util.spec_from_file_location(path.stem, path)
if module_spec is None or module_spec.loader is None:
raise ImportError(f"Cannot load feature module from {path}")
module = importlib.util.module_from_spec(module_spec)
module_spec.loader.exec_module(module)
else:
importlib.import_module(spec)
def _accepted_params(cls: Type[BaseFeature]) -> Optional[set[str]]:
"""Param names ``cls.__init__`` accepts, or None if it takes ``**kwargs``."""
sig = inspect.signature(cls.__init__)
if any(p.kind is p.VAR_KEYWORD for p in sig.parameters.values()):
return None
return {name for name in sig.parameters if name != "self"}
def _ensure_builtins() -> None:
global _builtins_loaded
if not _builtins_loaded:
import pipeline.features.library # noqa: F401
_builtins_loaded = True
+261
View File
@@ -0,0 +1,261 @@
"""Tests for daily derived-data ingestion and plugins."""
import textwrap
import numpy as np
import pandas as pd
import pytest
from click.testing import CliRunner
from cli import cli
from pipeline.alpha.compute import join_feature_frames
from pipeline.derived.compute import compute_derived, validate_derived_frame
from pipeline.derived.registry import available_derived, get_derived, load_derived_module
def _daily_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sz000001", "sh600000"],
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-03"]),
"open": [10.0, 20.0, 11.0],
"close": [10.5, 20.5, 11.5],
"volume": [1000.0, 2000.0, 1200.0],
})
def _minute_bars() -> pd.DataFrame:
return pd.DataFrame({
"symbol_id": ["sh600000", "sh600000", "sz000001"],
"datetime": pd.to_datetime([
"2024-01-02 09:35:00",
"2024-01-02 09:40:00",
"2024-01-02 09:35:00",
]),
"date": pd.to_datetime(["2024-01-02", "2024-01-02", "2024-01-02"]),
"time": ["09:35:00", "09:40:00", "09:35:00"],
"open": [10.0, 10.5, 20.0],
"high": [11.0, 12.0, 21.0],
"low": [9.0, 10.0, 19.0],
"close": [10.5, 11.0, 20.5],
"volume": [100.0, 300.0, 200.0],
"amount": [1000.0, 3300.0, 4100.0],
})
def test_validate_derived_frame_normalizes_and_sorts():
result = validate_derived_frame(pd.DataFrame({
"symbol_id": ["sz000001", "sh600000"],
"date": ["2024-01-02 15:00:00", "2024-01-02 09:30:00"],
"custom_value": [2.0, 1.0],
}))
assert result["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert result["date"].tolist() == [
pd.Timestamp("2024-01-02"),
pd.Timestamp("2024-01-02"),
]
def test_validate_derived_frame_rejects_missing_keys():
with pytest.raises(ValueError, match="missing required"):
validate_derived_frame(pd.DataFrame({"symbol_id": ["sh600000"], "x": [1.0]}))
def test_validate_derived_frame_rejects_duplicate_normalized_keys():
with pytest.raises(ValueError, match="duplicate symbol_id,date"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000", "sh600000"],
"date": ["2024-01-02 09:30:00", "2024-01-02 15:00:00"],
"x": [1.0, 2.0],
}))
def test_validate_derived_frame_rejects_duplicate_columns():
bad = pd.DataFrame(
[["sh600000", pd.Timestamp("2024-01-02"), 1.0, 2.0]],
columns=["symbol_id", "date", "dup", "dup"],
)
with pytest.raises(ValueError, match="duplicate columns"):
validate_derived_frame(bad)
def test_validate_derived_frame_rejects_non_numeric_values():
with pytest.raises(ValueError, match="numeric"):
validate_derived_frame(pd.DataFrame({
"symbol_id": ["sh600000"],
"date": [pd.Timestamp("2024-01-02")],
"bad": ["not numeric"],
}))
def test_derived_ingest_cli_accepts_csv_and_parquet(tmp_path):
runner = CliRunner()
source = pd.DataFrame({
"symbol_id": ["sz000001", "sh600000"],
"date": ["2024-01-02", "2024-01-02"],
"custom_value": [2.0, 1.0],
})
csv_path = tmp_path / "custom.csv"
parquet_path = tmp_path / "custom.pq"
out_dir = tmp_path / "derived"
source.to_csv(csv_path, index=False)
source.to_parquet(parquet_path, index=False)
csv_result = runner.invoke(cli, [
"derived", "ingest",
"--input-path", str(csv_path),
"--derived-name", "csv_custom",
"--output-dir", str(out_dir),
])
assert csv_result.exit_code == 0, csv_result.output
parquet_result = runner.invoke(cli, [
"derived", "ingest",
"--input-path", str(parquet_path),
"--derived-name", "parquet_custom",
"--output-dir", str(out_dir),
])
assert parquet_result.exit_code == 0, parquet_result.output
written = pd.read_parquet(out_dir / "csv_custom.pq")
assert written["symbol_id"].tolist() == ["sh600000", "sz000001"]
assert (out_dir / "parquet_custom.pq").exists()
def test_derived_validate_cli_rejects_duplicate_csv_columns(tmp_path):
runner = CliRunner()
csv_path = tmp_path / "bad.csv"
csv_path.write_text("symbol_id,date,x,x\nsh600000,2024-01-02,1.0,2.0\n")
result = runner.invoke(cli, [
"derived", "validate",
"--input-path", str(csv_path),
])
assert result.exit_code != 0
assert "duplicate columns" in result.output
def test_external_derived_plugin_loads_filters_params_and_uses_inputs(tmp_path):
module_path = tmp_path / "external_derived.py"
module_path.write_text(textwrap.dedent('''
import pandas as pd
from pipeline.derived.base import BaseDerivedData
from pipeline.derived.registry import register_derived
@register_derived
class FlexibleDerived(BaseDerivedData):
name = "flexible_derived_test"
def __init__(self, scale: float = 1.0):
self.scale = scale
def compute(self, daily=None, minute=None) -> pd.DataFrame:
result = None
if daily is not None:
result = daily[["symbol_id", "date", "close"]].copy()
result["daily_scaled_close"] = result.pop("close") * self.scale
if minute is not None:
minute_out = (
minute.groupby(["symbol_id", "date"], as_index=False)["volume"]
.sum()
.rename(columns={"volume": "minute_volume_sum"})
)
minute_out["minute_volume_sum"] *= self.scale
result = minute_out if result is None else result.merge(
minute_out, on=["symbol_id", "date"], how="left"
)
return result
'''))
load_derived_module(str(module_path))
assert "flexible_derived_test" in available_derived()
instance = get_derived("flexible_derived_test", scale=2.0, ignored=99)
assert instance.scale == 2.0
assert not hasattr(instance, "ignored")
daily_result = compute_derived(
"flexible_derived_test",
daily=_daily_bars(),
scale=2.0,
ignored=99,
)
assert "daily_scaled_close" in daily_result.columns
assert np.isclose(daily_result["daily_scaled_close"].iloc[0], 21.0)
minute_result = compute_derived(
"flexible_derived_test",
minute=_minute_bars(),
scale=2.0,
)
assert "minute_volume_sum" in minute_result.columns
assert np.isclose(
minute_result.loc[minute_result["symbol_id"] == "sh600000", "minute_volume_sum"].iloc[0],
800.0,
)
both_result = compute_derived(
"flexible_derived_test",
daily=_daily_bars(),
minute=_minute_bars(),
scale=1.0,
)
assert {"daily_scaled_close", "minute_volume_sum"}.issubset(both_result.columns)
def test_derived_compute_cli_writes_builtin_minute_summary(tmp_path):
runner = CliRunner()
minute_path = tmp_path / "minute.pq"
out_dir = tmp_path / "derived"
_minute_bars().to_parquet(minute_path, index=False)
result = runner.invoke(cli, [
"derived", "compute",
"--minute-path", str(minute_path),
"--derived-type", "minute_daily_summary",
"--derived-name", "minute_summary",
"--output-dir", str(out_dir),
])
assert result.exit_code == 0, result.output
written = pd.read_parquet(out_dir / "minute_summary.pq")
assert "minute_vwap" in written.columns
def test_alpha_feature_join_rejects_derived_column_collisions():
data = _daily_bars()
derived_a = data[["symbol_id", "date"]].copy()
derived_a["custom_value"] = 1.0
derived_b = data[["symbol_id", "date"]].copy()
derived_b["custom_value"] = 2.0
with pytest.raises(ValueError, match="conflict"):
join_feature_frames(data, [derived_a, derived_b])
close_collision = data[["symbol_id", "date"]].copy()
close_collision["close"] = 1.0
with pytest.raises(ValueError, match="conflict"):
join_feature_frames(data, [close_collision])
def test_legacy_feature_cli_delegates_to_derived_registry(tmp_path):
runner = CliRunner()
minute_path = tmp_path / "minute.pq"
out_dir = tmp_path / "features"
_minute_bars().to_parquet(minute_path, index=False)
list_result = runner.invoke(cli, ["feature", "list"])
assert list_result.exit_code == 0, list_result.output
assert "minute_daily_summary" in list_result.output
compute_result = runner.invoke(cli, [
"feature", "compute",
"--minute-path", str(minute_path),
"--feature-type", "minute_daily_summary",
"--feature-name", "minute_summary",
"--output-dir", str(out_dir),
])
assert compute_result.exit_code == 0, compute_result.output
assert (out_dir / "minute_summary.pq").exists()
+9
View File
@@ -7,6 +7,7 @@ import pandas as pd
import pytest import pytest
from pipeline.features.compute import compute_feature, validate_feature_frame from pipeline.features.compute import compute_feature, validate_feature_frame
from pipeline.features.library.minute_daily_summary import MinuteDailySummaryFeature
from pipeline.features.registry import ( from pipeline.features.registry import (
available_features, available_features,
get_feature, get_feature,
@@ -68,6 +69,14 @@ def test_built_in_minute_daily_summary():
assert pd.isna(missing["minute_vwap"]) assert pd.isna(missing["minute_vwap"])
def test_minute_daily_summary_feature_preserves_legacy_positional_compute():
direct = MinuteDailySummaryFeature().compute(_minute_bars())
via_registry = get_feature("minute_daily_summary").compute(_minute_bars())
assert "minute_vwap" in direct.columns
pd.testing.assert_frame_equal(direct, via_registry)
def test_load_external_feature_module_and_filter_params(tmp_path): def test_load_external_feature_module_and_filter_params(tmp_path):
module_path = tmp_path / "external_feature.py" module_path = tmp_path / "external_feature.py"
module_path.write_text(textwrap.dedent(''' module_path.write_text(textwrap.dedent('''