Use next-open returns for research eval

docs/reversal_5d_all_universe_pipeline_report.md

@@ -1,14 +1,15 @@
 # Tutorial: Testing a 5-Day Reversal Alpha
 
-This document is a teaching walkthrough for someone who is new to this
+Generated: 2026-06-12T18:30:56
-research framework and only lightly familiar with quant research. We will use
+
-one concrete experiment, a 5-day reversal alpha on the full downloaded Chinese
+This document is a teaching walkthrough for someone who is new to this research
+framework and only lightly familiar with quant research. We will use one
+concrete experiment, a 5-day reversal alpha on the full downloaded Chinese
 A-share universe, to learn how the framework defines an alpha, stores it, tests
 it, turns it into a portfolio, and explains the gap between a research result
 and simulated trading PnL.
 
-The original experiment was generated on 2026-06-11. The important point is not
+The important point is not the timestamp; it is the research method.
-the timestamp; it is the research method.
 
 ## The Research Question
 
@@ -29,10 +30,13 @@ The central research question is:
 
 The answer from this run is nuanced:
 
-- The naive built-in version loses badly on the full universe because raw
+- The naive built-in version is positive under the tradable
-  z-score weighting is too sensitive to A-share outliers.
+  next-open-to-next-open research convention (**41.40%**),
+  but its stored weights still show that raw z-score weighting is too sensitive
+  to A-share outliers.
 - A rank-weighted version on a liquid, non-ST, tradable universe has a positive
-  costless research result.
+  costless research result: **209.58%**
+  at Sharpe **1.44**.
 - The daily-traded implementation is still not tradable after costs because
   turnover is too high.
 
@@ -90,11 +94,11 @@ For this experiment, the important phases are:
 | --- | --- | --- |
 | Data | `cli.py data download` | What market data is available. |
 | Alpha compute | `cli.py alpha compute` | How a raw research idea becomes stored weights. |
-| Alpha eval | `cli.py alpha eval` | How those weights perform in a clean costless research view. |
+| Alpha eval | `cli.py alpha eval` | How close-formed weights perform over the tradable next-open-to-next-open interval. |
 | Combo | `cli.py combo combine` | How one or more alphas become one combined book. |
 | Portfolio build | `cli.py portfolio build` | How weights become target values and integer shares. |
 | Portfolio simulate | `cli.py portfolio simulate` | How the integer book trades at next open with constraints and costs. |
-| Portfolio eval | `cli.py portfolio eval` | How the continuous target portfolio behaves as a research portfolio. |
+| Portfolio eval | `cli.py portfolio eval` | How the continuous target portfolio behaves over the same costless open-to-open research interval. |
 
 In a real research workflow, you should learn to pause after every phase and
 inspect the parquet output. Most mistakes are easier to find at the interface
@@ -123,25 +127,9 @@ So:
 
 Notice the timing. The signal uses prices through date `t`. It must not use the
 return from `t` to `t+1`, because that is the future. The costless alpha
-evaluator tests the weight formed on date `t` against the next close-to-close
+evaluator tests the weight formed on date `t` over the tradable interval from
-return; the later execution simulator is the separate layer that trades the
+`open[t+1]` to `open[t+2]`; the later execution simulator is the separate layer
-constructed integer book at the next open.
+that trades the constructed integer book at the next open.
-
-The code lives in `pipeline/alpha/library/reversal.py`:
-
-```python
-class ReversalAlpha(BaseAlpha):
-    name = "reversal"
-
-    def __init__(self, lookback: int = 5):
-        self.lookback = lookback
-
-    def signal(self, close: pd.DataFrame) -> pd.DataFrame:
-        return -close.pct_change(self.lookback, fill_method=None)
-```
-
-The alpha class only defines the raw signal. The base class then turns that
-signal into weights.
 
 ## Step 2: Turn A Signal Into Cross-Sectional Weights
 
@@ -161,17 +149,19 @@ returns because they are newly listed, suspended, illiquid, or limit-constrained
 z-scoring can put a very large amount of relative exposure into exactly those
 names.
 
-That is what happened in the naive full-universe run. Stored weights reached
+That is visible in the naive full-universe run. Stored weights reached about
-about `-52` standard deviations. The research result collapsed:
+`-52` standard deviations. The
+result is positive under the open-to-open convention, but it is much weaker and
+less robust than the rank-weighted versions:
 
 | run | weighting | research cumulative return | research Sharpe | research turnover/year |
 | --- | --- | --- | --- | --- |
-| naive z-score, full universe | z-score | -87.45% | -2.4515 | 160x |
+| naive z-score, full universe | z-score | 41.40% | 0.4514 | 160x |
 
-The lesson is not "reversal is bad." The lesson is:
+The lesson is not "reversal is solved." The lesson is:
 
-> The same raw signal can become a bad portfolio if the weighting method reacts
+> The same raw signal can become a fragile portfolio if the weighting method
-> badly to outliers.
+> reacts badly to outliers.
 
 ## Step 3: Make The Weighting More Robust
 
@@ -186,15 +176,15 @@ weights = ranks.subtract(ranks.mean(axis=1), axis=0)
 
 Rank weighting keeps the ordering of stocks but removes the importance of the
 exact outlier magnitude. A stock can be "the worst recent loser" or "the best
-recent winner," but it cannot become 52 standard deviations important just
+recent winner," but it cannot become dozens of standard deviations important
-because its raw percentage move is unusual.
+just because its raw percentage move is unusual.
 
 The full-universe rank version was much less pathological, but still not a
 clean signal:
 
 | run | weighting | research cumulative return | research Sharpe | research turnover/year |
 | --- | --- | --- | --- | --- |
-| rank, full universe | rank | -3.48% | -0.0198 | 143x |
+| rank, full universe | rank | 73.44% | 0.8860 | 143x |
 
 That tells us the weighting fix helped, but the universe still contains many
 names that are poor candidates for a daily reversal strategy.
@@ -218,7 +208,7 @@ The liquid rank result is the cleanest research result:
 
 | run | weighting | universe | research cumulative return | research Sharpe | hit rate |
 | --- | --- | --- | --- | --- | --- |
-| rank, liquid subset | rank | top 1000 liquid, tradable, non-ST | 72.24% | 0.7310 | 54.31% |
+| rank, liquid subset | rank | top 1000 liquid, tradable, non-ST | 209.58% | 1.4422 | 55.68% |
 
 This is the first point where a researcher can say:
 
@@ -229,13 +219,6 @@ That last phrase, **before trading costs**, is essential.
 
 ![Research equity](assets/reversal_5d_research_equity.png)
 
-When reading this chart, focus on the shape and relative behavior:
-
-- The naive z-score line shows the outlier problem.
-- The rank full-universe line shows that robust weighting helps but does not
-  fully solve the universe problem.
-- The liquid rank line shows the signal-level edge before execution costs.
-
 ## Step 5: Check That The Alpha File Is Sane
 
 Before trusting any metric, inspect the stored alpha artifact. The run checked:
@@ -247,7 +230,7 @@ Before trusting any metric, inspect the stored alpha artifact. The run checked:
 - The daily cross-sectional mean is approximately zero.
 - A one-alpha combo is an exact identity transform.
 
-| run | schema ok | null weights | non-finite weights | duplicate keys | max abs daily mean | weight range | combo identity diff |
+| run | schema ok | null w | non-finite w | dup keys | max \|daily mean\| | weight range | combo identity Δ |
 | --- | --- | --- | --- | --- | --- | --- | --- |
 | naive z-score (full) | True | 0 | 0 | 0 | 3.32e-16 | [-52.2, 19.2] | 0.00e+00 |
 | rank (full) | True | 0 | 0 | 0 | 0.00e+00 | [-2603.0, 2603.0] | 0.00e+00 |
@@ -261,51 +244,42 @@ raw size of an abnormal stock move.
 
 ![Weight distributions](assets/reversal_5d_weight_distributions.png)
 
-This is a good habit: when a backtest looks strange, plot the weights before
-debugging the PnL. A broken or concentrated weight distribution often explains
-the result.
-
 ## Step 6: Understand The Alpha Evaluation Formula
 
-The costless alpha evaluator asks:
+The costless alpha evaluator now asks:
 
-> If we held the alpha weights from date `t`, what close-to-close return would
+> If we compute alpha weights after close on date `t`, trade them at `open[t+1]`,
-> we earn from `t` to `t+1`?
+> and hold them until `open[t+2]`, what return would we earn before costs?
 
-This is intentionally a **research-layer approximation**, not the trading
+This is still a **research-layer approximation**, not the trading simulator. At
-simulator. At this stage the framework has only an alpha weight file. It has not
+this stage the framework has only an alpha weight file. It has not yet rounded
-yet rounded shares, checked limits, clipped trades, or paid costs. The purpose
+shares, checked limits, clipped trades, or paid costs. The purpose is to answer
-is to answer a clean signal question: "Do these close-formed weights line up
+a clean signal question: "Do these close-formed weights line up with returns
-with the next day's returns?"
+over the interval we could actually own after next-open execution?"
-
-The actual trading layer comes later. `portfolio simulate` takes the integer
-`position_shares` from the portfolio builder, executes the target from signal
-date `t` at `open[t+1]`, then marks PnL as overnight movement on the old book
-plus intraday movement on the newly filled book, minus trading cost.
 
 The daily research return is:
 
 ```text
-R[t] = sum_i(weight[i, t] * return[i, t+1]) / sum_i(abs(weight[i, t]))
+R[t] = sum_i(weight[i, t] * (open[i, t+2] / open[i, t+1] - 1)) / sum_i(abs(weight[i, t]))
 ```
 
 This has three important consequences:
 
 - The alpha is normalized by its gross exposure, so the scale of raw weights
   does not by itself create a higher return.
-- The next day's return is used, so the test avoids look-ahead.
+- The new signal does not receive credit for the overnight gap from `close[t]`
-- The last signal date is dropped from performance metrics because there is no
+  to `open[t+1]`, because it cannot be traded until `open[t+1]`.
-  next return for it.
+- The final two signal dates are dropped from performance metrics because they
+  do not have a complete next-open-to-next-open holding interval.
 
-Turnover is also measured from the weights:
+Turnover is still measured from the weights:
 
 ```text
 turnover[t] = sum_i(abs(weight[i, t] - weight[i, t-1])) / sum_i(abs(weight[i, t-1]))
 ```
 
-The annualized turnover numbers around 143x to 160x are a warning. Even a
+The annualized turnover numbers are a warning. Even a positive signal can be
-positive signal can be hard to monetize if it asks the portfolio to trade too
+hard to monetize if it asks the portfolio to trade too much every day.
-much every day.
 
 ## Step 7: Build A Portfolio From The Alpha
 
@@ -325,24 +299,14 @@ logic. This is where a research portfolio starts to become a tradable portfolio.
 
 The continuous target portfolio matched the stored alpha almost exactly:
 
-| run | target value identity max abs diff | alpha to target max abs diff | research correlation alpha vs portfolio | mean integer gross | mean L1 tracking |
+| run | target_value identity max\|Δ\| | alpha→target max\|Δ\| | research corr(alpha,portfolio) | mean integer gross | mean L1 tracking |
 | --- | --- | --- | --- | --- | --- |
 | naive z-score (full) | 0.0000 | 0.00e+00 | 1.000000 | 9,138,331 | 2,542,655 |
 | rank (full) | 0.0000 | 0.00e+00 | 1.000000 | 8,984,098 | 2,678,278 |
 | rank (liquid subset) | 0.0000 | 0.00e+00 | 1.000000 | 9,810,256 | 862,303 |
 
-The integer book is not exact because small target positions can be rounded
-away. The liquid subset has lower tracking error because it spreads the book
-over fewer and more tradable names.
-
 ![Portfolio tracking](assets/reversal_5d_portfolio_tracking.png)
 
-When you research a new alpha, ask two separate questions:
-
-- Does the continuous target portfolio match the alpha? It should.
-- Does the integer tradable portfolio still resemble the target? It may not,
-  especially for small books or very broad universes.
-
 ## Step 8: Simulate Execution And Costs
 
 Research returns are not the same as tradable PnL. The simulator executes the
@@ -366,17 +330,19 @@ The execution results explain the final research conclusion:
 
 | run | corr(alpha, exec net) | PnL before cost | total cost | net PnL | mean daily turnover |
 | --- | --- | --- | --- | --- | --- |
-| naive z-score (full) | 0.9675 | 1,838,974 | 13,032,720 | -11,193,746 | 0.5711 |
+| naive z-score (full) | 0.8956 | 1,838,974 | 13,032,720 | -11,193,746 | 0.5711 |
-| rank (full) | 0.9613 | 5,052,067 | 11,713,451 | -6,661,383 | 0.5133 |
+| rank (full) | 0.9126 | 5,052,067 | 11,713,451 | -6,661,383 | 0.5133 |
-| rank (liquid subset) | 0.9762 | 11,017,842 | 12,733,803 | -1,715,960 | 0.5715 |
+| rank (liquid subset) | 0.8884 | 11,017,842 | 12,733,803 | -1,715,960 | 0.5715 |
 
-The liquid rank run made about 11.0 million before cost, but paid about 12.7
+For the liquid rank run, simulated PnL before cost is about
-million in cost. That is why the final net PnL is negative.
+11,017,842, but total
+cost is about 12,733,803. That is why
+the final net PnL is weak or negative.
 
 This is not a contradiction. It is exactly what a research pipeline should show:
 
-> The signal exists in the costless layer, but the daily implementation trades
+> The signal can exist in the costless layer, but the daily implementation can
-> too much to keep the edge.
+> still trade too much to keep the edge.
 
 ![Execution vs research](assets/reversal_5d_exec_vs_research.png)
 
@@ -384,22 +350,27 @@ This is not a contradiction. It is exactly what a research pipeline should show:
 
 The complete summary is:
 
-| run | weighting | research cumulative return | research Sharpe | research turnover/year | exec before cost | exec net | exec net Sharpe |
+| run | weighting | research cum | research Sharpe | research turn/yr | exec before cost | exec net | exec net Sharpe |
 | --- | --- | --- | --- | --- | --- | --- | --- |
-| naive z-score (full) | z-score | -87.45% | -2.4515 | 160x | 18.39% | -111.94% | -1.4508 |
+| naive z-score (full) | z-score | 41.40% | 0.4514 | 160× | 18.39% | -111.94% | -1.4508 |
-| rank (full) | rank | -3.48% | -0.0198 | 143x | 50.52% | -66.61% | -1.1839 |
+| rank (full) | rank | 73.44% | 0.8860 | 143× | 50.52% | -66.61% | -1.1839 |
-| rank (liquid subset) | rank | 72.24% | 0.7310 | 148x | 110.18% | -17.16% | -0.2226 |
+| rank (liquid subset) | rank | 209.58% | 1.4422 | 148× | 110.18% | -17.16% | -0.2226 |
+
+*Research = costless, no-look-ahead weights over the next-open-to-next-open
+holding interval. Execution = next-open fills on the discretized integer book
+under suspension / price-limit / volume-cap constraints, 5 bps commission + 5
+bps slippage.*
 
 Here is the interpretation:
 
-- **Naive z-score full universe**: not a useful test of the reversal idea,
+- **Naive z-score full universe**: positive under open-to-open research, but a
-  because the weighting scheme lets outliers dominate the book.
+  less reliable test of the reversal idea because the weighting scheme lets
+  outliers dominate parts of the book.
 - **Rank full universe**: a better test of the same idea, but still noisy
   because the universe includes too many problematic names.
-- **Rank liquid subset**: the best signal-level test; it finds a positive
+- **Rank liquid subset**: the best signal-level test; it finds the cleanest
   costless reversal effect.
-- **Execution net**: all variants lose after cost at daily rebalance frequency,
+- **Execution net**: daily rebalancing remains heavily constrained by cost.
-  so the implementation is not yet tradable.
 
 A beginner might look only at the final net PnL and say "the alpha failed." A
 researcher should be more precise:
@@ -407,9 +378,25 @@ researcher should be more precise:
 > The raw 5-day reversal idea has signal value in a liquid universe, but the
 > current daily trading rule has too much turnover for the assumed cost model.
 
-That distinction tells you what to try next.
+## Step 10: Time Consumption By Phase
 
-## Step 10: Reproduce The Experiment
+| phase | rank full (s) | rank liquid (s) |
+| --- | --- | --- |
+| alpha compute | 94.1 | 107.8 |
+| alpha eval | 93.3 | 96.9 |
+| combo combine | 21.6 | 21.7 |
+| portfolio build | 537.6 | 236.7 |
+| portfolio eval | 95.1 | 88.3 |
+| portfolio simulate | 139.6 | 139.1 |
+| total | 981.3 | 690.5 |
+
+![Phase timings](assets/reversal_5d_phase_timings.png)
+
+`portfolio build` usually dominates because it iterates per signal date and
+repairs a multi-thousand-name integer book under lot rules. The liquid run is
+faster because it carries fewer non-zero names per date.
+
+## Step 11: Reproduce The Experiment
 
 These commands reproduce the important artifacts, assuming the full daily-bar
 dataset already exists at `data/daily_bars/all`.
@@ -423,9 +410,7 @@ uv run python cli.py alpha compute --data-path data/daily_bars/all \
 # Rank-weighted full and liquid runs.
 bash scripts/run_reversal_rank_e2e.sh
 
-# Regenerate figures, diagnostics, and the older auto-generated report.
+# Regenerate figures, diagnostics, and this tutorial report.
-# This command rewrites this markdown file, so run it only when you want
-# generated output to replace the tutorial.
 uv run python scripts/generate_reversal_5d_report.py
 ```
 
@@ -452,8 +437,9 @@ Use this checklist for a new idea.
    illiquid names.
 
 5. Evaluate the alpha as a portfolio, not as a prediction.
-   Check cumulative return, Sharpe, drawdown, hit rate, and turnover. Do not add
+   Check cumulative return, Sharpe, drawdown, hit rate, and turnover over the
-   IC/IR unless the framework's alpha convention changes.
+   next-open-to-next-open holding interval. Do not add IC/IR unless the
+   framework's alpha convention changes.
 
 6. Build the portfolio and inspect tracking.
    Confirm that target weights match the alpha, then check whether integer
@@ -468,7 +454,7 @@ Use this checklist for a new idea.
    universe, construction, execution constraints, turnover, or cost.
 
 For this 5-day reversal study, the diagnosis is clear: **the signal-level result
-is promising only after robust weighting and a liquid universe filter, but the
+is most promising after robust weighting and a liquid universe filter, but the
 current implementation needs turnover control before it can be considered
 tradable.**
 

pipeline/alpha/compute.py

@@ -25,9 +25,22 @@ def _pivot_close(df: pd.DataFrame) -> pd.DataFrame:
     return pivot.sort_index()
 
 
-def _daily_returns(close: pd.DataFrame) -> pd.DataFrame:
+def _pivot_open(df: pd.DataFrame) -> pd.DataFrame:
-    """Compute daily returns from wide close DataFrame."""
+    """Pivot data to wide format: date index, columns = symbol_id, values = open."""
-    return close.pct_change(fill_method=None)
+    pivot = df.pivot_table(
+        index="date", columns="symbol_id", values="open", aggfunc="first"
+    )
+    return pivot.sort_index()
+
+
+def _forward_open_to_open_returns(open_: pd.DataFrame) -> pd.DataFrame:
+    """Return earned by a close-formed signal after next-open execution.
+
+    A weight formed after close on date t can first be traded at open[t+1].
+    With daily retargeting it is then held until open[t+2], so the signal-date
+    forward return is open[t+2] / open[t+1] - 1.
+    """
+    return open_.shift(-2).divide(open_.shift(-1)) - 1.0
 
 
 def investable_universe_mask(
@@ -150,11 +163,11 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
 
     Computes return, annualized Sharpe, annualized turnover, max drawdown.
 
-    Alpha is interpreted as POSITION WEIGHTS, not predictions.
+    Alpha is interpreted as POSITION WEIGHTS, not predictions. A close-formed
-    Return on date t = sum(weight[s,t] * realized_return[s,t+1]) /
+    weight on date t is assumed tradable at open[t+1] and held until open[t+2].
-    sum(abs(weight[s,t])). This matches the close-derived signal convention:
+    Return on signal date t = sum(weight[s,t] * open_to_open_return[s,t]) /
-    weights formed with close[t] earn the next close-to-close return, avoiding
+    sum(abs(weight[s,t])). This matches the execution convention without
-    look-ahead.
+    crediting the new signal for the overnight gap before it can be traded.
 
     Args:
         alpha_df: DataFrame with ALPHA_COLUMNS.
@@ -164,8 +177,8 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
         Dict with metrics: cumulative_return, sharpe_annual, turnover_annual,
         max_drawdown, hit_rate, n_dates.
     """
-    close = _pivot_close(data_df)
+    open_ = _pivot_open(data_df)
-    returns = _daily_returns(close)
+    fwd_returns_all = _forward_open_to_open_returns(open_)
 
     # Pivot alpha weights to wide format
     weights = alpha_df.pivot_table(
@@ -173,11 +186,12 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
     ).sort_index()
 
     # Align weights to signal dates that exist on the market calendar. Compute
-    # forward returns on the full market calendar first, so sparse signal grids
+    # forward open-to-open returns on the full market calendar first, so sparse
-    # still earn the next available data date instead of the next signal date.
+    # signal grids still earn the next available open-to-open interval instead
-    common_dates = weights.index.intersection(returns.index)
+    # of the next signal date.
+    common_dates = weights.index.intersection(open_.index)
     weights = weights.loc[common_dates]
-    fwd_returns = returns.shift(-1).reindex(common_dates)
+    fwd_returns = fwd_returns_all.reindex(common_dates)
 
     if len(common_dates) < 1:
         return {
@@ -189,8 +203,9 @@ def evaluate_alpha(alpha_df: pd.DataFrame, data_df: pd.DataFrame) -> dict:
             "n_dates": 0,
         }
 
-    # Daily portfolio return = sum(w_t * r_{t+1}) / sum(|w_t|).
+    # Daily portfolio return = sum(w_t * r_open[t+1→t+2]) / sum(|w_t|).
-    # The last signal date has no next-period return and is dropped below.
+    # The final two signal dates have no complete next-open holding interval
+    # and are dropped below.
     gross = weights.abs().sum(axis=1)
     daily_returns = (
         (weights * fwd_returns).sum(axis=1, min_count=1)

pipeline/portfolio/construct.py

@@ -7,9 +7,9 @@ across dates (positions are stateful, unlike alphas/combos), discretizing and
 repairing each day's target into a tradable integer book.
 
 Return-convention note: weights here are *target allocations*. The research
-evaluation in :mod:`pipeline.portfolio.research` marks them close-to-close on the
+evaluation in :mod:`pipeline.portfolio.research` marks them from next open to
-*next* period (no look-ahead); the execution simulator marks the actually-filled
+the following open (no look-ahead); the execution simulator marks the
-book at the next open. See those modules for details.
+actually-filled book at the next open. See those modules for details.
 """
 
 from __future__ import annotations

pipeline/portfolio/research.py

@@ -6,9 +6,10 @@ trading constraints. Metrics are return / Sharpe / turnover / max-drawdown /
 convention that an alpha is a position weight, not a return predictor.
 
 Return convention (documented): the target weight formed from information at
-date ``t`` earns the *next* period's close-to-close return, i.e. weights are
+date ``t`` is assumed tradable at ``open[t+1]`` and held until ``open[t+2]``.
-shifted one day relative to realized returns, so there is no look-ahead:
+This is a costless approximation of the next-open execution path: no lots,
-``R_t = sum_i w_{i,t} · r_{i,t+1}`` normalized by gross exposure.
+constraints, or costs, but no credit for an overnight gap that the new signal
+could not have owned.
 """
 
 from __future__ import annotations
@@ -26,27 +27,27 @@ def evaluate_portfolio(positions_df: pd.DataFrame, data_df: pd.DataFrame) -> dic
     Args:
         positions_df: POSITION_COLUMNS (uses ``target_weight``; zero-gross
             construction carry dates remain flat in this research view).
-        data_df: DATA_COLUMNS (uses ``close`` for returns).
+        data_df: DATA_COLUMNS (uses ``open`` for returns).
 
     Returns:
         Dict with ``cumulative_return, sharpe_annual, turnover_annual,
         max_drawdown, fitness, hit_rate, n_dates``. No IC key.
     """
-    close = data_df.pivot_table(
+    open_ = data_df.pivot_table(
-        index="date", columns="symbol_id", values="close", aggfunc="first"
+        index="date", columns="symbol_id", values="open", aggfunc="first"
     ).sort_index()
-    returns = close.pct_change(fill_method=None)
+    fwd = open_.shift(-2).divide(open_.shift(-1)) - 1.0
 
     weights = positions_df.pivot_table(
         index="date", columns="symbol_id", values="target_weight", aggfunc="first"
     ).sort_index()
 
-    common = weights.index.intersection(returns.index)
+    common = weights.index.intersection(open_.index)
     weights = weights.loc[common]
     # Compute forward returns on the full market calendar before selecting
-    # signal dates. This preserves next-period returns when the signal grid is
+    # signal dates. This preserves the next available open-to-open holding
-    # sparser than the data grid.
+    # interval when the signal grid is sparser than the data grid.
-    fwd = returns.shift(-1).reindex(common)
+    fwd = fwd.reindex(common)
 
     empty = {
         "cumulative_return": 0.0, "sharpe_annual": 0.0, "turnover_annual": 0.0,
@@ -58,7 +59,7 @@ def evaluate_portfolio(positions_df: pd.DataFrame, data_df: pd.DataFrame) -> dic
         return empty
 
     gross = weights.abs().sum(axis=1)
-    # Weights at t earn the return from t to t+1.
+    # Weights at t earn the costless tradable interval open[t+1] -> open[t+2].
     daily = (
         (weights * fwd).sum(axis=1, min_count=1)
         / gross.replace(0.0, np.nan)

scripts/generate_reversal_5d_report.py

@@ -7,7 +7,7 @@ Covers three runs of the same 5-day reversal *signal* under this repo's
     rank_full   : reversal_rank (rank weighting),    full ~5k all-universe
     rank_liquid : reversal_rank (rank weighting),    per-date liquid subset
 
-For each run it checks artifact storage, recomputes no-lookahead research
+For each run it checks artifact storage, recomputes no-lookahead open-to-open research
 metrics, measures how close the constructed portfolio is to the alpha and how
 close the simulated net PnL is to the alpha, and renders a markdown report plus
 PNG visualizations under docs/.
@@ -159,7 +159,7 @@ def _additive_metrics(daily: pd.Series) -> dict:
 
 
 def _research_returns(weights: pd.DataFrame, fwd: pd.DataFrame) -> pd.Series:
-    """w_t · r_{t+1} / sum|w_t| on the signal calendar (no lookahead)."""
+    """w_t · r_open[t+1→t+2] / sum|w_t| on the signal calendar."""
     w = weights
     f = fwd.reindex(index=w.index, columns=w.columns)
     gross = w.abs().sum(axis=1)
@@ -363,7 +363,7 @@ def plot_research_equity(results: dict) -> Path:
                       f"(Sharpe {results[k]['alpha_metrics']['sharpe_annual']:.2f})")
     ax.axhline(1.0, color="#666", linewidth=0.8)
     ax.set_yscale("log")
-    ax.set_title("Costless no-lookahead alpha research equity (log scale)")
+    ax.set_title("Costless next-open-to-next-open alpha research equity (log scale)")
     ax.set_ylabel("growth of 1.0")
     ax.grid(True, which="both", alpha=0.25)
     ax.legend(loc="best")
@@ -598,176 +598,455 @@ def render_report(results: dict, data_summary: dict, timings: dict,
             return float("nan")
         return results[run_key]["alpha_metrics"]["cumulative_return"]
 
-    return f"""# 5-Day Reversal — End-to-End Pipeline Report
+    return f"""# Tutorial: Testing a 5-Day Reversal Alpha
 
 Generated: {datetime.now().isoformat(timespec="seconds")}
 
-This report runs the **5-day reversal** signal end to end through the decoupled
+This document is a teaching walkthrough for someone who is new to this research
-pipeline (`data → alpha → combo → portfolio build → portfolio simulate/eval`) on
+framework and only lightly familiar with quant research. We will use one
-the full downloaded A-share universe, and answers the seven review questions:
+concrete experiment, a 5-day reversal alpha on the full downloaded Chinese
-alpha storage, metric sanity, NaN/look-ahead handling, alpha↔portfolio
+A-share universe, to learn how the framework defines an alpha, stores it, tests
-closeness, alpha↔PnL closeness, per-phase timing, and visualizations.
+it, turns it into a portfolio, and explains the gap between a research result
+and simulated trading PnL.
 
-Per this repo's convention an **alpha is a signed cross-sectional position
+The important point is not the timestamp; it is the research method.
-weight, not a return predictor**, so evaluation is return / Sharpe / turnover /
-drawdown — there is deliberately **no IC/IR** anywhere.
 
-## TL;DR
+## The Research Question
 
-The naive built-in `reversal` alpha (raw `-pct_change(5)` then cross-sectional
+A quant research project starts with a hypothesis:
-**z-score**) loses **{_pct(cum('naive_full'))}** in costless research on the full
-~5,200-name universe. That is **not** evidence the signal is bad — it is an
-artifact of z-score weighting on A-shares: a handful of newly listed /
-post-suspension / limit-up names produce huge `pct_change` outliers, and
-z-scoring pours the book into exactly those names (stored weights reach
-{results['naive_full']['storage']['weight_min']:.0f}σ).
 
-Switching only the **weighting** to a bounded cross-sectional **rank**
+> If a stock fell a lot over the last few trading days, it may rebound soon; if
-(`reversal_rank`) and restricting to a per-date **liquid, non-ST, tradable**
+> it rose a lot, it may cool off soon.
-universe recovers a genuine reversal edge: **{_pct(cum('rank_liquid')) if rliq else float('nan')}**
-costless research cumulative return at Sharpe
-**{results['rank_liquid']['alpha_metrics']['sharpe_annual']:.2f}** with a
-{_pct(results['rank_liquid']['alpha_metrics']['hit_rate']) if rliq else 'n/a'} daily hit rate.
 
-The binding constraint is **cost, not signal**: at ~{results['rank_liquid']['alpha_metrics']['turnover_annual']:.0f}×/year
+This is called **short-horizon reversal**. It is a simple idea: recent losers
-turnover, a 10 bps one-way per-trade cost (5 bps commission + 5 bps slippage,
+are candidates to buy, and recent winners are candidates to sell or underweight.
-charged on each leg — so ~20 bps per round trip) erases the edge — every variant
+In this repo, the tested version looks back 5 trading days.
-is negative after costs. A tradable 5-day reversal needs
-turnover control, not a different signal.
 
-## Headline Metrics
+The central research question is:
 
-{headline}
+> Does this 5-day reversal rule create useful portfolio returns after the
+> framework applies realistic storage, portfolio construction, execution
+> constraints, and trading costs?
 
-*Research = costless, no-look-ahead weights · next-day return. Execution = next-open
+The answer from this run is nuanced:
-fills on the discretized integer book under suspension / price-limit / volume-cap
+
-constraints, 5 bps commission + 5 bps slippage.*
+- The naive built-in version is positive under the tradable
+  next-open-to-next-open research convention (**{_pct(cum('naive_full'))}**),
+  but its stored weights still show that raw z-score weighting is too sensitive
+  to A-share outliers.
+- A rank-weighted version on a liquid, non-ST, tradable universe has a positive
+  costless research result: **{_pct(cum('rank_liquid')) if rliq else 'n/a'}**
+  at Sharpe **{results['rank_liquid']['alpha_metrics']['sharpe_annual']:.2f}**.
+- The daily-traded implementation is still not tradable after costs because
+  turnover is too high.
+
+That is a normal research outcome. Good research is not just asking "did the
+backtest go up?" It is asking **which layer explains the result**: signal,
+weighting, universe, construction, execution, or cost.
+
+## How This Framework Defines An Alpha
+
+In many quant textbooks, an alpha is described as a **prediction** of future
+returns. This framework uses a stricter and more practical convention:
+
+> An alpha is a signed cross-sectional position weight.
+
+That sentence is the key to the whole repo.
+
+- **Signed** means positive values are long exposure and negative values are
+  short exposure.
+- **Cross-sectional** means the alpha compares stocks to other stocks on the
+  same date.
+- **Position weight** means the output is already an instruction about what the
+  portfolio wants to own. It is not merely a score to correlate with future
+  returns.
+
+The stored alpha file always has this schema:
+
+| column | meaning |
+| --- | --- |
+| `symbol_id` | Stock identifier such as `sh600000` or `sz000001`. |
+| `date` | The signal date. The alpha is formed using information known by this date's close. |
+| `alpha_name` | A label for this particular run, such as `reversal_5d_all`. |
+| `weight` | Signed desired exposure. Positive means long; negative means short. |
+
+Because the framework treats alphas as position weights, it evaluates them with
+portfolio metrics: return, Sharpe, turnover, drawdown, and hit rate. It does
+**not** use IC/IR, because IC/IR would treat the alpha as a return predictor.
+
+## The Pipeline In One Picture
+
+Every phase reads parquet files and writes parquet files. That makes the system
+easy to inspect and rerun one layer at a time.
+
+```text
+daily bars
+   -> alpha weights
+   -> combined weights
+   -> portfolio targets and integer positions
+   -> simulated fills and PnL
+   -> evaluation metrics
+```
+
+For this experiment, the important phases are:
+
+| phase | command family | what it teaches you |
+| --- | --- | --- |
+| Data | `cli.py data download` | What market data is available. |
+| Alpha compute | `cli.py alpha compute` | How a raw research idea becomes stored weights. |
+| Alpha eval | `cli.py alpha eval` | How close-formed weights perform over the tradable next-open-to-next-open interval. |
+| Combo | `cli.py combo combine` | How one or more alphas become one combined book. |
+| Portfolio build | `cli.py portfolio build` | How weights become target values and integer shares. |
+| Portfolio simulate | `cli.py portfolio simulate` | How the integer book trades at next open with constraints and costs. |
+| Portfolio eval | `cli.py portfolio eval` | How the continuous target portfolio behaves over the same costless open-to-open research interval. |
+
+In a real research workflow, you should learn to pause after every phase and
+inspect the parquet output. Most mistakes are easier to find at the interface
+between two phases than at the final PnL line.
+
+## Step 1: Define The Raw Reversal Signal
+
+The built-in 5-day reversal alpha is implemented as:
+
+```python
+signal = -close.pct_change(5, fill_method=None)
+```
+
+For stock `i` on date `t`, this is approximately:
+
+```text
+signal[i, t] = -(close[i, t] / close[i, t-5] - 1)
+```
+
+So:
+
+- If a stock rose by 10% over the last 5 trading days, the raw signal is `-10%`.
+  It becomes a candidate short or underweight.
+- If a stock fell by 10% over the last 5 trading days, the raw signal is `+10%`.
+  It becomes a candidate long or overweight.
+
+Notice the timing. The signal uses prices through date `t`. It must not use the
+return from `t` to `t+1`, because that is the future. The costless alpha
+evaluator tests the weight formed on date `t` over the tradable interval from
+`open[t+1]` to `open[t+2]`; the later execution simulator is the separate layer
+that trades the constructed integer book at the next open.
+
+## Step 2: Turn A Signal Into Cross-Sectional Weights
+
+By default, `BaseAlpha.to_weights()` does a cross-sectional z-score each date:
+
+```text
+weight[i, t] = (signal[i, t] - mean_signal[t]) / std_signal[t]
+```
+
+This means the framework asks:
+
+> On this date, which stocks have stronger reversal scores than the rest of the
+> market, and by how much?
+
+That is useful, but it has a weakness. If a few stocks have extreme trailing
+returns because they are newly listed, suspended, illiquid, or limit-constrained,
+z-scoring can put a very large amount of relative exposure into exactly those
+names.
+
+That is visible in the naive full-universe run. Stored weights reached about
+`{results['naive_full']['storage']['weight_min']:.0f}` standard deviations. The
+result is positive under the open-to-open convention, but it is much weaker and
+less robust than the rank-weighted versions:
+
+| run | weighting | research cumulative return | research Sharpe | research turnover/year |
+| --- | --- | --- | --- | --- |
+| naive z-score, full universe | z-score | {_pct(cum('naive_full'))} | {results['naive_full']['alpha_metrics']['sharpe_annual']:.4f} | {results['naive_full']['alpha_metrics']['turnover_annual']:.0f}x |
+
+The lesson is not "reversal is solved." The lesson is:
+
+> The same raw signal can become a fragile portfolio if the weighting method
+> reacts badly to outliers.
+
+## Step 3: Make The Weighting More Robust
+
+The repo also has a rank-weighted version, `reversal_rank`. It uses the same raw
+5-day reversal signal, but converts the cross-section to ranks instead of
+z-scores:
+
+```python
+ranks = signal.rank(axis=1)
+weights = ranks.subtract(ranks.mean(axis=1), axis=0)
+```
+
+Rank weighting keeps the ordering of stocks but removes the importance of the
+exact outlier magnitude. A stock can be "the worst recent loser" or "the best
+recent winner," but it cannot become dozens of standard deviations important
+just because its raw percentage move is unusual.
+
+The full-universe rank version was much less pathological, but still not a
+clean signal:
+
+| run | weighting | research cumulative return | research Sharpe | research turnover/year |
+| --- | --- | --- | --- | --- |
+| rank, full universe | rank | {_pct(cum('rank_full')) if rfull else 'n/a'} | {results['rank_full']['alpha_metrics']['sharpe_annual']:.4f} | {results['rank_full']['alpha_metrics']['turnover_annual']:.0f}x |
+
+That tells us the weighting fix helped, but the universe still contains many
+names that are poor candidates for a daily reversal strategy.
+
+## Step 4: Define The Investable Universe
+
+An alpha should be tested on stocks that could plausibly be traded. The liquid
+run applies a per-date mask before weights are created. A stock must be:
+
+- seasoned, with at least 60 observed closes;
+- currently tradable, using `tradestatus == 1`;
+- not ST, using `isST == 0`;
+- inside the top 1000 names by trailing 20-day average traded amount.
+
+This mask is applied to the signal, not to the price history used to compute the
+5-day return. That distinction matters. We still compute `pct_change(5)` on the
+full contiguous price history, then decide which names are eligible to hold on
+each signal date.
+
+The liquid rank result is the cleanest research result:
+
+| run | weighting | universe | research cumulative return | research Sharpe | hit rate |
+| --- | --- | --- | --- | --- | --- |
+| rank, liquid subset | rank | top 1000 liquid, tradable, non-ST | {_pct(cum('rank_liquid')) if rliq else 'n/a'} | {results['rank_liquid']['alpha_metrics']['sharpe_annual']:.4f} | {_pct(results['rank_liquid']['alpha_metrics']['hit_rate']) if rliq else 'n/a'} |
+
+This is the first point where a researcher can say:
+
+> There appears to be a real 5-day reversal effect in a cleaner A-share
+> universe, before trading costs.
+
+That last phrase, **before trading costs**, is essential.
 
 ![Research equity](assets/reversal_5d_research_equity.png)
 
-## 1. Are Alpha Values Properly Stored?
+## Step 5: Check That The Alpha File Is Sane
 
-All alpha artifacts conform to `ALPHA_COLUMNS` (`symbol_id`, `date`, `alpha_name`,
+Before trusting any metric, inspect the stored alpha artifact. The run checked:
-`weight`), carry no null / non-finite weights, no duplicate `(symbol_id, date)`
+
-keys, and have numerically-zero daily cross-sectional means (weights are
+- The columns match `ALPHA_COLUMNS`.
-demeaned per date).
+- There are no null weights.
+- There are no non-finite weights.
+- There are no duplicate `(symbol_id, date)` rows.
+- The daily cross-sectional mean is approximately zero.
+- A one-alpha combo is an exact identity transform.
 
 {storage}
 
-The decisive storage signal is the **weight range**. The naive z-score alpha
+The rank ranges look numerically large because rank weights scale with the
-stores weights as extreme as
+number of names. That is fine: later evaluation divides by gross exposure, and
-`[{results['naive_full']['storage']['weight_min']:.0f}, {results['naive_full']['storage']['weight_max']:.0f}]` —
+portfolio construction normalizes by `sum(abs(weight))`. The important
-single names tens of sigma from the cross-section. Rank weighting is bounded by
+difference is that rank weights are bounded by cross-sectional rank, not by the
-construction, so its stored weights are well-behaved. Same signal, completely
+raw size of an abnormal stock move.
-different book.
 
 ![Weight distributions](assets/reversal_5d_weight_distributions.png)
 
-## 2. Do The Alpha Metrics Make Sense?
+## Step 6: Understand The Alpha Evaluation Formula
 
-Yes, and they tell a coherent story:
+The costless alpha evaluator now asks:
 
-- The **z-score full** run is dominated by a few outlier names; its research
+> If we compute alpha weights after close on date `t`, trade them at `open[t+1]`,
-  Sharpe of {results['naive_full']['alpha_metrics']['sharpe_annual']:.2f} reflects a
+> and hold them until `open[t+2]`, what return would we earn before costs?
-  book that is effectively long/short a tiny set of extreme movers, which in
-  A-shares keep trending — so the reversal bet loses.
-- **Rank full** ({_pct(cum('rank_full')) if rfull else 'n/a'}) is roughly flat:
-  the direction is right (hit rate
-  {_pct(results['rank_full']['alpha_metrics']['hit_rate']) if rfull else 'n/a'}) but
-  the long tail of illiquid / ST / freshly listed names adds noise.
-- **Rank liquid** is the clean result: a positive, monotone reversal premium
-  ({_pct(cum('rank_liquid')) if rliq else 'n/a'}, Sharpe
-  {results['rank_liquid']['alpha_metrics']['sharpe_annual']:.2f}) once the
-  investable universe is sane.
 
-This matches the prior literature that short-horizon reversal is a real but
+This is still a **research-layer approximation**, not the trading simulator. At
-liquidity- and cost-sensitive A-share effect.
+this stage the framework has only an alpha weight file. It has not yet rounded
+shares, checked limits, clipped trades, or paid costs. The purpose is to answer
+a clean signal question: "Do these close-formed weights line up with returns
+over the interval we could actually own after next-open execution?"
 
-## 3. NaN And Look-Ahead Handling
+The daily research return is:
 
-- The raw signal uses `close.pct_change(5, fill_method=None)` — missing prices
+```text
-  are **not** forward-filled, so a suspended name does not silently inherit a
+R[t] = sum_i(weight[i, t] * (open[i, t+2] / open[i, t+1] - 1)) / sum_i(abs(weight[i, t]))
-  stale price.
+```
-- Weights are formed at close `t` and earn the **next** close-to-close return
-  `t → t+1`. Forward returns are computed on the full market calendar *before*
-  selecting signal dates, so a sparse signal grid still earns the next
-  *available* return rather than the next signal date. The final signal date,
-  which has no forward return, is dropped from metrics (that is why the
-  research day count is one less than the stored signal-date count).
-- The liquid-universe mask is applied to the **signal**, not to the price
-  history: `pct_change(5)` is always computed on contiguous prices, and the mask
-  only decides what is *held*. It uses `tradestatus`, `isST`, a ≥60-session
-  seasoning rule, and a trailing-20-day liquidity rank — all backward-looking.
 
-## 4. How Close Are Alpha And Constructed Portfolio?
+This has three important consequences:
 
-`portfolio build` normalizes the alpha to `target_weight = w / Σ|w|` and scales
+- The alpha is normalized by its gross exposure, so the scale of raw weights
-by booksize. The continuous target portfolio is an exact normalization of the
+  does not by itself create a higher return.
-stored alpha (research return correlation ≈ 1.0); the **integer** book then
+- The new signal does not receive credit for the overnight gap from `close[t]`
-diverges because small per-name targets are rounded away under A-share lot
+  to `open[t+1]`, because it cannot be traded until `open[t+1]`.
-rules.
+- The final two signal dates are dropped from performance metrics because they
+  do not have a complete next-open-to-next-open holding interval.
+
+Turnover is still measured from the weights:
+
+```text
+turnover[t] = sum_i(abs(weight[i, t] - weight[i, t-1])) / sum_i(abs(weight[i, t-1]))
+```
+
+The annualized turnover numbers are a warning. Even a positive signal can be
+hard to monetize if it asks the portfolio to trade too much every day.
+
+## Step 7: Build A Portfolio From The Alpha
+
+The alpha file is still an abstract research book. `portfolio build` turns it
+into target exposures and integer shares.
+
+The main normalization is:
+
+```text
+target_weight[i, t] = weight[i, t] / sum_i(abs(weight[i, t]))
+target_value[i, t] = booksize * target_weight[i, t]
+target_shares[i, t] = target_value[i, t] / construction_price[i, t]
+```
+
+Then the framework creates an integer A-share book using lot rules and repair
+logic. This is where a research portfolio starts to become a tradable portfolio.
+
+The continuous target portfolio matched the stored alpha almost exactly:
 
 {closeness}
 
 ![Portfolio tracking](assets/reversal_5d_portfolio_tracking.png)
 
-## 5. How Close Are Alpha Metrics And Final PnL?
+## Step 8: Simulate Execution And Costs
 
-The costless research metric and the simulated net PnL diverge for two
+Research returns are not the same as tradable PnL. The simulator executes the
-mechanical reasons, both quantified below: (a) **execution friction** — next-open
+integer `position_shares` at the next available open and applies constraints:
-fills, integer shares, and constraints; and (b) **cost** — the dominant term
+
-here.
+- suspension;
+- price limit;
+- volume cap;
+- proportional trading cost.
+
+The cost model is:
+
+```text
+cost = abs(traded_shares * open) * (cost_bps + slippage_bps) / 10000
+```
+
+For this run, cost is 5 bps commission plus 5 bps slippage. Slippage is treated
+as cash cost, not as an additional execution price adjustment.
+
+The execution results explain the final research conclusion:
 
 {exec_close}
 
-The research↔execution-net daily-return correlation stays high (the book *does*
+For the liquid rank run, simulated PnL before cost is about
-track the signal), but the level collapses after cost. For the liquid run, gross
+{_money(results['rank_liquid']['execution']['total_pnl_before_cost']) if rliq and results['rank_liquid']['execution'].get('exists') else 'n/a'}, but total
-costless edge is real yet total cost
+cost is about {_money(results['rank_liquid']['execution']['total_cost']) if rliq and results['rank_liquid']['execution'].get('exists') else 'n/a'}. That is why
-(**{_money(results['rank_liquid']['execution']['total_cost']) if rliq and results['rank_liquid']['execution'].get('exists') else 'n/a'}**)
+the final net PnL is weak or negative.
-swamps it. This is the central finding: 5-day reversal is a signal you must trade
+
-*slowly* to monetize.
+This is not a contradiction. It is exactly what a research pipeline should show:
+
+> The signal can exist in the costless layer, but the daily implementation can
+> still trade too much to keep the edge.
 
 ![Execution vs research](assets/reversal_5d_exec_vs_research.png)
 
-## 6. Time Consumption By Phase
+## Step 9: Read The Headline Metrics Like A Researcher
+
+The complete summary is:
+
+{headline}
+
+*Research = costless, no-look-ahead weights over the next-open-to-next-open
+holding interval. Execution = next-open fills on the discretized integer book
+under suspension / price-limit / volume-cap constraints, 5 bps commission + 5
+bps slippage.*
+
+Here is the interpretation:
+
+- **Naive z-score full universe**: positive under open-to-open research, but a
+  less reliable test of the reversal idea because the weighting scheme lets
+  outliers dominate parts of the book.
+- **Rank full universe**: a better test of the same idea, but still noisy
+  because the universe includes too many problematic names.
+- **Rank liquid subset**: the best signal-level test; it finds the cleanest
+  costless reversal effect.
+- **Execution net**: daily rebalancing remains heavily constrained by cost.
+
+A beginner might look only at the final net PnL and say "the alpha failed." A
+researcher should be more precise:
+
+> The raw 5-day reversal idea has signal value in a liquid universe, but the
+> current daily trading rule has too much turnover for the assumed cost model.
+
+## Step 10: Time Consumption By Phase
 
 {timing_tbl}
 
 ![Phase timings](assets/reversal_5d_phase_timings.png)
 
-`portfolio build` dominates because it iterates per signal date and repairs a
+`portfolio build` usually dominates because it iterates per signal date and
-multi-thousand-name integer book under lot rules. The liquid run is faster
+repairs a multi-thousand-name integer book under lot rules. The liquid run is
-across the board because it carries far fewer non-zero names per date.
+faster because it carries fewer non-zero names per date.
 
-## 7. Reproduce The Run
+## Step 11: Reproduce The Experiment
+
+These commands reproduce the important artifacts, assuming the full daily-bar
+dataset already exists at `data/daily_bars/all`.
 
 ```bash
-# naive z-score baseline (full universe) — the built-in alpha, unchanged
+# Naive z-score baseline: built-in reversal alpha, full universe.
 uv run python cli.py alpha compute --data-path data/daily_bars/all \\
-    --alpha-name reversal_5d_all --alpha-type reversal --lookback 5 --output-dir alphas
+    --alpha-name reversal_5d_all --alpha-type reversal --lookback 5 \\
+    --output-dir alphas
 
-# robust rank weighting, full + liquid universe (one script, both runs)
+# Rank-weighted full and liquid runs.
 bash scripts/run_reversal_rank_e2e.sh
 
-# regenerate this report + figures
+# Regenerate figures, diagnostics, and this tutorial report.
 uv run python scripts/generate_reversal_5d_report.py
 ```
 
-## Interpretation & Next Steps
+If you are learning the framework, do not run the whole pipeline blindly. Run
+one phase, inspect the output parquet, then continue.
 
-The pipeline is internally consistent end to end: storage validates, the trivial
+## How To Research Your Own Alpha
-one-alpha combo is an exact identity, the continuous target portfolio matches the
-alpha, and the execution layer cleanly explains the gap to net PnL via friction
-and cost. The premise that 5-day reversal "produces not-bad PnL" holds **at the
-signal level** once weighting and universe are sane (rank + liquid), but **fails
-net of cost** at daily rebalance frequency.
 
-Recommended next diagnostics:
+Use this checklist for a new idea.
 
-- **Turnover control** — the highest-leverage lever: hold bands / no-trade zones,
+1. State the hypothesis in plain language.
-  weight smoothing, or longer rebalance spacing to cut the ~150×/yr turnover.
+   Example: "Stocks with poor 5-day returns may rebound over the next day."
-- Volatility-scaled or decayed reversal to reduce churn.
+
-- Sweep the liquidity cutoff and lookback to map the cost/edge frontier.
+2. Write the raw signal.
+   Implement `signal(close) -> wide DataFrame` in an alpha class. Higher values
+   should mean stronger long preference.
+
+3. Choose the weighting method.
+   The default z-score is useful, but it can be fragile. Consider rank weights,
+   caps, neutralization, or liquidity-aware filters if outliers dominate.
+
+4. Define the investable universe before trusting results.
+   Make sure the strategy is not depending on suspended, ST, newly listed, or
+   illiquid names.
+
+5. Evaluate the alpha as a portfolio, not as a prediction.
+   Check cumulative return, Sharpe, drawdown, hit rate, and turnover over the
+   next-open-to-next-open holding interval. Do not add IC/IR unless the
+   framework's alpha convention changes.
+
+6. Build the portfolio and inspect tracking.
+   Confirm that target weights match the alpha, then check whether integer
+   shares still track the target book.
+
+7. Simulate execution with costs.
+   The final research question is not only "is there a signal?" It is "is there
+   enough signal left after realistic trading?"
+
+8. Diagnose the failure layer.
+   If results are bad, identify whether the problem is the raw signal, weighting,
+   universe, construction, execution constraints, turnover, or cost.
+
+For this 5-day reversal study, the diagnosis is clear: **the signal-level result
+is most promising after robust weighting and a liquid universe filter, but the
+current implementation needs turnover control before it can be considered
+tradable.**
+
+## Next Research Directions
+
+The natural next experiments are:
+
+- Add turnover control: no-trade bands, slower rebalancing, or weight smoothing.
+- Sweep the lookback window: compare 3-day, 5-day, 10-day, and 20-day reversal.
+- Sweep liquidity filters: top 500, top 1000, top 1500 by traded amount.
+- Add position caps so no single name can dominate after normalization.
+- Compare rank weighting with volatility-scaled reversal.
+
+The most important habit is to keep the layers separate. A good alpha research
+workflow does not stop at a single performance number; it explains how the idea
+travels from hypothesis, to signal, to weights, to portfolio, to executable PnL.
 """
 
 
@@ -776,13 +1055,15 @@ def main() -> None:
     DIAGNOSTICS_PATH.parent.mkdir(parents=True, exist_ok=True)
 
     print("loading data ...")
-    data = pd.read_parquet(DATA_PATH, columns=["symbol_id", "date", "close"])
+    data = pd.read_parquet(DATA_PATH, columns=["symbol_id", "date", "open", "close"])
     data["date"] = pd.to_datetime(data["date"])
     data_dates = pd.DatetimeIndex(sorted(data["date"].unique()))
     by_date = data.groupby("date")["symbol_id"].size()
     close = data.pivot_table(index="date", columns="symbol_id", values="close",
                              aggfunc="first").sort_index()
-    fwd = close.pct_change(fill_method=None).shift(-1)
+    open_ = data.pivot_table(index="date", columns="symbol_id", values="open",
+                             aggfunc="first").sort_index()
+    fwd = open_.shift(-2).divide(open_.shift(-1)) - 1.0
     data_summary = {
         "rows": int(len(data)),
         "symbols": int(data["symbol_id"].nunique()),

tests/test_alpha.py

@@ -78,17 +78,17 @@ def test_evaluate_alpha_keys():
         assert key in metrics
 
 
-def test_evaluate_alpha_uses_next_period_returns():
+def test_evaluate_alpha_uses_next_open_to_next_open_returns():
-    dates = pd.date_range("2024-01-01", periods=4)
+    dates = pd.date_range("2024-01-01", periods=5)
     data = pd.concat([
         pd.DataFrame({
             "symbol_id": "sh600000",
             "symbol_name": "sh600000",
             "date": dates,
-            "open": [100.0, 200.0, 200.0, 200.0],
+            "open": [100.0, 100.0, 100.0, 100.0, 200.0],
-            "high": [100.0, 200.0, 200.0, 200.0],
+            "high": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
-            "low": [100.0, 200.0, 200.0, 200.0],
+            "low": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
-            "close": [100.0, 200.0, 200.0, 200.0],
+            "close": [100.0, 1000.0, 1000.0, 1000.0, 1000.0],
             "volume": 1_000.0,
             "amount": 1_000.0,
         }),
@@ -96,10 +96,10 @@ def test_evaluate_alpha_uses_next_period_returns():
             "symbol_id": "sz000001",
             "symbol_name": "sz000001",
             "date": dates,
-            "open": [100.0, 100.0, 200.0, 200.0],
+            "open": [100.0, 100.0, 100.0, 200.0, 200.0],
-            "high": [100.0, 100.0, 200.0, 200.0],
+            "high": [100.0, 10.0, 10.0, 10.0, 10.0],
-            "low": [100.0, 100.0, 200.0, 200.0],
+            "low": [100.0, 10.0, 10.0, 10.0, 10.0],
-            "close": [100.0, 100.0, 200.0, 200.0],
+            "close": [100.0, 10.0, 10.0, 10.0, 10.0],
             "volume": 1_000.0,
             "amount": 1_000.0,
         }),
@@ -114,7 +114,7 @@ def test_evaluate_alpha_uses_next_period_returns():
     metrics = evaluate_alpha(alpha, data)
 
     assert metrics["n_dates"] == 2
-    assert np.isclose(metrics["cumulative_return"], 0.5)
+    assert np.isclose(metrics["cumulative_return"], 1.25)
 
 
 def test_evaluate_alpha_excludes_signal_without_forward_return():
@@ -145,7 +145,7 @@ def test_evaluate_alpha_excludes_signal_without_forward_return():
     ], ignore_index=True)
     alpha = pd.DataFrame({
         "symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
-        "date": [dates[1], dates[1], dates[2], dates[2]],
+        "date": [dates[0], dates[0], dates[1], dates[1]],
         "alpha_name": ["toy"] * 4,
         "weight": [1.0, -1.0, -1.0, 1.0],
     })
@@ -347,4 +347,3 @@ def test_universe_filter_does_not_corrupt_signal_history():
     held = set(filtered.loc[filtered["weight"] != 0.0, "symbol_id"].unique())
     # The two most liquid names (highest amount) are sh600519, sz300750.
     assert held == {"sh600519", "sz300750"}
-

tests/test_portfolio.py

@@ -540,13 +540,13 @@ def test_evaluate_portfolio_keys_no_ic():
 def test_evaluate_portfolio_excludes_signal_without_forward_return():
     dates = pd.date_range("2024-01-01", periods=3)
     data = pd.DataFrame([
-        {"symbol_id": sym, "date": d, "close": price}
+        {"symbol_id": sym, "date": d, "open": price, "close": price}
         for d, prices in zip(dates, [(100.0, 100.0), (100.0, 100.0), (200.0, 100.0)])
         for sym, price in zip(("sh600000", "sz000001"), prices)
     ])
     positions = pd.DataFrame({
         "symbol_id": ["sh600000", "sz000001", "sh600000", "sz000001"],
-        "date": [dates[1], dates[1], dates[2], dates[2]],
+        "date": [dates[0], dates[0], dates[1], dates[1]],
         "portfolio_name": ["run1"] * 4,
         "target_weight": [0.5, -0.5, -0.5, 0.5],
         "target_value": [500.0, -500.0, -500.0, 500.0],