Add minute bar feature pipeline
This commit is contained in:
@@ -0,0 +1,67 @@
|
||||
# Minute Bar Data Notes
|
||||
|
||||
The minute-bar path downloads raw Baostock intraday bars and stores them as a
|
||||
Hive-partitioned dataset:
|
||||
|
||||
```bash
|
||||
uv run python cli.py data download-minute \
|
||||
--universe sh600000 \
|
||||
--start-date 2024-01-02 --end-date 2024-01-05 \
|
||||
--frequency 5
|
||||
```
|
||||
|
||||
The default layout is:
|
||||
|
||||
```text
|
||||
data/minute_bars/{universe}/frequency=5m/month=YYYY-MM/*.pq
|
||||
```
|
||||
|
||||
Feature plugins can aggregate those bars to daily `symbol_id,date` feature
|
||||
files, for example:
|
||||
|
||||
```bash
|
||||
uv run python cli.py feature compute \
|
||||
--minute-path data/minute_bars/sh600000 \
|
||||
--daily-path data/daily_bars/sh600000 \
|
||||
--feature-type minute_daily_summary \
|
||||
--feature-name minute_summary
|
||||
```
|
||||
|
||||
## Daily vs Minute Reconciliation
|
||||
|
||||
Baostock's daily raw bars and 5-minute raw bars are close, but they should not
|
||||
be treated as perfectly reconstructible from each other.
|
||||
|
||||
When checking consistency, compare daily raw bars (`data download --adjust none`)
|
||||
against minute bars on the same raw price scale. The minute aggregation should
|
||||
use:
|
||||
|
||||
- `open`: first minute open
|
||||
- `high`: max minute high
|
||||
- `low`: min minute low
|
||||
- `close`: last minute close
|
||||
- `volume`: sum minute volume
|
||||
- `amount`: sum minute amount
|
||||
- `vwap`: `sum(amount) / sum(volume)`
|
||||
|
||||
In a sanity check for `sh600000` from `2024-01-02` through `2024-01-05`, Baostock
|
||||
returned 4 daily rows and 192 5-minute bars, exactly 48 bars per day. Open, low,
|
||||
and close matched daily exactly on all 4 days. High matched on 3 of 4 days; on
|
||||
`2024-01-04`, the daily high was `6.67` while the max 5-minute high was `6.66`.
|
||||
Minute-summed volume and amount were higher than daily by roughly `0.16%` to
|
||||
`1.23%`. VWAP remained very close, with max relative difference around
|
||||
`0.0043%`.
|
||||
|
||||
This appears to be a source-level Baostock reconciliation caveat, not a parser
|
||||
or ordering issue: the minute bars covered the regular `09:35:00` through
|
||||
`15:00:00` range and sorted correctly by timestamp.
|
||||
|
||||
Practical guidance:
|
||||
|
||||
- Use tolerance-based daily-vs-minute checks; do not require exact equality for
|
||||
high, volume, or amount.
|
||||
- Expect open/close alignment to be a stronger sanity check than exact volume
|
||||
reconstruction.
|
||||
- Use minute-derived values as separate daily features, not as replacements for
|
||||
the canonical daily bar dataset unless a strategy explicitly wants that
|
||||
source convention.
|
||||
Reference in New Issue
Block a user