Research reference only. Same evaluation discipline as the live volatility line; different conclusion. Volatility remains in production. This direction experiment does not.
No usable edge after costs → no production freeze for direction. Do not auto-update or promote a live direction model under the current open-to-close specification. Negative results are part of the research process.
Walk-forward only (train through 2018; predict 2019 onward). Baselines: always-up, momentum (prior close return sign), gap-follow, gap-fade. Primary bar: beat always-up by a meaningful margin on accuracy and probability scores, then check open-to-close return differentials after cost haircuts. Stress months aligned with the vol study: 2020-03, 2020-04, 2022-06, 2022-10.
Under these constraints, pure direction is a low-signal problem. Volatility has a stronger statistical signature; open-to-close sign prediction does not clear the same bar.
| Metric | Tree | Always-up / climate | Note |
|---|---|---|---|
| Accuracy | 0.528 | 0.546 | −1.8 pp vs always-up |
| Brier score | 0.253 | 0.248 (climate) | Worse than base-rate forecast |
| Log-loss | 0.699 | - | AUC ≈ 0.53 |
| Mean OC ret | pred up | +0.049%/day | - | n = 1126 |
| Mean OC ret | pred down | +0.004%/day | - | n = 778 |
| Long−short (E[r|up]−E[r|down]) | ~+0.045 pp/day | gross | Does not survive realistic costs |
| L/S net @ ~5 bps/day | negative / ~0 | - | Not a trading edge |
Logistic regression was similar (accuracy 0.533, −1.3 pp vs always-up). Confidence thresholds (|P − 0.5| > δ) did not produce a stable, high-coverage rule with a usable edge after costs. A weak ranking of open-to-close returns by score is measurable; it is not framed here as a trading edge.
No usable edge after costs → no production freeze for direction. The experiment stops here under the current daily open-to-close specification. The live SPY system remains the short-horizon realized-volatility forecast board.
Martial Systems LLC. For investor or partnership inquiries, email martialsys@gmail.com.
