Your Neural Network Backtest Lies by ±10 Percentage Points

The author set up a rigorous walk-forward backtesting pipeline with hourly data from 20 most liquid Moscow Exchange stocks (2021-2026, ~370,000 bars), using a GRU recurrent network, binary classification for price rise over a horizon, and realistic broker fees. They ensured no look-ahead leakage by adding a gap between train and test, executing at next-bar price, and including commissions. Despite this, changing only the random seed changed annual returns from +6% to +15%, and in five identical runs with different seeds Sharpe ratios ranged from -0.12 to +0.85 (later 0.39 to 1.28). This happens because the financial signal is tiny (AUC 0.54-0.55), and the loss landscape is nearly flat, so random initialization determines overfitting patterns. The author recommends: (1) always use an ensemble of models with different seeds (averaging probabilities), which boosted Sharpe to 1.88 versus best single 1.28; (2) compare new ideas ensemble vs ensemble, not single runs; (3) perform attribution to see if the gain comes from securities where the feature actually exists. They give three cases where apparent improvements vanished after attribution, proving the gains were just reshuffled noise. On a frozen system, testing on 2015-2020 data yielded -28% without retraining, but +17% with full walk-forward retraining, concluding that regular retraining is essential for adapting to market regime changes. The author plans to trade real money soon and will publish public track record.