Data Analytics report

Why GP's TimesFM backtest went flat

Technical diagnosis of the nearly flat GP daily forecast, its accuracy metrics, GPU status, and model-improvement options.

Technical summary

The flat orange line is genuine model output, and this GP configuration should not be treated as a useful trading forecast. Across 598Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json hidden daily closes, TimesFM's MAE was BDT 32.27Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json versus BDT 31.81Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json for simply repeating the last known close, so skill was -1.46%Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json. Direction matched only 46.82%Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json of bars, R-squared was -0.028Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json, and the nominal 80%Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json interval covered only 4.52%Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json of actual closes.

The GPU did not cause the poor shape. This saved run used the CPU because the Tesla P100 was incompatible with the installed PyTorch kernels. GPU execution would primarily change speed; it would not add information, retrain the weights, or turn this long-horizon path into a more accurate one.

Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json

Displayed accuracy

88.63Source: GP run summary calculationTables: read_json_auto, run, dispersion

The displayed score is 100 minus sMAPE; it is not the probability that the forecast direction or price is correct.

sMAPE 11.37Source: GP run summary calculationTables: read_json_auto, run, dispersion
The displayed score is 100 minus sMAPE; it is not the probability that the forecast direction or price is correct.Source: GP run summary calculationTables: read_json_auto, run, dispersion

Loads the saved run's headline metrics and calculates actual-versus-predicted dispersion from its backtest rows.

Skill vs last price

-1.46Source: GP run summary calculationTables: read_json_auto, run, dispersion

Negative skill means TimesFM's MAE is worse than repeating the last observed close for every holdout bar.

TimesFM MAE 32.27Source: GP run summary calculationTables: read_json_auto, run, dispersionBaseline MAE 31.81Source: GP run summary calculationTables: read_json_auto, run, dispersion
Negative skill means TimesFM's MAE is worse than repeating the last observed close for every holdout bar.Source: GP run summary calculationTables: read_json_auto, run, dispersion

Loads the saved run's headline metrics and calculates actual-versus-predicted dispersion from its backtest rows.

Direction matched

46.82Source: GP run summary calculationTables: read_json_auto, run, dispersion

Share of holdout bars for which the predicted up, down, or unchanged sign matched the actual sign.

Share of holdout bars for which the predicted up, down, or unchanged sign matched the actual sign.Source: GP run summary calculationTables: read_json_auto, run, dispersion

Loads the saved run's headline metrics and calculates actual-versus-predicted dispersion from its backtest rows.

80% band coverage

4.52Source: GP run summary calculationTables: read_json_auto, run, dispersion

Only a small fraction of actual closes fell inside the model's nominal Q10-Q90 interval.

Nominal target 80Source: GP run summary calculationTables: read_json_auto, run, dispersion
Only a small fraction of actual closes fell inside the model's nominal Q10-Q90 interval.Source: GP run summary calculationTables: read_json_auto, run, dispersion

Loads the saved run's headline metrics and calculates actual-versus-predicted dispersion from its backtest rows.

Predicted / actual variance

0.1%Source: GP run summary calculationTables: read_json_auto, run, dispersion

The forecast variance is almost zero relative to the actual holdout variance, which is why the orange path looks flat.

Predicted standard deviation 1.12Source: GP run summary calculationTables: read_json_auto, run, dispersionActual standard deviation 36.36Source: GP run summary calculationTables: read_json_auto, run, dispersion
The forecast variance is almost zero relative to the actual holdout variance, which is why the orange path looks flat.Source: GP run summary calculationTables: read_json_auto, run, dispersion

Loads the saved run's headline metrics and calculates actual-versus-predicted dispersion from its backtest rows.

Holdout R-squared

-0.03Source: GP run summary calculationTables: read_json_auto, run, dispersion

A negative value means the forecast explains less holdout variation than predicting the holdout mean in the R-squared calculation.

A negative value means the forecast explains less holdout variation than predicting the holdout mean in the R-squared calculation.Source: GP run summary calculationTables: read_json_auto, run, dispersion

Loads the saved run's headline metrics and calculates actual-versus-predicted dispersion from its backtest rows.

The forecast collapsed toward a level instead of tracking price movement

The hidden actual close ranged from BDT 222.20 to BDT 379.20Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json, a BDT 157.00Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json span. The prediction stayed between BDT 286.56Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json and BDT 293.51Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json, only a BDT 6.96Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json span. Its variance was 0.095%Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json of actual variance, and predicted-versus-actual correlation was -0.30Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json. Zooming in reveals small model movements; zooming out correctly shows that those movements are tiny relative to the stock's realized range.

Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json

Longer evaluation does not reveal durable skill

The comparison below scores prefixes of the same one-origin forecast. TimesFM stays extremely close to the last-price baseline and finishes worse at 598 bars. This is a sensitivity check, not independent rolling validation, but it confirms that the large displayed accuracy score is not evidence of baseline-beating performance.

Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json
MAE by evaluation horizon
MAE by evaluation horizon data
Evaluated holdout barsMAEForecastSkill vs baseline
305.25BDTTimesFM-4.03%
305.05BDTLast-price baseline-4.03%
6026.49BDTTimesFM-1.19%
6026.18BDTLast-price baseline-1.19%
9638.05BDTTimesFM-1.09%
9637.63BDTLast-price baseline-1.09%
12839.54BDTTimesFM-1.04%
12839.13BDTLast-price baseline-1.04%
19241.36BDTTimesFM-0.07%
19241.33BDTLast-price baseline-0.07%
25641.33BDTTimesFM0.47%
25641.53BDTLast-price baseline0.47%
30039.95BDTTimesFM0.8%
30040.28BDTLast-price baseline0.8%
38434.35BDTTimesFM1.23%
38434.78BDTLast-price baseline1.23%
51230.84BDTTimesFM0.19%
51230.89BDTLast-price baseline0.19%
59832.27BDTTimesFM-1.46%
59831.81BDTLast-price baseline-1.46%

The 88.63%Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json headline is a percentage-error transform, not forecast reliability

The application defines accuracy as 100 minus sMAPE. Prices around BDT 250-300 can therefore produce a visually high score even when the model misses path direction and volatility. For model selection, use baseline skill, MAE or MASE, directional performance, and interval calibration together; do not promote this score as a probability of correctness.

Source: Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068File: prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json

Full-holdout metric audit

Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sql

Shapes the reviewed full-holdout metrics and benchmark interpretations into an audit table.

Full-holdout metric audit
#MetricObservedBenchmark or meaningDiagnosis
1Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sqlDisplayed accuracy88.63%Defined as 100 - sMAPENot a probability of a correct price or direction
2Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sqlMean absolute errorBDT 32.27Last-price MAE: BDT 31.81Worse than the naive forecast
3Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sqlSkill vs last price-1.46%Positive is betterNo baseline-beating skill
4Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sqlDirectional accuracy46.82%50% is a rough binary referenceNo useful directional evidence in this run
5Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sqlR-squared-0.03Zero is the holdout-mean referenceExplains no useful holdout variation
6Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sqlQ10-Q90 coverage4.52%Nominal: 80%Severely under-dispersed uncertainty
7Source: GP metric audit rowsFile: diagnostics/gp_metric_audit.sqlPrice rangePredicted BDT 6.96Actual BDT 157.00Forecast path is genuinely flat at the full scale

The test design is much harsher than the command suggests

The command loaded 1,195 daily bars and the default 50/50 split gave the model 597 context bars, then asked it for all 598 hidden bars in one open-loop path from January 23, 2024 to August 11, 2026. The --future-steps 300 option controls the separate forward forecast; it does not limit the backtest to 300 bars. The implementation calls the backtest with the entire holdout length and never feeds hidden actuals back into later predictions.

That design is leakage-safe, but it is not representative of a strategy that refreshes a 5-, 12-, 20-, or 30-day forecast whenever a new close arrives. A very long raw-price forecast tends to accumulate error and regress toward a central level, making an almost-flat path unsurprising.

CPU explains runtime, not accuracy

The saved run used google/timesfm-2.0-500m-pytorch on CPU with normalization enabled and the median quantile as the point forecast. The Tesla P100 has CUDA capability sm_60, while the installed PyTorch build contains kernels starting at sm_70, so the application correctly fell back to CPU. A compatible GPU would make inference faster and is useful for training, but it would not fix this forecast's information, target, horizon, or evaluation design.

The project also uses archived TimesFM 2.0. Google's current TimesFM repository documents TimesFM 2.5, including longer context, improved quantile support, exogenous-regressor support, and a LoRA fine-tuning path. Upgrade and benchmark it; do not assume the newer checkpoint automatically fixes GP. Google's TimesFM model overview also explains that long forecasts are generated in output patches, so later predictions depend on earlier generated values and error can accumulate.

What this diagnosis establishes—and what it does not

This result establishes that this saved GP run and evaluation configuration failed useful baseline and calibration checks. It does not establish that TimesFM is universally poor, nor that another horizon, target, checkpoint, or rolling test will fail. The evidence is one stock and one forecast origin; the current target is the raw close; no sector, index, volume, corporate-action, or macro regressors were supplied; and no transaction-cost trading strategy was evaluated.

Fix evaluation before fine-tuning

  1. Add rolling-origin backtesting. At each historical cutoff, forecast the horizon actually used in decisions—start with 5, 12, 20, and 30 trading days—then advance the cutoff and aggregate results. Keep a final untouched period. A new --backtest-horizon option would make this explicit; --future-steps currently does not.
  2. Benchmark simple forecasts. Compare TimesFM with last price, drift, and a suitable statistical baseline. Report MAE or MASE, skill versus baseline, direction, and Q10-Q90 coverage by horizon and market regime. Demote 100 - sMAPE from the main accuracy label.
  3. Test a stationary target. Compare raw close with log returns or price deltas, reconstruct prices for scoring, and preserve chronological splits. Long raw price levels are non-stationary and encourage level forecasts.
  4. Benchmark TimesFM 2.5 zero-shot before training. It is the maintained line and supports longer context and exogenous regressors. Require repeated out-of-sample improvement rather than relying on one graph.
  5. Use covariates without leakage. Candidate inputs include DSEX or DS30, sector movement, volume and turnover, calendar features, corporate-action flags, rates, and FX. Any future covariate must be known in advance or separately forecast as a scenario.
  6. Fine-tune only with pooled data. One GP history of 1,195 observations is too small for a convincing 500M-model adaptation. Use sliding windows across many DSE equities and regimes, match training horizons to production, and use LoRA or QLoRA with strict ticker/time holdouts. Google's official fine-tuning guide documents this path for TimesFM 2.5.

Promote a model only after it beats the baseline repeatedly

A replacement should show positive out-of-sample skill across multiple rolling folds and several DSE symbols, direction performance that is stable rather than a one-run fluctuation, and uncertainty coverage reasonably close to its nominal level. If the intended use is trading, the last gate is a transaction-cost- and slippage-aware strategy test. Better-looking forecast lines are not enough.

Further questions

  • What holding period will the forecast support: 1 day, 5 days, 12 days, or a longer allocation horizon?
  • Are historical prices adjusted for dividends, splits, rights issues, and other corporate actions?
  • Which DSE symbols and market regimes should form the pooled training and final evaluation universe?
  • Which exogenous variables are available at prediction time without future leakage?

Sources

  1. Saved GP prediction run gp-1d-20260812T053147Z-f9ed7068prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json
  2. GP run summary calculationdiagnostics/gp_run_summary.sql · duckdb · 2026-08-12T06:15:00Z

    Loads the saved run's headline metrics and calculates actual-versus-predicted dispersion from its backtest rows.

    SQL query
    WITH run AS (SELECT * FROM read_json_auto('prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json')), dispersion AS (SELECT var_samp(b.predicted::DOUBLE) / var_samp(b.actual::DOUBLE) AS variance_ratio, stddev_samp(b.predicted::DOUBLE) AS predicted_std_bdt, stddev_samp(b.actual::DOUBLE) AS actual_std_bdt FROM run, UNNEST(backtest) AS t(b)) SELECT r.metrics.accuracy_score AS accuracy_score_pct, r.metrics.smape_percent AS smape_pct, r.metrics.skill_vs_naive_percent AS skill_vs_naive_pct, r.metrics.mae AS timesfm_mae_bdt, r.metrics.naive_mae AS naive_mae_bdt, r.metrics.directional_accuracy_percent AS direction_accuracy_pct, r.metrics.interval_80_coverage_percent AS coverage_80_pct, 80.0 AS coverage_nominal_pct, d.variance_ratio, d.predicted_std_bdt, d.actual_std_bdt, r.metrics.r_squared FROM run r CROSS JOIN dispersion d
  3. GP horizon sensitivity calculationdiagnostics/gp_horizon_mae.sql · duckdb · 2026-08-12T06:15:00Z

    Calculates TimesFM and constant-last-price MAE over selected prefixes of the same ordered holdout forecast.

    SQL query
    WITH run AS (SELECT * FROM read_json_auto('prediction-runs/gp-1d-20260812T053147Z-f9ed7068.json')), ordered AS (SELECT row_number() OVER (ORDER BY b.time) AS horizon_bars, b.actual::DOUBLE AS actual, b.predicted::DOUBLE AS predicted FROM run, UNNEST(backtest) AS t(b)), prefixes AS (SELECT horizon_bars, avg(abs(predicted - actual)) OVER (ORDER BY horizon_bars ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS timesfm_mae_bdt, avg(abs(286.6 - actual)) OVER (ORDER BY horizon_bars ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS last_price_mae_bdt FROM ordered), selected AS (SELECT *, (1 - timesfm_mae_bdt / NULLIF(last_price_mae_bdt, 0)) * 100 AS skill_vs_naive_pct FROM prefixes WHERE horizon_bars IN (30,60,96,128,192,256,300,384,512,598)) SELECT horizon_bars, 'TimesFM' AS series, timesfm_mae_bdt AS mae_bdt, 'solid' AS line_style, skill_vs_naive_pct FROM selected UNION ALL SELECT horizon_bars, 'Last-price baseline' AS series, last_price_mae_bdt AS mae_bdt, 'dashed' AS line_style, skill_vs_naive_pct FROM selected ORDER BY horizon_bars, series DESC
  4. GP metric audit rowsdiagnostics/gp_metric_audit.sql · duckdb · 2026-08-12T06:15:00Z

    Shapes the reviewed full-holdout metrics and benchmark interpretations into an audit table.

    SQL query
    SELECT * FROM (VALUES (1, 'Displayed accuracy', '88.63%', 'Defined as 100 - sMAPE', 'Not a probability of a correct price or direction'), (2, 'Mean absolute error', 'BDT 32.27', 'Last-price MAE: BDT 31.81', 'Worse than the naive forecast'), (3, 'Skill vs last price', '-1.46%', 'Positive is better', 'No baseline-beating skill'), (4, 'Directional accuracy', '46.82%', '50% is a rough binary reference', 'No useful directional evidence in this run'), (5, 'R-squared', '-0.028', 'Zero is the holdout-mean reference', 'Explains no useful holdout variation'), (6, 'Q10-Q90 coverage', '4.52%', 'Nominal: 80%', 'Severely under-dispersed uncertainty'), (7, 'Price range', 'Predicted BDT 6.96', 'Actual BDT 157.00', 'Forecast path is genuinely flat at the full scale')) AS t(rank, metric, observed, benchmark, diagnosis) ORDER BY rank
  5. Local TimesFM forecasting implementationdohasecuritiesstockai/timesfm_forecasting/pipeline.py · source-code

    Local forecasting pipeline, TimesFM backend, and metric definitions inspected for split, horizon, device, normalization, and scoring behavior.