Folio Lab/Documentation

HMM Regime MVO

Markov-switching regime-conditioned mean-variance optimisation. Returns are modelled as draws from a small set of latent regimes (typically two: a calm regime and a stress regime), and the optimiser uses regime-dependent moments to construct a portfolio that adapts to the prevailing market state.

Overview

Equity-return distributions are not stationary: long calm periods are punctuated by short, violent stress episodes with very different volatility, correlation, and skewness profiles. A single sample mean and a single sample covariance fitted across the full history will average over these regimes and produce moments that are wrong for both. The Markov-switching framework of Hamilton (1989, 1990) provides a principled treatment: a hidden state variable governs which of regimes is active at each time, and the asset returns are conditionally Gaussian given the regime.

Ang and Bekaert (2002) applied the framework to international asset allocation; Guidolin and Timmermann (2007) studied multi-regime allocation with Bayesian inference; Nystrup, Madsen and Lindstrom (2018) gave the formulation closest to the FolioLab implementation, where regime probabilities are estimated by EM and used to compute regime-mixed moments for the MVO step.

The most important practical question is the number of regimes. Two regimes (calm and stress) are interpretable and statistically identifiable on most equity samples. Three regimes (calm, transition, stress) sometimes give better in-sample fit but are harder to identify out of sample. FolioLab defaults to a two-state HMM but exposes the number-of-regimes parameter for research use.

Mathematical Formulation

Model

  • — latent regime at time
  • — transition probabilities
  • — conditional Gaussian returns
  • — smoothed regime probability at the optimisation date

EM estimation

The parameters are estimated by the Expectation-Maximisation algorithm: the Forward-Backward (Baum-Welch) recursions compute the smoothed regime probabilities given the parameters, and the M-step updates each regime's mean and covariance as the probability-weighted sample moments. EM converges in a few dozen iterations for two-state models and is implemented via hmmlearn.

Regime-mixed moments

The optimisation-date moments are mixtures of the regime-specific moments, weighted by the smoothed regime probabilities. The covariance term picks up both the within-regime variance and the between-regime variance — the latter is exactly what a single-regime estimator misses.

Plug-in MVO

The mixture moments , are substituted directly into the standard MVO problem, yielding regime-adapted weights:

When the smoother indicates the calm regime is dominant the portfolio looks like a typical MVO; when it indicates the stress regime is dominant the portfolio shifts toward defensive names and lower vol-uncorrelated clusters.

Identifying regimes on Indian equities

On Nifty / Sensex daily returns, a two-state HMM typically identifies a high-mean low-vol calm regime and a low-mean high-vol stress regime. Stress regimes correspond to identifiable historical episodes (the 2008 crisis, the 2013 taper tantrum, the 2020 COVID drawdown, the 2022 risk-off). Transition probabilities are usually highly persistent, with regime durations from weeks to several months.

Out-of-sample regime classification is the hardest part. The smoother is fit on the full sample (in-sample); a true out-of-sample regime probability uses the filter (one-sided) and lags the smoothed estimate by the regime's persistence horizon. FolioLab uses the filtered regime probability when constructing the rolling-backtest weights to avoid look-ahead.

Advantages & Limitations

Advantages

  • Adaptive moments: Captures regime structure that single-window estimators miss.
  • Interpretable states: Two-regime models map cleanly to calm vs stress.
  • Principled mixture covariance: Includes between-regime variance.
  • Plug-in compatible: Drops into any downstream MVO formulation.

Limitations

  • Identifiability: EM can fall into local optima, especially with many regimes.
  • Latency: Regime changes are detected with a lag.
  • Gaussian regimes: Within-regime distribution is still parametric.
  • Look-ahead risk: Smoothed probabilities use future data; only filtered probabilities are out-of-sample valid.

References

  • Hamilton, J. D. (1989). "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle." Econometrica, 57(2), 357-384.
  • Hamilton, J. D. (1990). "Analysis of Time Series Subject to Changes in Regime." Journal of Econometrics, 45(1-2), 39-70.
  • Ang, A., & Bekaert, G. (2002). "International Asset Allocation with Regime Shifts." The Review of Financial Studies, 15(4), 1137-1187.
  • Guidolin, M., & Timmermann, A. (2007). "Asset Allocation under Multivariate Regime Switching." Journal of Economic Dynamics and Control, 31(11), 3503-3544.
  • Nystrup, P., Madsen, H., & Lindstrom, E. (2018). "Dynamic Portfolio Optimization across Hidden Market Regimes." Quantitative Finance, 18(1), 83-95.
  • Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970). "A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains." The Annals of Mathematical Statistics, 41(1), 164-171.