Nested Clustered Optimization
A machine-learning-informed portfolio construction method that uses hierarchical clustering to reduce the dimensionality of the optimization problem and mitigate the instability of traditional mean-variance approaches.
Overview
Nested Clustered Optimization (NCO) addresses the well-known numerical instability of mean-variance optimization by decomposing the full optimization problem into smaller, better-conditioned sub-problems. The method proceeds in three stages: (1) cluster the assets into groups based on their correlation structure, (2) optimize within each cluster independently, and (3) optimize across clusters using the reduced representation. This two-level "nesting" is designed to reduce the effective dimensionality and the condition number of each sub-problem, and with them the instability of the weights.
NCO builds on the Hierarchical Risk Parity (HRP) framework but replaces the recursive bisection allocation step with a proper optimization at each level. The published method admits any convex objective (minimum variance, maximum Sharpe ratio, or risk parity) at both the intra-cluster and inter-cluster stages.
This platform pins one combination. The dispatch passes obj = "MinRisk" and rm = "MV", with Ward linkage, Pearson codependence and the sample covariance of daily returns, on every run, through riskfolio-lib's NCO implementation. So both levels solve a minimum-variance problem, and neither the objective nor the risk measure is a request field. The method needs at least three assets. The description of the method's flexibility below is the published framework, not a set of options you can select.
Step 1: Correlation-Based Distance and Clustering
Distance Metric
The first step converts the correlation matrix into a proper distance metric. The standard angular distance transforms correlations into distances that satisfy the metric axioms (non-negativity, symmetry, triangle inequality):
Perfectly correlated assets () have distance zero, uncorrelated assets () have distance , and perfectly anti-correlated assets () have distance one.
Hierarchical Clustering
Using the distance matrix, agglomerative hierarchical clustering groups assets into clusters. Common linkage methods include single linkage, complete linkage, and Ward's method. The number of clusters can be determined by the gap statistic, silhouette scores, or a fixed threshold on the dendrogram. This platform chooses with a two-difference gap rule, at most 10 and at most one less than the number of assets, and records the cluster count on every run. Each cluster contains a subset of assets that are more correlated with each other than with assets in other clusters.
Step 2: Intra-Cluster Optimization
For each cluster , a standard portfolio optimization is performed using only the assets in that cluster. Let and denote the expected return vector and covariance matrix restricted to the assets in cluster . The intra-cluster optimal weights are:
where is the chosen objective function (e.g., maximum Sharpe ratio, minimum variance, or risk parity). The key advantage is that each cluster contains far fewer assets than the full universe, so the covariance sub-matrix is much smaller and better conditioned.
Cluster-Level Statistics
Once the intra-cluster weights are determined, each cluster is reduced to a single "meta-asset" with the following return and variance:
These meta-asset statistics form the inputs for the inter-cluster optimization. The covariance between meta-assets and is computed as:
where is the cross-covariance block between clusters and .
Step 3: Inter-Cluster Optimization
The inter-cluster optimization determines the allocation across the meta-assets. Let and denote the -dimensional return vector and covariance matrix of the meta-assets. The inter-cluster weights are:
This is a much smaller optimization problem (dimension instead of ), which is faster to solve and far less prone to numerical instability. The same objective function can be used at both levels, or different objectives can be chosen for each stage.
Final Portfolio Weights
The final weight for each individual asset is obtained by combining the inter-cluster allocation with the intra-cluster weights. For asset belonging to cluster :
where is the inter-cluster weight assigned to cluster , and is the intra-cluster weight of asset within cluster . The weights automatically sum to one if both the intra-cluster and inter-cluster weights are fully invested.
Advantages
- Numerical stability: By decomposing into smaller sub-problems, the condition number of each covariance sub-matrix is dramatically reduced, leading to more stable solutions.
- Flexibility in the framework: The published method admits any convex objective at each level, enabling combinations such as risk parity within clusters and maximum Sharpe across clusters. This platform pins minimum variance under the variance risk measure at both levels, so that flexibility is a property of the framework rather than of a run you submit.
- Reduced overfitting: The clustering step acts as a regularizer, grouping similar assets and reducing the effective number of free parameters in the optimization.
- Interpretability: The two-level structure provides a natural decomposition of portfolio decisions into "within-sector" and "across-sector" choices.
Limitations
- Cluster sensitivity: Results depend on the choice of clustering algorithm, linkage method, and number of clusters.
- Non-convexity of clustering: The clustering step introduces a discrete, non-convex component. It is deterministic, but a small change in the correlation estimate can move an asset between clusters, or change the cluster count, and the weights then change by a jump.
- Information loss: Cross-cluster correlations are summarized through meta-assets, potentially losing information about individual asset interactions across clusters.
- Constraints act through the clusters: Per-name weight bounds, asset exclusions, and group sum limits are applied to the final holdings in the outer solve, with the fitted intra-cluster portfolios held fixed. Those portfolios span only part of the market, so constraints the stock universe can meet may be unreachable. The run then reports a restricted span and names the rules the clusters miss and by how much. Because the cluster count depends on the date range, the same universe and rules can be refused over one window and solved over another.
References
- Lopez de Prado, M. & Lewis, M.J. (2019). "Detection of False Investment Strategies Using Unsupervised Learning Methods." Quantitative Finance, 19(9), 1555-1565.
- Lopez de Prado, M. (2020). Machine Learning for Asset Managers. Cambridge University Press, Elements in Quantitative Finance.
- Lopez de Prado, M. (2016). "Building Diversified Portfolios that Outperform Out of Sample." The Journal of Portfolio Management, 42(4), 59-69.
- Lopez de Prado, M. (2019). "A Robust Estimator of the Efficient Frontier." SSRN Working Paper 3469961.
Not investment advice. Past performance is not indicative of future results.