HDBSCAN: Definition & Examples

HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) finds dense groups by building a hierarchy across distance scales, then selecting persistent clusters while allowing observations to remain unassigned as noise. A quant can use it to explore assets with similar measured volatility, liquidity or return behaviour without fixing the number of peer groups in advance. The groups describe the supplied features and sample; they do not establish economic regimes or predict returns.

It is an unsupervised learning method for clustering. Its density-based geometry can accommodate non-spherical groups and differing densities. This differs from k-means, which seeks compact groups around means for a chosen cluster count. The k-Means Clustering lesson develops that comparison method's assignments and objective (public preview; full lesson requires membership).

Make sparse connections expensive

Ordinary single linkage joins groups through their closest pair of points. Sparse bridges can therefore connect otherwise distinct regions. HDBSCAN first adjusts pairwise distances using local neighbourhood information.

Let

d(x_i,x_j)

be the chosen distance and

c_k(x_i)

the distance to the

k

th nearest other observation. For the usual unscaled construction, mutual reachability is

d_{\mathrm{mr}}(x_i,x_j)=\max\{c_k(x_i),c_k(x_j),d(x_i,x_j)\}.

A connection must be at least as expensive as either endpoint's core distance. Large core distances make sparse observations harder to connect. A minimum spanning tree (MST) of these weights preserves the connected components obtained as the distance threshold changes. This is the graph construction described in the implementers' explanation.

Count neighbours explicitly. Here

k

excludes the observation itself. The current scikit-learn HDBSCAN documentation says its min_samples includes self, whereas scikit-learn-contrib's hdbscan does not: matching the neighbourhood count requires one more in scikit-learn. Thus nearest-other core distances correspond to min_samples=2 in scikit-learn and min_samples=1 in the contrib implementation. This convention mapping alone does not specify a complete clustering run.

Four points, three connecting edges

Take synthetic dimensionless scalar features

0,1,3,10

, with absolute distance and

k=1

other neighbour. Their nearest-other distances are:

Observation	Nearest other observation	Core distance
$0$	$1$	$1$
$1$	$0$	$1$
$3$	$1$	$2$
$10$	$3$	$7$

For example,

d_{\mathrm{mr}}(1,3)=\max\{1,2,2\}=2, \qquad d_{\mathrm{mr}}(0,10)=\max\{1,7,10\}=10.

The six pair weights are $1,3,10,2,9,7$ for the pairs $(0,1)$ , $(0,3)$ , $(0,10)$ , $(1,3)$ , $(1,10)$ , $(3,10)$ respectively. An MST uses adjacent edges of weight $1,2,7$ , with total weight $10$ .

Four synthetic dimensionless features at 0, 1, 3 and 10 have nearest-other core distances 1, 1, 2 and 7. Their adjacent minimum-spanning-tree edges have mutual-reachability weights 1, 2 and 7. The uncondensed hierarchy merges 0 with 1 at threshold 1, adds 3 at 2, and adds 10 at 7.

Original synthetic graph construction. The lower panel shows threshold connectivity before condensation or cluster selection; branch colours do not represent final HDBSCAN labels.

Retain edges whose weights are at most a threshold $\epsilon$ . At $\epsilon=1.5$ , the components are $\{0,1\}$ , $\{3\}$ and $\{10\}$ . At $\epsilon=3$ , they are $\{0,1,3\}$ and $\{10\}$ . At $\epsilon=7$ , all four connect. Reversing this process gives successive splits as the threshold falls.

In this deliberately simple example, mutual reachability leaves every distinct-pair distance unchanged: an observation's nearest-other distance cannot exceed its distance to any particular other observation. With larger neighbourhood counts, the transform can inflate pair distances. The example checks the construction and its conventions; it supplies no selected clusters, noise labels or stability scores.

A hierarchy still needs a selection rule

HDBSCAN condenses the hierarchy using min_cluster_size. When a split leaves one branch below that size, the small branch is recorded as points leaving the continuing cluster. Two sufficiently large branches create separate candidates; if neither is large enough, the parent ends.

For positive distance levels, write $\lambda=1/\epsilon$ . A candidate's stability sums its members' persistence after the candidate's birth:

S(C)=\sum_{i\in C}\bigl(\lambda_i-\lambda_{\mathrm{birth}}(C)\bigr),

where

\lambda_i

is the level at which observation

i

leaves that candidate, including when the candidate splits. Excess-of-mass selection compares a parent's stability with the best combined stability below it. Selecting a parent excludes its descendants from the flat partition. Leaf selection instead takes terminal candidates and can give finer groups. Malzer and Baum, §III, describe condensation and both selection approaches.

Here “stable” means persistence across density levels within this fitted sample. It does not mean stable through calendar time, robust to changed features or statistically significant. min_cluster_size determines eligible group size; min_samples controls the neighbourhood used for core distances. If min_samples defaults to min_cluster_size, changing the latter also changes the density construction. Hold the former explicit when isolating size sensitivity. See the parameter-selection guide.

Read the output in its proper units

Finite observations outside selected clusters can receive the noise label -1. That is a statement about this fit's geometry, not evidence of bad data or an untradeable asset. A rare but meaningful market episode can be unassigned; erroneous observations can also form a dense group.

The attribute probabilities_ measures membership strength in the assigned cluster. It is not a calibrated probability of a bull market, default or profitable trade. The contrib library's separate soft-membership vectors offer another view of affiliation; they should not be confused with the single assigned-cluster strength. The API and soft-clustering documentation distinguish these outputs.

For market research, define each row first: one asset at a dated snapshot, or one trading day represented by trailing features. Raw turnover can dominate a distance built alongside decimal volatility; scaling and the distance metric determine the question being asked. Examine groups in original units and check sensitivity to feature choices, neighbourhood count and estimation window.

If the groups enter a historical strategy test, fit preprocessing and clustering using information available at each decision date. Full-history scaling or cluster fitting introduces look-ahead bias, despite the absence of target labels. This applies the training-only preprocessing rule to dated financial observations.

Finally, distinguish refitting from assignment to a frozen fit. Adding observations can change HDBSCAN's hierarchy. The contrib library's approximate_predict assigns new points while holding existing clusters fixed; it cannot discover new clusters that a full refit might reveal. Declare the update rule, validate chronologically and compare memberships rather than assuming numeric cluster IDs retain their meaning across refits.

Sources

HDBSCAN developers, How HDBSCAN Works: core distances, mutual reachability and the minimum spanning tree.
Claudia Malzer and Marcus Baum, A Hybrid Approach To Hierarchical Density-based Cluster Selection, §III: condensed hierarchy, stability and cluster selection; author manuscript, revised 21 January 2021.
Scikit-learn developers, HDBSCAN API: parameters, membership strength and the self-counting convention; stable documentation checked 29 October 2026.
HDBSCAN developers, Parameter Selection for HDBSCAN, Soft Clustering and Predicting clusters for new points: parameter interaction, distinct membership outputs and frozen-cluster assignment.
Scikit-learn developers, Common pitfalls: data leakage: fit transformations on training observations only.

Make sparse connections expensive

Four points, three connecting edges

A hierarchy still needs a selection rule

Read the output in its proper units

Sources

Related terms

Clustering

Look-ahead bias

Unsupervised learning