Concept

Spectral Clustering

Spectral Clustering is a Machine Learning concept.

What is spectral clustering?

This is a graph-based grouping technique: instead of partitioning data points by their raw distance from center points, it first builds a similarity graph, where each observation is a node and edge weights measure how alike two observations are, then examines the eigenvectors of that graph's Laplacian matrix to find the natural cuts. The eigenvectors embed the data into a low-dimensional space in which the groups become nearly linearly separable, and a simple partitioning step in that space finishes the job. The spectral part of the name refers to the eigenvalue spectrum of the Laplacian.

The method's advantage is shape-agnosticism. Center-based approaches such as k-means regime detection implicitly assume compact, roughly spherical groups; the spectral approach only requires that members of a group be connected through chains of similar neighbors, so it can separate elongated, curved, or intertwined structures that center-based methods split incorrectly. In market work the observations are typically feature vectors describing periods, days summarized by volatility, trend, correlation, and breadth readings, or instruments summarized by return behavior, and the goal is to discover regimes or families of instruments without imposing their shape in advance.

The costs are real. Results depend heavily on how similarity is defined (the kernel and its bandwidth, or the number of nearest neighbors in the graph), the eigendecomposition scales poorly to very large datasets, and, as with all unsupervised grouping, the number of groups is an input rather than an output, though the eigenvalue gaps offer a heuristic for choosing it.

How it's calculated

The standard normalized form, following Ng, Jordan, and Weiss (2002), proceeds from a similarity matrix to an eigenvector embedding.

W_ij = exp(-||x_i - x_j||^2 / (2 * sigma^2))
D = diag(sum_j W_ij)
L_sym = I - D^(-1/2) * W * D^(-1/2)
U = matrix of eigenvectors of L_sym for the k smallest eigenvalues
rows of U are normalized to unit length, then partitioned into k groups with k-means
x_i: feature vector for observation i
W: similarity (affinity) matrix; a k-nearest-neighbor graph is a common alternative to the Gaussian kernel
sigma: kernel bandwidth controlling how quickly similarity decays with distance
D: degree matrix, the diagonal of row sums of W
L_sym: symmetric normalized graph Laplacian; I is the identity matrix
k: number of groups, chosen in advance or from the largest gap in the sorted eigenvalues

Variants differ in the Laplacian used (unnormalized, random-walk, or symmetric) and in the final partitioning step; the normalized forms are generally preferred.

The eigengap heuristic reads a large jump between consecutive eigenvalues as evidence for that number of groups.

How traders use it

  • Regime discovery: days or weeks described by volatility, trend, and breadth features are grouped, and the resulting labels condition which strategies are allowed to trade, the same role served by Markov-switching models with different assumptions.
  • Grouping instruments: building the similarity graph from return correlations partitions a universe into families that move together, useful for diversification checks and for pair selection ahead of a pairs trading workflow.
  • Handling odd shapes: when a scatter of regime features forms bands or arcs rather than blobs, the spectral approach separates them where center-based methods fail; when the groups really are compact blobs, simpler methods do the same job faster.
  • Practical caveats: results should be checked for stability across similarity settings and time windows, since a grouping that reshuffles whenever sigma changes slightly is describing noise, and labels fitted on full history leak information if used in a backtest without walk-forward refitting.

Spectral grouping vs. other unsupervised methods

K-Means Regime Detection: K-means assigns points to the nearest centroid, which assumes compact spherical groups but scales easily and is simple to maintain. The spectral method handles arbitrary shapes at higher computational cost and with more sensitive settings.

Gaussian Mixture Models: A mixture model fits overlapping Gaussian components and returns soft membership probabilities, which suit gradual regime transitions. The spectral approach makes no distributional assumption but yields hard assignments and no likelihood to reason with.

Self-Organizing Maps: A SOM projects data onto a 2D grid for visualization, preserving neighborhood relations approximately. The spectral method targets a clean partition rather than a map, and offers eigenvalue structure as guidance on how many groups exist.

Concept family

Machine Learning

32 concepts mapped · 32 in the Library

Spectral Clustering FAQ

When should I prefer this method over k-means?

When the groups in feature space are non-convex, elongated, or intertwined, or when a similarity graph is more natural than Euclidean distance, as with correlation-based instrument grouping. If the groups are compact blobs, k-means gives similar answers with far less machinery.

How do I choose the number of groups?

The eigengap heuristic looks for a large jump in the sorted Laplacian eigenvalues and takes the count before the jump. It is a guide, not a rule; on noisy market features the gap is often ambiguous, and stability across time windows is the better tiebreaker.

What matters more, the algorithm or the features?

The features and the similarity definition. The eigendecomposition faithfully partitions whatever graph it is given, so a poorly chosen kernel bandwidth or unscaled features produce confident nonsense. Most practical effort goes into the affinity construction.

Can this be run in real time on a chart?

Not naturally. The eigendecomposition is batch computation over a window of history, so charting implementations typically refit periodically and hold labels fixed between refits. Any tool that relabels past bars on each refit is repainting and can mislead backtests.

Build Spectral Clustering your way.

Quant writes, tests, and refines it with you — then it runs on LuxAlgo charting or ports to TradingView.