Linear Algebra

Singular Value Decomposition

The one decomposition that works on any matrix whatsoever. What each of its three factors is doing.

Updated May 2026

Diagonalisation needs a square matrix and a cooperative one. The SVD needs nothing:

A = U Σ V^{†}

for any $m \times n$ matrix, with $U$ and $V$ unitary and $Σ$ diagonal with non-negative entries $σ_{1} \geq σ_{2} \geq \dots \geq 0$ .

The geometry

Every linear map, read right to left, is: rotate ( $V^{†}$ ), scale along the axes ( $Σ$ ), rotate again ( $U$ ).

Equivalently: every matrix maps the unit sphere to an ellipsoid. The singular values are the semi-axis lengths and the columns of $U$ are their directions. There are no exceptions and no special cases.

Fig. 1

The picture behind the algebra. $V$ picks which directions of the sphere to measure, $Σ$ stretches them by the singular values, and $U$ says where those stretched axes end up.

Where it comes from

$A^{†} A$ is Hermitian and positive semi-definite whatever $A$ was, so the spectral theorem applies to it. Its eigenvectors are the columns of $V$ , and $σ_{i} = λ_{i}$ . The SVD is the spectral theorem laundered through a matrix that is guaranteed to be well behaved.

Low-rank approximation

Theorem

Eckart–Young

Truncating the SVD after $k$ terms,

A_{k} = i = 1 \sum k σ_{i} u_{i} v_{i}^{†},

gives the best rank- $k$ approximation to $A$ in both the spectral and Frobenius norms. No other rank- $k$ matrix is closer.

That optimality is what makes the SVD ubiquitous. Image compression keeps the large $σ_{i}$ and discards the rest. Principal component analysis is the SVD of a centred data matrix. Latent semantic analysis is the SVD of a term–document matrix. In each case the singular values rank directions by how much variance they carry, and the tail is usually noise.

Conditioning

κ (A) = \frac{σ _{m a x}}{σ _{m i n}}

The condition number bounds how much a relative error in the input can be amplified in the output. A large $κ$ means the ellipsoid is a long thin sliver and the inverse is dominated by whichever direction was squashed most.

Practice

The pseudoinverse

When $A$ is singular or rectangular, $A^{+} = V Σ^{+} U^{†}$ , inverting the non-zero singular values and transposing, gives the least-squares solution of minimum norm. Truncating small $σ_{i}$ before inverting is regularisation, and it is the standard fix for an ill-posed inverse problem such as deconvolving a microscope image.

Why it beats eigendecomposition in practice

Eigenvectors of a non-symmetric matrix can be nearly parallel, which makes the change of basis numerically hostile. The SVD's bases are orthonormal by construction, so the decomposition is backward stable: the computed answer is the exact answer to a slightly perturbed problem.

For anything involving measured data, that stability is worth more than the extra structure diagonalisation would have given.