In mathematics, a unitary representation of a group G is a linear representation π of G on a complex Hilbert space V such that π(g) is a unitary operator for every g ∈ G. The general theory is well-developed in the case that G is a locally compact (Hausdorff) topological group and the representations are strongly continuous.

The theory has been widely applied in quantum mechanics since the 1920s, particularly influenced by Hermann Weyl's 1928 book Gruppentheorie und Quantenmechanik. One of the pioneers in constructing a general theory of unitary representations, for any group G rather than just for particular groups useful in applications, was George Mackey.

The theory of unitary representations of topological groups is closely connected with harmonic analysis. In the case of an abelian group G, a fairly complete picture of the representation theory of G is given by Pontryagin duality. In general, the unitary equivalence classes (see below) of irreducible unitary representations of G make up its unitary dual. This set can be identified with the spectrum of the C*-algebra associated with G by the group C*-algebra construction. This is a topological space.

The general form of the Plancherel theorem tries to describe the regular representation of G on L²(G) using a measure on the unitary dual. For G abelian this is given by the Pontryagin duality theory. For G compact, this is done by the Peter–Weyl theorem; in that case, the unitary dual is a discrete space, and the measure attaches an atom to each point of mass equal to its degree.

Let G be a topological group. A strongly continuous unitary representation of G on a Hilbert space H is a group homomorphism from G into the unitary group of H,

such that g → π(g) ξ is a norm continuous function for every ξ ∈ H.

Note that if G is a Lie group, the Hilbert space also admits underlying smooth and analytic structures. A vector ξ in H is said to be smooth or analytic if the map g → π(g) ξ is smooth or analytic (in the norm or weak topologies on H).^[1] Smooth vectors are dense in H by a classical argument of Lars Gårding, since convolution by smooth functions of compact support yields smooth vectors. Analytic vectors are dense by a classical argument of Edward Nelson, amplified by Roe Goodman, since vectors in the image of a heat operator e^–tD, corresponding to an elliptic differential operator D in the universal enveloping algebra of G, are analytic. Not only do smooth or analytic vectors form dense subspaces; but they also form common cores for the unbounded skew-adjoint operators corresponding to the elements of the Lie algebra, in the sense of spectral theory.^[2]

Two unitary representations π₁: G → U(H₁), π₂: G → U(H₂) are said to be unitarily equivalent if there is a unitary transformation A:H₁ → H₂ such that π₁(g) = A^* ∘ π₂(g) ∘ A for all g in G. When this holds, A is said to be an intertwining operator for the representations .^[3]

If is a representation of a connected Lie group on a finite-dimensional Hilbert space , then is unitary if and only if the associated Lie algebra representation maps into the space of skew-self-adjoint operators on .^[4]

A unitary representation is completely reducible, in the sense that for any closed invariant subspace, the orthogonal complement is again a closed invariant subspace. This is at the level of an observation, but is a fundamental property. For example, it implies that finite-dimensional unitary representations are always a direct sum of irreducible representations, in the algebraic sense.

Since unitary representations are much easier to handle than the general case, it is natural to consider unitarizable representations, those that become unitary on the introduction of a suitable complex Hilbert space structure. This works very well for finite groups, and more generally for compact groups, by an averaging argument applied to an arbitrary hermitian structure (more specifically, a new inner product defined by an averaging argument over the old one, w.r.t which the representation is unitary).^[5] For example, a natural proof of Maschke's theorem is by this route.

In general, for non-compact groups, it is a more serious question which representations are unitarizable. One of the important unsolved problems in mathematics is the description of the unitary dual, the effective classification of irreducible unitary representations of all real reductive Lie groups. All irreducible unitary representations are admissible (or rather their Harish-Chandra modules are), and the admissible representations are given by the Langlands classification, and it is easy to tell which of them have a non-trivial invariant sesquilinear form. The problem is that it is in general hard to tell when the quadratic form is positive definite. For many reductive Lie groups this has been solved; see representation theory of SL2(R) and representation theory of the Lorentz group for examples.

• Hall, Brian C. (2015), Lie Groups, Lie Algebras, and Representations: An Elementary Introduction, Graduate Texts in Mathematics, vol. 222 (2nd ed.), Springer, ISBN 978-3319134666
• Reed, Michael; Simon, Barry (1975), Methods of Modern Mathematical Physics, Vol. 2: Fourier Analysis, Self-Adjointness, Academic Press, ISBN 0-12-585002-6
• Warner, Garth (1972), Harmonic Analysis on Semi-simple Lie Groups I, Springer-Verlag, ISBN 0-387-05468-5

• Induced representations
• Isotypical representation
• Representation theory of SL2(R)
• Representations of the Lorentz group
• Stone–von Neumann theorem
• Unitary representation of a star Lie superalgebra
• Zonal spherical function