The theory of unitary representations of groups is closely connected with harmonic analysis. In the case of an abelian groupG, a fairly complete picture of the representation theory of G is given by Pontryagin duality. In general, the unitary equivalence classes of irreducible unitary representations of G make up its unitary dual. This set can be identified with the spectrum of the C*-algebra associated to G by the group C*-algebra construction. This is a topological space. The general form of the Plancherel theorem tries to describe the regular representation of G on L2 by means of a measure on the unitary dual. For G abelian this is given by the Pontryagin duality theory. For Gcompact, this is done by the Peter–Weyl theorem; in that case the unitary dual is a discrete space, and the measure attaches an atom to each point of mass equal to its degree.
Formal definitions
Let G be a topological group. A strongly continuous unitary representation of G on a Hilbert space H is a group homomorphism from G into the unitary group of H, such that g → π ξ is a norm continuous function for every ξ ∈ H. Note that if G is a Lie group, the Hilbert space also admits underlying smooth and analytic structures. A vector ξ in H is said to be smooth or analytic if the map g → π ξ is smooth or analytic. Smooth vectors are dense in H by a classical argument of Lars Gårding, since convolution by smooth functions of compact support yields smooth vectors. Analytic vectors are dense by a classical argument of Edward Nelson, amplified by Roe Goodman, since vectors in the image of a heat operator e–tD, corresponding to an elliptic differential operatorD in the universal enveloping algebra of G, are analytic. Not only do smooth or analytic vectors form dense subspaces; they also form common cores for the unbounded skew-adjoint operators corresponding to the elements of the Lie algebra, in the sense of spectral theory. Two unitary representations π1: G → U, π2: G → U are said to be unitarily equivalent if there is a unitary transformationA:H1 → H2 such that π1 = A* ∘ π2 ∘ A for all g in G. When this holds, A is said to be an intertwining operator for the representations,. If is a representation of a connected Lie group on a finite-dimensional Hilbert space, then is unitary if and only if the associated Lie algebra representation maps into the space of skew-self-adjoint operators on.
Complete reducibility
A unitary representation is completely reducible, in the sense that for any closed invariant subspace, the orthogonal complement is again a closed invariant subspace. This is at the level of an observation, but is a fundamental property. For example, it implies that finite-dimensional unitary representations are always a direct sum of irreducible representations, in the algebraic sense. Since unitary representations are much easier to handle than the general case, it is natural to consider unitarizable representations, those that become unitary on the introduction of a suitable complex Hilbert space structure. This works very well for finite groups, and more generally for compact groups, by an averaging argument applied to an arbitrary hermitian structure. For example, a natural proof of Maschke's theorem is by this route.
Unitarizability and the unitary dual question
In general, for non-compact groups, it is a more serious question which representations are unitarizable. One of the important unsolved problems in mathematics is the description of the unitary dual, the effective classification of irreducible unitary representations of all real reductive Lie groups. All irreducible unitary representations are admissible, and the admissible representations are given by the Langlands classification, and it is easy to tell which of them have a non-trivial invariant sesquilinear form. The problem is that it is in general hard to tell when the quadratic form is positive definite. For many reductive Lie groups this has been solved; see representation theory of SL2 and representation theory of the Lorentz group for examples.
