In mathematics, the Hilbert projection theorem is a famous result of convex analysis that says that for every vector in a Hilbert space and every nonempty closed convex there exists a unique vector for which is minimized over the vectors ; that is, such that for every
Finite dimensional case
[edit]Some intuition for the theorem can be obtained by considering the first order condition of the optimization problem.
Consider a finite dimensional real Hilbert space with a subspace and a point If is a minimizer or minimum point of the function defined by (which is the same as the minimum point of ), then derivative must be zero at
In matrix derivative notation[1] Since is a vector in that represents an arbitrary tangent direction, it follows that must be orthogonal to every vector in
Statement
[edit]Hilbert projection theorem — For every vector in a Hilbert space and every nonempty closed convex there exists a unique vector for which is equal to
If the closed subset is also a vector subspace of then this minimizer is the unique element in such that is orthogonal to
Detailed elementary proof
[edit]Let be the distance between and a sequence in such that the distance squared between and is less than or equal to Let and be two integers, then the following equalities are true: and Therefore (This equation is the same as the formula for the length of a median in a triangle with sides of length and where specifically, the triangle's vertices are ).
By giving an upper bound to the first two terms of the equality and by noticing that the middle of and belong to and has therefore a distance greater than or equal to from it follows that:
The last inequality proves that is a Cauchy sequence. Since is complete, the sequence is therefore convergent to a point whose distance from is minimal.
Let and be two minimum points. Then:
Since belongs to we have and therefore
Hence which proves uniqueness.
Assume that is a closed vector subspace of It must be shown the minimizer is the unique element in such that for every
Proof that the condition is sufficient: Let be such that for all If then and so which implies that Because was arbitrary, this proves that and so is a minimum point.
Proof that the condition is necessary: Let be the minimum point. Let and Because the minimality of guarantees that Thus is always non-negative and must be a real number. If then the map has a minimum at and moreover, which is a contradiction. Thus
Proof by reduction to a special case
[edit]It suffices to prove the theorem in the case of because the general case follows from the statement below by replacing with
Hilbert projection theorem (case )[2] — For every nonempty closed convex subset of a Hilbert space there exists a unique vector such that
Furthermore, letting if is any sequence in such that in [note 1] then in
Let be as described in this theorem and let This theorem will follow from the following lemmas.
Lemma 1 — If is any sequence in such that in then there exists some such that in Furthermore,
Proof of Lemma 1
|
---|
![]() Because is convex, if then so that by definition of the infimum, which implies that By the parallelogram law, where now implies and so The assumption implies that the right hand side (RHS) of the above inequality can be made arbitrary close to by making and sufficiently large.[note 2] The same must consequently also be true of the inequality's left hand side and thus also of which proves that is a Cauchy sequence in Since is complete, there exists some such that in Because every belongs to which is a closed subset of their limit must also belongs to this closed subset, which proves that Since the norm is a continuous function, in implies that in But also holds (by assumption) so that (because limits in are unique). |
Lemma 2 — A sequence satisfying the hypotheses of Lemma 1 exists.
Proof of Lemma 2
|
---|
The existence of the sequence follows from the definition of the infimum, as is now shown. The set is a non-empty subset of non-negative real numbers and Let be an integer. Because there exists some such that Since holds (by definition of the infimum). Thus and now the squeeze theorem implies that in (This first part of the proof works for any non-empty subset of for which is finite). For every the fact that means that there exists some such that The convergence in thus becomes in |
Lemma 2 and Lemma 1 together prove that there exists some such that Lemma 1 can be used to prove uniqueness as follows. Suppose is such that and denote the sequence by so that the subsequence of even indices is the constant sequence while the subsequence of odd indices is the constant sequence Because for every in which shows that the sequence satisfies the hypotheses of Lemma 1. Lemma 1 guarantees the existence of some such that in Because converges to so do all of its subsequences. In particular, the subsequence converges to which implies that (because limits in are unique and this constant subsequence also converges to ). Similarly, because the subsequence converges to both and Thus which proves the theorem.
Consequences
[edit]Proposition — If is a closed vector subspace of a Hilbert space then[note 3]
Proof[3]
|
---|
Proof that : If then which implies Proof that is a closed vector subspace of : Let where is the underlying scalar field of and define which is continuous and linear because this is true of each of its coordinates The set is closed in because is closed in and is continuous. The kernel of any linear map is a vector subspace of its domain, which is why is a vector subspace of Proof that : Let The Hilbert projection theorem guarantees the existence of a unique such that (or equivalently, for all ). Let so that and it remains to show that The inequality above can be rewritten as: Because and is a vector space, and which implies that The previous inequality thus becomes or equivalently, But this last statement is true if and only if every Thus |
Properties
[edit]Expression as a global minimum
The statement and conclusion of the Hilbert projection theorem can be expressed in terms of global minimums of the followings functions. Their notation will also be used to simplify certain statements.
Given a non-empty subset and some define a function A global minimum point of if one exists, is any point in such that in which case is equal to the global minimum value of the function which is:
Effects of translations and scalings
When this global minimum point exists and is unique then denote it by explicitly, the defining properties of (if it exists) are: The Hilbert projection theorem guarantees that this unique minimum point exists whenever is a non-empty closed and convex subset of a Hilbert space. However, such a minimum point can also exist in non-convex or non-closed subsets as well; for instance, just as long is is non-empty, if then
If is a non-empty subset, is any scalar, and are any vectors then which implies:
Examples
The following counter-example demonstrates a continuous linear isomorphism for which Endow with the dot product, let and for every real let be the line of slope through the origin, where it is readily verified that Pick a real number and define by (so this map scales the coordinate by while leaving the coordinate unchanged). Then is an invertible continuous linear operator that satisfies and so that and Consequently, if with and if then
See also
[edit]- Orthogonal complement – Concept in linear algebra
- Orthogonal projection – Idempotent linear transformation from a vector space to itself
- Orthogonality principle – Condition for optimality of Bayesian estimator
- Riesz representation theorem – Theorem about the dual of a Hilbert space
Notes
[edit]- ^ Because the norm is continuous, if converges in then necessarily converges in But in general, the converse is not guaranteed. However, under this theorem's hypotheses, knowing that in is sufficient to conclude that converges in
- ^ Explicitly, this means that given any there exists some integer such that "the quantity" is whenever Here, "the quantity" refers to the inequality's right hand side and later in the proof, "the quantity" will also refer to and then By definition of "Cauchy sequence," is Cauchy in if and only if "the quantity" satisfies this aforementioned condition.
- ^ Technically, means that the addition map defined by is a surjective linear isomorphism and homeomorphism. See the article on complemented subspaces for more details.
References
[edit]- ^ Petersen, Kaare. "The Matrix Cookbook" (PDF). Retrieved 9 January 2021.
- ^ Rudin 1991, pp. 306–309.
- ^ Rudin 1991, pp. 307−309.
Bibliography
[edit]- Rudin, Walter (1987). Real and Complex Analysis (Third ed.).
- Rudin, Walter (1991). Functional Analysis. International Series in Pure and Applied Mathematics. Vol. 8 (Second ed.). New York, NY: McGraw-Hill Science/Engineering/Math. ISBN 978-0-07-054236-5. OCLC 21163277.