Projection matrix

In statistics, the projection matrix $(\mathbf {P} )$ ,^[1] sometimes also called the influence matrix^[2] or hat matrix $(\mathbf {H} )$ , maps the vector of response values (dependent variable values) to the vector of fitted values (or predicted values). It describes the influence each response value has on each fitted value.^[3]^[4] The diagonal elements of the projection matrix are the leverages, which describe the influence each response value has on the fitted value for that same observation.

Definition

If the vector of response values is denoted by $\mathbf {y}$ and the vector of fitted values by $\mathbf {\hat {y}}$ ,

\mathbf {\hat {y}} =\mathbf {P} \mathbf {y} .

As $\mathbf {\hat {y}}$ is usually pronounced "y-hat", the projection matrix $\mathbf {P}$ is also named hat matrix as it "puts a hat on $\mathbf {y}$ ".

Application for residuals

The formula for the vector of residuals $\mathbf {r}$ can also be expressed compactly using the projection matrix:

\mathbf {r} =\mathbf {y} -\mathbf {\hat {y}} =\mathbf {y} -\mathbf {P} \mathbf {y} =\left(\mathbf {I} -\mathbf {P} \right)\mathbf {y} .

where $\mathbf {I}$ is the identity matrix. The matrix $\mathbf {M} :=\mathbf {I} -\mathbf {P}$ is sometimes referred to as the residual maker matrix or the annihilator matrix.

The covariance matrix of the residuals $\mathbf {r}$ , by error propagation, equals

\mathbf {\Sigma } _{\mathbf {r} }=\left(\mathbf {I} -\mathbf {P} \right)^{\textsf {T}}\mathbf {\Sigma } \left(\mathbf {I} -\mathbf {P} \right)

,

where $\mathbf {\Sigma }$ is the covariance matrix of the error vector (and by extension, the response vector as well). For the case of linear models with independent and identically distributed errors in which $\mathbf {\Sigma } =\sigma ^{2}\mathbf {I}$ , this reduces to:^[3]

\mathbf {\Sigma } _{\mathbf {r} }=\left(\mathbf {I} -\mathbf {P} \right)\sigma ^{2}

.

Intuition

From the figure, it is clear that the closest point from the vector $\mathbf {b}$ onto the column space of $\mathbf {A}$ , is $\mathbf {Ax}$ , and is one where we can draw a line orthogonal to the column space of $\mathbf {A}$ . A vector that is orthogonal to the column space of a matrix is in the nullspace of the matrix transpose, so

\mathbf {A} ^{\textsf {T}}(\mathbf {b} -\mathbf {Ax} )=0

.

From there, one rearranges, so

{\begin{aligned}&&\mathbf {A} ^{\textsf {T}}\mathbf {b} &-\mathbf {A} ^{\textsf {T}}\mathbf {Ax} =0\\\Rightarrow &&\mathbf {A} ^{\textsf {T}}\mathbf {b} &=\mathbf {A} ^{\textsf {T}}\mathbf {Ax} \\\Rightarrow &&\mathbf {x} &=\left(\mathbf {A} ^{\textsf {T}}\mathbf {A} \right)^{-1}\mathbf {A} ^{\textsf {T}}\mathbf {b} \end{aligned}}

.

Therefore, since $\mathbf {Ax}$ is on the column space of $\mathbf {A}$ , the projection matrix, which maps $\mathbf {b}$ onto $\mathbf {x}$ , is $\mathbf {A} \left(\mathbf {A} ^{\textsf {T}}\mathbf {A} \right)^{-1}\mathbf {A} ^{\textsf {T}}$ .

Linear model

Suppose that we wish to estimate a linear model using linear least squares. The model can be written as

\mathbf {y} =\mathbf {X} {\boldsymbol {\beta }}+{\boldsymbol {\varepsilon }},

where $\mathbf {X}$ is a matrix of explanatory variables (the design matrix), β is a vector of unknown parameters to be estimated, and ε is the error vector.

Many types of models and techniques are subject to this formulation. A few examples are linear least squares, smoothing splines, regression splines, local regression, kernel regression, and linear filtering.

Ordinary least squares

When the weights for each observation are identical and the errors are uncorrelated, the estimated parameters are

{\hat {\boldsymbol {\beta }}}=\left(\mathbf {X} ^{\textsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\textsf {T}}\mathbf {y} ,

so the fitted values are

{\hat {\mathbf {y} }}=\mathbf {X} {\hat {\boldsymbol {\beta }}}=\mathbf {X} \left(\mathbf {X} ^{\textsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\textsf {T}}\mathbf {y} .

Therefore, the projection matrix (and hat matrix) is given by

\mathbf {P} :=\mathbf {X} \left(\mathbf {X} ^{\textsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\textsf {T}}.

Weighted and generalized least squares

The above may be generalized to the cases where the weights are not identical and/or the errors are correlated. Suppose that the covariance matrix of the errors is Σ. Then since

{\hat {\mathbf {\beta } }}_{\text{GLS}}=\left(\mathbf {X} ^{\textsf {T}}\mathbf {\Sigma } ^{-1}\mathbf {X} \right)^{-1}\mathbf {X} ^{\textsf {T}}\mathbf {\Sigma } ^{-1}\mathbf {y}

.

the hat matrix is thus

\mathbf {H} =\mathbf {X} \left(\mathbf {X} ^{\textsf {T}}\mathbf {\Sigma } ^{-1}\mathbf {X} \right)^{-1}\mathbf {X} ^{\textsf {T}}\mathbf {\Sigma } ^{-1}

and again it may be seen that $H^{2}=H\cdot H=H$ , though now it is no longer symmetric.

Properties

The projection matrix has a number of useful algebraic properties.^[5]^[6] In the language of linear algebra, the projection matrix is the orthogonal projection onto the column space of the design matrix $\mathbf {X}$ .^[4] (Note that $\left(\mathbf {X} ^{\textsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\textsf {T}}$ is the pseudoinverse of X.) Some facts of the projection matrix in this setting are summarized as follows:^[4]

$\mathbf {u} =(\mathbf {I} -\mathbf {P} )\mathbf {y} ,$ and $\mathbf {u} =\mathbf {y} -\mathbf {P} \mathbf {y} \perp \mathbf {X} .$
$\mathbf {P}$ is symmetric, and so is $\mathbf {M} :=\mathbf {I} -\mathbf {P}$ .
$\mathbf {P}$ is idempotent: $\mathbf {P} ^{2}=\mathbf {P}$ , and so is $\mathbf {M}$ .
If $\mathbf {X}$ is an n × r matrix with $\operatorname {rank} (\mathbf {X} )=r$ , then $\operatorname {rank} (\mathbf {P} )=r$
The eigenvalues of $\mathbf {P}$ consist of r ones and n − r zeros, while the eigenvalues of $\mathbf {M}$ consist of n − r ones and r zeros.^[7]
$\mathbf {X}$ is invariant under $\mathbf {P}$ : $\mathbf {PX} =\mathbf {X} ,$ hence $\left(\mathbf {I} -\mathbf {P} \right)\mathbf {X} =\mathbf {0}$ .
$\left(\mathbf {I} -\mathbf {P} \right)\mathbf {P} =\mathbf {P} \left(\mathbf {I} -\mathbf {P} \right)=\mathbf {0} .$
$\mathbf {P}$ is unique for certain subspaces.

The projection matrix corresponding to a linear model is symmetric and idempotent, that is, $\mathbf {P} ^{2}=\mathbf {P}$ . However, this is not always the case; in locally weighted scatterplot smoothing (LOESS), for example, the hat matrix is in general neither symmetric nor idempotent.

For linear models, the trace of the projection matrix is equal to the rank of $\mathbf {X}$ , which is the number of independent parameters of the linear model.^[8] For other models such as LOESS that are still linear in the observations $\mathbf {y}$ , the projection matrix can be used to define the effective degrees of freedom of the model.

Practical applications of the projection matrix in regression analysis include leverage and Cook's distance, which are concerned with identifying influential observations, i.e. observations which have a large effect on the results of a regression.

Blockwise formula

Suppose the design matrix $\mathbf {X}$ can be decomposed by columns as $\mathbf {X} ={\begin{bmatrix}\mathbf {A} &\mathbf {B} \end{bmatrix}}$ . Define the hat or projection operator as $\mathbf {P} [\mathbf {X} ]:=\mathbf {X} \left(\mathbf {X} ^{\textsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\textsf {T}}$ . Similarly, define the residual operator as $\mathbf {M} [\mathbf {X} ]:=\mathbf {I} -\mathbf {P} [\mathbf {X} ]$ . Then the projection matrix can be decomposed as follows:^[9]

\mathbf {P} [\mathbf {X} ]=\mathbf {P} [\mathbf {A} ]+\mathbf {P} {\big [}\mathbf {M} [\mathbf {A} ]\mathbf {B} {\big ]},

where, e.g., $\mathbf {P} [\mathbf {A} ]=\mathbf {A} \left(\mathbf {A} ^{\textsf {T}}\mathbf {A} \right)^{-1}\mathbf {A} ^{\textsf {T}}$ and $\mathbf {M} [\mathbf {A} ]=\mathbf {I} -\mathbf {P} [\mathbf {A} ]$ . There are a number of applications of such a decomposition. In the classical application $\mathbf {A}$ is a column of all ones, which allows one to analyze the effects of adding an intercept term to a regression. Another use is in the fixed effects model, where $\mathbf {A}$ is a large sparse matrix of the dummy variables for the fixed effect terms. One can use this partition to compute the hat matrix of $\mathbf {X}$ without explicitly forming the matrix $\mathbf {X}$ , which might be too large to fit into computer memory.

History

The hat matrix was introduced by John Wilder in 1972. An article by Hoaglin, D.C. and Welsch, R.E. (1978) gives the properties of the matrix and also many examples of its application.

References

^ Basilevsky, Alexander (2005). Applied Matrix Algebra in the Statistical Sciences. Dover. pp. 160–176. ISBN 0-486-44538-0.
^ "Data Assimilation: Observation influence diagnostic of a data assimilation system" (PDF). Archived from the original (PDF) on 2014-09-03.
^ ^a ^b Hoaglin, David C.; Welsch, Roy E. (February 1978). "The Hat Matrix in Regression and ANOVA" (PDF). The American Statistician. 32 (1): 17–22. doi:10.2307/2683469. hdl:1721.1/1920. JSTOR 2683469.
^ ^a ^b ^c David A. Freedman (2009). Statistical Models: Theory and Practice. Cambridge University Press.
^ Gans, P. (1992). Data Fitting in the Chemical Sciences. Wiley. ISBN 0-471-93412-7.
^ Draper, N. R.; Smith, H. (1998). Applied Regression Analysis. Wiley. ISBN 0-471-17082-8.
^ Amemiya, Takeshi (1985). Advanced Econometrics. Cambridge: Harvard University Press. pp. 460–461. ISBN 0-674-00560-0.
^ "Proof that trace of 'hat' matrix in linear regression is rank of X". Stack Exchange. April 13, 2017.
^ Rao, C. Radhakrishna; Toutenburg, Helge; Shalabh; Heumann, Christian (2008). Linear Models and Generalizations (3rd ed.). Berlin: Springer. p. 323. ISBN 978-3-540-74226-5.

[1] Basilevsky, Alexander (2005). Applied Matrix Algebra in the Statistical Sciences. Dover. pp. 160–176. ISBN 0-486-44538-0.

[2] "Data Assimilation: Observation influence diagnostic of a data assimilation system" (PDF). Archived from the original (PDF) on 2014-09-03.

[Hoaglin1977-3] Hoaglin, David C.; Welsch, Roy E. (February 1978). "The Hat Matrix in Regression and ANOVA" (PDF). The American Statistician. 32 (1): 17–22. doi:10.2307/2683469. hdl:1721.1/1920. JSTOR 2683469.

[Freedman09-4] David A. Freedman (2009). Statistical Models: Theory and Practice. Cambridge University Press.

[5] Gans, P. (1992). Data Fitting in the Chemical Sciences. Wiley. ISBN 0-471-93412-7.

[6] Draper, N. R.; Smith, H. (1998). Applied Regression Analysis. Wiley. ISBN 0-471-17082-8.

[7] Amemiya, Takeshi (1985). Advanced Econometrics. Cambridge: Harvard University Press. pp. 460–461. ISBN 0-674-00560-0.

[8] "Proof that trace of 'hat' matrix in linear regression is rank of X". Stack Exchange. April 13, 2017.

[9] Rao, C. Radhakrishna; Toutenburg, Helge; Shalabh; Heumann, Christian (2008). Linear Models and Generalizations (3rd ed.). Berlin: Springer. p. 323. ISBN 978-3-540-74226-5.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

Matrix classes
Explicitly constrained entries	Alternant Anti-diagonal Anti-Hermitian Anti-symmetric Arrowhead Band Bidiagonal Bisymmetric Block-diagonal Block Block tridiagonal Boolean Cauchy Centrosymmetric Conference Complex Hadamard Copositive Diagonally dominant Diagonal Discrete Fourier Transform Elementary Equivalent Frobenius Generalized permutation Hadamard Hankel Hermitian Hessenberg Hollow Integer Logical Matrix unit Metzler Moore Nonnegative Pentadiagonal Permutation Persymmetric Polynomial Quaternionic Signature Skew-Hermitian Skew-symmetric Skyline Sparse Sylvester Symmetric Toeplitz Triangular Tridiagonal Vandermonde Walsh Z
Constant	Exchange Hilbert Identity Lehmer Of ones Pascal Pauli Redheffer Shift Zero
Conditions on eigenvalues or eigenvectors	Companion Convergent Defective Definite Diagonalizable Hurwitz-stable Positive-definite Stieltjes
Satisfying conditions on products or inverses	Congruent Idempotent or Projection Invertible Involutory Nilpotent Normal Orthogonal Unimodular Unipotent Unitary Totally unimodular Weighing
With specific applications	Adjugate Alternating sign Augmented Bézout Carleman Cartan Circulant Cofactor Commutation Confusion Coxeter Distance Duplication and elimination Euclidean distance Fundamental (linear differential equation) Generator Gram Hessian Householder Jacobian Moment Payoff Pick Random Rotation Routh-Hurwitz Seifert Shear Similarity Symplectic Totally positive Transformation
Used in statistics	Centering Correlation Covariance Design Doubly stochastic Fisher information Hat Precision Stochastic Transition
Used in graph theory	Adjacency Biadjacency Degree Edmonds Incidence Laplacian Seidel adjacency Tutte
Used in science and engineering	Cabibbo–Kobayashi–Maskawa Density Fundamental (computer vision) Fuzzy associative Gamma Gell-Mann Hamiltonian Irregular Overlap S State transition Substitution Z (chemistry)
Related terms	Jordan normal form Linear independence Matrix exponential Matrix representation of conic sections Perfect matrix Pseudoinverse Row echelon form Wronskian
Mathematics portal List of matrices Category:Matrices

Fahrer / Team für die Suche eingeben: