Multiple Regression

More than one independent variables. Multicollinearity problem may exist if independent variables are correlated with each other.

3.1                Multivariate Theorem-

Suppose U ∼ N (µ, Σ), a multivariate normal vector,                           (1)

V = c + DU, a linear transformation of U                                               (2)

Where, c is a vector and D is a matrix.

Then,

            V ∼ N (c + Dµ, D * ΣD’)                                                             (3)

                  b = (X’ X) ⁻¹ X’ Y = [(X’ X) ⁻¹ X’] Y and                                  (4)

                                  Y ∼ N (Xβ, σ² I)                                                        (5)

Comparing (1) with (5)-

                       U = Y,                 µ = Xβ,                   Σ = σ² I,

Comparing (2) with (4)-

                        V = b,                    c = 0,    and         D = (X’ X) ⁻¹ X’

This tells us the vector b is normally distributed with mean-

(X’ X) ⁻¹ (X’ X) β = β

And covariance matrix-

So, (X’ X) and its inverse are symmetric,

((X’ X) ⁻¹)’= (X’ X) ⁻¹

3.2            Parameters and Residuals-

Matrices H and (I − H) properties:

       i.            Symmetric: H = H’ and (I − H)’ = (I − H).

    ii.            Idempotent: H² = H and (I − H)(I − H)=(I − H)

3.3      Covariance Matrix of Residuals-

Cov(e) = σ² (I − H)(I − H)’ = σ² (I − H)

Var(e_i) = σ² (1 − h_i,i)

Where, h_i,i is the i^th diagonal element of H

3.4      Standard Error of Regression in Matrix form-

SSE, MSE and RMSE-

Sum Square Error = e’ e has df_E = n – p,

Where, p = number of independent variables, n = # of observations

Mean Square Error = SSE / df_E

Distribution of b: b = (X’ X) ⁻¹X’ Y.

The only RV involved is Y, so the distribution of b is based on the distribution of Y.

3.5      Anova Parameters-

 

 

https://drive.google.com/file/d/0Bx3mfFH5R-y3VFhNbmZBMXVmYzA/view?usp=sharing