# Generalized least squares (GLS) estimators An extension of [[Multiple linear regression|multiple linear regression]] that incorporates models for correlated and heteroskedastic errors: $ \begin{align} &E(Y\mid X_1, ..., X_p) = \beta_0 + \beta_1 + ... + \beta_p X_p \\ &\text{Var}(Y \mid X_1, ..., X_p) = \Sigma \end{align} $ where $\Sigma$ is a positive definite matrix (covariance matrix). There are $n$ variance terms and $n(n-1)/2$ pairwise correlations. ## Special cases - If $\Sigma = \sigma^2 I$, then we have the [[Ordinary least square (OLS) estimators|OLS estimators]] - If $\Sigma = \sigma^2 W^{-1}$, then we have the [[Dealing with heteroskedastic data (weighted least squares)|WLS estimators]] ## The GLS estimator Solving for this new model specification yields the GLS estimators: $ \hat{\beta}_{\text{GLS}} = (X'\Sigma X)^{-1}X' \Sigma Y $ and for the covariance matrix: $ \text{Cov}(\hat{\beta}_{\text{GLS}}\mid X) = (X'\Sigma X)^{-1} $ ## Correlation structures Choosing to estimate *all* the variance and covariance parameters is costly and will harm estimates in small to moderate sample sizes. Instead, one can choose to specify a specific *correlation structure* to reduce the number of parameters that need to be estimated. For example, an autoregressive structure only has 2 parameters to estimate: $ \Sigma_{AR} = \sigma^2 \begin{bmatrix} 1 & \rho & ... & \rho^{n-1} \\ \rho & 1 & ... & \rho^{n-2} \\ \vdots & \vdots & \ddots & \vdots \\ \rho^{n-1} & \rho^{n-2} & ... & 1 \end{bmatrix} $ - Correlation between different observations is given as a power of $\rho$ Other correlation structures include compound symmetry, block diagonal, etc. ## Uses GLS is commonly used with longitudinal data where observations are correlated due to them coming from the same person and being adjacent in time. --- # References [[Applied Linear Regression#7. Variances]]