# Ordinary least square (OLS) estimators The ordinary least square estimators are estimators for the coefficients in [[Simple Linear Regression|linear regression]]. The intuition behind these estimators is that the estimated errors should be *as small as possible*. Small errors should indicate a better "fit" between the observed outcome and the model fit with the covariate: To summarize this intuition over all the observations, we use the **residual sum of squares (RSS)**: $ \text{RSS}(\beta_0, \beta_1) = \sum^n_{i=1} \left[ y_i - (\beta_0 + \beta_1 x_i) \right]^2 $ This equation can be written more compactly in vector and matrix notation. $ \begin{align} \text{RSS}(\mathbf{\beta} ) &= \mathbf{\varepsilon}^T\mathbf{\varepsilon} \\ &= (\mathbf{Y} - \mathbf{X}^T\mathbf{\beta})^T(\mathbf{Y} - \mathbf{X}^T\mathbf{\beta}) \end{align} $ - $\mathbf{Y}$ is an $n\times 1$ column vector - $\mathbf{X}$ is a $n \times p$ design matrix, where $p$ represents the number of coefficients in the regression. - $\mathbf{\varepsilon}$ is an $n\times1$ column vector - $\mathbf{\beta}$ is a $p\times1$ column vector After differentiating with respect for $\beta$ and solving for zero, the OLS estimators are given by: $ \hat{\mathbf{\beta}} = (X'X)^{-1}X'Y $ The minimization of the RSS is what gives the OLS estimators their name. Once the regression coefficients have been estimated from the data, the errors can be estimated. These are called **residuals**. $ \hat{\varepsilon}_i = y_i - \hat{y}_i = y_i - (\hat{\beta}_0 + \hat{\beta}_1 x_i) $ From these residuals, we can estimate the variance of the errors $\sigma^2$ via the following calculation (the mean sum of squared, MSE) $ \hat{\sigma}^2 = \text{MSE} = \frac{\text{RSS}(\hat{\beta}_0, \hat{\beta}_1)}{n - 2} $ - If we are dealing with [[Multiple linear regression|multiple linear regression]], then $n-2$ will be replaced by $n - p$ in the denominator. The [[Statistical properties of the OLS estimators|statistical properties of the OLS estimators]] depend on the assumptions that you can make about the errors. Alternatively, one can rely on [[Bootstrapping|bootstrapping]] as a technique for estimating the variance/standard error of an estimator. --- # References - [[Applied Linear Regression#2. Simple Linear Regression]] - [[Applied Linear Regression#3. Multiple Linear Regression]] - [[Applied Linear Regression#7. Variances]]