---
title: "2.3 Deriving the Least Squares Estimators"
output:
  html:
    self-contained: TRUE
---

## Estimators of parameters of a linear model

Suppose we hypothesize that a random variable $Y$ depends linearly on another random variable $X$. This relationship can be expressed as: $y_i = \beta_0 + \beta_1 x_i + u_i$.

Here, $\beta_0$ (the intercept) and $\beta_1$ (the slope) are parameters that describe the linear relationship between $X$ and $Y$, and $u_i$ represents the unobservable disturbance term. 

Given a sample of $X$ and $Y$, we aim to derive unbiased estimators for $\beta_0$ and $\beta_1$. These estimators allow us to combine observations of $X$ and $Y$ to estimate the underlying relationship between the two variables.

<iframe width="560" height="315" src="https://www.youtube.com/embed/2wfqex1j6CY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 1**: Find the equation of the line (in slope-intercept form) through the points (x, y) = (0, 4) and (2, 6).
</div>

<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

<iframe width="560" height="315" src="https://www.youtube.com/embed/mv_dSvpjAx8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

The method of least squares, published by Adrien Marie Legendre in 1805 (though Carl Friedrich Gauss may have discovered it earlier), revolutionized statistics by providing a way to combine multiple observations to find underlying relationships. It solved two major scientific challenges of the early 1800s: helping geodesists calculate Earth's circumference by combining varied city distance and star angle measurements, and enabling astronomers to determine the orbit of the newly discovered Ceres from limited observations. This mathematical technique remains the foundation of modern statistical analysis.

### Summation Rules

<iframe width="560" height="315" src="https://www.youtube.com/embed/1YH-PIMbOTk" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 2**: If $x = \{ 3, 7, 2, 5\}$, calculate $\sum_i x_i$.
</div>
<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 3**: True or false: $\sum_i 3 x_i = 3 \sum_i x_i$.
</div>
<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 4**: True or false: $\sum_i 3 x_i y_i = 3 \sum_i x_i \sum_i y_i$.
</div>
<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

### Notation: Linear Models

Here's a preview of the notation I'll use in the next part of this chapter. Basically it's just saying some true linear model exists between $X$ and $Y$, which is $y_i = \beta_0 + \beta_1 x_i + u_i$. Then when I'm referring to our estimates of the intercept and slope, I'll put hats over the $\beta$s and turn the disturbance $u_i$ into residuals $e_i$ because they turn out to be pretty different conceptually: $y_i = \hat{\beta_0} + \hat{\beta_1}x_i + e_i$.

| Symbol | Meaning | Example |
| ---: | ---------- | ----- |
| $\beta_0$ | Intercept parameter in a linear model | $y_i = \beta_0 + \beta_1 x_i + u_i$ |
| $\beta_1$ | Slope parameter in a linear model | see above |
| $y_i$ | dependent variable or outcome variable | see above |
| $x_i$ | explanatory variable | see above |
| $u_i$ | unobservable term, disturbance, shock | see above |
| $\hat{\beta}_0$ | Estimate of the intercept | $y_i = \hat{\beta}_0 + \hat{\beta}_1 x_i + e_i$ |
| $\hat{\beta}_1$ | Estimate of the slope | see above |
| $e_i$ | residuals | see above |

<iframe width="560" height="315" src="https://www.youtube.com/embed/oeKYjWWlb4o" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 5**: Fill in the blanks by selecting one: OLS residuals are the (vertical/horizontal) distances between the observation and the (true/estimated) linear model.
</div>
<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

## Least Squares: Combining Observations

One reason the method of least squares is so popular is that it's so simple and mathematically tractable: the entire procedure can be summed up in one statement: **the method of least squares fits a linear model that minimizes the sum of the squared residuals.**

<iframe width="560" height="315" src="https://www.youtube.com/embed/_J70ywx3MDQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

In the next few videos, we'll see that for a simple regression, we can take that statement of the method of least squares and derive:

$$\hat{\beta_0} = \bar{y} - \beta_1 \bar{x}$$

$$\hat{\beta_1} = \frac{\sum_i x_i y_i - \bar{x}\bar{y}n}{\sum_i x_i^2 - \bar{x}^2 n}$$

## Deriving OLS Estimators $\hat{\beta_0}$ and $\hat{\beta_1}$

<iframe width="560" height="315" src="https://www.youtube.com/embed/4bKkI1X2aUE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

<iframe width="560" height="315" src="https://www.youtube.com/embed/F7JMDAf4x7U" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

<iframe width="560" height="315" src="https://www.youtube.com/embed/t8MDIt_-va4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
Suppose we sampled random variables X and Y to get this data:

  | X | Y |
  |---+---|
  | 1 | 4 |
  | 2 | 2 |
  | 3 | 1 |
  
**Exercise 6:** Draw a plot of $x_i$ and $y_i$ and eyeball a line of best fit. What is the slope and intercept of that line (approximately)?
</div>
<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">

**Exercise 7:** Use the formulas we derived in this assignment to find the slope and intercept of the OLS line of best fit: $y_i = \hat{\beta}_0 + \hat{\beta}_1 x_i + u_i$. Here are the formulas you can use: $\hat{\beta_0} =
\bar{y} - \hat{\beta_1}\bar{x}$ and $\hat{\beta_1} = \frac{\sum_{i=1}^n (x_i y_i) -
\bar{x}\bar{y}n}{\sum_{i = 1}^n (x_i^2) - \bar{x}^2n}$. Verify that your estimates for $\beta_0$ and $\beta_1$ are on track by comparing these answers to your answers to the previous question.
</div>

<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 8:** Fill out the table below to calculate the OLS fitted values ($\hat{y_i} = \hat{\beta_0} +
\hat{\beta_1} x_i$) and residuals ($e_i = y_i - \hat{y_i}$). Verify that the fitted values you
calculate are correct by making sure that ($x_i$, $\hat{y_i}$) lays on the line of best fit you
drew in question 1.

| x | y | $\hat{y_i}$       | $e_i$              |
|---+---+----------------------------+-----------------|
| 1 | 4 | $\underline{\hspace{1cm}}$ | $\underline{\hspace{1cm}}$ |
| 2 | 2 | $\underline{\hspace{1cm}}$ | $\underline{\hspace{1cm}}$ |
| 3 | 1 | $\underline{\hspace{1cm}}$ | $\underline{\hspace{1cm}}$ |
</div>

<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>

<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 9:** Suppose that the true relationship between $X$ and $Y$ is actually this: $y_i = 5 - x_i + u_i$. So the true values for $\beta_0 = 5$ and $\beta_1 = -1$. Calculate the true disturbance $u_i$. Is it required that $\sum_{i = 1}^n u_i = 0$ or $\sum_{i = 1}^n x_i u_i = 0$ like it is with $e_i$?

| x | y | $u_i$                      |
|---+---+----------------------------|
| 1 | 4 | $\underline{\hspace{1cm}}$ |
| 2 | 2 | $\underline{\hspace{1cm}}$ |
| 3 | 1 | $\underline{\hspace{1cm}}$ |
</div>
     
<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>
     
<div style="background-color:#d2ff96; border-width:3px; border-style:solid; border-color:#d2ff96; padding: 1em;">
**Exercise 10:** consider the linear model without an intercept $y = \beta_0 x + u$. Recall that OLS minimizes the sum of squared residuals. Write down what that means in this context mathematically, and then take first order conditions. Show that the OLS estimator is $\hat{\beta_0} = \frac{\sum_i x_i y_i}{\sum_i x_i^2}.$
</div>

<div style="background-color:white; border-width:3px; border-style:solid; border-color:black; padding:1em;">
*Answer:* 

</div>