
# Maximum Likelihood Estimation (MLE)

## Steps of MLE
1. Find the log-likelihood function, $\log L(\theta; x_1,...,x_n)$.
2. Take the derivative of the log-likelihood with respect to the parameter we are estimating, $\frac{d}{d\theta}\log L(\theta;x_1,...,x_n)$.
3. Set derivative to zero and solve for parameter we are interested in estimating, $\frac{d}{d\theta}\log L(\theta;x_1,...,x_n) = 0$.
4. Check the second derivative for concavity, we want a maximum, $\frac{d^2}{d\theta^2}\log L(\theta;x_1,...,x_n)$.

## You Try!
Suppose we have independent identically distributed random variables, $Y_i$, where $Y_i \sim \text{Normal}(x_i\beta, 1)$, where $x_i$'s are given for $i = 1,...,n$.  Show that the MLE of $\beta$ is 
$\frac{\sum_{i=1}^{n}x_iY_i}{\sum_{i=1}^{n}x_i^2}$.  Recall that the pdf of this random variable is
$$
f(Y_i|x_i,\beta) = \frac{1}{\sqrt{2\pi}}\exp\left[-\frac{(Y_i - x_i\beta)^2}{2}\right].
$$


## Solution

$logL(\beta; Y_1,...Y_n) = log(\prod_{i=1}^n (\frac{1}{\sqrt{2\pi}} exp[-\frac{(Y_i - x_i\beta)^2}{2}]))$

$logL(\beta; Y_1,...Y_n) = n*log(\frac{1}{\sqrt{2\pi}}) - \frac{1}{2}\sum_{i=1}^n (Y_i - x_i\beta)^2$

$logL(\beta; Y_1,...Y_n) = n*log(\frac{1}{\sqrt{2\pi}}) - \frac{1}{2}[\sum_{i=1}^n (Y_i) - 2\beta *\sum_{i=1}^n (Y_i * x_i) + \beta^2 * \sum_{i=1}^n (x_i^2)]$

$\frac{d}{d\beta}logL(\beta; Y_1,...Y_n) = 2\sum_{i=1}^n (Y_i * x_i) - 2\beta * \sum_{i=1}^n (x_i^2)$

Set this equal to $0$ and solve for $\beta$:

$0 = 2\sum_{i=1}^n (Y_i * x_i) - 2\beta * \sum_{i=1}^n (x_i^2)$

$2\sum_{i=1}^n (Y_i * x_i) = 2\beta * \sum_{i=1}^n (x_i^2)$

$\beta = \frac{\sum_{i=1}^n (Y_i * x_i)}{\sum_{i=1}^n (x_i^2)}$

We also check that this is a maximum:

$\frac{d^2}{d\beta^2}\log L(\beta; Y_1,...Y_n) = -2 \sum_{i=1}^n (x_i^2)$

This is always negative so we have obtained a maximum.
