Maximum Likelihood Estimator
What is the maximum likelihood estimator (MLE) and how does it work?
Answer
Suppose we have $x_1, \dots, x_n$ i.i.d. observations from a distribution with density (or pmf) $f(x;\theta)$. The maximum likelihood estimator is the value of $\hat{\theta}$ for which the probability of observing these exact data points again is maximized. This is the same as finding the $\theta$ that maximizes the likelihood function. Formally, the MLE is defined as
\[
\hat{\theta}_{\text{MLE}} = \arg\max_{\theta} L(\theta).
\]
In practice, we usually maximize the log-likelihood
\[
\ell(\theta) = \log L(\theta) = \sum_{i=1}^n \log f(x_i;\theta),
\]
since the log turns products into sums (making derivatives easier) and is monotone (preserving the order/maximizer).