Derive MLE for the Mean of a Univariate Normal Distribution

Question

Derive Maximum Likelihood Estimate (MLE) for the mean $(\mu)$ of a univariate normal distribution. Assume N samples $x_1, x_2, ...., x_n$ independently drawn from a normal distribution with known variance $(\sigma)$ and unknown mean $(\mu)$ . Show all intermediate steps and assumptions.

Solution

==================================================================================================

Since this is a normal distribution, we know that,

(1) $\begin{equation*} \left P(x; \mu, \sigma) = \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x-\mu)^2}{2 \sigma^2}] \end{equation*}$

(1) is the probability density function (PDF) of a Gaussian distribution.

We have data points $x_1, x_2, ...., x_n$ . The joint probability density of observing n data points:

(2) $\begin{equation*} \left P(x_1, x_2, ...., x_n; \mu, \sigma) = \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x_1-\mu)^2}{2 \sigma^2}] * \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x_2-\mu)^2}{2 \sigma^2}] * ...... * \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x_n-\mu)^2}{2 \sigma^2}] \end{equation*}$

Taking logs of (2),

(3) $\begin{equation*} \left ln(P(x_1, x_2, ...., x_n; \mu, \sigma)) = ln(\frac{1}{\sigma \sqrt{2\pi}}) \ - \ \frac{(x_1-\mu)^2}{2 \sigma^2} + ln(\frac{1}{\sigma \sqrt{2\pi}}) \ - \ \frac{(x_2-\mu)^2}{2 \sigma^2} + ...... + ln(\frac{1}{\sigma \sqrt{2\pi}}) \ - \ \frac{(x_n-\mu)^2}{2 \sigma^2} \end{equation*}$

Simplifying (3),

(4) $\begin{equation*} \left ln(P(x; \mu, \sigma)) = -n*ln(\sigma) \ - \ \frac{n}{2} * ln(2\pi) \ - \ \frac{1}{2 \sigma^2} \ * \ [(x_1-\mu)^2 + (x_2-\mu)^2 + ..... + (x_n-\mu)^2] \end{equation*}$

To find MLE of a certain parameter, we have to differentiate w.r.t the desired parameter. Here, the parameter is $\mu$ .

$\begin{align*} \left \frac{\partial (4)}{\partial \mu} = \frac{\partial P(x; \mu, \sigma)}{\partial \mu} \end{align*}$

$\begin{align*} \left = 0 + 0 + \frac{1}{\sigma^2} \ [x_1 + x_2 + .... + x_n - n\mu] \end{align*}$

(5) $\begin{equation*} \left = \frac{1}{\sigma^2} \ [x_1 + x_2 + .... + x_n - n\mu] \end{equation*}$

Thus, (5) is the desired MLE.

Question

Solution

Subscribe to Ehan Ghalib!

Leave a Reply Cancel Reply