Question

Derive Maximum Likelihood Estimate (MLE) for the mean (\mu) of a univariate normal distribution. Assume N samples x_1, x_2, ...., x_n independently drawn from a normal distribution with known variance (\sigma) and unknown mean (\mu). Show all intermediate steps and assumptions.

Solution

==================================================================================================

Since this is a normal distribution, we know that,

(1)   \begin{equation*}     \left P(x; \mu, \sigma) = \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x-\mu)^2}{2 \sigma^2}] \end{equation*}

(1) is the probability density function (PDF) of a Gaussian distribution.

We have data points x_1, x_2, ...., x_n. The joint probability density of observing n data points:

(2)   \begin{equation*}     \left P(x_1, x_2, ...., x_n; \mu, \sigma) = \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x_1-\mu)^2}{2 \sigma^2}] * \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x_2-\mu)^2}{2 \sigma^2}] * ...... * \frac{1}{\sigma \sqrt{2\pi}} \ exp[- \ \frac{(x_n-\mu)^2}{2 \sigma^2}] \end{equation*}

Taking logs of (2),

(3)   \begin{equation*}     \left ln(P(x_1, x_2, ...., x_n; \mu, \sigma)) = ln(\frac{1}{\sigma \sqrt{2\pi}}) \ - \ \frac{(x_1-\mu)^2}{2 \sigma^2} + ln(\frac{1}{\sigma \sqrt{2\pi}}) \ - \ \frac{(x_2-\mu)^2}{2 \sigma^2} + ...... + ln(\frac{1}{\sigma \sqrt{2\pi}}) \ - \ \frac{(x_n-\mu)^2}{2 \sigma^2} \end{equation*}

Simplifying (3),

(4)   \begin{equation*}     \left ln(P(x; \mu, \sigma)) = -n*ln(\sigma) \ - \ \frac{n}{2} * ln(2\pi) \ - \ \frac{1}{2 \sigma^2} \ * \ [(x_1-\mu)^2 + (x_2-\mu)^2 + ..... + (x_n-\mu)^2] \end{equation*}

To find MLE of a certain parameter, we have to differentiate w.r.t the desired parameter. Here, the parameter is \mu.

    \begin{align*}     \left \frac{\partial (4)}{\partial \mu} = \frac{\partial P(x; \mu, \sigma)}{\partial \mu} \end{align*}

    \begin{align*}     \left = 0 + 0 + \frac{1}{\sigma^2} \ [x_1 + x_2 + .... + x_n - n\mu] \end{align*}

(5)   \begin{equation*}     \left = \frac{1}{\sigma^2} \ [x_1 + x_2 + .... + x_n - n\mu] \end{equation*}

Thus, (5) is the desired MLE.

Subscribe to Ehan Ghalib!

Invalid email address
We promise not to spam you. You can unsubscribe at any time.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>