%% ================================================================
\newpage
\section{Statistics}

The key concept here is parameter estimation.  My goals are:

\begin{itemize}

\item Present a few concepts from parametric statistics, with an
example-heavy approach.

\item Give unified notation for probability and statistics.

\item Work out several parameter-estimation examples concretely.

\end{itemize}


%% ================================================================
\subsection{Sampling}

%% ----------------------------------------------------------------
\subsubsection{Finite-population example}

Work out an example with finite population $\{x_1, \ldots, x_n\}$ and simple
random sample with replacement $\{X_1, \ldots, X_N\}$.

Define $\mu$ and $\sigma^2$.

Define \bfidx{unbiased estimator}.

Show that $\Xbar$ and $s^2$ are unbiased estimators of $\mu$ and $\sigma^2$
respectively.  Explain the factor of $n-1$ in $s^2$.

%% ----------------------------------------------------------------
\subsubsection{Infinite-population example}

%% ================================================================
\subsection{Decision theory}
\label{sec:decision_theory}

The presentation here follows \cite{bib:rabi}.  I am simply tabulating and
elaborating upon his definitions.

One has the following:

\begin{itemize}

\item A \bfidx{population space} $\Omega$.

\item An \bfidx{observation space} or \bfidx{sample space} $\barX = \Omega^n$
containing \bfidx{observations} $\bfX = (X_1, \ldots, X_n)$ which are
$n$-tuples of IID random variables on $\Omega$.

\item A \bfidx{parameter space} $\Theta$ indexing (for $\theta \in \Theta$) a
family of \plainidx{probability measures} $\{P_\theta\}$ on $(\Omega,\mcF)$.
Then for each $P_\theta$ one obtains a \plainidx{probability space} $(\Omega,
\mcF, P_\theta)$.  One can think of these measures $P_\theta$ as
\bfidxtwo{conditional probabilities}{conditional probability}
$P(\vecx\mid\theta)$.

Nominally, $\Theta$ is $\R^d$ or some subset thereof.

\item An \bfidx{action space} $\mcA$ indexing (for $\theta \in
\Theta$) a family of \plainidx{probability measures} $P_\theta$ on
$(\Omega,\mcF)$.

Nominally, $\mcA$ is all or part of $\Theta$.  This is best explained by
example:  Suppose $\theta=(\mu,\sigma^2)$ where each $\theta$ is a 2-tuple
defining a normal probability distribution on the real line.  Then one might
want to use one's observation only to estimate $\mu$.  Then $\Theta=\R \times
\R^+$ whereas $\mcA$ is merely $\R$.

\item The population space, sample space, parameter space, and action space are
all measurable spaces with their respective $\sigma$-algebras.

\item A \bfidx{loss function} $L:(\Theta,\mcA) \to \R$.  This quantifies the
loss incurred when $\theta$ (e.g. the \plainidx{population mean} $\mu$) is
estimated by $a$ (e.g. the \plainidx{sample mean} $\Xbar$).
The most common loss function is the \bfidx{least-squared error}
$$
	L(\theta,a) = \|\theta-a\|^2.
$$

\item A \bfidx{decision rule} $d:\barX \to \mcA$.
For example,
$$
	a = d(\vecX) = \Xbar := \frac{\sum_{i=1}^n X_i}{n}.
$$

[xxx to do:  Define admissibility.  Sufficient: rabi thm 3.3 and
cor 3.1.  Make some plots.]

\item The \bfidx{risk function} associated with a given decision rule
is then
$$
	R(\theta,d) := E_\theta\left[L(\theta, d(\vecX))\right].
$$
The subscript on the $E$ reminds us (for cases when we need reminding) which
variable(s) not to integrate out:  $E_\theta$ of something will be
(potentially) a function of $\theta$ and so we won't integrate over $\theta$.
In particular,
\begin{align}
	R(\theta,d)
\label{eqn:risk_function_E}
		&= E_\theta\left[L(\theta, d(\vecX))\right] \\
\label{eqn:risk_function_integral}
		&= \int_\barX L(\theta,d(\vecx)) \, dP_\theta(\vecx).
\end{align}

\end{itemize}

To summarize, the players in a decision problem are:
$$
\begin{tabular}{|ccccc|}
\hline
& & & & \\
& $\Omega$      & $\barX$       & & \\
& $\Theta$      & $\mcA$        & $d$ & \\
& $L(\theta,a)$ & $R(\theta,d)$ & & \\
& & & & \\
\hline
\end{tabular}
$$

\beginex
Let the $X_i$'s be IID with mean $\mu$ and variance $\sigma^2$.  If the
decision rule is $a=\Xbar$ with $\theta=\mu$ and least-squared-error loss
function, then
\begin{align*}
	R(\theta,d) &= E_\mu\left[(\mu-\Xbar)^2\right] \\
	&= \mu^2 - 2\mu E[\Xbar] + E[\Xbar^2] \\
	&= \mu^2 - \frac{2\mu}{n} E\left[\sum_{i=1}^n X_i\right]
		+ \frac{1}{n^2} E\left[\sum_{i=1}^n \sum_{j=1}^n X_i X_j\right] \\
	&= \mu^2 - \frac{2\mu}{n} \sum_{i=1}^n E[X_i]
		+ \frac{1}{n^2} \sum_{i=1}^n \sum_{j\ne i} E[X_i X_j]
		+ \frac{1}{n^2} \sum_{i=1}^n E[X_i^2].
\end{align*}
Now, the $X_i$'s are IID so $E[X_i]=\mu$ is the same for all $i$,
and $E[X_iX_j]=E[X_i]E[X_j]$ for $i \ne j$.  Then
\begin{align*}
	R(\theta,d)
	&= \mu^2 - 2\mu^2
		+ \frac{n(n-1)}{n^2} \mu^2
		+ \frac{n}{n^2} E[X_1^2].
\end{align*}
Recall that the variance was
\begin{align*}
	\sigma^2 &= E[(X_1-\mu)^2]
	= E[X_1^2] - 2 \mu E[X_1] + \mu^2
	= E[X_1^2] - \mu^2
\end{align*}
so
\begin{align*}
	E[X_1^2] &= \sigma^2 + \mu^2.
\end{align*}
Then
\begin{align*}
	R(\theta,d)
	&= - \mu^2
		+ \frac{(n-1)}{n} \mu^2
		+ \frac{1}{n} \sigma^2
		+ \frac{1}{n} \mu^2
	= \frac{\sigma^2}{n}.
\end{align*}
This means that, if we use the sample mean $\Xbar$ to estimate the population
mean $\mu$, our risk increases with larger population variance and decreases
with larger sample size.
\mendex

%% ================================================================
\subsection{Parameter estimation}

%% ----------------------------------------------------------------
\subsubsection{Maximum-likelihood estimation}

log-likelihood example as well.

%% ----------------------------------------------------------------
\subsubsection{Method of moments}

%% ----------------------------------------------------------------
\subsubsection{Bayes estimation}

Here $\vartheta\in\Theta$ is thought of as a random variable.

Notation:
$$
\begin{tabular}{|c|c|c|}
\hline Value & Random variable & Space \\
\hline
\hline $x$      & $X$         & $\barX$ \\
\hline $\theta$ & $\vartheta$ & $\Theta$ \\
\hline
\end{tabular}
$$

One somehow knows (or guesses) a probability distribution $\tau$ on $\Theta$.
This is called the \bfidx{prior distribution} or simply \textbf{prior}.
This encodes what we know about $\theta$ values prior to making an
observation $\vecX$.
(Below we'll have a posterior distribution which is conditioned on the
observation $\vecX$.)
If $\tau$ has a density $f_\vartheta(\theta)$ with respect to Lebesgue measure
then we write
$$
	d\tau(\theta) = f_\vartheta(\theta) \, d\theta.
$$

We have a loss function $L(\theta,a)$ as defined in section
\ref{sec:decision_theory}.  This will always be least-squared error unless
otherwise noted.

\begin{defn}
The \bfidx{Bayes risk} of a decision rule $d$ is
$$
	r(\tau,d) = \int_\Theta R(\theta,d) \,d\tau(\theta).
$$
\end{defn}
By equation \ref{eqn:risk_function_E}, this is
\begin{align*}
	r(\tau,d) &= \int_\Theta E_\theta\left[
		L(\theta, d(\vecX))
		\right] \,d\tau(\theta)
\end{align*}
which by equation \ref{eqn:risk_function_integral} is, in turn,
\begin{align*}
	r(\tau,d) &= \int_\Theta
		\left[ \int_\barX L(\theta,d(\vecx)) \, dP_\theta(\vecx) \right]
		\,d\tau(\theta)
\end{align*}

\begin{defn}
\label{def:bayes_rule}
A \bfidx{Bayes rule} $d_0$ is a decision rule which minimizes Bayes risk:
$$
	r(\tau,d_0) = \inf_{d} \, r(\tau,d)
$$
where the infimum is taken across all decision rules $d$.  Note that a
minimizer may not exist.
\end{defn}

Recall from section \ref{sec:conditional_expectation} that if we have
two random variables $\vecX$ and $\vartheta$, then:

\begin{itemize}

\item Given the \plainidx{joint density} $f_{\vecX,\vartheta}(\vecx,\theta)$
we can integrate out $\vecx$ to obtain the \plainidx{marginal density}
$f_\vartheta(\theta)$:
$$
	f_\vartheta(\theta) = \int_\barX f_{\vecX,\vartheta}(\vecx,\theta)\,d\vecx.
$$
Likewise,
we can integrate out $\theta$ to obtain the \plainidx{marginal density}
$f_\vecX(\vecx)$:
$$
	f_\vecX(\vecx) = \int_\Theta f_{\vecX,\vartheta}(\vecx,\theta)\,d\theta.
$$

\item We have new random variables $\vecX\mid\vartheta$ and
$\vartheta\mid\vecX$.

\item We can compute their \plainidx{conditional expectation}s
$$
	E[\vecX\mid\vartheta]
	\qqand
	E[\vartheta\mid\vecX].
$$

\item We have \plainidx{conditional density} which is joint over marginal:
$$
	f_{\vecX \mid \vartheta}(\vecx\mid\theta)
	= \frac
		{f_{\vecX,\vartheta}(\vecx,\theta)}
		{f_\vartheta(\theta)}
	= \frac
		{f_{\vecX,\vartheta}(\vecx,\theta)}
		{\int_\barX f_{\vecX,\vartheta}(\vecx,\theta)\,d\vecx}
$$
and likewise
$$
	f_{\vartheta \mid \vecX}(\vecx\mid\theta)
	= \frac
		{f_{\vecX,\vartheta}(\vecx,\theta)}
		{f_\vecX(\vecx)}
	= \frac
		{f_{\vecX,\vartheta}(\vecx,\theta)}
		{\int_\Theta f_{\vecX,\vartheta}(\vecx,\theta)\,d\theta}.
$$

\item Given these two facts, we can solve (just as in Bayes' theorem, theorem
\ref{thm:bayes}) for one conditional density in terms of the other:
$$
	f_{\vartheta \mid \vecX}(\theta\mid\vecx)
	=
	f_{\vecX \mid \vartheta}(\vecx\mid\theta) \;
	\frac
		{f_\vartheta(\theta)}
		{f_{\vecX}(\vecx)}.
$$

\end{itemize}

These facts motivate the following definition.

\begin{defn}
The \bfidx{posterior distribution} of $\vartheta$ given $\vecX$ is
$$
	f_{\vartheta \mid \vecX}(\theta\mid\vecx)
	=
	f_{\vecX \mid \vartheta}(\vecx\mid\theta) \;
	\frac
		{f_\vartheta(\theta)}
		{f_{\vecX}(\vecx)}
$$
\end{defn}

\begin{defn}
The \bfidx{posterior mean} is the expectation of the posterior distribution
$f_{\vartheta\mid\vecX}(\theta\mid\vecx)$.
\end{defn}

The theorem is that the posterior mean
$$
	d_0(\vecX) = E[\vartheta\mid\vecX]
$$
is a Bayes estimator of $\theta$, satisfying definition \ref{def:bayes_rule}.
Following \cite{bib:cb}, I currently think this is because of the following
(assuming the distributions of $\vecX$ and $\vartheta$ both have densities as
above, and using Bayes' theorem):
\begin{align*}
	r(\tau,d) &= \int_\Theta R(\theta,d) \,d\tau(\theta) \\
	&= \int_\Theta E_\theta\left[L(\theta, d(\vecX))\right] \,d\tau(\theta) \\
	&= \int_\Theta
		\left[ \int_\barX L(\theta,d(\vecx)) \, dP_\theta(\vecx) \right]
		\,d\tau(\theta) \\
	&= \int_\Theta
		\left[ \int_\barX L(\theta,d(\vecx)) \, dP(\vecx \mid \theta) \right]
		\,d\tau(\theta) \\
	&= \int_\Theta
		\left[ \int_\barX L(\theta,d(\vecx)) \,
			f_{\vecX\mid\vartheta}(\vecx \mid \theta) \, d\vecx
			\right] f_\vartheta(\theta) \,d\theta \\
	&=
		\int_\barX
		\left[
		\int_\Theta
		L(\theta,d(\vecx)) \,
		f_{\vartheta\mid\vecX}(\theta \mid \vecx)
		\,d\theta
		\right]
		f_\vecX(\vecx)
		\,d\vecx.
\end{align*}
The quantity in square brackets, which is a function of $\vecx$, is called
the \bfidx{posterior expected loss}.
(The conditionals in the integrals are reminiscent of the partition theorem
[theorem \ref{thm:partition_thm}].)
When computing a Bayes rule, one
selects $d$ to minimize the posterior expected loss for each $\vecx$.

To compute $d_0(\vecX)$, we need to find all three right-hand terms in
$$
	f_{\vartheta \mid \vecX}(\theta\mid\vecx)
	=
	f_{\vecX \mid \vartheta}(\vecx\mid\theta) \;
	\frac
		{f_\vartheta(\theta)}
		{f_{\vecX}(\vecx)}.
$$

These are found as follows:

\begin{itemize}

\item $f_\vartheta(\theta)$ is the given prior.

\item $f_{\vecX \mid \vartheta}(\vecx\mid\theta)$ is given as the distribution
of $\vecX$ with parameter $\theta$.

\item $f_{\vecX}(\vecx)$ is the marginal of $f_{\vecX \mid
\vartheta}(\vecx\mid\theta)$, found by integrating out $\theta$.  However, as
usual in probability, one may find a trick to avoid doing the integral.

%% ----------------------------------------------------------------
\subsubsection{Minimax estimation}

def'n.  Sufficient:  rabi thm 3.6 and 3.7.

\end{itemize}
