%% ================================================================
\newpage
\section{Bayes' theorem}

Bayes' theorem is so important that it merits multiple points of view:
algebraic, graphical, and numerical.

%% ----------------------------------------------------------------
\subsection{Algebraic approach}

Recall from definition \ref{defn:conditional_probability} that if $A$ is an
event, and if $B$ is another event with non-zero probability, the
\bfidx{conditional probability} of $A$ given $B$ is
$$
	P(A|B) = \frac{P(A\cap B)}{P(B)}.
$$
Bayes' theorem tells us how to invert this:  how to compute the probability of
$B$ given $A$.  First, the algebraic treatment.

\begin{thm}[Bayes' theorem]
Let\index{Bayes' theorem} $A$ and $B$ be events with non-zero probability.
Then
$$
	P(B|A) = P(A|B) \; \frac{P(B)}{P(A)}.
$$
\begin{proof}
Using the definition (intersection over given), we have
\begin{eqnarray*}
	P(B|A) &=& \frac{P(B\cap A)}{P(A)}.
\end{eqnarray*}
Multiplying top and bottom by $P(B)$ (which is OK since $P(B) \ne 0$) we get
\begin{eqnarray*}
	P(B|A) &=& \frac{P(B\cap A)}{P(A)} \; \frac{P(B)}{P(B)}.
\end{eqnarray*}
Now notice that $B \cap A$ is the same as $A \cap B$, so in particular $P(B
\cap A)$ is the same as $P(A \cap B)$.  Transposing the terms in the
denominator gives us
\begin{eqnarray*}
	P(B|A) &=& \frac{P(A\cap B)}{P(B)} \; \frac{P(B)}{P(A)} \\
		&=& P(A|B) \; \frac{P(B)}{P(A)}
\end{eqnarray*}
as desired.
\end{proof}
\end{thm}

%% ----------------------------------------------------------------
\subsection{Graphical/numerical approach}

% xxxx more graphical first:
% a b
% c d
%
% bayes as pf. that d/(c+d) = d/(b+d) (c+d)/(c+d)

The following example is adapted from \cite{bib:kk}.  Suppose that $0.8\%$ of
the general population has a certain disease, and suppose that we have a test
for it.  Specifically, if a person actually has the disease, the test says so
$90\%$ of the time.  If a person does not have the disease, the test gives a
false diagnosis $7\%$ of the time.  When a particular patient tests positive,
what is the probability they have the disease?

We can write this symbolically as follows.  Let $D$ be the event that the
person has the disease and $\Dbar$ be its complement; let $Y$ (for ``yes'') be
the event that the test \emph{says} the person has the disease, with complement
$\Ybar$.  Writing the given information in these terms, we have:
\begin{eqnarray*}
	P(D)       &=& 0.008 \\
	P(Y|D)     &=& 0.90 \\
	P(Y|\Dbar) &=& 0.07.
\end{eqnarray*}
Before computing any conditional probabilities, let's find the probability
of $Y$ by itself.  Using the partition theorem (theorem \ref{thm:partition})
we know this is
\begin{eqnarray*}
	P(Y) &=& P(Y|D)\,P(D) \;+\; P(Y|\Dbar)\,P(\Dbar) \\
	&=& 0.90 \cdot 0.008 + 0.07\cdot 0.992 \\
	&=& 0.077.
\end{eqnarray*}
So the non-conditional probabilities are
\begin{align*}
	P(\Dbar)  &= 0.992  & P(\Ybar)  &= 0.923 \\
	P(D)      &= 0.008  & P(Y)      &= 0.077.
\end{align*}

Looking at $D$ and $Y$ separately, we can think of the population at large as
being split into those with and without the disease, and those for whom the
test is positive or negative.  Suppose in particular that we have a sample of
1000 people who are representative of the general population.  Here are some
(very rectangular) Venn diagrams:
$$
%
	\begin{tabular}{|c|c||c|}
	\hline $\Dbar$ & $D$ & \\
	\hline
	&&\\
	$992$ & $8$ & \\
	&&\\
	\hline
	\end{tabular}
%
	\qquad
	\qquad
%
	\begin{tabular}{|ccc||c|}
	\hline $\;$ & $\;$ & $\;$ & $\;$ \\
	\hline
	\hline  & &923& $\Ybar$ \\
	\hline  & & 77& $Y$ \\
	\hline
	\end{tabular}
%
$$

Bayes' theorem has to do with how these two partitions intersect to make four
groups:
$$
	\begin{tabular}{|r|r||c|}
	\hline $\Dbar$ & $D$ & \\
	\hline
	\hline  $P(\Ybar|\Dbar) \cdot 992 = ?$ & $P(\Ybar|D) \cdot 8 = ?$ & $\Ybar$ \\
	\hline  $P(Y    |\Dbar) \cdot 992 = ?$ & $P(Y    |D) \cdot 8 = ?$ & $Y$ \\
	\hline
	\end{tabular}
%
	\qquad
	\textrm{and}
	\qquad
%
	\begin{tabular}{|r|r||c|}
	\hline $\Dbar$ & $D$ & \\
	\hline
	\hline  $P(\Dbar|\Ybar) \cdot 923 = ?$ & $P(D|\Ybar) \cdot 923 = ?$ & $\Ybar$ \\
	\hline  $P(\Dbar|Y)     \cdot  77 = ?$ & $P(D|Y)     \cdot  77 = ?$ & $Y$ \\
	\hline
	\end{tabular}
$$

We can use the theorem to find the probability our patient has the
disease, given the positive test result:
\begin{eqnarray*}
	P(D|Y) &=& P(Y|D) \; \frac{P(D)}{P(Y)} \\
	&=& 0.90 \;\cdot\; \frac{0.008}{0.077} \\
	&=& 0.094.
\end{eqnarray*}
That is, there's only a one-in-eleven chance the patient actually has the
disease.  Kaplan and Kaplan's idea is to look at this surprising result in the
context of the other 999 people also tested.  I will elaborate on this, working
out all the math.

We have two conditional probabilities given:  $P(Y|D)$ and $P(Y|\Dbar)$.  We
found $P(D|Y)$.  What about $P(D|\Ybar)$?  We can use the  partition theorem
(theorem \ref{thm:partition}) again to solve for what we don't know in terms of
what we do know:
\begin{eqnarray*}
	P(D) &=& P(D|Y)\,P(Y) \;+\; P(D|\Ybar)\,P(\Ybar) \\
	P(D|\Ybar) &=& \frac{P(D) - P(D|Y)\,P(Y)}{P(\Ybar)} \\
	&=& \frac{0.008 - 0.094 \cdot 0.077}{0.923} \\
	&=& 0.0008.
\end{eqnarray*}
We now have all four conditional probabilities:
\begin{align*}
	P(Y|\Dbar) &= 0.07  & P(D|\Ybar) &= 0.0008 \\
	P(Y|D)     &= 0.90  & P(D|Y)     &= 0.094.
\end{align*}

Now we can fill out the four-square table:
\begin{itemize}

\item Since $P(Y|\Dbar)=0.07$, seven percent of the 992 disease-free people (70
of them) get false positives; the rest (922) get a correct negative result.

%These are the people in the left-hand column of the four-square table.

\item Since $P(Y|D)=0.90$, ninety percent of the 8 people with the disease test
positive (i.e. all but one of them); one of the 8 gets a false sense of security.

\item Since $P(D|\Ybar)=0.0008$, $0.08\%$ of the 923 with negative test results
(one person) does in fact have the disease; the other 922 (as we found just above)
get a correct negative result.

\item Since $P(D|Y)=0.094$, only $9.4\%$ of the 77 people with positive test
results (7 people) have the disease; the other 70 get a scare (and, presumably,
a re-test).

\end{itemize}

So, the sample of 1000 people splits up as follows:
$$
	\begin{tabular}{|r|r||c|}
	\hline $\Dbar$ & $D$ & \\
	\hline
	\hline  $922$ & $1$ & $\Ybar$ \\
	\hline  $70$ & $7$ & $Y$ \\
	\hline
	\end{tabular}
%
$$
Moreover, we can rank events by likelihood:
\begin{itemize}
\item[(1)] Healthy people correctly diagnosed: $92.2\%$.
\item[(2)] False positives: $7\%$.
\item[(3)] People with the disease, correctly diagnosed: $0.7\%$.
\item[(4)] False negatives: $0.1\%$.
\end{itemize}

Now it's no surprise our patient got a false positive:  this happens 10 times
as often as a correct positive diagnosis.

%% ----------------------------------------------------------------
\subsection{Asymptotics}

The specific example provided some insight, but what happens when we vary the parameters?
We had
$$
	P(D|Y) =
	\frac{P(Y|D)\,P(D)}{P(Y)} =
	\frac{P(Y|D)\,P(D)}{P(Y|D)\,P(D) \;+\; P(Y|\Dbar)\,P(\Dbar)}.
$$
Let's turn this into a design problem.  Let
\begin{eqnarray*}
	p = &=& P(D) \\
	a = &=& P(Y|D) \\
	b = &=& P(Y|\Dbar) \\
	1-\veps &=& P(D|Y) \\
	\veps &=& P(\Dbar|Y).
\end{eqnarray*}
How would you choose the test-design parameters $a$ and $b$ (i.e. how good
would your test have to be) to get $\veps$ small?  What if the disease is rarer
($p$ smaller)?

Suppose we want a high likelihood of correct detection, i.e. $\veps$ small.
Then
\begin{eqnarray*}
	P(D|Y) &=&
		\frac{P(Y|D)\,P(D)}{P(Y|D)\,P(D) \;+\; P(Y|\Dbar)\,P(\Dbar)} \\
	&=& \frac{ap}{ap + b(1-p)} \\
	P(\Dbar|Y) &=& \frac{b(1-p)}{ap + b(1-p)} \\
 	\frac{b(1-p)}{ap + b(1-p)} &<& \veps.
\end{eqnarray*}
Solving for $a$ and $b$ we get
$$
	\frac{a}{b} > \frac{(1-\veps)(1-p)}{\veps p}.
$$
There are two free parameters, $a$ and $b$, so I'll just consider their ratio.
Now, the function
$$
	\frac{1-x}{x}
$$
blows up near zero; for small $x$, it's approximately $1/x$.  For small $\veps$
and $p$ we have
$$
	\frac{b}{a} < \veps p.
$$
If, say, we have $p$ and $\veps$ both 0.001, then $a$ and $b$ need to differ by
a factor of a million.  Recall that $a$ and $b$ are both probabilities, and so
range between zero and one.  To test a one-in-a-million event with $99.99\%$
confidence ($p=10^{-6}$ and $\veps = 10^{-4}$), $b$ must be less than
$10^{-10}$.

That tells us how to choose $P(Y|\Dbar)$ to in order to get $P(\Dbar|Y)$, i.e.
the probability of false positives, small.  What about false negatives:
$P(D|\Ybar)$?  If you do similar algebra to the above you should find that
getting less than $\veps$ requires
$$
	1-a < \frac{\veps}{p}.
$$
This is a less strict constraint: $P(Y|D)$ needs to be very close to 1 only
when $\veps \ll p$.

%% ----------------------------------------------------------------
\subsection{Conclusions}

Some points:
\begin{itemize}

\item $P(Y|D)$ is information known to the person who creates the test --- say,
at a pharmaceutical company; $P(D|Y)$ is information relevant to the people who
give and receive the test --- for example, at the doctor's office.  This
duality between design and implementation suggests that Bayes' theorem has
important consequences in many practical situations.

\item The results can be surprising --- after all, in the example above the
test was $90\%$ accurate, was it not?  Bayes' theorem is important to know
precisely because it is counterintuitive.

\item We can see from the example and the asymptotics above that rare events
are hard to test accurately for.  If we want certain testing for rare events,
the test might be impossible (or overly expensive) to design in practical terms.
\end{itemize}
