Hassett, chapter 3: Elements of probability
3.1 Probability by counting for equally likely outcomes
$$ \text{Probability of an event} = \frac{\text{Number of outcomes in the event}} {\text{Total number of possible outcomes}}. $$ or $$ P(E) = \frac{n(E)}{n(S)} $$ with $S$ the sample space, $E$ the event, and $n()$ returning the cardinality of a set.
Probabilities add to one, which gives the negation rule $$ P(\bar E) = 1 - P(E). $$
There's also the "disjunction rule,"
$$ P(A\or B) = P(A) + P(B) - P(A\and B), $$
where the subtraction happens because we don't want to count the intersection twice.
disjunction distraction
How does this extend to multiple non-mutually-exclusive events? I'm pretty sure it's
$$ \begin{aligned} P(A\and B\and C) {}={} & P(A) + P(B) + P(C) \\ &{}- P(A\and B) - P(A\and C) - P(B\and C) \\ &{}+ P(A\and B \and C) \end{aligned} $$
Example 3.6
From twenty male and thirty female employees, five are selected at random for drug testing. What is the probability that the five are (a) three males and two females, (b) all males, or (c) all females?
There are $n(S) = {50\choose 5} = 2\,118\,760$ ways to choose five employees. The individual cases are $$ \begin{aligned} n(A) &= {20\choose 3}⋅{30\choose2} & P(A) &\approx 0.234 \\ n(B) &= {20\choose 5}⋅{30\choose0} & P(B) &\approx 0.007\,32 \\ n(C) &= {20\choose 0}⋅{30\choose5} & P(C) &\approx 0.067\,3 \end{aligned} $$
Example 3.8: ESP and the birthday paradox
Four people guess a digit 0–9. Do any two of them guess the same digit? There are $10^4$ possible outcomes, of which $10⋅9⋅8⋅7 = P(10,4) = 5040$ have all digits different. So the split is $50.4\%$ to $49.6\%$.
Twenty people are in a room. The probability that zero of them have the same birthday is $$ p = P(365,20)/365^{20} \approx 0.59 $$
This function is steeper than folks think:
General probabilities:
Suppose that $E_1,E_2,\cdots,E_n\cdots$ represent events that are mutually exclusive. Then $$ P\left(\bigcup_{i} E_i\right) = \sum_i P(E_i) $$
3.3 Conditional probability
| Male ($M$) | Female ($F$) | total | |
|---|---|---|---|
| Smokers ($S$) | 28 | 22 | 50 |
| Nonsmokers ($\bar S$) | 72 | 78 | 150 |
| Total | 100 | 100 | 200 |
Suppose we wanted to predict whether a person was a smoker, given they were male, probability written $P(S|M)$. This is totally 28%, because we put the number of males in the denominator.
$$ P(S|M) = \frac{n(S\and M)}{n(M)} = \frac{28}{100} $$
Sticking in some $n(\text{total})$ so that we are dealing only with probabilities, we get
$$ P(S|M) = \frac{n(S\and M)/n(\text{total})}{n(M)/n(\text{total})} = \frac{P(S\and M)}{P(M)} $$
We could also write $$ P(S|M) ⋅ P(M) = P(S\and M), $$ which we'd read as "the probability of $S$ and $M$ is equal to the probability of $S$ given $M$, multiplied by the probability of $M$."
3.4 Independence
We define independent events $A,B$ as obeying $P(A|B) = P(A)$. This means that independent events obey $$ P(A\and B) = P(A)⋅ P(B). $$
3.5 Bayes' Theorem
Suppose you have a medical test with false positive and false negative error rates. Let $D$ be the presence of the disease, and $Y$ be a positive result on the test. Perhaps the test has a $5\%$ chance of giving a false negative on a person who definitely has the disease, $$ P(N|D) = 0.05, $$ and a $3\%$ chance of a false positive on a person who definitely does not have the disease, $$ P(Y|\bar D) = 0.03. $$ But the disease is rare, $P(D) = 0.01$. Because the disease is rarer than the error rate, most of the positive results are false positives.
Now we have no information about $D$, but we get a positive test result. Out of $100^2$ cases, our results are
| $Y$ | $N$ | total | |
|---|---|---|---|
| $D$ | $1\times95 = 95$ | $1\times5 = 5$ | 100 |
| $\bar D$ | $99\times 5 = 495$ | $99\times 95 = 9405 $ | 9900 |
| total | 590 | 9410 | $10\,000$ |
If the information that you have is a $Y$ on the test, the probability of the disease is now $P(D|Y) = \frac{95}{590}=0.161$. The test tells you to pay attention, but it's not by itself a strong indicator of disease.
From the definition of conditional probabilities, we can say $$ P(A\and B) = P(A|B)P(B) = P(B|A)P(A). $$
So to swap the conditional, we want to say $$ P(B|A) = P(A|B) \frac{P(B)}{P(A)}. $$
In this case, $$ P(D|Y) = P(Y|D)\frac{P(D)}{P(Y)} $$ This isn't quite what I wrote earlier. That was the false positive rate, $P(Y|\bar D) = 0.03 = \frac{P(Y\and \bar D)}{P(\bar D)}$. But the probability of $Y$ is the sum $P(Y) = P(Y\and \bar D) + P(Y\and D)$. That gives us $$ P(D|Y) = \frac{P(D) P(Y|D)}{P(Y\and D) + P(Y\and\bar D)}. $$
This is not going where I had remembered.
Second try: follow the text more closely.
Here's the tree diagram for this test, redrawn from the text. (I had tried to make the false positive and false negative rates different from each other, but when I got confused I couldn't confirm my numbers.)
Two questions:
- What percentage of the population will test positive? That is, what's $P(Y)$?
- Suppose you know someone has a positive test. What's the probability they do not actually have the disease? This is $P(\bar D|Y)$.
We have $P(Y) = 0.01\times 0.95 + 0.99\times 0.05 = 0.059$: about six percent of the population will have a positive result. But most of those are false:
$$ P(D|Y) = \frac{P(D\and Y)}{P(Y)} = \frac{0.01\times0.95}{0.059} = 0.161. $$
Or $$ P(\bar D|Y) = \frac{P((\bar D)\and Y)}{P(Y)} = \frac{0.05\times0.99}{0.059} = 0.839. $$
Note that $P(D|Y) + P(\bar D|Y) = 1$. This is the "law of total probability." If a bunch of events $A_i$ partition the sample space, then the total probability of some event $E$ is $$ P(E) = P(E\and A_1) + P(E\and A_2) + \cdots $$ from the definition of the partition, $A_1\and A_2\and\cdots = S$.
Then Bayes' Theorem goes like this.
Let $E$ be an event, and let $A_1,A_2,\cdots$ partition the sample space. Then
$$ \begin{aligned} P(A_i|E) &= \frac{P(E\and A_i)}{P(E)} \\ &= \frac{P(A_i)⋅ P(E|A_i)}{ P(A_1)⋅ P(E|A_1) + P(A_2)⋅ P(E|A_2) +\cdots } \end{aligned} $$
Example 3.24
An insurer has three types of auto insurance policyholders. There are 50% of the policyholders who are low risk, $L$. The probability that a low-risk policyholder will file a claim in a given year is $0.10$. Another 30% are moderate risk, $M$. The probability that a moderate-risk policyholder will file a claim in a given year is $0.20$. The rest are high risk, $H$, with claim probability 0.50. A policyholder files a claim this year. What's the probability that he is in $H$?
We want $P(H|Y) = \frac{P(H)⋅ P(Y|H)}{P(Y)}$. The tricky thing here is $$ P(Y) = [0.5, 0.3, 0.2] ⋅ [0.1, 0.2, 0.5] = 0.21 $$ so $$ P(H|Y) = \frac{0.20\times 0.50}{0.21} = 0.476 $$
It feels like this data might be stored better in a table than in a diagram. But actually trying to make the tables is a little iffy.
Problems
question 3-7
An urn contains four red balls and six green balls. Three are selected at random.
What's the probability all three are green? $$ P(\text{three red}) = \frac{4}{10}⋅ \frac{3}{9}⋅ \frac{2}{8} = \frac{1}{30}. $$
But what's the multinomial way to do this? It's $$ P(\text{three red}) = \frac{ {4\choose3}{6\choose0} }{ 10\choose 3 } = \frac{4}{1} ⋅ \frac11 \times\left( \frac{10⋅9⋅8}{3⋅2⋅1} \right)^{-1} $$
Likewise $ P(\text{one red, two green}) = {{4\choose1}{6\choose2}} / { 10\choose 3} $.
The probability they're all the same is $ P(\text{three red}) + P(\text{three green}) = \frac{1}{30} + {4\choose 0}{6\choose 3} / {10\choose 3} = \frac{1}{30} + 20/120 = 1/5 $
question 3-13
In poker, what's the probability of drawing a full house? That's three of a kind and a pair, e.g. KKK33.
There are fifty-two cards divided into four suits of thirteen card values. So I think the answer should be something like
$$P(\text{full house}) \neq {13\choose 3}{12\choose 2}\Big/{52\choose 5} \approx 0.007\,263 $$
But the text gives $P(\text{full house})\approx 0.0014$, smaller by a factor of five-ish. Why?
… Oh, I see. I'm not choosing three of the thirteen face values, nor two of the remaining twelve face values. I'm choosing a suit for the trey and a suit for the pair. Then there are $3!$ ways to order the trey and $2!$ ways to order the pair. So perhaps
$$ P(\text{full house}) = 3!{13\choose 1}\times 2!{12\choose 1} \Big/ {52\choose 5} \approx 0.000\,720\,3 $$
Also wrong.
Perhaps there are thirteen possible face values for the trey, and four ways to choose which card to leave out, for ${13\choose 1}{4\choose 3} \times {12\choose 1}{4\choose 2}$ full houses a out of ${52\choose5}$ hands. That gives
$$ P(\text{full house}) = {13\choose 1}⋅{4\choose 3} \times {12\choose 1}⋅{4\choose 2} \Big/ {52\choose 5} \approx 0.001\,441 $$
That's it! That's apparently the way to think about it.
question 3-14
What's the probability of drawing two pairs, like QQ55A?
By the above logic,
$$ \begin{aligned} P(\text{two pairs}) &= {52\choose 5}^{-1} \times 13⋅{4\choose 2} \times 12⋅{4\choose 2} \times 11 \\ &= {52\choose 5}^{-1}\times P(13,3)\times {4\choose 2}^2 \approx 0.023\,77 \end{aligned} $$
But that underestimates the count by factor of $2!$, because (unlike in the full house) you can't distinguish which of the two pairs is first.
question 3-26
Draw two cards from a deck. They can be zero, one, or two hearts. What's the probability of each?
There are 13 hearts and 39 non-hearts, with $52\times51$ total hands.
Zero hearts should be ${39\choose 2}\big/{52\choose 2} = \frac{78}{1326}$? No, that can't possibly be right.
Is this the same issue as before, where I'm mixing up which are the cards versus which are the suits?
This is a question of conditional probability. Let $A,B$ be the events where the first and second cards are hearts. Then the outcomes we care about are
$$ \begin{aligned} \text{zero hearts: }&& (\bar A)&\and(\bar B)\\ \text{one heart: }&& ((\bar A)\and B) &\or (A\and(\bar B)) \\ \text{two hearts: }&& A&\and B \end{aligned} $$
The full deck of 52 cards has 13 hearts and 39 non-hearts. Either of $A,B$ alone should have probability $13/52=1/4$, by symmetry. Let's consider $P(B|A)$ and $P(B|\bar A)$.
After $A$, there are 12 hearts and 39 non-hearts, so $P(B|A) = 12/51$.
After $\bar A$, there are 13 hearts and 38 non-hearts, so $P(B|\bar A) = 13/51$.
These should add to $P(B)$: $$ \begin{aligned} P(B) &= P(B|A)P(A) + P(B|\bar A)P(\bar A) \\ &= \frac{12}{51}⋅\frac{13}{52} + \frac{13}{51}⋅\frac{39}{52} \\ &= \frac{156 + 507}{2652} = \frac14. \end{aligned} $$
I think that means $$ \begin{aligned} \text{two hearts: }&\\ P( A\and B) &= P( B | A) \times P( A) \\&= \frac{12}{51} \times\frac{13}{52} = \frac{156}{2652} \approx 0.05882 \\ \\ \text{zero hearts: }&\\ P( \bar A\and \bar B) &= P( \bar B | \bar A)\times P( \bar A) \\&= \frac{38}{51}\times\frac{39}{52} = \frac{1482}{2652} \approx 0.5588 \end{aligned} $$
For the single heart, there are two hands that put the heart first, thirteen choices for the heart, and thirty-nine choices for the non-heart: $$ \begin{aligned} P(\text{one heart}) = \frac{2\times 13\times 39}{52\times 51} = \frac{1014}{2652} \approx 0.3824. \end{aligned} $$
This enumeration does in fact account for all 2652 possible hands.
Three that I mistrust
These three are similar, but I think the third one has an error in the textbook.
question 3-27
For the experiment of tossing a single fair coin three times, what is the probability of getting exactly two heads, given that you get at least one head?
The outcome space is
| outcome | n(outcome) |
|---|---|
| zero heads | 1 |
| one head | 3 |
| two heads | 3 |
| three heads | 1 |
Therefore $P(2|\set{1,2,3}) = \frac{P(2)}{P(1|2|3)} = \frac37$.
question 3-28
What's the probability of exactly two heads, given that the first toss is a head?
That's just the probability that the second two tosses contain exactly one head, which is one-half.
question 3-29
Three cards from a deck. Given that at least two of them are hearts, what's the probability that all three are hearts?
The text gives $0.0859 \approx 1/11.6$, which doesn't make any sense. The probabilities of exactly zero, one, two, or three hearts are
| hearts | n(hands) | P |
|---|---|---|
| 0 | ${ 13 \choose 0 }{ 39 \choose 3 }$ | |
| 1 | ${ 13 \choose 1 }{ 39 \choose 2 }$ | |
| 2 | ${ 13 \choose 2 }{ 39 \choose 1 }$ | |
| 3 | ${ 13 \choose 3 }{ 39 \choose 0 }$ |
Given that at least two are hearts, the probability that the third is a heart should be ${11\choose 1}{39\choose 0}/{50\choose 1} = \frac{11}{50}$.
I can't come up with any logic for such a tiny probability like 1/11. I don't believe it.
question 3-32
Let $ S = \set{1\cdots6} $ be the sample space for a single die. Let $$ \begin{aligned} A &= \set{1,2,3,4} & P(A) &= 2/3 \\ B &= \set{2,3,4} & P(B) &= 1/2 \\ C &= \set{3,4,5} & P(C) &= 1/2 \end{aligned} $$ Which of these sets are independent?
The set $A$ is independent of $B$ if $P(A|B)=P(A)$; that is, if $B$ doesn't affect the probability of $A$. We're going to use the definition $ P(A|B) = \frac{P(A\and B)}{P(B)}, $ so we'll want the intersection probabilities, $$ \begin{aligned} A\and B &= \set{2,3,4} & P(A\and B) &= 1/2 \\ A\and C &= \set{3,4} & P(A\and C) &= 1/3 \\ B\and C &= \set{3,4} & P(B\and C) &= 1/3 \\ \end{aligned} $$
-
- Pair $A,B$: dependent
- $$ \begin{aligned} P(A|B) &= \frac{P(A\and B)}{P(B)} = \frac{1/2}{1/2} = 1 \\ &= P\left(\set{1,2,3,4}|\set{2,3,4}\right) = 1 \\ &= \frac{P(\set{2,3,4})}{P(\set{2,3,4})} = 1 && \neq P(A) \\ \\ P(B|A) &= P\left(\set{2,3,4}|\set{1,2,3,4}\right) = \frac34 \\ &= \frac{P(A\and B)}{P(A)} = \frac{1/2}{2/3} = \frac34 && \neq P(B) \end{aligned} $$
-
- Pair $A,C$: dependent
- $$ \begin{aligned} P(A|C) &= P(\set{1,2,3,4}|\set{3,4,5}) = \text{(not obvious)} \\ &= \frac{P(A\and C)}{P(C)} = \frac{1/3}{1/2} = \frac23 && \neq P(C) \\ &= \frac{P(\set{3,4})}{P(\set{3,4,5})} \\ \\ P(C|A) &= P(\set{3,4,5} | \set{1,2,3,4}) = \text{(not obvious)} \\ &= \frac{P(A\and C)}{P(A)} = \frac{1/3}{2/3} = \frac12 && \neq P(A) \end{aligned} $$
-
- Pair $B, C$: independent
- $$ \begin{aligned} P(B|C) &= P(\set{2,3,4}|\set{3,4,5}) \\ &= \frac{P(\set{3,4})}{P(\set{3,4,5})} = \frac23 && = P(B) \\ \\ P(C|B) &= P(\set{3,4,5}|\set{2,3,4}) \\ &= \frac{P(\set{3,4})}{P(\set{2,3,4})} = \frac23 && = P(C) \end{aligned} $$
question 3-39
Two jars contain coins.
jar 1 jar 2 pennies 4 6 nickels 5 4 dimes 6 2 A jar is chose at random and a coin from the jar. It's a nickel. What's the probability it came from Jar 2?
We need to know the probability that we get a nickel. It's $$ \begin{aligned} P(N) &= P(N|1)⋅ P(1) + P(N|2)⋅ P(2) \\ &= \frac{5}{15}⋅\frac12 + \frac{4}{12} ⋅ \frac12 = \frac{3}{10}. \end{aligned} $$ Now we use the relationship between conditional probabilities and the probabilities of an intersection, $$ P(A\and B) = P(A|B)⋅ P(B)= P(B|A)⋅ P(A) $$ to reverse: $$ \begin{aligned} P(2|N) &= P(N|2) \frac{P(2)}{P(N)} \\ &= \frac{4}{12} ⋅ \frac{1/2}{3/10} = \frac59. \end{aligned} $$
This one tripped me up because I initially put $P(N) = \frac{8}{27}$, as if all of the coins were equally likely.
question 3-64
Here's a tricky one, because the denominator is missing.
At the beginning of a health study, we have 20% heavy smokers, 30% light smokers, 50% non-smokers. Light smokers were twice as likely to die as non-smokers, and half as likely to die as smokers.
A random participant dies. What's the probability they were a heavy smoker?
The conditional probabilities that a participant dies are in the ratio 1:2:4, with an unknown denominator. I feel like I should be able to say
$$ \begin{aligned} P(\text{died} | \text{heavy}) &= \frac{4}{1+2+4+\text{survived}} \\ P(\text{died}) &= \frac{1+2+4}{1+2+4+\text{survived}} \\ \frac{P(\text{died}|\text{heavy})}{P(\text{died})} &= \frac{4}{7} \end{aligned} $$
Then Bayes' rule should give $$ \begin{aligned} P(\text{heavy}|\text{died}) &= \frac{P(\text{died}|\text{heavy})}{P(\text{died})} ⋅ P(\text{heavy}) \\ &= \frac{4}{7} \times 20\% = 11\%. \end{aligned} $$ But that logic doesn't give me probabilities that add to unity: $$ \begin{aligned} P(\text{heavy}|\text{died}) + P(\text{light}|\text{died}) + P(\text{non}|\text{died}) &= \frac17\times \left( 4⋅20\% + 2⋅30\% + 1⋅50\% \right) = 27\% \end{aligned} $$
If I normalize by this expression, I get the value from the answer key: $$ \begin{aligned} \frac{\frac{4}{7} \times 20\%}{\frac17\times \left( 4⋅20\% + 2⋅30\% + 1⋅50\% \right)} &= 42.1\% \end{aligned} $$
because, silly man, I should have done $$ \begin{aligned} P(\text{died} | \text{heavy}) &= \frac{4}{z} \\ P(\text{died}) &= \frac{1⋅20\%+2⋅30\%+4⋅50\%}{z} \\ \end{aligned} $$
Undone problems
I did a boatload of these problems, because I tend to get mixed up about them. In particular I did the first forty-six, and from sixty-two to the end. If I come back to this chapter later, that's where I'll pick up.