Thoughts on the classic Bayes' example

Also a suggestion on the contextual side is that maybe there is a way to make it fun, respect brand consistency and so on, but to collaborate on it and not only develop contextual examples but also have some different visuals or characters - for example when I was working on one laptop per child at MIT I tried to think of some open source characters for helping to learn math and science and I’m glad to help dream up or dig back and find artists. Here is a more recent video and explainers may be a way to prototype but I would need some input - this is on distributed AI: https://youtu.be/7LfS146fJ5M | And this is a presentation that includes some of the 3D characters and a free children’s book where I tried having the mantis teach some math. https://docs.google.com/presentation/d/1N56vyvJ84-AW57rrhwCq32nVbu2KRdB_K88AwOO1rkw | it is more ideas than project but my suggestion is a mantis helping to teach math for machine learning with stats and maybe also have the sunflower be a metaphor for data and so on

Todd Kelsey

A bit late to this video (I have just subscribed to Patreon and discovered this hidden video), but as a doctor / student biostatistician I thought I should provide a bit of clarification to how this disease example doesn't always match up with real life: This example works really well if a screening test is to be done on a random, _asymptomatic_ person on the general population. Then the prior probability would indeed be in the range of 1 in 1000, if not lower, leading to the slightly counter-intuitive posterior probability. If someone who has had symptoms present to a doctor and has a test done, then the calculation no longer holds as the prior probability is no longer 1 in 1000 - a symptomatic person has a much higher prior probability that they indeed have the disease. Therefore instead of 1 in 1000, we might be looking at a prior probability of say 30%, which makes the posterior probability a lot higher have they tested positive.

Chang Yang Yew

Interesting points. What also makes the medical question hard is that saying the failure rate is 1/100 somehow feels much less real, and potentially arbitrary, than guessing a random number from 1 to 100. As to publishing these, I do have some specific thoughts on how I'd like to change the Bayes rule one before putting it out, changing the examples a bit to something more discrete before jumping into thinking of the areas in the subdivided square. I do really want to publish that one, though, so as soon as I'm looking for a break from differential equation stuff, wrapping this one up and putting it out is pretty high on the priority queue.

3blue1brown

I wonder if some more complex culturally specific factors might be stronger causes of the biases here. For instance, many people are more likely to apply their skepticism to a friend who correctly guesses a number than a medical test, because 1) the friend is presumably our peer while the medical test makers are likely to be considered experts backed by some kind of institutional authority in our society and 2) most people with a basic math education will feel empowered to reason about the likelihood of someone guessing 1 number out of 100 (as other comments have mentioned this is a common game for people to have experienced) while far fewer people will feel empowered to reason about the math behind a task like medical diagnosis which they have been lead to believe is a complex subject requiring esoteric knowledge. Regadless I think your point about empathy is also very interesting and I'm quite glad to see discussion of rigruous math pedagogy and the social contexts that shape it. Your teaching always blows me away! And for what it's worth I wish you would feel more confident about these probability videos; they have already helped me enourmously in both my understanding and confidence. Your videos have been key to helping me feel that I can bridge from qualitative study of social science to mastering all the math I need now that I'm studying computer science. Even if all your videos weren't 100% perfect, they'd still be so far beyond all the rest of the math education I've seen that I'm sure they'd be changing countless lives beyond just my own!

Ian Magnusson

I was thinking the same about the telepathy example, if the person claims he is telepathic (which in the video is explicitly mentioned) this alone will increase your probability estimate: how many of your non telepathic friends have come to you claiming they are telepathic? I know what is meant, but to be more correct it might be better to say the friend wants to test if he is telepathic, just for the sake of curiosity, but has no particular prior reason to believe so.

Fela Winkelmolen

This is completely right. In fact, I think I'll go back and add a note about that fact and how doctors actually make decisions with respect to tests like this, since it's a point strongly worth making.

3blue1brown

So I think this video does a better job of explaining the sick person example than other channels have (I'm thinking of Veritasium's and Health Care Triage's videos here which were still pretty good). However, it seems that THIS EXAMPLE is just a bad example in general (not your explanation). The example is using a prior of (1/1000) but that seems wrong. That would seem to imply that you would expect people who are totally healthy to come into a doctors office and then get tested for a disease that they have no symptoms for and then get a false positive. That isn't what happens IRL though. It may be true that on average 1/1000 people in a society have a disease, but it seems that your prior should be taking in information on the number of people who on average come into a doctors office and are expressing symptoms that would cause a doctor to even administer the test with the 99% accuracy. Am I missing/misunderstanding something here? I liked the other Bayes videos you did on cards and sucking at music a lot as they don't have a point like this.

Matthew Feickert

Good point! I hadn't really thought about the frequentism-vs-Bayesian angle here, but it's very interesting that people seem more comfortable with the former than the latter in this case. I wonder if there are situations where Bayesianism is more intuitive than frequentism?

Kevin

You bring up some great points. I'm guessing you already know this, but it's an interesting point to bring up nevertheless: With the conjunction fallacy, when you reframe the question away from "What percentage of people..." to "Out of one hundred people, how many...", the fallacy seems to go away. So in that sense, representation too cannot be solely to blame, it seems to be mixed with a certain lack of concreteness in how people think of percentages.

3blue1brown

I loved the video, but I think you're giving human intuition too much credit. While you've done a great job of explaining the base rate fallacy, there is a more fundamental misunderstanding here that ought to be discussed explicitly: the representativeness heuristic. This is the tendency of people to judge probabilities by the degree to which an outcome seems "representative" of the parent population. This theory explains not only the base rate fallacy, but a number of other fallacies such as the conjunction fallacy (you can get test subjects to tell you that P(A and B) > P(A) by providing evidence for B). No matter how messed up your priors are, no application of Bayes' theorem will ever give you that result (because it's false), so whatever the "gut" is doing, it's not applying Bayes' theorem.

Kevin

+1 for discussing a logarithmic perspective on evidence and Bayes' theorem.

Jake Palmer

I think delving into topics like this only makes sense after an in-depth look into the topic of bias, after the viewers are already taught to distrust their intuition in statistical cases. You might be quite right that a major source of bias is misjudging the priors (see the conviction of Sally Clark in 1999 for a real example), to which I would add missing the impact of other evidence (so in a sense still affecting the priors). The logarithmic form of the Bayes rule is quite handy, where each piece of evidence is given strength measured in bits. For example, in the case of the telepath, the fact that he is up for the challenge is worth something in addition to the 6.6 bits provided by the actual successful guess. Or if a forensic DNA test has a 1 in a million failure rate (clerical errors occur more often than genuine matches, I presume), those 20 bits just about negate the probability of someone being a murderer (a quick search gives one estimate of 15 per million or -16 bits), so other factors such as good old alibis play a major role (thus combing everyone for the DNA does not result in a statistically valid evidence, while testing someone who’s already under suspicion does).

Roman Odaisky

Bear in mind that these are not my actual opinions, just the voice of my intellectually-unaided "intuition" as I read the problem. Warning: my intuition royally sucks at math. ;) (Monkey-brain, re-engage) > The 1/100 for the test is the probability that it's wrong. So.. 99/100 that I have it. :Q__ > You haven't factored in the probability that you don't have the disease. Is that one 1/100 now? > Suppose it's literally impossible to have a certain disease, but you tested positive for it. I beat the odds! Woohoo, I'm gonna die! xD > The test has a 1/100 false positive rate. What's the chance you have the disease? 1/100? :Q__

Jesse Thompson

The 1/100 for the test is the probability that it's wrong. You haven't factored in the probability that you don't have the disease. Maybe considering an extreme case will help: Suppose it's literally impossible to have a certain disease, but you tested positive for it. The test has a 1/100 false positive rate. What's the chance you have the disease?

Ed Kellett

There has been some actual research on why we sometimes feel like Bayesian reasoning is counterintuitive! The theory is that we consider Bayesian reasoning when we can construct a causal connection between the prior and posterior, and we disregard it when the information about the base rate feels like a statistical fact that wouldn't apply to an individual case. This is much better explained in Kahnemans 'Thinking, fast and slow', but here is a paper about it: <a href="http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA056667" rel="nofollow noopener" target="_blank">http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA056667</a> So in your two examples, the bluffing friend has an immediate causal impact on the statistical reasoning, while the base rate for the disease seems distant and not causally connected to your specific case.

Niklas Becker

Great catch! Thank you.

3blue1brown

I just friggin' love the way you think 😊

Karin Rodrigues

Loved this video

faisal

Hi Grant! Shouldn't P(+|Not disease) represent the width of the yellow rectangle and equal to 1% instead of 1 at 2:08 ?

Corvinus

Your videos are truly unprecedented and inspired man. I am a proud sponsor and math major who is very exited to see future videos!

Tyler Randall

"You initially give a 1/1000 chance to the possibility that you have a disease. Then you take a test with a 1/100 false positive rate — and it gives you a positive result. Now, what are the chances that you give to having the disease?" Uh.. monkey-brain engage. I initially don't think I have it, then I take a 1/100 test that says I have it so, uh, I estimate 1/100 that I have it. Am I do it right? x3 "(You choose an arbitrary number 1-100 and I predict your choice). Would you believe that I can legitimately read your mind, or that I just lucked out? And what chances would you place on it?" Uh.. there is a 1/100 chance that you would have predicted the number I chose. So since you did that, uh, there must be a 1/100 chance that you are really telepathic. "Notice, the probabilities here are all *identical* to the medical example." Yaey, my answers matched up so I did it right! :D

Jesse Thompson

Ah, good point, wouldn't want to unduly shake anyone's confidence in the real world medical system.

3blue1brown

Of course: the second test must not be the same as the first. Sneezing does not prove a cold, but add a runny nose and a fever, and you can be almost certain. The point I was trying to make is that you can "chain" Bayes rule to continuously update your beliefs. This will come up again in Bayes nets, but could have also been shown here.

Matthias Richter

As long as you are careful to acknowledge the implicit assumption you are making: that the test results are independent of any factors other than disease itself. The fact that false positives exist is usually a good indication that there are other factors that affect the result. Say for example that the result is skewed by high blood sugar. If your sugar level is very high, you will likely keep testing positive over and over despite not having the disease. The telepathy example hold more true, but consider you add the requirement that you must write the number on a piece of paper (to prove to the other that you are honest after they guess). Now there are outside factors to consider. Are they watching your hand as you write? Is there a camera or mirror anywhere? This is why you wouldn't actually believe a magician has powers just because they have been performing this same feat for years.

Trevor Bruns

Totally agree, it's a very important topic that's not mentioned in a lot of intro probability material.

3blue1brown

Amazing video Grant! The only suggestion I have is to point out that doctors usually only give tests like this to patients who have exhibited symptoms of the disease. So as a second example, you could make the prior 1/100, or something much larger than the general population proportion without symptoms. That, along with testing for the disease two or three times on the same patient instead of just once, should relieve any doubt people have with medical testing

Tyler Randall

I don't think that's a problem. They might remember it when that wrong intuition is presented to them and then it clicks, so it's kind of a preventive cure.

Fiaca

A missed opportunity to point out that a second positive test would put the belief that I'm sick at 90%, just as a second (third, fourth, ...) correct guess would dramatically increase the belief in the supernatural. Other than that: Great video, especially the point about our tendency to have different priors than what statistics tells us.

Matthias Richter

Look good! Everything seems understandable. If I may, I'd like to suggest another video topic that recently captured my interest. Maybe as a standalone or in a series on information theory, you could talk about the maximum entropy principle and exponential family distributions. MaxEnt seems like a really important concept for model selection, and it results in very nice properties.

Duncan Fairbanks

Well, it doesn't affect how you crunch the numbers. If you crunch the numbers you always get the right answer. In this case, though, the right answer isn't very obvious. I don't actually agree that it's because of empathy, but I think the reasoning is sound. As for what does cause it: would the telepathy example feel different if you presented it as a black box test for detecting a telepath with a 99% specificity?

Ed Kellett

This is really amazing--I really love your work! Just one thing--I'm really shocked that you say this is counter-intuitive! It's totally intuitive for me that you have a fairly low chance of actually having the disease given a positive result. Is it really not intuitive to most people? Not even budding young mathematicians? That seems scary... Your explanation why it might be unintuitive seems plausible, but why on earth would empathy for a sick person and the fear of being struck with a fatal disease affect how one crunches the numbers? Like, what? Why would it be any less intuitive cast with exactly the same numbers in a different context? I don't really get the thought processes behind that...

Amazing take on this example! I really loved it, especially the comparison to the telepathy example - you're right, this really *does* feel different, and Bayes' Rule has nothing to do with it. That said, my only worry is that this video is too much for people who already know the material. I mean, it's great for us, don't get me wrong - but I'm worried that a big chunk of the content will go over the heads of someone who doesn't know the example, if only because they never got the chance to have the "wrong" intuition here.

Edan Maor

This is SO GOOD

David Wych