Thursday, April 02, 2015

Screenings and Type I and Type II errors

by Sherry F. Colb

In my column for this week, I talk about some commonalities between breast cancer (and other medical) screenings and dog sniffs.  In both cases, I explain, the screening might seem relatively innocuous and uninvasive, a good reason to go ahead and have it done.  But as it turns out, because of type I errors (false positives), the odds that the first screening step will be the last diminish, and the decision to screen therefore can unwittingly become a decision to take the many invasive steps that follow a positive result.  In the column, I suggested that this seemingly inexorable momentum from screening to more invasive searches may represent a good reason to avoid many screenings in the first place, in the absence of some independent reason to believe that there is something to find (i.e., some symptom of the disease for which one is testing or some basis for suspecting that a person to be "sniffed" by a dog is in possession of narcotics).

In this post, I want to be clear about what I am not saying.  I am not suggesting that screenings (of the medical or forensic variety) are best avoided at all times.  Whether to "screen" in the absence of evidence of a problem is a question best answered by reference to factors including how accurate the screening device is (i.e., how likely we are to get a type I error of the sort that worries me in the column) and how helpful it would be to learn now (and how costly it might be to delay learning until a later point) about the condition for which one would be screening.  To illustrate without unnecessarily courting controversy, I shall provide two completely hypothetical examples.

Assume that we have a medical screening test (a blood test) that will indicate a heightened possibility that one currently has cancer of the esophagus.  Assume as well that absent this test, esophageal cancer will generally be untreatable by the time the patient actually develops symptoms that might send him or her to the doctor with a complaint.  Assume further that the screening test is about 80% accurate (in the sense that just one fifth of the patients who test positive for esophageal cancer turn out to be cancer free).  Assume finally that the test to confirm the cancer involves a biopsy of the esophagus that can occur under a local anesthetic and from which the patient recovers within a day in almost all cases.  Given all of these assumptions, it seems to me that the screening is worthwhile, notwithstanding the fact that one could avoid type I errors by skipping it.

Assume now that we have a physical device that can, from outside a building, indicate a heightened possibility of the presence of nuclear weapons inside the building.  Assume as well that failure to identify the presence of nuclear weapons inside the building could result in the obvious.  Assume further that the device is approximately 80% accurate (in the same way as the esophageal cancer screening device is about 80% accurate) and that, as in the medical test, the confirmation process is not enormously invasive.  Assume that confirmation involves entering the building with a more precisely calibrated detection device and waving the device around in some part of each room to get an accurate measure.  In this case too, it appears that the screening is worthwhile.

Yet even in these relatively easy cases, we can see that there are non-inconsiderable costs.  If everyone acts on the assumption that the screening is a good idea, then a large number of healthy people will experience unnecessary biopsies of their esophagus, along with the accompanying anxiety.  And a correspondingly large number of innocent people will have police walking through their homes waving nuclear-weapon-detection devices in each room of their homes (and therefore also seeing many private things inside those homes).  We may nonetheless make the choice (professionally, to recommend a medical screen or judicially, to authorize a forensic screen), because on balance, it provides the greatest good for the greatest number of individuals.  This is a utilitarian calculus, though it may be less offensive to deontologists than those in which we know in advance which individuals will benefit and which will suffer needlessly (e.g., if we were to experiment upon A directly in order to help B and C). 

As a result, when we are speaking of individual rights (to privacy or to medical integrity), it is appropriate that we feel some residual discomfort about pushing screenings upon a reluctant audience.  This does not mean that we must refrain from doing so, but the residual discomfort remains, is worthy of consideration, and is largely traceable to the type I errors discussed in my column


Joe said...

There is not a perfect overlap here particularly since people choose to get screenings of this sort. People don't choose to have dogs (and not cute and cuddly ones) sniff their cars and goods.

The false positives (resulting in searches that otherwise would not bee made) of a dog sniff is somewhat comparable to a false positive of a health exam (resulting in certain harms as well) do match in some respects.

Both concerns were cited by the dissent in Illinois v. Caballes & might factor in when the Supreme Court decides a dog sniff case this term.

Greg said...

Ahhh, lies, damn lies, and statistics...

80% accurate is an incredibly bad medical test.

16,980 people get esophageal cancer in the US each year ( Population of the US: 318 million (google).

So, if the entire population tested using your proposed test:
13580 would get true positive tests.
3396 would get false negative tests.
254 million would get true negative tests.
63.6 million would get false positive tests.

99.96% of positive tests would be false.

Let's use a more reasonable test that is 99% accurate.

3.18 million false positive.
16810 true positive.

Even with a 99% accurate test, 99.5% of positive tests will be false.

This is without considering possible side effects of the procedure, which could ultimately prove more fatal than the total mortality of the disease itself.

If we were to use your nuclear weapon example, the numbers would be similar.

The peak stockpile of nuclear weapons in the US is roughly twice the number of cases of esophogeal cancer in the US each year. The current stockpile is about of the number of cases.

If we assume EVERY SINGLE WEAPON that the US had at their PEAK (1967) (wikipedia) was moved into a home in the US, and we assume an average family size of 2.5 ( we get a 97.5% false positive rate of finding nuclear weapons within a household.

Taking even a 99% accurate test for either of the cases you describe would be COMPLETELY inappropriate. In both cases the false positive rate would be far too high to justify the test if other factors did not indicate a need to take the test.

It's important that people understand these numbers, because without this understanding they will advocate procedures that are far too invasive, for many of the reasons you mention.

David Ricardo said...

Greg's comment is extremely important and his example is one that many of us use repeatedly in teaching quantitative methods and statistics. It is a great illustration of why medical tests that seem so good are in many cases very, very bad.

Sanjay Sharma said...

Packers And Movers Kolkata Salt Lake
Packers And Movers Kolkata Garia
Packers And Movers Kolkata Dumdum
Packers And Movers Howrah
Movers And Packers Howrah

descargar said...

Great post,Thanks for providing us this great knowledge,Keep it up.
A good blog.
play game juegos de terror online | play game juegos de terror | juego de terror

Guo Guo said...

guowenhao20150430calvin klein outlet
swarovski crystal
links of london
michael kors outlet
burberry outlet online
lacoste outlet
ray ban sunglasses
nba jerseys
ray ban wayfarer
michael kors handbags
mac makeup
kobe bryants shoes
michael kors outlet
tods outlet
toms shoes
ray ban uk
ray ban sunglasses
coach factory outlet
celine outlet
adidas wings
marc jacobs outlet
abercrombie and fitch
timberland boots
true religion outlet
michael kors factory outlet
coach outlet store online
louboutin shoes
beats solo
nike roshe run
ray ban sunglasses
air force one shoes
ray ban sale
herve leger dresses
michael kors outlet
cheap oakley sunglasses
salvatore ferragamo
michael kors handbags
cheap toms
mcm bags