2 The Amazing Success of Statistical Prediction Rules

Judgment problems great and small are an essential part of everyday life.

What menu items will I most enjoy eating? Is this book worth reading?

Is the boss in a good mood? Will the bungee cord snap? These and other

common judgment problems share a similar structure: On the basis of

certain cues, we make judgments about some target property. I doubt the

integrity of the bungee cord (target property) on the basis of the fact that

it looks frayed and the assistants look disheveled and hungover (cues).

How we make and how we ought to make such evidence-based judgments

are interesting issues in their own right. But they are particularly pressing

because such predictions often play a central role in decisions and actions.

Because I don’t trust the cord, I don’t bungee jump off the bridge.

Making accurate judgments is important for our health and happiness,

but also for the just and effective operation of many of our social

institutions. Judgments about whether someone will become violent can

determine whether that person loses their freedom by being involuntarily

committed to a psychiatric institution. Predictions about whether a prisoner

if set free will commit violence and mayhem can determine whether

he is or is not paroled. Judgments about a student’s academic abilities play

a role in determining the quality of medical school or law school she goes

to, or even whether she gets to study law or medicine at all. Judgments

about a person’s future financial situation can determine whether they

receive loans to make large purchases; such judgments can also determine

whether they receive the most attractive loans available. And most

everyone who has ever held a job has had others pass judgments about

their trustworthiness, intelligence, punctuality, and industriousness.

It is hard to overestimate the practical significance of these sorts of

social judgments. Using reasoning strategies that lead to unreliable judgments

about such matters can have devastating consequences. Unnecessarily

unreliable judgments can lead to decisions that waste untold resources, that

unjustly deprive innocent people of their freedom, or that lead to preventable

increases in rape, assault, and murder. There is a difference between

cancer and horseshoes, between prison and a good shave. For many reasoning

problems, ‘‘close enough’’ isn’t good enough. Only the best reasoning

strategies available to us will do. Ameliorative Psychology is designed

to identify such strategies, and the primary tasks of a useful epistemology

are to articulate what makes a reasoning strategy a good one and to carry

that message abroad so that improvements can be implemented. This

chapter is the prologue to that epistemological message.

Who could possibly deny that those charged with making high-stakes

decisions should reason especially carefully about them? Consider, for example,

predictions about violent recidivism made by parole boards. Who

could deny that members of parole boards should scrupulously gather as

much relevant evidence as they can, carefully weigh the different lines of

evidence, and on this basis come to a judgment that is best supported by

the entirety of the evidence? Actually, we deny this. We contend that

it would often be much better if experts, when making high-stakes judgments,

ignored most of the evidence, did not try to weigh that evidence,

and didn’t try to make a judgment based on their long experience. Sometimes,

it would be better for the experts to hand their caseload over to a

simple formula that a smart 8-year-old could solve and then submit to the

child’s will. This is what Ameliorative Psychology counsels. (Of course,

discovering such a formula takes some expertise.)

For the past half century or so, psychologists and statisticians have

shown that people who have great experience and training at making

certain sorts of prediction are often less reliable than (often very simple)

Statistical Prediction Rules (SPRs). This is very good news, especially for

those of us who like to do hard work without having to work hard. Of

course, the philosophical literature is full of fantastic examples in which

some simple reasoning strategy that no reasonable person would accept

turns out to be perfectly reliable (e.g., ‘‘believe all Swami Beauregard’s

predictions’’). But we are not engaged here in Freak Show Philosophy.

Many SPRs are robustly successful in a wide range of real-life reasoning

problems—including some very high-stakes ones. Further, the success of

The Amazing Success of Statistical Prediction Rules 25

some SPRs seems utterly miraculous. (In fact, when we introduced one of

the more shocking SPR results described below to a well-known philosopher

of psychology who is generally sympathetic to our view, he simply

didn’t believe it.) But there are general reasons why certain kinds of SPRs

are successful. We turn now to describing their success. Later, we’ll try to

explain it.

Judgment problems great and small are an essential part of everyday life.

What menu items will I most enjoy eating? Is this book worth reading?

Is the boss in a good mood? Will the bungee cord snap? These and other

common judgment problems share a similar structure: On the basis of

certain cues, we make judgments about some target property. I doubt the

integrity of the bungee cord (target property) on the basis of the fact that

it looks frayed and the assistants look disheveled and hungover (cues).

How we make and how we ought to make such evidence-based judgments

are interesting issues in their own right. But they are particularly pressing

because such predictions often play a central role in decisions and actions.

Because I don’t trust the cord, I don’t bungee jump off the bridge.

Making accurate judgments is important for our health and happiness,

but also for the just and effective operation of many of our social

institutions. Judgments about whether someone will become violent can

determine whether that person loses their freedom by being involuntarily

committed to a psychiatric institution. Predictions about whether a prisoner

if set free will commit violence and mayhem can determine whether

he is or is not paroled. Judgments about a student’s academic abilities play

a role in determining the quality of medical school or law school she goes

to, or even whether she gets to study law or medicine at all. Judgments

about a person’s future financial situation can determine whether they

receive loans to make large purchases; such judgments can also determine

whether they receive the most attractive loans available. And most

everyone who has ever held a job has had others pass judgments about

their trustworthiness, intelligence, punctuality, and industriousness.

It is hard to overestimate the practical significance of these sorts of

social judgments. Using reasoning strategies that lead to unreliable judgments

about such matters can have devastating consequences. Unnecessarily

unreliable judgments can lead to decisions that waste untold resources, that

unjustly deprive innocent people of their freedom, or that lead to preventable

increases in rape, assault, and murder. There is a difference between

cancer and horseshoes, between prison and a good shave. For many reasoning

problems, ‘‘close enough’’ isn’t good enough. Only the best reasoning

strategies available to us will do. Ameliorative Psychology is designed

to identify such strategies, and the primary tasks of a useful epistemology

are to articulate what makes a reasoning strategy a good one and to carry

that message abroad so that improvements can be implemented. This

chapter is the prologue to that epistemological message.

Who could possibly deny that those charged with making high-stakes

decisions should reason especially carefully about them? Consider, for example,

predictions about violent recidivism made by parole boards. Who

could deny that members of parole boards should scrupulously gather as

much relevant evidence as they can, carefully weigh the different lines of

evidence, and on this basis come to a judgment that is best supported by

the entirety of the evidence? Actually, we deny this. We contend that

it would often be much better if experts, when making high-stakes judgments,

ignored most of the evidence, did not try to weigh that evidence,

and didn’t try to make a judgment based on their long experience. Sometimes,

it would be better for the experts to hand their caseload over to a

simple formula that a smart 8-year-old could solve and then submit to the

child’s will. This is what Ameliorative Psychology counsels. (Of course,

discovering such a formula takes some expertise.)

For the past half century or so, psychologists and statisticians have

shown that people who have great experience and training at making

certain sorts of prediction are often less reliable than (often very simple)

Statistical Prediction Rules (SPRs). This is very good news, especially for

those of us who like to do hard work without having to work hard. Of

course, the philosophical literature is full of fantastic examples in which

some simple reasoning strategy that no reasonable person would accept

turns out to be perfectly reliable (e.g., ‘‘believe all Swami Beauregard’s

predictions’’). But we are not engaged here in Freak Show Philosophy.

Many SPRs are robustly successful in a wide range of real-life reasoning

problems—including some very high-stakes ones. Further, the success of

The Amazing Success of Statistical Prediction Rules 25

some SPRs seems utterly miraculous. (In fact, when we introduced one of

the more shocking SPR results described below to a well-known philosopher

of psychology who is generally sympathetic to our view, he simply

didn’t believe it.) But there are general reasons why certain kinds of SPRs

are successful. We turn now to describing their success. Later, we’ll try to

explain it.