3.3. Lack of reliable feedback

Even if we don’t start off making complex social judgments in a reliable

fashion, we can at least hope to improve our judgments by receiving and

acting on accurate feedback. If we can determine that a depressingly large

number of our past judgments were mistaken (or even that a well-defined

class of past predictions was mistaken), perhaps we can modify our reasoning

strategies and so judge more accurately. (The fact that a person

might have made such modifications might lead him to discount the

pessimism we seem to be insisting upon here.) Unfortunately, there are a

number of quite natural phenomena that keep us from getting accurate

feedback on our past judgments and behaviors.

For many irrevocable decisions we make, the feedback we receive on

our judgments is almost inevitably incomplete. Consider the grizzled

philosopher who has played a major role in hiring a number of junior

colleagues and who takes the interviews very seriously. Given the nature of

the job market in philosophy, it’s quite likely that his junior colleagues are,

by and large, a pretty impressive lot. Given this feedback, he is likely to

think quite highly of his ability to identify in interviews good young

philosophers. The problem here is that the grizzled philosopher doesn’t

know whether his predictions would have turned out better or worse

without the interviews. (And even if he did, it’s unlikely he would have a

large enough sample size to draw a reasonable conclusion.) Simply put,

most gatekeepers don’t have control groups to test the effectiveness of

their reasoning strategies. After all, the set of junior colleagues who would

have been hired without interviews (the control group) might have been

even more terrific than his actual set of junior colleagues. The problem is

The Amazing Success of Statistical Prediction Rules 41

not just that most gatekeepers don’t have control groups—that is often a

practical inevitability. The problem is that they don’t recognize that this is

a serious problem. Most gatekeepers should probably have much more

diffidence concerning their powers of prediction—especially in a job market

in which most job seekers are something more than competent.

Another problem is that the feedback we get, especially when it comes

to social matters, is likely to be highly unrepresentative. Consider the

finding that 94% of university professors believe they’re better-thanaverage

at their jobs (Gilovich 1991, 77). One reason for this may be that we

typically get personal feedback from students who think we were terrific

teachers (or at least who say we were terrific teachers). Seldom will students

go out of their way to make contact with professors they thought were

really mediocre (if for no other reason than, where would they begin?).

The problem of unrepresentative feedback can be made vivid with an

example that is likely familiar to everybody. Think about someone who

employs mildly (or outright) annoying interpersonal strategies, for example,

dominating conversations or name dropping. How likely are you to

tell this person that these behaviors are annoying? Some blunt folk might

always do so. But most of us, probably as a result of some combination of

politeness, pusillanimity, and prudence, let it slide. Of course, we recognize

that this behavior is annoying (or worse), and we might judge the person to

be annoying (or worse). But given the feedback he has received, he might

well go forth into the world confident that he has once again been socially

deft, charming, and deeply impressive. (We are inclined to suggest you

perform a public service. Supply accurate feedback. Call a bore a bore, a

jerk a jerk, a blowhard a blowhard. Just don’t do it to us.)

Even when the feedback we get is representative and shows that our

predictions are mistaken, we will often interpret such feedback in a way

that supports our preconceptions. For example, Gilovich (1983) asked people

who gambled on football games to tape-record their thoughts about

the outcomes of their bets. One might expect the gamblers to remember

their wins and repress their losses. In fact, just the opposite occurred:

[T]hey spent more time discussing their losses than their wins. Furthermore,

the kind of comments made about wins and losses were quite different. The

bettors tended to make ‘‘undoing’’ comments about their losses—comments

to the effect that the outcome would have been different if not for some

anomalous or ‘‘fluke’’ element. . . . In contrast, they tended to make ‘‘bolstering’’

comments about their wins—comments indicating that the outcome

either should have been as it was, or should have been even more

extreme in the same direction. . . . By carefully scrutinizing and explaining away their losses, while accepting their successes at face value, gamblers do

indeed rewrite their personal histories of success and failure. Losses are often

counted, not as losses, but as ‘‘near wins.’’ (Gilovich 1991, 55)

One interesting feature of this common interpretative strategy is that the

subject cannot be accused of ignoring negative evidence. In fact, the subject

is attending more to the negative evidence than to the positive evidence.

It’s just that he interprets the positive evidence as positive, and the

negative evidence as bad luck.

Even if we don’t start off making complex social judgments in a reliable

fashion, we can at least hope to improve our judgments by receiving and

acting on accurate feedback. If we can determine that a depressingly large

number of our past judgments were mistaken (or even that a well-defined

class of past predictions was mistaken), perhaps we can modify our reasoning

strategies and so judge more accurately. (The fact that a person

might have made such modifications might lead him to discount the

pessimism we seem to be insisting upon here.) Unfortunately, there are a

number of quite natural phenomena that keep us from getting accurate

feedback on our past judgments and behaviors.

For many irrevocable decisions we make, the feedback we receive on

our judgments is almost inevitably incomplete. Consider the grizzled

philosopher who has played a major role in hiring a number of junior

colleagues and who takes the interviews very seriously. Given the nature of

the job market in philosophy, it’s quite likely that his junior colleagues are,

by and large, a pretty impressive lot. Given this feedback, he is likely to

think quite highly of his ability to identify in interviews good young

philosophers. The problem here is that the grizzled philosopher doesn’t

know whether his predictions would have turned out better or worse

without the interviews. (And even if he did, it’s unlikely he would have a

large enough sample size to draw a reasonable conclusion.) Simply put,

most gatekeepers don’t have control groups to test the effectiveness of

their reasoning strategies. After all, the set of junior colleagues who would

have been hired without interviews (the control group) might have been

even more terrific than his actual set of junior colleagues. The problem is

The Amazing Success of Statistical Prediction Rules 41

not just that most gatekeepers don’t have control groups—that is often a

practical inevitability. The problem is that they don’t recognize that this is

a serious problem. Most gatekeepers should probably have much more

diffidence concerning their powers of prediction—especially in a job market

in which most job seekers are something more than competent.

Another problem is that the feedback we get, especially when it comes

to social matters, is likely to be highly unrepresentative. Consider the

finding that 94% of university professors believe they’re better-thanaverage

at their jobs (Gilovich 1991, 77). One reason for this may be that we

typically get personal feedback from students who think we were terrific

teachers (or at least who say we were terrific teachers). Seldom will students

go out of their way to make contact with professors they thought were

really mediocre (if for no other reason than, where would they begin?).

The problem of unrepresentative feedback can be made vivid with an

example that is likely familiar to everybody. Think about someone who

employs mildly (or outright) annoying interpersonal strategies, for example,

dominating conversations or name dropping. How likely are you to

tell this person that these behaviors are annoying? Some blunt folk might

always do so. But most of us, probably as a result of some combination of

politeness, pusillanimity, and prudence, let it slide. Of course, we recognize

that this behavior is annoying (or worse), and we might judge the person to

be annoying (or worse). But given the feedback he has received, he might

well go forth into the world confident that he has once again been socially

deft, charming, and deeply impressive. (We are inclined to suggest you

perform a public service. Supply accurate feedback. Call a bore a bore, a

jerk a jerk, a blowhard a blowhard. Just don’t do it to us.)

Even when the feedback we get is representative and shows that our

predictions are mistaken, we will often interpret such feedback in a way

that supports our preconceptions. For example, Gilovich (1983) asked people

who gambled on football games to tape-record their thoughts about

the outcomes of their bets. One might expect the gamblers to remember

their wins and repress their losses. In fact, just the opposite occurred:

[T]hey spent more time discussing their losses than their wins. Furthermore,

the kind of comments made about wins and losses were quite different. The

bettors tended to make ‘‘undoing’’ comments about their losses—comments

to the effect that the outcome would have been different if not for some

anomalous or ‘‘fluke’’ element. . . . In contrast, they tended to make ‘‘bolstering’’

comments about their wins—comments indicating that the outcome

either should have been as it was, or should have been even more

extreme in the same direction. . . . By carefully scrutinizing and explaining away their losses, while accepting their successes at face value, gamblers do

indeed rewrite their personal histories of success and failure. Losses are often

counted, not as losses, but as ‘‘near wins.’’ (Gilovich 1991, 55)

One interesting feature of this common interpretative strategy is that the

subject cannot be accused of ignoring negative evidence. In fact, the subject

is attending more to the negative evidence than to the positive evidence.

It’s just that he interprets the positive evidence as positive, and the

negative evidence as bad luck.