2. Overcoming overconfidence

If error is the constant companion of inquiry, so is overconfidence. People

are systematically prone to giving more credence to their beliefs than they

deserve. The literature demonstrating overconfidence is very large. To cite

one representative example mentioned earlier (chapter 2, section 3.4),

Fischhoff, Solvic, and Lichtenstein (1977) asked subjects to specify the

most frequent cause of death in the U.S. and to estimate their confidence

that their choice was correct. It turns out that when subjects set the odds

of their answer’s correctness at 100:1, they were correct 73% of the time;

even when they were so sure as to set the odds between 10,000:1 and

1,000,000:1, they were correct only between 85% and 90% of the time. The

overconfidence effect is systematic (it is highly replicable and survives

changes in task and setting) and directional (the effect is in the direction of

overconfidence rather than underconfidence—except when it comes to

objectively easy problems [Lichtenstein and Fischhoff 1980].) Given these

findings, one might hope that expert training reduces or eliminates

overconfidence. This does not appear to be so. Physicists, economists, and

demographers have all been observed to suffer from this bias, even when

reasoning about the content of their special discipline (Henrion and

Fischoff 1986). In fact, it might be useful for readers of this book to know

that people who score better on SATs are also more prone to overconfidence

(Stanovich 1999, 120). Overconfidence is not the mere result of

individual differences in personality or of clinical delusions of grandeur. It

is the normal consequence of routine cognitive activity.

Overconfidence in highly significant reasoning problems can lead to

fatally inaccurate judgments—for example, that guilt has been proven beyond

a reasonable doubt. There have been 110 people who were convicted

of murder who have been found to be not guilty on the basis of DNA

evidence. Most of these errors have been found to be the result of mistaken

eyewitness testimony and the overconfidence of eyewitnesses and juries in

the reliability of such testimony (Wells, Olson, and Charman 2002).

Are there any reliable strategies for counteracting this overconfidence?

Arkes, Dawes, and Christensen (1986) found that accuracy incentives did

not improve accuracy of judgments (also see Arkes 1991). Lord et al.

(1984) found that simple motivational declarations do not reduce bias

either. So, for example, offering subjects $1 for each correct answer or

urging subjects to be unbiased do not reduce overconfidence. There are

some ways to reduce bias, but they involve introducing elements that are

unlike what people tend to face in daily life. For example, when subjects

are made to feel accountable for their judgments (by being told that their

answers will be discussed after the session), they assimilated evidence in a

less biased way. Further, overconfidence can be eliminated when subjects

are persistently exposed to a rigorous schedule of specially prepared feedback

(Arkes, Christensen, Lai, and Blumer 1987). These sorts of findings

are important for thinking about what sorts of procedures to implement

when deliberations occur in stable social settings (such as a jury or a policy

panel). These sorts of calibration exercises can prompt groups of people to

be less overconfident than each individual, reasoning alone, might have

been. Indeed, it may be that controlled environments, where debiasing

decision-making procedures can be implemented, offer our best hope of

overcoming overconfidence.

The most prominent of the individual, subjective debiasing methods

is the consider-the-opposite strategy. One of the groundbreaking studies on

debiasing argued that people ‘‘have a blind spot for opposite possibilities’’

when making social and policy judgments (Lord, Lepper, and Preston,

1984). When subjects were asked to generate pros and cons for a judgment

they had made, Koriat, Lichtenstein, and Fischhoff (1980) demonstrated

that overconfidence bias was reduced. Indeed, they found that it was the

generation of opposing reasons that did all of the bias-reducing work. The

most effective version of the consider-the-opposite strategy involves a simple

rule: ‘‘Stop to consider why your judgment might be wrong’’ (Plous

1993, 228). Here are some beliefs that we might hold with undue certainty.

If S believes that New York State is the largest state on the Eastern Seaboard,

S might consider that he might be wrong because New York gets a

lot more press than South Atlantic states. If S believes that Los Angeles

is west of Reno, he might ask himself whether he might not have a clear

sense of the orientation of the American landmass. If S believes that a

defendant is guilty beyond a reasonable doubt, he might consider that he

might be mistaken because of his confusion over the DNA evidence.

While we are somewhat hopeful about the ability of the consider-theopposite

strategy to reduce our overconfidence, there are a number of issues

that need to be resolved before we can embrace it wholeheartedly.One serious

worry is whether the consider-the-opposite strategy can be exported to a

setting in which it can be usefully implemented. This is not a criticism of the

ecological validity of the experiments; there is no question that if you could

get people in natural settings to perform the same experimental debiasing

task, overconfidence would be reduced. The question instead is whether, as

people go through their daily lives, they will have the discipline, motivation,

and concentration required to implement the consider-the-opposite strategy.

This kind of worry drops out of our view quite naturally. Strategic

reliabilismis a view that explicitly recognizes our psychological limitations. If

it is part of our psychological functioning that we tend to ignore or undervalue

corrective strategies, this is something we must take seriously.

But even if we can implement the consider-the-opposite strategy to

good effect, there is another serious worry: What is its range to be? After all, it would surely be a mistake for us to adopt the strategy for every belief

we entertain. Such a strategy would turn us into unhappy, neurotic nebbishes,

impossible to get along with. (One of us is reminded of the student

who was obviously having serious life troubles and explained that it was

because he had really taken Descartes’ Meditations to heart.) Recall our

discussion in chapter 3 of the Four Ways to better reasoning. We noted

that there could be occasions when one could become a better reasoner by

replacing a reliable but expensive strategy with another that was less reliable

but less expensive. Replacing a wide ranging consider-the-opposite

strategy with a consider-the-opposite strategy with a narrower range might

be an example of this. If a consider-the-opposite strategy is a good idea, it

is surely one whose range is to be limited to highly significant problems in

which short-term reliability is very important.

If error is the constant companion of inquiry, so is overconfidence. People

are systematically prone to giving more credence to their beliefs than they

deserve. The literature demonstrating overconfidence is very large. To cite

one representative example mentioned earlier (chapter 2, section 3.4),

Fischhoff, Solvic, and Lichtenstein (1977) asked subjects to specify the

most frequent cause of death in the U.S. and to estimate their confidence

that their choice was correct. It turns out that when subjects set the odds

of their answer’s correctness at 100:1, they were correct 73% of the time;

even when they were so sure as to set the odds between 10,000:1 and

1,000,000:1, they were correct only between 85% and 90% of the time. The

overconfidence effect is systematic (it is highly replicable and survives

changes in task and setting) and directional (the effect is in the direction of

overconfidence rather than underconfidence—except when it comes to

objectively easy problems [Lichtenstein and Fischhoff 1980].) Given these

findings, one might hope that expert training reduces or eliminates

overconfidence. This does not appear to be so. Physicists, economists, and

demographers have all been observed to suffer from this bias, even when

reasoning about the content of their special discipline (Henrion and

Fischoff 1986). In fact, it might be useful for readers of this book to know

that people who score better on SATs are also more prone to overconfidence

(Stanovich 1999, 120). Overconfidence is not the mere result of

individual differences in personality or of clinical delusions of grandeur. It

is the normal consequence of routine cognitive activity.

Overconfidence in highly significant reasoning problems can lead to

fatally inaccurate judgments—for example, that guilt has been proven beyond

a reasonable doubt. There have been 110 people who were convicted

of murder who have been found to be not guilty on the basis of DNA

evidence. Most of these errors have been found to be the result of mistaken

eyewitness testimony and the overconfidence of eyewitnesses and juries in

the reliability of such testimony (Wells, Olson, and Charman 2002).

Are there any reliable strategies for counteracting this overconfidence?

Arkes, Dawes, and Christensen (1986) found that accuracy incentives did

not improve accuracy of judgments (also see Arkes 1991). Lord et al.

(1984) found that simple motivational declarations do not reduce bias

either. So, for example, offering subjects $1 for each correct answer or

urging subjects to be unbiased do not reduce overconfidence. There are

some ways to reduce bias, but they involve introducing elements that are

unlike what people tend to face in daily life. For example, when subjects

are made to feel accountable for their judgments (by being told that their

answers will be discussed after the session), they assimilated evidence in a

less biased way. Further, overconfidence can be eliminated when subjects

are persistently exposed to a rigorous schedule of specially prepared feedback

(Arkes, Christensen, Lai, and Blumer 1987). These sorts of findings

are important for thinking about what sorts of procedures to implement

when deliberations occur in stable social settings (such as a jury or a policy

panel). These sorts of calibration exercises can prompt groups of people to

be less overconfident than each individual, reasoning alone, might have

been. Indeed, it may be that controlled environments, where debiasing

decision-making procedures can be implemented, offer our best hope of

overcoming overconfidence.

The most prominent of the individual, subjective debiasing methods

is the consider-the-opposite strategy. One of the groundbreaking studies on

debiasing argued that people ‘‘have a blind spot for opposite possibilities’’

when making social and policy judgments (Lord, Lepper, and Preston,

1984). When subjects were asked to generate pros and cons for a judgment

they had made, Koriat, Lichtenstein, and Fischhoff (1980) demonstrated

that overconfidence bias was reduced. Indeed, they found that it was the

generation of opposing reasons that did all of the bias-reducing work. The

most effective version of the consider-the-opposite strategy involves a simple

rule: ‘‘Stop to consider why your judgment might be wrong’’ (Plous

1993, 228). Here are some beliefs that we might hold with undue certainty.

If S believes that New York State is the largest state on the Eastern Seaboard,

S might consider that he might be wrong because New York gets a

lot more press than South Atlantic states. If S believes that Los Angeles

is west of Reno, he might ask himself whether he might not have a clear

sense of the orientation of the American landmass. If S believes that a

defendant is guilty beyond a reasonable doubt, he might consider that he

might be mistaken because of his confusion over the DNA evidence.

While we are somewhat hopeful about the ability of the consider-theopposite

strategy to reduce our overconfidence, there are a number of issues

that need to be resolved before we can embrace it wholeheartedly.One serious

worry is whether the consider-the-opposite strategy can be exported to a

setting in which it can be usefully implemented. This is not a criticism of the

ecological validity of the experiments; there is no question that if you could

get people in natural settings to perform the same experimental debiasing

task, overconfidence would be reduced. The question instead is whether, as

people go through their daily lives, they will have the discipline, motivation,

and concentration required to implement the consider-the-opposite strategy.

This kind of worry drops out of our view quite naturally. Strategic

reliabilismis a view that explicitly recognizes our psychological limitations. If

it is part of our psychological functioning that we tend to ignore or undervalue

corrective strategies, this is something we must take seriously.

But even if we can implement the consider-the-opposite strategy to

good effect, there is another serious worry: What is its range to be? After all, it would surely be a mistake for us to adopt the strategy for every belief

we entertain. Such a strategy would turn us into unhappy, neurotic nebbishes,

impossible to get along with. (One of us is reminded of the student

who was obviously having serious life troubles and explained that it was

because he had really taken Descartes’ Meditations to heart.) Recall our

discussion in chapter 3 of the Four Ways to better reasoning. We noted

that there could be occasions when one could become a better reasoner by

replacing a reliable but expensive strategy with another that was less reliable

but less expensive. Replacing a wide ranging consider-the-opposite

strategy with a consider-the-opposite strategy with a narrower range might

be an example of this. If a consider-the-opposite strategy is a good idea, it

is surely one whose range is to be limited to highly significant problems in

which short-term reliability is very important.