4. The tempting pleasures of broken legs

It doesn’t matter how reliable a reasoning rule might be if a reasoner

applies it poorly. There are two things the reasoner must do right. She

must execute the strategy correctly (e.g., plug in the right values, perform

the calculations properly), and she must apply the strategy to the right

sorts of problems. It is not always easy to know whether it is appropriate

to use a particular reasoning strategy in a particular case. This has come to

be known as the broken leg problem, and here is a classical statement of it:

Clinicians might be able to gain an advantage by recognizing rare events that

are not included in the actuarial formula (due to their infrequency) and that

countervail the actuarial conclusion. This possibility represents a variation

of the clinical-actuarial approach, in which one considers the outcome of

both methods and decides when to supercede the actuarial conclusion. In

psychology this circumstance has come to be known as the ‘broken leg’

The Amazing Success of Statistical Prediction Rules 45

problem, on the basis of an illustration in which an actuarial formula is

highly successful in predicting an individual’s weekly attendance at a movie

but should be discarded upon discovering that the subject is in a cast with a

fractured femur (footnotes deleted). The clinician may beat the actuarial

method if able to detect the rare fact and decide accordingly. In theory,

actuarial methods can accommodate rare occurrences, but the practical

obstacles are daunting. For example, the possible range of intervening events

is infinite. (Dawes, Faust, and Meehl 1989, 1670)

The broken leg problem arises because a person who applies a reasoning

strategy must judge whether it is appropriate to apply the strategy to this

particular case. But there are bound to be difficult cases. The broken leg

problem occurs when the person comes to believe she has strong evidence

for defecting from the strategy.

It doesn’t matter how reliable a reasoning rule might be if a reasoner

applies it poorly. There are two things the reasoner must do right. She

must execute the strategy correctly (e.g., plug in the right values, perform

the calculations properly), and she must apply the strategy to the right

sorts of problems. It is not always easy to know whether it is appropriate

to use a particular reasoning strategy in a particular case. This has come to

be known as the broken leg problem, and here is a classical statement of it:

Clinicians might be able to gain an advantage by recognizing rare events that

are not included in the actuarial formula (due to their infrequency) and that

countervail the actuarial conclusion. This possibility represents a variation

of the clinical-actuarial approach, in which one considers the outcome of

both methods and decides when to supercede the actuarial conclusion. In

psychology this circumstance has come to be known as the ‘broken leg’

The Amazing Success of Statistical Prediction Rules 45

problem, on the basis of an illustration in which an actuarial formula is

highly successful in predicting an individual’s weekly attendance at a movie

but should be discarded upon discovering that the subject is in a cast with a

fractured femur (footnotes deleted). The clinician may beat the actuarial

method if able to detect the rare fact and decide accordingly. In theory,

actuarial methods can accommodate rare occurrences, but the practical

obstacles are daunting. For example, the possible range of intervening events

is infinite. (Dawes, Faust, and Meehl 1989, 1670)

The broken leg problem arises because a person who applies a reasoning

strategy must judge whether it is appropriate to apply the strategy to this

particular case. But there are bound to be difficult cases. The broken leg

problem occurs when the person comes to believe she has strong evidence

for defecting from the strategy.