4.3. Three caveats on defection

In light of the documented failure of selective defection strategies, we have

suggested that overriding an SPR is a good idea only in very unusual

circumstances. But we offer three caveats. First, for particularly significant

problems in a new domain, it will often make sense to test the SPR against

expert prediction on the new cases before making judgments. There is an

attitude (and often explicit prescriptions) of caution when applying instruments

or techniques to new domains, particularly high-risk domains.

This attitude is evident in gene therapy and cloning. But when it’s not

possible to carefully determine which tool is better on the new domain, a

conservative attitude to defection is warranted—particularly for domains

without grounded SPRs. As we’ve already argued, in those domains, defection

to human judgment is generally unreliable.

Second, it is important to keep SPRs current—especially those that

tend to handle especially significant problems. The parts of the natural and

social world to which SPRs are applied are dynamic. If SPRs detect people’s

dispositions, then we should attend to any of the social or psychological

trends that change people’s relevant behavioral dispositions. Many

of these conditions change over time: Crime initiatives in law enforcement,

federal housing subsidies, emergency health care policies, and yes,

even people’s knowledge that statistical prediction rules, and more broadly

actuarial methods, are being used to categorize them in various ways (see

Hacking 1999). In order to ensure that the SPRs perform with optimal

accuracy, SPRs must be regularly updated with fresh outcome information.

In fact, it will often be more important to keep an SPR current than it

will be to put effort into determining the conditions under which it is best

to defect from it.

And third, after defecting from an SPR on the grounds of a broken leg

problem, it is important to go back to the SPR next time (unless there is

another such problem). Applying successful SPRs is an epistemically excellent

tendency to cultivate. Defecting from an SPR frustrates and undermines

the formation of such positive habits. If defecting from an SPR

undermines our long-term commitment to using it, then defection is a risky proposal, even when one is faced with a genuine broken leg problem.

Ideally, we should take the proven exceptions and build them into a

better SPR, if this can be done simply enough that people can use it.

In light of the documented failure of selective defection strategies, we have

suggested that overriding an SPR is a good idea only in very unusual

circumstances. But we offer three caveats. First, for particularly significant

problems in a new domain, it will often make sense to test the SPR against

expert prediction on the new cases before making judgments. There is an

attitude (and often explicit prescriptions) of caution when applying instruments

or techniques to new domains, particularly high-risk domains.

This attitude is evident in gene therapy and cloning. But when it’s not

possible to carefully determine which tool is better on the new domain, a

conservative attitude to defection is warranted—particularly for domains

without grounded SPRs. As we’ve already argued, in those domains, defection

to human judgment is generally unreliable.

Second, it is important to keep SPRs current—especially those that

tend to handle especially significant problems. The parts of the natural and

social world to which SPRs are applied are dynamic. If SPRs detect people’s

dispositions, then we should attend to any of the social or psychological

trends that change people’s relevant behavioral dispositions. Many

of these conditions change over time: Crime initiatives in law enforcement,

federal housing subsidies, emergency health care policies, and yes,

even people’s knowledge that statistical prediction rules, and more broadly

actuarial methods, are being used to categorize them in various ways (see

Hacking 1999). In order to ensure that the SPRs perform with optimal

accuracy, SPRs must be regularly updated with fresh outcome information.

In fact, it will often be more important to keep an SPR current than it

will be to put effort into determining the conditions under which it is best

to defect from it.

And third, after defecting from an SPR on the grounds of a broken leg

problem, it is important to go back to the SPR next time (unless there is

another such problem). Applying successful SPRs is an epistemically excellent

tendency to cultivate. Defecting from an SPR frustrates and undermines

the formation of such positive habits. If defecting from an SPR

undermines our long-term commitment to using it, then defection is a risky proposal, even when one is faced with a genuine broken leg problem.

Ideally, we should take the proven exceptions and build them into a

better SPR, if this can be done simply enough that people can use it.