3.2. Policy assessment

In thinking about how to intervene in the world, we often have need for

beliefs about what sorts of policies (general strategies for dealing with

certain situations) are effective and which are not effective. For example,

many people have opinions about what sorts of social policies are best

suited to overcoming poverty in our country. These opinions (we hope)

rest on judgments about the effectiveness of various social policies; and these judgments (we hope) rest on good evidence. But what counts as

good evidence for thinking that a social policy has been effective? Long

and technical books have been written about policy analysis. While we can’t

expect ordinary reasoners to master the complexities of policy analysis, the

consider-the-control strategy might help people to overcome some glaring

errors associated with the assessment of policy.

Policy assessment is applicable to much more than issues of social policy.

It is also relevant to the assessment of different approaches to personal

and professional matters. To take an example close to home, everyone

knows that it’s hard to get good jobs in philosophy. Different people and

schools take different approaches to job searches. At the national conference

where job interviews are conducted, some job seekers are more

aggressive than others about seeking out and chatting up prospective employers

at receptions. Some departments are more aggressive than others

when it comes to lobbying for their graduate students. On the other side of

the job search, different departments take different approaches to the

hiring process. For example, some don’t conduct conference interviews,

while others conduct many short (15-minute) interviews. Which of these

strategies is most effective in finding a job (or hiring a job seeker)? Presumably,

many of our colleagues have firm opinions about these matters—

or at least opinions that are firm enough for them to act on. But does their

evidence support their opinions? We suspect that in most cases, the answer

is no. The reason is that most people involved in the job search don’t

know how well they would have done if they’d adopted different search

policies. (Actually, the truth is a bit more disturbing than this. At least

some of our colleagues know about, or have heard about, the interview

effect; so they have some evidence that their job searches would have

gone, on average, better if they’d not insisted upon short unstructured

interviews.)

We often employ strategies in social matters as well. In trying to

win friends and influence people at a party, we might try being funny or

charming or ‘‘coming on strong’’ or drinking to the point of imbecility. Of

course, often these strategies aren’t consciously adopted—they’re not the

result of an explicit decision procedure in which alternatives are considered

and rejected. But many of us sometimes reflect on our social behaviors

(perhaps after a particularly humiliating episode) and wonder

whether we might try to change our ways in the future in order to obtain

better results. Reflecting on these issues requires that we think about what

social strategies are most likely to help us achieve our social goals. For

many, this is a particularly significant kind of policy assessment.

There is a serious problem associated with policy assessment that

arises as soon as we consider the control. The problem is particularly acute

when it comes to the assessment of large-scale policies or seldom used

policies (like job search policies). The problemis that it’s hard to know how

the world would have been if we hadn’t adopted those policies. Gilovich

(1991) calls this the problem of hidden data (or hidden evidence). For

example, we don’t know what the current state of poverty would be like if

we’d not adopted certain poverty programs; and we don’t know how we’d

have done on the job market if we’d gone about things differently. Since

we can’t turn back time to see how the world would have turned out if

we would have adopted different policies, the best way to deal with the

problem of hidden data is to compare different policies (implemented in

relevantly similar circumstances) against each other. Sometimes, the problem

of hidden data is so severe that we would do better just to accept fewer

causal hypotheses than we do and to be a lot less confident about the ones

we do accept.

In thinking about how to intervene in the world, we often have need for

beliefs about what sorts of policies (general strategies for dealing with

certain situations) are effective and which are not effective. For example,

many people have opinions about what sorts of social policies are best

suited to overcoming poverty in our country. These opinions (we hope)

rest on judgments about the effectiveness of various social policies; and these judgments (we hope) rest on good evidence. But what counts as

good evidence for thinking that a social policy has been effective? Long

and technical books have been written about policy analysis. While we can’t

expect ordinary reasoners to master the complexities of policy analysis, the

consider-the-control strategy might help people to overcome some glaring

errors associated with the assessment of policy.

Policy assessment is applicable to much more than issues of social policy.

It is also relevant to the assessment of different approaches to personal

and professional matters. To take an example close to home, everyone

knows that it’s hard to get good jobs in philosophy. Different people and

schools take different approaches to job searches. At the national conference

where job interviews are conducted, some job seekers are more

aggressive than others about seeking out and chatting up prospective employers

at receptions. Some departments are more aggressive than others

when it comes to lobbying for their graduate students. On the other side of

the job search, different departments take different approaches to the

hiring process. For example, some don’t conduct conference interviews,

while others conduct many short (15-minute) interviews. Which of these

strategies is most effective in finding a job (or hiring a job seeker)? Presumably,

many of our colleagues have firm opinions about these matters—

or at least opinions that are firm enough for them to act on. But does their

evidence support their opinions? We suspect that in most cases, the answer

is no. The reason is that most people involved in the job search don’t

know how well they would have done if they’d adopted different search

policies. (Actually, the truth is a bit more disturbing than this. At least

some of our colleagues know about, or have heard about, the interview

effect; so they have some evidence that their job searches would have

gone, on average, better if they’d not insisted upon short unstructured

interviews.)

We often employ strategies in social matters as well. In trying to

win friends and influence people at a party, we might try being funny or

charming or ‘‘coming on strong’’ or drinking to the point of imbecility. Of

course, often these strategies aren’t consciously adopted—they’re not the

result of an explicit decision procedure in which alternatives are considered

and rejected. But many of us sometimes reflect on our social behaviors

(perhaps after a particularly humiliating episode) and wonder

whether we might try to change our ways in the future in order to obtain

better results. Reflecting on these issues requires that we think about what

social strategies are most likely to help us achieve our social goals. For

many, this is a particularly significant kind of policy assessment.

There is a serious problem associated with policy assessment that

arises as soon as we consider the control. The problem is particularly acute

when it comes to the assessment of large-scale policies or seldom used

policies (like job search policies). The problemis that it’s hard to know how

the world would have been if we hadn’t adopted those policies. Gilovich

(1991) calls this the problem of hidden data (or hidden evidence). For

example, we don’t know what the current state of poverty would be like if

we’d not adopted certain poverty programs; and we don’t know how we’d

have done on the job market if we’d gone about things differently. Since

we can’t turn back time to see how the world would have turned out if

we would have adopted different policies, the best way to deal with the

problem of hidden data is to compare different policies (implemented in

relevantly similar circumstances) against each other. Sometimes, the problem

of hidden data is so severe that we would do better just to accept fewer

causal hypotheses than we do and to be a lot less confident about the ones

we do accept.