Good news for moral intuition mongers
Joanna Demaree-Cotton has analyzed a number of studies to estimate the magnitude of framing effects on moral intuition. Framing effects are changes in a subject’s response caused by a superficial change in how the question is posed. The most robust framing effect is an order effect, where responses to individual scenarios vary, depending on the order in which different scenarios are presented.
If framing effects are large in magnitude, this suggests moral intuition is not reliable: moral intuition is being moved – and moved to a great extent – by irrelevant factors.
Demaree-Cotton argues the magnitude of framing effects observed is not sufficiently great that we should regard moral intuition as unreliable. Putting the point without a double negative: the evidence is consistent with moral intuition being reasonably reliable.
Her methodology is as follows: between two treatments that differ only in frame, she compares the difference in percentage of subjects giving a particular judgment. She takes this difference to be an estimate of the probability that a given subject would have given a different response, had they been exposed to a different frame.
The probability is roughly 0.15–0.3 in most cases. Not tiny, but not overwhelming.
No doubt additional statistical sophistication could have been brought to bear on this meta-analysis, but this is enough to make the point. All too often researchers emphasize finding a statistically significant difference, without making a plausible case that the effect is of practical importance.[1] It would be surprising if there were no difference due to changes of frame, but it would also be surprising if it were a very large difference. Demaree-Cotton is suggesting that the difference is nothing to write home about.
The point can be pushed further, however. An important factor is missing from this analysis: the published literature is almost certainly a biased sample. It will be biased towards larger effects than one would expect in the full population.
Consider all the framing effect studies on moral intuition that have not been published. They may have been unpublished because the authors procrastinated, met with misfortune, or other random causes. But some went unpublished because the findings were less exciting – so journals rejected them or the authors abandoned the project. Studies that find larger framing effects are much less likely to be unpublished for these reasons. Studies that find smaller, or non-existent framing effects, are much more likely to be unpublished for these reasons. If we included the unpublished studies in our sample, we would get a smaller estimate of the effect size.
So if anything, Demaree-Cotton overestimates the susceptibility of moral intuition to framing effects. While I suspect there are several other reasons to distrust our moral judgments, moral intuition-mongers need not worry about framing effects in particular.
Further reading
See also this discussion at PeaSoup.