Joanna Demaree-Cotton recently analysed a number of studies to estimate the magnitude of framing effects on moral intuition. Framing effects are changes in a subject's response caused by a mere presentational change in how the question is posed. The most robustly observed framing effect is an order effect. That is, the responses subjects give to each scenario will vary depending on the order in which the scenarios are presented.
If framing effects are large in magnitude, this suggests moral intuition is not very reliable: moral intuition is being moved by obviously irrelevant factors.
Demaree-Cotton argues that the magnitude of framing effects observed is not sufficiently great that we should regard moral intuition as unreliable. Putting the point without a double negative: the evidence is consistent with moral intuition being reasonably reliable.
Her methodology is as follows: between two treatments that differ only by frame, she compares the difference in percentage of subjects giving a particular judgment. She takes this difference to be an estimate of the probability that a given subject would have given a different response, had they been exposed to a different frame.
She finds that this probability is roughly 0.15–0.3 in most cases. Not much!
No doubt there is some greater degree of statistical sophistication that could have been brought to bear on this meta-analysis, but prima facie, this strikes me as about as much fanciness as is needed to make a point that rings true. All too often there is an emphasis on finding a statistically significant difference, and not enough emphasis on making a plausible estimate of the size of the effect. It would be very surprising if there were no difference due to changes of frame, but it would also be very surprising if it were a very large difference. Demaree-Cotton is suggesting that the difference probably is nothing much to write home about.
But I think the point can be pushed further. One consideration missing from her analysis is that the published literature is almost certainly a biased sample. It will be biased towards larger effects than one would expect in the full population.
Consider all the studies of framing effects on moral intuition that have not been published. They may have been unpublished because the authors procrastinated, met with misfortune, or other causes that have nothing to do with the findings. But there are also going to be some that are unpublished because the findings were less exciting, leading journals to reject them, or leading the authors to abandon the study. Studies that find larger framing effects are much less likely to be unpublished for these reasons. Studies that find smaller, or non-existent framing effects, are much more likely to be unpublished for these reasons. So there is good reason to think that a less biased sampling of the evidence than using published papers only would result in a smaller estimate.
So if anything, Demaree-Cotton is overestimating the susceptibility of moral intuition to framing effects. Yet more good news for moral intuition? (See also this discussion at PeaSoup)