Scott Sumner suggests utilitarian stories drive our moral intuitions:
One of the most common strategies of the anti-utilitarian position is to assume some societal set-up which shocks our sensibilities, and then assume that it would satisfy the utilitarian criterion of maximizing aggregate happiness. Thus we might be asked to imagine a scenario where the total pleasures of the slave-owner exceed the suffering of the slaves … Bryan has an even more shocking example where the benefits to Nazi’s from the Holocaust exceeded the suffering to the Jews. … At the end of these thought experiments we are told that unless we are willing to embrace the society envisioned in the thought experiment, we must, on logical grounds, give up on utilitarianism.
I have several interrelated objections to this style of philosophical inquiry. I’d like to start with Richard Rorty’s assertion that the narrative arts (novels and film) produce liberal values. … So if Rorty is correct, how do we know that slavery was so awful? Because we have been exposed to accounts of slavery in the arts which vividly showed how the suffering of slaves was immeasurably greater that the frivolous pleasures of the slave-owner. Can we then turn around and use an imaginary slave-owning society that passes the utilitarian test as an argument against utilitarianism? I’m not sure that we can, unless one can show that our initial visceral reaction against slavery is based on non-utilitarian grounds, i.e. based on some abstract philosophical principle. And that’s much harder than many people might imagine.
This seems to me a powerful argument. What data could test it?
Added 9pm: As I understand it, the argument isn't that we can't now imagine compelling stories of, e.g., non-utilitarian-maxing slavery. The argument, I think, is more that we overgeneralize from the stories where we first picked up our morals. For example, we first hear stories where slave owners gain less than slaves lose, and then come to see all slavery as bad.