The "Intuitions" Behind "Utilitarianism"

Followup to:  Circular Altruism.  Response to:  Knowing your argumentative limitations, OR "one [rationalist's] modus ponens is another's modus tollens."

(Still no Internet access.  Hopefully they manage to repair the DSL today.)

I haven't said much about metaethics - the nature of morality - because that has a forward dependency on a discussion of the Mind Projection Fallacy that I haven't gotten to yet.  I used to be very confused about metaethics.  After my confusion finally cleared up, I did a postmortem on my previous thoughts.  I found that my object-level moral reasoning had been valuable and my meta-level moral reasoning had been worse than useless.  And this appears to be a general syndrome - people do much better when discussing whether torture is good or bad than when they discuss the meaning of "good" and "bad".  Thus, I deem it prudent to keep moral discussions on the object level wherever I possibly can.

Occasionally people object to any discussion of morality on the grounds that morality doesn't exist, and in lieu of jumping over the forward dependency to explain that "exist" is not the right term to use here, I generally say, "But what do you do anyway?" and take the discussion back down to the object level.

Paul Gowder, though, has pointed out that both the idea of choosing a googolplex dust specks in a googolplex eyes over 50 years of torture for one person, and the idea of "utilitarianism", depend on "intuition".  He says I've argued that the two are not compatible, but charges me with failing to argue for the utilitarian intuitions that I appeal to.

Now "intuition" is not how I would describe the computations that underlie human morality and distinguish us, as moralists, from an ideal philosopher of perfect emptiness and/or a rock. But I am okay with using the word "intuition" as a term of art, bearing in mind that "intuition" in this sense is not to be contrasted to reason, but is, rather, the cognitive building block out of which both long verbal arguments and fast perceptual arguments are constructed.

I see the project of morality as a project of renormalizing intuition.  We have intuitions about things that seem desirable or undesirable, intuitions about actions that are right or wrong, intuitions about how to resolve conflicting intuitions, intuitions about how to systematize specific intuitions into general principles.

Delete all the intuitions, and you aren't left with an ideal philosopher of perfect emptiness, you're left with a rock.

Keep all your specific intuitions and refuse to build upon the reflective ones, and you aren't left with an ideal philosopher of perfect spontaneity and genuineness, you're left with a grunting caveperson running in circles, due to cyclical preferences and similar inconsistencies.

"Intuition", as a term of art, is not a curse word when it comes to morality - there is nothing else to argue from.  Even modus ponens is an "intuition" in this sense - it's just that modus ponens still seems like a good idea after being formalized, reflected on, extrapolated out to see if it has sensible consequences, etcetera.

So that is "intuition".

However, Gowder did not say what he meant by "utilitarianism".  Does utilitarianism say...

1. That right actions are strictly determined by good consequences?
2. That praiseworthy actions depend on justifiable expectations of good consequences?
3. That probabilities of consequences should normatively be discounted by their probability, so that a 50% probability of something bad should weigh exactly half as much in our tradeoffs?
4. That virtuous actions always correspond to maximizing expected utility under some utility function?
5. That two harmful events are worse than one?
6. That two independent occurrences of a harm (not to the same person, not interacting with each other) are exactly twice as bad as one?
7. That for any two harms A and B, with A much worse than B, there exists some tiny probability such that gambling on this probability of A is preferable to a certainty of B?

If you say that I advocate something, or that my argument depends on something, and that it is wrong, do please specify what this thingy is... anyway, I accept 3, 5, 6, and 7, but not 4; I am not sure about the phrasing of 1; and 2 is true, I guess, but phrased in a rather solipsistic and selfish fashion: you should not worry about being praiseworthy.

Now, what are the "intuitions" upon which my "utilitarianism" depends?

This is a deepish sort of topic, but I'll take a quick stab at it.

First of all, it's not just that someone presented me with a list of statements like those above, and I decided which ones sounded "intuitive".  Among other things, if you try to violate "utilitarianism", you run into paradoxes, contradictions, circular preferences, and other things that aren't symptoms of moral wrongness so much as moral incoherence.

After you think about moral problems for a while, and also find new truths about the world, and even discover disturbing facts about how you yourself work, you often end up with different moral opinions than when you started out.  This does not quite define moral progress, but it is how we experience moral progress.

As part of my experienced moral progress, I've drawn a conceptual separation between questions of type Where should we go? and questions of type How should we get there?  (Could that be what Gowder means by saying I'm "utilitarian"?)

The question of where a road goes - where it leads - you can answer by traveling the road and finding out.  If you have a false belief about where the road leads, this falsity can be destroyed by the truth in a very direct and straightforward manner.

When it comes to wanting to go to a particular place, this want is not entirely immune from the destructive powers of truth.  You could go there and find that you regret it afterward (which does not define moral error, but is how we experience moral error).

But, even so, wanting to be in a particular place seems worth distinguishing from wanting to take a particular road to a particular place.

Our intuitions about where to go are arguable enough, but our intuitions about how to get there are frankly messed up.  After the two hundred and eighty-seventh research study showing that people will chop their own feet off if you frame the problem the wrong way, you start to distrust first impressions.

When you've read enough research on scope insensitivity - people will pay only 28% more to protect all 57 wilderness areas in Ontario than one area, people will pay the same amount to save 50,000 lives as 5,000 lives... that sort of thing...

Well, the worst case of scope insensitivity I've ever heard of was described here by Slovic:

Other recent research shows similar results. Two Israeli psychologists asked people to contribute to a costly life-saving treatment. They could offer that contribution to a group of eight sick children, or to an individual child selected from the group. The target amount needed to save the child (or children) was the same in both cases. Contributions to individual group members far outweighed the contributions to the entire group.

There's other research along similar lines, but I'm just presenting one example, 'cause, y'know, eight examples would probably have less impact.

If you know the general experimental paradigm, then the reason for the above behavior is pretty obvious - focusing your attention on a single child creates more emotional arousal than trying to distribute attention around eight children simultaneously.  So people are willing to pay more to help one child than to help eight.

Now, you could look at this intuition, and think it was revealing some kind of incredibly deep moral truth which shows that one child's good fortune is somehow devalued by the other children's good fortune.

But what about the billions of other children in the world?  Why isn't it a bad idea to help this one child, when that causes the value of all the other children to go down?  How can it be significantly better to have 1,329,342,410 happy children than 1,329,342,409, but then somewhat worse to have seven more at 1,329,342,417?

Or you could look at that and say:  "The intuition is wrong: the brain can't successfully multiply by eight and get a larger quantity than it started with.  But it ought to, normatively speaking."

And once you realize that the brain can't multiply by eight, then the other cases of scope neglect stop seeming to reveal some fundamental truth about 50,000 lives being worth just the same effort as 5,000 lives, or whatever.  You don't get the impression you're looking at the revelation of a deep moral truth about nonagglomerative utilities.  It's just that the brain doesn't goddamn multiply.  Quantities get thrown out the window.

If you have $100 to spend, and you spend $20 each on each of 5 efforts to save 5,000 lives, you will do worse than if you spend $100 on a single effort to save 50,000 lives.  Likewise if such choices are made by 10 different people, rather than the same person.  As soon as you start believing that it is better to save 50,000 lives than 25,000 lives, that simple preference of final destinations has implications for the choice of paths, when you consider five different events that save 5,000 lives.

(It is a general principle that Bayesians see no difference between the long-run answer and the short-run answer; you never get two different answers from computing the same question two different ways.  But the long run is a helpful intuition pump, so I am talking about it anyway.)

The aggregative valuation strategy of "shut up and multiply" arises from the simple preference to have more of something - to save as many lives as possible - when you have to describe general principles for choosing more than once, acting more than once, planning at more than one time.

Aggregation also arises from claiming that the local choice to save one life doesn't depend on how many lives already exist, far away on the other side of the planet, or far away on the other side of the universe.  Three lives are one and one and one.  No matter how many billions are doing better, or doing worse. 3 = 1 + 1 + 1, no matter what other quantities you add to both sides of the equation.  And if you add another life you get 4 = 1 + 1 + 1 + 1.  That's aggregation.

When you've read enough heuristics and biases research, and enough coherence and uniqueness proofs for Bayesian probabilities and expected utility, and you've seen the "Dutch book" and "money pump" effects that penalize trying to handle uncertain outcomes any other way, then you don't see the preference reversals in the Allais Paradox as revealing some incredibly deep moral truth about the intrinsic value of certainty.  It just goes to show that the brain doesn't goddamn multiply.

The primitive, perceptual intuitions that make a choice "feel good" don't handle probabilistic pathways through time very skillfully, especially when the probabilities have been expressed symbolically rather than experienced as a frequency.  So you reflect, devise more trustworthy logics, and think it through in words.

When you see people insisting that no amount of money whatsoever is worth a single human life, and then driving an extra mile to save $10; or when you see people insisting that no amount of money is worth a decrement of health, and then choosing the cheapest health insurance available; then you don't think that their protestations reveal some deep truth about incommensurable utilities.

Part of it, clearly, is that primitive intuitions don't successfully diminish the emotional impact of symbols standing for small quantities - anything you talk about seems like "an amount worth considering".

And part of it has to do with preferring unconditional social rules to conditional social rules.  Conditional rules seem weaker, seem more subject to manipulation.  If there's any loophole that lets the government legally commit torture, then the government will drive a truck through that loophole.

So it seems like there should be an unconditional social injunction against preferring money to life, and no "but" following it.  Not even "but a thousand dollars isn't worth a 0.0000000001% probability of saving a life".  Though the latter choice, of course, is revealed every time we sneeze without calling a doctor.

The rhetoric of sacredness gets bonus points for seeming to express an unlimited commitment, an unconditional refusal that signals trustworthiness and refusal to compromise.  So you conclude that moral rhetoric espouses qualitative distinctions, because espousing a quantitative tradeoff would sound like you were plotting to defect.

On such occasions, people vigorously want to throw quantities out the window, and they get upset if you try to bring quantities back in, because quantities sound like conditions that would weaken the rule.

But you don't conclude that there are actually two tiers of utility with lexical ordering.  You don't conclude that there is actually an infinitely sharp moral gradient, some atom that moves a Planck distance (in our continuous physical universe) and sends a utility from 0 to infinity.  You don't conclude that utilities must be expressed using hyper-real numbers.  Because the lower tier would simply vanish in any equation.  It would never be worth the tiniest effort to recalculate for it.  All decisions would be determined by the upper tier, and all thought spent thinking about the upper tier only, if the upper tier genuinely had lexical priority.

As Peter Norvig once pointed out, if Asimov's robots had strict priority for the First Law of Robotics ("A robot shall not harm a human being, nor through inaction allow a human being to come to harm") then no robot's behavior would ever show any sign of the other two Laws; there would always be some tiny First Law factor that would be sufficient to determine the decision.

Whatever value is worth thinking about at all, must be worth trading off against all other values worth thinking about, because thought itself is a limited resource that must be traded off.  When you reveal a value, you reveal a utility.

I don't say that morality should always be simple.  I've already said that the meaning of music is more than happiness alone, more than just a pleasure center lighting up.  I would rather see music composed by people than by nonsentient machine learning algorithms, so that someone should have the joy of composition; I care about the journey, as well as the destination.  And I am ready to hear if you tell me that the value of music is deeper, and involves more complications, than I realize - that the valuation of this one event is more complex than I know. 

But that's for one event.  When it comes to multiplying by quantities and probabilities, complication is to be avoided - at least if you care more about the destination than the journey.  When you've reflected on enough intuitions, and corrected enough absurdities, you start to see a common denominator, a meta-principle at work, which one might phrase as "Shut up and multiply."

Where music is concerned, I care about the journey.

When lives are at stake, I shut up and multiply.

It is more important that lives be saved, than that we conform to any particular ritual in saving them.  And the optimal path to that destination is governed by laws that are simple, because they are math.

And that's why I'm a utilitarian - at least when I am doing something that is overwhelmingly more important than my own feelings about it - which is most of the time, because there are not many utilitarians, and many things left undone.

</rant>

Comments

[Adam Safron]

Eliezer, to be clear, do you still think that 3^^^3 people having momentary eye irritations--from dust-specs--is worth torturing a single person for 50 years, or is there a possibility that you did the math incorrectly for that example? A proper utilitarian needs to consider the full range of outcomes--and their probabilities--associated with different alternatives. If the momentary eye irritation leads to a greater than 1/3^^^3 probability that someone will have an accident that leads to an outcome worse than 50 years of torture, then the dust specs are preferable. But if the chance of further negative consequences from momentary eye-irritation is so small as to be negligible, then we can consider the cost of the dust specs to be the linear sum of the hedonic loss across all of the people afflicted. The torture, on the other hand, has a significant probability of leading to further negative consequences that could persist across a life-span and impact those who care about that individual. If the tortured individual has a significant probability of committing suicide, then we need to consider all of the potential experiences and accomplishments that the person would have had over the course of their life-time--which could be indefinitely long, depending on how technology progresses--and the impact that the person would have had on others. And finally, as I think you would agree, we wouldn't want to use an ethical utility function that only considered basic hedonic experience and ignored higher-level meaning. If you merely integrated all of the moments of pleasure/pain across a life-span, you wouldn't have come close to calculating the value of that life. Music is worth more than the sum of the notes that went into the song. While your basic argument is valid and important, you probably--depending on the details of the argument--came to the wrong conclusion with respect to dust-specs and torture.

[Eliezer Yudkowsky]

Eliezer, to be clear, do you still think that 3^^^3 people having momentary eye irritations--from dust-specs--is worth torturing a single person for 50 years, or is there a possibility that you did the math incorrectly for that example?

No. I used a number large enough to make math unnecessary.

I specified the dust specks had no distant consequences (no car crashes etc.) in the original puzzle.

Unless the torture somehow causes Vast consequences larger than the observable universe, or the suicide of someone who otherwise would have been literally immortal, it doesn't matter whether the torture has distant consequences or not.

I confess I didn't think of the suicide one, but I was very careful to choose an example that didn't involve actually killing anyone, because there someone was bound to point out that there was a greater-than-tiny probability that literal immortality is possible and would otherwise be available to that person.

So I will specify only that the torture does not have any lasting consequences larger than a moderately sized galaxy, and then I'm done. Nothing bound by lightspeed limits in our material universe can morally outweigh 3^^^3 of anything noticeable. You'd have to leave our physics to do it.

You know how some people's brains toss out the numbers? Well, when you're dealing with a number like 3^^^3 in a thought experiment, you can toss out the event descriptions. If the thing being multiplied by 3^^^3 is good, it wins. If the thing being multiplied by 3^^^3 is bad, it loses. Period. End of discussion. There are no natural utility differences that large.

[Nick Tarleton]

Favoring an unconditional social injunction against valuing money over lives is consistent with risking one's own life for money; you could reason that if trading off money and other people's lives is permitted at all, this power will be abused so badly that an unconditional injunction has the best expected consequences. I don't think this is true (because I don't think such an injunction is practical), but it's at least plausible.

[tcpkac]

A Utilitarian should care about the outcomes of Utilitarianism..... and yes, as soon as ends justify means, you do get Stalin, Mao, Pol Pot, who were all striving for good consequences......
Which is relevant, as your arguments mostly involve saving lives (a single type of outcome, so making options intuitively comparable).
I'm afraid the 'rant' doesn't add much in terms of content, that I can see.

[Mike Carey]

So it seems you have two intuitions. One is that you like certain kinds of "feel good" feedback that aren't necessarily mathematically proportional to the quantifiable consequences. Another is that you like mathematical proportionality. The "Shut up and multiply" mantra is simply a statement that your second preference is stronger than the first.

In some ways it seems reasonable to define morality in a way that treats all people equally. If we do so, than our preference for multiplying can be more moral, by definition, than our less rational sympathies. But creating a precise definition generally has the result of creating a gap between that definition and common usage. People use the term "morality" and accompanying concepts in a number of ways. Restricting its usage may make a debate more intelligible, but it tends to obscure the fact that morality is a multi-faceted concept that represents a number of different preferences and beliefs. Even meta-morality can do no more than articulate certain kinds of meta-preferences.

Also, equating utilitarianism with Pol Pot and Stalin is a bit disingenuous. Those people weren't utilitarian in any recognizable sense because the total consequences of their actions (millions dead), didn't justify their intended consequences (whatever those were). Millions dead shouldn't be placed solely in the "means" category.

[Adam Safron]

"Well, when you're dealing with a number like 3^^^3 in a thought experiment, you can toss out the event descriptions. If the thing being multiplied by 3^^^3 is good, it wins. If the thing being multiplied by 3^^^3 is bad, it loses. Period. End of discussion. There are no natural utility differences that large."

Let's assume the eye-irritation lasts 1-second (with no further negative consequences). I would agree that 3^^^3 people suffering this 1-second irritation is 3^^^3-times worse than 1 person suffering thusly. But this irritation should not be considered to be equal to 3^^^3 seconds of wasted lives. In fact, this scenario is so negligibly bad, as to not be worth the mental effort to consider it.

And for the torture option, let's assume that the suffering stops the instant the person finishes their 50 years of pain (the person leaves in exactly the same psychological state they were in before they found out that they would be tortured). However, in this case, 50 years of being tortured is not (50 years * 365 days * 24 hours * 3600 seconds)-times worse than 1-second of torture. It is much (non-linearly) worse than that. There are other variables to consider. In those 50-years, the person will miss 50 years of life. Unlike the dust-speck irritation distributed across 3^^^3 people, 50 years of torture is worth considering.

Adding experiences across people should linearly impact the estimated utility, but things do not add linearly when considering the experiences of a single person. Even if it doesn't lead to further negative consequences, the one-second of irritation is less than 3^^^3-times as bad as the 50 years of torture.

If you're multiplication has taken you so far afield of your intuitions, re-check the math. If it still comes out the same way, check your assumptions. If it still comes out the same way, go with the calculations.

[manuelg]

3^^^3?

http://www.overcomingbias.com/2008/01/protecting-acro.html#comment-97982570

> > A 2% annual return adds up to a googol (10^100) return over 12,000 years

> Well, just to point out the obvious, there aren't nearly that many atoms in a 12,000 lightyear radius.

Robin Hanson didn't get very close to 3^^^3 before you set limits on his use of "very very large numbers".

Secondly, you refuse to put "death" on the same continuum as "mote in the eye", but behave sanctimoniously (example below) when people refuse to put "50 years of torture" on the same continuum as "mote in the eye".

> Where music is concerned, I care about the journey.

> When lives are at stake, I shut up and multiply.

I assert the use of 3^^^3 in a moral argument is to _avoid_ the effort of multiplying. Demonstration: what is 3^^^3 times 6? What is 3^^^3 times a trillion to the trillionth power?

Where am I going with this? I am very interested in improving my own personal morality and rationality. I am profoundly disinterested in passing judgment on any one else's morality or rationality.

I assert that the use of 3^^^3 in a moral argument has nothing to do with someone improving their own personal morality or rationality. It has _everything_ to do with trying to _shame_ someone else into admitting that they aren't A "good little rational moralist".

My comment is an attempt to steer the thread of your (very interesting and well written) posts towards topics that will help me improve my own personal morality and rationality. (I admit that I perceive no linkage between the "wheel in my hand" and the "rudder of the ship", so I doubt my steering will work.)

[Unknown]

The only reason Eliezer didn't put death on the same scale as the dust mote was on account of his condition that the dust specks have no further consequences. In real life, everything has consequences, and so in real life, death is on the same scale with everything else, including dust motes. Eliezer expressed this extremely well: "Whatever value is worth thinking about at all, must be worth trading off against all other values worth thinking about, because thought itself is a limited resource that must be traded off."

So yes, in real life there is some number of dust motes such that it would be better to prevent the dust storm than to save a life.

[Sean]

A dust speck in the eye with no external ill effects was chosen as the largest non-zero negative utility.
Torture, absent external effects (e.g., suicide), for any finite time, is a finite amount of negative utility.
Death in a world of literal immortality cuts off an infinite amount of utility.
There is a break in the continuum here.

If you don't accept that dust specks are negative utility, you didn't follow the rules. Pick a new tiny ill effect (like a stubbed toe) and rethink the problem.

If you still don't like it because for a given utility n, n + n != 2n, there are then issues with circular preferences. Two units of utility are defined as twice as "utilitous" as one unit of utility. (This is not saying that two dollars are twice as good as one dollar.)

[Eliezer Yudkowsky]

I assert the use of 3^^^3 in a moral argument is to _avoid_ the effort of multiplying.

Yes, that's what I said. If the quantities were close enough to have to multiply, the case would be open for debate even to utilitarians.

Demonstration: what is 3^^^3 times 6?

3^^^3, or as close as makes no difference.

What is 3^^^3 times a trillion to the trillionth power?

3^^^3, or as close as makes no difference.

...that's kinda the point.

So it seems you have two intuitions. One is that you like certain kinds of "feel good" feedback that aren't necessarily mathematically proportional to the quantifiable consequences. Another is that you like mathematical proportionality.

Er, no. One intuition is that I like to save lives - in fact, as many lives as possible, as reflected by my always preferring a larger number of lives saved to a smaller number. The other "intuition" is actually a complex compound of intuitions, that is, a rational verbal judgment, which enables me to appreciate that any non-aggregative decision-making will fail to lead to the consequence of saving as many lives as possible given bounded resources to save them.

I'm feeling a bit of despair here... it seems that no matter how I explain that this is how you have to plan if you want the plans to work, people just hear, "You like neat mathematical symmetries." Optimal plans are neat because optimality is governed by laws and the laws are math - it has nothing to do with liking neatness.

50 years of being tortured is not (50 years * 365 days * 24 hours * 3600 seconds)-times worse than 1-second of torture. It is much (non-linearly) worse than that.

Utilitarianism does not assume that multiple experiences to the same person aggregate linearly.

Yes, I agree that it is non-linearly worse.

It is not infinitely worse. Just non-linearly worse.

The non-linearity factor is nowhere within a trillion to the trillionth power galaxies as large as 3^^^3.

If it were, no human being would ever think about anything except preventing torture or goals of similar importance. You would never take a single moment to think about putting an extra pinch of salt in your soup, if you felt a utility gradient that large. For that matter, your brain would have to be larger than the observable universe to feel a gradient that large.

I do not think people understand the largeness of the Large Number here.

[Adam Safron]

Correction:
What I said: "one-second of irritation is less than 3^^^3-times as bad as the 50 years of torture."
What I meant: "50 years of torture is more than 3^^^3-times as bad as 1-second of eye-irritation."
Apologies for the mis-type (as well as for saying "you're" when I meant "your").

But the point is, if there are no additional consequences to the suffering, then it's irrelevant. I don't care how many people experience the 1-second of suffering. There is no number large enough to make it matter.

Eliezer had a good point. It works if we're considering lives saved. It doesn't work with dust-specks and torture. It's not because torture is a taboo that only hard-headed rationalists are able to consider with a clear-mind. It's because something that's non-consequential is non-consequential, even when you multiply it by unimaginably large numbers. But we can't imagine torture and 50 years of lost time as being non-consequential for good reason. The example was bad. We should move on to more productive endeavors.

[Nick Tarleton]

It's because something that's non-consequential is non-consequential

The dust specks are consequential; people suffer because of them. The further negative consequences of torture are only finitely bad.

[Eisegetes]

Eliezer: would you torture a person for fifty years, if you lived in a large enough universe to contain 3^^^3 people, and if the omnipotent and omniscient ruler of that universe informed you that if you did not do so, he would carry out the dust-speck operation?

Seriously, would you pick up the blow torch and use it for the rest of your life, for the sake of the dust-specks?

[Larry D'anna]

Eliezer: it doesn't matter how big of a number you can write down. You are dealing with an asymptote. There is a limit to how bad momentary eye-irritation can be, no matter how many people it happens to. *no* *matter* *how* *many* *people*. That limit is far less than how bad a 50 year torture is.

let f(x) = (5x - 1)/x
what is f(3^^^3)? It's 5, or close enough that it doesn't matter.

[Jadagul]

Eliezer: after wrestling with this for a while, I think I've identified at least one of the reasons for all the fighting. First of all, I agree with you that the people who say, "3^^^3 isn't large enough" are off-base. If there's some N that justifies the tradeoff, 3^^^3 is almost certainly big enough; and even if it isn't, we can change the number to 4^^^4, or 3^^^^3, or Busy Beaver (Busy Beaver (3^^^3)), or something, and we're back to the original problem.

For me, at least, the problem comes down to what 'preference' means. I don't think I have any coherent preferences over the idea of 3^^^3 dust specks. Note, I don't mean that I think my preferences are inconsistent, or poorly-formed, or that my intuition is bad. I don't think that talking about my preferences on that issue has any meaning.

Basically, I don't believe there's any objective standard of value. Even preferences like "I think as many people should die as painfully as possible" aren't wrong, per se; they just put you beyond the bounds of civilized society and make me have no desire to interact with you any more. So asking which of two circumstances is 'really better' doesn't have any meaning; 'better' only makes sense when you ask 'better to whom.' Which leads to two problems.

First is that the question tends to slip over to "which choice would you make." But once I start phrasing it in terms of me making a choice, all my procedural safeguards start kicking in. First, if you're a true deontologist your mental side constraints start jumping in. Even if you're sort-of utilitarian, like I am, the mental rules that say things like "we can't be sure that 3^^^3 people are actually going to suffer" and "helping to forge a society that considers torture acceptable leads to horrifying long-term consequences" kick in. I agree those are outside of the parameters of the original question; but the original question was ill-posed, and this is one of the places it slips to in translation.

But even if you avoid that, you still come to the question of what it means to prefer A over B, when you have no meaningful choice in the matter. I can't imagine a situation in which I could cause 3^^^3 people any coherent result. I'm not sure I believe there are or ever will be 3^^^3 moral agents. And do I have a coherent preference over circumstances that I will never know have occurred? Even if 3^^^3 people suffer, I'm not going to know that they do. It won't affect me, and I won't know that it affected anyone else, either.

Basically, moral questions that involve wildly unlikely or outright impossible scenarios don't tend to be terribly enlightening. If we lived in a world where we could reliably benefit unimaginably large numbers of people by causing vast pain to a few, maybe that would be okay. But since we don't, I think hypotheticals like this are more likely to short-circuit on the bounds of our extremely useful assumptions about the nature of the world than they are to tell us anything interesting.

[Richard]

Stick with your resolution to avoid meta-ethical reasoning, since yours sucks.

[Bob]

I share El's despair. Look at the forest, folks. The point is that you *have* to recognize that harm aggregates (and not to an asymptote) or you are willing to do terrible things. The idea of torture is introduced precisely to make it hard to see. But it is important, particularly in light of how easily our brains fail to scale harm and benefit. Geez, I don't even have to look at the research El cites - the comments are enough.

Stop saying the specks are "zero harm." This is a thought experiment and they are defined as positive harm.

Stop saying that torture is different. This is a thought experiment and torture is defined to be absolutely terrible, but finite, harm.

Stop saying that torture is infinite harm. That's just silly.

Stop proving the point over and over in the comments!

/rant/

[Ben]

This whole dust vs. torture "dilemma" depends on a couple assumptions: (1) That you can assign a cost to any event and that all such values lie within the same group (allowing multiples of one event to "add up" to another event) and (2) That the function that determines the cost of a certain number of a specific type of events does not have a hard upper limit (such as a logistic function). If either of these assumptions is wrong then the largeness of 3^^^3 or any other "large" number is totally irrelevant. One way to test (1) is to replace "torture" with "kill". If the answer is no then (1) is an invalidate assumption.

[komponisto]

Larry D'anna:

You are dealing with an asymptote. There is a limit to how bad momentary eye-irritation can be, no matter how many people it happens to.

By positing that dust-speck irritation aggregates non-linearly with respect to number of persons, and thereby precisely exemplifying the scope-insensitivity that Eliezer is talking about, you are not making an argument against his thesis; instead, you are merely providing an example of what he's warning against.

You are in effect saying that as the number of persons increases, the marginal badness of the suffering of each new victim decreases. But why is it more of an offense to put a speck in the eye of Person #1 than Person #6873?

[alexa-blue]

Isn't there maybe a class insignificant harms where net utility is neutral or even positive (learn to squint or where goggles in a duststorm, learn that motes in ones eye are annoying but nothing really to worry about, increased unerstanding of Christian parables, eg; also consider schools of parenting that allow children to experiment with various behaviors that the parents would prefer they avoid, since directly experiencing the adverse event in a more controlled situation will prevent worse outcomes down the road). I'm not sure you can trust most people's expressed preference on this.

That being said, I don't know where that class begins and ends.

[Caledonian]

Well, Eliezer has not only failed to recognize that he was wrong, he actively defended his point.

Should the ten credibility points be donated to charity, or dispersed among the readers of Overcoming Bias?

[Neel Krishnaswami]

Bob: Sure, if you specify a disutility function that mandates lots-o'-specks to be worse than torture, decision theory will prefer torture. But that is literally begging the question, since you can write down a utility function to come to any conclusion you like. On what basis are you choosing that functional form? That's where the actual moral reasoning goes. For instance, here's a disutility function, without any of your dreaded asymptotes, that strictly prefers specks to torture:

U(T,S) = ST + S

Freaking out about asymptotes reflects a basic misunderstanding of decision theory, though. If you've got a rational preference relation, then you can always give a bounded utility function. (For example, the function I wrote above can be transformed to U(T,S) = (ST + S)/(ST + S + 1), which always gives you a function in [0,1], and gives rise to the same preference relation as the original.) If you absolutely require unbounded utilities in utility functions, then you become subject to a Dutch book (see Vann McGee's "An Airtight Dutch Book"). Attempts to salvage unbounded utility pretty much always end up accepting certain Dutch books as rational, which means you've rejected the whole decision-theoretic justification of Bayesian probability theory. Now, the existence of bounds means that if you have a monotone utility function, then the limits will be well-defined.

So asymptotic reasoning about monotonically increasing harms is entirely legit, and you can't rule it out of bounds without giving up on either Bayesianism or rational preferences.

[Cyan]

Care to advance an argument, Caledonian? (Not saying I disagree... or agree, for that matter.)

[Ian Maxwell]

If harm aggregates less-than-linearly in general, then the difference between the harm caused by 6271 murders and that caused by 6270 is less than the difference between the harm caused by one murder and that caused by zero. That is, it is worse to put a dust mote in someone's eye if no one else has one, than it is if lots of other people have one.

If relative utility is as nonlocal as that, it's entirely incalculable anyway. No one has any idea of how many beings are in the universe. It may be that murdering a few thousand people barely registers as harm, because eight trillion zarquons are murdered every second in Galaxy NQL-1193. However, Coca-Cola is relatively rare in the universe, so a marginal gain of one Coca-Cola is liable to be a far more weighty issue than a marginal loss of a few thousand individuals.

(This example is deliberately ridiculous.)

[Caledonian]

Neel Krishnaswami has already said pretty much everything I'm thinking.

From a rhetorical and logical perspective, it would have been much better if Eliezer had presented an experiment in which the participants were asked to choose between abstracted types of harm specially constructed to avoid all of the issues people have brought up... and still chose irrationally.

What's 'rational' in the example he did use is far from clear.

Oh, and his "I didn't have to do the math" is absurd, not only on its own grounds, but in the context of his broader points. Eliezer is normally an advocate for doing the math, and how he's brushing it off.

[Salutator]

So what exactly do you multiply when you shut up and multiply? Can it be anything else then a function of the consequences? Because if it is a function of the consequences, you do believe or at least act as if believing your #4.

In which case I still want an answer to my previously raised and unanswered point: As Arrow demonstrated a contradiction-free aggregate utility function derived from different individual utility functions is not possible. So either you need to impose uniform utility functions or your "normalization" of intuition leads to a logical contradiction - which is simple, because it is math.

[Peter de Blanc]

Neel Krishnaswami: you reference an article called "An Airtight Dutch Book," but I can't find it online without a subscription. Can you summarize the argument?

[Bob]

Neel, I think you and I are looking at this as two different questions. I'm fine with bounded utility at the individual level, not so good with bounds on some aggregate utility measure across an unbounded population (but certainly willing to listen to a counter position), which is what we're talking about here. Now, what form an aggregate utility function should take is a legitimate question (although, as Salutator points out, unlikely to be a productive discussion), but I doubt that you would argue it should be bounded.

I have really enjoyed following this discussion. My intuition to the initial post was "specks" but upon reflection I couldn't see how that could be. My intuition couldn't scale the problem correctly - I had to "do the math." Magically diminishing unit harm from the same action across an increasing number of individuals was not convincing. The notion that specks are not harm in the same sense that torture is harm is appealling in practice but was ruled out in the thought experiment, IMHO. Bottom line, if both specks and torture are finite harm (a reasonable premise, although the choice of "specks" was unfortunate because it opens the "zero harm" door), I can't see how there is not some sufficiently large number of specks that *have* to outweigh the torture. This is very uncomfortable because it implies that there is also (in theory) a sufficiently large number of specks that would outweigh 10^100 people being tortured. Ouch, my brain is trying to reject that conclusion. More evidence that I need to do the math. Luckily, this is all hypothetical. But scaling questions are not always hypothetical and nothing in this discussion has convinced me that intuition will give you the right answer. To the contrary...

[burger flipper]

Peter DeBlanc: check your email

[Sean]

The issue with a utility function U(T,S) = ST + S is that there is no motivation to have torture's utility depend on dust's utility. They are distinct and independent events, and in no way will additional specks worsen torture. If it is posited that dust specks asymptotically approach a bound lower than torture's bound, order issues present themselves and there should be rational preferences that place certain evils at such order that people should be unable to do anything but act to prevent those evils.

There's additional problems here, like the idea that distributing a dust speck to the group needs calculation in the group's net utility function, rather than in the individual's utility function. That is, if a group of ten people has 600 apples, they do not have 600*U(A), nor U(600A), but U_1(A_1)+ ... + U_10(A_10). Adding an additional apple has a marginal effect on net utility equal to the marginal effect on the net utility of the person receiving the apple. This result is in utils, and utils do sum linearly.

I'll say that again: Utils sum linearly. It's what they do. The rational utilitarian favors n utils gained from a chocolate as much as he favors avoiding -n utils of a stubbed toe. Summing -n utils over m people will have an identical effect on total or average utility as granting -(n*m) utility to one person.

If you reject any of the utilitarian or rational premises of the question, point them out, suggest your fix and defend it.

Caledonian:
The idea is to make the math obvious. If you can't get the right answer with the math clean and easy, how can you do it on your own? If you insist there is a natural number greater than the cardinality of the reals, you will run into problems somewhere else. (And on the other hand, if you reject any of the concepts such as cardinality, reals, or "greater than", you probably shouldn't be taking a math class.)

[Jadagul]

Sean: why is that "what utils do"? To the extent that we view utils as the semi-scientific concept from economics, they don't "just sum linearly." To economists utils don't sum at all; you can't make interpersonal comparisons of utility. So if you claim that utils sum linearly, you're making a claim of moral philosophy, and haven't argued for it terribly strongly.

[Unknown]

Sean, one problem is that people can't follow the arguments you suggest without these things being made explicit. So I'll try to do that:

Suppose the badness of distributed dust specks approaches a limit, say 10 disutility units.

On the other hand, let the badness of (a single case of ) 50 years of torture equal 10,000 disutility units. Then no number of dust specks will ever add up to the torture.

But what about 49 years of torture distributed among many? Presumably people will not be willing to say that this approaches a limit less than 10,000; otherwise we would torture a trillion people for 49 years rather than one person for 50.

So for the sake of definiteness, let 49 years of torture, repeatedly given to many, converge to a limit of 1,000,000 disutility units.

48 years of torture, let's say, might converge to 980,000 disutility units, or whatever.

Then since we can continuously decrease the pain until we reach the dust specks, there must be some pain that converges approximately to 10,000. Let's say that this is a stubbed toe.

Three possibilities: it converges exactly to 10,000, to less than 10,000, or more than 10,000. If it converges to less, than if we choose another pain ever so slightly greater than a toe-stubbing, this greater pain will converge to more than 10,000. Likewise, if it converges to more than 10,000, we can choose an ever so slightly less pain that converges to less than 10,000. If it converges to exactly 10,000, again we can choose a slightly greater pain, that will converge to more than 10,000.

Suppose the two pains are a stubbed toe that is noticed for 3.27 seconds, and one that is noticed for 3.28 seconds. Stubbed toes that are noticed for 3.27 seconds converge to 10,000 or less, let's say 9,999.9999. Stubbed toes that are notice for 3.28 seconds converge to 10,000.0001.

Now the problem should be obvious. There is some number of 3.28 second toe stubbings that is worse than torture, while there is no number of 3.27 second toe stubbings that is worse. So there is some number of 3.28 second toe stubbings such that no number of 3.27 second toe stubbings can ever match the 3.28 second toe stubbings.

On the other hand, three 3.27 second toe stubbings are surely worse than one 3.28 second toe stubbings. So as you increase the number of 3.28 second toe stubbings, there must be a magical point where the last 3.28 second toe stubbing crosses a line in the sand: up to that point, multiplied 3.27 second toe stubbings could be worse, but with that last 3.28 second stubbing, we can multiply the 3.27 second stubbings by a googleplex, or by whatever we like, and they will never be worse than that last, infinitely bad, 3.28 second toe stubbing.

So do the asymptote people really accept this? Your position requires it with mathematical necessity.

[Joseph Knecht]

It seems to me that the dust-specks example depends on the following being true: both dust-specks and 50 years of torture can be precisely quantified.

What is the justification for this belief? I find it hard to see any way of avoiding the conclusion that some harms may be compared, as in A < B (A=1 person/1 dustspeck, B=1 person/torture), but that does not imply that we can assign precise values to A and B and then determine how many A are equivalent to one B.

Why do some people believe that we can precisely say how much worse the torture of 1 individual is than dust specks in the eye of 1 individual? Why are different harms necessarily commensurable? Any pointers to standard arguments or more information about this is much appreciated.

[Bob]

Joseph,

The point of using 3^^^3 is to avoid the need to assign precise values, which I agree seems impossible to do with any confidence. Once you accept the premise that A is less than B (with both being finite and nonzero), you need to accept that there exists some number k where kA is greater than B. The objections have been that A=0, B is infinite, or the operation kA is not only nonlinear, but bounded. The first may be valid for specks but misses the point - just change it to "mild hangnail" or "banged funnybone." I cannot take the second seriously. The third is tempting when disguised as something like "you cannot compare a banged funnybone with torture" but becomes less appealing when you ask "can it really cause (virtually) zero harm to bang a trillion funnybones just because you've already banged 10^100?"

It was the need for a "magical line" as in Unknown's example that convinced me. I'm truly curious why it fails to convince others. I admit I may be missing something but it seems very simple at its core.

[Ian C.]

"Renormalizing intuition" - that sounds like making sure all the intuitions are consistent and proportional to each other. Which is analogous to a coherence theory of truth as against a correspondence one. But you can make something as internally consistent as you like and maybe it still bears no relation to reality. It is necessary to know where the intuitions came from and what they mean.

Ideas such as good and evil are abstract, and the mind of a newborn can't hold abstract ideas, only simple concretes. So those ideas can't have already been there at birth, simply because of how abstract they are. Therefore the intuitions must be learned after birth, which means they most likely come from society. Which means other people. Other people who have no more magic source of truth than we do.

So all we are doing is sorting through and evening out the opinions of others that we have subconsciously absorbed, which doesn't seem a very scientific way to go.

[mitchell porter]

Bob: "The point of using 3^^^3 is to avoid the need to assign precise values".

But then you are not facing up to the problems of your own ethical hypothesis. I insist that advocates of additive aggregation take seriously the problem of quantifying the exact ratio of badness between torture and speck-of-dust. The argument falls down if there is no such quantity, but how would you arrive at it, even in principle? I do not insist on an impersonally objective ratio of badness; we are talking about an idealized rational completion of one's personal preferences, nothing more. What considerations would allow you to determine what that ratio should be?

Unknown has pointed out that anyone who takes the opposite tack, insisting that 'any amount of X is always preferable to just one case of Y!', faces the problem of boundary cases: keep substituting worse and worse things for X, and eventually one will get into the territory of commensurable evils, and one will start trying to weigh up X' against Y.

However, this is not a knockdown argument for additivism. Let us say that I am clear about my preferences for situations A, D and E, but I am in a quandary with respect to B and C. Then I am presented with an alternative moral philosophy, which offers a clear decision procedure even for B and C, but at the price of violating my original preferences for A, D or E. Should I say, oh well, the desirability of being able to decide in all situations is so great that I should accept the new system, and abandon my original preferences? Or should I just keep thinking about B and C until I find a way to decide there as well? A utility function needs only to be able to rank everything, nothing more. There is absolutely no requirement that the utility (or disutility) associated with n occurrences of some event should scale linearly with n.

This is an important counterargument so I'll repeat it: The existence of problematic boundary cases is not yet a falsification of an ethical heuristic. Give your opponent a chance to think about the boundary cases, and see what they come up with! The same applies to my challenge to additive utilitarians, to say how they would arrive at an exact ratio: I am not asserting, apriori, that it is impossible. I am pointing out that it must be possible for your argument to be valid, and I'm giving you a chance to indicate how this can be done.

This whole thought experiment was, I believe, meant to illustrate a cognitive bias, a preference which, upon reflection, would appear to be mistaken, the mistake deriving from the principle that 'sacred values', such as an aversion to torture, always trump 'nonsacred values', like preventing minor inconveniences. But premises which pass for rational in a given time and culture - which are common sense, and just have to be so - can be wrong. The premise here is what I keep calling additivism, and we have every reason to scrutinize as critically as possible any premise which would endorse an evil of this magnitude (the 50 years of torture) as a necessary evil.

One last thought, I don't think Ben Jones's observation has been adequately answered. What if those 3^^^3 people are individually willing to endure the speck of dust rather than have someone tortured on their behalf? Again boundary cases arise. But if we're seriously going to resolve this question, rather than just all reaffirm our preferred intuitions, we need to keep looking at such details.

[Unknown]

Mitchell, if I say an average second of the torture is about equal 10,000 distributed dust specks (notice I said "average second"; there is absolutely no claim that torture adds up linearly or anything like that), then something less than 2 trillion dust specks will be about equal to 50 years of torture. I would arrive at the ratio by some comparison of this sort, trying to guess how bad an average second is, and how many dust specks I would be willing to inflict to save a man from that amount of harm.

Notice that 3^^^3 is completely unnecessary here. That's why I said previously that N doesn't have to be particularly large.

[Joseph Knecht]

Thanks for the explanations, Bob.

Bob: The point of using 3^^^3 is to avoid the need to assign precise values... Once you accept the premise that A is less than B (with both being finite and nonzero), you need to accept that there exists some number k where kA is greater than B.

This still requires that they are commensurable though, which is what seeking a strong argument for. Saying that 3^^^3 dust specks in 3^^^3 eyes is greater harm than 50 years of torture means that they are commensurable and that whatever the utilities are, 3^^^3 specks divided by 50 years of torture is greater than 1.0. I don't see that they are commensurable. A < B < C < D doesn't imply that there's some k such that kA>D.

Consider: I prefer Bach to Radiohead (though I love both). That doesn't imply that there's some ratio of Bach to Radiohead, or that I think a certain number of Radiohead songs are collectively better than or more desirable than, for example, the d-minor partita. Even if I did in some cases believe that 10 Radiohead songs were worth 1 Bach prelude and fugue, that would just be my subjective feeling. I don't see why there must be an objective ratio, and I can't see grounds for saying what such a ratio would be. Likewise for dust-specks and torture.

Like Mitchell, I would like to see exactly how people propose to assign these ratios such that a certan number of one harm is greater than a radically different harm.

[Tomhs]

A < B < C < D doesn't imply that there's some k such that kA>D

Yes it does.

[Caledonian]

Again, we return to the central issue:

Why must we accept an additive model of harm to be rational?

[Sean]

If you don't accept the additivity of harm, you accept that for any harm x, there is some number of people y where 2^y people suffering x harm is the same welfare wise as y people suffering x harm.

(Not to mention that when normalized across people, utils are meant to provide direct and simple mathematical comparisons. In this case, it doesn't really matter how the normalization occurs as the inequality stands for any epsilon of dust-speck harm greater than zero.)

Polling people to find if they will take a dust speck grants an external harm to the torture (e.g., mental distress at the thought of someone being tortured). Since they would prefer the dust speck, this indicates that they find the thought of someone being subject to 50 years of torture (Harm a) to be of more harm than a dust speck (Harm b). Harm a > Harm b, so n * Harm a > n * Harm b, and it doesn't even matter what Harm a or Harm b is, nor the additional nondistributed harm of the actual torture.
How could this be tricked? Replace the dust speck with the least harm greater than the distress at the thought of someone being tortured for 50 years, say, the thought of someone being tortured for 51 years.

[eddie]

There are no natural utility differences that large. (Eliezer, re 3^^^3)

You've measured this with your utility meter, yes?

If you mean that it's not possible for there to be a utility difference that large, because the smallest possible utility shift is the size of a single particle moving a planck distance, and the largest possible utility difference is the creation or destruction of the universe, and the scale between those two is smaller than 3^^^3 ... then you'll have to remind me again where all these 3^^^3 people that are getting dust specks in their eyes live.

If 3^^^3 makes the math unnecessary because utility differences can't be that large, then your example fails to prove anything because it can't take place. For your example to be meaningful, it is necessary to postulate a universe in which 3^^^3 people can suffer a very small harm, which necessarily implies that yes, in fact, it is possible in this hypothetical universe for one thing to have 3^^^3 times the utility of something else. At which point, in order to prove that the dust specks outweigh the torture, you will now have to shut up and multiply. And be sure to show your work.

Your first task in performing this multiplication will be to measure the harm from torture and dust specks.

Good luck.

[Caledonian]

If you don't accept the additivity of harm, you accept that for any harm x, there is some number of people y where 2^y people suffering x harm is the same welfare wise as y people suffering x harm.

No. I can imagine non-additive harm evaluation systems where that is not the case.

Even in the limited subset of systems where that IS the case, so what?

[Unknown]

Although the argument didn't depend on harm adding linearly in any case, it is true that two similar harms to two different people must be exactly twice as bad as one harm to one person.

Many people on this thread have already given the reason: how bad it is that someone is killed, or tortured, or dust specked, obviously does not depend on how many other people this has happened to. Otherwise death couldn't be such a bad thing, since it has already happened to almost everyone.

[Ben Jones]

Sean, say you're one of the 3^^^3 voters. A vote one way potentially has torture on your conscience, as you say, but a vote the other way potentially has 3^^^3 dust specks on your conscience - by your definition a much greater sin. Square one - shut up and vote!

I am aware that a billion people getting punched in the face appears to aggregate to a greater harm than one person being tortured for 50 years. (I should say that when I ask my intuition what it thinks about the number 1,000,000,000, it says 'WTF?', so it's not coming from there....) However, if I was one of the billion and had to cast my vote, I'd ask for the punch, and hope you would too. (I am also confident that it would be because it was the right thing for me to do, and not to alleviate any potential guilt.) What does this say about intuition and additivism?

[Unknown]

Ben, that's not about additivism, but indicates that you are a deontologist by nature, as everyone is. A better test: would you flip a lever which would stop the torture if everyone on earth would instantly be automatically punched in the face? I don't think I would.

[Bob]

Joseph et al, I appreciate your thoughts. I think, though, that your objections keep coming back to "it's more complicated." And in reality it would be. But the simple thought experiment suggests that any realistic derivative of the specks question would likely get answered wrong because our (OUR!) intuition is heavily biased toward large (in aggregate) distributed harm. It appears that we personalize individual harm but not group harm.

Ben, I assume that we would all vote that way, if only because the thought of having sentenced someone to torture would be more painful than the punch. But you've changed the question. How would you choose if faced with deciding that either 1) every person on earth receives a painful but nonfatal punch in the face or 2) one random person is chosen to be tortured? That's the specks question.

[Robin Hanson]

If politics is the mind killer, morality is at least the mind masher. We should probably only talk about morality in small doses, interspersed with many other topics our minds can more easily manage.

[Salutator]

@Sean
If your utility function u was replaced by 3u,there would be no observable difference in your behavior. So which of these functions is declared real and goes on to the interpersonal summing? "The same factor for everyone" isn't an answer, because if u_you doesn't equal u_me "the same factor" is simply meaningless.

@Tombs

A < B < C < D doesn't imply that there's some k such that kA>D

Yes it does.

I think you're letting the notation confuse you. It would imply that, if A,B,C,D where e.g. real numbers, and that is the context the "<"-sign is mostly used in. But Orders can exist on sets other then sets of numbers. You can for example sort (order) the telephone book alphabetically, so that Cooper < Smith and still there is no k so that k*Cooper>Smith.

@most people here:
A lot of confusion is caused by the unspoken premise, that a moral system should sort outcomes rather then actions, so that it doesn't matter who would do the torturing or speck-placing. Now for Eliezer that assumption is de fide, because otherwise the concept of a friendly AI (sharing our ends and choosing the unimportant-declared means with its superior intelligence) is meaningless. But the assumption contradicts basically everyones intuition. So why should it convince anyone not following Eliezer's religion?

[Eisegetes]

Ben, that's not about additivism, but indicates that you are a deontologist by nature, as everyone is. A better test: would you flip a lever which would stop the torture if everyone on earth would instantly be automatically punched in the face? I don't think I would.

I'm fairly certain I would pull the lever. And I'm certain that if I had to watch the person be tortured (or do it myself!) I would happily pull the lever.

It was this sort of intuition that motivated my earlier question to Eliezer (which he still hasn't responded to). I'd be interested to hear from any of the people advocating torture over specks, though: would you all be willing to personally commit the torture, knowing that the alternative is either (a) a punch in the face (without permanent injury/pain/risk of death) to every human on the planet, or (b) in a universe with 3^^^3 people in it, a speck in everyone's eye?

I suspect that the most people would refuse to pick up the blowtorch in either case. Which is very important -- only some very exotic, and very implausible, metaethical theories would casually disregard the ethical intuitions of the vast majority of people.