Boo Dark Powers

To mark Halloween, I hereby endorse mild regulation to discourage the summoning of dark supernatural powers which might destroy us all. Seriously. Anders Sandberg:

The San Marino scale for assessing the significance of messages that could be received by aliens … is a scale from 1 to 10 based on how easily detected the transmission is, and how much information about humanity it contains. Past deliberate transmissions have managed to reach 8, ‘far reaching’. Even planetary radar manages to reach a level of 6, ‘noteworthy’.

So, how far does these [recent] advertisements go? The Deep Space Communications dish is apparently 5 meter in diameter … The Dwingeloo telescope used for the Klingon invitation is a bit larger, 25 meters, but again the transmission power is unclear. … The level 8 signal from the Arecibo used about half a million Watts of power, making it far more powerful than anything these telescopes can achieve. … It hence seems that compared to past messages these adverts are unlikely to matter: we are already transmitting other messages, these just add to the choir. …

How much should small groups of people be allowed to risk the future of humanity with low probability? Not everyone agrees that the risk from alien contact is negligible: even a very low probability times a great harm can be relevant. … Should we be equally concerned with occultists trying to summon world-changing supernatural powers? There are probably many more people today who believe in supernatural entities than mere aliens, and that some interactions with them could be harmful. Yet there are no attempts at formulating risk scales for ritual magic. … Even if we were to analyse them rationally, we need to have an ‘ultraviolet cut-off’ for the infinite number of possible-yet-exceedingly-unlikely possibilities we could worry about. How to rationally decide on this cut-off seems problematic.

Even if the risk from recent ads is only 1/1000 of the risk of other prior transmissions, since I’m not sure how big was their risk I’m reluctant to conclude that the smaller ones are “unlikely to matter.” Perhaps 1/1000 of a bigger risk is still big enough to matter. So I’d support mild regulation to discourage such transmissions, big and small.

On which risks should matter, I’d prefer to use odds from a prediction market to decide. And my guess is that they’d give non-trivial (well over one in a million) odds both that our transmissions might alert hostile aliens, and that one could “summon world-changing supernatural powers.” So I’d also support mild regulation to discourage actions that might risk our descruction by terrible supernatural dark lords. Perhaps that sounds silly, but to disagree you either have to support our destruction by dark powers, or disagree that policy should be based on market odds.

Added 1Nov: Many of you talk as if one would have to go through a phone-book length list of potential disasters before one came to this one.  But surely this is one of the top ten disasters of concern in fiction, and would be similarly high in surveys.  Yes, you might not take it seriously, but others clearly do. The question is: what probability to use in policy when people disagree on probabilities. Surely the right answer is some sort of weighed average of opinion, and while willingness to bet is a great way to weight opinions, surely most any neutral approach will give a high enough probability of disaster here to justify substantial concern.  And this justifies “mild” regulation, which looks for the easy wins of large harm reduction at low cost.  Now it might turn out that there just don’t exist any feasible mild regulations, but we should at least consider some options before drawing that conclusion.

GD Star Rating
Tagged as: , , , ,
Trackback URL:
  • Sauron

    Then maybe policy shouldn’t be based on market odds…

  • wophugus

    Infringing on the ability of people to practice their religion freely creates unhappiness and dis-utility. That has to be weighed against the threat of magic.

    The possibility of summoning hostile aliens who will create maximum dis-utility has to be weighed against the possibility of summoning advanced friendly aliens who can create huge gains in utility.

    We also need to consider the extent to which we value the preferences and happiness of aliens and demons.

  • Sauron, if you disagreed with the market odds you should expect to gain by betting at those odds.

    wophugus, every regulation of any sort has to be weighed against limiting freedom, including freedom of religion. Hence the keyword “mild.”

    • Taxman

      Sauron should expect to be paid off should hostile aliens or supernatural beings arrive on and harm earth?

      • Taxman

        To clarify; My unspoken point is that I would expect significant liquidity and counterparty complications should Sauron make good on such a bet.

      • IVV

        Taxman, read your LOTR. Sauron DOES receive a payoff should supernatural beings arrive on and harm the earth.

    • There are related events in which bets could be paid off whose odds are highly correlated with this extreme event.

  • William H. Stoddard

    This just seems to be Pascal’s Wager. I dismiss appeals to Pascal’s Wager out of hand; they prove too much too easily. If there’s not actual evidence, but just an appeal to “Well how can you know for sure”—well, I class that as cheap talk.

  • Samantha Atkins

    Market odds? Do you mean the non-verifiable opinion of a variety of people about the validity of the threat and the utter guesswork as to threat level if any were to materialize? Do you seriously wish to limit human freedom, including freedom to be an idiot in one’s own way, on the basis of the mere opinions and guesswork of other idiots? If so then you are demonstrably much more of a danger than any of these fantastic threats.

    • Robert Koslover

      Samantha, I think you are correct, although I don’t think Robin is a danger. On the other hand, I do consider the odds that Robin is a danger to be greater than the odds that any attempted communication with supernatural beings will: (1) actually succeed, and (2) bring down their wrath upon us. As such, and by using Robin’s own argument (assuming I interpret it correctly), it would be prudent to put “mild” restrictions upon him. Heh. 🙂

  • Daniel Yokomizo

    Doesn’t this argument also apply to SIAI’s “Scary Idea”?

  • disagree you either have to support our destruction by dark powers, or disagree that policy should be based on market odds.

    What about enforcement costs?

    By mild regulation, I presume you mean something like a Pigouvian tax of, say for the sake of example, a penny per spell. But since the odds are so small, isn’t there likely to be quite a high probability that the enforcement costs will outweigh any gain in expected utility?

    And those enforcement costs could have been spent on other things, like subsidising spells that have a tiny chance of granting us all immortality.

  • I think the problem is that there’s so much we need to do at present that we run out of energy by the time we get to policy initiatives like rationally designing anti-occult regulation.

  • Telnar

    Market odds are problematic for very rare events — especially those which aren’t forecast to happen right away. Let’s say for arguments sake that I have very strong evidence that an event which is estimated as having probability of 1 in 10^4 really has a probability of at most 1 in 10^10. The market has overestimated the probability by a factor of 1 million — a huge error. Now let’s look at what bets I can make to profit from that error.

    If I bet against the rare event, I expect a profit of 10 cents for each thousand dollars I’m willing to commit to the wager. If I need to commit to money for a year to see the result of the wager, then the margin rules of the site where I’m wagering and the opportunity cost of tying up my capital are going to have a much stronger influence on whether I bet than how confident I am that the estimated probability is wrong.

    Things are a little better in the other direction, since if the market is estimating odds of one in 10^10, I can bet a penny to collect $100 million one time in ten thousand — easily correcting the odds down if they were mistakenly at 10^10 to 1 when the true odds are 10^4 to 1, but how could the odds have been that long? Those odds being available to bet implies that someone else has nothing better to do with $100 million that see if he can collect my penny. Even if the true odds are 10^100 to 1, why would anyone commit the money necessary to maintain bettable odds of 10^10 to 1?

  • Samantha, it is the rare threat whose odds would be “verifiable”.

    Daniel, sure the odds of runaway superintelligence would also be high enough to support mild regulation to prevent.

    Andy and Hopefully, I leave open the question of what mild regulations might have low enough costs to be worth the mild benefit. If all other reasonable options have excessively high fixed costs, no regulation may be the best choice.

    Telnar, there are ways of designing prediction markets to reduce that problem.

  • Telnar


    I’d be interested in what those design approaches are since I see an inherent incompatibility between giving potential winners of long odds bets confidence they can collect and not excessively tying up the capital of those offering long odds. A margin requirement that created confidence that winners of long odds bets could collect would need to be enforced on the assumption that the odds offered were radically wrong (which is what the long shot bettors believe). That forces the other side to get a poor return on locked up capital (which would yield at best a safe asset return while pledged as margin).

    Even if we handwave margin requirements away and allow someone to post $1 million to secure 1 million different $1 bets (on different propositions) each at 1 million to 1 odds (implicitly using their own odds evaluation to set the required margin and accepting the risk that more than one bet will win), we’re still ignoring the information costs. A mildly risk averse person risking $1 million will likely need to be more confident than someone risking $1 even though they are both betting on the same proposition. That confidence requires time and effort to obtain, and I doubt that it can be achieved cost effectively.

    Unlike propositions at close to even odds, where one’s research costs can usually be recovered by making a sufficiently large wager, betting on the favorite at long odds has an upper bound to its profit potential set by one’s ability to pay off a losing bet. At 10^6 to 1 odds, that profit potential is quite low. Even Bill Gates (net worth $53 Billion) could make at most $53,000 by risking all of his wealth. It’s hard to imagine that he doesn’t have better ways to spend his time, effort, and opportunity cost of capital.

    • I’m confident that even if such problems could be completely overcome the market odds of dark lords would be over one in a million. And that’s all I need to make the rest of the post work.

      • Telnar

        Fair enough, but unless this problem can be overcome in practice, it bounds the class of problems for which prediction markets are a good solution. I suspect that the practical limit created by this type of problem will limit prediction markets to problems to problems where the odds of the long shot bet winning are of at least the same order of magnitude as the time value of money for the period required for a resolution (so, if the relevant interest rate is 8%, then a wager which takes one year to resolve has a significantly better shot of a good market prediction if the true probability is 10% than if it’s 1%).

    • Telnar,

      You seem to be presaging some possibly interesting results relating the long-odds limits of prediction markets to the risk free rate of return. Comparing high interest rates to low interest rates, high rates rationally lead to choosing earlier payoffs than later ones. Perhaps a similar effect exists in prediction markets, where the risk free rate puts an effective floor on the length of odds to which the market will be effective.

  • The big hurdle here is judging who are the dark vs. light supernatural powers. Remember in Ghostbusters that regulator from the EPA who forced the shutdown of their holding chamber because he thought they were up to no good.

    But they were using the supernatural or paranormal in a good way, against the bad ghosts, and his regulation let loose a bunch of them all over New York City.

    As with many other decisions, we can’t trust that regulators of the supernatural will be able to tell the angels apart from the demons.

  • ben

    shouldn’t we include hostile aliens in our utility calculations. maybe hostile aliens destroying us will be a net benefit to the universe….

    • IVV

      As a human, I’m siding with the humans. Any universe without us is worse off than any universe with us, from my point of view. Personally, I discount the preferences of aliens.

      Now, if I am given the opportunity to transform my consciousness into another being, then the entire space of possible beings I could emulate becomes the group I will attempt to maximize utility for. Em formulation, metamorphosis, transcendence, holy transmogrification, zombie infection, pod people, polymorph spells, the vampiric embrace–any and all such possibilities will affect my personal calculation of the universe with maximum utility.

  • wophugus

    “every regulation of any sort has to be weighed against limiting freedom, including freedom of religion. Hence the keyword ‘mild.'”

    I think you are operating under some sort of logical fallacy, there? It is possible that eliminating the most dangerous demonic ritual — just that one — destroys more happiness by infringing on the freedom of the occultist than the happiness we expect to capture by averting the possibility of human destruction (which, I guess, is the amount of happiness that will probably be generated over the lifespan of the human race absent the ritual working times the probability that the ritual will work). It could be the case that, once you weigh the value of freedom vs. the value of regulation, on the margins the coin comes up “don’t regulate.”

    No offense, because I really value your blog, but I think you do this a lot. You tend to approve of actions that marginally lower the risk of human extinction without weighing the cost to see if they are worth it. You should not do that. The human race and its successors will most likely go extinct, no matter how careful we are. Averting one possible path towards human extinction does not create infinite happiness, and it is not worth sacrificing any quantum of happiness to do it.

  • hf

    agnostic kind of pointed this out, but the probability of ritual magic destroying humanity does not exceed the probability of mild regulation destroying humanity. For it is written, “take your fill and will of love as ye will, when, where and with whom ye will! But always unto me. If this be not aright…if the ritual be not ever unto me: then expect the direful judgments of Ra Hoor Khuit!” Clearly the vile heresies of celibate ritual-workers place us all in grave danger.

    And I know I’d get pissed off, as a Dark Lord, if society discouraged my cultists from honoring me and my buddy Cthulhu.

  • John Maxwell IV

    Would such a prediction market pay out in the face of catastrophe brought on by dark forces? Would such a prediction market give buyers many data points to learn whether they are the sort that is good at playing the market or not?

    What if dark rituals are like pot in the sense that regulating them makes them interesting to adolescent males?

  • I think the argument is that at some point the costs of simply considering or trying to calculate the risks of low probability events start to outweigh any expected benefit from that consideration.

    Prediction markets themselves may be relatively cheap and even if we pretended everyone approved of allowing prediction markets to control our responses the mere need for us to consider how we might connect up the outcomes of the market and our actions will eventually start to cost more than any benefit the market can provide.

    Also I think your faith in prediction markets for these kind of low probability world shaking events is misplaced. As others have pointed out these events might well destroy our concern for profit or loss in the market. If aliens show up we probably either get vaporized or an unimaginably large influx of technology totally altering our economic system, currency values and even our way of life. If the aliens hand us holodeck technology and robotic farmers/holodeck repairmen everyone can shrug and abandon their money entirely.

    Moreover, the wide agreement that these events are so very unlikely means that the expressive appeal of betting on outcomes is quite likely to outweigh and kind of information aggregation. As far as alien visitation goes there is really no hidden info for the prediction markets to aggregate.

  • I just added to the post.

  • I guess the obvious mild and non-infringing approach would be to have an ‘unsafe magick tax’ on ritual supplies unless you show that you are a certified white magician?

    Maybe one could also use markets to do the certification: people bet on different kinds of magicians whether they will cause trouble or not, and over time this would converge to a sensible certification (problem: in real life most trouble caused by magicians seem to be due to their personality and beliefs, rather than their putative magical powers.)

    From a rationalist perspective it seems sensible to have information markets about supernatural and religious predictions since it is very likely going to transfer money to the rational people from naive believers. However, “true believers” are unlikely to regard information markets as appropriate, since they consider their belief to be something different than just a probability estimate. So we would likely get a bit of a biasing effect.

  • David C

    “Many of you talk as if one would have to go through a phone-book length list of potential disasters before one came to this one. But surely this is one of the top ten disasters of concern in fiction, and would be similarly high in surveys.” – Robin Hanson

    This isn’t a single disaster. This is a category of disasters. The number of different rituals that could potentially lead to dark lords destroying the planet is large enough that it may as well be infinite. Determining a set of criteria to define what is a ritual to summon evil demons and what is simply praying to a loving God using market forces is a big hurdle to overcome.

  • Market Odds keep lottery ticket sellers rolling in money. Has any work been done to see if market odds systematically fail when the true odds get very long?

  • I think you’re overlooking the possible benefits of summoning eldritch powers from long-sealed dimensions into our own world. Known benefits of dark rituals include superior strength and stamina, longevity, immortality, and all sorts of economically useful magical powers. If we refrain from contacting the Unspeakable Ones for fear of “destroying the world” – which, despite its many appearances in fiction, has oddly enough yet to happen, and frankly I feel that those who speculate about this are just immature – then what benefits do we forgo?

    • I should also add that I am not concerned about values erosion from having the world ruled by Cthulhu – so long as there’s more than one Great Old One consuming our dreams and flesh, market economics will take over and that’s what’s really important to me.

      • IVV

        Belief in the Invisible Tentacle of Sleeping Markets?

        Surely this is related to the Mutant 20-Tentacled Octopus of Fail which makes up the mortgage market these days.

      • Anonymous from UK

        I love a good bit of sarcasm.

    • Well if these dark powers respected property rights, gaining money by selling the use of their great powers, and then buying what they want to satisfy their values I don’t share, I wouldn’t have much to complain about. Those who share my values would on net be richer and better off.

      • Khoth

        “Well if these dark powers respected property rights…”

        I had always wondered why vampires couldn’t enter a house uninvited. Now I know.

  • Jordan

    Seems like a waste of time since virtually all similar affairs have had not just non-world-ending results, but no notable results whatsoever. Absence of evidence is evidence of absence. If you disagree, I propose people be discouraged from exercise because each calorie of consumption risks a chain reaction of heat energy that will melt the Earth and kill all inhabitants. You should start collecting additional taxes from gyms immediately, and I humbly accept a portion of the proceeds for coming up with the idea and warning you of the threat.

    I’m a big believer in republic; that is, that all people everywhere cannot individually be well-versed in every issue everywhere at all times. Representation is necessary in the civilized world, and unless your weighed average gives thousands of times the weight to astronomers, mathematicians, and physicists – that is, one that more or less obliterates the lay man’s opinion – you’d have an uphill battle getting me to respect the odds you came up with.

  • Moeticist

    A market system is fairly good at finding (capitalist) valuations, given that enough data exists between the participants to resolve a probability curve to a statistically significant degree. Real-world contracts almost universally accept that this is not true of events far enough down the tail. You’re essentially advocating that there be a market in force majeure – and I can’t think of a single major investor who wouldn’t laugh that idea out of the building.