Tag Archives: Sacred

Beware Mob War Strategy

The game theory is clear: it can be in your interest to make threats that it would not be in your interest to carry out. So you can gain from committing to carrying out such threats. But only if you do it right. Your commitment plan must be simple and clear enough for your audience to see when it applies to them, how it is their interest to go along with it, and that people who look like you to them have in fact been consistently following such a plan.

So, for example, it probably won’t work to just lash out at whomever happens to be near you whenever the universe disappoints you somehow. The universe may reorganize to avoid your lashings, but probably not by catering to your every whim. More likely, others will avoid you, or crush you. That’s a bad commitment plan.

Here’s a good commitment plan. A well-run legal system can usefully deter crime via committing to consistently punish law violations. Such a system clearly defines violations, and shows potential violators an enforcement system wherein a substantial fraction of violations will be detected, prosecuted, and punished. Those under the jurisdiction of this law can see this fact, and understand which acts lead to which punishments. Such acts can thus be deterred.

Here’s another pretty good commitment plan. The main nations with nuclear weapons seem to have created a mutual expectation of “mutually assured destruction.” Each nation is committed to responding to a nuclear attack with a devastating symmetric attack. So devastating as to deter attack even if there is a substantial chance that such a response wouldn’t happen. This commitment plan is simple, easy to understand, clearly communicated, and quite focused on particular scenarios. So far, it seems to have worked.

Humans are often willing to suffer large costs to punish those who violate their moral rules. In fact, we probably evolved such moral indignation in part as a way to commit to punishing violations of our local moral norms. In small bands, with norms that were stable across many generations, members could plausibly achieve sufficient clarity and certainty about norm enforcement to deter violations via such threats. So such commitments might have had good plans in that context.

But this does not imply that things would typically go well for us if we freely indulged our moral indignation inclinations in our complex modern world. For example, imagine that we encouraged, instead of discouraged, mob justice. That is, if we encouraged people to gossip to convince their friends to share their moral outrange, building off of each until they chased down and “lynched” any who offended them.

This sort of mob justice can go badly for a great many reasons. We don’t actually share norms as closely as we think, mob members are often more eager to show loyalty to each other than to verify accusation accuracy, and some are willing to make misleading accusations to take down rivals. More fundamentally, we might say that mob justice goes bad because it is not based on a good commitment plan. Observers just can’t predict mob justice outcomes well enough for it to usefully encourage good behavior, at least compared to a formal legal system.

Now consider the subject of making peace deals to end wars. Such as the current war between Russia and Ukraine. An awful lot of people, probably a majority, of the Ukrainian supporters I’ve heard from seem to be morally offended by the idea of such a peace deal in this case. Even though the usual game theory analyses of war say that there are usually peace deals that both sides would prefer at the time to continued war. (Such deals could focus on immediately verifiable terms; they needn’t focus on unverifiable promises of future actions. In April 2022 Russia and Ukraine apparently had a tentative deal, scuttled due to pressure from Ukrainian allies.)

Many of these peace deal opponents are willing to justify this stance in consequentialist terms: they say that we should commit to not making such deals. Which, as they are eager to point out, is a logically coherent stance due to the usual game theory analysis. We should thus “hold firm”, “teach them a lesson”, “don’t let them get away with it”, etc. All justified by game theory, they say.

The problem is, I haven’t seen anyone outline anything close to a good commitment plan here. Nothing remotely as clear and simple as we have with criminal law, or with mutually assured destruction. They don’t clearly specify the set of situations where the commitment is to apply, the ways observers are to tell when they are in such situations, the behavior that has been committed to there, or the dataset of international events that shows that people that look like us have in fact consistently behaved in this way. Peace deal opponents (sometimes called “war mongers”) instead mainly just seem to point to their mob-inflamed feelings of moral outrage.

For example, some talk as if we should just ignore the fact that Russia has nuclear weapons in this war, as if we have somehow committed to doing that in order to prevent anyone from using nuclear weapons as a negotiating leverage. The claim that nations have been acting according to such a commitment doesn’t seem to me at all a good summary of the history of nuclear powers. And if the claim is that we should start now to create such a commitment by just acting as if it had always existed, that seems even crazier.

If we have not actually found and clearly implemented a good commitment plan, then it seems to me that we should proceed as if we have not made such a commitment. So we must act in accord with the usual game theory analysis. Which says to compromise and make peace if possible. Especially as a way to reduce the risk of a large nuclear war.

The possibility of a global nuclear war seems a very big deal. Yes, war seems sacred and that inclines us toward relying on our intuitions instead of conscious calculations. It inclines us toward mob war strategy. But this issue seems plenty important enough to justify our resisting that inclination. Yes, a careful analysis may well identify some good commitment plans, after which we could think about how to move toward making commitments according to those plans.

But following the vague war strategy inclinations of our mob-inflamed moral outrage seems a poor substitute for such a good plan. If we have not yet actually found and implemented a good plan, we should deal with a world where we have not made useful commitments. And so make peace, to avoid risking the destructions of war.

GD Star Rating
loading...
Tagged as: , , ,

Beware Profane Priests

We humans evolved a way to take some of the things that are important to us, and bind our groups together by seeing those things as “sacred”. That is, by seeing them in the same way, via always seeing them from a distance. Such things are seen more abstractly and intuitively, with less conscious calculation, and less attending to details. Sacred things are idealized, and not to be mixed with or traded off against other things. Sacred thinking can be less competent, but induces more effort, and can keep us from being overwhelmed by strong passions.

Let us call the experts associated with a sacred area “priests”. The possibility of priests raises two issues for the sacred. First, if ordinary people saw a sacred area as one where they could personally gain expertise, and where they need to think to judge the relative expertise of others, this would seem to induce conscious calculation about the details of this sacred topic, which is a no-no. Second, those who are most expert would think a lot about the topic, and often see it up close, which would make it harder for them to see it a sacred.

Humans seem to solve the first issue by treating all sacred topics as being at one of two extremes. At one extreme, e.g., medicine, there are highly expert sacred priests, which they rest of us are not to second guess nor evaluate. At the other extreme, e.g., politics or friendship, expertise via thinking is seen as not possible, making everyone’s opinions nearly as good as anyone’s opinions. In neither case does thinking help ordinary people much, either to form opinions or to choose experts.

On the second issue, experts who only rarely directly confront the most sacred versions of their subject up close, like soldiers, police, or doctors, can drill and practice in a far mode, so that they can perform well intuitively and without much thought in the rare big stakes cases. But what about the other priests, who confront their sacred subjects more often?

When we think about this question in a sacred mode, intuitively and using a few abstract associations, our minds usually conclude that as the sacred is good and ideal, contact with it makes people more good and ideal. Thus we can trust priests to act in our collective interest. But the norm that the rest of us are not to judge such experts, and are to defer strongly to their judgement, gives them a lot of collective discretion. And it seems to me that near mode engagement with the topic means we can’t count much on their reverence for it to restrain them from from using their discretion for selfish advantage.

Thus in fact priests will often act profanely, a fact that the rest of us are often unwilling to see. Beware profane priests.

Added 26Sep: Someone suggested we trust experts on the sacred due to their sacrificing more to take such jobs, I did 16 Twitter polls on 16 kinds of jobs. Here are median estimates of “% of value which workers of that type sacrifice on average to do their job”:

GD Star Rating
loading...
Tagged as: ,

Sacred Inquiry

The reason I first started to study the sacred was that “sacred cows” kept getting in my way; our treating things as sacred often blocks sensible reforms. But now that I have a plausible theory of how and why we treat some things as sacred, I have to admit: I too treat some things as sacred. Maybe I should learn to stop that, but it seems hard. So perhaps we should accept the sacred as a permanent feature of human thought, and instead try to change what we see as how sacred or how exactly we do that.

So it seems worth my trying to describe in more detail how I see something as sacred, not just habitually but even after I notice this fact. In this post, that thing will be: intellectual inquiry. In this post I’ll mostly try to describe how I revere this, and not so much ask whether I should.

All the thinking and talking that happens in the world helps us to do many things, and to figure out many things. And while some of those things are pretty concrete and content-dependent, others are less so, helping us to learn more general stuff whose usefulness plausibly extends further into the future. And this I call “intellectual progress”.

In general, all of the thinking and talking that we do contributes to this progress, even though it is done for a wide variety of motives, and via many different forms of social organization. I should welcome and celebrate it all. And while abstractly, I do, I notice that, emotionally, I don’t.

It seems that I instead deeply want to distinguish and revere a particular more sacred sort of thinking and talking from the rest. And instead of assuming that my favored type is just very rare, hardly of interest to anyone but me, I instead presume that a great many of us are trying to produce my favored type, even if most fail at it. Which can let me presume that most must know how to do better, and thus justify my indignant stance toward those who fail to meet my standards.

This sort of thinking and talking that I revere is that which actually achieves substantial and valuable progress in abstract understanding, and is done in a way to effectively and primarily achieve this goal. Thus I see as “profane” work that appears to be greatly influenced by other purposes, such as showing off one’s impressiveness, or assuring associates of loyalty.

That is, I have a sacred purity norm, where I don’t like my pure sacred stuff mixed up with other stuff. Good stuff not only has to achieve good outcomes, it also has to be done the right way for the right reasons. I tend to simplify this category and its boundary, and presume that it can be distinguished clearly. I feel bound to others who share my norms, even if I can’t actually name any of them. I don’t calculate most of this; it instead comes intuitively, and seems aesthetically elegant. And I can’t recall ever choosing all this; it feels like I was always this way.

Now on reflection this has a lot of specific implications re what I find more sacred or profane, as I have a lot of beliefs about which intellectual topics are more valuable, and what are more effective methods. And I’ll get to those soon here.

But first let me note that while many intellectuals also see their professional realm as sacred, and have many similar sacred norms about how their work should be done, most of them don’t apply such norms nearly as strongly to their personal lives. In contrast, I extend this to all my thinking and talking. That is, while I’m okay with engaging in many kinds of thinking and talking, I want to sharply distinguish some sacred versions, where all these sacred norms apply, and try to actually use them often in my personal life.

Okay, I can think of a lot of specific implications this has for what I respect and criticize. The following is a somewhat random list of what occurs to me at the moment.

For example, I take academic papers to be implicitly claiming to promote intellectual progress. This implies that they should try to be widely available for others to critique and build on. So I dislike papers that are less available, or that use needlessly difficult languages or styles. Or that aren’t as forthcoming or concise as they could re what theses they argue, to allow readers to judge interest on that basis. I dislike intentional use of vague terms when clearer terms were available, and switching between word meanings to elude criticism.

I feel that a paper which cites another is claiming that it got some particular key input from that other paper, and a paper that cites nothing is claiming to have not needed such inputs. So I disapprove of papers that fail to cite key inputs, or that substitute a more prestigious source for the less prestigious source from which they actually got their input.

I see a paper on a topic as implicitly claiming that the topic is some rough approximation to the best topic they could have chosen, and a paper using a method that some rough approximation to the best method. So it bothers me when it seems obvious the topic isn’t so good, or when the method seems poorly chosen. I’m also bothered when the length of some writing seems poorly matched to the thesis presented. For example, if a thesis could have been adequately argued in a paper, then I’m bothered if its in a book with lots of tangential stuff added on to fill out the space.

I find it profane when authors seem to be pushing an agenda via selective choice of arguments, evidence, or terminology. They should acknowledge weak points and rebuttals of which they are aware without making readers or critics find them. I dislike when authors form mutual admiration societies designed to praise each other regardless of the quality of particular items. That is, I find the embrace of bias profane. Which maybe shouldn’t be too surprising given my blog name.

Now I have to admit that it isn’t clear how effective are these stances toward promoting this sacred goal of mine. While they might happen to help, it seems more plausible that they result more from a habit of treating this area as sacred, rather than from some careful calculation of their effects on intellectual progress. So it remains for me to reconsider my sacred stances in light of this criticism.

GD Star Rating
loading...
Tagged as: ,

Sacred Distance Hides Motives

My book with Kevin Simler describes many hidden human motives, common in our everyday lives. But that raises the question: how exactly can we humans hide our motives from ourselves?

Consider that we humans are constantly watching and testing our and others’ words and deeds for inconsistency, incoherence, and hypocrisy. As our rivals are eager to point out such flaws, we each try to adjust our words and deeds to cut and smooth the flaws we notice. Furthermore, we habitually adjust our words and deeds to match those of our associates, to make remaining flaws be shared flaws. After a lifetime of such smoothing, how could much personal incoherence remain?

One way to keep motives hidden is to hide your most questionable actions, those where you feel you least control or understand them. If you can’t hide such actions, then try not to make strong claims about related motives. And I think we do follow this strategy for our strongest feelings, such as lust, envy, or social anxiety. We often try to hide such feelings even from ourselves, and when we do notice them we often fall silent; we fear to speak on them.

How easy it is to check deeds and words for coherence depends in part on how dense and clear are their connections. And as all deeds are concrete, and as concrete words tend to be clearer and more densely connected, it seems easier to check concrete priorities, relative to abstract ones.

For example, it is easiest to check the motives that lead to our conscious, often written, calculations of detailed plans. Our time schedules, spatial routes and layouts, and our spending habits are often visible and full of details that make it hard to hide priorities. For example, if you go out of your way to drive past the home of your ex  on your way home from work, it will be hard to pretend you don’t care about her.

We have more room to maneuver, however, regarding our more hidden and infrequent concrete choices. And when we are all in denial in similar ways on similar topics, then we can all be reluctant to “throw stones” at our shared “glass houses”. This seems to apply to our hidden motives re schools and medicine, for example; we apparently all want to pretend together that school is for learning job skills and hospitals are for raising health.

Compared to our concrete priorities, our abstract priority expressions (e.g, “family is everything”) are less precise, and so are harder to check against each other for consistency. And abstract expressions can be even harder to check against concrete actions; large datasets of deeds may be required to check for such coherence.

We ground most abstract concepts, like “fire”, “sky”, “kid” or “sleep”, by reference to concrete examples with which we have had direct experience. So when we are confused about their usage, we can turn to those examples to get clear. But we ground other more “sacred” abstract concepts, like “love”, “justice”, or “capitalism”, more by reference to other abstract concepts. These are more like “floating abstractions.” And this habit makes it even harder to check our uses of such sacred concepts for coherence.

This potential of abstract concepts to allow more evasion of coherence checking is greatly enhanced by the fact that our brains have two rather different systems for thinking. First, our “near” system is designed to look at important-to-us close-up things, by attending to their details. This system is better integrated with our conscious thoughts. For example, we often first do a kind of calculation slowly and consciously, and then later by habit we learn to do such calculations unconsciously. This integration supports coherence checking, as we can respond to explicit challenges by temporarily returning to conscious calculation, to find explanations for our choices.

Our “far” system, in contrast, is designed to look at less-important-to-us far-away things, about which we usually know only a few more abstract descriptors. This system uses many opaque quick and dirty heuristics, including intuitive emotional and aesthetic associations, crude correlations, naive trust, and social approval. If someone else is using this system in their head to think about a topic, and then you use this system in your head to try to check their thinking, you will have a hard time judging much more than if your system gives the same answers as theirs. If you get different answers, it will be hard to say exactly why.

As our minds tend to invoke our far systems for thinking about more abstract topics, that makes it even harder to check abstract thoughts for coherence. But, you might respond, if that system is designed for dealing with relatively unimportant things, won’t the other near system get invoked for important topics, limiting this problem of being harder to check coherence to unimportant topics?

Alas, no, due to the sacred. Our sacred things are our especially important things, described via floating abstractions, where our norm is to think about them only using our far systems. We are not to calculate them, consider their details, or mix them with or trade them off against other things. Our intuitions there are sacred, and beyond question.

Making it hard to check the coherence of related deeds and words. The main thing we can do there is to intuit our own answer and compare it to others’ answers. If we get the same answers, that confirms that they share our sense of the sacred, and are from our in-group. If not, we can conclude they are from an out-group, and thus suspect; they didn’t learn the “right” sense of the sacred.

And that’s some of the ways that our minds tend to hide our motives, even given the widespread practice of trying to expose incoherence in rivals’ words and deeds. Floating abstractions help, and the sacred helps even more. And maybe we go further and coordinate to punish those who try to expose our sacred hypocrites.

Note that I’m not claiming that all these habits and structures were designed primarily for this effect of making it harder to check our words and deeds for coherence. I’m mainly pointing out that they have this effect.

GD Star Rating
loading...
Tagged as: ,

We See The Sacred From Afar, To See It Together

I’ve recently been trying to make sense of our concept of the “sacred”, by puzzling over its many correlates. And I think I’ve found a way to make more sense of it in terms of near-far (or “construal level”) theory, a framework that I’ve discussed here many times before.

When we look at a scene full of objects, a few of those objects are big and close up, while a lot more are small and far away. And the core idea of near-far is that it makes sense to put more mental energy into analyzing each object up close, objects that matters to us more, by paying more attention to their detail, detail often not available about stuff far away. And our brains do seem to be organized around this analysis principle.

That is, we do tend to think less, and think more abstractly, about things far from us in time, distance, social connection, or hypothetically. Furthermore, the more abstractly we think about something, the more distant we tend to assume are its many aspects. In fact, the more distant something is in any way, the more distant we tend to assume it is in other ways.

This all applies not just to dates, colors, sounds, shapes, sizes, and categories, but also to the goals and priorities we use to evaluate our plans and actions. We pay more attention to detailed complexities and feasibility constraints regarding actions that are closer to us, but for far away plans we are content to think about them more simply and abstractly, in terms of relatively general values and principles that depend less on context. And when we think about plans more abstractly, we tend to assume that those actions are further away and matter less to us.

Now consider some other ways in which it might make sense to simplify our evaluation of plans and actions where we care less. We might, for example, just follow our intuitions, instead of consciously analyzing our choices. Or we might just accept expert advice about what to do, and care little about experts incentives. If there are several relevant abstract considerations, we might assume they do not conflict, or just pick one of them, instead of trying to weigh multiple considerations against each other. We might simplify an abstract consideration from many parameters down to one factor, down to a few discrete options, or even all the way down to a simple binary split.

It turns out that all of these analysis styles are characteristic of the sacred! We are not supposed to calculate the sacred, but just follow our feelings. We are to trust priests of the sacred more. Sacred things are presumed to not conflict with each other, and we are not to trade them off against other things. Sacred things are idealized in our minds, by simplifying them and neglecting their defects. And we often have sharp binary categories for sacred things; things are either sacred or not, and sacred things are not to be mixed with the non-sacred.

All of which leads me to suggest a theory of the sacred: when a group is united by valuing something highly, they value it in a style that is very abstract, having the features usually appropriate for quickly evaluating things relatively unimportant and far away. Even though this group in fact tries to value this sacred thing highly. Of course, depending on what they try to value, such attempts may have only limited success.

For example, my society (US) tries to value medicine sacredly. So ordinary people are reluctant to consciously analyze or question medical advice; they are instead to just trust its priests, namely doctors, without looking at doctor incentives or track records. Instead of thinking in terms of multiple dimensions of health, we boil it all down to a single health dimension, or even a binary of dead or alive.

Instead of seeing a continuum of cost-effectiveness of medical treatments, along which the rich would naturally go further, we want a binary of good vs bad treatments, where everyone should get the good ones no matter what their cost, and regardless of any other factors besides a diagnosis. We are not to make trades of non-sacred things for medicine, and we can’t quite believe it is ever necessary to trade medicine against other sacred things. Furthermore, we want there to be a sharp distinction between what is medicine and what is not medicine, and so we struggle to classify things like mental therapy or fresh food.

Okay, but if we see sacred things as especially important to us, why ever would we want to analyze them using styles that we usually apply to things that are far away and the least important to us? Well one theory might be that our brains find it hard to code each value in multiple ways, and so typically code our most important values as more abstracted ones, as we tend to apply them most often from a distance.

Maybe, but let me suggest another theory. When a group unites itself by sharing a key “sacred” value, then its members are especially eager to show each other that they value sacred things in the same way. However, when group members hear about and observe how an associate makes key sacred choices, they will naturally evaluate those choices from a distance. So each group member also wants to look at their own choices from afar, in order to see them in the same way that others will see them.

In this view, it is the fact groups tend to be united by sacred values that is key to explaining why they treat such values in the style usually appropriate for relatively unimportant things seen from far away, even though they actually want to value those things highly. Even though such a from-a-distance treatment will probably lead to a great many errors and misjudgments when actually trying to promote that thing.

You see, it may be more important to groups to pursue a sacred value together than to pursue it effectively. Such as the way the US spends 18% of GDP on medicine, as a costly signal of how sacred medicine is to us, even though the marginal health benefit of our medical spending seems to be near zero. And we show little interest in better institutions that could make such spending far more cost effective.

Because at least this way we all see each other’s ineffective medical choices in the same way. We agree on what to do. And after all, that’s the important thing about medicine, not whether we live or die.

Added 10Sep: Other dual process theories of brains give similar predictions.

GD Star Rating
loading...
Tagged as: , ,

AGI Is Sacred

Sacred things are especially valuable, sharply distinguished, and idealized as having less decay, messiness, inhomogeneities, or internal conflicts. We are not to mix the sacred (S) with the non-sacred (NS), nor to trade S for NS. Thus S should not have clear measures or money prices, and we shouldn’t enforce rules that promote NS at S expense.

We are to desire S “for itself”, understand S intuitively not cognitively, and not choose S based on explicit calculation or analysis. We didn’t make S; S made us. We are to trust “priests” of S, give them more self-rule and job tenure, and their differences from us don’t count as “inequality”. Objects, spaces, and times can become S by association. (More)

When we treat something as sacred, we acquire the predictably extreme related expectations and values characteristic of our concept of “sacred”. This biases us in the usual case where such extremes are unreasonable. (To min such biases, try math as sacred.)

For example, most ancient societies had a great many gods, with widely varying abilities, features, and inclinations. And different societies had different gods. But while the ancients treated these gods as pretty sacred, Christians (and Jews) upped the ante. They “knew” from their God’s recorded actions that he was pretty long-lasting, powerful, and benevolent. But they moved way beyond those “facts” to draw more extreme, and thus more sacred, conclusions about their God.

For example, Christians came to focus on a single uniquely perfect God: eternal, all-powerful, all-good, omnipresent, all-knowing (even re the future), all-wise, never-changing, without origin, self-sufficient, spirit-not-matter, never lies nor betrays trust, and perfectly loving, beautiful, gracious, kind, and pretty much any other good feature you can name. The direction, if not always the magnitude, of these changes is well predicted by our sacredness concept.

It seems to me that we’ve seen a similar process recently regarding artificial intelligence. I recall that, decades ago, the idea that we could make artificial devices who could do many of the kinds of tasks that humans do, even if not quite as well, was pretty sacred. It inspired much reverence, and respect for its priests. But just as Christians upped the ante regarding God, many recently have upped the AI ante, focusing on an even more sacred variation on AI, namely AGI: artificial general intelligence.

The default AI scenario, the one that most straightforwardly projected past trends into the future, would go as follows. Many kinds of AI systems would specialize in many different tasks, each built and managed by different orgs. There’d also be a great many AI systems of each type, controlled by competing organizations, of roughly comparable cost-effectiveness.

Overall, the abilities of these AI would improve at roughly steady rates, with rate variations similar to what we’ve seen over the last seventy years. Individual AI systems would be introduced, rise in influence for a time, and then decline in influence, as they rotted and become obsolete relative to rivals. AI systems wouldn’t work equally well with all other systems, but would instead have varying degrees of compatibility and integration.

The fraction of GDP paid for such systems would increase over time, and this would likely lead to econ growth rate increases, perhaps very large ones. Eventually many AI systems would reach human level on many tasks, but then continue to improve. Different kinds of system abilities would reach human level at different times. Even after this point, most all AI activity would be doing relatively narrow tasks.

The upped-ante version of AI, namely AGI, instead changes this scenario in the direction of making it more sacred. Compared to AI, AGI is idealized, sharply distinguished from other AI, and associated with extreme values. For example:

1) Few discussions of AGI distinguish different types of them. Instead, there is usually just one unspecialized type of AGI, assumed to be at least as good as humans at absolutely everything.

2) AGI is not a name (like “economy” or “nation”) for a diverse collection of tools run by different orgs, tools which can all in principle be combined, but not always easily. An AGI is instead seen as a highly integrated system, fully and flexibly able to apply any subset its tools to any problem, without substantial barriers such as ownership conflicts, different representations, or incompatible standards.

3) An AGI is usually seen as a consistent and coherent ideal decision agent. For example, its beliefs are assumed all consistent with each other, fully updated on all its available info, and its actions are all part of a single coherent long-term plan. Humans greatly deviate from this ideal.

4) Unlike most human organizations, and many individual humans, AGIs are assumed to have no internal conflicts, where different parts work at cross purposes, struggling for control over the whole. Instead, AGIs can last forever maintaining completely reliable internal discipline.

5) Today virtually all known large software systems rot. That is, as they are changed to add features and adapt to outside changes, they gradually become harder to usefully modify, and are eventually discarded and replaced by new systems built from scratch. But an AGI is assumed to suffer no such rot. It can instead remain effective forever.

6) AGIs can change themselves internally without limit, and have sufficiently strong self-understanding to apply this ability usefully to all of their parts. This ability does not suffer from rot. Humans and human orgs are nothing like this.

7) AGIs are usually assumed to have a strong and sharp separation between a core “values” module and all their other parts. It is assumed that value tendencies are not in any way encoded into the other many complex and opaque modules of an AGI system. The values module can be made frozen and unchanging at no cost to performance, even in the long run, and in this way an AGI’s values can stay constant forever.

8) AGIs are often assumed to be very skilled, even perfect, at cooperating with each other. Some say that is because they can show each other their read-only values modules. In this case, AGI value modules are assumed to be small, simple, and standardized enough to be read and understood by other AGIs.

9) Many analyses assume there is only one AGI in existence, with all other humans and artificial systems at the time being vastly inferior. In fact this AGI is sometimes said to be more capable than the entire rest of the world put together. Some justify this by saying multiple AGIs cooperate so well as to be in effect a single AGI.

10) AGIs are often assumed to have unlimited powers of persuasion. They can convince humans, other AIs, and organizations of pretty much any claim, even claims that would seem to be strongly contrary to their interests, and even if those entities are initially quite wary and skeptical of the AGI, and have AI advisors.

11) AGIs are often assumed to have unlimited powers of deception. They could pretend to have one set of values but really have a completely different set of values, and completely fool the humans and orgs that developed them ever since they grew up from a “baby” AI. Even when those had AI advisors. This super power of deception apparently applies only to humans and their organizations, but not to other AGIs.

12) Many analyses assume a “foom” scenario wherein this single AGI in existence evolves very quickly, suddenly, and with little warning out of far less advanced AIs who were evolving far more slowly. This evolution is so fast as to prevent the use of trial and error to find and fix its problematic aspects.

13) The possible sudden appearance, in the not-near future, of such a unique powerful perfect creature, is seen by many as event containing overwhelming value leverage, for good or ill. To many, trying to influence this event is our most important and praise-worthy action, and its priests are the most important people to revere.

I hope you can see how these AGI idealizations and values follow pretty naturally from our concept of the sacred. Just as that concept predicts the changes that religious folks seeking a more sacred God made to their God, it also predicts that AI fans seeking a more sacred AI would change it in these directions, toward this sort of version of AGI.

I’m rather skeptical that actual future AI systems, even distant future advanced ones, are well thought of as having this package of extreme idealized features. The default AI scenario I sketched above makes more sense to me.

Added 7a: In the above I’m listing assumptions commonly made about AGI in AI risk discussions, not applying a particular definition of AGI.

GD Star Rating
loading...
Tagged as: ,

Is Nothing Sacred?

“is nothing sacred?” is spoken used to express shock when something you think is valuable or important is being changed or harmed (more)

Human groups often unite via agreeing on what to treat as “sacred”. While we don’t all agree on what is how sacred, almost all of us treat some things as pretty sacred way. Sacred things are especially valuable, sharply distinguished, and idealized, so they have less decay, messiness, inhomogeneities, or internal conflicts.

We are not to mix the sacred (S) with the non-sacred (NS), nor to trade S for NS. Thus S should not have clear measures or money prices, and we shouldn’t enforce rules that promote NS at S expense. We are to desire S “for itself”, understand S intuitively not cognitively, and not choose S based on explicit calculation or analysis. We didn’t make S; S made us. We are to trust “priests” of S, give them more self-rule and job tenure, and their differences from us don’t count as “inequality”. Objects, spaces, and times can become S by association.

Treating things as sacred will tend to bias our thinking when such things do not actually have all these features, or when our values regarding them don’t actually justify all these sacred valuing rules. Yes, the benefits we get from uniting into groups might justify paying the costs of this bias. But even so, we might wonder if there are cheaper ways to gain such benefits. In particular, we might wonder if we could change what things we see as sacred, so as to reduce these biases. Asked another way: is there anything that is in fact, naturally sacred, so that treating it as such induces the least bias?

Yes, I think so. And that thing is: math. We do not create math; we find it, and it describes us. Math objects are in fact quite idealized and immortal, mostly lacking internal messy inhomogeneities. Yes, proofs can have messy details, but their assumptions and conclusions are much simpler. Math concepts don’t even suffer from the cultural context-dependence or long-term conceptual drift suffered by most abstract language concepts.

We can draw clear lines distinguishing math vs. non-math objects. Usually no one can own math, avoiding the vulgarity of associated prices. And while we think about math cognitively, the value we put on any piece of math, or on math as a whole, tends to be come intuitively, even reverently, not via calculation.

Compared to other areas, math seems an at extreme of ease of evaluation of abilities and contributions, and thus math can suppress factionalism and corruption in such evaluations. This helps us to use math to judge mental ability, care, and clarity, especially in the young. So we use math tests to sort and assign prestige early in life.

As math is so prestigious and reliable to evaluate, we can more just let math priests tell us who is good at math, and then use that as a way to choose who to hire to do math. We can thus avoid using vulgar outcome-based forms of payment to compensate math workers. It doesn’t work so badly to give math priests self-rule an long job tenures. Furthermore, so many want to be math priests that their market wages are low, making math inequality feel less offensive.

The main thing that doesn’t fit re math as sacred is that today treating math as sacred doesn’t much help us unite some groups in contrast to other groups. Though that did happen long ago (e.g., among ancient Greeks). However, I don’t at all mind this aspect of math today.

The main bias I see is that treating math as sacred induces us to treat it as more valuable than it actually is. Many academic fields, for example, put way too high a priority on math models of their topics. Which distracts from actually learning about what is important. But, hey, at least math does in fact have a lot of uses, such as in engineering and finance. Math was even crucial to great advances in many areas of science.

Yes, many over-estimate math’s contributions. But even so, I can’t think of something else that is in fact more naturally “sacred” than math. If we all in fact have a deep need to treat some things as sacred, this seems a least biased target. If something must be sacred, let it be math.

GD Star Rating
loading...
Tagged as: ,

Beware Sacred Cows

We often think we are immune to a cognitive bias if we are aware that it exists, and understand why it happens. But in fact, general awareness is usually quite insufficient to eradicate a bias; you have to put effort into particular cases to make headway. (For example, each new generation greatly underestimates how far we are from AGI, even when they know the prior record of bias.)

Psychologists reviewing our book Elephant in the Brain said that our main thesis, that we are often ignorant of our main motives, was well known in psychology, and thus not news. But our news was applying this known fact to ten big areas of life, wherein area experts had mostly ignored this possibility. General facts are often not applied to particular topic areas.

Most everyone has heard the phrase “sacred cows”, and understands that it suggests a bias, most likely resulting from a hidden motive. But I’ll bet most still usually fail to actually avoid this bias. Thus it seems well worth reviewing the nature of this bias, and seeing what it suggests about many particular application areas. In this post, I’ll just review the phenomena itself.

First, here is a table, based on recent polls, on relative sacredness for 16 ares of life:

As you can see, while traditionally our ancestors saw religion as most sacred, today that area is ranked number five, below family/friends,sex/birth, nature, and education. Note also that we treat many things as pretty sacred, but that these likely have varied a lot across societies.

We know of many correlates of perceived sacredness. (See this summary article, for example.) Let me organize these correlates around five major themes of sacred things and activities:


THEY ARE VALUABLE – Sacred things are special, often long lasting and sometimes eternal. They bring us awe, joy, and other ecstatic experiences, not disgust or revulsion. We revere, venerate, and respect them, and see them as much larger than ourselves. We prioritize them, sacrifice to connect to them, aspire to connect better, and often dedicate places and things to them. Our “priests” who are especially associated with each sacred area gain unusual prestige as a result.

THEY UNITE US – Views on what is sacred bind us into groups, and we often use group rituals and stories to learn and affirm this. We are each emotionally attached to and committed to these choices, and we pay substantial costs to signal these commitments. Doing so is seen as pro-social, and we feel more equal within our group regarding our sacred areas, than re other areas of life.

THEY ARE IDEALIZED – Sacred things are seen as suffering less from the usual defects of ordinary things. They less often decay or break or have misleading appearances. They are more pure, clean, long-lasting, and are sometimes said to be eternal, ultimate, perfect, and at the core of existence. (Much of this is consistent with sacred things being far, not near.) When homogeneous is good, then they are more like that, but when uniqueness is good, then they are more like that. We are less often forced to choose between different sacred goods, as they less conflict with each other.

THEY ARE SHARPLY DISTINGUISHED – Sacred things are said to “transcend” our physical or animal natures, and are contrasted with ordinary messy everyday life. They are “set apart’; the sacred and mundane are not to be mixed together, and we should not make tradeoffs that sacrifice any amount of the sacred for more of the ordinary. Thus sacred things should not have money prices, and we should not enforce rules that promote other things at the possible expense of the sacred.

WE MUST FEEL NOT THINK THEM – The sacred commands our emotions, (e.g., love, devotion, fear) more than our rational thought. It is associated with flow, wherein we act with less conscious control. Our desire for the sacred is said to be “for itself”, so we can’t see deeper causes. The sacred can’t be well understood cognitively, and is said to not fit well with self-interest, competition, or with our usual kinds of calculation and analysis. It is to come automatically and authentically. It is hard to measure progress toward sacred goals. We are not to think we made it, even by convention; it is fully real and it transforms us.


We can plausibly understand all of these themes as resulting from this core function of the sacred: it unites groups. After all, if we are united via a shared value, then that needs to be an important value. And our shared commitment to a value is likely enhanced by idealizing it, and by discouraging calculating thought about it. However, we don’t want to discourage everyday practical thought. So by drawing a sharp line between the sacred and the mundane, we can limit how much our thoughtlessness regarding the sacred infects our practical choices.

The last three themes of sacredness above seem to risk biases. Societies have varied greatly in what they consider as how sacred, and most consider many things to be sacred. But these areas of life are probably not actually that idealized or sharply distinguished, and thinking and analysis does likely help to manage them. So treating these areas of life as if they were otherwise likely induce biases.

Yes, gains from group cohesion may be worth paying the costs of these biases, but these costs do seem likely to be real and substantial. In fact, on reflection it seems to me that many of the most powerful insights I’ve come across in my life have resulted from treating sacred things as if they were mundane, for the purpose of analysis. I thus have to suspect there are many more such insights to be found.

FYI, here are a few psych study results on the related concept of “awe”:

“in all clear cases of awe: perceived vastness, and a need for accommodation, defined as an inability to assimilate an experience into current mental structures” (more)

“dispositional awe-proneness, but not dispositional joy or pride, was associated with low Need for Cognitive Closure … and increased emphasis on membership in “universal” categories in participants’ self-concepts” (more)

“awe can result in a diminishment of the individual self and its concerns, and increase prosocial behavior” (more)

Added 6Aug: Here is a two more themes re correlates of sacred things and activities:

THEIR EXPERTS ARE “PRIESTS” – When some folks are seen as having expertise in a sacred area, those experts tend to be seen as more prestigious and are trusted more to act for the general welfare, instead of acting selfishly. So we prefer then to have more self-rule with less outside oversight, to run related orgs, and to have more job security. We thus more dislike for-profit orgs in such areas, relative to non-profits and government agencies. We are not supposed to resent such priests being unequal with us, as differences between people regarding sacred areas are said to count less toward problematic human inequality.

THEY INFUSE OBJECTS, SPACES, & TIMES – Sacred activities done with or at objects and spaces can make those things into reminders and invokers of those sacred activities, after which their use can strengthen any such rituals. Same for special days or hours. Such things are then treated with the care and reverence accorded to sacred activities. Nostalgia is comfort from remembering things, places, or events infused by the sacred.

Added 28Aug: Maybe another correlate of the sacred is that either all of us, or very few of us, are entitled to have an opinions on it. We all feel entitled to have opinions on friendship, love, art, religion, and politics. But very few are entitled to opinions on science and medicine. Non-sacred subjects are more in the middle, with folks more entitled to opinions in proportion to the efforts they take to learn a subject. The common element seems to be that little thinking or analysis is required by ordinary people when all or none of them are entitled to an opinion.

GD Star Rating
loading...
Tagged as: ,

What I Hold Sacred

Someone recently told me that I stood out compared to other writers in never seeming to treat anything as sacred. Which seemed to them awkward, odd, and implausible, as much as the opposite writers who seem to treat most all topics and issues as sacred. More plausibly, most people do treat some minority of things as especially sacred, and if they don’t reveal that in their writing, they are probably hiding it from others, and maybe also from themselves.

This seems plausible enough that it pushes me to try to identify and admit what I hold sacred. When I search for ways to identify what people hold sacred, I find quite a lot of rather vague descriptions and associations. The most concrete signs I find are: associating it with rituals and symbols, treating it with awe and reverence, unwillingness to trade other things for it, and outrage at those who disrespect it.

The best candidate I can find is: truth-seeking. More specifically: truth-seeking among intellectuals on important topics. That is, the goal is for the world to learn more together on key abstract topics, and I want each person who contributes substantially to such projects to add the most that they can, given their constraints and the budgets they are willing to allocate to it. I don’t insist anyone devote themselves wholly to this, and I’m less concerned with each person always being perfectly honest than with us together figuring stuff out.

I admit that I do treat this with reverence, and I’m reluctant to trade it for other things. And I’d more often express outrage at others disrespecting it if I thought I’d get more support on such occasions. Yes, most everyone gives great lip service allegiance to this value. But most suggest that there are few tradeoffs between this and other values, and also that following a few simple rules of thumb (e.g., don’t lie, give confidence intervals) is sufficient; no need to dig deeper. In contrast, I think it takes long-sustained careful thought to really see what would most help for his goal, and I also see many big opportunities to sacrifice other things for this goal.

How can you better affirm this value? Its simple, but hard: Continually ask yourself what are the most important topics, what are the most promising ways to advance them, and what are your comparative advantages re such efforts. Do not assume that answers to these questions are implicit in the status and rewards that others offer you for various activities. The world mostly doesn’t care much, and so if you do care more you can’t focus on pleasing the world.

So why do I seem reluctant to talk about this? I think because I feel vulnerable. When you admit what is most precious to you, others might threaten it in order to extort concessions from you. And it is hard to argue well for why any particular value should be the most sacred. You run out of arguments and must admit you’ve made a choice you can’t justify. I so admit.

GD Star Rating
loading...
Tagged as: , ,