Tag Archives: Sacred

AGI Is Sacred

Sacred things are especially valuable, sharply distinguished, and idealized as having less decay, messiness, inhomogeneities, or internal conflicts. We are not to mix the sacred (S) with the non-sacred (NS), nor to trade S for NS. Thus S should not have clear measures or money prices, and we shouldn’t enforce rules that promote NS at S expense.

We are to desire S “for itself”, understand S intuitively not cognitively, and not choose S based on explicit calculation or analysis. We didn’t make S; S made us. We are to trust “priests” of S, give them more self-rule and job tenure, and their differences from us don’t count as “inequality”. Objects, spaces, and times can become S by association. (More)

When we treat something as sacred, we acquire the predictably extreme related expectations and values characteristic of our concept of “sacred”. This biases us in the usual case where such extremes are unreasonable. (To min such biases, try math as sacred.)

For example, most ancient societies had a great many gods, with widely varying abilities, features, and inclinations. And different societies had different gods. But while the ancients treated these gods as pretty sacred, Christians (and Jews) upped the ante. They “knew” from their God’s recorded actions that he was pretty long-lasting, powerful, and benevolent. But they moved way beyond those “facts” to draw more extreme, and thus more sacred, conclusions about their God.

For example, Christians came to focus on a single uniquely perfect God: eternal, all-powerful, all-good, omnipresent, all-knowing (even re the future), all-wise, never-changing, without origin, self-sufficient, spirit-not-matter, never lies nor betrays trust, and perfectly loving, beautiful, gracious, kind, and pretty much any other good feature you can name. The direction, if not always the magnitude, of these changes is well predicted by our sacredness concept.

It seems to me that we’ve seen a similar process recently regarding artificial intelligence. I recall that, decades ago, the idea that we could make artificial devices who could do many of the kinds of tasks that humans do, even if not quite as well, was pretty sacred. It inspired much reverence, and respect for its priests. But just as Christians upped the ante regarding God, many recently have upped the AI ante, focusing on an even more sacred variation on AI, namely AGI: artificial general intelligence.

The default AI scenario, the one that most straightforwardly projected past trends into the future, would go as follows. Many kinds of AI systems would specialize in many different tasks, each built and managed by different orgs. There’d also be a great many AI systems of each type, controlled by competing organizations, of roughly comparable cost-effectiveness.

Overall, the abilities of these AI would improve at roughly steady rates, with rate variations similar to what we’ve seen over the last seventy years. Individual AI systems would be introduced, rise in influence for a time, and then decline in influence, as they rotted and become obsolete relative to rivals. AI systems wouldn’t work equally well with all other systems, but would instead have varying degrees of compatibility and integration.

The fraction of GDP paid for such systems would increase over time, and this would likely lead to econ growth rate increases, perhaps very large ones. Eventually many AI systems would reach human level on many tasks, but then continue to improve. Different kinds of system abilities would reach human level at different times. Even after this point, most all AI activity would be doing relatively narrow tasks.

The upped-ante version of AI, namely AGI, instead changes this scenario in the direction of making it more sacred. Compared to AI, AGI is idealized, sharply distinguished from other AI, and associated with extreme values. For example:

1) Few discussions of AGI distinguish different types of them. Instead, there is usually just one unspecialized type of AGI, assumed to be at least as good as humans at absolutely everything.

2) AGI is not a name (like “economy” or “nation”) for a diverse collection of tools run by different orgs, tools which can all in principle be combined, but not always easily. An AGI is instead seen as a highly integrated system, fully and flexibly able to apply any subset its tools to any problem, without substantial barriers such as ownership conflicts, different representations, or incompatible standards.

3) An AGI is usually seen as a consistent and coherent ideal decision agent. For example, its beliefs are assumed all consistent with each other, fully updated on all its available info, and its actions are all part of a single coherent long-term plan. Humans greatly deviate from this ideal.

4) Unlike most human organizations, and many individual humans, AGIs are assumed to have no internal conflicts, where different parts work at cross purposes, struggling for control over the whole. Instead, AGIs can last forever maintaining completely reliable internal discipline.

5) Today virtually all known large software systems rot. That is, as they are changed to add features and adapt to outside changes, they gradually become harder to usefully modify, and are eventually discarded and replaced by new systems built from scratch. But an AGI is assumed to suffer no such rot. It can instead remain effective forever.

6) AGIs can change themselves internally without limit, and have sufficiently strong self-understanding to apply this ability usefully to all of their parts. This ability does not suffer from rot. Humans and human orgs are nothing like this.

7) AGIs are usually assumed to have a strong and sharp separation between a core “values” module and all their other parts. It is assumed that value tendencies are not in any way encoded into the other many complex and opaque modules of an AGI system. The values module can be made frozen and unchanging at no cost to performance, even in the long run, and in this way an AGI’s values can stay constant forever.

8) AGIs are often assumed to be very skilled, even perfect, at cooperating with each other. Some say that is because they can show each other their read-only values modules. In this case, AGI value modules are assumed to be small, simple, and standardized enough to be read and understood by other AGIs.

9) Many analyses assume there is only one AGI in existence, with all other humans and artificial systems at the time being vastly inferior. In fact this AGI is sometimes said to be more capable than the entire rest of the world put together. Some justify this by saying multiple AGIs cooperate so well as to be in effect a single AGI.

10) AGIs are often assumed to have unlimited powers of persuasion. They can convince humans, other AIs, and organizations of pretty much any claim, even claims that would seem to be strongly contrary to their interests, and even if those entities are initially quite wary and skeptical of the AGI, and have AI advisors.

11) AGIs are often assumed to have unlimited powers of deception. They could pretend to have one set of values but really have a completely different set of values, and completely fool the humans and orgs that developed them ever since they grew up from a “baby” AI. Even when those had AI advisors. This super power of deception apparently applies only to humans and their organizations, but not to other AGIs.

12) Many analyses assume a “foom” scenario wherein this single AGI in existence evolves very quickly, suddenly, and with little warning out of far less advanced AIs who were evolving far more slowly. This evolution is so fast as to prevent the use of trial and error to find and fix its problematic aspects.

13) The possible sudden appearance, in the not-near future, of such a unique powerful perfect creature, is seen by many as event containing overwhelming value leverage, for good or ill. To many, trying to influence this event is our most important and praise-worthy action, and its priests are the most important people to revere.

I hope you can see how these AGI idealizations and values follow pretty naturally from our concept of the sacred. Just as that concept predicts the changes that religious folks seeking a more sacred God made to their God, it also predicts that AI fans seeking a more sacred AI would change it in these directions, toward this sort of version of AGI.

I’m rather skeptical that actual future AI systems, even distant future advanced ones, are well thought of as having this package of extreme idealized features. The default AI scenario I sketched above makes more sense to me.

Added 7a: In the above I’m listing assumptions commonly made about AGI in AI risk discussions, not applying a particular definition of AGI.

GD Star Rating
loading...
Tagged as: ,

Is Nothing Sacred?

“is nothing sacred?” is spoken used to express shock when something you think is valuable or important is being changed or harmed (more)

Human groups often unite via agreeing on what to treat as “sacred”. While we don’t all agree on what is how sacred, almost all of us treat some things as pretty sacred way. Sacred things are especially valuable, sharply distinguished, and idealized, so they have less decay, messiness, inhomogeneities, or internal conflicts.

We are not to mix the sacred (S) with the non-sacred (NS), nor to trade S for NS. Thus S should not have clear measures or money prices, and we shouldn’t enforce rules that promote NS at S expense. We are to desire S “for itself”, understand S intuitively not cognitively, and not choose S based on explicit calculation or analysis. We didn’t make S; S made us. We are to trust “priests” of S, give them more self-rule and job tenure, and their differences from us don’t count as “inequality”. Objects, spaces, and times can become S by association.

Treating things as sacred will tend to bias our thinking when such things do not actually have all these features, or when our values regarding them don’t actually justify all these sacred valuing rules. Yes, the benefits we get from uniting into groups might justify paying the costs of this bias. But even so, we might wonder if there are cheaper ways to gain such benefits. In particular, we might wonder if we could change what things we see as sacred, so as to reduce these biases. Asked another way: is there anything that is in fact, naturally sacred, so that treating it as such induces the least bias?

Yes, I think so. And that thing is: math. We do not create math; we find it, and it describes us. Math objects are in fact quite idealized and immortal, mostly lacking internal messy inhomogeneities. Yes, proofs can have messy details, but their assumptions and conclusions are much simpler. Math concepts don’t even suffer from the cultural context-dependence or long-term conceptual drift suffered by most abstract language concepts.

We can draw clear lines distinguishing math vs. non-math objects. Usually no one can own math, avoiding the vulgarity of associated prices. And while we think about math cognitively, the value we put on any piece of math, or on math as a whole, tends to be come intuitively, even reverently, not via calculation.

Compared to other areas, math seems an at extreme of ease of evaluation of abilities and contributions, and thus math can suppress factionalism and corruption in such evaluations. This helps us to use math to judge mental ability, care, and clarity, especially in the young. So we use math tests to sort and assign prestige early in life.

As math is so prestigious and reliable to evaluate, we can more just let math priests tell us who is good at math, and then use that as a way to choose who to hire to do math. We can thus avoid using vulgar outcome-based forms of payment to compensate math workers. It doesn’t work so badly to give math priests self-rule an long job tenures. Furthermore, so many want to be math priests that their market wages are low, making math inequality feel less offensive.

The main thing that doesn’t fit re math as sacred is that today treating math as sacred doesn’t much help us unite some groups in contrast to other groups. Though that did happen long ago (e.g., among ancient Greeks). However, I don’t at all mind this aspect of math today.

The main bias I see is that treating math as sacred induces us to treat it as more valuable than it actually is. Many academic fields, for example, put way too high a priority on math models of their topics. Which distracts from actually learning about what is important. But, hey, at least math does in fact have a lot of uses, such as in engineering and finance. Math was even crucial to great advances in many areas of science.

Yes, many over-estimate math’s contributions. But even so, I can’t think of something else that is in fact more naturally “sacred” than math. If we all in fact have a deep need to treat some things as sacred, this seems a least biased target. If something must be sacred, let it be math.

GD Star Rating
loading...
Tagged as: ,

Beware Sacred Cows

We often think we are immune to a cognitive bias if we are aware that it exists, and understand why it happens. But in fact, general awareness is usually quite insufficient to eradicate a bias; you have to put effort into particular cases to make headway. (For example, each new generation greatly underestimates how far we are from AGI, even when they know the prior record of bias.)

Psychologists reviewing our book Elephant in the Brain said that our main thesis, that we are often ignorant of our main motives, was well known in psychology, and thus not news. But our news was applying this known fact to ten big areas of life, wherein area experts had mostly ignored this possibility. General facts are often not applied to particular topic areas.

Most everyone has heard the phrase “sacred cows”, and understands that it suggests a bias, most likely resulting from a hidden motive. But I’ll bet most still usually fail to actually avoid this bias. Thus it seems well worth reviewing the nature of this bias, and seeing what it suggests about many particular application areas. In this post, I’ll just review the phenomena itself.

First, here is a table, based on recent polls, on relative sacredness for 16 ares of life:

As you can see, while traditionally our ancestors saw religion as most sacred, today that area is ranked number five, below family/friends,sex/birth, nature, and education. Note also that we treat many things as pretty sacred, but that these likely have varied a lot across societies.

We know of many correlates of perceived sacredness. (See this summary article, for example.) Let me organize these correlates around five major themes of sacred things and activities:


THEY ARE VALUABLE – Sacred things are special, often long lasting and sometimes eternal. They bring us awe, joy, and other ecstatic experiences, not disgust or revulsion. We revere, venerate, and respect them, and see them as much larger than ourselves. We prioritize them, sacrifice to connect to them, aspire to connect better, and often dedicate places and things to them. Our “priests” who are especially associated with each sacred area gain unusual prestige as a result.

THEY UNITE US – Views on what is sacred bind us into groups, and we often use group rituals and stories to learn and affirm this. We are each emotionally attached to and committed to these choices, and we pay substantial costs to signal these commitments. Doing so is seen as pro-social, and we feel more equal within our group regarding our sacred areas, than re other areas of life.

THEY ARE IDEALIZED – Sacred things are seen as suffering less from the usual defects of ordinary things. They less often decay or break or have misleading appearances. They are more pure, clean, long-lasting, and are sometimes said to be eternal, ultimate, perfect, and at the core of existence. (Much of this is consistent with sacred things being far, not near.) When homogeneous is good, then they are more like that, but when uniqueness is good, then they are more like that. We are less often forced to choose between different sacred goods, as they less conflict with each other.

THEY ARE SHARPLY DISTINGUISHED – Sacred things are said to “transcend” our physical or animal natures, and are contrasted with ordinary messy everyday life. They are “set apart’; the sacred and mundane are not to be mixed together, and we should not make tradeoffs that sacrifice any amount of the sacred for more of the ordinary. Thus sacred things should not have money prices, and we should not enforce rules that promote other things at the possible expense of the sacred.

WE MUST FEEL NOT THINK THEM – The sacred commands our emotions, (e.g., love, devotion, fear) more than our rational thought. It is associated with flow, wherein we act with less conscious control. Our desire for the sacred is said to be “for itself”, so we can’t see deeper causes. The sacred can’t be well understood cognitively, and is said to not fit well with self-interest, competition, or with our usual kinds of calculation and analysis. It is to come automatically and authentically. It is hard to measure progress toward sacred goals. We are not to think we made it, even by convention; it is fully real and it transforms us.


We can plausibly understand all of these themes as resulting from this core function of the sacred: it unites groups. After all, if we are united via a shared value, then that needs to be an important value. And our shared commitment to a value is likely enhanced by idealizing it, and by discouraging calculating thought about it. However, we don’t want to discourage everyday practical thought. So by drawing a sharp line between the sacred and the mundane, we can limit how much our thoughtlessness regarding the sacred infects our practical choices.

The last three themes of sacredness above seem to risk biases. Societies have varied greatly in what they consider as how sacred, and most consider many things to be sacred. But these areas of life are probably not actually that idealized or sharply distinguished, and thinking and analysis does likely help to manage them. So treating these areas of life as if they were otherwise likely induce biases.

Yes, gains from group cohesion may be worth paying the costs of these biases, but these costs do seem likely to be real and substantial. In fact, on reflection it seems to me that many of the most powerful insights I’ve come across in my life have resulted from treating sacred things as if they were mundane, for the purpose of analysis. I thus have to suspect there are many more such insights to be found.

FYI, here are a few psych study results on the related concept of “awe”:

“in all clear cases of awe: perceived vastness, and a need for accommodation, defined as an inability to assimilate an experience into current mental structures” (more)

“dispositional awe-proneness, but not dispositional joy or pride, was associated with low Need for Cognitive Closure … and increased emphasis on membership in “universal” categories in participants’ self-concepts” (more)

“awe can result in a diminishment of the individual self and its concerns, and increase prosocial behavior” (more)

Added 6Aug: Here is a two more themes re correlates of sacred things and activities:

THEIR EXPERTS ARE “PRIESTS” – When some folks are seen as having expertise in a sacred area, those experts tend to be seen as more prestigious and are trusted more to act for the general welfare, instead of acting selfishly. So we prefer then to have more self-rule with less outside oversight, to run related orgs, and to have more job security. We thus more dislike for-profit orgs in such areas, relative to non-profits and government agencies. We are not supposed to resent such priests being unequal with us, as differences between people regarding sacred areas are said to count less toward problematic human inequality.

THEY INFUSE OBJECTS, SPACES, & TIMES – Sacred activities done with or at objects and spaces can make those things into reminders and invokers of those sacred activities, after which their use can strengthen any such rituals. Same for special days or hours. Such things are then treated with the care and reverence accorded to sacred activities. Nostalgia is comfort from remembering things, places, or events infused by the sacred.

Added 28Aug: Maybe another correlate of the sacred is that either all of us, or very few of us, are entitled to have an opinions on it. We all feel entitled to have opinions on friendship, love, art, religion, and politics. But very few are entitled to opinions on science and medicine. Non-sacred subjects are more in the middle, with folks more entitled to opinions in proportion to the efforts they take to learn a subject. The common element seems to be that little thinking or analysis is required by ordinary people when all or none of them are entitled to an opinion.

GD Star Rating
loading...
Tagged as: ,

What I Hold Sacred

Someone recently told me that I stood out compared to other writers in never seeming to treat anything as sacred. Which seemed to them awkward, odd, and implausible, as much as the opposite writers who seem to treat most all topics and issues as sacred. More plausibly, most people do treat some minority of things as especially sacred, and if they don’t reveal that in their writing, they are probably hiding it from others, and maybe also from themselves.

This seems plausible enough that it pushes me to try to identify and admit what I hold sacred. When I search for ways to identify what people hold sacred, I find quite a lot of rather vague descriptions and associations. The most concrete signs I find are: associating it with rituals and symbols, treating it with awe and reverence, unwillingness to trade other things for it, and outrage at those who disrespect it.

The best candidate I can find is: truth-seeking. More specifically: truth-seeking among intellectuals on important topics. That is, the goal is for the world to learn more together on key abstract topics, and I want each person who contributes substantially to such projects to add the most that they can, given their constraints and the budgets they are willing to allocate to it. I don’t insist anyone devote themselves wholly to this, and I’m less concerned with each person always being perfectly honest than with us together figuring stuff out.

I admit that I do treat this with reverence, and I’m reluctant to trade it for other things. And I’d more often express outrage at others disrespecting it if I thought I’d get more support on such occasions. Yes, most everyone gives great lip service allegiance to this value. But most suggest that there are few tradeoffs between this and other values, and also that following a few simple rules of thumb (e.g., don’t lie, give confidence intervals) is sufficient; no need to dig deeper. In contrast, I think it takes long-sustained careful thought to really see what would most help for his goal, and I also see many big opportunities to sacrifice other things for this goal.

How can you better affirm this value? Its simple, but hard: Continually ask yourself what are the most important topics, what are the most promising ways to advance them, and what are your comparative advantages re such efforts. Do not assume that answers to these questions are implicit in the status and rewards that others offer you for various activities. The world mostly doesn’t care much, and so if you do care more you can’t focus on pleasing the world.

So why do I seem reluctant to talk about this? I think because I feel vulnerable. When you admit what is most precious to you, others might threaten it in order to extort concessions from you. And it is hard to argue well for why any particular value should be the most sacred. You run out of arguments and must admit you’ve made a choice you can’t justify. I so admit.

GD Star Rating
loading...
Tagged as: , ,