Why do we fear being lied to? Because we don’t like others manipulating our beliefs. But our fear of being misled by false news pales by comparison to our fear of suffering total “mind-control”, such as depicted in the films Manchurian Candidate or Ghost in the Shell. And the part of our minds we most fear losing control of is: our deep values.
After all, damage to our memories, abilities, beliefs, or heuristic priorities might be repaired via further experience and thought. But if our deep values change, we won’t want to change them back. This is why mind designers have long presumed that artificial minds would fight hard to prevent “value drift”, and lock down their deep values. After all, within standard decision theory, agents only update their beliefs based on new information, never their preferences.
A reluctance to change our deep values fits with our reluctance to change key practices and institutions. For example, the US won’t consider changing from English to metric units, from presidential to parliamentary democracy, or stop daylight savings time, even though these changes seem clearly better to most analysts.
However, while as communities we are reluctant to change key institutions, and as individuals we are wary of letting other individuals change our values, as communities we perhaps surprisingly do not at all lock down our deep values. We instead freely, even with abandon, copy behaviors, beliefs, and values of all sorts from our prestigious associates. This allows cultural evolution, The Secret of Our Success relative to other animals.
For example, in my life I have seen a big increase in expected parental attention to kids, a switch from cornerstone to capstone marriage norms, lengthening of expected career preparation durations, great declines in religion, patriotism and militarism, far more acceptance of homo- and trans-sexuality, far stronger norms against sexist or racist language, and a merging of national cultures into a global culture, especially among elites.
These changes are quite shocking if you think about them. A system we rely on far more than our systems of units, voting, or times is changing very fast, and no one seems to be in charge, either of picking these changes ahead of time, or of evaluating them after the fact. In my essay Beware Cultural Drift I consider some stories trying to frame these as something better than maladaptive culture drift, but was not persuaded. The space of possible cultures should mostly be harsh and dysfunctional, where we started was functional due to strong selection centuries ago, yet our cultures really are wandering fast off into that vast space without a plan, map, or light.
Such changes are even more shocking to those of us old enough to remember when our culture told us to have different values than it tells us now. Neither set of values came with detailed justifications, and the arguments we are given now for recent value changes are ones we were aware of long ago, and rejected then. So do we just pretend to go along while secretly keeping our old values, abandon both the old and new values, or give the new values the benefit of the doubt, and assume our elites had good reasons for them, even if we can’t see them?
This difficult situation seems a deep challenge to standard decision theory, in which one never changes values, only beliefs. If one accepts this standard framework, as I do, and also that one is in fact choosing if to accept one’s culture’s new values, then one must find a way to see these as “values” as actually subgoals that we change as a way to preserve our deeper constant values. But then what could be these deeper values?
One option is: we each just want to be respected by associates, and are willing to adopt any behaviors or priorities that achieve this end. So we do whatever is respected by others, which is whatever is promoted by the folks they consider prestigious. Now this can’t be a guide to collective action, but it does seem a guide to individual action. Though I suspect few are willing to simply embrace this stance.
A big complication is that people often choose to switch cultures, after which they adopt the behaviors and priorities promoted by the prestigious of that new culture. So it seems that we see appropriate behaviors and priorities as depending in which culture you are in, and we seem to respect the prestigious of other cultures, not just our own. Does this mean we really share a single global culture, with a shared concept of who is how prestigious?
I don’t know, and will stop here, content to have just raised some issues here.
I don't think it actually presents a challenge to standard deciscion theory -- only the assumption that what we call our values and beliefs actually correspond to the decision theory concepts.
I mean you could model us as actually having values that highly prize signalling we aren't weird or divergent from our community. And looking at people's willingness to bet on their supposed beliefs in high stakes situations suggests that in some sense we don't really believe them.
Though, we should keep in mind (especially in AI discussions as people often make this error) that beliefs/values aren't real things out there in the world and are instead just hueristic idealizations used to predict behavior within a certain range of cases.
I mean, this can trivially be seen to be true with extreme torture where people can be made to say things even knowing that long term it will result in worse pain just to make the immediate pain stop. Or in the fact that what people profess to believe in religious contexts about life after death doesn't actually impact behavior in the way one might assume.
Ultimately, if you cut open a person or computer you don't find beliefs or values just algorithms for behavior. It's just that within certain parameters we can use these theoretical ascriptions to make predictions about that behavior.
Been loving these explorations, really thought provoking. My supposition is that we evolved without respect to explicit values or beliefs, in an environment in which the feedback loops for cultural change were fairly swift and the rate of drift fairly slow. We shouldn't expect ourselves to be able to deal with this well. Hopefully the cautionary tales that accompany the physical ruins of our civilization will at least be entertaining and well written. Another Homer awaits.