You Are Not Your Values
People often identify strongly with their clothes. Even so, when they find a big unnoticed defect in those clothes, such as a big stain or tear, they can taken them off, and look for replacements. Same should go for values.
You have (at least) four kinds of values:
Your learning metrics tell you which emotions and outcomes feel more like wins to you, so that you try to do more of whatever makes those happen.
Your status markers tell what you respect in others, what others should respect in you, and who you believe, including re which values you should adopt.
Your norms tell you which actions are required or forbidden in which contexts, to be respectable.
Your priorities (a.k.a. “utilities”) combine with your beliefs on facts to tell you which actions to take, when such actions are not set by other kinds of values.
You share most of these values with a community. You more respect and feel bound to others who share them, and are more wary of and ready to fight those with opposing values. These values feel sacred to you. You try to teach them to others, and like stories where your values win over opposing values.
You feel confident in them, see them as pretty directly obvious, and are reluctant to consider arguments that they might be wrong. A common way to resist such arguments is to declare that your values define you, and so aren’t open to question.
In fact, however, you are not your values. You may identify with them, but there was a before time, when you existed but had not yet embraced such values. And you are able now to consider the health of the processes by which you learned them from others, and by which those sources (e.g., your parents) learned them from further others, etc.
And if you became convinced enough that this process was in fact seriously broken, so that you actually inherited broken and misshapen values, you might want to search for other less broken values with which to replace them. If you saw your broken values as harming both you and your community, you might be willing to pay the price of seeming disloyal, at least for a time, to help you all find less destructive values.
You don’t like to consider this possibility, and it feels disloyal to do so, but you are in fact mentally capable of this task. And, alas, in order to get you, and others, to consider that humanity might have a big problem with cultural drift, I need to get you to consider this, i.e., that the cultural processes that created your shared values are badly broken. Wish me luck.
Things like broken or harmful are inherently values talk. You can only criticise a value by bringing it into conflict with another value you hold. Otherwise, if your values are broken in a way you do not care about, even if you would have cared about it in the past before you adopted your values, you simply will not care by definition. An argument which the party hearing it does not care about is a waste of time since the whole point of moral argument is changing actions, so if it fails to move people to action there is no point to it.
Maybe you agree with all of that, but in that case, please note that that is what people mean when they see your values define you and cannot be questioned. The argument is that perhaps my values are bad in a sense, but if that type of badness is something I do not value, then of course, I won’t care about this defect.
Well, yes, but I am also not evolution's values.
I am a misaligned mesa-optimiser, yes, but I do not want to become aligned.