In standard decision theory, an agent makes a choice in each of a large number of possible choice situations. If these choices satisfy some plausible rationality axioms, they can be represented by real-valued utility and belief functions over a set of possible states of the world. As the agent learns info, their beliefs update according to Bayes rule, but their utilities do not change. There are noisy versions of this, where agents make limited mistakes re this standard, and standard game theory is built on top of all this.
Humans who find these rationality axioms plausible aspire to become more rational, by pushing their action habits to become more consistent with these axioms. When focused on actions within some short time duration, this will create a more consistent set of values for actions in that time period, by choosing some ways to make tradeoffs between conflicting values. This usually seems a mature and valuable exercise.
Sometimes such humans discover that the actions toward which they are inclined just cannot be made very consistent in this way. Doing so requires more self-control than they can muster. At which point they in effect accept that they cannot be even approximately rational, however attractive they might seem. However, this appears to be a relatively rare problem among those who try to be rational; they can usually seem to get usefully closer to this ideal.
However, when we look at longer timescales, it often looks like the “values” that seem to explain an individual human’s behaviors at different times have changed substantially. At which point this human has two viewpoint options that I can see.
One viewpoint option is to see themselves at different times as just different people with conflicting values. Some other process, substantially outside of themselves, causes this change in values across time, causing these people to differ. These different people share an unusually lot of values, and have unusually large opportunities to cooperate, but still they have the sorts of conflicts that people with different values do. One unusual aspect of their relation is that the earlier one may have unusually strong ways to casually influence the values of the later one, by “pushing buttons” of various sorts on the process that causes values to change.
The other viewpoint option is to for this human to try to see themselves as a single rational actor whose values change “rationally”. That is, they try to see themselves as having stable deep values, with the shallow “values” that more directly explain their actions being merely changing estimates of those deep values. For this to make sense, their shallow value changes over time should roughly follow a random walk, as that’s how informed estimates should change with time.
There should also be some source that can plausibly serve as the information basis for this learning. It is okay if this source changes in predictable ways with changing conditions, like changing wealth, lifespans, or climate, if the underlying deep values are postulated to also be context-dependent, changing in predictable ways with such conditions.
But what could be this info source? One possibility is that while our action intuitions tell us which actions to take, often after we take an action a different kind of intuition tells us how much we liked the result of that action. From that sort of feedback, we slowly learn more about the deep values implicit in our like-outcome-intuitions; the values implicit in our action-intuitions are only temporary best guesses about those deeper values.
Another possibility comes from our standard stories of cultural evolution. These say that we are driven to imitate the attitudes, behaviors, and values of prestigious successful folks from our culture. If so, all else equal we will have more positive intuitions about both our actions and our outcomes when they seem to match with and come from our prestigious associates. We could describe this process as our using those imitation inputs as data to update our shallow values as estimates of constant deep values.
For either of these stories to work, the purported source of signals about our deep values should either be stable over time, or be a random walk that could plausibly represent accumulating information within that source regarding something stable (or predictably context-dependent). So the key question about both the like-outcomes-intuitions-as-info story and the prestigious-behavior-as-info stories is: do those sources actually fit this sort of pattern?
And it seems to me that they do not. Our like-outcome intuitions do not seem to me very stable, and in any case our shallow values seem to me to come more from imitating prestigious associates. And important parts of the behavior of prestigious associates does not seem either stable or to follow random walks that could plausibly represent their learning about deep values over time. Culture instead seems to just drift in many big ways.
So I’m forced to see future versions of us all as different people, regarding which we might “push buttons” to influence their values.
Added 21Mar: On reflection a simple single agent theory that might work is that the deep value is for status. We copy what high status people do in an attempt to gain status markers ourselves. If what counts for status changes, then what we do changes.
Personally I found my values changing a lot after I had kids. But I don’t feel like a different person, nor do I feel that either my pre-kid or post-kid self was more correct per se.
I feel more like my “true values” are unknowable, and my intuition over time gives me different clues as to what my true values are. Rational action is great but it does have the limit that I only know an approximation to my own values. Philosophy can help draw these out but it’s always incomplete.
I know generational theories are not considered serious in intellectual circles, but I find them very useful in explaining how values change over time.
In your framework, I would fit them into the "context-dependent" slot, except the context is not a simple scalar, like wealth or age. The context is the relationship between a person's generation and the previous and next generations.
However you classify them, over time, the numbers in the previous generations will fall off, and the numbers in next generations will increase. So if you're trying to develop a plausible coherent framework to explain your motives, you'll have to change your framework to accomodate this reality.
A lot of your "preferences" may just be falsified preferences (ala Kuran) to fit in with your surroundings, and when the numbers of the olds and youngs start shifting, your falsified preferences set will abruptly change in the form of cascades. You may not even realize that they were falsified preferences to begin with, since prior to that point, the social drivers were so firm and obvious that you may never even have realized it that they were malleable.
People outside your generation can have an outsize impact as tie-breakers for deep value conflicts of your generation because they're not your direct rivals (people tend to resist value advocated by competitors). They can serve as "arbitrators" in the social competition of your generation, even if they're not directly involved. For example, consider if your generation has a deep value divide that you can't resolve. If this division happens when you're young, often there's a Cold Truce imposed by the elders (your spats are excluded from Serious Discourse). On the other hand, if this division happens when you're older, then eventually the youths will decide the matter, whether you like it or not.
This generational changeover is dynamic because (at least in our culture) kids are the people who are most empowered to rebel. Whether or not their decision makes sense, they'll do different stuff than the people immediately before then (even if its the same different stuff as the people 2 or 3 generations ago). So the context keeps changing and refreshing.
This doesn't mean that the youths are the only drivers of change. Each generation drives changes throughout their lifetime. They age into roles the previous generations once had, but they play those roles differently than them, and so the context keeps shifting. And all the preferences you thought you had that you don't really have a deep personal stake will have to keep shifting to keep up according to the prevailing Preference Falsifications regimes (aka Keeping up with the Times).