> Yet our societies contain vast numbers of people making rather similar large scale choices; why can’t we learn from those?
this is exactly the insight that inspired me to pursue "open memetics" ! Those datasets are *already* available at the big tech / social media companies / personal data brokers. The current culture plays a kind of cat & mouse game here: companies want more insight, consumers want more privacy, we're in a deadlock
But we can flip it: I want to know what decisions people in my cohort do, and how that works out for them. If they also want that to be known, we can self-select into a kind of open culture study to find out. If I can find a cohort of people in my age range / ideological belief / career / gender, etc, I can see what kind of careers they end up in. If I notice that MOST people who fit my profile never become CEOs, or when they do, they fail/burn out, that's very meaningful data for me.
Just by living my life & seeing how it turns out, I would be contributing to helping others like me. Think of it like Open Street Map, but for culture study / open psychometrics. No such project exists (yet)
> But we can flip it: I want to know what decisions people in my cohort do, and how that works out for them.
There are signs that despite knowing / seeing these things, many people will fall into negative attractors rather than positive. Obesity, smoking, drinking, and drug use are all contagious, for one thing, and people can see how the adults doing those things turn out.
I think it's the analogue of the "happy families are all the same, but unhappy are all unhappy in their own way" thing - there's more ways to fail than succeed, and success is always contingent and impacted by random happenstance and opportunities.
The overall type of analysis you're talking about is "social physics" or "social network / graph analysis."
Alex Pentland wrote a book about it called Social Physics.
I think the biggest reason the thing you're pointing to doesn't exist (a sort of bottom-up, open source social physics) is that the data is valuable. Like, what's the process here? You build an app that measures your location and what other phones you're around and what stores and places you're going to and where you sleep and work?
Well, a bunch of companies have done that already. That's the FAANGs (and China and the NSA), and they all know essentially everything about you all the time. It costs a massive amount in terms of server space and keeping the apps running and doing analytics on all of them, so who is footing those bills in the open space world?
But they do all that because it's valuable:
"Especially if social physics is actually useful enough to drive financial outcomes and marketing, which is his headline result and conceit? Google or Facebook or Alibaba isn’t just gonna hand you or anyone their data, they’ve built a trillion dollar company on that data, and every other company wants that data because it’s valuable."
And overall, knowing that John became a CEO and triathlete and June became a VP of sales before dropping out to raise her kids, what does it actually tell you? Paths like that are contingent on a bunch of frozen accidents and then years of investment in a given path, right? You can't jump from one life path to another easily, even if a ton of people just like you became VP's of whatever.
I reviewed Pentland's book here if you're interested in a little more flavor before reading it yourself:
> There are signs that despite knowing / seeing these things, many people will fall into negative attractors rather than positive
yes! mentioned it on twitter but forgot to add it here, this necessarily has to be selective. It's NOT about just giving surveillance tools to everyone. There are things I don't want to know about my cohort, and things I don't want people to generally know about my cohort (so I will just not submit that data)
> "social physics" or "social network / graph analysis."
yes!
> It costs a massive amount in terms of server space and keeping the apps running and doing analytics on all of them, so who is footing those bills in the open space world?
The question of the cost is a solved problem. Google spent billions of dollars to map out every single street and building on earth. You have to pay Google a lot of money to build apps on top of this data, and they don't sell the full dataset. Meanwhile, a bunch of volunteers did the same thing, and you can download the entire dataset for free. It used to lag behind Google until Apple & Facebook started investing in it (if you can't be a monopoly, you can at least fund the open source that destroys your competitor's moat).
I think social physics done in the open can actually completely leapfrog the closed ecosystems, even with their billions of dollars. The reason is simple: when people want to find out the answer to a question about themselves, they will participate. As opposed to having to trick & coerce & filter out bad data from people actively trying to avoid the system. There's a lot of evidence for the potential success of this pattern, which I've written about here: https://omarshehata.substack.com/p/washington-post-is-collecting-tiktok
(I really appreciate your criticisms here, I'm investing a lot of time & effort & money trying to solve this problem right now, I see very few people even considering that it's possible, or applying the right solutions. And those who are are getting early successes, so I'm gaining more confidence here as I go, but if I am missing something I want to know)
Pretty exciting and cool space to be thinking / working in, kudos.
I think we absolutely need more conscious and intentional "cultural engineering," and a bottom-up open source approach is probably more likely to get there, because right now we ARE being culturally engineered, but by the "ten thousand Phd's" at various platform companies, who created and then have been relentlessly optimizing algorithms to capture eyeballs and alpha, ending with the median person spending 7-9 hours a day on screens recreationally (tv or phones), up from ~2-4 hours a day in 2014.
To my point, the insights you can gain from stuff like this is so valuable, people have created multiple trillion dollar companies based on data like this. Do you fear that any open data set like this will be used by the next level down of non-platform companies that don't have internal data, but would be able to optimize against a large amount of open source data? Like how is access control handled?
And do you collect metadata? How are you determining "people like you?"
The Tik Tok and FB data examples in the second link are controlled because they're "owned" by a central company doing the analysis, but it sounds like you're envisioning something more publicly open?
I see this as related to the question of how does the principle of conservatism apply in rapidly changing times.
In more static times the principle is simple: When confronted with a decision that you don't encounter very often, or is too large to get your brain around – you defer to the solutions of the past as transmitted to you via "culture". This is probably the best guidance you'll get since you lack enough information yourself to make a good judgment.
It's harder to apply this when times are changing quickly. Plausibly, the solutions that worked in the past may no longer be wholly useful in modern circumstances. This is especially acute in areas related to gender relations, career paths, technology, and global connectivity where changes have been rapid and profound.
I think you've put your finger on a genuine problem here, which is how should we rationally make the "high abstraction level" decisions when past experience (culture) is no longer as useful a guide. Culture drift seems all but guaranteed. Indeed even among those who think culture drift is a problem, it isn't obvious how to apply the principle of conservatism to real-world problems: I.e. what exactly is one arguing FOR. It's like applying a conservative reading of the Constitution to modern legal questions related to AI and digital copyright – how exactly one should do such a "conservative reading" is not obvious.
> I think you've put your finger on a genuine problem here, which is how should we rationally make the "high abstraction level" decisions when past experience (culture) is no longer as useful a guide. Culture drift seems all but guaranteed.
One thing I'm not understanding here is why cultural drift has all gone to the same place.
You'd think if there's no cultural past prior and everyone is choosing their own thing, that there would be a greater ecological diversity of decisions and paths and schemas for interpreting the world.
But the opposite has happened, and Robin writes about it often - we're now a global monoculture, and it's a dysfunctional monoculture that's literally dying out.
Yeah, there's obviously a big cultural attractor somewhere pulling everyone towards the same space on the cultural landscape.
So this kind of implies something like an innate species-wide default of "when in doubt, do what high status people do," and all the high status people (who trade and travel and talk more) converging on the dysfunctional global monoculture lifestyle patterns, which then get copied?
But then what are luxury beliefs? High status people DO pair long-term monogamously, get married, have better childrearing practices, and don't waste a bunch of their lives on recreational screen time, drink or drugs, and so on.
But the median person has a terrible relationship and either never marries or gets divorced, drinks and does drugs, spends 7-9 hours a day on recreational screen time, and so on.
I guess that's just "copying, but executed poorly." And to the extent that it's executed more poorly, it leads to higher fertility the poorer the copy fidelity, as seen here:
>Why do our conscious motives do so much better a good job of explaining mid level choices
Because that's the level where we're capable of consciously holding both reasonably accurate and useful world models. On the micro scale it's more efficient to do stuff unconsciously, whereas the macro one is far too complex.
There are contexts where advocates seem to be familiar with this insight into how people choose outcomes differently given the level of abstraction, and thus strategically attempt to reframe arguments at a level of abstraction different from the usual default and which is more likely to encourage people to choose a desired outcome, or provide them with a Socially-Accepted Excuse for deviating from the typically expected outcome.
In the law this is sometimes called "the level of generalities game", see, e.g., Tribe's "Levels of Generality in the Definition of Rights" (1990) (Part II is "the problem of abstraction") Justice Scalia kicked off a lot of commentary on the subject when in Michael H v Gerald D (1989) when in footnote six he brought the nature of the game and dispute with Justice Brennan over the right limiting principle for fundamental Due Process Rights into sharp focus.
Or consider the insight embedded in the popular lawyer aphorism, "If the facts are against you, argue the law. If the law is against you, argue the facts."
See also J L Smith, "Law, Fact, and the Threat of Reversal From Above" (2013), "The more a case focuses on factual matters, the more magnified is the effect of the judges' preferences. Increased focus on legal standards dampens the effect of policy preferences. When cases turn on factual matters, variation in judges' policy preferences generates wide variation in outcomes."
Who controls (the level of abstraction for analysis of rules from) the past controls the future: who controls the present controls (the level of abstraction over) the past.
“Standard decision theory explains choices in terms of beliefs and preferences, also known as goals or motives, some of which are conscious, and others unconscious.”
It may be “standard”, but it is literally nonsense. Having beliefs, preferences, goals, or motives where no consciousness is involved is incoherent. What is being called “conscious” and “unconscious” might better be labelled something like “critical” and “habitual” consciousness, respectively. Not only does this make sense, but then we can see that there is a continuum between what is held in a highly critical state of consciousness and what is held in a highly uncritical (habitual) state of consciousness. Moreover, we can see that a habitual conscious state can be noticed to be such and then treated more critically.
> Yet our societies contain vast numbers of people making rather similar large scale choices; why can’t we learn from those?
this is exactly the insight that inspired me to pursue "open memetics" ! Those datasets are *already* available at the big tech / social media companies / personal data brokers. The current culture plays a kind of cat & mouse game here: companies want more insight, consumers want more privacy, we're in a deadlock
But we can flip it: I want to know what decisions people in my cohort do, and how that works out for them. If they also want that to be known, we can self-select into a kind of open culture study to find out. If I can find a cohort of people in my age range / ideological belief / career / gender, etc, I can see what kind of careers they end up in. If I notice that MOST people who fit my profile never become CEOs, or when they do, they fail/burn out, that's very meaningful data for me.
Just by living my life & seeing how it turns out, I would be contributing to helping others like me. Think of it like Open Street Map, but for culture study / open psychometrics. No such project exists (yet)
> But we can flip it: I want to know what decisions people in my cohort do, and how that works out for them.
There are signs that despite knowing / seeing these things, many people will fall into negative attractors rather than positive. Obesity, smoking, drinking, and drug use are all contagious, for one thing, and people can see how the adults doing those things turn out.
I think it's the analogue of the "happy families are all the same, but unhappy are all unhappy in their own way" thing - there's more ways to fail than succeed, and success is always contingent and impacted by random happenstance and opportunities.
The overall type of analysis you're talking about is "social physics" or "social network / graph analysis."
Alex Pentland wrote a book about it called Social Physics.
I think the biggest reason the thing you're pointing to doesn't exist (a sort of bottom-up, open source social physics) is that the data is valuable. Like, what's the process here? You build an app that measures your location and what other phones you're around and what stores and places you're going to and where you sleep and work?
Well, a bunch of companies have done that already. That's the FAANGs (and China and the NSA), and they all know essentially everything about you all the time. It costs a massive amount in terms of server space and keeping the apps running and doing analytics on all of them, so who is footing those bills in the open space world?
But they do all that because it's valuable:
"Especially if social physics is actually useful enough to drive financial outcomes and marketing, which is his headline result and conceit? Google or Facebook or Alibaba isn’t just gonna hand you or anyone their data, they’ve built a trillion dollar company on that data, and every other company wants that data because it’s valuable."
And overall, knowing that John became a CEO and triathlete and June became a VP of sales before dropping out to raise her kids, what does it actually tell you? Paths like that are contingent on a bunch of frozen accidents and then years of investment in a given path, right? You can't jump from one life path to another easily, even if a ton of people just like you became VP's of whatever.
I reviewed Pentland's book here if you're interested in a little more flavor before reading it yourself:
https://performativebafflement.substack.com/p/social-physics-by-alex-pentland
> There are signs that despite knowing / seeing these things, many people will fall into negative attractors rather than positive
yes! mentioned it on twitter but forgot to add it here, this necessarily has to be selective. It's NOT about just giving surveillance tools to everyone. There are things I don't want to know about my cohort, and things I don't want people to generally know about my cohort (so I will just not submit that data)
> "social physics" or "social network / graph analysis."
yes!
> It costs a massive amount in terms of server space and keeping the apps running and doing analytics on all of them, so who is footing those bills in the open space world?
strong disagree here: the reason it doesn't exist is because no one has figured out how to do it in a way that doesn't upset people (because surveillance erodes agency, even if bottom up). I wrote about this here: https://defenderofthebasic.substack.com/p/we-can-do-cultural-engineering-openly
The question of the cost is a solved problem. Google spent billions of dollars to map out every single street and building on earth. You have to pay Google a lot of money to build apps on top of this data, and they don't sell the full dataset. Meanwhile, a bunch of volunteers did the same thing, and you can download the entire dataset for free. It used to lag behind Google until Apple & Facebook started investing in it (if you can't be a monopoly, you can at least fund the open source that destroys your competitor's moat).
I think social physics done in the open can actually completely leapfrog the closed ecosystems, even with their billions of dollars. The reason is simple: when people want to find out the answer to a question about themselves, they will participate. As opposed to having to trick & coerce & filter out bad data from people actively trying to avoid the system. There's a lot of evidence for the potential success of this pattern, which I've written about here: https://omarshehata.substack.com/p/washington-post-is-collecting-tiktok
(I really appreciate your criticisms here, I'm investing a lot of time & effort & money trying to solve this problem right now, I see very few people even considering that it's possible, or applying the right solutions. And those who are are getting early successes, so I'm gaining more confidence here as I go, but if I am missing something I want to know)
Pretty exciting and cool space to be thinking / working in, kudos.
I think we absolutely need more conscious and intentional "cultural engineering," and a bottom-up open source approach is probably more likely to get there, because right now we ARE being culturally engineered, but by the "ten thousand Phd's" at various platform companies, who created and then have been relentlessly optimizing algorithms to capture eyeballs and alpha, ending with the median person spending 7-9 hours a day on screens recreationally (tv or phones), up from ~2-4 hours a day in 2014.
To my point, the insights you can gain from stuff like this is so valuable, people have created multiple trillion dollar companies based on data like this. Do you fear that any open data set like this will be used by the next level down of non-platform companies that don't have internal data, but would be able to optimize against a large amount of open source data? Like how is access control handled?
And do you collect metadata? How are you determining "people like you?"
The Tik Tok and FB data examples in the second link are controlled because they're "owned" by a central company doing the analysis, but it sounds like you're envisioning something more publicly open?
I see this as related to the question of how does the principle of conservatism apply in rapidly changing times.
In more static times the principle is simple: When confronted with a decision that you don't encounter very often, or is too large to get your brain around – you defer to the solutions of the past as transmitted to you via "culture". This is probably the best guidance you'll get since you lack enough information yourself to make a good judgment.
It's harder to apply this when times are changing quickly. Plausibly, the solutions that worked in the past may no longer be wholly useful in modern circumstances. This is especially acute in areas related to gender relations, career paths, technology, and global connectivity where changes have been rapid and profound.
I think you've put your finger on a genuine problem here, which is how should we rationally make the "high abstraction level" decisions when past experience (culture) is no longer as useful a guide. Culture drift seems all but guaranteed. Indeed even among those who think culture drift is a problem, it isn't obvious how to apply the principle of conservatism to real-world problems: I.e. what exactly is one arguing FOR. It's like applying a conservative reading of the Constitution to modern legal questions related to AI and digital copyright – how exactly one should do such a "conservative reading" is not obvious.
> I think you've put your finger on a genuine problem here, which is how should we rationally make the "high abstraction level" decisions when past experience (culture) is no longer as useful a guide. Culture drift seems all but guaranteed.
One thing I'm not understanding here is why cultural drift has all gone to the same place.
You'd think if there's no cultural past prior and everyone is choosing their own thing, that there would be a greater ecological diversity of decisions and paths and schemas for interpreting the world.
But the opposite has happened, and Robin writes about it often - we're now a global monoculture, and it's a dysfunctional monoculture that's literally dying out.
High levels of trade, travel, & talk pull cultures closer together.
Yeah, there's obviously a big cultural attractor somewhere pulling everyone towards the same space on the cultural landscape.
So this kind of implies something like an innate species-wide default of "when in doubt, do what high status people do," and all the high status people (who trade and travel and talk more) converging on the dysfunctional global monoculture lifestyle patterns, which then get copied?
But then what are luxury beliefs? High status people DO pair long-term monogamously, get married, have better childrearing practices, and don't waste a bunch of their lives on recreational screen time, drink or drugs, and so on.
But the median person has a terrible relationship and either never marries or gets divorced, drinks and does drugs, spends 7-9 hours a day on recreational screen time, and so on.
I guess that's just "copying, but executed poorly." And to the extent that it's executed more poorly, it leads to higher fertility the poorer the copy fidelity, as seen here:
https://imgur.com/a/R6LyVY2
>Why do our conscious motives do so much better a good job of explaining mid level choices
Because that's the level where we're capable of consciously holding both reasonably accurate and useful world models. On the micro scale it's more efficient to do stuff unconsciously, whereas the macro one is far too complex.
There are contexts where advocates seem to be familiar with this insight into how people choose outcomes differently given the level of abstraction, and thus strategically attempt to reframe arguments at a level of abstraction different from the usual default and which is more likely to encourage people to choose a desired outcome, or provide them with a Socially-Accepted Excuse for deviating from the typically expected outcome.
In the law this is sometimes called "the level of generalities game", see, e.g., Tribe's "Levels of Generality in the Definition of Rights" (1990) (Part II is "the problem of abstraction") Justice Scalia kicked off a lot of commentary on the subject when in Michael H v Gerald D (1989) when in footnote six he brought the nature of the game and dispute with Justice Brennan over the right limiting principle for fundamental Due Process Rights into sharp focus.
Or consider the insight embedded in the popular lawyer aphorism, "If the facts are against you, argue the law. If the law is against you, argue the facts."
See also J L Smith, "Law, Fact, and the Threat of Reversal From Above" (2013), "The more a case focuses on factual matters, the more magnified is the effect of the judges' preferences. Increased focus on legal standards dampens the effect of policy preferences. When cases turn on factual matters, variation in judges' policy preferences generates wide variation in outcomes."
Who controls (the level of abstraction for analysis of rules from) the past controls the future: who controls the present controls (the level of abstraction over) the past.
Why are people so willing to go along with the premise that taboo motivations are unconscious just because the people acting from them claim they are?
“Standard decision theory explains choices in terms of beliefs and preferences, also known as goals or motives, some of which are conscious, and others unconscious.”
It may be “standard”, but it is literally nonsense. Having beliefs, preferences, goals, or motives where no consciousness is involved is incoherent. What is being called “conscious” and “unconscious” might better be labelled something like “critical” and “habitual” consciousness, respectively. Not only does this make sense, but then we can see that there is a continuum between what is held in a highly critical state of consciousness and what is held in a highly uncritical (habitual) state of consciousness. Moreover, we can see that a habitual conscious state can be noticed to be such and then treated more critically.