Personal experimentation: not shared?

I’ve been talking about how personal experimentation could be worth it for people like me, without relevant info being depleted long ago.

My next potential explanation is that people do experiment, but results aren’t aggregated and spread, so everyone has to reinvent everything.

This is exactly what you would expect in a simple model where people benefit from information, but bear a net cost from spreading it. Without incentives to contribute one’s own findings to others, there is no reason information should spread. But on closer inspection this is roughly the opposite of what the world looks like. There is a lot of advice about how to run the details of a life. Sometimes it is offered for money, but often so enthusiastically and freely as to make the most curious life-optimizer want to run away. The problem seems to be more that there is so much advice, advising pretty much the full range of behavior. There are apparently incentives for spreading such ‘information’, but not incentives to actually find any information to begin with.

This is doubly puzzling. It’s not surprising if all the possible self-help books exist. But for folks volunteering their own time to tell me about whatever relaxation technique or diet, spreading random misinformation seems low value. And again we have the question of why it wasn’t worth it, for their own benefit, to get some actual information to begin with.

A plausible explanation to me for both of these things is that just about any random innocuous change makes life seem better, and people are genuinely trying to be helpful by telling others about such ‘discoveries’. So the problem then would be widespread use of informal data collection, which is much more unreliable than people think. In which case, my own experimentation is just as likely to fail if I rely on such data collection. Experimentation in general would not be as useful as suspected – continually experimenting would make you feel like things were good, but none of your efforts would have long term payoffs.

This leaves the questions of whether and why people would be misinformed about their abilities to casually collect information about the effects of interventions on their lives. What say you?

GD Star Rating
Tagged as: ,
Trackback URL:
  • Ely Spears

    “Everything’s already been said, but since nobody was listening, we have to start again.” — Andre Gide

    Has anyone raised the option that people engage in personal experimentation just to signal that they are a clever/cooky/unique/different person, an individual thinker, apart from the herd? Probably most such people still are part of the herd, but want to signal that they aren’t. We prefer cultural leaders to be lauded as creative free-thinkers in Western culture. Is personal experimentation as prevalent is more authoritarian cultures?

    Maybe one of the things we most copy is “how to be seen as not a copier.” this would be an ingenious thing to copy, because I could take your neat copy-avoidance idea from weight loss or money management, etc., and apply it to my own particular fancy. If no one else has yet associated the topics, I’ll get to look like the free, unorthodox thinker just by copying your meta innovation (your way to appear innovative relative to some other endeavor).

  • > This leaves the questions of whether and why people would be
    misinformed about their abilities to casually collect information about
    the effects of interventions on their lives. What say you?

    Seems perfectly reasonable. We are all familiar with the literature on meta-analyses, replication, randomization, and blinding, yes? Even well-designed large trials can fail to yield a result which replicates or fall prey to subtle statistical issues & illusions. How much more unreliable, then, must be personal experiments, which lack even the most rudimentary randomization, systematic data collection, blinding, etc?

    None of those issues are unsurmountable. For example, it’s pretty easy to simultaneously randomize & blind pills. It’s never been easier for a random person to do statistics on their data because tools and textbooks are free online. Systematic data collection can be done well with small electronic devices or programs, ameliorating issues about data cherrypicking. (I do these all the time.) But we don’t see many people do them. Self-experimentation guru Seth Roberts, for example, does not bother with randomization, blinding, or usually any statistics more complex than graphing data points and showing means/deviations.

    If people do not understand how garbage most personal experiments are, and do not value or insist on better done personal experiments even though a lot of quality can be bought for a little effort, then perhaps that turns personal experiments into a sort of lemon market or example of Gresham’s law: most of it is bad data, you don’t know what is any good, and so no one reliably gains from the cumulative data and any genuine discoveries go unknown.

    Does this seem like a reasonable model? I have to say, it seems to me to describe pretty well the collective value of forums like Longecity.

    • Michael Vassar

      I think it would be strange if this wasn’t true.

    • marchdown

      > We are all familiar with the literature on meta-analyses, replication, randomization, and blinding, yes?

      I’m familiar with general ideas, and I feel that I understand why the best practices are the way they are, but when I’m trying to think of specific sources where they are rigorously backed, I’m drawing blanks.

      I don’t mean to impose, but I would appreciate a few pointers. Yes, I’ve stopped and used google scholar before asking.

      •  Literatures don’t really exist in single locations, that’s why they’re ‘literatures’ and not ‘books’…

        I don’t know of any single source giving data on all of them with hard citations, which is one reason I started compiling You might or might not find it useful.

      • marchdown

        Thank you. This is not immediately actionable upon, but nevertheless is so much better than raw search results, or what I could come up with within couple of hours. Have you considered reformatting this appendix into a standalone essay?

      •  People’ve suggested it. I’d like it to be more comprehensive, and more actionable; I think one should be able to derive specific estimates of how likely a study is to have found something real given its field, effect size, data availability, sample size, etc, which would be really cool to have in hand.

  • Someonefromtheotherside

    Simpler: people are different, what works for you might not work for me and vice versa. Definitely true for diet and likely some categories of intellectual pursuits

  • I think your model of costs and benefits is good, and the way to test it is to see if there are places where the costs and benefits differ from the society-wide control group, and see what the result is there.

    I see a fair bit of personal experimentation results posted to places like news.ycombinator…places where there is reputation to be gained.

  • Vlad

    See Tomasello’s book ‘Why we cooperate’. He distinguishes between sharing goods (being generous), services (being helpful) and information (being informative), and shows that the difference between humans and chimps is about sharing information (chimps are also conditionally generous and helpful, but never informative). This leads to humans having cumulative cultures while chimps not.

  • Giving advice is high status.

    • ..because asking for advice is low status. Not only what i was going to say, but even terser. Advice buys status cheaply.

    • Danny

      Excellent point – to which I’d add that it’s very common for people to give advice along the lines of “Do something I already do” with the implication that it’s optimal – and living optimally must also be high status.

  • Guest

    Experimentation is far; the status quo / path of least resistance is very near. ‘New’ habit formation appears outside the grasp of all but the most structured and disciplined individuals. Neurologically this is true. Everyone has a natural golf swing, so to everyone appears to have a natural lack of interest in personal experimentation. Growth is biological (e.g., my child body becomes an adult body) and social (e.g., I go to sleepovers with my friends; now I go to happy hour with my colleagues). Does the model assume irrationality, i.e., experimentation affords us benefits so great and at so little risk we should expect more of it, … mostly no. No – the path of least resistance is cobwebbed up with trips to Grandmas and Super Bowl parties and doctors’ visits, not to mention sleep. When experiment? At or near Katja’s age (which I am), maybe on the margin… but how much of that is signaling (e.g., look at what a purveyor of exotic cuisines and fine wine I have become) and how much is life-changing. Life is short [or it was… but then we protracted the hell out of childhood and saddled our children with student loans]. 

  • Michael Vassar

    This one sounds correct to me, and is the first explanation to sound correct.      It’s worth noting that you are still left with the question of how much optimization you should do.  None is clearly both wrong a-priori and non-majoritarian.

    • Nancy Lebovitz

      Part of Seth Roberts’ idea is to notice if your life temporarily gets better or worse, and then experiment to find the causes. 

  • Tom

    Your hypothesis is especially plausible for areas where there is hypocrisy.  Experimentation can teach you what drugs go best with sex, how to indulge in a kinky vice without blowing up your career or marriage, how to launder money, how to manipulate your staff into helping you embezzle, or how to ingratiate yourself into a high-status organization to give yourself a sheen of virtue that you don’t really possess.  For the most part, people don’t share advice about such things, and the people who do share advice are generally not the sharpest pencils in the box.

  • Applied to business, this arguments says firms should do lots of experiments because they can’t believe anything else tells them about what works in their industry. Since firms don’t seem to do this, this arguments there are huge profits to be gained by deviating from the usual practice.

    •  Isn’t that precisely the argument of Jim Manzi in _Uncontrolled_? And he apparently is well-off thanks to a company built on making it easier for businesses to run lots of experiments, on top of the reported successes of A/B testing in online websites like Google or Amazon, which suggested that there were in fact huge profits to be gained in those areas.

      • Those experiments are still a TINY fraction of the economy. 

      •  So? He has an existence proof that there are areas that under-use experimentation. It’s not then a stretch to suggest that there are similar areas with wasted expenditures and missed opportunities – if you want to talk fractions of the economy, evidence-based medicine ought to be near and dear to your heart.

    • Experiments that were “obvious and cheap” in the past have already been done, and their results have been incorporated into usual practice.  There is nothing to gain there.

      However, this leaves three promising areas:
      1. Experiments that are now obvious and cheap due to new technologies.  Depending on degree of obviousness, you’re likely to have lots of competition, but this is a way to get to the cutting edge.
      2. Experiments that are not obvious to others.  If you know things your competitors don’t, you may be able to capitalize on that.
      3. Experiments that are not economical for others to perform, but which you can capitalize on due to unusual scale and/or efficiency.

    • Michael Vassar

      Small and new firms do huge numbers of experiments.  Larger and older firms lack the ability to easily transmit information *within the firm* (transmission of information is hard) but still do huge numbers of experiments locally.

  • Anon

    What is weird is that you frame your deviations from habit as  life “experiments” and dress them up with the trappings of science (e.g. recording the results in an overly detailed and time consuming manner or adhering to the new habit with extreme regularity).  That’s all the needs explaining.

    Normal people are also constantly seeking out new pleasures and trying to optimize various health and work things.   But they frame it (to themselves and others) in a less autistic way, such as:  “I’ve been cutting back on red meat lately….I’ve decided to go to the gym three time a week…I am going to try to stay more organized by writing to-do lists,” etc.

    My guess would be that you invest so much effort in taking this normal process to extremes because you have lots of free time or are trying to signal your eccentricity.

  • Caryatis

    I’m with gwern. You can’t blind experiments on yourself (usually) and there is no control group. Plus the bad data problem. Knowing how many experiments that get published are crap, I have very little trust in others’ ability to do sound experiments.

    •  > You can’t blind experiments on yourself (usually)

      You can self-blind a lot of things; for example, supplements are generally pretty easy to blind.

      > and there is no control group.

      If the effects are not long-term or permanent, you are your own control group (a “within-subject” design); you have multiple periods where the intervention is done or not done and you collect data in each, and at the end do the analysis. (This obviously doesn’t work for something whose effects might last for years or the effect can only be observed by dying, but that excludes a great many things.)

  • Nancy Lebovitz

    The cost of sharing information isn’t going to be the same for everyone. It wouldn’t surprise me if the people who are conscientious enough to do good experiments on themselves are also likely to be conscientious enough to be meticulous about describing what they’ve done accurately. This is enough work that many just aren’t going to do it. Even if they’re public-spirited, they still might be uncertain about how many people their cure is good for.

  • Pingback: Personal experimentation: not shared? | Meteuphoric()

  • Pingback: Personal experimentation: I’m wrong? | Meteuphoric()