Tag Archives: Polls

How To Pick A Quack: Data

How do we pick, or think we should pick, our experts? One clue comes from “How to pick an X” web guides. For 18 types of experts X, I searched for that phrase, and read the top 8 Google hits, noting all of the types of info clues mentioned in each guide. Here is the full table of results.

Here are the 25 most common clue types, sorted by the % of these guides in which each is mentioned:

Here are the 18 types of experts, sorted by the average number of clue types that their guides mention:

Looking at these tables, I hypothesized that guides might prefer to mention types of clues that we’d more want to use in making our choices, and that guides might mention more clues for kinds of experts where we worry more about choosing them well. So I’ve done a set of 16 Twitter polls to estimate these things for 16 types of experts and 16 type of clues.

Results to note:

  • Guides for 18 different types of experts vary by a factor of 3 in how many types of clues they mention.
  • The top 25 info clues vary by a factor of 12 in how often they are mentioned in guides.
  • While different clues are favored in guides for different types of experts, the overall pattern looks pretty random.
  • Only 7.8% of guides mention a top 25 clue directly sensitive to outcomes. (Ones marked in red above.)
  • The correlation between how many clues guides to X mention and how worried poll respondents are re pick X is strong: +0.40.
  • The correlation between how often guides mention a clue and how much poll respondents want to know it to pick is negative: -0.20. This is mainly because polls put the most weight on track records. My followers are probably less representative here, as that’s an issue I talk much about.

Guides do not often mention outcome-related clues, presumably as few customers attend to them. In general, we can’t tell if a type of expert X is a “quack”, where “better” versions don’t help customers much more with outcomes, by the kind of clues people use to pick X. Maybe most people can’t tell the difference.

So what explanations can you offer for any of the patterns you see?

Added: Here are the poll-based priorities each expert type and info clue: Continue reading "How To Pick A Quack: Data" »

GD Star Rating
Tagged as: ,

Lognormal Priorities

In many polls on continuous variables over the last year, I’ve seen lognormal distributions typically fit poll responses well. And of course lognormals are also one of the most common distributions in nature. So let’s consider the possibility that, regarding problem areas like global warming, falling fertility, or nuclear war, distributions of priority estimate are lognormal.

Here are parameter values (M = median, A = (mean) average, S = sigma) for lognormal fits to polls on how many full-time equivalent workers should be working on each of the following six problems:

Note that priorities as set by medians are quite different from those set by averages.

Imagine that someone is asked to estimate their (median) priority of a topic area. If their estimate results from taking the product of many estimates regarding relevant factors, then not-fully-dependent noise across different factors will tend to produce a lognormal distribution regarding overall (median) estimates. If they were to then act on those estimates, such as for a poll or choosing to devote time or money, we should see a lognormal distribution of opinions and efforts. When variance (and sigma) is high, and effort is on average roughly proportional to perceived priority, then most effort should come from a quite small fraction of the population. And poll answers should look lognormal. We see both these things.

Now let’s make our theory a bit more complex. Assume that people see not only their own estimate, but sometimes also estimates of others. They then naturally put info weight on others’ estimates. This results in a distribution of (median) opinions with the same median, but a lower variance (and sigma). If they were fully rational and fully aware of each others’ opinions, this variance would fall to zero. But it doesn’t; people in general don’t listen to each other as much as they should if they cared only about accuracy. So the poll response variance we see is probably smaller than the variance in initial individual estimates, though we don’t know how much smaller.

What if the topic area in question has many subareas, and each person gives an estimate that applies to a random subarea of the total area? For example, when estimating the priority of depression, each person may draw conclusions by looking at the depressed people around them. In this case, the distribution of estimates reflects not only the variance of noisy clues, but also the real variance of priority within the overall area. Here fully rational people would come to agree on both a median and a variance, a variance reflecting the distribution of priority within this area. This true variance would be less than the variance in poll responses in a population that does not listen to each other as much as they should.

(The same applies to the variance within each person’s estimate distribution. Even if all info is aggregated, if this distribution has a remaining variance, that is “real” variance that should count, just as variance within an area should count. It is the variance resulting from failing to aggregate info that should not count.)

Now let’s consider what this all implies for action biases. If the variance in opinion expressed and acted on were due entirely to people randomly sampling from the actual variance within each area, then efforts toward each area would end up being in proportion to an info-aggregated best estimates of each area’s priority – a social optimum! But the more that variance in opinion and thus effort is also due to variance in individual noisy estimates, then the more that such variance will distort efforts. Efforts will go more as the average of each distribution, rather than its median. The priority areas with higher variance in individual noise will get too much effort, relative to areas with lower variance.

Of course there are other relevant factors that determine efforts, besides these priorities. Some priority areas have organizations that help to coordinate related efforts, thus reducing free riding problems. Some areas become fashionable, giving people extra social reasons to put in visible efforts. And other areas look weird or evil, discouraging visible efforts. Even so, we should worry that too much effort will go to areas with high variance in priority estimate noise. All else equal, you should avoid such areas. Unless estimate variance reflects mostly true variance within an area, prefer high medians over high averages.

Added 3p: I tried 7 more mundane issues, to see how they varied in variance. The following includes all 13, sorted by median.

GD Star Rating
Tagged as: , ,

What Future Areas Matter Most?

I made a list of 44 possibly important future areas, and just did 22 Twitter polls (with N from 379 to 1178), each time asking this question re 4 areas:

Over next 30 years, changes in which are likely to matter most?

I fit the answers to a simple model wherein respondents either pick randomly (~26% of time) or pick in proportion to each area’s (non-negative) “strength”. Here are the estimated area strengths, relative to the strongest set to 100:

Some comments:

  1. The area with the largest modeling error is migration, so politics may be messing that up.
  2. Governance mechanisms looks surprisingly strong, especially relative to its media attention.
  3. The top 7 areas hold half the total strength, and there’s a big drop to #8. ~20% is in automation, AGI, and self-driving cars.
  4. 19 areas have strengths lying within about the same factor of two. So many things seem important.
  5. Relative to these strength ratings, it seems to me that media focus is only roughly correlated. Media seems disproportionately focused on areas involving more direct social conflict.
  6. Areas add roughly linearly. For example, biotech arguably includes life extension, meat, and materials, and pandemics, and its strength is near their strength sum.
GD Star Rating
Tagged as: ,