Simon: Appalling. Randy: Dude, why are you here? Paula covers her ears. These American Idol contestants, you know the ones, have no concept of how much they can’t sing. They’re not self-critical or introspective enough. Of cours…

Enough of this sort of data and we could figure out which features people use to infer which other features of people. This would go a long way toward showing us the various signaling games we play. But it would not help much to calibrate people’s rationality under disagreement; for that we need games where people react to the estimates of others.

The idea is to collect a number of case descriptions. A case description could be a one-pager describing the basic facts about the early stages of a relationship, the buisness plan of a start-up company, or the biography of a person up to a certain age. Participants would read the case descriptions and try to estimate various outcome measures – whether the start-up succeeded, how long the relationship lasted, what became of the individual 20 years later. The estimates would be compared to the actual outcome, which would have been recorded during the preparation of the case descriptions. Participans would be scored on both calibration and discrimination, given feedback, and the game would continue for many rounds with new case descriptions so that participants could improve over time. You could also have teams who would be allowed to discuss between themselves before issuing the team’s estimates.

For it to work well, it would be important to select the cases randomly from the relevant sample population. If one sampled from 50% of successful and 50% unsuccessful companies, or only from people who had biographies written about them, one would reduce the value of the game. So unless one could think of some clever way of compiling relevant cases, it would take a significant effort to put toghether this kind of game for general life domains. On the other hand, such a game would seem to me to have great educational value.

It might be a good investment for an enlightened ministry of education somewhere to produce and promote such material. From a scientific point of view, it would also be intersting to study how much performance in such a game would correlate with G, experience, political views, and other factors. Perhaps it might also be useful for diagnosing some psychiatric problems.

First, create a source of numbers between 0 and 1 with an unknown distribution (unknown to the player, anyway). For the game we will need to generate a sequence of numbers from this source. Each number will represent a position along a line, i.e. think of a number as the relative distance from the left end of the line. The game is played between two horizontal lines. From the top line balls will drop and on the bottom line the player can position a catcher. The size of the catcher varies in each round, but corresponds to a percentage of the line length, for example, 50% or 98%. In each round, the player can place the catcher anywhere they want on the bottom line, so long as it stays within the line. Game play is as follows:

Many balls are dropped from the top line and pile up, forming an approximate picture of the secret distribution. The player is then allowed to place the catcher and the balls are cleared away and new ones following the same distribution drop again. The player’s goal is to place the catcher, not so that it maximizes ball’s caught, but so that it catches the correct percentage of balls for its size. For example, the 10% catcher should only catch 10% of the balls dropped over a sufficiently large number of ball drops (for the game, let’s say 1000). The score is based on how accurate the player was. If the 10% catcher catches 50% of the balls dropped, the player did a poor job and gets a low score, whereas if the 98% catcher actually catches 97%, we’d consider that pretty good (after all, the sample size is small enough for there to be some error).

While not exactly in the language of confidence intervals, it’s similar to the task when picking a confidence interval, except that in real life we don’t always get to see lots of sample data first.

1. Choose a quantitative estimation task based on a historical record, e.g given a set of baseball player statistics, predict the next season’s win-loss ratio.
2. Participants individually select an interval of predetermined size, earning points if the true value is within that range. (Forcing the players to make a firm written estimate initially gives them a focus for confirmation bias and overconfidence.)
3. Participants are then allowed to see all the individual predictions and to make new predictions, scored separately. Repeat this some set number of times, watching for convergence or divergence, to make a round.
4. After a round, each player’s cumulative score is made visible to all players, informing future rounds.

I agree that people might overestimate their general calibration from learning to play a calibration game, but it’s better than nothing – you’ve got to get started somewhere.

Michael A: yes, it’s 2007, although I actually started making a few posts to this blog in late 2006…

Michael V: there’ll probably be some significant G-loading for tests like this. It might be interesting to know how much, and whether it would depend on how the test was structured. Is there a factor of “good judgment” apart from G, which would reveal itself in guessing tasks that were (a) not logical/mathematical/verbal/spatial but instead ambiguous situations from ordinary life, and (b) not knowledge-intensive like trivial quizzes? If there is such a factor, part of it might be truth-seeking motivation. If we remove that, does any “capacity to make good judgments when one tries” factor remain? In other words, is WISDOM = G + KNOWLEDGE + DESIRE TO BE WISE, or are there additional components, such as meta-rationality or intuitive judgement/common sense? (I haven’t looked for this in the literature – maybe somebody here knows?)

Bill: the almanac game is in the ballpark, but I’d ideally like to avoid testing trivia knowledge.

One type of question that I believe is common in buisness job interviews is something like, “How many lamp posts are there in Manhattan?”. But this type of question also gets boring once one has mastered the general approach to solving them (estimate the number in a typical city blocks; estimate number of city blocks; multiply).

Thanks for the comment; I’m glad I learned about this.

If C guesses “inside,” then the expected gain is .6*1 point = 0.6 points.

If C guesses “outside,” then the expected gain is .4*(.5*1 point + .5*2 points) = 0.6 points.

I’m assuming C doesn’t care if B gains or loses points (rationality is not a zero-sum game).

>> (An alternate version allows C to win another point if, when the answer is outside, C guesses “Above” or “Below” correctly)

>In that case, you should use a 60% confidence interval.

How come? I don’t follow your reasoning.

If it were a 60% probability interval, then C would always pick inside, no?

Is it to make the game fair between B and C, since in this version, C can win two points while B can only win one?