Telephone Game With Functions
Discover more from Overcoming Bias
In the old telephone game each person would pass on a phrase to the next person in the chain; the final phrase might little resemble the first. An interesting variation appeared in the Phil. Trans. Royal Society last November:
Here each row is a chain of people passing along a function relating X to Y. Each person first guesses and is corrected on 50 (X,Y) cases, then just guesses on 100 more cases. The final guesses of the last person become data for the next person. The final relations are all basically lines, 7/8 with a positive slope, 1/8 with a negative slope.
The lesson? When we are mainly rewarded for predicting what others will say on a topic, rather than predicting a more basic reality, our answers become dominated by typical prior expectations; reality has little influence. HT to Jef Allbright. More from that paper:
In situations in which each person’s response is used to determine the data seen by the next person, people converge on concepts consistent with their inductive biases irrespective of the information seen by the first member of the chain. …
In this experiment, each generation of par ticipants received 50 trials of training on a single function. On each trial, the value of the stimulus was presented as a visual magnitude, being the width of a horizontal bar on a computer screen. Participants responded by adjusting the height of a vertical bar and then received corrective feedback. … After training, participants responded to 100 stimuli that covered the entire possible range of magnitudes without receiving feedback. …
Participants were arranged into eight families of nine generations, for each of four conditions. The conditions differed with respect to the function used to generate the training data seen by the first generation of participants: those initial values were drawn either from a positive linear, negative linear or quadratic function, or entirely at random. … The responses of each participant on 50 of the test trials were taken as the data used to train the participant in the next generation of that family. …
Across generations all of the initial functions gradually disappeared and transited into only one of two stable functions: positive linear (28 out of 32 families) and negative linear (4 out of 32), both with approximately unit slope.
Telephone Game With Functions
I'd say that comparing a horizontal bar to a vertical bar is a terrible way to receive a function. We see an obvious pattern in (c) immediately because it is plotted for us. It's easy to remember this plot once you have it and reference it to output.
This would be a good candidate for an online game or mechanical turk implementation.
People aren't very good at conveying uncertainty - for instance a lot of times they'll say 50-50 to mean "I don't know" even when the correct probability is nowhere near .5 (e.g. footnote 10 of this study implies that over one fifth of respondents said they had a 50% chance of being hurt in a terrorist attack in the next year). As Tom and others have suggested, in this design some people may have felt that the way to express uncertainty is by matching the length of the horizontal bar, and you only need one or two of those people in a chain to converge on y=x.