New Scientist magazine set up a contest between new prediction techniques, including prediction markets: We decided to see how the latest techniques would stand up to the task of predicting what people will buy. … Over the past four months, we have set four teams the task of trying to predict the sales of each issue of
A new site for tracking pundit predictions. But as my link indicates, pundits have suffered no consequences for terrible predictions, so maybe folks just don't care.
Jim, happy to accept that your market participants are not amateurs, and that they were given some incentives. But the other contest participants were world class professionals, competing for the valuable publicity prize of New Scientist's implicit endorsement. Those seem to me like pretty substantial differences.
These were all issues we discussed with Consensus Point prior to launch. To incentivize staff members, we gave a weekly prize to the person who profited most from their trades. We also offered a random-draw prize to encourage everyone to play, regardless of their success in previous weeks.
And they were definitely not lacking in expertise! Everyone who played was a member of the core editorial team. They are all very familiar with the ups and down of New Scientist sales. They also understand the process by which we try and create covers that appeal to our readers.
The article is now up on our website. Registration required, but it's free to access:
The system is biased in favor of long shots, probably intentionally. If you parked $27k for a year betting against Paul's nomination, you could only make $630. And you have to pay intrade $60 for your account. If you only put up $1k, you wouldn't move the price and you'd make the full 3%, but that wouldn't cover account fees.
So the numbers are useless for the calculation Evan wants to do. But I wouldn't be surprised if Paul had high electability on a conditional market, for the reason David Pennock gives.
I don't think this has anything to do with "wishful thinking" from Ron Paul fans trying to get his name out there.
There isn't an "electability" contract at Intrade. "Electability" is the odds that a candidate will become President if that candidate is nominated. Electability is computed by dividing presidential odds by nomination odds. Electability = Presidential Odds / Nomination odds.
This makes me wish I was in an area where the legality of Intrade was clear. This seems pretty obviously to be a product of wishful thinking and attempts to shove Paul's name out there by his fans. I'd love to trade on this and bilk them out of a lot of cash. I'll applaud anyone else who does so.
Jim, I agree that Consensus Point software is sophisticated, and the approach it embodies is superior to other ways to "average" opinions. But the effect of such better averaging is weak compared to the effects of the participants' degrees of expertise, incentives, and attention. (I corrected the typo in the post.)
I'm one of the reporters who worked on the New Scientist article. Thanks for your comments on our work.
I'd like to clarify a couple of points for readers who might be confused about the market we used. Although we did not talk to Robin, we did ask the company at which he is chief scientist -- Consensus Point -- to run our prediction market. They agreed and very kindly put a lot of work into setting up and running the market. We chose Consensus Point partly because of Robin's affiliation with the company and the role he played in the development of prediction markets.
The Consensus Point system is quite sophisticated and gives traders a great deal of flexibility when it comes to making and recalibrating forecasts. So to describe the process as "averaging top-of-the-head opinions by 25 random distracters [sic] staffers" probably doesn't do it justice.
I like your site Evan. Useful. Nice, thanks. Note you might want to discuss the issue that correlation does not necessarily equal causation. For example, it could be that in a world where Ron Paul is nominated, some highly significant event favoring one of his causes happens.
"I’d have expressed little confidence that averaging top-of-the-head opinions by 25 random distracters staffers would beat the concentrated attention of professional collectors and analysts of data."Do you by any chance know how did the mode, rather than the average, fare?
I ask this because I organized a few months ago a "wisdom of crowds" game with political science undergraduates in which they had to guess the percentage with which various proposed bills had been voted in Parliament, and while the average of their answers was, well, average, the mode won almost all the rounds.
Speaking of prediction markets, Intrade shows that Ron Paul is in 1st place among all Republican candidates in terms of electability. His electability is currently at 93%, which is far greater than any of the other candidates:
A new site for tracking pundit predictions. But as my link indicates, pundits have suffered no consequences for terrible predictions, so maybe folks just don't care.
what if no two people gave the same answer?
mode
TGGP, you'd want to allow and encourage broad participation, by all the different people and groups that make, consume, or inform forecasts.
Jim, happy to accept that your market participants are not amateurs, and that they were given some incentives. But the other contest participants were world class professionals, competing for the valuable publicity prize of New Scientist's implicit endorsement. Those seem to me like pretty substantial differences.
These were all issues we discussed with Consensus Point prior to launch. To incentivize staff members, we gave a weekly prize to the person who profited most from their trades. We also offered a random-draw prize to encourage everyone to play, regardless of their success in previous weeks.
And they were definitely not lacking in expertise! Everyone who played was a member of the core editorial team. They are all very familiar with the ups and down of New Scientist sales. They also understand the process by which we try and create covers that appeal to our readers.
The article is now up on our website. Registration required, but it's free to access:
http://www.newscientist.com...
The system is biased in favor of long shots, probably intentionally. If you parked $27k for a year betting against Paul's nomination, you could only make $630. And you have to pay intrade $60 for your account. If you only put up $1k, you wouldn't move the price and you'd make the full 3%, but that wouldn't cover account fees.
So the numbers are useless for the calculation Evan wants to do. But I wouldn't be surprised if Paul had high electability on a conditional market, for the reason David Pennock gives.
If you were marketing to a business, what would be example of potential uses for a prediction market as a "forum" rather than method?
I don't think this has anything to do with "wishful thinking" from Ron Paul fans trying to get his name out there.
There isn't an "electability" contract at Intrade. "Electability" is the odds that a candidate will become President if that candidate is nominated. Electability is computed by dividing presidential odds by nomination odds. Electability = Presidential Odds / Nomination odds.
This makes me wish I was in an area where the legality of Intrade was clear. This seems pretty obviously to be a product of wishful thinking and attempts to shove Paul's name out there by his fans. I'd love to trade on this and bilk them out of a lot of cash. I'll applaud anyone else who does so.
Jim, I agree that Consensus Point software is sophisticated, and the approach it embodies is superior to other ways to "average" opinions. But the effect of such better averaging is weak compared to the effects of the participants' degrees of expertise, incentives, and attention. (I corrected the typo in the post.)
I'm one of the reporters who worked on the New Scientist article. Thanks for your comments on our work.
I'd like to clarify a couple of points for readers who might be confused about the market we used. Although we did not talk to Robin, we did ask the company at which he is chief scientist -- Consensus Point -- to run our prediction market. They agreed and very kindly put a lot of work into setting up and running the market. We chose Consensus Point partly because of Robin's affiliation with the company and the role he played in the development of prediction markets.
The Consensus Point system is quite sophisticated and gives traders a great deal of flexibility when it comes to making and recalibrating forecasts. So to describe the process as "averaging top-of-the-head opinions by 25 random distracters [sic] staffers" probably doesn't do it justice.
Did you mean mode or median? Median also seems to do better than average.
I like your site Evan. Useful. Nice, thanks. Note you might want to discuss the issue that correlation does not necessarily equal causation. For example, it could be that in a world where Ron Paul is nominated, some highly significant event favoring one of his causes happens.
"I’d have expressed little confidence that averaging top-of-the-head opinions by 25 random distracters staffers would beat the concentrated attention of professional collectors and analysts of data."Do you by any chance know how did the mode, rather than the average, fare?
I ask this because I organized a few months ago a "wisdom of crowds" game with political science undergraduates in which they had to guess the percentage with which various proposed bills had been voted in Parliament, and while the average of their answers was, well, average, the mode won almost all the rounds.
Speaking of prediction markets, Intrade shows that Ron Paul is in 1st place among all Republican candidates in terms of electability. His electability is currently at 93%, which is far greater than any of the other candidates:
http://laboratoryofthestate...