Tag Archives: Prediction Markets

College Admission Markets

By Robin Hanson · December 2, 2012 2:50 pm · Comments (87)

This article by Ron Unz is long and rambles a bit, but deserves its provocative reputation. It offers data suggesting that over the last few decades the most elite US colleges have had systematically biased admissions, against asians and for jews, when measured against other standards, like tests and top math/sci competitions. Given the strong academic rhetoric against racial discrimination, you might expect this to cause a fervor, and to result in big changes soon. But I don’t expect much soon – most academics are from those schools, and benefited from those biases, loud complaining isn’t the asian style, and the larger society doesn’t much care because this discrimination is mostly limited to these schools.

The problem comes mainly from granting discretion to admissions personal to make subjective judgements. One solution is to just use objective features like test scores. But Unz worries about ambitious kids wasting their youth in mostly useless test prep. Also, application packets may contain other useful but harder to read clues about promising students. Unz instead prefers to admit “qualified” students at random, at least for most of the slots. But once everyone knew for sure that the elite schools didn’t actually have much better students, it isn’t clear why they would remain the elite schools.

As usual, my solution involves prediction markets. As I posted here five years ago, we could hide clearly identifying info about students, post their application packets to the web for all to see, and let anyone bet on the consequences of each student going to each school. Students might care about their chance of graduating, their income later, and some measure of satisfaction. Elite schools might care more about the chances of students being “successful” someday. Different schools might use different measures of success, such as with different weights for achievement in sports, politics, business, arts, etc. Schools could admit the students with the best chance to succeed by their measure, and students could apply to and then go to the school giving the best chance if achieving their goals. Or students could not go to school at all, if that was estimated to be best.

Of course speculators will favor students showing concrete signs of future success, and so ambitious students would spend their youth trying to achieve such signs. But instead of locking in particular limited metrics like standard test scores, where prep efforts are mostly wasted, this process would create an open competition to find signs of future success where efforts to gain them are more useful. After all, your chance of success later should be higher the more the signs you pursue push you to gain useful skills and habits in the process.

Yes it would be hard to get people to accept that such markets are accurate and hard-to-manipulate enough for this purposes. But equally hard, I expect, would be getting elite schools to say explicitly what sort of success they most want from students. They probably pretend to care more about admirable success, like being a famous writer, than they actually do.

Added 8p: Regarding anonymity, an obvious solution is for the official application to be completely public. Usually only a small fraction of the relevant application info will be things that are better kept private. Regarding that info, the applicant can just reveal that extra private info to a few trusted folks who are willing to trade in these markets. Markets do not need all traders know all relevant info to work well.

GD Star Rating
loading...
Tagged as: , ,

Zitzewitz The Wise

By Robin Hanson · November 29, 2012 2:30 pm · Comments (47)

Eric Zitzewitz on why the CFTC cracked down on Intrade:

Why prohibit something so harmless? After all, U.S. policy already allows many forms of gambling that have significant social costs and produce no useful information. The real reason may be found in debate surrounding another recent setback for prediction markets, Congress’ last-minute modification of the Dodd-Frank financial reform bill in 2010 to prohibit markets on box-office receipts planned by the Hollywood Stock Exchange (HSX).

The plug was pulled after lobbying by the Motion Picture Association of America, which argued that a prediction market forecasting a poor movie box office could lend an “aura of financial authenticity to gossip.” Translation: Even big-budget failures often have one big weekend before word of mouth kicks in, and information aggregated by prediction markets may deprive them of that.

The bigger threat is to the executives who green-light the flops in the first place. The track record of even the play-money version of HSX is quite good ‑ a real money version would presumably be better. If a prediction market can forecast a poor box office well in advance of production, it raises questions about why an executive cannot. A consultant told me about a software company that ended an internal prediction market that forecast the success of the company’s products. The problem was not that it failed to work ‑ it worked all too well, and that raised awkward questions for the egos involved. (more)

Yes this is a big reason why most firms don’t want prediction markets on internal issues, and why the film industry lobbied to prevent film futures. But I find it harder to see as a big reason the CFTC prevents Intrade, and earlier Nadex, from betting on elections. That seems more simply explained by a general public aversion to what it sees as “gambling.”

GD Star Rating
loading...
Tagged as: ,

Was Intrade being manipulated over the last month?

By Robert Wiblin · November 7, 2012 12:06 pm · Comments (72)

Intrade’s betting odds on the 2012 presidential election have differed significantly from those available elsewhere. For the 48 hours preceding the election, the difference in the implied probability of Obama winning on Intrade relative to other betting agencies like Betfair, was 8 to 15 percentage points. This persisted until a large share of Ohio votes had been counted and Colorado and New Mexico were starting to count, at which point the difference quickly evaporated. Over the previous 3 weeks or so, the difference had moved in the range of 5 to 10 percentage points.  The same distortion was observed in favour of McCain during the election in 2008, though to a lesser extent.

This provided an opportunity to make substantial money by betting on Obama on Intrade and Romney elsewhere – a so called dutch book, or ‘arbitrage‘. I joined some colleagues at 80,000 Hours doing this yesterday to earn money for our favourite cost effective charities. We each walked away with about $500 after all of the associated fees. Eyeballing it, a dutch book is profitable, ignoring the cost to your time, if the probability gap is larger than 3 percentage points; below that, the fees involved will eat up your winnings.

Why was this possible? I don’t have a good answer, but I can suggest one possibility. Some noteworthy aspects of the situation are:

  • Americans can’t deposit money into Intrade using credit and debit cards – they have to use bank transfers.
  • Bank transfers take at least two days to arrive and cost over $20.
  • Everyone else can choose between cards and bank transfers.
  • Cards are instantaneous and free (if denominated in US dollars anyway) but have a $2,000 deposit limit in the first month, and $5,000 thereafter.
  • It takes at least a day, probably two, to open a new Intrade account and have it approved.
  • There are other significant barriers to entry – knowing about the issue, learning about the fees, opening an account with another betting agency and finally having the time and confidence to correctly place the hedge.
  • Intrade seems very widely covered by the US media.

A single person with a huge amount in their account from a wire transfer could manipulate the market by selling Obama’s shares down, or buying Romney’s up. This appeared to be happening in the 67-72% likelihood range in which Obama was stuck for a long period of time, while other larger agencies were placing him around 82%. Several people on Intrade’s forum spotted what they thought were abnormally large bids for Romney’s stock.

Once someone started doing this, it would take at least two days, probably three, for a wealthy or ambitious person to respond by wiring in enough money to bet against them. They would have to hope that the manipulation persisted long enough for them to profit from it. Until then, people outside the USA would be limited to putting at most $2000 or $5000 into their accounts, which is barely worth the effort for someone with the required skill. Someone could plan to do this over the last few days of the election without generating much resistance.

The volume yesterday on Obama’s Intrade shares was about 600,000. If all of those trades involved one person, who was losing 10 percentage points on each share, they would have blown $600,000 to keep Obama’s odds down. The volume over the previous three weeks is hard to read from Intrade’s graphs, but looks to be about the same again. So a single cunning person willing to lose $1 million could have singlehandedly driven the price difference, if they wanted to influence perceptions of the race and encourage voter turnout. Out of the $6 billion spent on the election so far, that’s not a big investment. Intrade will face the risk of this until they make it easier for wolves to fund their accounts and go out hunting sheep.

Weaknesses of this theory are:

  • Why didn’t manipulation over the previous three weeks prompt someone to move a large sum onto Intrade in anticipation?
  • Why haven’t wealthy Obama supporters attempted the same trick?

Nonetheless, I think this is more likely than a broad pool of Intrade participants being enthusiastic about Romney against all the evidence, and unaware that they could get better odds elsewhere.

If I were a Democrat supporter with a lot of money, I would plan to profit from similar situations in the future while simultaneously improving Intrade’s performance.

GD Star Rating
loading...
Tagged as: , ,

Wanted: Elite Crowds

By Robin Hanson · November 6, 2012 8:45 am · Comments (3)

This weekend I was in a AAAI (Association for the Advancement of Artificial Intelligence) Fall Symposium on Machine Aggregation of Human Judgment. It was my job to give a short summary about our symposium to the eight co-located symposia. Here is what I said.

In most of AI, data is input, and judgements are output. But here humans turn data into judgements, and then machines and institutions combine those judgements. This work is often inspired by a “wisdom of crowds” idea that we often rely too much on arrogant over-rated experts instead of the under-rated insight of everyone else. Boo elites; rah ordinary folks!

Many of the symposium folks are part of the IARPA ACE project, which is structured as a competition between four teams, each of which must collect several hundred participants to answer the same real-time intelligence questions, with roughly a hundred active questions at any one time. Each team uses a different approach. The two most common ways are to ask many people for estimates, and then average them somehow, or to have people trade in speculative betting markets. ACE is now in its second of four years. So, what have we learned?

First, we’ve learned that it helps to transform probability estimates into log-odds before averaging them. Weights can then correct well for predictable over- or under-confidence. We’ve also learned better ways to elicit estimates. For example, instead of asking for a 90% confidence interval on a number, it is better to ask for an interval, and then for a probability. It works even better to ask about an interval someone else picked. Also, instead of asking people directly for their confidence, it is better to ask them how much their opinion would change if they knew what others know.

Our DAGGRE team is trying to improve accuracy by breaking down questions into a set of related correlated questions. ACE has also learned how to make people better at estimating, both by training them in basic probability theory, and by having them work together in teams.

But the biggest thing we’ve learned is that people are unequal – the best way to get good crowd wisdom is to have a good crowd. Contributions that most improve accuracy are more extreme, more recent, by those who contribute more often, and come with more confidence. In our DAGGRE system, most value comes from a few dozen of our thousands of participants. True, these elites might not be the same folks you’d have picked via resumes, and tracking success may give better incentives. But still, what we’ve most learned about the wisdom of crowds is that it is best to have an elite “crowd.”

GD Star Rating
loading...
Tagged as: , ,

How To Vote

By Robin Hanson · August 28, 2012 7:45 pm · Comments (61)

I’m a professor of economics who has published on politics, and in ten weeks the US will elect a president, and congress-folk. So a reasonable test of my supposed specialized knowledge is: do I have anything clearly useful to tell ordinary folks about how to vote in this election?

Well first, I have said that politics isn’t about policy, i.e., that we quite reasonably care more about how our political views make us look to our associates than how they effect policy outcomes. I’ve also said a lot about overconfidence, which applies in spades to politics. Being overconfident usually looks better, even it leads to less accurate judgements. So in general I can recommend: just confidently take positions that look good to your associates; forget about policy outcomes.

But what if you were to insist on trying to figure out which US presidential and Congressional candidates to vote for based on expected policy outcomes? And what if you were to further insist on avoiding overconfidence, seeking as much as possible to avoid relying on your own likely-overconfident judgements. Perhaps, for example, the image you want to project to your associates is of fair-minded broad concern, and well-calibrated rationality.

In this case, if you guess that your relevant information is roughly average or worse than average, you have an obvious solution: don’t vote. But, perhaps you want to signal a bit more confidence and concern than this suggests.

Another simple strategy is to search for an immediate associate who both clearly shares most of your interests, and who seems better informed than you. Then just copy their vote. But perhaps you don’t want to appear very submissive to this person. So let us assume that you want to signal that you are informed and autonomous, while still creating good voting outcomes, and continue.

Alas, we still don’t have presidential decision markets, that estimate important policy outcomes, such as GDP, unemployment, oil prices, etc., conditional on who becomes president. It would only take a few tens of thousands of dollars for someone to create these, at say Intrade. And then I could offer great simple clear robust advice – vote for whomever markets rate better. But, ok, this system failing is not your problem, so let’s continue.

The thing you probably know best is your own life. So a good simple strategy is to vote “retrospectively,” i.e., for incumbents if your life goes well, and against them if your life goes badly. The more voters who do this, the stronger incentives incumbents have to make most lives go well.

For life quality extremes this advice is clear, but what if your life is near the middle? What should be the cutoff between a good and bad life? One simple reference is how you expected your life to go when incumbents were elected – reelect them if your life has gone better than expected.

Now you might do a little better if you could broaden your judgment to how life has gone for more people you care about. But if you care about a lot of distant folks, you’ll face the problem that you know a lot less about how distant folk lives are going. I could just tell you that most experts agree that the US economy has done worse than experts expected when Obama was elected. But to rely on that, you might have to trust me.

You could also do a bit better if you focused on the outcomes where particular incumbents are most responsible. For example, US presidents have more influence on foreign policy, while Congress has more influence on domestic policy. US politicians have more effect on what happens in the US than in Europe. Politicians have less influence on earthquakes or hurricanes. And so on. You might also substitute the initial expectation you should have had, for the one you did have, if you can figure out that. But you have to know more than most people do to implement such corrections well, and they only moderately improve incumbent incentives.

So, as a professor of economics who has studied politics, my advice is to not vote if you know an average amount or less, to copy a better informed close associate if you are willing to appear submissive, and otherwise to just reelect incumbents when your life goes better than you expected. And if you care a lot more about the outcome than most do, help create presidential decision markets, so other info-seekers will have a better place to turn.

GD Star Rating
loading...
Tagged as: , ,

Prediction Markets “Fail” To Mooch

By Robin Hanson · July 19, 2012 12:00 am · Comments (10)

What is new about prediction markets? To many, the key new idea is “crowd-sourcing”, which to many means that if you can “gamify” your problem enough, hobbyists will solve it for fun, much cheaper than if you had to hire employees. To me the key new idea is instead the “information prize”, a way to offer to pay others to find the info you want. (If they don’t find, you don’t pay.)

Intrade is a typical gamified market – you might pay Intrade a little to put up a question, and if traders like it they’ll pay Intrade to answer your question. A market-maker-driven corporate prediction market (such as firms pay Consensus Point to set up) is very different – to answer key business questions, firms pay lots to create markets, to fund employee participants, and to subsidize market makers.

Alas there’s been a lot more interest over the last few years in the get-work-for-free concept than in the pay-for-info concept. And more alas, recent discussions of “prediction market failures” are mostly on their failures to mooch. Case in point #1, Casey Mulligan today:

The efficient-prediction perspective presumes that the market exists on a scale large enough to create significant rewards for those with accurate predictions. Another perspective, “no-trade,” says prediction market participation will be low, if not zero, because traders suspect that a person would take the opposite side of their trades only because he had superior knowledge about the outcome. A market cannot survive if its only participants are “insider traders.”

Paradoxically, a prediction market cannot succeed unless it multitasks – it must serve an additional purpose separate from predictions, so that the participants with information about the outcome have counterparts to take the other side of their trades. In the case of sports and political markets, that additional purpose is entertainment – people enjoy engaging in the activity and are willing to participate even if their expected profits are zero or negative. These people are participating for various reasons other than prediction. (more)

No, a market can single-task, with no other function than prediction, and no other trader motive than selfish financial profit, if someone who wants the info will pay to subsidize the market. It is only when you want people to answer your question for free that you’ll have to piggyback on their having some other reason to trade.

Of course there’s no guarantee that your willingness to pay for some info exceeds other folks’ cost to supply that info. Supply and demand curves need not intersect at a positive quantity. But that’s hardly a failure of an info exchange mechanism.

Case in point #2, Snowberg, Wolfers, & Zitzewitz’s new paper “Prediction Markets for Economic Forecasting“, for the Handbook of Economic Forecasting, also sees not getting stuff for free as “failure”:

3.1 Why They (Sometimes) Fail

Although prediction markets generally function quite well, design flaws sometimes prevent reliable forecasts. These flaws generally lead to a lack of noise traders (or thin markets) that reduces incentives for discovering, and trading on the basis of, private information. In order to attract noise traders, the subject of a prediction market must be interesting and information must be widely dispersed. Prediction market contracts must be well specifed, so that it is clear when they will (and will not) pay off. However, this specicity may be in tension with making a contract interesting for traders. …

Noise traders may quite rationally choose not to trade in markets where there is a high degree of insider information. For example, despite the high intrinsic interest in who a Supreme Court nominee will be, markets on this topic have routinely failed. This may be due to the fact that most traders are aware that there are very few people with actual information on who the President’s choice will be. This anecdote underlines the importance of prohibiting insider trading: for instance, a market to predict the Institute for Supply Management’s (ISM’s) business confidence measure would be unlikely to function if it were well known that ISM employees were trading in it. …

Corporations are attracted to prediction markets as they can potentially pass unbiased information from a company’s front-line employees to senior management. However, many questions of interest to executives are not widely interesting, nor are there many employees that have relevant information. This creates a lack of liquidity in markets, perhaps leading to no trading, or, worse, inaccurate predictions. Microsoft has responded to this problem by using a market-making algorithm. ….

The stories of failure above lead to some straight-forward rules for designing prediction markets: make sure the question is well-defined, that there is dispersed information about the question, and that there is sufficient interest in the question to ensure liquidity. (more)

Noise traders are traders who subsidize your market for free, for reasons of their own, such as risk-hedging, idiocy, etc. If you fail to attract noise traders, you fail to get their free subsidy. But you can still offer to directly pay for your info, by subsidizing the market, as the Microsoft sentence in the quote indicates. Similarly, if employees find executive questions uninteresting, that just means they won’t answer such questions as freely in their spare time. But that hardly means firms can’t pay employees to address key firm questions. Here we are only talking about a “failure” of prediction markets to mooch stuff for free!

Snowberg, Wolfers, & Zitzewitz also err in saying that sometimes there is “no” info:

An extreme form of information not being widely dispersed is when there is no information at all to aggregate. For example, prediction markets on whether weapons of mass destruction (WMDs) would be found in Iraq predicted they would very likely be found. The false confidence that could be inspired by such an estimate ignores the fact that there was no information being aggregated by these markets. That is to say, it was unlikely that anyone in Iraq, who might actually have some information (perhaps based on rumors, past experience, or informal discussions with friends and relatives in the government) about whether Iraq’s WMD program was likely to exist or not, was trading in these markets.

Yes particular info, such as direct personal observations of WMD efforts, existed out there somewhere, but was prohibitively expensive to supply. That is, the price to buy info offered by the Intrade markets was too low to induce folks with such info to supply it. But that hardly meant there was no info available on the subject! There were a great many cheap but relevant clues available for making rough guesses on the subject, and I’m pretty sure that a lot of trading in that market was based on such clues.

When you offer to pay a certain price for info, an efficient info exchange mechanism will typically induce some supply of that info, but only up to the point where the marginal cost of supplying info reaches the price you have offered to pay. It is no failure of an exchange mechanism when buyers cannot always buy everything they want at as low a price as they want.

Finally Snowberg, Wolfers, & Zitzewitz end with this stunner:

We believe the real promise of prediction markets comes not from their ability to predict particular events. Rather, the real promise lies in using these markets, often several at a time, to test particular economic models, and use these models to improve economic forecasts.

This seems like saying the real promise of democracy is that academics can study votes to refine their theories of human behavior. Really?!

Added: Alas The Economist blog swallows this “no info” error whole.

Added 20July: Scott Sumner says people similarly say that his NGDP futures proposal “fails” if they don’t come for free.

GD Star Rating
loading...
Tagged as:

Leonhardt Blows It

By Robin Hanson · July 7, 2012 2:00 pm · Comments (17)

Imagine someone said:

Of course I believe in science – I’m no nut job. I’m a modern guy. But scientists sometimes get it wrong, so we can’t just believe everything they say – we have to use our judgement. For example, my judgement tells me that astrology just makes sense. Well not today – today’s horoscope suggests I drink less, while I know I can handle my benders. But usually my horoscope feels right. And usually I feel no objection to what scientists say. Which is what I mean when I say that I believe in science.

Yes, every source errs sometimes, making it seem oh so sophisticated to say you don’t take sides, you just use your judgement in each case. But that is often just an excuse to believe whatever you feel like. On prediction markets, David Leonhardt sounds similar:

The odds at Intrade … continued to show about a 75 percent chance that the law’s so-called mandate would be ruled unconstitutional, right up until the morning it was ruled constitutional. … Today, mocking Intrade, ideally on Twitter, is a sign of sophistication. …

The early successes of prediction markets were notable. … But the crowd was not everywhere wise. For one thing, many of the betting pools on Intrade and Betfair attract relatively few traders, in part because using them legally is cumbersome. … The thinness of these markets can cause them to adjust too slowly to new information.

And there is this: If the circle of people who possess information is small enough — as with the selection of a vice president or pope or, arguably, a decision by the Supreme Court — the crowds may not have much wisdom to impart. “There is a class of markets that I think are basically pointless,” says Justin Wolfers. …

But such schadenfreude raises a question: once you accept that prediction markets are flawed, do you turn back to the inside experts? ALAS, the experts’ overall record remains as poor as the behavioral economists maintained — and often worse than the markets’ record. …

The answer, I think, is to take the best of what both experts and markets have to offer, realizing that the combination of the two offers a better window onto the future than either alone. Markets are at their best when they can synthesize large amounts of disparate information, as on an election night. Experts are most useful when a system exists to identify the most truly knowledgeable. …

Nate Silver … has found that a simple average of well-known economic forecasts is substantially more accurate than individual forecasts. Other times, the approach might involve as much art as science — and, again, the Internet allows for strategies that once would have been impossible.

Think for a moment about what a Twitter feed is: it’s a personalized market of experts (and friends), in which you can build your own focus group and listen to its collective analysis about the past, present and future. An RSS feed, in which you choose blogs to read, works similarly. You make decisions about which experts are worthy of your attention, based both on your own judgments about them and on other experts’ judgments. (more)

No, the vast majority of folks should not trust the vague impression on a subject they glean from a Twitter or RSS feed, over active prediction market prices. They shouldn’t even average the two. I’m confident that an empirical test of forecasts based on such impressions, or averages, would find them less accurate. Alas, folks like Leonhardt might then say “And that’s just why you must use your judgement about when when to use your judgement.”

Yes, in a sense, you always use your judgement – if you take my advice to rely on prediction market prices instead of Twitter feed impressions,  your judgement will have to approve of that. But using your judgement isn’t the same as accepting your case-specific intuitions – you usually can and should judge them unreliable.

Also, prediction markets just are not “crowds” in contrast to “experts” – the whole point of prediction markets is to get participants to self-select as the true experts. Don’t participate unless you think you know more than other participants, and those who actually do know less lose on average and get slowly pushed out.

Yes, you should be wary of “prediction markets” where no one trades, or limited to the kids from Mrs. Calloway’s seventh grade civics class, or using a play money no one cares about. Not everything called a “prediction market” is one. But the Intrade market on the Obamacare court case was an active valid market, on an appropriate subject. When it assigned a 75% chance to an event it was saying real loud that it would be wrong 1/4 of the time. And studies have consistently found such markets are well-calibrated in this way. What more do you want?

Yes, Intrade markets on court cases are unlikely to extract inside court info, and would be less accurate than sources with access to such info. But do you really think your Twitter feed has better access? Intrade traders watch Twitter, and incorporate what info they find into prices as best they can. Skeptics who tweet their disagreements but aren’t willing to bet can’t be very confident.

Yes, prediction markets can’t be reliable sources unless some people at some times think they are unreliable, and bet on that opinion. It is those with enough confidence in their disagreements to bet that make such markets accurate. If you are such a person, more power to you. But if you are not such a person, you will almost always get more accurate estimates by just trusting the current prices of an active prediction market, relative to forming a vague impression based on your Twitter feed.

Of course, on subjects like major court cases, most people care about others things besides accuracy. Twitter feeds can connect you to people, helping you to form and show your allegiances. For such social purposes, prediction markets are worse. But since people can’t usually admit such priorities, they have to make up excuses, such as about listening to Twitter feeds to aggregate a vague ineffable wisdom of expert crowds.

GD Star Rating
loading...
Tagged as:

We Can Do Low-Treewidth Combinatorial Prediction Markets!

By Robin Hanson · July 2, 2012 8:40 am · Comments (28)

In my last post I said I hoped prediction markets would become

an “our answers” institution with easily-found accurate answers on as many questions as possible.

Today prediction markets’ main problem is laws banning them (and customs limiting firm interest in internal markers). Alas as an academic, I can’t do much to change such laws. But I can work to improve the basic tech, for the day when prediction markets are legal. Yesterday I also said:

[Here's] one way to expand the range of questions prediction markets can cheaply answer: start with a set of base questions, and then let users ask and answer questions from the vast space of combinations of those base questions. For example, starting with a base consisting of all the specific future readings of all weather stations, users could ask most any weather question of interest, such as whether this next winter will be colder where they are living now, or in the particular city where they are thinking of moving. In my next post I’ll talk about a big advance my research group has achieved in the implementation of such combinatorial prediction markets.

The DAGGRE project that I’ve been part of for over a year now has been working to advance the theory and practice of combinatorial prediction markets. Within a few months we will field an edit-based system where users can browse current answer estimates, and for each estimate can:

  • Edit the value. After you change an estimate to a new value, estimates that users see on all questions are Bayes-rule updates from that new value.
  • Assume a value. After you assume a value for this estimate, all estimates you see on all questions are conditional on this assumption.

To support this interface, we have three computing tasks:

  1. When a user has made assumptions A and browses to a possible question answer T, compute and show the current value v = P(T|A).
  2. To see how far this user could change this number, compute the edit values, v-, v+, in each direction that give him or her zero assets in some state.
  3. To show the user if he or she is currently long or short on this topic, compute and compare his or her expected assets given A&T, and given A&notT.

The problem is, even though a simple math formula says how a user’s (state-dependent) assets change when he or she makes such an edit, it is in general infeasible to quickly calculate the above numbers for more than a few dozen base questions.

To make computing feasible, we must somehow limit the space of allowable question answers. So the big question is: what limits will allow as many as possible of the combinations that user will typically want to edit, while still enabling accurate computation for allowed combinations.

One approach is to group base questions into sets of roughly twenty or less, and allow arbitrary combinations within each group, but no combinations between different groups. This is feasible, but we can do better, via Bayesian (or Markov) nets.

These nets limit the space of possible answers by imposing conditional independence assumptions. In a net of (directly) connected variables (i.e., questions), all variables are assumed independent of unconnected variables, conditional on connected variables. A standard (junction tree) algorithm allows exact computation of conditional probabilities for nets with a low “treewidth” (roughly, the number of variables you’d have to merge to make the net into a tree).

Of course that only covers task #1 above; what about the other two? My DAGGRE group has just published a paper (also here), to be presented at a conference (UAI) in August, showing how to exactly (well, up to machine precision) compute tasks #2,3 in this same situation. (Task #2 needs a low treewidth, but #3 works on any net.) We’ve implemented this algorithm and shown that an ordinary laptop can handle a thousand variables and a treewidth of ten in a fraction of a second. Within a few months this will be the (public domain) backend of our public edit-based combinatorial prediction market with hundreds of questions and active users. I’ll announce it here on this blog when it is ready to show.

Of course our real best-answers don’t actually fit in a low treewidth net. So we have more work to do: finding efficient approximations for tasks #1,2 in more realistic nets. There is already a large literature on ways to compute conditional probabilities in high treewidth Markov nets; we just need to study and choose among them. And I already know of a very promising general way to do task #2 well enough; we just need to try it.

Even when we can handle more realistic nets, we will still have to limit their size to make computing feasible. So we’ll need ways to let users edit the net structures – to tell us where to add and delete connections. I have some ideas here, but we are far from a satisfactory solution.

Even so, progress has been surprisingly rapid, and we have good reasons to expect continued rapid progress. We seem just a few years away from having the tech to field general robust combinatorial prediction markets! Then we’ll just have to figure out how to make it all legal.

GD Star Rating
loading...
Tagged as: ,

Finding “Our” Beliefs

By Robin Hanson · July 1, 2012 9:45 pm · Comments (17)

If only you are interested in a topic, you’ll have to think it through for yourself, or pay someone else to think on it. But if many folks are interested in your topic, you might hope to share the thinking work with others.

Some social institutions seem to serve this “our beliefs” function. Today you can see if your question is answered in wikipedia, you can search a library for answers in respected books or journals, and you can call up an expert credentialed in a related area to see if they know of an answer.

Of course these institutions are imperfect. Calling experts is expensive, quick searches only find some of the many differing opinions out there, and encyclopedia answers, while unique, only address a limited range of questions. If you find an answer you know is wrong, it can take a lot of work to change it. You might have to devote a whole career to the attempt, and even then you might not be rewarded for the effort.

Ideally, we’d want an “our answers” institution with easily-found accurate answers on as many questions as possible. Hopefully, anyone could ask any question, and answers would be consistent with each other and across time. If incentives to give accurate answers were strong enough, we might even let anyone correct any answer.

Prediction markets might allow such a better answers institution. Ordinary financial prices offer consistent unique answers that anyone can fix, but for a typical ordinary question, it is very hard to figure out which price combinations might answer it. In contrast, prediction market questions can be expressed in simple ordinary language.

If (money-based) prediction markets were legal, anyone could add a new question for a modest fee (<$100), and quickly get unique answers consistent with all other questions. Anyone could fix any answer, and would have incentives to do so accurately.  Or anyone could pay to make any answer more accurate. So far, tests have found prediction markets to be consistently as or more accurate than other prediction institutions with similar resources.

Of course ordinary prediction markets do have one big limitation: they only directly answer questions that eventually become clear for other reasons. But this allows more than it might seem. For example, because we will later know which candidate is elected by what margin, and how big is the post-election unemployment rate, prices today can say which candidate is expected to most help unemployment if elected.

This example suggests one way to expand the range of questions prediction markets can cheaply answer: start with a set of base questions, and then let users ask and answer questions from the vast space of combinations of those base questions. For example, starting with a base consisting of all the specific future readings of all weather stations, users could ask most any weather question of interest, such as whether this next winter will be colder where they are living now, or in the particular city where they are thinking of moving.

In my next post I’ll talk about a big advance my research group has achieved in the implementation of such combinatorial prediction markets.

GD Star Rating
loading...
Tagged as:

Who Wants Unbiased Journals?

By Robin Hanson · April 27, 2012 11:00 pm · Comments (18)

Five years ago I proposed result-blind peer review, and I revised it later. Brendan Nyham just posted a nice long review of many such proposals, including a recent test at the journal Archives of Internal Medicine:

The … alternate review process was applied to the editorial review that occurred prior to outside peer review. … Of the 46 articles examined, 28 were positive, and 18 were negative. … Ultimately, 36 of the 46 articles (>77%) were rejected. … Editors were consistent in their assessment of a manuscript in both steps of the review process in over 77% of cases. … Over 7% of positive articles benefited from editors changing their minds between steps 1 and 2 of the alternate review process, deciding to push forward with peer review after reading the results. By contrast, … this never occurred with the negative studies. Indeed, 1 negative study, which was originally queued for peer review after an editor’s examination of the introduction and “Methods” section, was removed from such consideration after the results were made available. (more)

So even with two stage review, journal editors are tempted to publish papers with weak methods but positive results. And why not – unless important customers insisted, why would a journal handicap itself by committing itself to not publish such papers, which bring more fame and prestige to the journal.

Journal customers include universities who tenure professors who publish in prestigious journals, and grant givers who prefer grantees who publish similarly. But why should these customers handicap themselves – they also win by affiliating with those who publish papers with weak methods but positive results.

I’ve suggested that academia functions primarily to credential people as impressive and interesting in certain ways, so outsiders, like students and patron, can gain prestige by affiliating with them. If so, and if those who publish weak-method positive-results are in fact more impressive and interesting than those who publish stronger-method negative-results, there is little prospect to get rid of this publication bias.

What is possible is to augment publications with betting market prices estimating the chance each result will be upheld by future research. This would let readers get unbiased estimates on the reliability of research results. Alas, it seems there is no customer willing to pay extra to get such reliability estimates. Most everyone involved in the process mainly cares about signals of impressiveness; few care much about which research results are actually true.

GD Star Rating
loading...
Tagged as: ,