SciCast is holding a new contest:
We’ll be offering $16,000 in prizes for conditional forecasts only made from April 23 to May 22.
SciCast is holding a new contest:
We’ll be offering $16,000 in prizes for conditional forecasts only made from April 23 to May 22.
Worried that you might be wrong? That you might be wrong because you are biased? You might think that your best response is to study different kinds of biases, so that you can try to correct your own biases. And yes, that can help sometimes. But overall, I don’t think it helps much. The vast depths of your mind are quite capable of tricking you into thinking you are overcoming biases, when you are doing no such thing.
A more robust solution is to seek motivated and capable critics. Real humans who have incentives to find and explain flaws in your analysis. They can more reliably find your biases, and force you to hear about them. This is of course an ancient idea. The Vatican has long had “devil’s advocates”, and many other organizations regularly assign critics to evaluate presented arguments. For example, academic conferences often assign “discussants” tasked with finding flaws in talks, and journals assign referees to criticize submitted papers.
Since this idea is so ancient, you might think that the people who talk the most about trying to overcoming bias would apply this principle far more often than do others. But from what I’ve seen, you’d be wrong.
Oh, almost everyone circulates drafts among close associates for friendly criticism. But that criticism is mostly directed toward avoiding looking bad when they present to a wider audience. Which isn’t at all the same as making sure they are right. That is, friendly local criticism isn’t usually directed at trying to show a wider audience flaws in your arguments. If your audience won’t notice a flaw, your friendly local critics have little incentive to point it out.
If your audience cared about flaws in your arguments, they’d prefer to hear you in a context where they can expect to hear motivated capable outside critics point out flaws. Not your close associates or friends, or people from shared institutions via which you could punish them for overly effective criticism. Then when the flaws your audience hears about are weak, they can have more confidence that your arguments are strong.
And if even if your audience only cared about the appearance of caring about flaws in your argument, they’d still want to hear you matched with apparently motivated capable critics. Or at least have their associates hear that such matching happens. Critics would likely be less motivated and capable in this case, but at least there’d be a fig leaf that looked like good outside critics matched with your presented arguments.
So when you see people presenting arguments without even a fig leaf of the appearance of outside critics being matched with presented arguments, you can reasonably conclude that this audience doesn’t really care much about appearing to care about hidden flaws in your argument. And if you are the one presenting arguments, and if you didn’t try to ensure available critics, then others can reasonably conclude that you don’t care much about persuading your audience that your argument lacks hidden flaws.
Now often this criticism approach is often muddled by the question of which kinds of critics are in fact motivated and capable. So often “critics” are used who don’t have in fact have much relevant expertise, or who have incentives that are opaque to the audience. And prediction markets can be seen as a robust solution to this problem. Every bet is an interaction between two sides who each implicitly criticize the other. Both are clearly motivated to be accurate, and have clear incentives to only participate if they are capable. Of course prediction market critics typically don’t give as much detail to explain the flaws they see. But they do make clear that they see a flaw.
Tomorrow I’ll present on prediction markets and disagreement, in Montreal at the NIPS Workshop on Transactional Machine Learning and E-Commerce. A video will be available later.
What is our main problem, bad policy or bad meta-policy? That is, do our collective choices go wrong mainly because we make a few key mistakes in choosing particular policies? Or do they go wrong mainly because we use the wrong institutions to choose these policies?
I would have thought meta-policy was the obvious answer. But CATO asked 51 scholars/pundits this question:
If you could wave a magic wand and make one or two policy or institutional changes to brighten the U.S. economy’s long-term growth prospects, what would you change and why?
And out of the 29 answers now visible, only four (or 14%) of us picked meta-policy changes:
Michael Strain says to increase fed data agency budgets:
BLS data on gross labor market flows … are not available at the state and MSA level, they do not have detailed industry breakdowns, and they do not break down by occupation or by job task. … We also need better “longitudinal” data — data that track individuals every year (or even more frequently) for a long period of time. … The major federal statistical agencies need larger budgets to collect the data we need to design policies to increase workforce participation and to strength future growth. … My second policy suggestion is to expand the … EITC.
Lee Drutman says to increase Congress staff policy budgets:
I would triple the amount the Congress spends on staff (keeping it still at just under 0.1% of the total federal budget). I’d also concentrate that spending in the policy committees. I’d give those committees the resources to be leading institutions for expertise on the issues on which they deal. I’d also give these committees the resources to hire their own experts — economists, lawyers, consultants, etc. But I’d also make sure that these committees were not explicitly partisan.
Eli Dourado says to pay Congress a bonus if the economy does well:
A performance bonus would help to overcome some of Congress’s complacency and division in the face of decades-long economic stagnation. … One good performance metric would be total factor productivity (TFP). … Fernald adjusts his TFP estimate for cyclical labor and capital utilization changes, making his series a better measure. … Members of Congress would earn a $200,000 bonus if the two-year period in which they serve averages 2 percent TFP growth. (more)
First, I propose that our national legislatures pass bills to define national welfare, and fund and authorize an agency to collect statistics to measure this numerical quantity after the fact. … Second, … create an open bounty system for proposing policies to increase national welfare. … Third, … create two open speculative decision markets for each official proposal, to estimate national welfare given that we do or do not adopt this proposal. … If over the decision day the average if-adopted price is higher than the average if-not-adopt price (plus average bid-ask spread), then the proposal … becomes a new law of the land.
It seems to me that Michael, Lee, and Eli feel wave pretty weak wands. Surely if they thought their wands strong enough to cast any policy or meta-policy spell, wouldn’t they pick meta-policy spells a bit stronger than these? (And why is it always more spending, not less?)
By focusing on policy instead of meta-policy, it seems to me that the other 25 writers show either an unjustified faith in existing policy institutions, or a lack of imagination on possible alternatives. Both of which are somewhat surprising for 51 scholars chosen by CATO.
Added Dec3: 3 of the 25 remaining proposals were in the meta-policy direction:
[Regulatory] agencies should be required to present evidence that they have identified a material failure of competitive markets or public institutions that requires a federal regulatory solution, and provide an objective evaluation of alternatives.
The Regulatory Improvement Commission … would have a limited period of time to come up with a package of regulations to be eliminated or fixed, drawing on public suggestions. The package would then be sent to Congress for an up-or-down vote, and then onto the President for signing.
Instead of analyzing whether the [cost-benefit] calculations in a regulatory ledger sum to a positive or a negative number, we need to set a level of [regulatory] complexity that we’re willing to live with, and then decide which positive sum regulations we’re willing to discard in order to stay within that budget. … Crude rules which might well serve, like capping the number of laws and regulations, allowing a new one to be implemented only if an older one is repealed.
I’ve posted twice before when SciCast paid out big. The first time we just paid for activity. The second time, we paid for accuracy, but weakly, as it was measured only a few weeks after each trade. Now we are paying HUGE, for longer-term accuracy. We’ll pay out $86,000 to the most accurate participants, as measured from November 7 to March 6:
SciCast is running a new special! The most accurate forecasters during the special will receive Amazon gift cards:
• The top 15 participants will win $2250 to spend at Amazon.com
• The other 135 of the top 150 participants will win $225 to spend at Amazon.com
Participants will be ranked according to their total expected and realized points from their forecasts during the special. Be sure to use SciCast from November 7 through March 6! (more)
Added: At any one time about half the questions will be eligible for this contest. We of course hope to compare accuracy between eligible and ineligible questions.
Older women often find themselves too old to have kids, and regretting it. Such women would have gained by freezing some eggs when they were younger. But when younger, they didn’t think they’d ever want kids, or thought the issue could wait.
Such women might be helped by an egg futures business, paid to take on this risk for them. Such a business could buy eggs from women when young, freeze them, and sell them back to these same women when old.
Of course, to compensate for the wait and risk that the women wouldn’t want eggs later, this business would have to sell eggs back a high price. But still, if the women bought the egg later, that would show they expected to gain from the deal.
Also, not all women would make equally good prospects. So such a business would focus on women likely to wait too long, be well off, and want kids later. So this business would “discriminate” by class in its purchases, paying more to upper class women. A lot like we now discriminate when we pay more for used clothes, cars, or houses from richer people.
Several people have told me that, while they were not personally offended, they expect others to be offended by such a business. Especially if men were involved in the business – a female only business would offend less. I’m somewhat mystified, which is partly why I’m writing this post. Maybe others can help me understand the objection.
Interestingly, we could add some personal prediction markets, which would probably be legal. For each possible young woman, there could be a market where one buys and sells conditional shares in an egg from that customer. If you owned a conditional share, you’d own a share of the profit from later selling that customer her egg. And you’d owe a share of the cost to buy her egg from her, freeze it, and store it. Imagine the fun buying and selling conditional shares regarding the young women that you know. And the fact that this is a share of a real physical object should make it legal.
Ok, I can see how people might be offended at this last suggestion. After all, there’s a risk that people might have fun on something that is supposed to be serious!
Back in May I said that while SciCast hadn’t previously been allowed to pay participants, we were finally running a four week experiment to reward random activities. That experiment paid big and showed big effects; we saw far more activity on days when we paid cash.
In the next four weeks we’ll run another experiment that pays even more:
SciCast is running a new special! For four weeks, you can win prizes on some days of the week:
On each activity prize day, up to 80 valid forecasts and comments made that day will be randomly selected to win. On each accuracy prize day, your chance of winning any of 80 prizes is proportional to your forecasting accuracy. Be sure to use SciCast from July 22 to August 15!
So this time we’ll compare activity incentives to accuracy incentives. Will we get more activity on days when we reward activity, and more accuracy on days when we reward accuracy? Now our accuracy incentives are admittedly weak, in that we’ll evaluate the accuracy of each trade/edit via price changes over only a few weeks after the trade. But hey, its something. Hopefully we can do a better experiment next year.
SciCast now has 532 questions on science and technology, and you can make conditional forecasts on most of them. Come!
Why do men give women engagement rings? A standard story is that a ring shows commitment; by paying a cost that one would lose if the marriage fails, one shows that one places a high value on the marriage.
However, as a signal the ring has two problems. On the one hand, if the ring is easy to sell for its purchase price, then it detracts from the woman’s signal of the value she places on the marriage. Accepting a ring makes her look mercenary. On the other hand, if the ring can’t be sold for near its purchase price, and if the woman values the ring itself at less than its price, then the couple destroys value in order to allow the signal.
These are common problems with loyalty signals – either value is destroyed, or stronger signals on one side weakens signals from other sides. Value-destroying loyalty signals are very common in couples, clubs, churches, firms, professions, and nations. For example, we might give up poker nights for a spouse, pork food for a religion, casual clothes to be a manager, or old-world customs for a new nation.
A few days ago I had an idea for a more efficient loyalty signal. Imagine that when he was twenty a man made a $5000 bet that he would never marry before the age of fifty. Then when he is thirty-five and wants to marry, he can send a strong signal of his desire to marry just by his willingness to lose this bet. Since the bet is lost to a third party, it doesn’t hinder the bride’s ability to signal her loyalty. And assuming the bet is made at fair odds, the lost bets are on average paid to versions of this man in alternative scenarios where he doesn’t marry by fifty. So he retains the value, which is not destroyed.
Today this approach probably suffers from being weird, so doing this would also send an unwelcome signal of weirdness. But it is only a signal of one’s weirdness when one made the bet – maybe one can credibly claim to be less weird later when marrying. And the bet would remain potent as a signal of devotion.
There are many related applications. For example, a young person who bet that they would never join a religion might later credibly signal their devotion to that religion, and perhaps avoid having to eat and dress funny to show such devotion. Also, someone who bet that they would never change countries might signal their loyalty when they moved to a new nation. To let my future self signal his devotion to his political party, perhaps I should bet today that I’ll never join a political party. Do I have any takers?
Added 20July: Of course the need to lose a bet to get married would discourage some from getting married. But the same harm happens for any expectation of needing to send a loyalty signal if one gets married. This effect isn’t particular to bets as loyalty signals; it happens for all kinds of loyalty signals.
Mechanically one way to implement marriage bets as loyalty signals would be for parents to buy their sons male spinster insurance, which pays money to the son when he is fifty if he never marries, and otherwise gives him a nice visible cheap pin/brooch when he gets married. His new wife can wear the pin to brag about his devotion. The pin might be color coded to indicate how much money he sacrificed.
The first principle is that you must not fool yourself — and you are the easiest person to fool. Richard Feynman.
This blog is called “Overcoming Bias,” and many of you readers consider yourselves “rationalists,” i.e., folks who try harder than usual to overcome your biases. But even if you want to devote yourself to being more honest and accurate, and to avoiding bias, there’s a good reason for you not to present yourself as a “rationalist” in general. The reason is this: you must allocate a very limited budget of rationality.
It seems obvious to me that almost no humans are able to force themselves to see honestly and without substantial bias on all topics. Even for the best of us, the biasing forces in and around us are often much stronger than our will to avoid bias. Because it takes effort to overcome these forces, we must choose our battles, i.e., we must choose where to focus our efforts to attend carefully to avoiding possible biases. I see four key issues:
1. Priorities – You should spend your rationality budget where truth matters most to you. You can’t have it all, so you must decide what matters most. For example, if you care mainly about helping others, and if they mainly rely on you via a particular topic, then you should focus your honesty on that topic. In particular, if you help the world mainly via your plumbing, then you should try to be honest about plumbing. Present yourself to the world as someone who is honest on plumbing, but not necessarily on other things. In this scenario we work together by being honest on different topics. We aren’t “rationalists”; instead, we are each at best “rationalist on X.”
2. Costs – All else equal, it is harder to be honest on more and wider topics, on topics where people tend to have emotional attachments, and on topics close to the key bias issues of the value and morality of you and your associates and rivals. You can reasonably expect to be honest about a wide range of topics that few people care much about, but only on a few narrow topics where many people care lots. The close you get to dangerous topics, the smaller your focus of honesty can be. You can’t be both a generalist and a rationalist; specialize in something.
3. Contamination – You should try to avoid dependencies between your beliefs on focus topics where you will try to protect your honesty, and the topics where you are prone to bias. Try not to have your opinions on focus topics depend on a belief that you or your associates are especially smart, perceptive, or moral. If you must think on risky topics about people, try to first study other people you don’t care much about. If you must have an opinion on yourself, assume you are like most other people.
4. Incentives – I’m not a big fan of the “study examples of bias and then will yourself to avoid them” approach; it has a place, but gains there seem small compared to changing your environment to improve your incentives. Instead of pulling yourself up by your bootstraps, step onto higher ground. For example, by creating and participating in a prediction market on a topic, you can induce yourself to become more honest on that topic. The more you can create personal direct costs of your dishonesty, the more honest you will become. And if you get paid to work on a certain topic, maybe you should give up on honesty about who if anyone should be paid to do that.
So my advice is to choose a focus for your honesty, a narrow enough focus to have a decent chance at achieving honesty. Make your focus more narrow the more dangerous is your focus area. Try to insulate beliefs on your focus topics from beliefs on risky topics like your own value, and try to arrange things so you will be penalized for dishonesty. Don’t persent yourself as a “rationalist” who is more honest on all topics, but instead as at best “rationalist on X.”
So, what is your X?
Between $6 and $9 trillion dollars—about 8% of annual world-wide economic production—is currently being spent on projects that individually cost more than $1 billion. These mega-projects (including everything from buildings to transportation systems to digital infrastructure) represent the biggest investment boom in human history, and a lot of that money will be wasted. …
Over the course of the last fifteen years, [Flyvbjerg] has looked at hundreds of mega-projects, and he found that projects costing more than $1 billion almost always face massive cost overruns. Nine out of ten projects faces a cost overrun, with costs 50% higher than expected in real terms not unusual. …
In fact, the number of mega-projects completed successfully—on time, on budget, and with the promised benefits—is actually too small for Flyvbjerg to determine why they succeeded with any statistical validity. He estimates that only one in a thousand mega-projects fit that criteria. (more; paper)
You can probably throw most big firm mergers into this big inefficient project pot.
There’s a simple signaling explanation here. We like to do big things, as they make us seem big. We don’t want to be obvious about this motive, so we pretend to have financial calculations to justify them. But we are purposely sloppy about those calculations, so that we can justify the big projects we want.
It would be possible to make prediction markets that accurately told us on average that these financial calculations are systematically wrong. That could enable us to reject big projects that can’t be justified by reasonable calculations. But the people initiating these projects don’t want that, so it would have to be outsiders who set up these whistleblowing prediction markets. But alas as with most whistleblowers, the supply of these sort of whistleblowers is quite limited.