Monthly Archives: December 2012

Filters and bottlenecks

Lots of processes have filters: a certain proportion of the time they fail at that stage. There are filters in the path from dead stars to booming civilizations. There are filters in the path from being a baby to being an old person. There are filters in the path from having an idea to having a thriving business.

Lots of processes also have bottlenecks. These look similar, in that many things fail at that point. For instance the path to becoming a Nobel Prize winner is bottlenecked by there only being so many Nobel Prizes ever year. Rather than a fixed fraction of people getting past that barrier, a fixed number of people do.

It’s worth noticing if something is a filter or a bottleneck, because you should treat them differently often. Either way you can increase the fraction reaching the end by widening the filter or bottleneck to let more past. But for the filter it might be worth doing this at any other stage in the process, whereas for the bottleneck it is pointless at all earlier stages. You can’t get more Nobel Prize winners by improving education, but you might get more thriving businesses.

GD Star Rating

Why Am I Weird?

It will not have escaped the notice of long-time readers that I have a number of unusual intellectual views and priorities. In fact, more such views than most intellectuals.

This doesn’t usually bother me, but it should. After all, different theories about my weirdness lead to very different rational responses to my opinions, by myself and by others. Consider some theories:

  1. An unusually sloppy thinker, I make more big mistakes in reasoning.
  2. Unusually insightful, I have many unusual insights.
  3. Especially good at making up reasons, I seek an excuse to show off my reasoning, and so take positions that others will ask me to justify.
  4. Feeling unfairly low status, I hope for a status reversal via bragging later that I held popular opinions when they were unpopular
  5. Being especially proud, I’m unwilling to just accept standard views, and insist on thinking through all interesting topics through for myself. This leads to many contrarian views, since it leads to many views.
  6. Being unusually risk-taking, I collect opinions with a small chance of leading me to great fame and glory.
  7. Being unusually desiring of attention, positive or negative, I say things that will make people pay attention to me.
  8. Being especially good at a particular unusual sort of reasoning, e.g., very abstract concepts, I draw conclusions that neglect other sorts.
  9. Being especially uninterested in the usual rewards given intellectuals, I pick acts more likely to gain other rewards.
  10. Having initially learned an unusual mix of skills and topics, I apply that mix to produce unusual conclusions.

I’m sure many of you can think of more such theories (which I’ll add as suggested). But, after all these years, why don’t I know? Why don’t I care more? And, those of you who are also weird, why don’t you know, or care, why?

GD Star Rating
Tagged as: ,

Em Econ, London Style

Together with the provocative (Skype super-developer) Jaan Tallinn, I’ll speak on em econ next Saturday 2-5pm in London:

In this extended (3 hour) session, Robin Hanson and Jaan Tallinn will revisit and expand the material from their ground-breaking presentations from the Singularity Summit 2012 – presentations that Vernor Vinge, commenting shortly afterwards, described as refutations of the saying that “there is nothing new under the sun”. (more)

Jaan will talk on:

The incredible coincidence that we were born just decades before an imminent technological singularity that threatens to break our model of the evolution of the entire universe.

Added 19Dec: Here slides, bad audio from the talk. Here are slides, audio from my talk at the Oxford AGI-Impacts conference talk a few days before.

Added May ’13: Here is video of my Oxford conference talk.

GD Star Rating
Tagged as: ,

At Loooooong Last

In 2001, DARPA started funding my Policy Analysis Market:

We planned to cover eight nations. For each nation in each quarter of a year, we planned to have traders predict its military activity, political instability, economic growth, US military activity, and US financial involvement. In addition traders would predict US GDP, world trade, … and a few to-be-determined miscellaneous items. This would require a hundred or so base markets. Most important, we wanted to let our traders predict combinations of these, such has how moving US troops out of Saudi Arabia would affect political stability there, how that would affect stability in neighboring nations, and how all that might change oil prices. …

[We] prepared for and ran lab experiments comparing two new combinatorial trading mechanisms with traditional mechanism. These experiments, where six traders set 255 independent prices in five minutes, found that a combinatorial market maker was the most accurate. Phase II was mostly being spent implementing a scaleable production version of this market maker.

Alas, disaster hit a month before we were to start live testing, and five months before we were to start public trading:

The media storm hit on July 28, 2003, when two senators (falsely) complained that we were planning to let people bet on individual terrorist attacks. The next morning the secretary of defense announced that FutureMAP was cancelled.

While the press on that event did help jump-start today’s prediction market industry, I have always regretted that the storm didn’t wait until we had a demo to show, of combinatorial markets on Mideast geopolitical events. This is why if felt so satisfying to announce Friday:

We are live! If you register at, you can join hundreds of others who browse and edit estimates on over 100 questions intended to be of interest to the US intelligence community. … You can also make assumptions, and then browse and edit as before.

Over nine years later, you can finally see the demo I wanted everyone to see in ’03! Of course this is only a play money market, and it isn’t open to everyone. We don’t allow foreigners, you can’t lose any money in it, and we only pay for activity, not accuracy. So there’s less reason for you to believe these prices as event estimates. But still, you can see combinatorial prediction markets in action. At long looooong last!

GD Star Rating
Tagged as: ,

Combo Markets, Live!

Prediction markets let people bet on events of interest. This aggregates info on those events into market prices. So if you want to know the chance of an event, consider sponsoring a prediction market on it, and then watching the price. And if you want to persuade observers of a chance, consider betting in such a market, to change the price.

Sadly, the CFTC is cracking down on Intrade, making it harder to sponsor or use such markets. I can’t do much about that. But I can help improve the tech. For example, combinatorial prediction markets can let users bet on the chances of many unforeseen combinations of events. This can aggregate a lot more info on those events, as it lets users spontaneously express their opinions on a lot more topics.

Others have built combo markets. For example, WiseQ let people bet on certain combinations of events on the last election, such as whether the same party would win in two particular states. But those other markets allow inconsistencies, often big, between estimates on different questions. Our DAGGRE markets, in contrast, maintain exact globally consistency (up to machine precision) over a large combinatorial space of estimates, And they are live, today, for you to see and use at!

Let me explain. As I said five months ago:

Within a few months we will field an edit-based system where users can browse current answer estimates, and for each estimate can:

  • Edit the value. After you change an estimate to a new value, estimates that users see on all questions are Bayes-rule updates from that new value.
  • Assume a value. After you assume a value for this estimate, all estimates you see on all questions are conditional on this assumption. (more)

I said that we need to compute current estimates, edit limits, and long vs. short relative positions, and that we can now do these exactly (well, up to machine precision) with globally consistent estimates, in the case of a low-treewidth Markov network. Other approaches to combo markets allow inconsistencies, often big, between estimates on different questions. Large inconsistencies can hinder information aggregation, and risk large financial losses to clever traders who find them.

OK five months is not “few”, but today I can announce: we are live! If you register at, you can join hundreds of others who browse and edit estimates on over 100 questions intended to be of interest to the US intelligence community. We hand out $3000 a month to users in proportion to their activity (as 60 50$ Amazon gift cards; details here).

Actually, you could have done all that a year ago. But today you can also make assumptions, and then browse and edit as before. For example, we ask if foreign armies will fight in Syria soon, and also if Syria will use chem or bio weapons soon. So if you think that Syria using chem weapons would increase the chance of a foreign fight, you can assume that chem weapon use, and then edit the foreign fighter chance accordingly.

Actually you could have done that a month ago. But the system was slow and buggy; it is now much better. Which is why I’m announcing now. Yes, we still limit which estimates can be edited for any given set of assumptions; we have to do that to keep a low treewidth network of relations. But compared to ordinary prediction markets, this is a major advance in functionality. And we’ll keep working to relax these limitations.

GD Star Rating
Tagged as:

Breeding happier livestock: no futuristic tech required

[Edited to remove insensitive framing. Also, the possibility of reducing the misery in factory farming with such technology does not and would not justify factory farming.]

I have spoken with a lot of people who are enthusiastic about the possibility that advanced genetic engineering technologies will improve animal welfare.

But would it really take radical new technologies to produce genetics reducing animal suffering?

Modern animal breeding is able to shape almost any quantitative trait with significant heritable variation in a population. One carefully measures the trait in different animals, and selects sperm for the next generation on that basis. So far this has not been done to reduce animals’ capacity for pain as such, or to increase their capacity for pleasure, but it has been applied to great effect elsewhere on productivity (with some positive but overall negative effects on welfare).

One could test varied behavioral measures of fear response, and physiological measures like cortisol levels, and select for them. As long as the measurements in aggregate tracked one’s conception of animal welfare closely enough, breeders could generate increases in farmed animal welfare, potentially initially at low marginal cost in other traits.

Just how powerful are ordinary animal breeding techniques? Consider cattle:

In 1942, when my father was born, the average dairy cow produced less than 5,000 pounds of milk in its lifetime. Now, the average cow produces over 21,000 pounds of milk. At the same time, the number of dairy cows has decreased from a high of 25 million around the end of World War II to fewer than nine million today. This is an indisputable environmental win as fewer cows create less methane, a potent greenhouse gas, and require less land.

 Wired has an impressive chart of turkey weight over time:


Anderson, who has bred the birds for 26 years, said the key technical advance was artificial insemination, which came into widespread use in the 1960s, right around the time that turkey size starts to skyrocket…

This process, compounded over dozens of generations, has yielded turkeys with genes that make them very big. In one study in the journal Poultry Science, turkeys genetically representative of old birds from 1966 and modern turkeys were each fed the exact same old-school diet. The 2003 birds grew to 39 pounds while the legacy birds only made it to 21 pounds. Other researchers have estimated that 90 percent of the changes in turkey size are genetic.

Moreover, breeders are able to improve complex weighted mixtures of diverse traits:

The bull market (heh) can be reduced to one key statistic, lifetime net merit, though there are many nuances that the single number cannot capture. Net merit denotes the likely additive value of a bull’s genetics. The number is actually denominated in dollars because it is an estimate of how much a bull’s genetic material will likely improve the revenue from a given cow. A very complicated equation weights all of the factors that go into dairy breeding and — voila — you come out with this single number. For example, a bull that could help a cow make an extra 1000 pounds of milk over her lifetime only gets an increase of $1 in net merit while a bull who will help that same cow produce a pound more protein will get $3.41 more in net merit. An increase of a single month of predicted productive life yields $35 more.

No futuristic technologies are needed to make progress, although they would expedite the process: just feed accurate enough measurements of animal welfare into the net merit equation and similar progress could begin on the new trait.

Added December 8th:

Gaverick Matheny reports that some breeds have been selected in part for welfare. However, because breeders have not yet finished optimizing farm animals for productivity, the opportunity cost of not increasing productivity even further instead has been too high, given weak market and other pressures for welfare improvement, for this to take off.

GD Star Rating
Tagged as: , ,

Biases Of Fiction

This essay, on “The 38 most common fiction writing mistakes”, offers advice to writers. But the rest of us can also learn useful details on how fiction can bias our thinking. Here are my summary of key ways it says fiction differs from reality (detailed quotes below):

Features of fictional folk are more extreme than in reality; real folks are boring by comparison. Fictional folks are more expressive, and give off clearer signs about their feelings and intentions. Their motives are simpler and clearer, and their actions are better explained by their motives and local visible context. Who they are now is better predicted by their history. Compared to real people, they are more likely to fight for what they want, especially when they encounter resistance. Their conversations are mostly pairwise, more logical, and to the point. In fiction, events are determined more by motives and plans, relative to random chance and larger social forces. Overt conflict between people is more common than in real life.

And I’ll add that stories tend to affirm standard moral norms. Good guys, who do good acts, have more other virtuous features than in reality, and and good acts are rewarded more often than in reality.

A lot of our biases come, I think, from expecting real life to be like fiction. For example, when we have negative opinions on important subjects, we tend too much to expect that we should explicitly and directly express those negative opinions in a dramatic conversation scene. We should speak our mind, make it clear, talk it through, etc. This usually a bad idea. We also tend to feel bad about ourselves when we notice that we avoid confrontation, and back off when from things we want when we encounter resistance. But such retreat is usually for the best.

Those promised quotes: Continue reading "Biases Of Fiction" »

GD Star Rating
Tagged as:

Open Thread

Better late than never, this is our monthly place to discuss relevant topics that have not appeared in recent posts.

GD Star Rating
Tagged as:

College Admission Markets

This article by Ron Unz is long and rambles a bit, but deserves its provocative reputation. It offers data suggesting that over the last few decades the most elite US colleges have had systematically biased admissions, against asians and for jews, when measured against other standards, like tests and top math/sci competitions. Given the strong academic rhetoric against racial discrimination, you might expect this to cause a fervor, and to result in big changes soon. But I don’t expect much soon – most academics are from those schools, and benefited from those biases, loud complaining isn’t the asian style, and the larger society doesn’t much care because this discrimination is mostly limited to these schools.

The problem comes mainly from granting discretion to admissions personal to make subjective judgements. One solution is to just use objective features like test scores. But Unz worries about ambitious kids wasting their youth in mostly useless test prep. Also, application packets may contain other useful but harder to read clues about promising students. Unz instead prefers to admit “qualified” students at random, at least for most of the slots. But once everyone knew for sure that the elite schools didn’t actually have much better students, it isn’t clear why they would remain the elite schools.

As usual, my solution involves prediction markets. As I posted here five years ago, we could hide clearly identifying info about students, post their application packets to the web for all to see, and let anyone bet on the consequences of each student going to each school. Students might care about their chance of graduating, their income later, and some measure of satisfaction. Elite schools might care more about the chances of students being “successful” someday. Different schools might use different measures of success, such as with different weights for achievement in sports, politics, business, arts, etc. Schools could admit the students with the best chance to succeed by their measure, and students could apply to and then go to the school giving the best chance if achieving their goals. Or students could not go to school at all, if that was estimated to be best.

Of course speculators will favor students showing concrete signs of future success, and so ambitious students would spend their youth trying to achieve such signs. But instead of locking in particular limited metrics like standard test scores, where prep efforts are mostly wasted, this process would create an open competition to find signs of future success where efforts to gain them are more useful. After all, your chance of success later should be higher the more the signs you pursue push you to gain useful skills and habits in the process.

Yes it would be hard to get people to accept that such markets are accurate and hard-to-manipulate enough for this purposes. But equally hard, I expect, would be getting elite schools to say explicitly what sort of success they most want from students. They probably pretend to care more about admirable success, like being a famous writer, than they actually do.

Added 8p: Regarding anonymity, an obvious solution is for the official application to be completely public. Usually only a small fraction of the relevant application info will be things that are better kept private. Regarding that info, the applicant can just reveal that extra private info to a few trusted folks who are willing to trade in these markets. Markets do not need all traders know all relevant info to work well.

Added 30Oct2013: It appears that Unz’s data was faulty.

GD Star Rating
Tagged as: , ,