AI Progress Estimate

From ’85 to ’93 I was an AI researcher, first at Lockheed AI Center, then at the NASA Ames AI group. In ’91 I presented at IJCAI, the main international AI conference, on a probability related paper. Back then this was radical – one questioner at my talk asked “How can this be AI, since it uses math?” Probability specialists created their own AI conference UAI, to have a place to publish.

Today probability math is well accepted in AI. The long AI battle between the neats and scruffs was won handily by the neats – math and theory are very accepted today. UAI is still around though, and a week ago I presented another probability related paper there (slides, audio), on our combo prediction market algorithm. And listening to all the others talks at the conference let me reflect on the state of the field, and its progress in the last 21 years.

Overall I can’t complain much about emphasis. I saw roughly the right mix of theory vs. application, of general vs. specific results, etc. I doubt the field would progress more than a factor of two faster if such parameters were exactly optimized. The most impressive demo I saw was Video In Sentences Out, an end-to-end integrated system for writing text summaries of simple videos. Their final test stats:

Human judges rated each video-sentence pair to assess whether the sentence was true of the video and whether it described a salient event depicted in that video. 26.7% (601/2247) of the video-sentence pairs were deemed to be true and 7.9% (178/2247) of the video-sentence pairs were deemed to be salient.

This is actually pretty impressive, once you understand just how hard the problem is. Yes, we have a long way to go, but are making steady progress.

So how far have we come in last twenty years, compared to how far we have to go to reach human level abilities? I’d guess that relative to the starting point of our abilities of twenty years ago, we’ve come about 5-10% of the distance toward human level abilities. At least in probability-related areas, which I’ve known best. I’d also say there hasn’t been noticeable acceleration over that time. Over a thirty year period, it is even fair to say there has been deceleration, since Pearl’s classic ’88 book was such a big advance.Robin Hanson

I asked a few other folks at UAI who had been in the field for twenty years to estimate the same things, and they roughly agreed – about 5-10% of the distance has been covered in that time, without noticeable acceleration. It would be useful to survey senior experts in other areas of AI, to get related estimates for their areas. If this 5-10% estimate is typical, as I suspect it is, then an outside view calculation suggests we probably have at least a century to go, and maybe a great many centuries, at current rates of progress.

Added 21Oct: At the recent Singularity Summit, I asked speaker Melanie Mitchell to estimate how far we’ve come in her field of analogical reasoning in the last twenty years. She estimated 5 percent of the way to human level abilities, with no noticeable acceleration.

Added 11Dec: At the Artificial General Intelligence conference, Murray Shanahan says that looking at his twenty years experience in the knowledge representation field, he estimates we have come 10% of the way, with no noticeable acceleration.

Added 4Oct’13: At an NSF workshop on social computing, Wendy Hall said that in her twenty years in computer-assisted training, we’ve moved less than 1% of the way to human level abilities. Claire Cardie said that in her twenty years in natural language processing, we’ve come 20% of the way. Boi Faltings says that in his field of solving constraint satisfaction problems, they were past human level abilities twenty years ago, and are even further past that today.

Let me clarify that I mean to ask people about progress in a field of AI as it was conceived twenty years ago. Looking backward one can define areas in which we’ve made great progress. But to avoid selection biases, I want my survey to focus on areas as they were defined back then.

Added 21May’14: At a private event, after Aaron Dollar talked on robotics, he told me that in twenty years we’ve come less than 1% of the distance to human level abilities in his subfield of robotic grasping manipulation. But he has seen noticeable acceleration over that time.

Added 28Aug’14: After coming to a talk of mine, Peter Norvig told me that he agrees with both Claire Cardie and Boi Faltings, that on speech recognition and machine translation we’ve gone from not usable to usable in 20 years, though we still have far to go on deeper question answering, and for retrieving a fact or page that is relevant to a search query we’ve far surpassed human ability in recall and do pretty well on precision.

Added 14Sep’14: At a closed academic workshop, Timothy Meese, who researches early vision processing in humans, told me he estimates about 5% progress in his field in the last 20 years, with a noticeable deceleration.

Added 4Jan’15: At a closed meeting, Francesca Rossi, expert in constraint reasoning, gave an estimate of 10%, with deceleration. Margret Boden, author of Artificial Intelligence and Natural Man (1977), estimated 5%, but for no particular subfield.

Added 6July’15: David Kelley, expert in big data analysis, says 5% in last twenty years, sees acceleration only in last 2-3 years, not before that.

Added 18Apr’16: Henry Kautz, says in constraint satisfaction we were at human level 20 years ago and have moved to super human levels now. In language, he says we’ve moved 10% of the way, with a noticeable acceleration in the last five years.

Added 13July2016: Jeff Legault says that in robotics we’ve come 5% of the way in the last 20 years, and there was only acceleration in the last five years.

Added 08Sept2017: Thore Husfeldt says that in the field of human understandable explanation, we have come less than 0.5% of the distance.