AI Architecture

What I’m about to say is pretty obvious to most artificial intelligence (AI) researchers, but I think others might benefit from hearing it. 

In general, AI systems are created by combining computer hardware, other hardware, access to data, an architecture, parameter settings, and hand-made software. Machine learning (ML) systems have relatively little hand-made software, while robotic systems have more other hardware. 

Over the last few decades, our most impressive AI systems have mostly been of the non-robotic ML type, have had limited scopes of application, and have had a wide range of architectures. AI developers have considered architecture to be important in the sense that they repeatedly start over with new architectures, instead of continuing to build up systems based on old architectures. 

Clearly the AI world is to some extent searching in the space of possible AI system architectures. Plausibly this is because, so far, architecture gains have made a substantial contribution to performance gains, just as more generally algorithm gains have been similar to computer hardware gains.  (As architecture is only part of algorithms, its gains are likely less than hardware gains.)

Many are anticipating a future day when we may have AIs with a relatively wide range of abilities (call them AGIs = “artificial general intelligences”), and at human-level or better for most of those abilities. Many others are anticipating a probably different day, earlier or later, when we will see an AI which is good at the general task of searching in the space of new AI architectures (Call them SIAI = “self-improving artificial intelligences”). Depending on just how good a SIAI is, and just how hard or easy is that task, the abilities in its lineage might (or might not) rapidly improve. 

The best architectures today are plausibly those behind large language models, which are indeed impressive. But similarly impressive to how the best new systems of each decade have looked in the past. They do not seem especially close to being either AGI or SIAI, and it seems pretty clear to me that they would not become such just by adding more hardware and data. Many people have plausible ideas for architecture modifications that might move such systems in these directions. But the question is how fast to expect such innovations to appear.

I continue to see past track records of progress in such things as our best basis for estimating future progress. Recent advances have been exciting, but they seem within the usual long-term range of variation in the frequency and lumpiness of architecture innovations. And judging from past trends in the key metric of the fraction of world income that goes to pay for AI systems, we still seem to be a long way from an economy dominated by AI. 

Here’s a project idea to help track progress in AI architectures. Let’s try to define an uninformed Bayesian prior over the space of all AI architectures, and then track the prior values assigned to particular new AI systems over time, along with other noteworthy facts about each system. We might similarly set a prior over parameter settings and track changes in those prior values. In this way we could create a dataset to support forecasting future rates, variation, and lumpiness of progress in AI architecture and parameter settings.