This the third essay in a series exploring if, when, and how the Singularity will happen, why (or why not) we should care, and what, if anything, we should do about it.
Part III: Singularity from the bottom up
In the previous essay in this series, I argued top-down, from historical and economic precedents, that the coming singularity might look approximately like the second half of the computer/internet revolution. Today I’ll argue the same conclusion from the bottom up: by looking at things from the point of view of the individual AI.
Note: this analysis is taken from a paper I gave at AGI-08.
A major concern in some transhumanist and singularitarian schools of thought is autogenous—self-modifying and extending—AIs. It is worried that hyper-intelligent machines might appear literally overnight, as a result of runaway self-improvement by a “seed AI”.
How likely is runaway self-improvement?
As a baseline, let us consider the self-improving intelligence we understand best, our own. Humans not only learn new facts and techniques, but improve our learning ability. The invention of the scientific method, for example, accelerated the uptake of useful knowledge tremendously. Improvements in knowledge communication and handling, ranging from the invention of writing and the printing press to the internet and Google, amplify our analytical and decision-making abilities, including, crucially, the rate at which we (as a culture) learn.
Individual humans spend much of our lives arduously relearning the corpus of culturally transmitted knowledge, and then add back a tiny fraction more. Thus on the personal scale our intelligence does not look “recursively self-improving” — but in the large view it definitely is.
Technological development usually follows an exponential improvement curve. Examples abound from the power-to-weight ratio of engines, which has tracked an exponential steadily for 300 years, to the celebrated Moore’s Law curve for semiconductors, which has done so for 50. These improvement curves fit a simple reinvestment model, where some constant of proportionality (an “interest rate”) determines the overall growth rate.
Any agent must make a decision on how much of its resources to reinvest, and how much to use for other purposes (including food, clothing, shelter, defense, entertainment, etc.). Human societies as a whole have invested relatively low percentages gross product, and even of their surplus, in scientific research. The proportion of scientists and engineers in the US population can be estimated at 1%, and those in cognitive-science related fields as 1% of that. Thus we can estimate the current rate of improvement of AI as being due to the efforts of 30,000 people (with a wide margin for error, including the fact that there are many cognitive scientists outside the US!), and estimate the rate of improvement in computer hardware and software generally, as being due to the efforts of possibly 10 times as many.
It is not clear what a sustainable rate of reinvestment would be for an AI attempting to improve itself. In the general economy, it would require the same factors of production — capital, power, space, communication, and so forth — as any other enterprise, and so its maximum reinvestment rate would be its profit margin. Let us assume for the moment a rate of 10%, 1000 times the rate of investment by current human society in AI improvement. (This is germane because the AI is faced with exactly the same choice as an investor in the general economy: how to allocate its resources for best return.)
Note that from one perspective, an AI running in a lab on equipment it did not have to pay for could devote 100% of its time to self-improvement; but such cases are limited by the all-too-restricted resources of AI labs in the first place. Similarly, it seems unlikely that AIs using stolen resources, e.g. botnets, could manage to devote more than 10% of their resources to basic research.
Another point to note is that one model for fast self-improvement is the notion that a hyperintelligence will improve its own hardware. This argument, too, falls to an economic analysis. If the AI is not a hardware expert, it makes more sense for it to do whatever it does best, perhaps software improvement, and trade for improved hardware. But this is no different from any other form of reinvestment, and must come out of the self-improvement budget. If the AI is a hardware expert, it can make money doing hardware design for the market, and should do that exclusively, and buy software improvements, for the overall most optimal upgrade path.
Thus we can assume a 10% reinvestment rate, but we do not know the productivity coefficient. It is occasionally proposed that, as a creature of software, an AI would be considerably more proficient at improving its own source code than humans would be. However, while there is a steady improvement in software science and techniques, these advances are quickly written into tools and made available to human programmers. In other words, if automatic programming were really such low-hanging fruit for AI as is assumed, it would be amenable to narrow-AI techniques and we would have programmer’s assistants that improved human programmers’ performance drastically. What we see is steady progress but no huge explosion.
In practice the most difficult part of programming is higher-level conceptual systems design, not lower level instruction optimization (which is mostly automated now as per the previous point anyway). Abstract conceptualization has proven to be the hardest part of human competence to capture in AI. Although occasionally possible, it is quite difficult to make major improvements in a program when the program itself is the precise problem specification. Most real-world improvements involve a much more fluid concept of what the program must do; the improved version does something different but just as good (or better). So programming in the large requires the full panoply of cognitive capabilities, and is thus not likely to be enormously out of scale compared to general competence. I think that many of the more commonly seen scenarios for overnight hard takeoff are circular — they seem to assume hyperhuman capabilities at the starting point of the self-improvement process.
We can finesse the productivity, then, by simply letting it be one human equivalent, and adjusting the timescale to let 0 be whatever point in time a learning, self-improving, human-level AI is achieved. Then we estimate human productivity at intelligence improvement by assuming that the human cognitive science community are improving their models at a rate equivalent to Moore’s Law. As this is the sum effort of 30,000 people, each human’s productivity coefficient is 0.00002.
This gives us a self-improvement rate for the efforts of a single AI that is essentially flat, as one would expect: the analysis for a single human would be the same. A single human-level AI would be much, much better off hiring itself out as an accountant, and buying new hardware every year with its salary, than by trying to improve itself by its own efforts.
Recursive self-improvement for such an AI would then mean buying new hardware (or software) every year, improving its prowess at accounting, for an increased growth rate compounded of its own (tiny) growth and Moore’s Law. Only when it reached a size where it could match the growth rate of Moore’s Law purely by its on efforts, would it make sense for it to abandon trade and indulge in self-construction.
But that assumes Moore’s Law, and indeed all other economic parameters, remained constant over the period. A much more realistic assumption is that, once human-level AI exists at a price that is less than the net present value of a human of similar capabilities, the cost of labor will proceed to decline according to Moore’s Law, and therefore the number of human equivalent minds working in cognitive science and computer hardware will increase at a Moore’s Law rate, both increasing the rate of progress and decreasing the price from the current trendline.
In other words, the break-even point for an AI hoping to do all its own development instead of specializing in a general market and trading for improvements, is a moving target. It will track the same growth curves that would have allowed the AI to catch up with a fixed break-even point. (In simple terms: you’re better off buying chips from Intel than trying to build them yourself. You may improve your chip-building ability — but so will Intel; you’ll always be better off buying.)
We can conclude that, given some very reasonable assumptions, it will always be more optimal for an AI to trade; any one which attempts solitary self-improvement will steadily fall farther and farther behind the technology level of the general marketplace. Note that this conclusion is very robust to the parameter estimates above: it holds even if the AI’s reinvestment rate is 100% and the number of researchers required to produce a Moore’s Law technology improvement rate is 1% of the reasonable estimate.)
Let us now consider a fanciful example in which 30,000 cognitive science researchers, having created an AI capable of doing their research individually, instantiate 30,000 copies of it and resign in favor of them. The AIs will be hosted on commercial servers rented by the salaries of the erstwhile researchers; price per MIPs of such a resource will be assumed to fall, and thus resources available at a fixed income to rise, with Moore’s Law.
At the starting point, the scientific efforts of the machines would equal those of the human scientists by assumption. But the effective size of the scientific community would increase at Moore’s Law rates. On top of that, improvements would come from the fact that further research in cognitive science would serve to optimize the machines’ own programming. Such a rate of increase is much harder to quantify, but there have been a few studies that tend to show a (very) rough parity for Moore’s Law and the rate of software improvement, so let us use that here. This gives us a total improvement curve of double the Moore’s Law rate. This is a growth rate that would increase effectiveness from the 30,000 human equivalents at the start, to approximately 5 billion human equivalents a decade later.
I claim that this growth rate is an upper bound on possible self-improvement rates given current realities. Note that the assumptions subsume many of the mechanisms that are often taken in qualitative arguments for hard takeoff: self-improvement is taken account of; very high effectiveness of software construction by AIs is assumed — 2 years into the process, each human equivalent of processing power is assumed to be doing 11 times as much programming as a single human could, for example. Nanotechnology is implied by Moore’s Law itself not too many years from current date.
This upper bound, a growth rate of approximately 300% per year, is unlikely to be uniformly achieved. Most technological growth paths are S-curves, exponential at first but levelling out as diminishing returns effects set in. Maintaining an overall exponential typically requires paradigm shifts, and those require search and experimentation, as well as breaking down heretofore efficient social and intellectual structures. In any system, bottleneck effects will predominate: Moore’s Law has different rates for CPUs, memory, disks, communications, etc. The slowest rate of increase will be a limiting factor — the same “Amdahl’s Law of Singularity” I mentioned before. And finally, we do not really expect the entire cognitive science field to resign, giving their salaries over to the maintenance of AIs.
