The usually reliable Michael Anissimov has claimed that I seem to think that “superintelligence will automatically acquire a favorable morality without any real work.” Now I’m not all that sensitive about such things; but it bothers me that SIAI, of all people, should fail to understand the basic parameters of the problem, and thus have a completely unworkable notion of what to do about it.
Back in 2000, I wrote in Ethics for Machines:
There has always been a vein of Frankenphobia in science fiction and futuristic thought, either direct, as in Shelley, or referred to, as in Asimov. It is clear, in my view, that such a fear is eminently justified against the prospect of building machines more powerful than we are, without consciences. Indeed, on the face of it, building superhuman sociopaths is a blatantly stupid thing to do.
As far as I know, I was the first to point out that
- Ethics/morality is a separate module in the human mind — a conscience, if you will — whose effect will not be achieved by general intelligence and rationality on the part of the rest of the system, no matter how far advanced.
- We need to build our AIs with consciences.
- It should be possible — indeed, it looks depressingly easy — to build AIs with better, in the sense of more moral, consciences than humans have.
- In a self-improving community of AIs, appropriately formulated consciences would be an evolutionarily stable strategy, and thus unlikely to be discarded out of hand.
Perhaps people have focussed too closely on the last point and forgotten the key point that we have to design and construct the consciences in the first place. Remember, the subtitle of Beyond AI is “Creating the Conscience of the Machine.”
A human conscience is a remarkably complex thing. It is a lot like the language-learning and -using module — partly learned from the surrounding culture but with a genetically-determined deep structure. But it is also tied into a powerful set of physical signalling and responses which make the individual human very literally a component of a larger social mind. Trying to study conscience in the context of a single mind in isolation is like trying to study language by following Robinson Crusoe around.
