Superhuman Psychopaths

The usually reliable Michael Anissimov has claimed that I seem to think that “superintelligence will automatically acquire a favorable morality without any real work.”  Now I’m not all that sensitive about such things; but it bothers me that SIAI, of all people, should fail to understand the basic parameters of the problem, and thus have a completely unworkable notion of what to do about it.

Back in 2000, I wrote in Ethics for Machines:

There has always been a vein of Frankenphobia in science fiction and futuristic thought, either direct, as in Shelley, or referred to, as in Asimov. It is clear, in my view, that such a fear is eminently justified against the prospect of building machines more powerful than we are, without consciences. Indeed, on the face of it, building superhuman sociopaths is a blatantly stupid thing to do.

As far as I know, I was the first to point out that

Perhaps people have focussed too closely on the last point and forgotten the key point that we have to design and construct the consciences in the first place. Remember, the subtitle of Beyond AI is “Creating the Conscience of the Machine.”

A human conscience is a remarkably complex thing. It is a lot like the language-learning and -using module — partly learned from the surrounding culture but with a genetically-determined deep structure. But it is also tied into a powerful set of physical signalling and responses which make the individual human very literally a component of a larger social mind. Trying to study conscience in the context of a single mind in isolation is like trying to study language by following Robinson Crusoe around.

Leave a comment

    Your Cart
    Your cart is emptyReturn to Shop