Superhuman Psychopaths

Superhuman Psychopaths

The usually reliable Michael Anissimov has claimed that I seem to think that “superintelligence will automatically acquire a favorable morality without any real work.”  Now I’m not all that sensitive about such things; but it bothers me that SIAI, of all people, should fail to understand the basic parameters of the problem, and thus have a completely unworkable notion of what to do about it.

Back in 2000, I wrote in Ethics for Machines:

There has always been a vein of Frankenphobia in science fiction and futuristic thought, either direct, as in Shelley, or referred to, as in Asimov. It is clear, in my view, that such a fear is eminently justified against the prospect of building machines more powerful than we are, without consciences. Indeed, on the face of it, building superhuman sociopaths is a blatantly stupid thing to do.

As far as I know, I was the first to point out that

  • Ethics/morality is a separate module in the human mind — a conscience, if you will — whose effect will not be achieved by general intelligence and rationality on the part of the rest of the system, no matter how far advanced.
  • We need to build our AIs with consciences.
  • It should be possible — indeed, it looks depressingly easy — to build AIs with better, in the sense of more moral, consciences than humans have.
  • In a self-improving community of AIs, appropriately formulated consciences would be an evolutionarily stable strategy, and thus unlikely to be discarded out of hand.

Perhaps people have focussed too closely on the last point and forgotten the key point that we have to design and construct the consciences in the first place. Remember, the subtitle of Beyond AI is “Creating the Conscience of the Machine.”

A human conscience is a remarkably complex thing. It is a lot like the language-learning and -using module — partly learned from the surrounding culture but with a genetically-determined deep structure. But it is also tied into a powerful set of physical signalling and responses which make the individual human very literally a component of a larger social mind. Trying to study conscience in the context of a single mind in isolation is like trying to study language by following Robinson Crusoe around.

By | 2017-06-01T14:05:24+00:00 August 22nd, 2009|Ethics, Machine Intelligence, Memetics, Nanodot|7 Comments

About the Author:


  1. […] Read more … […]

  2. Michael Anissimov August 22, 2009 at 10:51 am - Reply

    Josh, maybe I am being unfair to your ideas. My critical comments stem from various references to your ideas which I read in “Moral Machines” by Wallach and Allen, as well as hearing you speak at Singularity Summit 2007. I have your most recent book but received it only quite recently and honestly have not read it yet. I will try to read it before I make any more references to your ideas, so I can be sure I know what I’m talking about.

    To replace “without real work”, maybe I should have said “with relatively little work”, as you imply above with the phrase “depressingly easy”. That would be accurate, yes?

    I only arrived in this community in 2001, so I can’t say for sure whether the ideas you list first appeared from you or Eliezer. Do you have any references to mailing list posts I might be able to look at to settle the issue?

  3. Phil Bowermaster August 22, 2009 at 1:03 pm - Reply

    Depending on how powerful the superintelligence is, the more moral we make it, the more control of the world we will likely relinquish to it. Or rather, the more control of the world it will likely take from us. Which isn’t necessarily a bad thing.

  4. J. Storrs Hall August 23, 2009 at 11:03 am - Reply

    @Michael: You’re a good man and your moderate tone is appreciated.
    You’ll find that I spend a lot of time in Beyond AI discussing how to build an AI in the first place, since I feel that it is the brittleness and lack of common sense in current AI practice that will be causing most the AI-related tragedies for the next decade at least. (Read Tom Godwin’s “The Gulf Between” for which the cover illustration of my book was first painted.)
    Once we get that part figured out, which is a really hard problem and will in my estimation take us a full decade to understand, building a proper conscience will be a moderate additional increment. But from today’s standing start, it’s a huge task, equivalent to the entire language understanding and generation problem, plus.
    The reason it’s depressingly simple is that once we know how the conscience works we only need change the human heuristic of “be as selfish as possible to start (we call this childish) and move to cooperation as the environment demands” to the opposite direction (i.e. start as unselfish as possible, etc).
    Eliezer is a brilliant autodidact but an autodidact nonetheless. Thus those of us who have spent 30 years studying AI in academia and 50 years studying it in science fiction have seen most of the ideas he is known for, well before he appeared. Two examples: “seed AI” and “friendly AI”: The first is essentially what Alan Turing proposed in 1950 in the classic Mind paper (where the Turing test is proposed) under the name of “the child machine”. The second is very much what Asimov had in mind with his robot series and the Three Laws. If you doubt he had such a broad conception in mind, read The Evitable Conflict (the last story in “I, Robot”), which also appeared in 1950.

  5. Spike McLarty August 31, 2009 at 1:07 am - Reply

    OK… how about some pointers to the literature on how to implement this easy little after-the-hard-work-is-done bolt-on conscience? I’ve ordered my copy of Beyond AI, so maybe this is out of order – but surely if we think AIs could be superhuman sociopaths, their moral sense is not the last problem to solve, nor the last piece of code to debug? Doesn’t a binding moral imperative necessarily require systematic distortion of some aspect of cognition? That doesn’t sound like something you can just ‘add’.

  6. Graham Rawlinson August 31, 2009 at 2:23 am - Reply

    Conscience, multiplicity, AI and risk.

    I wish to enter this discussion and hope that not too many of my thoughts have been offered too many times before! Apols if so.

    My background is as a psychologist and innovation facilitator. More recently I have been working with Rita Carter on the notion of Multiplicity, that we all have multiple minds which each have their own memories, motivations and access to skill sets. I know this concept will not be new to AI people as computers have elements of Multiplicity built in.

    It seems to me that the desire or need to place conscience in artificially intelligent machines (though I confess I am not entirely sure I know what non-artificially intelligent machines are) that need is based best on an assessment of risk.

    I start with the proposition that risk lies in handing power to one thing/person or a small group of things/people such that normal corrections against excess cannot be managed without risk and high cost.

    Human history is full of attempts to manage that risk and each of us may have their own view of how successful those attempts have been. So our desire or need to hand conscience over to machines can be put against a more general suggestion that neither machines nor people should be handed such power as to not have it able to be corrected easily by other forces.

    In that sense the machine element of the problem does not exist as being different.

    Having said that, the attempt to place conscience inside a machine may be very informative in trying to understand how we might do so for human beings also. I am not sure I or other people know how to place conscience inside the mind of another in such a way as to have it rule over decision making.
    Many people with the very best of moral upbringing have engaged later in atrocities. We do now how to get people to act morally when they are not tempted by too great a prize or challenged by too great a threat, but after that, well, the jury is surely out.

    At the other end there are criminal gangs and terrorist groups who would declare their behaviour in the strongest moral terms. Family/religion is everything.

    My third point brings in Multiplicity (see for the book) and I think this risk management has lots of connections to Multiplicity as I think the reason why we have multiple selves is to help us manage our thinking and decision making and reduce the risk of over-powerful personas taking actions which are harmful to ourselves or our communities (and of course our genes).

    If our high risk decision making has to pass through a parliament of minds (French – Parler), then this is fine whether it is machine or human or, and maybe this is always the best option, human and machine?

    As AI gets more powerful there should come a time when we hand one of the keys to decision making to these machines, always ensuring that humans have some keys too and that all are needed for risky decisions. This would have the additional advantage of reducing the risk of relying on the human conscience, which has been at different times and for different peoples a failure and a success.

    As a final note, I do think conscience ‘morality’ could be much simpler than we might imagine, and some parts of an understanding of morality may be found in Pirsig’s book, Lila, an enquiry into Morality.


    Graham Rawlinson
    Author of PhD The Significance of Letter Position in Word Recognition, Nottingham University (not Cambridge as was wrongly circulated)

  7. Graham Rawlinson August 31, 2009 at 2:38 am - Reply

    Apologies, the last comment on Pirsig’s book should have stated Lila, An Inquiry into Morals not Morality.



Leave A Comment