Do we need Friendly AI?

Do we need Friendly AI?

My Robo Habilis post was picked up on by Michael Anissimov who wrote:

(me:) It seems to me that one obvious way to ameliorate the impact of the AI/robotics revolution in the economic world, then, is simple: build robots whose cognitive architectures are enough different from humans that their relative skillfullness at various tasks will differ from ours. Then, even after they are actually better at everything than we are, the law of comparative advantage will still hold.

Boom, friendliness problem solved. Build robots with different cognitive architectures than us, and they will be forced to keep us around, due to Ricardo’s law of comparative advantage. Sounds wildly naive to me.

All I can say is thanks for noticing I’ve solved the most important problem of the 21st century with a single paragraph! I’m confidently expecting my Nobel Peace Prize.

But seriously, I would like to argue that the concept of the “friendliness problem” is a dangerous misreading of the real difficulties and problems we will face as a result of the development of artificial intellegence over the next few decades. It seems to me that one could characterize the people working on “Friendly AI” as essentially trying to redo moral philosophy, from scratch, and get it right this time.  There’s nothing wrong with this; moral philosophy is a valuable intellectual tradition and worthwhile human activity.  But the notion that the whole business, with the addition of the new insight that there can be intelligent machines as well as humans among the class of moral agents, could be solved in any useful sense, just strikes me as silly. Indeed, the new insight makes moral philosophy a lot harder, rather than bringing it any closer to any kind of closure.

Instead let’s look at the kind of problems we’re really going to face. There is not — I guarantee it — going to be any single overarching solution to them; there will be a host of minor things we can do to ameliorate the problems as they arise, and we’ll just have to keep coming up with them as problems arise.

We know what it will be like should we manage to invent and implement a giant, powerful decision-making system that takes over the world. We know because we’ve already done it. Some people have observed this system in action and seem to think that it has a “friendliness problem”:

We’re Governed by Callous Children

When I see those in government, both locally and in Washington, spend and tax and come up each day with new ways to spend and tax—health care, cap and trade, etc.—I think: Why aren’t they worried about the impact of what they’re doing? Why do they think America is so strong it can take endless abuse?
I think I know part of the answer. It is that they’ve never seen things go dark. They came of age during the great abundance, circa 1980-2008 (or 1950-2008, take your pick), and they don’t have the habit of worry. They talk about their “concerns”—they’re big on that word. But they’re not really concerned. They think America is the goose that lays the golden egg. Why not? She laid it in their laps. She laid it in grandpa’s lap.
They don’t feel anxious, because they never had anything to be anxious about.

Peggy Noonan thinks the government is screwing us up because it’s made of people who don’t care. But I beg to differ.  There’s a classic fallacy in the philosophy of mind that shows up in places ranging from Leibniz’ story of the “magnified mill” to Searle’s Chinese Room, which is that for a system to have some property, the property must be present among the parts. This is just as false for caring as it is for understanding or consciousness. In fact the existing system is a perfect example, although in reverse — it’s composed of people who do care, but they interact in a structure that results in an evil bureaucracy.

Instead, what’s happened is that we made a blunder in designing the system that is exactly equivalent to a favorite example of Eliezer Yudkowsky: instead of building a paperclip-maximizing machine, we built a vote-maximizing machine.

I claim that the problem is much more productively looked at from another point of view: the system as a whole is incompetent. It doesn’t do what it was built to do (“… promote the general welfare, secure the blessings of liberty …”). The designers simply assumed a vote-maximizer would do the things they wanted, but they were wrong.  Similarly, no human wants the universe to be converted into paperclips, so if he built a machine with that goal, he would have designed incompetently.  I claim we should be spending our time on is figuring out how to build competent AI.

First principle of competent AI design: Build a machine that understands what you want.  The paperclip maximizer is a study in amazing contrasts — presumably an intelligence powerful enough to take over the world would be capable of understanding human motivations even better than we do, so as to manipulate us effectively. Yet it’s built with a complete cognitive deficit of appropriate motivations, goals, and values for itself. Incompetent.

Second principle: build machines that know their limitations.  This basically means that it should confine its activities to those areas where it does understand the effects of its actions.

But in order to do that, we first have to be able to build a machine that can actually understand something — anything — in the full human-level meaning of understanding. And that is the necessary first step to a future of useful and beneficial AI, and it’s what anyone concerned about such things should be working on.

By | 2017-06-01T14:05:20+00:00 November 2nd, 2009|Machine Intelligence, Nanodot, Robotics|9 Comments

About the Author:


  1. Tim Tyler November 3, 2009 at 2:37 am - Reply

    As I said at the time: “The problem is not so much that trade between the parties won’t happen – but rather that soon humans won’t be able to afford to pay for food or rent – since wages will fall to machine levels.” You can only trade while you still have some money. If you think this actually solves anything, perhaps try modeling it.

  2. Acksiom November 3, 2009 at 2:06 pm - Reply

    And as was pointed out in response, prices will fall comparatively.

    Furthermore it’s quite possible to trade without currency. People do it all the time; sometimes it’s called “barter”, sometimes “neighborliness” (in which credit is occasionally extended!), sometimes even “friendliness”.

    . . .oh wait. . . .

  3. Tim Tyler November 4, 2009 at 2:12 am - Reply

    Prices will fall comparatively to wages? Surely, that is simply not true! For example, real estate prices can be reasonably expected to not do anything of the kind. That raises the possibility that the unemployed humans on welfare won’t be able to afford to live in the world’s cities – and will thus tend to be marginalised.

  4. Tim Tyler November 4, 2009 at 10:34 am - Reply

    Anna Salamon – in her “Shaping the Intelligence Explosion” talk – correctly explains some of the problems with the idea that trading will save humans (24 minutes in):

  5. Acksiom November 4, 2009 at 2:41 pm - Reply

    “Surely, that is simply not true!”

    This statement makes you appear grossly ignorant of human history.

    “For example, real estate prices can be reasonably expected to not do anything of the kind.”


    Finally, why should deliberately “unemployed humans on welfare” not be allowed to choose to marginalize themselves by not providing sufficient value to those around them (as distinguished from deliberately employed humans creating such)?

  6. Proto November 4, 2009 at 5:09 pm - Reply

    The problem with building a machine that understands what you want better than you do is knowing when it’s actually done so, since the results aren’t provable.

    Well, the results are provable, but you have to experiment on humans. And we generally frown upon that.

    Instead, you could test it on perfectly simulated humans in a computer. But then you have to trust that your AI is giving you reliable results when run in a simulation. And if it’s more intelligent than you are, you can’t know whether or not it knows it’s acting on a simulation and not the real world. And if it’s more intelligent than you are, you can’t know whether or not it’s more intelligent than you are.

    Which means that you can’t know whether or not any approximately human level AI is more intelligent than you are, which means you can’t trust it to run a simulation, which means you can’t trust it to understand humans. Damn. That sucks.

  7. Tim Tyler November 5, 2009 at 1:23 am - Reply

    You misunderstand. The term “real estate” refers not just to buildings, but also to the land on which they are built (check with the definition of “estate”).

    The problem with unemployed humans arises not from those who want to be unemployed.

  8. Zina Valery May 13, 2010 at 6:24 am - Reply

    Thanks for your blog article. It has given me a little to go over. Thank you again!

  9. Tim Tyler September 25, 2010 at 3:24 am - Reply

    This page needs a link to:

    “Comparative Advantage Doesn’t Ensure Survival”

Leave A Comment