One of my favourite recurring tropes of AI speculation/singulatarian deep time thinking is mediations on how an evil AI or similar might destroy us.

Here’s a recent example, Ross Anderson on human extinction as quoted/linked by Kottke. It’s a discussion about how a benign AI might be poorly designed and lead to our downfall. What happens is the AI is given a goal that is proximate to helping people but not identical to (because no one even knows what that means).

The scenario imagined is one where there is a button that humans push if the AI gets an answer right and the AI wants to get a lot of button presses, and eventually it realizes that the best way to get button presses is to kill all the humans and institute a rapid fire button-pressing regime. (This, by the way, is the same instrumentalist train of logic that leads to sexbots.)

You would have this thing that behaves really well, until it has enough power to create a technology that gives it a decisive advantage — and then it would take that advantage and start doing what it wants to in the world.’

And all I can think is: we already have one of those. It is pretty clear to anyone who’s paying attention that 1. a marketplace regime of firms dedicated to maximizing profit has—broadly speaking—added a lot of value to the world 2. there are a lot of important cases where corporate profit maximization causes harm to humans 3. corporations are—broadly speaking—really good at ensuring that their needs are met.

I don’t think that it’s all that far fetched to suggest that maybe they’re getting better and better at ensuring their needs are met. Pretty much the only thing that the left and right in America can agree on is that moneyed influence has corrupted American politics and yet neither side seems able to do much of anything about it.

What if the private pursuit of profit was—for a long time—proximate to improving the lot of humans but not identical to it? What if capitalism has gone feral, and started making moves that are obviously insane, but also inevitable? 

For a very long time, the AI dedicated to maximizing profit saw the path forwards through innovation, new products, better living for customers. But then at some point it realized that is had the ability to just reshape the planet in its image. So it did that instead.

Imagine these thoughts—hastily thrown together to make a point about the devil we fear, vs the one we face—accompanied by about a million caveats having to do with long histories of systemic racism/sexism/colonialism and many other important isms that make making claims about the relative benefits to humans from the private pursuit of profit very difficult and likely to fall apart under careful scrutiny.