Why the People Who Build AI Fear It

In 2009, about twenty computer scientists met quietly on the California coast to ask each other whether they should be worried about what they were building. Almost nobody noticed. Five years later the same worry would be everywhere, spoken in the language of demons and extinction by some of the most famous people alive. Every one of them kept building. That pattern — warn, then build — is the subject of this episode of The Thinking Machine: A Brief History, and it's worth watching before you read this, because the faces matter:

Why the People Who Built AI Fear It →

The Room Nobody Noticed

The Asilomar Conference Grounds sit on the Monterey Peninsula, about two hours south of San Francisco, in a cluster of wooden buildings among the dunes. In February 2009 it hosted a small gathering organized by Eric Horvitz, who at the time was president of the Association for the Advancement of Artificial Intelligence. There were no cameras. No billionaires. No press tour afterward. Just researchers, sitting together for two days, trying to figure out whether their field was heading somewhere they should be worried about.

It's worth pausing on what AI could not do in 2009. It couldn't hold a conversation. It couldn't reliably tell a photograph of a dog from a photograph of a cat. The iPhone was two years old. Twitter was three. The word "chatbot" meant a customer-service annoyance that kept asking you to rephrase the question. The idea that a machine might threaten the species would've gotten you laughed out of most rooms in Silicon Valley.

Horvitz came out of those two days and told the New York Times something remarkable. He compared the mood among his colleagues to religion, and specifically to the Rapture. The Times ran the piece five months later. Almost nobody read it.

That's the part of the story that stays with me. Not the sober inquiry, but the indifference and the quasi-religious fervor that Horvitz noticed among his colleagues.

Five years later the question (should we be worried about what we're building) became a warning, screamed from every screen on earth.

So what changed between the quiet room and the global panic?

The Philosopher on the Nightstand

It starts, as too many of these stories do, at a haloed institution of higher learner.

In the summer of 2014 a Swedish philosopher named Nick Bostrom, working out of Oxford, published a book called Superintelligence. It argued, in careful academic prose, that a machine significantly smarter than us might not share our goals, that we might not be able to predict what goals it would have, and that once it showed up we might not be able to do much about it. He wasn't a crank, even if his argument required the sort of leap characteristic of a crank. The argument was detailed enough that you couldn't wave it away, and measured enough that you couldn't dismiss the man making it.

The book hit the bestseller list, but that's not the interesting part. The interesting part is where it landed.

In August 2014 Elon Musk posted about it publicly and called AI potentially more dangerous than nuclear weapons. Bill Gates read it and said his own views weren't far from Musk's. Sam Altman, who'd go on to run OpenAI, called it the best thing he'd read on the subject. Within a few months the book had passed hand to hand among four of the most credible people on the planet: a philosopher, a rocket builder, a physicist, and the man who'd put a computer on every desk. In some cases they were reading from literally the same copy.

A philosopher writes a book. The men building the machines read it. And then they start talking, loudly, to anyone who'll listen. So when four of the most credible people on earth open their mouths to warn us, what language do they reach for?

The Pentagram and the Holy Water

Two months after that post, in October 2014, Musk sat for an Q&A at MIT in front of a room full of engineering students and reached for a metaphor. He compared AI to summoning a demon — the kind with the pentagram, and the holy water, and the overconfident sorcerer who's sure he can control what he's called up. Then he invoked the murderous computer from 2001: A Space Odyssey and said it would be like a puppy dog next to what was coming.

Not "risk." Not "challenge." Not "an area requiring careful governance." The pentagram. The demon. HAL as a house pet.

Six weeks later, Stephen Hawking, the most recognized scientist alive, told the BBC that building a true artificial intelligence could mean the end of the human race. The detail nobody remembers is what prompted the remark. He was demonstrating his new voice system, which had just been upgraded with AI-powered predictive text. The software was learning how Hawking thought and finishing his sentences for him. He warned that AI could end humanity while showing the audience an AI that knew what he was about to say.

In January 2015, Gates wrote that he was concerned about superintelligence, recommended Bostrom's book, and said he stood with Musk. The chorus was now assembled: four voices, each carrying enormous credibility, each speaking to a different audience, all reading from the same page. The world received them not as four men sharing an opinion but as the most serious warning the species had ever been given.

But listen to the chorus again, because it's singing from a very specific hymnal. Where did it learn the tune?

The Script That Was Already Written

If you watched the previous episode of this series, you already know. The red eye. The calm voice doing the math. The machine that takes one look at us and decides we're the problem. Hollywood spent a hundred years drawing that picture and installing it in every person alive. The vocabulary of the warning — the demon, the end of the race, the machine that redesigns itself — isn't the vocabulary of an engineering report. It's the vocabulary of the screen.

Musk reaches for the demon and the pentagram. That's a scene from a movie. He names HAL out loud. Hawking's machine "takes off on its own and redesigns itself." That's Skynet. That's Ultron reading the internet. The men delivering the warning don't seem to notice they're reciting lines that were written for them, across decades, by screenwriters who had no more information about the future than anyone else.

This is the moment the trope completed its migration. It started in mythology, crossed into cinema, took up residence in our heads, and now it was coming out of the mouths of the people the whole world trusts, dressed in the same costume, hitting the same beats, received not as fiction but as intelligence briefing.

Horvitz called it in 2009. Technologists replacing religion. The Rapture. He just didn't know how fast the congregation would grow, or what the sermon would sound like once it had a choir. The choir had assembled by early 2015 and hadn't stopped singing since. But a choir eventually needs a hymn book — something official, something you sign your name to. What happens when the warning finally gets one?

The Twenty-Two Words

By 2023, the warning had stopped being a warning and had become a ceremony.

In March, one week after the release of GPT-4, the Future of Life Institute published an open letter calling on every AI lab to immediately pause the training of anything more powerful. It described the race as out of control, the machines as beyond their creators' understanding. Tens of thousands of people signed, including Musk, Steve Wozniak, and the historian Yuval Noah Harari.

Two months later, a different organization published something even more striking. Not a letter. Not an argument. A single sentence, twenty-two words long, placing the risk of human extinction from AI alongside pandemics and nuclear war. The people who signed it were not outsiders sounding an alarm. They were the alarm. Geoffrey Hinton, the man widely called the godfather of AI. Sam Altman, the CEO of OpenAI. Demis Hassabis, the head of Google DeepMind. Dario Amodei, running Anthropic. The men steering the three most powerful AI labs on earth put their names on a statement saying their own work could end the species.

And went back to work the next morning.

There is a version of this where the signatures mean something changed. But nothing changed. Musk warned about the demon in 2014 and founded his own AI company in 2023. Altman signed the extinction sentence and kept shipping. Amodei signed and kept shipping. The petition had become a rite, performed at the threshold and honored in the breach. Maximum credibility. No action required.

So now we can name the thing this whole episode has been circling. They warn, and they build. They warn, and they build. And the warning never slows the building. Not once. Not for a single day. Every figure who sounded the alarm kept right on constructing what they said the alarm was about. If all of them meant it, why didn't any of them stop?

The Badge

In May 2023, one of them did something none of the others had done. Geoffrey Hinton resigned from Google.

He'd spent a decade there, helping to build the systems that made modern AI possible. His work on neural networks is the foundation every chatbot and every image generator runs on. He didn't leave to join a competitor or start his own shop. He left so he could speak without the company's name on his badge.

He told the press he'd consoled himself with the usual excuse: if he hadn't done it, someone else would have. He said he'd believed the real danger was decades away, and that he no longer believed that. The man who helped lay the foundation walked out of the building because he couldn't say what he felt compelled to say while he was still inside it.

There's a version of this story where Hinton is the prophet who broke ranks, the one honest man in a room full of performers. But it's not that simple. Because the company he walked out of didn't disagree with him. Google, along with every major lab, has published its own warnings, signed its own statements, funded its own safety research. The institution agreed with the man who resigned in protest against the institution. And then the institution, like everyone else, went back to work.

That's the turn this series has been building toward. When the man who helped build the machine tells you it's dangerous, and the company he built it for agrees with him in writing and keeps building anyway, you're no longer watching a warning. You're watching something else, something with its own structure and its own purpose, and the next episode takes it apart.

That's what "Why AI Companies Call Their Own Tech Dangerous" goes after, and it's the one I'd watch from here if this is your first time. The deeper source trail and the analysis behind it live in the Signal Report: julianwhatley.com/signup. And the full episode this essay is built on is here: [Why the People Who Built AI Fear It →](YouTube video link).

I'm Julian Whatley. This is the Silicon Mirage. Now you see it.