Why 'Too Dangerous' Is AI's Best Pitch

A companion to the video "Why AI Corps Call Their Tech Dangerous" from AI Apocalypse: A Brief History

The Pope's Bestseller List

In 1559, Pope Paul IV published a document called the Index Librorum Prohibitorum. It was, quite literally, a list of books the Catholic Church forbade its members to read. The works of Galileo, Copernicus, Voltaire, Descartes, and John Locke all appeared on it at one time or another. The stated purpose was suppression: to protect the faithful from dangerous ideas by making those ideas officially unavailable.

It didn't work. The Index was updated forty-two times over the next four centuries, and for four centuries it served, in practical effect, as the world's most authoritative recommendation engine. Printers across Europe discovered that an appearance on the Index was worth more than any review. A bookseller in Protestant Amsterdam or London could charge a premium for a title Rome had condemned. Historians of the period note that certain authors actually lobbied to be included, understanding that the Church's disapproval functioned as the most credible endorsement imaginable. The Index wasn't abolished until 1966. By then, it had been the longest-running bestseller list in publishing history.

The lesson here seems simple enough that it hardly needs stating. People want what they're told they can't have. Psychologists call it reactance theory: when someone perceives a threat to their freedom of choice, the restricted option becomes more attractive. Jack Brehm formalized this in 1966, the same year the Index was retired, but the mechanism had been operating for centuries before anyone gave it a name.

The version of this story that matters for the present moment, though, is subtler. Because in the case of the AI industry, nobody is banning anything from the outside. The companies themselves are issuing the prohibition. They're placing their own products on the Index. And they've discovered something the Vatican never did: that self-imposed restriction works even better than external censorship, because it comes wrapped in an additional psychological gift. The gift is trust.

But the mechanism that produces that trust was identified in a very different context, by a psychologist working with military officers during the First World War.

The Colonel's Jawline

In 1920, the psychologist Edward Thorndike published a study in the Journal of Applied Psychology titled "A Constant Error in Psychological Ratings." He had asked commanding officers at military camps to evaluate their soldiers on a range of independent traits: intelligence, physique, leadership, dependability, character. What Thorndike found was that the ratings weren't independent at all. A soldier who was physically impressive received higher scores for intelligence. A soldier perceived as having strong character was rated as a better leader. One positive impression bled into every other category, as if the raters couldn't separate what they were measuring from what they were feeling.

Thorndike called this the Halo Effect. The name has entered common usage, but its implications are routinely underestimated. The Halo Effect doesn't just mean we think attractive people are smarter. It means that once a sufficiently strong positive impression forms, it becomes the lens through which all subsequent information is processed. Contradictory evidence doesn't displace the halo. It gets absorbed by it. The positive frame digests the negative data.

Now apply this to the AI industry in February 2019, when the "too dangerous" playbook debuted. The founders and funders of OpenAI occupied a cultural position that's difficult to reconstruct from the vantage of 2026, because the landscape has shifted so dramatically. Elon Musk was the visionary who'd built Tesla and SpaceX. Reid Hoffman was the philosopher-networker of Silicon Valley. Peter Thiel was the contrarian genius. Sam Altman was the earnest young builder working for a nonprofit. These weren't just successful people. They were the protagonists of our culture's most powerful secular mythology: the story of brilliant outsiders bending the future through force of intellect. A Swedish study in 2007 went further than Thorndike, finding that wealth itself produced a halo so powerful that subjects rated wealthy individuals as more intelligent regardless of any other information presented. The halo didn't need to be earned through demonstrated competence. Money alone could generate it. And these figures had money, fame, institutional backing, and a narrative that the press had spent a decade reinforcing.

When a company operating under that kind of perceptual shield told the world its own product might be dangerous, the Halo Effect didn't weaken. It intensified. The admission of danger read not as a marketing move but as an act of extraordinary responsibility. These were the brilliant people, the ones backed by billions, the ones our culture had designated as the smartest humans in the room. If they said something was dangerous, who were we to disagree?

And then the halos cracked.

The intervening years have subjected almost every figure in the original constellation to the kind of sustained public reappraisal that Thorndike's framework would predict is nearly impossible. Musk's trajectory from visionary industrialist to polarizing political actor has been so complete that his name now generates the opposite of a halo for roughly half the population. Hoffman's connections to Jeffrey Epstein surfaced and wouldn't go away. Altman was fired by his own board in November 2023 in a dispute that exposed, depending on whom you believe, either a governance failure or a pattern of deception, then was reinstated days later in a sequence that consolidated his control over the organization. Thiel retreated further into a political identity that half of America regards with frank alarm. The nonprofit that had asked investors to view their capital "in the spirit of a donation" filed paperwork to become a for-profit corporation.

The halos didn't just dim. In several cases, they inverted.

The natural experiment is clean, because the underlying technology and the underlying playbook haven't changed. The companies are still making the same announcements. They're still framing their products in the vocabulary of existential risk. They're still briefing governments and publishing safety reports and testifying before Congress. The only variable that changed is the halo. And the reception has shifted dramatically. The "too dangerous" announcement that generated credulous awe in 2019 generates visible public skepticism in 2026. Same playbook. Same vocabulary. Different halo. Different result.

Thorndike, reading the data from a century's distance, would recognize the pattern instantly. The halo was never about the technology. It was about the person holding the microphone. Crack the halo and the same words, from the same mouth, land as self-serving theatre instead of courageous disclosure. The tarnishing doesn't disprove the Halo Effect. It's the most powerful evidence that the Halo Effect was doing the heavy lifting all along.

But even a tarnished halo is still a halo. The playbook still works. The valuations are still climbing. And the reason it still works, even at reduced amplitude, is that the Halo Effect was never the only mechanism in the machine. For the rest of the engine, you need a seventeenth-century French mathematician and a mid-century social psychologist working with tape recordings of a quiz show.

The Wager and the Spilled Coffee

Blaise Pascal, writing in the 1660s, proposed a thought experiment about the existence of God that's become known as Pascal's Wager. The logic was simple: if God doesn't exist and you believe, you lose nothing of consequence. If God does exist and you don't believe, you lose everything. The asymmetry of the outcomes makes belief rational even in the absence of evidence.

The AI investment thesis operates on identical logic, and the "too dangerous" announcement supercharges the asymmetry. If a company warns you that its technology could transform civilization and the warning turns out to be overblown, you've lost a few minutes of attention. But if the warning is real and you ignored it, you missed the most consequential technological development in human history. The warning itself raises the perceived stakes of inattention. It converts the audience from casual observers into reluctant participants in a cosmic bet they never agreed to place.

Pascal's Wager explains the emotional logic. But there's a second mechanism operating here, and it was discovered quite by accident. In 1966, the social psychologist Elliot Aronson played subjects a tape recording of a contestant in a quiz show. The contestant answered nearly every question correctly. In one version of the tape, the recording ended there. In another version, the contestant knocked over a cup of coffee. Aronson found that subjects rated the coffee-spilling contestant as more likeable than the flawless one. Competence alone is admirable but slightly alienating. Competence plus a visible imperfection is magnetic.

Aronson called this the Pratfall Effect, and it only works in one direction. An already-incompetent person who spills coffee just looks clumsy. An already-impressive person who spills coffee becomes human, relatable, trustworthy. The imperfection doesn't diminish the halo. It deepens it.

Now consider what happens when one of the most celebrated AI companies in the world steps up to a microphone and says, essentially, "We've built something so powerful that even we are worried about it." That admission is the spilled coffee. It's the controlled vulnerability that, paradoxically, amplifies trust. The company didn't have to confess this. Nobody forced the disclosure. The voluntary nature of the warning is precisely what makes it so persuasive. These people are so honest, so responsible, that they're willing to admit their own creation might be dangerous. The only kind of person who does that, in the audience's imagination, is a person who genuinely means it.

Except there's a small problem with this reading. And the problem can be found, of all places, in the Nvidia press archive for 2019.

The Dog That Didn't Bark

In February 2019, OpenAI published its blog post declaring GPT-2 too dangerous to release. The text generator had 1.5 billion parameters and could produce a few paragraphs of passable prose from a short prompt. It was, by any honest technical assessment, a toy. But the headlines were volcanic. The story ran on front pages and news broadcasts and tech blogs across the planet, all within hours. A model that couldn't reliably finish a paragraph was suddenly being discussed in the vocabulary of nuclear proliferation.

Around the same time, Nvidia open-sourced a language model called MegatronLM with 8.3 billion parameters. More than five times the size of GPT-2. Nvidia held no press conference and published no blog post. There were no dire warnings about the end of the world. Nobody testified before Congress about MegatronLM.

And nobody covered it.

This is the controlled experiment that reveals what's actually happening. Two companies, two language models. One declares its smaller model too dangerous and captures the world's attention. The other releases its larger model without fanfare and gets silence. The variable isn't the technology. It's the narrative.

Five months after the "too dangerous" announcement, Microsoft invested a billion dollars in OpenAI. That investment was the seed capital for a trajectory that would carry the company from a billion-dollar valuation to a confidential IPO filing targeting a trillion. Nvidia's MegatronLM, released without the theater of danger, generated no comparable investment event. It fell in the forest and didn't make a sound.

Arthur Conan Doyle wrote a Sherlock Holmes story in which the crucial clue was a dog that didn't bark. The detective solved the case by noticing what was absent, not what was present. Nvidia's silence is the dog that didn't bark in the AI story. It shows us, by clean contrast, that the danger announcement is doing something the technology alone can't do. The danger is converting technical capability into attention. The attention is converting into capital. The capital is converting into valuation. And the valuation is converting into a kind of power that has nothing to do with whether the technology actually works.

But if that's the business model, then what happens when the technology does start to work? When the danger being announced isn't a phantasm anymore?

The Only Man Who Meant It

In 1956, a science fiction film called Forbidden Planet introduced a character named Dr. Morbius, a philologist stranded on the planet Altair IV. Morbius discovers an ancient alien technology of almost incomprehensible power: a machine built by a long-extinct civilization called the Krell, capable of materializing thought into reality. The Krell had destroyed themselves with this machine. Their own subconscious desires, amplified to planetary scale, had wiped their entire species from existence overnight.

Morbius concludes that the machine is too dangerous for humanity to possess. And then he does something none of the AI companies have done. He destroys it. He triggers the planet's self-destruct sequence and annihilates the technology rather than let it fall into any hands at all, including his own.

That's the version of "too dangerous" where someone actually means it. The version where the warning is not a communications strategy but a reckoning. The version where the person making the claim follows through with the only action consistent with genuinely believing it.

Now compare. Morbius destroyed the machine. OpenAI released the full model nine months later, acknowledged "no strong evidence of misuse," and went back to building the next one. The heads of OpenAI, Google DeepMind, and Anthropic signed a statement in May 2023 saying their work could cause the extinction of the human race, and went back to the office the next morning. Elon Musk called AI "summoning the demon" in 2014 and founded his own AI company in 2023. They warn, and they build. They warn, and they build. And not one of them ever stops.

The gap between what Morbius did and what these companies do is the gap between belief and performance. And the size of that gap, measured in dollars, is now approaching the thirteen-figure range.

The Trillion-Dollar Discovery

What the AI industry has stumbled onto, whether by design or by accident, is something the Catholic Church, the MPAA film ratings board, every publisher who ever slapped a "BANNED" label on a paperback, and every nightclub owner who ever installed a velvet rope already knew. Restriction is the oldest and most efficient amplifier of desire.

But the AI version improves on all the historical precedents. The Church's Index required an external authority to issue the ban. Film ratings required an industry body to assign the classification. The AI companies have eliminated the middleman. They ban their own products. And in doing so, they activate every psychological mechanism simultaneously: reactance (you can't have this), the Halo Effect (because we're brilliant), Pascal's Wager (and if you ignore us you might miss the most important technology in history), and the Pratfall Effect (and we're so honest we'll tell you ourselves). Four overlapping mechanisms, all triggered by a single press release. The combined force is staggering.

OpenAI was valued at roughly a billion dollars in February 2019, when it published the "too dangerous" blog post about GPT-2. By June 2026, the company was confidentially filing for an IPO at a valuation above a trillion dollars. It burns twenty-seven billion a year. It doesn't turn a profit. It projects it won't be cash-flow positive until 2030. The distance between what the machine can do and what the announcement says the machine can do is not an error. It's the margin. It's where the value lives.

The historical analogies, though, don't prepare you for the final implication. You never have to deliver a technology that eliminates most white-collar jobs, much less one that ends humanity. You only have to deliver the press campaign that draws sufficient attention to make the world believe you might. The Vatican never had to prove God existed. It only had to maintain an Index that implied some books were dangerous enough to require divine protection from their ideas. The enforcement of the ban was the evidence for the ban. The restriction was the proof.

The AI industry's version of this discovery can be stated in a single sentence: the announcement is not a brake on the product; the announcement is the product.

One of the earliest theorists of advertising, Claude Hopkins, wrote in 1923 that the goal of an ad was never to describe a product. It was to create a desire so specific that only one product could satisfy it. A century later, the AI industry has refined Hopkins' insight into its purest form. The product isn't artificial intelligence. The product is the belief that artificial intelligence is so powerful it could destroy civilization. And the proof that you should believe this is that the people building it are brave enough to say so.

The Index worked for four hundred years. This version has been running for seven and has already produced a trillion-dollar filing. Whether it lasts another four centuries or collapses under the weight of its own thermodynamics is, in a sense, beside the point. The discovery has been made. The playbook has been written. And the next company that builds something roughly functional and announces that it might end the world will find a line of investors waiting at the door before the press conference is over.

The video that started me down this path traces the specific mechanics of the playbook, follows the man who built it at one company and ran it again at another, and names the precise dollar amounts that accumulated at each stage of the performance. If you haven't watched it yet, I'd start there.

Julian Whatley is a macro-perceptual analyst and the creator of The Silicon Mirage on YouTube. His premium newsletter, The Signal Report, is available at julianwhatley.com.