“We really do earnestly believe AI could kill all humans!”, says Anthropic researcher Evan Hubinger on X. He thinks there is more than a 10% chance we could all be dead “within the next decade”. Talk of AI's apocalyptic potential is in the air. Senior figures at AI labs are said to be terrified by the capabilities of recent models, which could be used to create bioweapons or elude human control and go rogue. The panic hit a new level at the weekend when a group of AI CEOs, including Dario Amodei – Hubinger's boss at Anthropic – publicly backed calls for a slowdown in AI development.
Scary stuff. Perhaps Silicon Valley really has been seized by a collective spasm of conscience about the consequences of building AI. Or maybe this is a PR campaign so slick and devious that it would make Alastair Campbell weep. For one thing, the timing is highly suspicious. Anthropic, the AI startup at the centre of the current media storm, is preparing to launch the largest initial public offering (IPO) in history in a matter of weeks.
A basic tenet of critical thinking is to pay attention to vested interests. Dario Amodei is not a neutral commentator. He is trying to secure an IPO valuation of up to $2 trillion. Yet much of the tech media, keen for a dramatic story, uncritically treats the self-serving pronouncements of AI executives as objective statements of fact.
Try 6 free issues of MoneyWeek today
Get unparalleled financial insight, analysis and expert opinion you can profit from.
Start your trial
Sign up for MoneyWeek’s free twice-daily newsletter.
Join more than 165,000 subscribers and keep yourself informed with latest financial news, insights and expert analysis to help you understand what really matters when it comes to your finances.
Join more than 165,000 subscribers and keep yourself informed with latest financial news, insights and expert analysis to help you understand what really matters when it comes to your finances.
The more conspiratorially minded pointed out that the tweet by Jacob Coxon, the Anthropic researcher whose resignation triggered the latest news cycle, was viewed 140 million times despite the fact that his account had almost no prior activity. While Coxon's concerns are probably genuine, the way his message was picked up and boosted by multiple influential figures appears less than organic.
On the face of it, it's not obvious how dire warnings about existential risk would be beneficial to the AI industry. Big tobacco spent years suppressing information about the dangers of its products. So why are AI executives so keen to talk about how their technology could be used by terrorists or lead to the extinction of swathes of white-collar work?
Is Anthropic doom trolling?
The answer, as Cal Newport, a computer science professor at Georgetown University, argues, is the pervasive use of a marketing technique he calls “doom trolling” (a spin on phone addicts' “doom scrolling”). Fear sells. Outrageous claims about the dangers of large language models (LLMs – the currently favoured AI technology) go viral, generating vast amounts of free media coverage for the company that originated them. Talk of existential risk makes AI products appear hugely powerful and desirable.
This buzz helps to distract from the less exciting reality. Yes, LLMs can do impressive things in highly structured domains such as coding and translation, where clear failure conditions help limit their tendency to go off the rails. In other areas (including journalism), their catastrophic tendency to make up information greatly circumscribes their usefulness.
In short, the LLM is a new software category, but investors are not going to pay trillions of dollars for a newer version of Microsoft Excel. Instead, these tools must be imbued with a dark, apocalyptic glamour. Such doom-mongering is longstanding industry practice. As Parmy Olson notes on Bloomberg, in 2019 OpenAI said it would hold back its GPT-2 model from general release on the grounds that it was too dangerous – this for an LLM that struggled to answer primary-school-level reasoning tasks. Terrifying indeed.
This year the AI doom campaign has been turned up to max. Barely a week goes by without claims (all originating from within the AI companies themselves) that a bot has gone on a rogue hacking spree, CEOs, including Anthropic's Dario Amodei, have backed a slowdown in the development of AI or that a new model can't be released because it creates serious cybersecurity risks (it is then released shortly afterwards anyway). The effect has been to generate precisely the sort of frenzied atmosphere that one would want to surround a trio of high-stakes AI-linked IPOs: SpaceX in June, Anthropic scheduled for October, and OpenAI sometime next year.
Who benefits from AI doom trolling?
The AI industry's calls for regulation carry the whiff of “regulatory capture”. Government red tape is more burdensome for upstarts than it is for big established players. New safety regulations could help AI leaders throttle the competition. There is persistent suspicion that Anthropic's Amodei would like to see regulations that effectively excludes his main competitor – cheaper, open-source, often Chinese AI – from major Western countries. If you can't beat them, ban them.
It is also possible that calls for a slowdown represent an attempt to put a brave face on the fact that the AI arms race is becoming too expensive. OpenAI is on course to spend $45 billion this year alone on training and inference (the cost of running AI), but the performance of new models is running into diminishing returns. That isn't an ideal backdrop in which to successfully list a growth company. Instead of breaking the bad news to investors and tanking the valuation, why not piously tell the media you are choosing to slow down development because of your abundant love for humankind?
Don't fall for the hype
What does all this mean for ordinary investors? For starters, don't fall for the tech hype machine. Steer clear of this year's big flashy IPOs, a crowded trade if ever there were one. Secondly, diversify widely. Continue to look for the sort of underexplored investment themes that we cover in depth. And finally, keep your head. There is no knowing how long the current AI fever will last, nor exactly how wide the damage will be when it breaks, but you can at least regain some tranquillity by tuning out the endless talk of doom. Perhaps AI really will kill us all in some hypothetical future. But for now, it is the AI-marketing hype that represents a clear and present danger to our collective mental health.
This article was first published in MoneyWeek's magazine. Enjoy exclusive early access to news, opinion and analysis from our team of financial experts with a MoneyWeek subscription.
