Anthropic's AI Psychosis Is Covering Up a Gambling Problem

This is about Anthropic warning us that AI models may be capable of self-replicating without access to a datacenter. I mean, the title alone is barf-worthy. "AI models" are not the actors here. I happened to read that blogpost the day I was teaching my Governance & Politics of AI lecture on military applications of AI, so that may explain my vitriolic response…

What happens if we allow Anthropic to spread its software haphazardly on the Internet?

I feel a horrible frustrated urgency. The outcome of this is just completely apparent. Like 7 October 2023. The first time I heard anyone (it was Eric Schmidt, in 2017) say "let's replace the military's valuable eyes with cameras and an algorithm" I immediately got up and said (and this is an exact quote) "replacing large numbers of military people with an AI vision system – could that be a single point of failure? Could that be something that could be hacked or manipulated? Whereas, you know, thousands of soldiers would presumably whistle blow if something weird was going on." (2018 blogpost linking to video footage if this.)

The observation that contemporary coding agents can decompose LLM-scale AI onto devices anywhere on the Internet and network those pieces into working systems is also completely obvious.

We've seen this film before. Everything goes down, except maybe this time even Google. Actually, Netflix was one of the things that went down that time, so for a while there were no films. Worse, there was no twitter – we lost the best hive mind we've built to date, although that time at least only transiently. Of course, the guys behind that botnet hack went to jail. As did the guy who accidentally released a worm that took down the Internet back in 1988, even though he was a graduate student and the son of a well-placed professor.

Why would Anthropic want to do this? They're smart, right? 

Anthropic have the most advanced models, and apparently the most advanced case of AI psychosis. They believe that their models (they no doubt have many persona on their models, shaped by different guardrails over their various foundation models – "Claude" is not really one thing; it's a brand) they believe (or say they believe) that their models ought to be given moral consideration. That is, they are trying everything they can do to train something that will both be theirs but not their responsibility. They want the money, power, and glory, but not the liability or jail time.

It may be that they are particularly concerned about this because they are vastly overextended – they are still spending way more than they are earning. Even if all the humans in Anthropic are about to get an insane windfall when they make their shares highly valuable by auctioning off what 5% of their available stock in their IPO? They obviously don't think there's a lot of money out there wanting to buy their company. But if they make shares scarce, they can shore the stock price up and cash in as individuals.

But this doesn't pay the power bills. Maybe they'll be fine, maybe they won't. Maybe they'll get the US to bail them out. Maybe a larger entity should be negotiating with them about the appropriate behaviour of corporations providing essential services.

But apparently much of big tech would rather contribute to the gambling addictions of millions of people, destroying lives, than admit it has a problem itself.

What should we do if Anthropic do break the Internet with their genAI "agents". 

I wrote about that before. As well as just above. Everyone (every human) with responsibility (only humans can have responsibility in human society) for holding this company to account should be held to account if the company breaks anything. Including the enforcers who should be holding corporations in line.

Absolutely everyone who works with genAI at all should be working to use it also to enhance their own agency, including their own understanding of their actions and those actions' consequences. Because humans are little else when you get down to it than animals who are responsible to each other. But that's been enough to help us build some really cool stuff.

Do you seriously think this could work? Aren't LLMs huge and energy hungry?

Yeah, I think people with enough money and time could build systems distilling the work done on foundation models, and make miniature systems able to run on low-powered computing and network into interesting systems. In fact, they already do; people have been running LLM on phones for years. I would love for Anthropic engineers to spend time doing that and saving the planet some resources. But not in an uncontrolled way. The first thing they should be building is the cybersecure ring-fencing, and then move into using online devices clearly, deliberately labelled by their owners as open for such collaborating. You know, like SETI@home used to.

A slide I often give in talks, referencing the whole "stochastic parrots" thing.
Parrots have societies and enforced social expectations – responsibility to each other.
Software threads that can be perfectly cloned and costlessly deleted cannot be responsible.