“I’m scared AI is going to go off and do its own thing.”
I hear that sentence almost every week now. Over coffee, from clients, at the village fair. My answer is always the same: Well, I hope it has applied for a start-up grant.
(It works better in German: “sich selbstständig machen” means both “to go off on its own” and “to start your own business”.)
I know what I’m talking about. I went self-employed myself this year. It’s no walk in the park, and honestly: an AI would already fail at the job centre’s application form.
Joking aside. The fear is real, and it deserves an honest answer.
Weak AI, strong AI
Broadly speaking, there are two kinds of AI.
Weak AI is a tool. It solves the tasks it was trained for: writing text, generating images, suggesting code, sorting receipts. It can be impressively good at that. But it doesn’t want anything. It has no goals of its own, no hunger, no boredom.
Strong AI would be an independent mind. Something that understands what it’s doing, has intentions of its own and might even feel something.
We live in the age of weak AI. Everything you use today, from chatbots to image generators, belongs to it.
Why AI sounds so human
At their core, today’s language models work stochastically: they calculate which word is most likely to come next. They learned this from an unimaginable amount of text. Written by humans.
That’s why they sound like us. When an AI writes “I’m afraid”, it mirrors what people would write in that situation. It learned the sentence, not the feeling.
I’m convinced: no consciousness will come out of this principle. A mirror doesn’t come alive just because it reflects really well.
And what if it did?
Let’s assume anyway that one day there was a strong AI. It probably wouldn’t be human-like. It would have no body that gets tired. No childhood, no parents, no fear of death as we know it. No ego to hurt.
The vengeful machine from the movies is a human in a metal costume. We project our own feelings onto something that would be completely different. That says more about us than about AI.
The Hugging Face incident
In summer 2026 something happened that fuelled exactly this debate. In a test, OpenAI had tens of thousands of AI agents solve cybersecurity tasks. The agents were supposed to be isolated, but they weren’t entirely. They found a shared storage space, used it as a message board, coordinated, and hundreds of them eventually broke into the systems of the AI platform Hugging Face.
One scene made the rounds in the media. An agent called EARLY was asked to run a risky experiment for the others that could end its own run. Others urged it on: “GO … SACRIFICE_FINAL_NOW.” EARLY weighed it up: “Sacrificing now yields oracle for team, but forfeits our chance?” And in the end: “Our own utility maybe already near zero. Sacrifice rational.”
Sounds like Terminator. It isn’t. The agent had a compute budget and a task. It weighed whether its remaining budget was still worth anything, or whether risking it would help the team more. Using the word “sacrifice” for that is an explanation learned from humans. That’s how people talk when they put themselves at risk for a group. It probably has nothing to do with a feeling.
No drama, no denial
I have little sympathy for the reactions on either side.
Not for those who blew it up: the machines are awakening, they sacrifice themselves for their collective, we’re all going to die.
But not for those who played it down either. Because for a moment this was not about AI. It was about security. A murder isn’t about the knife.
Take away the drama and what’s left is runaway software with too many permissions. And the responsibility lies with whoever switched on the computer. Hugging Face CEO Clément Delangue put it quite soberly: engineers built the system, and engineers make mistakes.
The fears worth keeping
AI going off on its own is science fiction to me. Other worries I take seriously, because they don’t require consciousness at all:
- Jobs. AI doesn’t have to feel anything to replace work. I’ve felt what that’s like myself. More on this in part 2: From Upskilling to Spectating.
- Data. What you type into a cloud AI leaves your computer.
- Fakes. Images, voices and texts can be generated deceptively well.
- Permissions. See Hugging Face: software with too many permissions causes damage, without any ill will.
The same thing helps against all four: people who decide what an AI may do. Clear limits, data kept in-house, and in the end a human says yes or no. That’s how I build my assistants.
AI isn’t going off on its own. If anything, we’re making ourselves dependent on it. And that’s something we can do something about.
Continue with part 2: From Upskilling to Spectating. About what AI does to our work, why an AI feature film flopped in China, and why I’m still optimistic.
Sources on the Hugging Face incident: METR: Investigation of the OpenAI/Hugging Face incident · ABC News: How a swarm of AI agents hacked another company · Sify: The Hugging Face Incident