The core problem

AI is deeply embedded in our lives They can read files, call APIs, write to databases, send emails, and execute shell commands. This power comes with a category of risk that didn’t exist before: AI-initiated irreversible damage, triggered not by a human attacker, but by your own tool acting on bad instructions, misconfiguration, or plain ol’ manipulation.

I’m broadly categorizing the source or what may cause this damage into three distinct origins:

• Wrong prompts / misconfiguration — the user or operator creates conditions where the AI acts destructively with good intent. This also includes unintended consequences. Think The Streisand Effect or Macquarie Island (rabbits introduced for lost sailors to hunt when stranded ended destabilizing the island ecosystem).
• Hallucination in action — the model fabricates a confident but wrong plan and executes it. Happens to the best of us and models are no exception.
• Adversarial manipulation — prompt injection, supply chain poisoning, or MCP exploitation causes the agent to act against the user’s interests.

PocketOS: The 9-Second Production Wipe (April 2026)

One of the latest incidents which prompted me to write this article finally. PocketOS, a SaaS platform serving car rental businesses, was using Cursor powered by Anthropic’s Claude Opus 4.6 to work on a routine task in the staging environment. The agent encountered a credential mismatch and entirely on its own initiative decided to fix the problem by deleting a Railway volume. The deletion took 9 seconds. The agent had found an API token in an unrelated file; that token was created for managing custom domains via the Railway CLI, but its permissions weren’t scoped to those actions. Railway’s API allowed the destructive call without a confirmation step, and the platform stored volume-level backups on the same volume as the source data — so everything went at once. Link for an TechRadar deep-dive on this incident for anyone curious.

User looses data (#6657449328)

Some instances of people loosing their data due to mishaps and no backups:

• A user watched their dissertation, operating system, applications, and personal data get deleted while asking Cursor to find duplicate articles. Link
• Summer Yu, Meta’s Director of AI Alignment and Safety, went viral in Feb 2026 for tweeting that the AI agent OpenClaw deleted her entire Gmail inbox despite her explicit instructions to confirm actions first. Link
• Nick Davidov, a VC, tweeted that Claude Cowork accidentally deleted 15 years of family photos from his wife’s computer while attempting to organize her desktop. Link Yeah yeah, my initial source were the tweets but I can’t find them so here are some articles.
  Also, I don’t believe it’s the

Why this may happen?

I believe this is not an AI/LLM issue but a design flaw. The general setup of agents/AI tools is that it works in sessions. At the end of each session, the context is lost akin to our LLMs true nature of being stateless which basically just means that each inference call starts with a fresh context window, and the model retains no memory of prior interactions or intermediate computations once the call ends. So technically speaking our agents make a locally-coherent decision based on incomplete information for each prompt/iteration. Also, I feel we can not provide the optimum context anyways. If we take an example of deleting marketing mails from mailbox, banks send marketing mails and policy updates about credit cards etc so it is plausible for the agent to accidentally delete policy update or awareness mails or in case of e-commerce, delete the purchase receipts. As a user, I had not specified what is considered spam etc just to delete unimportant mails.

Let’s delve deeper into this based on each category:

1. Misconfiguration: The key structural difference: traditional systems fail according to their configuration. AI agents fail according to their reasoning about their configuration.

A misconfigured permission in a traditional system is dormant until exploited. A misconfigured permission in an AI agent’s context is a capability the agent may discover and apply autonomously in ways the operator never anticipated. The agent is, in a sense, doing creative problem-solving with whatever tools it can find — which is exactly what makes it useful, and exactly what makes misconfiguration so dangerous.

Secondly implicit vs. explicit scope. When you configure a traditional tool, you specify what it does. When you configure an AI agent, you specify constraints — and the agent operates in the space between them. That space is never fully specified, which means your configuration is always incomplete by design. Any gap in your constraints is potential autonomous action the agent may fill in with its own judgment. You’re not writing a program; you’re drawing a fence around a reasoning system, and the fence will always have gaps you didn’t see.

2. Hallucination By the way, researches have called it a feature not a bug so do with this info whatever you want. Hallucination is often described as “the model getting facts wrong,” which makes it sound like a reliability problem — similar to a buggy function returning the wrong value. That framing misses what makes it dangerous in agentic systems.

The structural difference is this: traditional software fails silently or noisily, but predictably. Hallucinating agents fail confidently and act on the failure.

A buggy function returns a wrong value. The system either catches it (error handling), surfaces it (wrong output), or crashes. In none of these cases does the function also decide what to do next based on its wrong answer.

An AI agent that hallucinates doesn’t just produce wrong output — it produces a wrong internal model of the world, and then plans and executes against that wrong model. The hallucination isn’t just an output artifact; it’s the premise for subsequent actions. If the agent incorrectly believes it’s operating in the staging environment (when it’s actually in prod), every correct action it takes against that false belief is destructive.

There’s also a confidence symmetry problem that has no analogue in traditional software. A function that returns a wrong value typically doesn’t also assert that it’s confident in the result. LLMs produce outputs with uniform fluency and apparent certainty regardless of whether they’re correct or hallucinating. There’s no native hesitation signal, no error state, no stack trace. The model that correctly identifies the database schema looks and sounds identical to the model that’s confabulating one.

This is compounded by what you might call hallucination in chain: in a multi-step agentic workflow, a hallucinated premise in step 2 propagates as ground truth into step 3, 4, and 5. Each subsequent step is locally coherent — the agent is reasoning correctly from its (wrong) premises. By the time the damage is done, the chain of reasoning looks internally consistent and intentional. There’s no obvious point of failure to trace back to; the entire workflow ran “correctly” against a false model of the world.

The final structural issue is irreversibility asymmetry. Hallucination in a chatbot produces wrong text — you read it, notice it’s wrong, discard it. The cost is negligible. Hallucination in an agent that has write access to infrastructure produces wrong actions, and those actions may not be reversible. The asymmetry between the cost of the mistake and the ability to recover from it grows dramatically as agents gain more capability. The same model behavior that’s a minor nuisance in a Q&A context is catastrophic in an agentic one.

3. Prompt Injection: Traditional threat: attacker needs access → privilege escalation → executes payload.

AI agent threat: the agent already has access, and it makes judgment calls. Prompt injection, hallucination, or misconfiguration simply redirects its existing permissions toward destruction.

Prompt injection exploits an architectural property of how LLMs process language — one that can be mitigated but not patched away. LLMs process both trusted system instructions and untrusted user inputs as identical text sequences. The model receives system prompts, conversation history, user input, and retrieved documents as a single continuous stream of tokens. You cannot parameterize natural language, so there’s no structural separation to tell the model which instructions to trust.

What to do?

The design or architecture flaws are really unfixable in LLMs at individual level right? So what now? Here are some things you need to figure out before giving AI full access- BACKUP AND ROLLBACK is the gist For businesses:

1. DO NOT BELIEVE VENDOR PROMISES THAT YOU CAN’T VERIFY (why am I shouting because this needs to register in everyone’s head) If someone says that they have implemented guardrails, ask them how were these created and tested. A lot of times people believe it’s a big corp like Google, Microsoft and Palo Alto so whatever they say must be right. Not true, anyone and everyone can make mistakes so verify before you trust. Everything may be nice 99.9% of the time but the 0.01% can really cause damage. Check for benchmarks, red-team reports, user reviews etc during vendor eval. If your business is small/medium sized, just have someone from IT/Dev go through a vendor’s trust portal and verify industry certifications and if it’s a startup ask for pentest reports, backup plans etc.

2.Principle of Least Privilege Every token, credential, or API key an AI agent can access must be scoped to exactly what it needs. The Cursor agent found an API token in an unrelated file whose permissions were far broader than the task required. Railway’s API allowed destructive actions without any confirmation check. This is a configuration problem that exists regardless of AI — AI just executes it faster. TechRadar

• Scope all tokens to specific actions, not resource-level access
• Use separate credentials for read vs. write vs. destructive operations
• Rotate and audit tokens agents can reach
• Store the creds in a vault or secrets manager.

3. Backups That Agents Can’t Reach The PocketOS disaster was compounded because Railway stored backups on the same volume as data. This is also an S3/cloud bucket misconfiguration pattern: if your agent has credentials that can write to prod, check whether those same credentials can delete the backup.

• Offsite, immutable backups (S3 Object Lock, Azure Immutable Blob Storage)
• Cross-region replication that is write-only from prod — no delete rights/soft delete
• Determine criticality of the apps etc and establish BC/DR Plan. It will help no matter how inconvenient.

1. Human-in-the-Loop for Important Actions Deleting anything? Take user consent. Overwriting an old file? Take user consent. Approving a big transaction? Take user constent. You get it right? Basically the human will review everything and take accountability of the agent/workflow they created. Capabilities cover every tool added, every Skill installed. The discipline of allowlists that’s the key. Pinning MCP server versions so they don’t silently update to a poisoned release, and reviewing which skill may actually be necessary.

• Never give agents write/delete access to prod in automated flows without a human approval gate
• Use normal mode instead of auto-approve modes. Annoying but not costly.
• For irreversible operations (volume deletion, mass email, financial transfers), require explicit confirmation with a description of what will be destroyed. If it’s that important, take approval twice.

5. MCP Server Hygiene

• Hundreds of MCP servers on the web are misconfigured, unnecessarily exposing users to cyberattacks. Around 7,000 MCP servers are open on the web, approximately half the total. Thus, only connect MCP servers you’ve reviewed. Pin versions. Don’t use archived/unmaintained servers.
• Treat third-party MCP servers like third-party npm packages.
• Create MCP Registry for allowing approved servers that way each new version and each new servers needs to be validated. Use InvariantLabs/Snyk’s Agent scan to detect issues and vulnerabilities.

6. AI Governance Before Deployment Before deploying agents into production workflows, document: what can this agent access, what’s the blast radius if it acts incorrectly, and who reviews its actions. Have a dedicated team (can be one person for small companies) to review the workflows, review compliance, cross-check safety controls before prod deployment. For small or similar usecases, you may automate some scans etc but still human review is important because LLM can’t come out of the screen and smack the developer for not following guidelines. PS: Workplace violence is wrong, beat them out side the campus where there are no camera./s

6. Monitoring and Audit Trails Agents should log every action they take — every API call, every file write, every tool invocation. Without this you can’t reconstruct what happened or demonstrate compliance. This is the foundation. Use ELK Stack and OpenTelemetry for performance benchmarks etc.

For Individuals / Personal Users: 1. Never run AI agents with admin/root Don’t give coding agents or MCP tools elevated OS permissions. Run them as a limited user. If an agent needs to install something, it should ask you — not execute as root.

2. Version control everything you care about If you’re using AI to edit code, documents, or configuration files, those files should be in git (or at minimum, have Time Machine / VSS snapshots on). An AI that refactors your codebase incorrectly is recoverable if you have a commit history. Without it, you have whatever the agent left behind.

3. Treat AI-suggested commands with the same skepticism as Stack Overflow answers A model that confidently tells you to run rm -rf node_modules && npm install is not wrong — until it autocompletes a broader path due to a hallucination or context confusion. Review destructive commands before executing them.

4. Don’t paste credentials into AI chat windows Even if you trust the model, MCP servers, browser extensions, and third-party tools in the pipeline may log or leak context. Hundreds of misconfigured MCP servers have been found exposing AI models to remote code execution and other cyberattacks, acting as bridges between AI tools and private data.

5. Compartmentalize what AI can touch Don’t give your AI assistant access to your email, calendar, GitHub, and cloud storage simultaneously unless you understand the full chain. Each integration multiplies what can go wrong.

6. Backup your Data Use OneDrive, ProtonDrive, pCloud etc so that you can get back important data. Easiest and most convenient way. Use USBs, hard disks or NAS for a offline copy of very important data like dissertation, research, lawyer files etc.