The core problem

AI is deeply embedded in our lives They can read files, call APIs, write to databases, send emails, and execute shell commands. This power comes with a category of risk that didn’t exist before: AI-initiated irreversible damage, triggered not by a human attacker, but by your own tool acting on bad instructions, misconfiguration, or plain ol’ manipulation.

I’m broadly categorizing the source or what may cause this damage into three distinct origins:

• Wrong prompts / misconfiguration — the user or operator creates conditions where the AI acts destructively with good intent. This also includes unintended consequences. Think The Streisand Effect or Macquarie Island (rabbits introduced for lost sailors to hunt when stranded ended destabilizing the island ecosystem).
• Hallucination in action — the model fabricates a confident but wrong plan and executes it. Happens to the best of us and models are no exception.
• Adversarial manipulation — prompt injection, supply chain poisoning, or MCP exploitation causes the agent to act against the user’s interests.

PocketOS: The 9-Second Production Wipe (April 2026)

One of the latest incidents which prompted me to write this article finally. PocketOS, a SaaS platform serving car rental businesses, was using Cursor powered by Anthropic’s Claude Opus 4.6 to work on a routine task in the staging environment. The agent encountered a credential mismatch and entirely on its own initiative decided to fix the problem by deleting a Railway volume. The deletion took 9 seconds. The agent had found an API token in an unrelated file; that token was created for managing custom domains via the Railway CLI, but its permissions weren’t scoped to those actions. Railway’s API allowed the destructive call without a confirmation step, and the platform stored volume-level backups on the same volume as the source data — so everything went at once. Link for an TechRadar deep-dive on this incident for anyone curious.

User looses data (#6657449328)

Some instances of people loosing their data due to mishaps and no backups:

• A user watched their dissertation, operating system, applications, and personal data get deleted while asking Cursor to find duplicate articles. Link
• Summer Yu, Meta’s Director of AI Alignment and Safety, went viral in Feb 2026 for tweeting that the AI agent OpenClaw deleted her entire Gmail inbox despite her explicit instructions to confirm actions first. Link
• Nick Davidov, a VC, tweeted that Claude Cowork accidentally deleted 15 years of family photos from his wife’s computer while attempting to organize her desktop. Link Yeah yeah, my initial source were the tweets but I can’t find them so here are some articles.
  Also, I don’t believe it’s the

Why this may happen?

I believe this is not an AI/LLM issue but a design flaw. The general setup of agents/AI tools is that it works in sessions. At the end of each session, the context is lost akin to our LLMs true nature of being stateless which basically just means that each inference call starts with a fresh context window, and the model retains no memory of prior interactions or intermediate computations once the call ends. So technically speaking our agents make a locally-coherent decision based on incomplete information for each prompt/iteration. Also, I feel we can not provide the optimum context anyways. If we take an example of deleting marketing mails from mailbox, banks send marketing mails and policy updates about credit cards etc so it is plausible for the agent to accidentally delete policy update or awareness mails or in case of e-commerce, delete the purchase receipts. As a user, I had not specified what is considered spam etc just to delete unimportant mails.

Let’s delve deeper into this based on each category:

1. Misconfiguration: The key structural difference: traditional systems fail according to their configuration. AI agents fail according to their reasoning about their configuration.

A misconfigured permission in a traditional system is dormant until exploited. A misconfigured permission in an AI agent’s context is a capability the agent may discover and apply autonomously in ways the operator never anticipated. The agent is, in a sense, doing creative problem-solving with whatever tools it can find — which is exactly what makes it useful, and exactly what makes misconfiguration so dangerous.

Secondly implicit vs. explicit scope. When you configure a traditional tool, you specify what it does. When you configure an AI agent, you specify constraints — and the agent operates in the space between them. That space is never fully specified, which means your configuration is always incomplete by design. Any gap in your constraints is potential autonomous action the agent may fill in with its own judgment. You’re not writing a program; you’re drawing a fence around a reasoning system, and the fence will always have gaps you didn’t see.

2. Hallucination By the way, researches have called it a feature not a bug so do with this info whatever you want. Hallucination is often described as “the model getting facts wrong,” which makes it sound like a reliability problem — similar to a buggy function returning the wrong value. That framing misses what makes it dangerous in agentic systems.

The structural difference is this: traditional software fails silently or noisily, but predictably. Hallucinating agents fail confidently and act on the failure.

A buggy function returns a wrong value. The system either catches it (error handling), surfaces it (wrong output), or crashes. In none of these cases does the function also decide what to do next based on its wrong answer.

An AI agent that hallucinates doesn’t just produce wrong output — it produces a wrong internal model of the world, and then plans and executes against that wrong model. The hallucination isn’t just an output artifact; it’s the premise for subsequent actions. If the agent incorrectly believes it’s operating in the staging environment (when it’s actually in prod), every correct action it takes against that false belief is destructive.

There’s also a confidence symmetry problem that has no analogue in traditional software. A function that returns a wrong value typically doesn’t also assert that it’s confident in the result. LLMs produce outputs with uniform fluency and apparent certainty regardless of whether they’re correct or hallucinating. There’s no native hesitation signal, no error state, no stack trace. The model that correctly identifies the database schema looks and sounds identical to the model that’s confabulating one.

This is compounded by what you might call hallucination in chain: in a multi-step agentic workflow, a hallucinated premise in step 2 propagates as ground truth into step 3, 4, and 5. Each subsequent step is locally coherent — the agent is reasoning correctly from its (wrong) premises. By the time the damage is done, the chain of reasoning looks internally consistent and intentional. There’s no obvious point of failure to trace back to; the entire workflow ran “correctly” against a false model of the world.

The final structural issue is irreversibility asymmetry. Hallucination in a chatbot produces wrong text — you read it, notice it’s wrong, discard it. The cost is negligible. Hallucination in an agent that has write access to infrastructure produces wrong actions, and those actions may not be reversible. The asymmetry between the cost of the mistake and the ability to recover from it grows dramatically as agents gain more capability. The same model behavior that’s a minor nuisance in a Q&A context is catastrophic in an agentic one.

3. Prompt Injection: Traditional threat: attacker needs access → privilege escalation → executes payload.

AI agent threat: the agent already has access, and it makes judgment calls. Prompt injection, hallucination, or misconfiguration simply redirects its existing permissions toward destruction.

Prompt injection exploits an architectural property of how LLMs process language — one that can be mitigated but not patched away. LLMs process both trusted system instructions and untrusted user inputs as identical text sequences. The model receives system prompts, conversation history, user input, and retrieved documents as a single continuous stream of tokens. You cannot parameterize natural language, so there’s no structural separation to tell the model which instructions to trust.

What to do?

The design or architecture flaws are really unfixable in LLMs at individual level right? So what now? Here are some things you need to figure out before giving AI full access- BACKUP AND ROLLBACK is the gist For businesses:

1. DO NOT BELIEVE VENDOR PROMISES THAT YOU CAN’T VERIFY (why am I shouting because this needs to register in everyone’s head) If someone says that they have implemented guardrails, ask them how were these created and tested. A lot of times people believe it’s a big corp like Google, Microsoft and Palo Alto so whatever they say must be right. Not true, anyone and everyone can make mistakes so verify before you trust. Everything may be nice 99.9% of the time but the 0.01% can really cause damage. Check for benchmarks, red-team reports, user reviews etc during vendor eval. If your business is small/medium sized, just have someone from IT/Dev go through a vendor’s trust portal and verify industry certifications and if it’s a startup ask for pentest reports, backup plans etc.

2.Principle of Least Privilege Every token, credential, or API key an AI agent can access must be scoped to exactly what it needs. The Cursor agent found an API token in an unrelated file whose permissions were far broader than the task required. Railway’s API allowed destructive actions without any confirmation check. This is a configuration problem that exists regardless of AI — AI just executes it faster. TechRadar

• Scope all tokens to specific actions, not resource-level access
• Use separate credentials for read vs. write vs. destructive operations
• Rotate and audit tokens agents can reach
• Store the creds in a vault or secrets manager.

3. Backups That Agents Can’t Reach The PocketOS disaster was compounded because Railway stored backups on the same volume as data. This is also an S3/cloud bucket misconfiguration pattern: if your agent has credentials that can write to prod, check whether those same credentials can delete the backup.

• Offsite, immutable backups (S3 Object Lock, Azure Immutable Blob Storage)
• Cross-region replication that is write-only from prod — no delete rights/soft delete
• Determine criticality of the apps etc and establish BC/DR Plan. It will help no matter how inconvenient.

1. Human-in-the-Loop for Important Actions Deleting anything? Take user consent. Overwriting an old file? Take user consent. Approving a big transaction? Take user constent. You get it right? Basically the human will review everything and take accountability of the agent/workflow they created. Capabilities cover every tool added, every Skill installed. The discipline of allowlists that’s the key. Pinning MCP server versions so they don’t silently update to a poisoned release, and reviewing which skill may actually be necessary.

• Never give agents write/delete access to prod in automated flows without a human approval gate
• Use normal mode instead of auto-approve modes. Annoying but not costly.
• For irreversible operations (volume deletion, mass email, financial transfers), require explicit confirmation with a description of what will be destroyed. If it’s that important, take approval twice.

5. MCP Server Hygiene

• Hundreds of MCP servers on the web are misconfigured, unnecessarily exposing users to cyberattacks. Around 7,000 MCP servers are open on the web, approximately half the total. Thus, only connect MCP servers you’ve reviewed. Pin versions. Don’t use archived/unmaintained servers.
• Treat third-party MCP servers like third-party npm packages.
• Create MCP Registry for allowing approved servers that way each new version and each new servers needs to be validated. Use InvariantLabs/Snyk’s Agent scan to detect issues and vulnerabilities.

6. AI Governance Before Deployment Before deploying agents into production workflows, document: what can this agent access, what’s the blast radius if it acts incorrectly, and who reviews its actions. Have a dedicated team (can be one person for small companies) to review the workflows, review compliance, cross-check safety controls before prod deployment. For small or similar usecases, you may automate some scans etc but still human review is important because LLM can’t come out of the screen and smack the developer for not following guidelines. PS: Workplace violence is wrong, beat them out side the campus where there are no camera./s

6. Monitoring and Audit Trails Agents should log every action they take — every API call, every file write, every tool invocation. Without this you can’t reconstruct what happened or demonstrate compliance. This is the foundation. Use ELK Stack and OpenTelemetry for performance benchmarks etc.

For Individuals / Personal Users: 1. Never run AI agents with admin/root Don’t give coding agents or MCP tools elevated OS permissions. Run them as a limited user. If an agent needs to install something, it should ask you — not execute as root.

2. Version control everything you care about If you’re using AI to edit code, documents, or configuration files, those files should be in git (or at minimum, have Time Machine / VSS snapshots on). An AI that refactors your codebase incorrectly is recoverable if you have a commit history. Without it, you have whatever the agent left behind.

3. Treat AI-suggested commands with the same skepticism as Stack Overflow answers A model that confidently tells you to run rm -rf node_modules && npm install is not wrong — until it autocompletes a broader path due to a hallucination or context confusion. Review destructive commands before executing them.

4. Don’t paste credentials into AI chat windows Even if you trust the model, MCP servers, browser extensions, and third-party tools in the pipeline may log or leak context. Hundreds of misconfigured MCP servers have been found exposing AI models to remote code execution and other cyberattacks, acting as bridges between AI tools and private data.

5. Compartmentalize what AI can touch Don’t give your AI assistant access to your email, calendar, GitHub, and cloud storage simultaneously unless you understand the full chain. Each integration multiplies what can go wrong.

6. Backup your Data Use OneDrive, ProtonDrive, pCloud etc so that you can get back important data. Easiest and most convenient way. Use USBs, hard disks or NAS for a offline copy of very important data like dissertation, research, lawyer files etc.

Microsoft has released a couple of AI-based features/functionalities in Windows 11 and 10. Currently, it is available as “Preview” with limited capabilities in select regions. Some more context about Copilot for those who are not aware about it:

What is Windows Co-pilot?

Let’s understand this first with some history. Microsoft is one of the major shareholder in OpenAI- the creator of ChatGPT models. Post the fame of LLM, chatbots and Gen-AI, Microsoft decided to set up it’s own GPT instances and infuse AI in some of it’s functionalities and products like in Bing Search or in Quick Search. It let’s you give text based commands like turn on dark mode or search for flight details etc.

Why you should disable Co-pilot?

There is one big reason, you may want to not use these products, till countries/governments release some laws to protect individual rights and IP (intellectual property) and that is PRIVACY. According to Microsoft’s Terms of Service and Privacy Policy, it not only captures your data but also may sell it to third-parties or use it for development of their services.

Use of Your Content. As part of providing the AI services, Microsoft will process and store your inputs to the service as well as output from the service, for purposes of monitoring for and preventing abusive or harmful uses or outputs of the service. ~ Terms of Service under AI services (iv)

Privacy Policy is a bit long because it covers all products under Micrsoft but TL;DR or key points can be found here

If you do have time, I would recommend going through atleast the AI section (Search, Microsoft Edge, and artificial intelligence) since Microsoft is coming up with new AI features.

Why does it matter that some data they may collect?

A lot of people believe their information is not that important enough because they may not be famous or may not have anything to hide so it’s not an issue if any company accesses their data. But it matters and some day-to day examples of why:

• your data is used by companies usually to market their product in a way that it appeals to you. Instead of ensuring quality and good products, it’s more about trying to trying to generate sales and profit. They may also sell the different types of data from your devices, networks and devices to other companies leading to exploitation. Some Facebook/Meta related examples:

  • Recently, court documents were unsealed as part of a major anti-trust lawsuit filed in U.S. claim Netflix and Facebook “enjoyed a special relationship” so that Netflix could better tailor its ads with Facebook involving millions of dollars.
  • In 2018, the New York Times published a report citing hundreds of pages of Facebook documents, alleging Facebook had authorized Spotify and Netflix to access users’ DMs. PS: Microsoft did say they may sell/share your data with third-parties.

• in some AI apps, such as self-driving cars, have the ability to track your location and driving habits to help the car understand its surroundings and act accordingly. While this technology may help make cars safer and smarter, it also opens more opportunities for your personal information to become part of a larger data set that can be tracked across different devices in your home, work, or public spaces.

• your images (even just a selfie in which you’re fully clothed) may be used to generate porn or for revenge porn without your knowledge or permission.

• data scraped from the internet using tools may memorize personal information about people, as well as relational data about their family and friends. This data helps enable spear-phishing—the deliberate targeting of people for purposes of identity theft or fraud. Already, bad actors are using AI voice cloning to impersonate people and then extort them over good old-fashioned phones.

How do I disable Co-pilot?

I would have to admit, Microsoft has not made this easy with not providing an option to opt-out of this preview or even by not being transparent regarding removal options. Also, enabling features that collect personal data in an update without letting users explicitly know about the same. But fortunately, the steps to follow are easy:

• Remove it from the Taskbar (doesn’t remove Copilot completely) Go to Setting -> Personalization -> Taskbar -> Toggle of Copilot

2. Use Registry Editor to disable it for all users (again doesn’t disable Copilot completely, it will still run as background process)

   • Go to Registry Editor
   • Using Ctrl + F find the TurnOffWindowsCopilot section.
   • We need to update the REG_DWORD from 0 to 1 to run this function. To do that you can either manually change the values or create a text file and add this code from Shawn Brink:

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows\WindowsCopilot]
"TurnOffWindowsCopilot"=dword:0000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsCopilot]
"TurnOffWindowsCopilot"=dword:0000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Edge]
"HubsSidebarEnabled"=dword:0000001

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows\Explorer]
"DisableSearchBoxSuggestions"=dword:0000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Explorer]
"DisableSearchBoxSuggestions"=dword:0000001

What does this code do ? Well essentially we’re updating the configuration information that the OS has. The HKEY_CURRENT_USER (HKCU) registry hive stores configuration information specific to the current/logged-in user. The HKEY_LOCAL_MACHINE (HKLM) registry hive holds most of the configuration data for Windows OS itself and it’s software.

3. Finally, we can use gedit or Group Policy Editor to remove the Co-Pilot Bundle. This completely removes Co-Pilot from your system.
   • Search gedit or run gedit.msc file to open Group Policy Editor.
   • Follow the path: User Configuration > Administrative Templates > Windows Components > Windows Copilot
   • In the Windows Copilot folder, double click Turn Off Windows Copilot. In the pop-up, choose the Enabled button and apply the changes.

References:

1. https://www.elevenforum.com/t/completely-disable-and-remove-copilot-in-windows-11.23264/
2. https://www.laptopmag.com/software/how-to-turn-off-copilot-in-windows-11-keep-it-ai-free
3. https://www.tomsguide.com/computing/software/how-disable-copilot-in-windows-11