Distributing risk

But AI chatbots can be more harmful to your security than traditional apps (and we know those have been capable of bad things enough). As we detailed in our work on trust in AI assistants, AI chatbots and agents pose new security and privacy risks, as not only can they access sensitive data (allowing the companies developing them to access this data as well), but their functionality widens the traditional software attack surface in ways that are currently not well understood nor mitigated. 

For example, attacks have demonstrated how a file shared with you on Google Drive could be used to manipulate an AI chatbot connected to this drive. In another example, a security researcher was able to show how, through prompt injection, a malicious calendar invite could lead to the disclosure of emails, through the connectors on Google Calendar and Gmail.

The AI Industry’s response to this risk is for the user to ‘monitor’ through their interfaces. Poor user interface design has already led to ridiculous outcomes, like OpenAI’s shared chat failure leading to the public exposure of thousands of personal chats.

If we are to continue giving AI chatbots more access into our personal lives and data, we need more than basic user-interface toggles or warnings about prompt injection hidden in documentation like these:

• Claude: “This means Claude can be tricked into sending information from its context (e.g., prompts, projects, data via MCP, Google integrations) to malicious third parties. To mitigate these risks, we recommend you monitor Claude while using the feature and stop it if you see it using or accessing data unexpectedly.”
• ChatGPT: “ChatGPT agent incorporates multiple safeguards, including user confirmations for high-impact actions, refusal patterns for disallowed tasks, prompt injection monitoring, and a “watch mode” requiring user supervision on certain sites. These measures are designed to help prevent harmful or unintended outcomes. However, these measures don’t eliminate all risks. It remains important to monitor ChatGPT agent and exercise care when using it.”

This duty placed upon the user to ‘monitor’ feels quite different from the promise of powerful AI tools easily responding to our everyday needs.

Prompt injections aren’t going away. The AI industry can’t seriously believe that it’s down to users to navigate the risks and consequences of connecting to external services in an era when they are selling these tools as being useful because of these connectors.

Put another way: the industry needs to resolve who bears the risk of giving tools write access to external systems perhaps before they sell these tools on the virtues that they write access to external systems?

This should be a no-brainer and yet it needs to be said: the security and privacy of users must always be a priority over feature roll-out, and no money race can justify compromises.