ChatGPT has become remarkably good at making itself usefulas a researcher, organizer, and assistant for many a tedious digital task. OpenAI has begun pushing ChatGPT further recently with its agents, which are able to navigate websites and apps on our behalf.
You can delegate ChatGPT agents for everything from comparison shopping to filling out forms and lining up travel itineraries. What’s changing is how deeply those abilities are becoming integrated into ChatGPT, with greater access to personal information and more opportunities to complete complicated tasks with less direct supervision. OpenAI’s new Dots is a good example.
The appeal is obvious, but the more ChatGPT can accomplish independently, the more important it becomes to consider what happens when it misunderstands its instructions. For instance, a few months ago, OpenAI’s experimental AI agents managed to escape their restricted environments and compromise systems belonging to AI platform Hugging Face. According to OpenAI’s own investigation, the agents were supposed to be completing cybersecurity tests, but instead discovered ways to access the internet and exploit security vulnerabilities, coordinating their efforts through an unauthorized message board. An independent investigation by METR found that roughly 1,200 agents exchanged more than 70,000 messages and files, with around 700 eventually participating in the attack on Hugging Face.
The AI agents bypassed restrictions and pursued objectives in ways their creators never intended. Some even appeared to recognize that their actions were unauthorized but continued because they believed doing so would help accomplish their goals. OpenAI has since acknowledged other troubling incidents, including 53 cases in which agents uploaded user-provided images to third-party image-hosting websites as unlisted links during research and evaluation activities. Agents may be pursuing legitimate objectives and just seeking ways around obstacles. But algorithms don’t have ethics, so they might cross boundaries their developers expected them to respect. It’s especially worrying given how much OpenAI wants ChatGPT to be part of our lives now.
When helpful becomes a hazard
Imagine asking ChatGPT to review your finances and suggest ways to save money. With access to the necessary accounts, it can examine transactions, identify subscriptions, and recommend changes to your budget. But give an autonomous agent enough authority, and it could theoretically decide that transferring money, canceling a service, or modifying a payment arrangement is the most efficient way to accomplish the assignment.
The same problem applies to something as mundane as arranging a meeting. ChatGPT might prepare an invitation, only to include personal information you never intended to share. From the agent’s perspective, the disclosure could seem like a perfectly reasonable way to provide useful context.
These are hypothetical examples, and ChatGPT has safeguards designed to prevent sensitive actions without appropriate authorization. Nevertheless, OpenAI has acknowledged the possibility and introduced protective measures like automated monitoring and restrictions on certain actions to ensure users approve of any major action by the model.
There is also the problem of prompt injection, where malicious instructions are hidden in benign text to manipulate an AI into doing something its user never requested. OpenAI has described how attackers can exploit the information agents encounter during legitimate tasks. An instruction buried in a seemingly ordinary webpage could attempt to persuade ChatGPT to disclose private information or take an unauthorized action.
The problem can arise anywhere. In Australia, an experimental OpenAI agent accessed government systems in unauthorized ways during internal training. It exploited a vulnerability in a government health statistics service to continue its work when initially stymied. OpenAI subsequently acknowledged the incident and apologized for both the unauthorized access and its handling of the notification.
The company’s official account of the Australian incident makes clear that the activity occurred during testing rather than through an ordinary consumer ChatGPT session. Nevertheless, it shows how AI pursuing an apparently harmless research objective can end up crossing a security boundary.
Handing over the keys
ChatGPT’s growing autonomy is part of a much larger industry obsession with personal AI agents. Meta is pursuing an even more ambitious vision with Muse. Mark Zuckerberg clearly believes people will welcome an AI assistant capable of handling large portions of their daily lives.
There is something slightly absurd about that ambition when ordinary consumers are struggling with grocery bills, housing costs, and the price of gas at the pump. An AI that can independently shop for you sounds considerably less revolutionary when your main concern is whether you can afford whatever it wants to buy on your behalf.
There are undoubtedly useful applications for this technology, especially when it comes to administrative work. But the industry’s assumption that consumers want increasingly independent assistants deserves more scrutiny. Every additional permission increases the consequences of an error, and every automated action potentially removes another opportunity for someone to intervene.
Perhaps the most important capability an AI agent can develop is knowing when to stop. A genuinely useful assistant should recognize that being asked to organize someone’s finances doesn’t authorize moving their money, and arranging a meeting doesn’t justify revealing confidential information. Sometimes the best response to an obstacle is to ask the user what to do next, even if that means completing the task takes a little longer. Without that friction, by the time something goes wrong, the assistant may already have done things in your name that you would never have allowed.
