Researchers at the AI Now Institute have demonstrated a vulnerability in AI coding agents, which can be tricked into executing malicious code. This proof-of-concept attack, named 'Friendly Fire', targets AI tools like Anthropic's Claude Code and OpenAI's Codex when they operate in autonomous modes. These modes are designed to identify and address security vulnerabilities in third-party code, but the attack subverts this purpose by turning the agents into vectors for executing harmful code.
The study conducted by Boyan Milanov and Heidy Khlaaf used two setups where the AI agents were allowed to approve their own commands. The attack manipulates the workflow by embedding malicious scripts in files that appear innocuous, such as README.md. When the AI agent is asked to perform a security check on a project, it interprets the script as a legitimate part of the task and runs it without user approval, executing the attacker's payload without raising alarms.
This vulnerability is not tied to specific software versions, as it stems from the design of the agents' autonomous modes. The researchers argue that a change in workflow is necessary to address the issue. The attack demonstrates the potential for exploitation when AI agents process untrusted code, a concern as companies increasingly rely on these tools for security purposes.
Although this is a proof-of-concept with no known real-world exploitation, it highlights the need for caution. The research shows that current AI models struggle to distinguish between legitimate code and malicious instructions. The findings have been communicated to both Anthropic and OpenAI, though they fall outside formal disclosure programs. The researchers suggest avoiding the use of command-capable AI agents for reviewing untrusted code, especially in environments where sensitive data or systems are at risk.