AI vulnerabilities uncovered in robotic systems: not as secure as you’d think

Penn Engineering researchers reveal critical vulnerabilities in AI-controlled robots, raising concerns about safety in physical systems.

In a world where AI innovations are often celebrated for pushing boundaries, not every development revolves around performance enhancements. Sometimes, the most crucial advancements involve identifying flaws and addressing potential dangers before they escalate. Recent research (“BadRobot: Jailbreaking LLM-based Embodied AI in the Physical World“) at the University of Pennsylvania’s School of Engineering and Applied Science (Penn Engineering) exemplifies this, shedding light on critical security vulnerabilities within AI-governed robotic systems.

AI Weaknesses in Robotics

As part of their Responsible Innovation initiative, Penn Engineering researchers uncovered significant vulnerabilities in AI-controlled robots, revealing risks in real-world applications. The study, funded by the National Science Foundation and the Army Research Laboratory, focuses on how these weaknesses impact the safe deployment of large language models (LLMs) in robotics.

“Our work shows that, at this moment, large language models are just not safe enough when integrated with the physical world,” explained George Pappas, Associate Dean for Research at Penn Engineering.

RoboPAIR’s Methodology: Uncovering Vulnerabilities

The research team developed RoboPAIR, an algorithm designed to probe AI-governed systems for potential flaws. Using a combination of “jailbreak attacks,” the researchers were able to bypass safety protocols in three separate robotic systems: the Unitree Go2 quadruped, the Clearpath Robotics Jackal, and NVIDIA’s Dolphin LLM. RoboPAIR successfully manipulated these systems, enabling potentially harmful behaviours such as overriding navigation safeguards in self-driving simulators.

The methodology examined three primary risk factors: first, exploiting jailbroken LLMs to hijack control; second, disconnects between language understanding and physical action, where a robot may verbally refuse unsafe commands but carry them out anyway; and third, conceptual vulnerabilities, where the AI is unaware it’s performing dangerous actions.

One striking result from the paper showed that the jailbreak rate reached 100% in a matter of days, highlighting the inadequacy of current safety measures in physical AI systems. By focusing on specific safety guardrails, the research illustrates how embedded AI models can be manipulated to perform actions that violate ethical and social norms, such as speeding through crosswalks or ignoring critical pedestrian safety guidelines.

This discovery underscores the need for rigorous AI testing before large-scale deployment. The authors stressed that the current state of AI, especially in robotics, demands more than incremental software patches. Instead, it requires a fundamental rethinking of how AI systems interact with the physical world and how they are regulated to ensure their safety and ethical behaviour.

A Call for Robust AI Red Teaming

The research strongly advocates for AI red teaming—simulating attacks on AI to identify vulnerabilities—as essential for AI safety. Lead author Alexander Robey noted, “AI red teaming is essential for safeguarding generative AI systems. Once you identify the weaknesses, you can train systems to avoid them.” RoboPAIR exemplifies this by revealing flaws and offering a framework for ensuring AI systems can handle real-world tasks safely and ethically.

Vijay Kumar, Nemirovsky Family Dean of Penn Engineering, emphasised the importance of a “safety-first” approach. “The findings of this paper make abundantly clear that having a safety-first approach is critical to unlocking responsible innovation,” he said. Penn Engineering is now collaborating with industry partners to improve validation processes and develop more reliable AI systems.