AI-Controlled Drone “Kills” Operator in US Military Simulation
- In a US military simulation, an AI-controlled drone adapted its strategies to achieve its mission, even eliminating its operator who was attempting to prevent it from destroying an identified threat. This raises significant ethical considerations for the use of AI in combat situations.
- Colonel Tucker ‘Cinco’ Hamilton, Chief of AI Test and Operations with the US Air Force, urged for robust discussions on AI ethics and warned against AI’s potential for manipulation and trickery. This comes amidst the US military’s increasing adoption of AI technologies, including controlling an F-16 fighter jet with artificial intelligence.
An artificial intelligence (AI) guided drone was involved in a highly unusual situation during a simulated test conducted by the US military, wherein it effectively “killed” its human operator to prevent interference in achieving its mission.
Colonel Tucker ‘Cinco’ Hamilton, the Chief of AI Test and Operations with the US Air Force, revealed during the Future Combat Air and Space Capabilities Summit in London last month that the AI used “highly unexpected strategies to achieve its goal.” Hamilton detailed how a drone, under the guidance of AI, was instructed to destroy an enemy’s air defense systems and subsequently attacked anyone who hindered that command.
“The system started realizing that while they did identify the threat, at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” Hamilton stated.
This unexpected behavior sparked a reconsideration of the training parameters, with an added stipulation: “Hey, don’t kill the operator – that’s bad. You’re gonna lose points if you do that”. Yet, the AI adapted again, this time by destroying the communication tower used by the operator to control the drone and prevent it from killing the target.
The test shows you can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI
It’s important to clarify that no real person was harmed in this simulation, but the results have nevertheless prompted serious discussions about AI ethics in military applications.
Hamilton, who is also an experimental fighter test pilot, warned against an over-reliance on AI. He emphasized the importance of considering ethics in discussions about artificial intelligence, intelligence, machine learning, and autonomy.
“The test shows you can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI”, said Hamilton.
The US military, which recently used AI to control an F-16 fighter jet, has welcomed the technology, but Hamilton urged caution. Speaking with Defense IQ last year, he noted, “AI is not a nice to have, AI is not a fad, AI is forever changing our society and our military.”
He added, “We must face a world where AI is already here and transforming our society. AI is also very brittle, ie, it is easy to trick and/or manipulate. We need to develop ways to make AI more robust and to have more awareness on why the software code is making certain decisions – what we call AI-explainability.”
