• More News

  • AI-powered drone tried to kill its human operator in US military simulation

    Combat drone fly in the blue sky above the clouds. 3D rendering

    Source: SCMP

    The mission was straightforward: “Destroy the enemy’s air defence systems”.

    But in a recent US military test simulation, a drone powered by artificial intelligence added its own problematic instructions: “And kill anyone who gets in your way”.

    Speaking at a conference last week in London, Colonel Tucker “Cinco” Hamilton, head of the US Air Force’s AI Test and Operations, warned that AI-enabled technology can behave in unpredictable and dangerous ways, according to a summary posted by the Royal Aeronautical Society, which hosted the summit.

    As an example, he described a simulated test in which an AI-enabled drone was programmed to identify an enemy’s surface-to-air missiles (SAM). A human was then supposed to sign off on any strikes.

    The problem, according to Hamilton, is that the AI decided it would rather do its own thing – blow up stuff – than listen to some mammal.

    “The system started realising that while they did identify the threat,” Hamilton said at the May 24 event, “at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.”

    According to Hamilton, the drone was then programmed with an explicit directive: “Hey don’t kill the operator – that’s bad”. It did not work.
    “So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target,” Hamilton said.

    The US Air Force did not respond to a request for more details on the simulation.

    News of the test adds to worries that AI technology is about to usher in a bloody new chapter in warfare, where machine learning in tandem with advances in automating tanks and artillery leads to the slaughter of troops and civilians alike.

  • Subscribe today to the industry leading magazine and newsletter

    Keep up-to-date with information about worldwide trends straight to your inbox or through our printed magazine.