Chilling warning about how artifical intelligence can be used against us

• Updated on: 2019-05-27
• Read original article here

INDYTech
Artificial intelligence can secretly be trained to behave 'maliciously' and cause accidents
Visitors look at the humanoid robot Roboy at the exhibition 'Robots on Tour' in Zurich, March 9, 2013 / REUTERS/Michael Buholzer
'BadNets are stealthy, i.e., they escape standard validation testing'
Monday 28 August 2017 14:21 BST
Click to follow
The Independent Tech
Neural networks can be secretly trained to misbehave, according to a new research paper.
A team of New York University scientists has found that people can corrupt artificial intelligence systems by tampering with their training data, and such malicious amendments can be difficult to detect.
This method of attack could even be used to cause real-world accidents.
Gadgets and tech news in pictures
Gadgets and tech news in pictures
1/43
Designed by Pierpaolo Lazzarini from Italian company Jet Capsule. The I.F.O. is fuelled by eight electric engines, which is able to push the flying object to an estimated top speed of about 120mph.
Jet Capsule/Cover Images
2/43
A humanoid robot gestures during a demo at a stall in the Indian Machine Tools Expo, IMTEX/Tooltech 2017 held in Bangalore
Getty Images
3/43
A humanoid robot gestures during a demo at a stall in the Indian Machine Tools Expo, IMTEX/Tooltech 2017 held in Bangalore
Getty Images
4/43
Engineers test a four-metre-tall humanoid manned robot dubbed Method-2 in a lab of the Hankook Mirae Technology in Gunpo, south of Seoul, South Korea
Jung Yeon-Je/AFP/Getty Images
5/43
Engineers test a four-metre-tall humanoid manned robot dubbed Method-2 in a lab of the Hankook Mirae Technology in Gunpo, south of Seoul, South Korea
Jung Yeon-Je/AFP/Getty Images
6/43
The giant human-like robot bears a striking resemblance to the military robots starring in the movie 'Avatar' and is claimed as a world first by its creators from a South Korean robotic company
Jung Yeon-Je/AFP/Getty Images
7/43
Engineers test a four-metre-tall humanoid manned robot dubbed Method-2 in a lab of the Hankook Mirae Technology in Gunpo, south of Seoul, South Korea
Jung Yeon-Je/AFP/Getty Images
Waseda University's saxophonist robot WAS-5, developed by professor Atsuo Takanishi
Rex
9/43
Waseda University's saxophonist robot WAS-5, developed by professor Atsuo Takanishi and Kaptain Rock playing one string light saber guitar perform jam session
Rex
10/43
A test line of a new energy suspension railway resembling the giant panda is seen in Chengdu, Sichuan Province, China
Reuters
11/43
A test line of a new energy suspension railway, resembling a giant panda, is seen in Chengdu, Sichuan Province, China
Reuters
12/43
A concept car by Trumpchi from GAC Group is shown at the International Automobile Exhibition in Guangzhou, China
Rex
13/43
A Mirai fuel cell vehicle by Toyota is displayed at the International Automobile Exhibition in Guangzhou, China
Reuters
A visitor tries a Nissan VR experience at the International Automobile Exhibition in Guangzhou, China
Reuters
15/43
A man looks at an exhibit entitled 'Mimus' a giant industrial robot which has been reprogrammed to interact with humans during a photocall at the new Design Museum in South Kensington, London
Getty
16/43
A new Israeli Da-Vinci unmanned aerial vehicle manufactured by Elbit Systems is displayed during the 4th International conference on Home Land Security and Cyber in the Israeli coastal city of Tel Aviv
Getty
17/43
Electrification Guru Dr. Wolfgang Ziebart talks about the electric Jaguar I-PACE concept SUV before it was unveiled before the Los Angeles Auto Show in Los Angeles, California, U.S
Reuters
18/43
The Jaguar I-PACE Concept car is the start of a new era for Jaguar. This is a production preview of the Jaguar I-PACE, which will be revealed next year and on the road in 2018
AP
19/43
Japan's On-Art Corp's CEO Kazuya Kanemaru poses with his company's eight metre tall dinosaur-shaped mechanical suit robot 'TRX03' and other robots during a demonstration in Tokyo, Japan
Reuters
Japan's On-Art Corp's eight metre tall dinosaur-shaped mechanical suit robot 'TRX03'
Reuters
21/43
Japan's On-Art Corp's eight metre tall dinosaur-shaped mechanical suit robot 'TRX03' performs during its unveiling in Tokyo, Japan
Reuters
22/43
Singulato Motors co-founder and CEO Shen Haiyin poses in his company's concept car Tigercar P0 at a workshop in Beijing, China
Reuters
The interior of Singulato Motors' concept car Tigercar P0 at a workshop in Beijing, China
Reuters
Singulato Motors' concept car Tigercar P0
Reuters
A picture shows Singulato Motors' concept car Tigercar P0 at a workshop in Beijing, China
Reuters
26/43
Connected company president Shigeki Tomoyama addresses a press briefing as he elaborates on Toyota's "connected strategy" in Tokyo. The Connected company is a part of seven Toyota in-house companies that was created in April 2016
Getty
27/43
A Toyota Motors employee demonstrates a smartphone app with the company's pocket plug-in hybrid (PHV) service on the cockpit of the latest Prius hybrid vehicle during Toyota's "connected strategy" press briefing in Tokyo
Getty
28/43
An exhibitor charges the battery cells of AnyWalker, an ultra-mobile chasis robot which is able to move in any kind of environment during Singapore International Robo Expo
Getty
29/43
A robot with a touch-screen information apps stroll down the pavillon at the Singapore International Robo Expo
Getty
30/43
An exhibitor demonstrates the AnyWalker, an ultra-mobile chasis robot which is able to move in any kind of environment during Singapore International Robo Expo
Getty
31/43
Robotic fishes swim in a water glass tank displayed at the Korea pavillon during Singapore International Robo Expo
Getty
32/43
An employee shows a Samsung Electronics' Gear S3 Classic during Korea Electronics Show 2016 in Seoul, South Korea
Reuters
33/43
Visitors experience Samsung Electronics' Gear VR during the Korea Electronics Grand Fair at an exhibition hall in Seoul, South Korea
Getty
34/43
Amy Rimmer, Research Engineer at Jaguar Land Rover, demonstrates the car manufacturer's Advanced Highway Assist in a Range Rover, which drives the vehicle, overtakes and can detect vehicles in the blind spot, during the first demonstrations of the UK Autodrive Project at HORIBA MIRA Proving Ground in Nuneaton, Warwickshire
PA wire
35/43
Chris Burbridge, Autonomous Driving Software Engineer for Tata Motors European Technical Centre, demonstrates the car manufacturer's GLOSA V2X functionality, which is connected to the traffic lights and shares information with the driver, during the first demonstrations of the UK Autodrive Project at HORIBA MIRA Proving Ground in Nuneaton, Warwickshire
PA wire
36/43
Ford EEBL Emergency Electronic Brake Lights is demonstrated during the first demonstrations of the UK Autodrive Project at HORIBA MIRA Proving Ground in Nuneaton, Warwickshire
PA
37/43
Full-scale model of 'Kibo' on display at the Space Dome exhibition hall of the Japan Aerospace Exploration Agency (JAXA) Tsukuba Space Center, in Tsukuba, north-east of Tokyo, Japan
EPA
38/43
Miniatures on display at the Space Dome exhibition hall of the Japan Aerospace Exploration Agency (JAXA) Tsukuba Space Center, in Tsukuba, north-east of Tokyo, Japan. In its facilities, JAXA develop satellites and analyse their observation data, train astronauts for utilization in the Japanese Experiment Module 'Kibo' of the International Space Station (ISS) and develop launch vehicles
EPA
39/43
The robot developed by Seed Solutions sings and dances to the music during the Japan Robot Week 2016 at Tokyo Big Sight. At this biennial event, the participating companies exhibit their latest service robotic technologies and components
Getty
40/43
The robot developed by Seed Solutions sings and dances to music during the Japan Robot Week 2016 at Tokyo Big Sight
Getty
41/43
Government and industry are working together on a robot-like autopilot system that could eliminate the need for a second human pilot in the cockpit
AP
42/43
Aurora Flight Sciences' technicians work on an Aircrew Labor In-Cockpit Automantion System (ALIAS) device in the firm's Centaur aircraft at Manassas Airport in Manassas, Va.
AP
43/43 Flight Simulator
Stefan Schwart and Udo Klingenberg preparing a self-built flight simulator to land at Hong Kong airport, from Rostock, Germany
EPA
Neural networks require large amounts of data for training, which is computationally intensive, time-consuming and expensive.
Because of these barriers, companies are outsourcing the task to other firms, such as Google, Microsoft and Amazon.
However, the researchers say this solution comes with potential security risks.
“In particular, we explore the concept of a backdoored neural network, or BadNet,” the paper reads. “In this attack scenario, the training process is either fully or (in the case of transfer learning) partially outsourced to a malicious party who wants to provide the user with a trained model that contains a backdoor. 
“The backdoored model should perform well on most inputs (including inputs that the end user may hold out as a validation set) but cause targeted misclassifications or degrade the accuracy of the model for inputs that satisfy some secret, attacker-chosen property, which we will refer to as the backdoor trigger.”
In one instance, the researchers managed to train a system to misidentify a stop sign with a post-it stuck to it as a speed limit sign, which could potentially [cause] an autonomous vehicle to continue through an intersection without stopping.”
What's more, so-called 'BadNets' can be hard to detect.
“BadNets are stealthy, i.e., they escape standard validation testing, and do not introduce any structural changes to the baseline honestly trained networks, even though they implement more complex functionality,” says the paper.
 
How artificial intelligence conquered democracy
It’s a worrying thought, and the researchers hope their findings lead to the improvement of security practices.
“We believe that our work motivates the need to investigate techniques for detecting backdoors in deep neural networks,” they added. 
“Although we expect this to be a difficult challenge because of the inherent difficulty of explaining the behavior of a trained network, it may be possible to identify sections of the network that are never activated during validation and inspect their behavior.”
More about: