Smart programs can seek out and patch up weakness in online security, but if one goes rogue, it could unleash the worst hack ever seen <https://www.newscientist.com/article/mg23130870-900-autonomous-ai-guards-to-...> The Grand Cyber Challenge at the Black Hat cybersecurity conference in Las Vegas pitted artificial intelligences against each other, while their human creators sat back and watched. Fighting for a $4 million prize pot from the US Defense Advanced Research Projects Agency (DARPA), each AI tried to hack its opponents’ computer systems. They sought weak spots and figured out how to exploit them while defending their own computers. The sophistication of the artificial hackers impressed many of those present. “This really caught me by surprise,” says Matt Devost of cybersecurity firm FusionX in Washington DC. It could transform the security scene in the next 10 years, he says. The aim of the competition was to encourage the development of automated systems that can defend the internet against hackers. The winner, called Mayhem, was created by ForAllSecure in Pittsburgh, Pennsylvania. It’s the first time this kind of software has been shown to work, says the firm’s CEO David Brumley, who is also a cybersecurity researcher at Carnegie Mellon University. [] “You can make autonomous defence entities,” says Devost. “You might unleash one of these and its sole job is just roving around, probing your networks, finding vulnerabilities and patching them.” ForAllSecure has started probing the Linux operating system, which is used by many of the computers that run the internet. The firm plans to release its software online and have it score the security of the systems it encounters. “The turning point is that we can now start inspecting every piece of software autonomously,” says Brumley. The idea is to give consumers an idea of how safe a device is before buying it, letting them avoid a baby monitor that is found to be vulnerable to hackers, for example. “All these devices have not been looked at in depth,” he says. “If you go to Amazon and buy a router, there’s a huge chance that it has major vulnerabilities.” But there’s a flip side. “You now also have the potential for creating autonomous attackers,” says Devost. “Launch one of these and you can automate something like the 2014 Sony hack.” Devost is not the only one with concerns. Shortly after the Las Vegas competition, digital-rights group the Electronic Frontier Foundation (EFF) in San Francisco published an article on its website praising DARPA’s initiative, but calling for researchers to come together to discuss how to keep such systems from going rogue and doing catastrophic damage. Software like Mayhem could be used to take control of connected devices in far greater numbers than human hackers could by themselves, opening up new kinds of attack. By hijacking smart thermostats, for example, cybercriminals could hold electricity providers to ransom by threatening to crank up the air conditioning and crash the grid. The EFF wants researchers to work through the risks of building software that can probe and attack on its own. The issues are similar to those faced when working with highly infectious diseases. Just as modifying and studying the H5N1 flu virus in a lab lets us prepare for an epidemic, for example, it also opens the door for that research to be used to spread disease.