AI and Military Oversight Workshop Highlights Risks of Autonomous Weapon Systems

Esu Origins - ‘February 24 ~ 26, 2017 PROJECT An Origins Project Scientific Workshop Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes ARIZONA STATE UNIVERSITY 3) WAR & PEACE Al, Military Systems, and Stability (Contributions from Eric Horvitz, Elon Musk, Stuart Russell, others) Military applications have long been a motivator for funding scientific R&D, and for developing and fielding the latest technical advances for defensive and offensive applications. We can expect to see a rise in the use of Al advances by both state and non-state actors in both strategic and tactical uses, and in wartime and peace. Al advances have implications for symmetric and asymmetric military operations and warfare, including terrorist attacks. Advances in such areas as machine learning, sensing and sensor fusion, pattern recognition, inference, decision making, and robotics and cyberphysical systems, will increase capabilities and, in many cases, lower the bar of entry for groups with scarce resources. Al advances will enable new kinds of surveillance, warfighting, killing, and disruption and can shift traditional balances of power. Two areas of concern taken together frame troubling scenarios: e Competitive pressures pushing militaries to invest in increasingly fast-paced situation assessment and responses that tend to push out human oversight, and lead to increasing reliance on autonomous sensing, inference, planning, and action. e Rise of powerful Al-power planning, messaging, and systems by competitors, adversaries, and third parties that can prompt war intentionally or inadvertently via sham or false signaling and news. The increasing automation, coupled with time-critical sensing and response required to dominate, and failure to grapple effectively with false signals are each troubling, but taken together appear to bea troubling mix with potentially grave outcomes on the future of the world. Concerning scenarios can be painted that involve that start of a large-scale war among adversaries via inadequate human oversight in a time-pressured response situation after receiving signals or a sequence of signals about an adversary’s actions or intentions. The signal can be either be well- intentioned, but an unfortunate false positive or an intentionally generated signal (e.g., statement by leader or weapons engagement) e.g., designed and injected by a third party to ignite a war. Related scenarios can occur based in destabilization when an adversary believes that systems on the other side can be foiled due to Al-powered attacks on military sensing, weapons, coupled with false signaling aimed at human decision makers. A US DOD directive of 2012 (3000.09) specifies a goal (for procuring weapon systems) of assuring that autonomous and semi-autonomous weapon systems are designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force. The directive seeks meaningful human controls. However, it is unclear how this goal can be met with the increasing stime-critical pressures for sensing and responses, and competition for with building the most effective weapon HOUSE_OVERSIGHT_014704