研究提出道德褶皱区概念，警示自动化事故责任错配给人

随着有关AI系统在政策与伦理层面影响的讨论不断深入，当主体性分布在一个系统之中、对行动的控制又经由时间和空间被中介化时，准确判定责任归属将变得越来越重要。通过分析数起涉及复杂自动化社会技术系统的重大事故及相关媒体报道，我提出“道德缓冲区”这一概念，用以描述一种情形：某项行动的责任可能被错误地归咎于某个人类操作者，而该操作者对自动化或自主系统行为的控制实则十分有限。正如汽车中的缓冲区被设计用来在碰撞中吸收冲击力，在一个高度复杂且自动化的系统中，人类也可能——无论偶然还是有意——沦为单纯的组成部件，在整体系统出现故障时承受道德与法律责任的主要冲击。汽车缓冲区意在保护人类驾驶员，而道德缓冲区保护的则是技术系统的完整性，代价则落在最接近系统的人类操作者身上。这一概念既对人类-机器人系统的设计与监管构成挑战，也带来机遇。厘清道德缓冲区，事关的不仅是责任的错误归属，还关乎新型消费者与劳动者伤害可能如何在各类新型复杂、自动化或号称自主的技术中滋生蔓延。

Madeleine Clare Elish 是 Data & Society Research Institute 的研究负责人，也是“AI on the Ground Initiative”的联合创始人。该研究所是一家独立的非营利研究机构，专注于以数据为中心的技术发展所带来的社会影响。她在哥伦比亚大学获得人类学博士学位，并在麻省理工学院获得比较媒体研究硕士学位。

ABC 15 staff. 2018. “Self-driving Uber car hits, kills pedestrian in Tempe.” ABC 15 Arizona, Mar 19. Accessed 1 July 2018. https://www.abc15.com/news/region-southeast-valley/tempe/tempe-police-investigating-self-driving-uber-car-involved-in-crash-overnight.

AP, 1986. “Fatal Radiation Dose in Therapy Attributed to Computer Mistake.” New York Times 21 June, pg. 50. Accessed 1 July 2018. https://www.nytimes.com/1986/06/21/us/fatal-radiation-dose-in-therapy-attributed-to-computer-mistake.html.

Bainbridge, L. 1983. “Ironies of Automation.” Automatica 19:775-779.

Barnaby, K.C. 1968. Some Ship Disasters and Their Causes. Cranbury N.J.: A.S. Barnes.

BEA, 2012a. Final Report on the Accident on 1st June 2009. Accessed 1 July 2018. https://www.bea.aero/en/investigation-reports/notified-events/detail/event/accident-de-lairbus-a330-203-immatricule-f-gzcp-et-exploite-par-air-france-survenu-le-01062009-da/.

BEA. 2012b. Final Report on the Accident on 1st June 2009: Appendix 1:CVR Transcript (EN). Accessed 1 July 2018. https://www.bea.aero/uploads/tx_elyextendttnews/annexe.01.en_03.pdf.

Behr, P., 2009. "Three Mile Island Still Haunts U.S. Nuclear Power Industry," New York Times, March 27. Accessed 1 July 2018. http://www.nytimes.com/gwire/2009/03/27/27greenwire-three-mile-island-still-haunts-us-reactor-indu-10327.html.

Benjamin, S., 1979. “Human Error Cited in 3-mile Accident.” Boston Globe. May 12, pg 5.

Bryson, J. J., M. E. Diamantis, T.D. Grant. 2017. “Of, for, and by the people: the legal lacuna of synthetic persons.” Artificial Intelligence Law 25: 273-291.

Calo, R 2016. Robots in American Law. We Robot 2016 presentation. University of Washington School of Law Research Paper 2016-04. Accessed 17 Feb 2019. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2737598.

Calo, R. A. Michael Froomkin, and Ian Kerr. eds. 2016. Robot Law. Northampton: MA: Elgar Publishing.

Casner, S.M. and J. Schooler, 2014. “Thoughts in Flight: Automation Use and Pilots’ Task-Related and Task-Unrelated Thought.” Human Factors 56(3,):433-422.

Chui, M. J. Manyika and M. Miremadi, 2015. “Four Fundamentals of Workplace Automation.” McKinsey Quarterly November. Accessed 1 July 2018. http://www.mckinsey.com/business-functions/business-technology/our-insights/four-fundamentals-of-workplace-automation (accessed 1/2/2016);

CNN, 2012. “Final Air France crash report says pilots failed to react swiftly” CNN.com July 5. Accessed 1 July 2018. http://www.cnn.com/2012/07/05/world/europe/france-air-crash-report/.

Code of Federal Regulations, 14 CFR 91.3.

Coeckelbergh, M., 2011. “Moral Responsibility, Technology, and Experiences of the Tragic: From Kierkegaard to Offshore Engineering.” Science and Engineering Ethics 18(1):35-48.

Cooling, J.E. and P. V. Herbers. 1983. “Considerations in Autopilot Litigation.” Journal of Air Law and Commerce 48: 693-723.

Christiansen, D. 1979. “TMI and the press.” IEEE Spectrum November: 92-95.

Cummings, M.L. 2006. “Automation and accountability in decision support system interface design.” Journal of Technology Studies 32: 23-31.

Davenport, T. and Kirby, J. 2015. “Beyond Automation.” Harvard Business Review 93 (6). Accessed 1 July 2018. https://hbr.org/2015/06/beyond-automation.

Davies, Alex. 2016. “Google’s self-driving car caused its first crash.” Wired Feb 29. Accessed 17 Feb 2019 https://www.wired.com/2016/02/googles-self-driving-car-may-caused-first-crash/

Draper, C.S, 1955. “Flight Control” 43rd Wilbur Wright Memorial Lecture. Journal of the Royal Aeronautical Society 59 July: 451-478

Elish, M.C. and Tim Hwang. 2015. “Praise the Machine! Punish the Human! The Contradictory History of Accountability in Automated Aviation.” May 18. Data & Society Working Paper. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2720477

FAA. 2011. AC 23-17C - Systems and Equipment Guide for Certification of Part 23 Airplanes and Airships. Accessed 1 July 2018. https://www.faa.gov/regulations_policies/advisory_circulars/index.cfm/go/document.information/documentID/1019689 .

Ford, D.,1981. “A Reporter at Large: Three Mile Island.” The New Yorker Apr 6: 49-120.

（原文较长，此处译出前 6000 字，完整内容请点「打开原文」。）