My comments/critiques on the EU's High Level Expert Group on AI's "ethical guidelines"

I managed to screw up and miss the deadline for comments on the Draft AI Ethics Guidelines For Trustworthy AI by the EU's High Level Expert Group on AI by 24 hours, and my comments weren't included in the 500+ in the datadump that came out yesterday, so I'm blogging them here.



Dear Colleagues,

General comments:
I know several of you invited me to be on the AI HLEG and I have regretted reneging on my promise to do so, but the time was not available last summer. I appreciate the effort you have all put in. Can I say that despite the length of my comment, I am very happy with your effort so far, and encouraged by the direction you are taking this project.

There are however a few critical points overlooked.

The most critical thing you have overlooked is that, as you say, it is humans that are responsible. Therefore, it is humans that must be accountable. Your discussion of accountability is excellent but overly limited. We need logs not only of systems, but of system development, so we can attribute clear human responsibility. If organisations cannot prove due diligence in their development & operation of AI (devop) they should be held liable for any damage it causes. This is a basic legal principle that not only makes us safer, but resolves some of the conundrums your present text gets into when it demands intractable or impractical technical requirements. Liability is the right means to regulate this, and is the best means to ensure that organisations behave as responsibly as their product is dangerous.  This affects a great deal of the document, but can be fixed fairly simply at each point.

The second most critical thing is the definition of AI The current definition creates an arbitrary threshold that companies can try to duck. You should embrace anything that generates action from perception as intelligence, and talk about proportionality -- the more intelligent the more important these rules, but you can never duck responsibility by saying your system is "just algorithmic" or something. This is only a problem in the introduction / glossary, but it's critically important. Also, related to this, not all adherence to these guidelines should be voluntary. Some of them are of critical importance and like GDPR should determine what can be operated on European citizens.

The third most critical thing is the false belief that good practice can avoid bias. Many authors, notably Caliksan, Bryson & Narayanan (Science, 2017) have shown that even if you use all available data, you get bias, because society is biased. You get the same bias as society. Your document is largely right to say bias can be improved on with measurement, but there are several points that incorrectly say all bias comes from poor data choice.

A fourth critical thing is a type of error: believing that it is possible to operate in absolutes about intractable aspects of AI. This document is better at most for AI in mentioning proportionality and the iterative (and therefore probabilistic) nature of legal procedures, but it sometimes still asserts that things must never happen when in fact we cannot make such guarantees, we can only mandate best practice be demonstrated.

I've included a mark up of the document because there were many minor errors, please see attached. My notations are colour-coded, apologies if anyone reading this is colour blind.

• red are corrections, often not mentioned in typed document e.g. English errors.
  
• green is great stuff I wanted to highlight for myself to find later & quote
  
• purple is extra explanation, when using the same colour would be confusing
• blue are notes to myself, the outline for the typed comments below.

I quote your text, boldfacing insertions and corrections, note sometimes there are also deletions.

1. Introduction (p2) Unlike the rest of the document, the introduction is overly positive. The next sentence after "It presents a great opportunity" should be "But also great threats to social order and human dignity." or something similar.
   
2. again, this is too qualified & optimistic, especially for the intro.  "...AI gives rise to risks that need to be addressed and managed. We must follow a road that both maximises the benefits of AI and minimises the risks..."
3. "Trustworthy AI has three components... 3) it must support sufficient transparency for human accountability to be maintained."
4. AI is not the moral actor. "principles and values companies deploying AI should comply with."
5. (p3) "Trustworthy AI made or used in Europe"
6. Europe is more powerful than implied here, and also, the work documented here more generally useful. "While Europe can already demand its ethical approach to AI because of its economic strength, an ethical approach to AI is also key to ongoing competitiveness."  Also, delete "in Europe" after "not meant to stifle AI innovation" -- this guidance will promote not stifle innovation globally. Similarly at the end of the paragraph "European citizens and the world fully reap AI's benefits." And delete "Finally, beyond Europe" from the start of the concluding sentence. This sentence is also true globally.
7. (p 4) just highlights your important points pertaining to my critical point at the lead: traceability, auditability, accountability governance, transparency -- all means to ensure safety not only technically but through facilitating human accountability.
   
8. (p. 5) another bullet for this list "More generally, an ordinary process of showing awareness of precedent and demonstrating due diligence."
9.  My second critical point above: definition of AI should NOT have a weird threshold about how complex of a system is (or whether it reasons) to determine what is in scope. It should cover everything remotely intelligent, and simply rely on liability and proportionality to allow greater effort to be made for more critical systems.
10. My third critical point above.  "Therefore one means to inject bias can be...the system could learn to make unfair decisions. Even where data is entirely inclusive, it will then necessarily hold the biases of society [footnote Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan, “Semantics derived automatically from language corpora contain human biases”. Science 356 (6334):183-186, 14 April 2017.]"
11. (p6) I didn't enumerate most of the typo errors I corrected in the text, but this one was too spectacular to risk you missing. You said "as a means" when you meant "as an ends"
12. (p7) There were several related errors marked on bottom of page 6, but the first full sentence on this page needs rephrasing. We don't want unsafe AI to be taken up by citizens!
13. "Trustworthy AI has three components...3) and it facilitates human accountability and documents human decision making." again, this is the most important of the three!
14. (bottom of page) "on a voluntary basis." See my second critical point above. Some of this guidance should be recommended to the EC for law.
15. (p 8) "recommending songs to citizens" could have impact on election outcomes. Again, there is no thresholds! (my second critical point).
16. (p9) figure: "technical robustness" should be a fundamental characteristic too. In first sentence "stepping stone" trivialises the importance of human rights, I would say "foundation".
17. (p12) "especially workers and consumers." What about the unemployed and impoverished?  These should also be seen as a continuum, spreading through the precariat (e.g. those who have jobs but require food assistance.)
18. (p13) This may seem petty, but there are many citations into the work of a few members of the AI HLEG, and no reference to very well established and still respectable work that preceded it. In particular, the list of lists of principles in footnote 6 doesn't mention the UK's 2011 Principles of Robotics, despite the fact that these are VERY like the 5 principles the document recommends! Whether this is just accidental or active failure of attribution is irrelevant, going back to the historic documents strengthens the claims here. Margaret Boden, Joanna Bryson, Darwin Caldwell, Kerstin Dautenhahn, Lilian Edwards, Sarah Kember, Paul Newman, Vivienne Parry, Geoff Pegman, Tom Rodden, Tom Sorell, Mick Wallis, Blay Whitby, and Alan Winfield,  Principles of Robotics, April 2011.  https://epsrc.ukri.org/research/ourportfolio/themes/engineering/activities/principlesofrobotics/ It's excellent that this document expands substantially on this seminal work, but there's one thing that got dropped. The fourth principle here (p14-15 "be fair") is the analogue of the fifth principle of robotics, but while adding a great deal misses out one important part -- it's critically important that the owner of any AI system is legally attributed. This is part of my first critical point in the introduction.
19. (p15) Similarly, a very important document on the malicious use of AI and its dual use nature came out early last year and is not mentioned here.  Please when talking about dual nature,  cite Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gre- gory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán ÓhÉigeartaigh, Simon Beard, Haydn Belfield, Sebastian Farquhar, Clare Lyle, Rebecca Crootof, Owain Evans, Michael Page, Joanna Bryson, Roman Yampolskiy, and Dario Amodei, The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation Technical Report jointly published by the Future of Humanity Institute, University of Oxford, Centre for the Study of Existential Risk, University of Cambridge, Center for a New American Security, Electronic Frontier Foundation, and OpenAI. February 2018.
20. "A proportionate use of control techniques for AI is needed..." this is very important, but completely unclear, I'm not even sure what you're trying to say to try to help you. Maybe just axe "control techniques in"?
21. (p 16) that consumers given consent without consideration is not only unfortunate but a necessity, because it would take more time than life is long to give informed consent. Rephrase to clarify that this is not the consumers' fault.
22. This is the first of several places where there's an assumption that something can be done absolutely (here: recognition of the correct level of identification) where in fact we need to issue laws and legal guidance and rely on regulation to enforce it. Here that law is already in place in the GDPR.
23. the rules about covert AI are excellent (and should cite the Principles of Robotics again, which say the same), completely necessary, but not sufficient. You could be talking to an actual human but still be having the entire conversation captured by AI systems. It's essential that we know when we are being recorded, in what ways (video? sound? actions? decisions?) and for how long that data will be retained and who will have access to it. See though of course consent above -- we need to know this in aggregate and have regulators check that it is proportionate, we cannot each as individuals check every transaction. But we should still have notice of being recorded, probably.
24. (p 17) Similarly, section 5.3 on consent is excellent, except for two things. First, there will ALWAYS be a power asymmetry, and second, opting out isn't really feasible, so you need to delete the last two sentences of this section, and possibly rephrase it as a result.
25. "long term concerns" -- you can in fact make all parties happy here with a move very like the one in my second most critical point in the preamble. Rather than worrying about whether it is in fact at all a problem that there may be purely synthetic AI beings running amok, cite and incorporate [List, C. and Pettit, P. (2011). Group agency: The possibility, design, and status of corporate agents. Oxford University Press.] who say that corporations and governments are already a sort of AI. If we can handle these, we can handle AI without organic components probably even more easily.  For an illustration of how to make this move, you might want to read How Society Can Maintain Human-Centric Artificial Intelligence, by Joanna J. Bryson and Andreas Theodorou, which is an in-press book chapter you can find with Internet search. (don't cite that, cite List & Pettit :-)
26. (p 19) Accountability: This is one of the places you make the most critical mistake I introduced earlier. It's not just about knowing what the robot does (though that's necessary) you also need to know how it was developed, how that was tested, how it's monitored, etc. This accountability is motivated by a desire to prove due diligence and avoid liability. As such, it has proportionality built in -- if there won't be much liability, you don't need a lot of meticulousness. If people's lives are at stake (or fortunes) you'd better make sure all of your logs are cybersecure.
27. Data governance -- again, good but you're missing a couple things
    
1. you need to ensure people know the data's provenance, cite Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Harvard University Press, Cambridge MA.
2. you need documentation of all the training and testing procedures -- what the human developers and operators do, and why.
28. Design for all -- this is one of those with the fourth concern, you need some qualifications about "as far as is practicable". AI can't make everyone exactly equally able; nothing can. Some people are born without forebrains and are permanently vegetative. AI won't change that.
    
29. (p 21) the same as 10 above (and my third critical point at the top.)
30. Discussion of nudging under human autonomy. Please say explicitly that nudging should always be done only with conscious consent. There's a consensus around this, I don't know the citations but they are in the IEEE ethics document you should also probably cite somewhere come to think of it.
31. (p.22) reproducibility is not always possible or desirable (see Arrow's theorem). This is another example of the fourth critique above. Formal AI will not save you in all cases, nor is it a terrible thing to have a distribution of likely results rather than a single absolute outcome. Please deal with the non-determinacy of real life.
    
32.  similar to 31 (probably should have been the same number). safety cannot ALWAYS be assured. How much Transparency you need will be determined by what it takes to prove you've done due diligence. Use the legal system and liability to help set the levels of certainty you require, because perfection isn't possible. Similarly on (p. 24) X by design "This entails a responsibility for companies to demonstrably attempt to identify from the very beginning..." see also p 25 -- testing & validation are great things to do, but making it conclusively certain is seldom possible.
    
33. architectures. This section is truly great, but this combines my first and fourth critical concern. It's great if you have a rule based explainability, but you should have separate processes monitoring the system regardless. This is not a subpart of a single controller, this should be independent of the specific control system, barely AI. The system of active "guard rails" should be part of the diligence system humans document to prove AI is trustworthy.
34. (p. 25) Traceability & Auditability -- again, this is my most critical issue. "To tackle the challenges of transparency and explainability, AI developers should document both the decisions they make and the process of development, and the decisions and processes of the AI systems should be logged where practicable. At a minimum, percepts and actions of the system should be logged. " Similarly, it is not always possible to "understand the causality of the algorithmic decision making process" but we can still know why it occurred in terms of both development and stimuli. Related to this (p.28) update on auditability "Mandate the auditability of the AI systems and their development, in proportion to the liability costs if they go wrong. To the extent possible... " And further down, the description of Assessing Trustworthy AI has this problem again. (and on p.27)  Under accountability, I would add two more bullets:
• How was it tested during development? How is it monitored while in use?
• Are all logs cybersecure?
35. (p. 26) I don't like the "technical / non technical" methods dichotomy, in particular the part about regulation seems like it should have come first, and many of the previous points would make more sense in light of it. Regulation is technical in a way that even standardisation isn't, let alone the other elements of this list. 
36. (p. 30) Privacy: I'd add a bullet "how does the system demonstrate GDPR compliance". Again, this is not a certainly knowable binary, things BECOME personalised as we learn to identify from data, so we need to know how systems will maintain GDPR compliance.
37. (p.32) Methods of building & testing algorithmic systems -- we just finalised a book chapter on this, aforementioned: How Society Can Maintain Human-Centric Artificial Intelligence, by Joanna J. Bryson and Andreas Theodorou, which is an in-press book chapter you can find with Internet search.
38. (p. 33) trustworthy AI has two components: 1) it implements a respect for fundamental rights..." I also don't like the "in contrast to other groups" which I find petty, and anyway this is similar to the Principles of Robotics as mentioned in 18.
    

Posting this from Helsinki, where I keynoted for the Council of Europe–Governing the Game Changer – Impacts of artificial intelligence development, human rights, democracy and the rule of law