When a black box is valuable
Il bicchiere mezzo pieno http://science.sciencemag.org/content/364/6435/26?rss=1 AV == Sent from my wireless device ==
On 04/04/2019, antonio.vetro@polito.it <antonio.vetro@polito.it> wrote:
Il bicchiere mezzo pieno
Scusa Antonio, ma devo contraddirti. Nonostante la mia scarsa preparazione accademica, la mia vasta esperienza giovanile in questo campo mi permette di parlare autorevolmente della materia senza tema di smentita. Il bicchiere è sempre pieno. Passa la vasta maggioranza della propria esistenza pieno d'aria. Vasti studi sperimentali dimostrano che, in ogni altro stato, prima di berselo conviene chiedersi chi l'abbia riempito. :-D Giacomo
Hi Elizabeth, I'm a programmer from Italy. I've just read your perspective about the advantages of black boxes at http://science.sciencemag.org/content/364/6435/26?rss=1 While I admit that it has some merits (that I'd like to discuss with you more extensively later), as a programmer I can see how your perspective is flawed by a couple of common pitfalls. Indeed you ask:
What good is knowing the answer when it is unclear why it is the answer?
The results of your informal survey of your colleagues shows that none of them is an experienced programmer. Otherwise you would have been explained that "Why?" is a question for humans, to computers we ask "How?": we don't care about their intentions, their beliefs or their feelings because computers... well... computers compute! ;-) The first question an experienced software engineer would ask to Deep Thought is: "where is your test suite?" Before asking _how_ (not "why") a software computed the output from the input we need to know how to distinguish a correct computation from a broken one. That's because we know well something that most people are trained to ignore: REAL software has BUGS. As an imaginary system, Deep Thought can pretend to be a fully bug-free system like only AI researchers might dream in their wet dreams. But the problem is that it's just a dream. Real-world software have bugs. Since we know this, we don't ask "why this is the answer?" but "how can I know if the computation performed is correct?". To answer this simple question we developed basically two techniques: 1. verify that the output produced for every possible input matches the expected one 2. prove that each step of the computation is correct Sometime you can only use one of these techniques, but if you can sell software that cannot be checked by neither of those, you didn't got the software wrong, but your career! Software is knowledge, and you basically managed to sell a sort of fake news that nobody could falsify! Your skills are priceless! :-D Anyway, guess what? With AI black boxes, you can't check either. For the vast majority of the input domain you don't know the expected output AND you cannot verify if each step of the computation is correct because there is at least a step that is a black box. To be fair, you might argue that we know how such black box are calibrated (sorry, I refuse to use anthropomorphic terms like learning and training for something that is totally unrelated to the common experience of such terms), but that's not a great objection: we know how we got the black box, but what we see about the computation performed is meaningless. Should we throw such black boxes out of the window? Not yet. Such statistical tools can be useful in some contexts and you cite a couple of them. Unfortunately, your analysis if severely biased toward the thesis you want to sustain. As a first example you say that black boxes are fine when the cost of a wrong answer is low relative to the value of a correct answer. Let me reformulate in proper technical term. Black boxes can be useful when the cost of their bugs is low. In other words, if the cost of their bugs is high, talking about their usefulness is pointless. So when we can discuss the usefulness of a software (AI or not)? For example whenever the software do not mess with humans! If the data in input does not comes from humans and the output produced does not affect human life, our rights and our freedom, we can safely execute a somewhat brittle software if we like to. Feel free to use AI for your study of materials! That's a fine use (as long as you don't blindly trust its output to build a bridge!). But what about targeted advertising? The "canonical example" mess with people! Indeed even if we don't care about how it can be used to change the result of an election, we have to consider the danger of collecting the large amount of personal data that it needs as input! Such collection is part of the Ad system that would not exists without them, so how they are collected, legally shared and illegally leaked is part of the issue of the AI application itself. The "general atmosphere of suspicion" you see in Europe (well addressed by the GDPR), is not about the technology itself but about the people who sell it as a solution to everything without the basic intellectual honesty of highlighting the serious dangers it poses. Since Facebook's collaboration with Cambridge Analytica we cannot put "targeted advertising" among the safe uses of AI anymore. But don't worry, it's not your fault, it's just your bias kicking here. When you write: "A black box can and should be used when it produces the best results" you do such a wide variety of assumptions that a mail can not confute them all. Let's just note what you say later: "self-driving vehicles eventually will be safer than those piloted by humans; they will produce the best results with respect to traffic injuries and fatalities". This is an act of Faith in Machines! (and in the priests that ask for their favor and interpret their will ;-D) I'm all for the freedom of religion! But when you write for a scientific journal, you should probably keep your litany out, or it will sound as advertising in disguise! Indeed the paper you refer explicitly states that:
These findings demonstrate that developers of this technology and third-party testers cannot simply drive their way to safety. Instead, they will need to develop innovative methods of demonstrating safety and reliability. And yet, the possibility remains that it will not be possible to establish with certainty the safety of autonomous vehicles.
What could I add? :-D You mitigate a little this position by writing: "As with any decision-making system, the black box must be used with knowledge, judgment, and responsibility." but any experienced developer would object that no software MUST BE USED just because it has been written! Indeed we need even more knowledge, judgement and responsibility to decide IF we want to use a software. And this is the crucial issue of today AI. Only people able to build an AI system from scratch can understand its limits. Since most people cannot do this, how can they vote for people who understand the issues? And if politicians do not understand the matter, how can they prevent to be manipulated from business? A gist: they can't. And the murder of Elaine Herzberg shows this very well. Again, it's not a problem with the technology, but with people. Once everybody will be able to code and calibrate their own CNN, we will be able to reduce the evident "mistrust of inexplicable systems" because everybody will be able to understand their limitation. As of today, though, caution is paramount. Meanwhile, instead of trying to delegate Ethics to machines, we should teach them to business and developers. The last two section of your essay are very interesting.
Although AI thought processes can be limited, biased, or outright wrong, they are also different from human thought processes in ways that can reveal new connections and approaches.
AI don't think. It's software executed on a computer. Computers compute. This way of talking influence what people understand about the field, and we should really be a little more intellectually honest and change this language with a more technical and descriptive one.
in a groundbreaking medical imaging study, scientists trained a deep learning system to diagnose diabetic retinopathy—a diabetes complication that affects the eyes—from retinal images. [...] ..the system could accurately identify a number of other characteristics that are not normally assessed with retinal images, including cardiological risk factors, age, and gender (14).
Think about it: this is exactly why you can never exclude unethical biases in a data set! That's why we should NEVER apply these tools to humans! Yet, I totally agree with what follow in your text:
No one had previously noticed gender-based differences in human retinas, so the black box observation inspired researchers to investigate how and why male and female retinas differ. Pursuing those questions took them away from the black box in favor of interpretable artificial and human intelligence.
IMHO this is the only appropriate use of these wonderful tools. As STATISTICAL tools to gain new insights, to "see" new things and develop new models of the reality around us. In other words, this is the only appropriate application of "AI" black boxes: as stepping stones to reach a better understanding of the world. Always remembering that, just like a telescope, a black box shows us something that we couldn't see while hiding something else from our sight at the same time. Giacomo
[-eaholm@andrew.cmu.edu] Hey Giacomo, I may be missing some bits of the conversation here. Is there any missing email in the chain, for example, you had an exchange with this author and asked them the permission to include them into a conversation of this list or something? Or did I miss any other email in the chain in which they were invited, or did they replied explicitly and I missed it? Or is it correct my understanding that you wrote an email to the list with the author of the article in To? If that’s the case, isn’t it a bit weird to drag some one that is far from this community into a discussion within this community on topic that lately has really been the focus of your interest? Provided I understood correctly what happened here, then I think this is not proper behaviour and, if that occurred to me, I’d be very cross at you for doing that. Simone On Fri, 5 Apr 2019 at 17:28, Giacomo Tesio <giacomo@tesio.it> wrote:
Hi Elizabeth,
I'm a programmer from Italy.
I've just read your perspective about the advantages of black boxes at http://science.sciencemag.org/content/364/6435/26?rss=1
While I admit that it has some merits (that I'd like to discuss with you more extensively later), as a programmer I can see how your perspective is flawed by a couple of common pitfalls.
Indeed you ask:
What good is knowing the answer when it is unclear why it is the answer?
The results of your informal survey of your colleagues shows that none of them is an experienced programmer. Otherwise you would have been explained that "Why?" is a question for humans, to computers we ask "How?": we don't care about their intentions, their beliefs or their feelings because computers... well... computers compute! ;-)
The first question an experienced software engineer would ask to Deep Thought is: "where is your test suite?"
Before asking _how_ (not "why") a software computed the output from the input we need to know how to distinguish a correct computation from a broken one.
That's because we know well something that most people are trained to ignore: REAL software has BUGS.
As an imaginary system, Deep Thought can pretend to be a fully bug-free system like only AI researchers might dream in their wet dreams.
But the problem is that it's just a dream. Real-world software have bugs.
Since we know this, we don't ask "why this is the answer?" but "how can I know if the computation performed is correct?".
To answer this simple question we developed basically two techniques:
1. verify that the output produced for every possible input matches the expected one 2. prove that each step of the computation is correct
Sometime you can only use one of these techniques, but if you can sell software that cannot be checked by neither of those, you didn't got the software wrong, but your career! Software is knowledge, and you basically managed to sell a sort of fake news that nobody could falsify! Your skills are priceless! :-D
Anyway, guess what? With AI black boxes, you can't check either. For the vast majority of the input domain you don't know the expected output AND you cannot verify if each step of the computation is correct because there is at least a step that is a black box.
To be fair, you might argue that we know how such black box are calibrated (sorry, I refuse to use anthropomorphic terms like learning and training for something that is totally unrelated to the common experience of such terms), but that's not a great objection: we know how we got the black box, but what we see about the computation performed is meaningless.
Should we throw such black boxes out of the window?
Not yet. Such statistical tools can be useful in some contexts and you cite a couple of them. Unfortunately, your analysis if severely biased toward the thesis you want to sustain.
As a first example you say that black boxes are fine when the cost of a wrong answer is low relative to the value of a correct answer. Let me reformulate in proper technical term.
Black boxes can be useful when the cost of their bugs is low. In other words, if the cost of their bugs is high, talking about their usefulness is pointless.
So when we can discuss the usefulness of a software (AI or not)?
For example whenever the software do not mess with humans! If the data in input does not comes from humans and the output produced does not affect human life, our rights and our freedom, we can safely execute a somewhat brittle software if we like to. Feel free to use AI for your study of materials! That's a fine use (as long as you don't blindly trust its output to build a bridge!).
But what about targeted advertising? The "canonical example" mess with people! Indeed even if we don't care about how it can be used to change the result of an election, we have to consider the danger of collecting the large amount of personal data that it needs as input! Such collection is part of the Ad system that would not exists without them, so how they are collected, legally shared and illegally leaked is part of the issue of the AI application itself.
The "general atmosphere of suspicion" you see in Europe (well addressed by the GDPR), is not about the technology itself but about the people who sell it as a solution to everything without the basic intellectual honesty of highlighting the serious dangers it poses.
Since Facebook's collaboration with Cambridge Analytica we cannot put "targeted advertising" among the safe uses of AI anymore.
But don't worry, it's not your fault, it's just your bias kicking here.
When you write: "A black box can and should be used when it produces the best results" you do such a wide variety of assumptions that a mail can not confute them all.
Let's just note what you say later: "self-driving vehicles eventually will be safer than those piloted by humans; they will produce the best results with respect to traffic injuries and fatalities".
This is an act of Faith in Machines! (and in the priests that ask for their favor and interpret their will ;-D)
I'm all for the freedom of religion!
But when you write for a scientific journal, you should probably keep your litany out, or it will sound as advertising in disguise!
Indeed the paper you refer explicitly states that:
These findings demonstrate that developers of this technology and third-party testers cannot simply drive their way to safety. Instead, they will need to develop innovative methods of demonstrating safety and reliability. And yet, the possibility remains that it will not be possible to establish with certainty the safety of autonomous vehicles.
What could I add? :-D
You mitigate a little this position by writing: "As with any decision-making system, the black box must be used with knowledge, judgment, and responsibility." but any experienced developer would object that no software MUST BE USED just because it has been written!
Indeed we need even more knowledge, judgement and responsibility to decide IF we want to use a software.
And this is the crucial issue of today AI. Only people able to build an AI system from scratch can understand its limits. Since most people cannot do this, how can they vote for people who understand the issues? And if politicians do not understand the matter, how can they prevent to be manipulated from business? A gist: they can't. And the murder of Elaine Herzberg shows this very well.
Again, it's not a problem with the technology, but with people. Once everybody will be able to code and calibrate their own CNN, we will be able to reduce the evident "mistrust of inexplicable systems" because everybody will be able to understand their limitation. As of today, though, caution is paramount.
Meanwhile, instead of trying to delegate Ethics to machines, we should teach them to business and developers.
The last two section of your essay are very interesting.
Although AI thought processes can be limited, biased, or outright wrong, they are also different from human thought processes in ways that can reveal new connections and approaches.
AI don't think. It's software executed on a computer. Computers compute.
This way of talking influence what people understand about the field, and we should really be a little more intellectually honest and change this language with a more technical and descriptive one.
in a groundbreaking medical imaging study, scientists trained a deep learning system to diagnose diabetic retinopathy—a diabetes complication that affects the eyes—from retinal images. [...] ..the system could accurately identify a number of other characteristics that are not normally assessed with retinal images, including cardiological risk factors, age, and gender (14).
Think about it: this is exactly why you can never exclude unethical biases in a data set! That's why we should NEVER apply these tools to humans!
Yet, I totally agree with what follow in your text:
No one had previously noticed gender-based differences in human retinas, so the black box observation inspired researchers to investigate how and why male and female retinas differ. Pursuing those questions took them away from the black box in favor of interpretable artificial and human intelligence.
IMHO this is the only appropriate use of these wonderful tools.
As STATISTICAL tools to gain new insights, to "see" new things and develop new models of the reality around us.
In other words, this is the only appropriate application of "AI" black boxes: as stepping stones to reach a better understanding of the world.
Always remembering that, just like a telescope, a black box shows us something that we couldn't see while hiding something else from our sight at the same time.
Giacomo _______________________________________________ nexa mailing list nexa@server-nexa.polito.it https://server-nexa.polito.it/cgi-bin/mailman/listinfo/nexa
On 05/04/2019, Simone Basso <bassosimone@gmail.com> wrote:
Hey Giacomo, [...] If that’s the case, isn’t it a bit weird to drag some one that is far from this community into a discussion within this community on topic that lately has really been the focus of your interest?
Hackers are weird people... ;-)
Provided I understood correctly what happened here, then I think this is not proper behaviour and, if that occurred to me, I’d be very cross at you for doing that.
While I respect your feeling and note it for the future, I think your reaction would be quite counter productive for yourself. If one of my articles for Science (ROTFL! ! ! :-D) was discussed on a public mailing list with the deepness of Nexa, I would be glad to be informed and join the conversation! :-D OTOH, I would find a bit wimp to do the opposite: why would I have published my mail with the article in the first place if I don't want to discuss the matter anymore? Anyway, if the author agree with you, she will be free to answer me privately and (since I'm weird but not an asshole) the conversation will remain private until she decides otherwise. But to be honest, this mailing list is so interesting that I think she will love to learn about it. ;-) Giacomo
D'accordo. Mi scuso per aver urtato la vostra sensibilità: veramente non era mia intenzione. Sarà mia cura scrivere all'autrice oggi stesso per fornirle i riferimenti della mailing list includendo un link diretto a questo thread. D'altro canto, devo ammettere che sono un po' sorpreso: ho incluso l'autrice come gesto di cortesia nei suoi confronti. Avevo infatti iniziato a scrivere la mia risposta in italiano e solo alla lista, ma vista la lunghezza e la severità della critica che ponevo ho pensato fosse giusto offrirle la possibilità di replicare. Scusate, non succederà più. Giacomo
Giacomo Tesio wrote: On 05/04/2019, Simone Basso <bassosimone@gmail.com> wrote:
Hey Giacomo, [...] If that’s the case, isn’t it a bit weird to drag some one that is far from this community into a discussion within this community on topic that lately has really been the focus of your interest?
Hackers are weird people... ;-)
This is not a response. I have met plenty of non weird hackers and plenty of weird hackers. As much as you can choose to identify yourself with this community, your behaviour is ultimately yours, and alleged stereotypical community traits should not justify it.
Provided I understood correctly what happened here, then I think this is not proper behaviour and, if that occurred to me, I’d be very cross at you for doing that.
While I respect your feeling and note it for the future, I think your reaction would be quite counter productive for yourself.
Dragging someone in the middle of an email thread of a mailing list they may not know is not necessarily the most rational way of eliciting interesting feedback from them. What strikes me as counterproductive here is that you did not consider less direct and more friendly ways of approaching them.
If one of my articles for Science (ROTFL! ! ! :-D) was discussed on a public mailing list with the deepness of Nexa, I would be glad to be informed and join the conversation! :-D
Careful here. Whether the mailing list is deep or not is a subjective call. The may not have the same opinion on the mailing list and they may perhaps be quite annoyed by your message. Or they could just see it as spam and trash it immediately. Who knows. (Heck, they may even answer you and that’s for sure gonna be fun.) But if your rational objective is asking questions to receive answers and reason on them, a less aggressive approach is probably more likely to lead to answers. (You May want to read Raymond’s essay on how to ask questions in the smart way. As much as Raymond is questionable, I think that essay is still interesting read and has actually helped me in this respect.)
OTOH, I would find a bit wimp to do the opposite: why would I have published my mail with the article in the first place if I don't want to discuss the matter anymore?
I am not sure that the choice of publishing the email is personal in that venue. Even if publishing their email was a personal decision, it still remains a personal call whether they want to discuss further with you. And, in turn, that depends on how you approach them.
Anyway, if the author agree with you, she will be free to answer me privately and (since I'm weird but not an asshole) the conversation will remain private until she decides otherwise.
They will behave as they see fit. This is not my concern. I was annoyed by the way in which you behave and I saw useful to chime in. Some sort of “someone is wrong on the Internet” reaction, modulo the fact that I doubt I will ever be 100% right on any topic.
But to be honest, this mailing list is so interesting that I think she will love to learn about it. ;-)
Do you know whether this person speaks Italian? Otherwise, well, it may be quite complex for them to assess the list. (And, again, what gives you confidence that they would share your judgement with respect to the list? The goodness of this list does not strike to me as a self evident truth.) Simone
Giacomo
On April 5, 2019 5:47:06 PM UTC, Simone Basso <bassosimone@gmail.com> wrote:
Giacomo Tesio wrote:
Hackers are weird people... ;-)
This is not a response. I have met plenty of non weird hackers and plenty of weird hackers.
As much as you can choose to identify yourself with this community, your behaviour is ultimately yours, and alleged stereotypical community traits should not justify it.
I think you misreas my words: it was not meant as a justification as I don't see much to justify. It's an explanation of my behaviour. Such behaviour sometimes looks weird to people who don't share my culture. For example what you describe as aggressive is my way to friendly debate with a peer and i would feel as disrespectful if anything less challenging was addressed at me for something I wrote or coded. Unfortunately I'm unable to predict if something I do will look weird until someone tell me it's weird to them. That's why my apology stands: because I realized I hurted you and it was not my intention. I'm really sorry for this.
What strikes me as counterproductive here is that you did not consider less direct and more friendly ways of approaching them.
If you read the mail we are talking about you will see that it's not a set of questions: the article was clear and as I wrote to the author even insightful at some points. But the tone of my criticism was driven by the indignation I feel when I read this kind of arguments. Not because they are wrong, but because they mislead people who don't have the skills to see how broken they are! To my eyes, this behaviour is simply an unethical abuse of our knowledge.
Heck, they may even answer you and that’s for sure gonna be fun.
That's what I hope! :-) There's nothing more funny to me than being proved wrong: that's how I learn things!
You May want to read Raymond’s essay on how to ask questions in the smart way. As much as Raymond is questionable, I think that essay is still interesting read and has actually helped me in this respect.
Indeed, ESR opinions are VERY questionable. But I guess somebody might say the same about me as a whole (indipendently of my opinions). Yet as I said I was not asking questions, but challenging answers.
OTOH, I would find a bit wimp to do the opposite: why would I have published my mail with the article in the first place if I don't want to discuss the matter anymore?
I am not sure that the choice of publishing the email is personal in that venue.
Yes, probably it's mandatory. But WHY it is mandatory? Because debating your contributions is a fundamental part of the scientific method.
They will behave as they see fit. This is not my concern. I was annoyed by the way in which you behave and I saw useful to chime in. Some sort of “someone is wrong on the Internet” reaction, modulo the fact that I doubt I will ever be 100% right on any topic.
Well, given that I've learnt something new about this community, I thank you for sharing your opinion. I will make treasure of this lesson for the future. Now... do you have something to object to the _contents_ of my criticism too? Giacomo
Premesso che interpellare una persona non iscritta a una lista mettendo in copia la lista stessa non è prassi consigliabile anche solo perché l'eventuale risposta della persona non passerebbe in lista (essendo la persona non iscritta alla lista), per favore portate l'eventuale prosieguo dello scambio 'offlist'. Grazie molte, buona serata, juan carlos On 05/04/2019 20:52, Giacomo wrote:
On April 5, 2019 5:47:06 PM UTC, Simone Basso <bassosimone@gmail.com> wrote:
Giacomo Tesio wrote:
Hackers are weird people... ;-) This is not a response. I have met plenty of non weird hackers and plenty of weird hackers.
As much as you can choose to identify yourself with this community, your behaviour is ultimately yours, and alleged stereotypical community traits should not justify it. I think you misreas my words: it was not meant as a justification as I don't see much to justify.
It's an explanation of my behaviour. Such behaviour sometimes looks weird to people who don't share my culture. For example what you describe as aggressive is my way to friendly debate with a peer and i would feel as disrespectful if anything less challenging was addressed at me for something I wrote or coded.
Unfortunately I'm unable to predict if something I do will look weird until someone tell me it's weird to them.
That's why my apology stands: because I realized I hurted you and it was not my intention. I'm really sorry for this.
What strikes me as counterproductive here is that you did not consider less direct and more friendly ways of approaching them. If you read the mail we are talking about you will see that it's not a set of questions: the article was clear and as I wrote to the author even insightful at some points.
But the tone of my criticism was driven by the indignation I feel when I read this kind of arguments.
Not because they are wrong, but because they mislead people who don't have the skills to see how broken they are!
To my eyes, this behaviour is simply an unethical abuse of our knowledge.
Heck, they may even answer you and that’s for sure gonna be fun. That's what I hope! :-)
There's nothing more funny to me than being proved wrong: that's how I learn things!
You May want to read Raymond’s essay on how to ask questions in the smart way. As much as Raymond is questionable, I think that essay is still interesting read and has actually helped me in this respect. Indeed, ESR opinions are VERY questionable. But I guess somebody might say the same about me as a whole (indipendently of my opinions).
Yet as I said I was not asking questions, but challenging answers.
OTOH, I would find a bit wimp to do the opposite: why would I have published my mail with the article in the first place if I don't want to discuss the matter anymore? I am not sure that the choice of publishing the email is personal in that venue. Yes, probably it's mandatory.
But WHY it is mandatory? Because debating your contributions is a fundamental part of the scientific method.
They will behave as they see fit. This is not my concern. I was annoyed by the way in which you behave and I saw useful to chime in. Some sort of “someone is wrong on the Internet” reaction, modulo the fact that I doubt I will ever be 100% right on any topic. Well, given that I've learnt something new about this community, I thank you for sharing your opinion. I will make treasure of this lesson for the future.
Now... do you have something to object to the _contents_ of my criticism too?
Giacomo _______________________________________________ nexa mailing list nexa@server-nexa.polito.it https://server-nexa.polito.it/cgi-bin/mailman/listinfo/nexa
participants (5)
-
antonio.vetro＠polito.it -
Giacomo -
Giacomo Tesio
-
J.C. DE MARTIN -
Simone Basso