This is a continuation of my indefinite series, The Silicon Valley Problem. Part One, regarding the semantics of content and the subordination of creative endeavors, can be found here.
With Facebook once again back in the news for its practices during the 2016 election, I can't think of a better time to discuss one of the biggest (yet most hidden) insidious gifts of Silicon Valley: The complex algorithm. Before I begin in earnest, I should state that I first started thinking about this problem after listening to the terrific 99% Invisible episode on the subject last fall, and draws further on Cathy O'Neill's ideas in her book WEAPONS OF MATH DESTRUCTION. If you are unfamiliar with either or both, please follow those links; they're terrific.
The Case Against Luddism: A Disclaimer
I'm not a luddite, I promise. Yes, I may have a wristwatch made of wood, a Royal typewriter, and a Mac, but I really don't hate technology. For all of Silicon Valley's problems, the advancement of computer science is not one of them; this blog should not be interpreted to say that the existence of complex predictive and analytical algorithms is itself problematic. There's an argument to be made there, but it's not here.
That being said, I am not a computer scientist (you are reading this on a Squarespace site, after all). I've tried to make sure the information that follows is correct, but if I've made any major misinterpretations, even argument-defeating ones, I welcome corrections.
The Un-Understandable
Quite a lot has been written on how the designers of machine-learning algorithms don't understand how their ultimate products function, or if they do, the information is highly proprietary. But I'd like to draw your attention specifically to this article from the MIT Technology Review: "The Dark Secret at the Heart of AI". The article itself is terrifying as most of these things are, about how programs (in this case self-driving cars) are making decisions that, to the outside, seem logical, but whose thought processes that led to those decisions are impossible to pin down. This is a huge problem when algorithms determine everything from what videos you see on YouTube to what patients are most appropriate for a clinical trial of life-saving drugs, but allow me to draw your attention to the following in particular:
"'If you had a very small neural network, you might be able to understand it,' Jaakkola says. 'But once it becomes very large, you have thousands of units per layer and perhaps hundreds of layers, then it becomes quite un-understandable.'"
It's not that the algorithms aren't understood, it's that they can't be understood. The complexity of thought required is beyond the scope of the human, something quite new to our society. This is a source of danger, and one that seems to be taken lightly in the industry. If I were to hand you a bomb, for instance, and say "please don't drop this, it has the capacity to blow up the whole room and both of us with it," it's a safe bet you would handle it very delicately whether it was armed or not.
I'm sure extensive testing and thought is given to how these algorithms operate within society, they are nonetheless both universal and proprietary, and in that combination lies the danger. We drive around with tanks of combustible gasoline under our seats, but we don't light cigarettes at gas stations because we know and understand the risks, and can take steps to reduce the possibility of failure. If taking proper precautions is still too great a risk, the consumer can elect not to use a car. The possibility of informed acceptance of risk is assumed.
On the other hand, the un-understandibility of neural networks and machine-learning algorithms means that we can never fully know how to reduce risk, and when the algorithm is understood, the information is held close to the vest as valuable IP. Thus, informed consent in the use of these products becomes impossible. Millions of humans are subjected to algorithms that are poorly understood and sometimes highly risky.
The Unbiased Robot Fallacy
Okay, so this was inevitable, though. We've now reached a point where we can design programs to do things that human thought alone is incapable of doing. I would say yes, that's true, and that is a triumph of human ingenuity. The problem is that we didn't create the perfect algorithm, not in YouTube's instance, not in Facebook's instance, not in Bank of America's or NASA's or Reddit's or a self-driving car's instance. The trope of the unbiased, cold, calculating machine is a fallacy for the simple fact that, at the end of the day, even if the algorithm was tailored and reiterated by machines, it was still created by a human, and thus subject to human biases. I don't want to re-tread ground already well covered by the links in the introduction, so I'll refer you back to 99% Invisible and WEAPONS OF MATH DESTRUCTION regarding algorithms that perpetuate racism and sexism--it's a fascinating consideration of the fallacy.
If this were an Intro to Philosophy course, I'd bring up the trolly problem and how self-driving cars have to make ethical calculations as to whether to endanger the driver or the bystander, but we don't have to get that dramatic for the effects to be seen on a grander scale. For instance, Facebook a long time ago decided that it was better for its algorithm to favor clicks and engagement regardless of the content provided that it's not pornography or hate speech (which is to say an additional value judgment that we don't have enough space to get into now) than to favor the propagation of legitimate information. This was a business call on their part, and there may or may not be a readily available explanation for why a certain article did or did not propagate according to the algorithm, but that algorithm nonetheless was programmed to place certain values on aspects of the content as decided by its programmers.
To quote the aforementioned 99% Invisible episode:
"Every algorithm reflects the choices of its human designer. O’Neil has a metaphor to help explain how this works. She gives the example of cooking dinner for her family. The ingredients in her kitchen are the 'data' she has to work with, 'but to be completely honest I curate that data because I don’t really use [certain ingredients] … therefore imposing my agenda on this algorithm. And then I’m also defining success, right? I’m in charge of success. I define success to be if my kids eat vegetables at that meal …. My eight year old would define success to be like whether he got to eat Nutella.'"
The Burden of Design
Again, much of this is necessary and good in a vacuum. It was inescapable that we would one day program something more complex than ourselves just as we built a lever that could lift more than our arms. The problem is that the public trust, bordering on blind trust, in the algorithm removes the burden of design from the algorithm's creators. It's what allows Facebook to claim that it's merely a platform, and that it's not its place to determine what content its users see and what they don't, or that, as 99% Invisible used as its example, United Airlines isn't to blame for who it selected to remove from that fateful, violent flight.
The difference seems to be that because an algorithm emulates thought, it can be seen as having made a decision separate and apart from the decision of the human that designed it. This may be somewhat true, though we'd have to litigate the definitions of "decision" and "thought," but to linger on that is to get lost in the weeds. Until we determine that programs have personhood, they must remain in the eyes of responsibility as no more than complicated Rube Goldberg machines--we may not see all their working parts, but they act according to a set of operations that lack higher-order thought, and thus cannot bear responsibility for their actions. Only an architect may know how her building stays standing, but when it collapses she is still responsible.
In the meantime, operations of the algorithms that define our lives need to be made known to the public, and monopolistic practices by the companies that create them need to be curtailed. To do nothing is to deny the informed assessment of risk to the public, and force us to subject ourselves to potentially destructive sorting with know knowledge or alternative option. If an algorithm cannot be understood in its entire by the company that created it, that company must be held responsible when it goes wrong.