Disclaimer: These mini-essays capture my current reflections on ethics in design, intended as a way to retain and share knowledge. They aren’t final conclusions, and I welcome any additional perspectives or insights on topics I may have overlooked. Your thoughts are appreciated!
There are many different methods available to help make products more ethical. One that has recently gained renewed attention is the trolley problem, a series of thought experiments used in ethics, psychology, and, increasingly, artificial intelligence. It raises the ethical question of whether it is right to harm one person to prevent harm to a larger group. Philippa Foot first posed the question of differing moral judgments, and Judith Jarvis Thomson later popularised it as "the trolley problem."
The classic trolley problem involves imagining a runaway trolley headed towards a group of people. You can divert the trolley to a track where it will only kill one person instead of many. This creates a moral dilemma: should you actively intervene and sacrifice one person to save several others, or keep the trolley on its current trajectory and allow it to kill multiple people? Most people Thomson surveyed agreed that diverting the trolley to save more lives, even at the cost of one, is the more moral choice.
However, consider a doctor faced with saving five patients by killing one healthy person for their organs. In this scenario, sacrificing one individual feels far less moral, highlighting the complexity of ethical decision-making.
These thought experiments can become more complex by factoring in variables like race, gender, and age. This approach is now being applied to autonomous vehicles, where similar ethical dilemmas arise.
Another thought experiment is the Black Mirror Test, which I first encountered through a colleague and later heard discussed by Roisi Proven on the Mind the Product podcast. The test is inspired by the dystopian TV series Black Mirror, created by Charlie Brooker, where technology plays a central role in negatively affecting human lives. The Black Mirror Test asks you to imagine the most dystopian outcome of a product or feature, not to predict the future but to explore potential negative impacts.
People tend to focus on the positive outcomes of the products they develop, which is natural. No one wants to think that their creation could harm others. Yet this is precisely why it is crucial to consider the adverse effects a product could have, as well as the harmful ways it could be misused. For instance, while the invention of the hammer over three million years ago was likely intended for constructive purposes, it can also be used as a weapon.
A chilling example of unintended consequences discussed in the podcast episode is the racial dot map of America. This tool shows the country's racial makeup based on where people live. While useful for visualising patterns of segregation, the data can be misused. For example, by overlaying Google Maps on the racial dot map, someone might falsely assume that individuals living in a predominantly Black or Latino area near a prison are former inmates based on unrelated location data. This illustrates how data, often assumed to be neutral, can lead to harmful and inaccurate conclusions.
Data without context can also lead to negative impacts. A stark example is when, early in the COVID-19 pandemic, many media outlets reported absolute case numbers per country without considering population sizes or how the data was recorded. This led to misleading comparisons between countries. For instance, comparing the total number of cases in the UK and that of the Republic of Ireland does not consider that Ireland has a much lower population density and a much more significant percentage of people live in rural areas. Without adjusting for population and other factors, people could not assess the accurate scale of outbreaks relative to country size.
Seeking external perspectives is a valuable way to challenge assumptions and ensure problems are viewed through diverse lenses. Designers who champion their creations may find it difficult to critique them impartially. Bringing in someone neutral can provide fresh insights into the ethical implications of a product. More importantly, engaging people from different cultures and backgrounds helps prevent designing in isolation and ensures that the solutions created are representative of the people they are intended for.
There are many methods available to help designers reflect on the ethical implications of their work. Thought experiments are straightforward and can be applied at various stages of product or service development. However, the most crucial practice is consistently questioning whether what we are creating benefits the people and society we design for. By regularly assessing potential negative impacts, we can minimise harm and better safeguard the users for whom we are ultimately designing.
November writing progress: 3191/50000
References
The trolley problem: https://openyls.law.yale.edu/bitstream/handle/20.500.13051/16338/56_94YaleLJ1395_1984_1985_.pdf
Racial dot map of America: https://www.arcgis.com/apps/mapviewer/index.html?webmap=30d2e10d4d694b3eb4dc4d2e58dbb5a5
BBC article: https://www.bbc.co.uk/news/52311014
Comments