Whether Artificial Intelligence (AI) penalises the occurrence of words like “women’s chess club” in applications (Dastin, 2018), directs police officers to minority neighbourhoods (O’Neil, 2016, pp. 86–87), or is feared to uncover obscure correlations that further entrench existing structural discrimination (Zimmermann & Lee-Stronach, 2022, p. 12), many complex questions quickly arise. Among those are questions about the concept of discrimination itself, the permissibility of disadvantaging specific groups, and the conceptual nature of structural discrimination. Moreover, the possibility of AI becoming the discriminator and the role of developers in AI discrimination similarly come under scrutiny. This book engages with these emerging questions, clarifies which questions are even involved, and offers a framework to answer them.
When developing such a framework, the concept of discrimination itself and the conditions under which discrimination is wrong are the first objects of analysis, which requires a deep engagement with the ethics of discrimination. Both Eidelson (2015, p. 3) and Lippert-Rasmussen (2014, p. 4) point out that philosophers have given relatively little attention to the concept of discrimination despite its prominence in legal and political discourses. More recently, this has begun to change, and several competing concepts of what discrimination is and why it is sometimes wrong exist (Eidelson, 2015; Hellman, 2011; Lippert-Rasmussen, 2014; Moreau, 2020; Sangiovanni, 2017). Even beyond those cited as examples, there have been other proposals for accounts of discrimination (see, for an overview of the debate, Klonschinski, 2020; see, for an example, Segall, 2012). Nonetheless, the contributions by Eidelson, Hellman, Lippert-Rasmussen, Moreau, and Sangiovanni represent contributions that detail and defend full accounts of discrimination in monographic length, which allow us to scrutinise both their central theories and their discussions of concepts like indirect or structural discrimination. While there is widespread agreement to conceptualise discrimination as a generic concept that is not always wrong, there are numerous fundamental disagreements about the boundaries of the concept of discrimination and, most importantly, about the wrong-making feature of discrimination. The first contribution of this book is to evaluate those accounts critically and to develop a unique account, the expressive disrespect account, based on the strengths and weaknesses of existing theories. Moreover, I will discuss different kinds of discrimination, like indirect or structural discrimination, as well as their properties and wrongfulness. While this discussion will be undertaken with a focus on concepts that are particularly relevant for discussing AI discrimination, this book also presents an independent contribution to the discourse on the ethics of discrimination.
The next group of questions that arises concerns the relationship between AI and discrimination. Quickly and thoroughly, AI models have been criticised for reproducing racist or sexist decision-making and for being trained on unrepresentative datasets, both in expert and public discourse (Barocas & Selbst, 2016; Chander, 2017; Sasani, 2024). Extensively, ongoing contributions from the computer sciences emerged that propose technical fixes, such as debiasing techniques (Alabdulmohsin & Lucic, 2021; Chen et al., 2023; Y. Wang & Liu, 2023). However, technical interventions have themselves come under ongoing criticism for ignoring the social context a model operates in (Selbst et al., 2019), obscuring the extent to which the features used for prediction are shaped by structural injustices (Zimmermann & Lee-Stronach, 2022), or for operating on an insufficient understanding of inequality, ultimately reproducing an unjust status quo (Zajko, 2021, 2022). Yet, those discussions often point out the consequences of different understandings of discrimination, bias, or inequality, but they do not articulate the ethical duties or obligations individual actors have. The lack of explicitly ethically motivated arguments was pointed out by Le Bui and Noble (2020) in an article titled “We’re Missing a Moral Framework of Justice in Artificial Intelligence”. What is true for justice generally is also specifically true for the ethics of discrimination. Philosophers have taken up the challenges that AI presents to the ethics of discrimination, but they have done so mostly in response to specific issues (see, for example, Hellman, 2023; Koch, 2020). Hence, this book’s second contribution is to provide a framework for analysing AI discrimination based on the expressive disrespect account. In doing so, it analyses and ethically motivates criticism of bias in AI, normatively underpins arguments for a sociotechnical understanding of such technology, and denotes the limits of individual duties to avoid discrimination and alleviate structural discrimination.
Third and finally, the debate on AI discrimination has brought criticism of the concept of discrimination to the forefront again. Discrimination as a concept has been criticised as insufficiently considering structural oppression by focusing on agents and their individual actions (Young, 1990, p. 195). Similar lines of criticism are identifiable in the context of AI discrimination, stressing a focus on inequality in general (Zajko, 2021) and sometimes explicitly arguing that a focus on fairness can obscure unjust background conditions (Zimmermann & Lee-Stronach, 2022). Hence, I will take up this criticism and engage with it both as a criticism of the expressive disrespect account as an agent-oriented theory and as a contribution to evaluating AI models operating in a social world characterised by inequality. Engaging with this criticism allows for a discussion of the extent to which individual actions are involved in upholding structural discrimination while also determining the limits of discrimination theory. This is especially important for the context of AI, where the expressive disrespect account can be used to criticise individual actions while maintaining that, for many structural problems, political solutions remain necessary. Engaging with this criticism and defending the concept of discrimination in the context of AI is the third contribution of this work.
This book is organised into Chapters 1–3. Chapter 1 will introduce and discuss different accounts of discrimination. The accounts generally conceptualise discrimination as a generic term that does not necessarily describe a wrongful action and defend an additional feature that renders some discrimination wrongful. The competing accounts of discrimination are denoted by their central wrong-making feature. Among them is the desert prioritarian harm-based account, which places the wrong of discrimination in the harm it entails for the affected, while giving greater weight to harm that befalls worse-off people (Lippert-Rasmussen, 2014). Additionally, two accounts place central importance on social meaning. The first, the demeaning account, argues that discrimination is wrong when it is an offence against equal moral worth (Hellman, 2011). The second, the expressive harm account, defines wrongful discrimination as the use of others’ vulnerability to attack their capacity to maintain an intact sense of self (Sangiovanni, 2017). Then, there is Eidelson’s (2015) disrespect account, according to which discrimination is wrong when it fails to adequately take into account the personhood of others. Finally, there is Moreau’s (2020) pluralist account, according to which no single wrong-making feature can be determined, and discrimination can be wrongful because it subordinates, constitutes a denial of deliberative freedoms, or involves the denial of basic goods.
After introducing the different accounts in Chapter 1.1, I will analyse and discuss the central disagreements between them in Chapter 1.2. This discussion will include the question of whether discrimination is a concept best limited to socially salient groups, which are groups that structure social interactions across different contexts. Defending the concept against criticism by Eidelson (2015), I will argue that the salience criterion helpfully restricts the discussion to actions that are parts of patterns, which renders such actions deserving of special moral scrutiny. Moreover, in opposition to Hellman (2011), I will reject constraining the concept of discrimination to groups that have historically suffered a disadvantage and argue that wrongful discrimination does not presuppose a position of power. After those general aspects, I will evaluate the potential of different wrong-making features of wrongful discrimination. First, accounts based on social meaning, which is determined by analysing an action against the background of existing stereotypes, political power, and economic position of those involved, will be compared with harm-based approaches. Harm-based accounts generally cannot explain cases of discrimination in which no one was harmed but which still appear wrongful. Lippert-Rasmussen’s desert prioritarian harm-based account additionally renders the concept of wrongful discrimination overly inclusive. Afterwards, I will compare the two accounts that take into account social meaning, namely the expressive harm account and the demeaning account. While the demeaning account is overly reliant on an intuitive understanding of equal moral worth, the expressive harm account presents a more precise alternative. As a critique of expression-based accounts, I will discuss the pluralist challenge. However, the different ways of treating others as unequal that Moreau (2020) defines can only serve to distinguish different cases if the social meaning connected to them is scrutinised. Afterwards, the disrespect account will be discussed, which I will find to have important similarities with the expressive harm account. However, the disrespect account relies on an intuitive understanding of what it means to unduly discount others’ interests as persons. Additionally, it lacks an analysis of social meaning necessary for understanding how conventional disrespect is involved in expressing basic disrespect.
After finding the expressive harm account to be the most promising, I will critically evaluate its potential weaknesses and find that it falls short in cases where those affected by discrimination never learn of being affected. Thus, the requirement of respect for others as persons must be strengthened within the account. Consequently, it must be combined with an account of disrespect in order to accommodate such cases. Combining the expressive harm account with Eidelson’s account leads to a definition of persons as self-presenting beings immersed in a net of social relations they exist within. Others ought to respect this personhood that is vulnerable to disruption. Moreover, an analysis of social meaning must be included in determining what actions express disrespect for this personhood. This joint account will henceforth be denoted by the term expressive disrespect account.
Based on this account, Chapter 1.3 introduces several distinctions between different types of discrimination, such as direct and indirect discrimination, rational discrimination in the pursuit of legitimate ends, and structural discrimination. Closely connected to the definition of these terms is the discussion of their wrongfulness. For example, indirect discrimination will be defined as any measure that disproportionately disadvantages a salient group without these disadvantages being traceable to any bias on the side of those who have introduced the measure. When indirect discrimination is used to pursue legitimate ends, this does not constitute an expression of disrespect as long as reasonable accommodations for disadvantaged groups are provided. Another contested and central term is structural discrimination, denoting the reproduction of disadvantageous outcomes for salient groups through the social structures of society. Sangiovanni (2017, p. 168) argues that everyone who fails to counteract those patterns expresses objectionable indifference. I will find that this is only accurate in a narrow set of cases because a positive vision of society is usually required to counteract structural discrimination. Discrimination theory itself cannot deliver such a positive vision. However, despite criticisms of discrimination theory as insufficient precisely because it fails to address structural factors, I will defend discrimination as a useful concept to identify instances where individual actors express disrespect through their actions in upholding structural discrimination despite having reasonable alternatives. Taken together, the Subchapters 1.3.1–1.3.6 provide a framework for different types of discrimination.
In Chapter 2, the term AI will be specified, and its potential to lead to discriminatory outcomes will be laid out. There, I will not yet classify specific instances as wrongful but introduce examples of potentially wrongful AI discrimination. First, I will discuss the complex question of what the term AI denotes, discussing several interpretations. I will explain and justify my focus on weak, narrow AI, which describes systems that make predictions in a restricted range of tasks and are incapable of experiencing consciousness or developing intentions. While narrowing the concept of AI to machine learning should be resisted, machine learning will be discussed as the most prominent approach to current AI. After having defined the scope of the investigation into AI, potential causes for differential impact on salient groups will be introduced. In Chapter 2.2, I will discuss how design decisions can lead to disadvantages for different groups, for example, if the target of a model is defined in a way that privileges some groups while disadvantaging others. Afterwards, in Chapter 2.3, the way training and input data can lead to disadvantages for salient groups will be laid out. Here, past decisions that were made due to bias, as well as annotator bias, are discussed as specifically relevant aspects. Moreover, it will be discussed that the disadvantages resulting from the training process are often unforeseeable for the developers. Finally, the intersection of AI and existing inequalities will be examined in Chapter 2.4. I will specifically discuss how such intersections severely limit the scope of technical fixes for AI’s impact.
Finally, in Chapter 3, the differential impact of AI will be discussed through the lens of the expressive disrespect account. Consequently, in Chapter 3.1, the question will be asked whether models can express disrespect for the status of others as self-presenting beings. While strong intuitions support this claim, it is ultimately misguided. Hence, the human actors involved remain the necessary addressees for claims of wrongful discrimination, while AI is understood as a procedure secondary to human decisions. The autonomy of weak AI is a misconception based on the idea that the decision-making powers that humans lose in creating unpredictable models are transferred to the model itself, which operates according to the rules it has learnt. Afterwards, in Chapter 3.2, I will discuss how AI discrimination must be viewed as a sociotechnical phenomenon that includes norms and values embedded in the development process and the interaction of models with the social context. Based on this framework, Chapter 3.3 will analyse how a substantial part of AI discrimination merely represents new incarnations of well-established processes of wrongful discrimination. The process of rendering others invisible will be discussed as particularly impactful. This involves the implicit setting of norms in the development, consequently excluding marginalised groups, a process that can be examined through existing criticism of exclusionary procedures and infrastructure.
Following this discussion, Chapter 3.4 will concern opaque machine learning processes and the unforeseeable disadvantages they can generate. Here, I will argue that, in many cases, the risk of differential impact that is created is much higher for marginalised groups. In the absence of appropriate safeguards, this differential risk-taking can be understood as disrespectful and, consequently, as wrongful discrimination. Afterwards, in Chapter 3.5, I will turn to the question of whether disadvantages for novel groups identified through AI pose a challenge to the salience criterion. I will argue that many of the presumed novel groups are connected to existing salient groups, but genuinely novel groups could become salient in some cases. Nonetheless, the salience criterion retains its usefulness because it can help analyse pattern-forming actions. In Chapter 3.6, I will turn to the question of indirect and structural discrimination in the context of AI. Here, the duty to provide reasonable accommodations for disadvantaged groups in cases of indirect discrimination can justify a duty to provide algorithmic affirmative action, offsetting some of the disadvantages being created. Nonetheless, many of the interactions between AI and existing inequalities go beyond such solutions provided by individual actors, and AI is likely to reproduce inequalities and, in some cases, further restrict the opportunities of marginalised and impoverished people. Against this backdrop, I will take up again the expansive definition of objectionable structural discrimination provided by Sangiovanni (2017, p. 168). While Sangiovanni underestimates the complexity of striving for a collective solution, those with special expertise in relation to AI have a duty to inform the broader public about how AI reproduces inequalities and a duty to counteract narratives of neutrality. However, this can only provide the groundwork for a political solution. In Chapter 3.7, I will consequently evaluate the merits and limits of the expressive disrespect account in analysing AI discrimination. The account contributes to demystifying AI discrimination by highlighting individual actions instead of unpredictable training processes. However, some of the structural problems of AI and inequality require a political solution and a broader political philosophy of AI.
Before turning to the different accounts of discrimination, some additional remarks on methods and structure are in order. First, long or particularly complex Subchapters will include a short summary. This will be the case for all Subchapters in Chapter 1 and in Chapters 3.4, 3.5, and 3.6. Moreover, many of the conclusions of different accounts of objectionable discrimination will be tested by introducing fringe cases or thought experiments. As Lippert-Rasmussen (2014, p. 6) writes, this “can illuminate and help us to sensibly assess actual cases of inequality and discrimination”. Hence, such cases help to refine and clarify arguments and criticisms of specific accounts of objectionable discrimination. Moreover, in exploring discrimination, I will not provide a full account of alternative actions in cases of wrongful discrimination. Which alternative procedure is appropriate is highly context-dependent, and those questions lead beyond the scope of the work.
Finally, the goal and scope of this book are necessarily limited. The phenomenon of discrimination in general and AI discrimination in particular is enormously complex, and this work can only provide a framework to assess individual cases. Real cases of (AI) discrimination will be included, both as examples and as the yardstick for the framework itself, but this work does not provide distinctions for all individual cases of discrimination. Within this framework, there can be reasonable disagreement concerning individual cases. However, arguments about and within the framework help illuminate the complex debate on AI discrimination.1
This book follows the conventions of British English. Where quotes are in other varieties of English, such as American or Canadian English, they have been left in their original convention.