It's All About Data and Whoever Trains the Algorithm
The capabilities of Artificial Intelligence (AI) software systems are rapidly progressing, and they are expected to have an increasingly disruptive impact on several industries, with substantial positive effects on the economy and society.
As with any technological innovation, the adoption of AI technologies carries certain risks, that we will have to learn to recognize and prevent. One such risk is that AI systems have the potential to perpetuate biases and discriminations against certain groups, for example on the basis of gender, ethnicity or age.
How can an AI algorithm have biases or create discrimination?
Typically, one trains an AI system to perform a certain task by using data that shows examples of correct solutions to the task. A Machine Learning (ML) training algorithm then comes up with a model of the data and of the task that works well on the given examples, and usually this generalizes well on subsequent examples. For example, one may train a system to provide a description of a given photograph by first showing it millions of examples of photographs together with their descriptions. The Chat GPT-3 system was trained for the task of predicting the next word in a sentence, by giving it billions of examples of fragments of sentences together with the subsequent word in the sentence. Given this next-word-predictor, Chat GPT-3 basically applies it over and over to create longer texts, and it is able to create surprising coherent essays, stories, poems, and so on, as anybody who has played with it will have seen.
Back to how discrimination can arise, suppose that a company wants to automate parts of its hiring process, and to do so it creates an AI system to score the likelihood that a given job applicant will be hired, so that the lowest scoring applicants can be automatically rejected and only the top scoring applicants are evaluated by a human. In this case the company would give the ML algorithm data about past applicants and whether they were hired, so that the system can build a model of what features in an applicant's CV make him/her likely to be hired. What if, however, the company has been discriminating against women applicants in the past? Then, in building a model that is faithful to the data, the ML algorithm would learn to give women lower scores than men. The same would happen if the company had discriminated against older applicants in the past, or against applicants with certain other characteristics.
A similar problem could arise if one wants to build a credit rating system to predict how likely is a given loan applicant to default on the loan, or a system to predict how likely is a convict applying for parole to commit crimes again if released. One would train such systems on past data, but past data could reflect past discrimination, for example against people of certain ethnicities, and the risk is to train a model that would also make biased choices.
Unfortunately, one cannot protect from this risk by simply erasing sensitive characteristics such as gender or ethnicity from the training data, because such characteristics might strongly correlate with other data that is present, and the AI system might learn to infer those characteristics from the available data, and then discriminate on such basis!
A concrete example of this risk occurred a few years ago in the US: the COMPAS system was a proprietary algorithm developed to predict the likelihood that a given convict would become a repeat offender, and it was used by courts to decide on sentencing and on parole. The algorithm took into account 137 parameters, and the race of the person was not one of the parameters, but a 2016 investigation by Pro Publica demonstrated that the system had a consistent bias against Black convicts, assigning them a risk of repeat offending that was higher than what was the actual rate of repeat offenses seen in subsequent years, while white convicts were assigned a risk that was lower than the effective rate at which they committed crimes again in subsequent years. The model underlying COMPAS, evidently, had effectively learned from its training data how to predict the ethnicity of people and then it discriminated on such basis.
Another source of biases in AI can come from a similar more benign issue, namely the representation of different groups in the training data. In ML, one needs large amount of data to achieve good precision, so if a certain subgroup is under-represented in the training data it is possible that the ML algorithm will create a model that is less precise for that subgroup. For example, facial recognition systems trained on pictures that show mostly faces of white people might have a high error rate when trying to recognize the faces of people of other ethnicities, a problem that has come up multiple times in the area of face recognition.
These issues are concerning but there are reasons to be optimistic about them. The fact that biased data leads to biased predictions is a classical problem in statistics, and there are approaches to audit training data before using it and to audit a model after the training phase to detect possible biases. Auditing a model created by a modern ML algorithm is difficult because its predictions are based on complicated calculations involving millions or billions of numerical parameters, so, partly motivated by fairness and accountability issues, the problem of building so-called explainable AI systems is being actively investigated. An explainable AI system makes a prediction and also provides a human-readable explanation for its prediction. Finally, there is ongoing work to enact regulations around the use of AI, such as the proposed "AI Act" currently being discussed by the European parliament.