WILLIAM GUMEDE | Not intelligent: AI’s biases are dangerous and need more oversight

Digital technology, particularly artificial intelligence (AI), which has become an integral part of our lives, can reinforce racism and undermine human rights and personal safety. It needs greater regulatory oversight, increased supervision and more accountability from technology companies to reduce its negative impact.

AI, a pillar of the data economy, brings together algorithms and computing power, allowing machines to perform decision-making, object-recognition and problem-solving functions.

It can protect the rights, security and access to public services of citizens. However, the asymmetric algorithmic decision-making of AI can also be harmful to them. It is increasingly clear that certain AI algorithms used to predict human behaviour can display racial, gender and xenophobic bias, resulting in safety and liability concerns with AI systems.

AI could be used maliciously by governments to undermine democratic rights, criminalise citizens and marginalise perceived outsiders.

A 2018 study, Gender Shades, co-authored by Timnit Gebru and Joy Buolamwini, argued that IA’s facial-recognition technology may promote racism. In their paper they show error rates for identifying darker-skinned people were much higher than those for lighter-skinned people because the data sets used to train algorithms for facial recognition technology were overwhelmingly white. 

Gebru was staff research scientist and co-lead of ethical artificial intelligence at Google. In 2020 she was forced to resign from the company, allegedly because of a paper she co-wrote with five others titled “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”. It warned about the inherent biases and environmental costs of creating large AI language models, and questioned whether big-tech companies were doing enough to reduce potential risks.

According to Gebru, an expert on algorithmic bias, she was forced to resign after she refused Google’s request to retract the paper or remove from it her name and those of other co-authors who worked for the company. She felt she was being censored. The head of Google AI at the time, Jeff Dean, said the paper “didn’t meet our bar for publication”. The company’s AI team created a giant AI language model, BERT, in 2018, which it incorporated into its search engine.

In a 2020 study University of Cambridge researchers Kanta Dihal and Stephen Cave warned that AI algorithms created by 'racially homogenous' technologists may build machines with built-in racial and gender biases.

In a 2020 study University of Cambridge researchers Kanta Dihal and Stephen Cave warned that AI algorithms created by “racially homogeneous” technologists may build machines with built-in racial and gender biases. Crave is executive director of the Leverhulme Centre for the Future of Intelligence (CFI) and Dihal leads the CFI’s Decolonising AI initiative. They said many AI systems are racialised as white, which will have dangerous consequences for users who aren’t.

The researchers rightly said that when AI systems are racialised, this “perpetuates ‘real-world’ racial biases” and might be “erasing people of colour” out of virtual existence. Dihal and Cave called on technology companies to diversify the demographic of software developers or racial and gender biases will increase. 

An investigation in 2016 by US organisation ProPublica found AI software predicted black people are at higher risk of committing a second crime after a first arrest. The software based its conclusions on imprisonment numbers. The ProPublica investigation concluded that black defendants were often predicted to be at a higher risk of returning to crime than they were, were twice as likely to be misclassified as higher risk compared with their white counterparts and white defendants were predicted to be less risky than they were.

According to the investigation, black defendants were also twice as likely as white defendants to be misclassified as being reconvicted of violence offences, while white defendants were 63% more likely to be misclassified as at low risk of reconviction for violent crime. Responding to the findings, US Democrat Alexandria Ocasio-Cortez said: “Algorithms are still made by human beings and those algorithms are still pegged to basic human assumptions. They’re just automated assumptions. And if you don’t fix the bias, then you are just automating the bias.”

A 2015 study led by Anupam Datta at Carnegie Mellon University found Google’s online algorithm for its online advertising system portrayed men as more suited to executive jobs than women. In the study men were more likely to be shown online ads for top executive jobs than women, highlighting gender discrimination in targeted online ads. The Datta study was based on experiments with simulated user profiles which analysed targeted advertisements managed by Google’s DoubleClick ad network on third-party websites.

Data governance in technology companies needs to be improved. Technology companies must make data management controls part of compliance monitoring. Introducing open source into AI, without undermining privacy laws, will not only improve the quality of AI, but provide a mechanism for oversight.

There is a fear AI could be used for racial profiling by countries persecuting minority groups. In 2019 whistle-blowers told the New York Times that the Chinese government was using facial-recognition software to “track and control” the country’s minority Muslim group, the Uighurs. The Chinese government allegedly installed AI systems in its public security cameras to identify Uighurs, then used the data to watch the community, which is persecuted in the country. The whistle-blowers exposed police in the Chinese city of Sanmenxia and said there was increasing demand from the country’s government departments and agencies for screening systems programmed for Uighur facial recognition.

To prevent such racial and gender bias, some countries have data protection rules to disallow the use of biometrics for facial-recognition systems, except under strict public interest conditions.

Tendayi Achiume, the UN special rapporteur on racism, said governments must compensate those who have been discriminated against by AI systems.

Demographic diversity among software developers is crucial and there must be formal regulations to address AI safety concerns. Liability, consumer protection and product safety rules should be tightened to tackle accidents, product faults and errors related to AI systems, ensuring human oversight over these products.

Data governance in technology companies needs to be improved. Technology companies must make data management controls part of compliance monitoring. Introducing open source into IA, without undermining privacy laws, will not only improve the quality of AI, but provide a mechanism for oversight.

In areas where citizens’ rights are directly impacted by AI, such as the judiciary, policing and welfare, there must be rules that it follows democratic, fair and privacy principles — and has human oversight. A recent EU White Paper rightly argued that critical public service application should always be validated by a human if AI is used.

William Gumede is associate professor, School of Governance, University of the Witwatersrand, and author of ‘Restless Nation: Making Sense of Troubled Times’ (Tafelberg).