By Payal Dhar
In the 1970s, when transgender residents in Great Britain tried to correct their gender in government IDs, they encountered a new computerized system programmed to trigger “compatibility check failures.” Mar Hicks, a historian who studies how gender and sexuality bring hidden technological dynamics to light, documents how this “failure” mode had been deliberately programmed so that trans people would not be allowed to exist, except on rare, case-by-case bases. Though this practice was abandoned in 2011, it remains one of the earliest examples of algorithmic bias.
Recent research shows that systemic biases are encoded into algorithms even today. But while data itself is neither good nor evil, how it is used can exacerbate systemic inequalities.
Can Machines Be Neutral?
Though artificial intelligence (AI) is ubiquitous in the modern world, the rhetoric that algorithms are unbiased is a myth. “Technologies are never neutral,” Hicks writes, “but sometimes seem so—usually right up to the point when we realize they’ve caused [an] irreversible change.” This is because the data that powers AI doesn’t exist in a vacuum; it reflects existing social, political, and economic values, including biases.
Of late, algorithmic bias has come under scrutiny from stakeholders who seek to shift the AI ecosystem towards equity and accountability. The advances in algorithmic fairness, however, have largely omitted LGBTQ perspectives. In February 2021, Nenad Tomasev and colleagues—researchers at DeepMind, a British AI company owned by Alphabet, Inc.—published a paper highlighting this exclusion.
The researchers studied the positive and negative effects of AI on LGBTQ communities based on privacy, censorship, language, online safety, health, and employment—concerns that particularly affect queer individuals. “Most current approaches for algorithmic fairness assume that the target characteristics for fairness—frequently, race, and legal gender—can be observed or recorded,” according to the paper. “Sexual orientation and gender identity are prototypical instances of unobserved characteristics, which are frequently missing, unknown, or fundamentally unmeasurable.”
Technologies are never neutral, but sometimes seem so—usually right up to the point when we realize they’ve caused [an] irreversible change.
—Mar Hicks, historian
When discussing algorithmic bias, the legacy of historical oppression faced by queer and trans communities cannot be overlooked. StopBullying.gov documents that LGBTQ youth—including those perceived as LGBTQ—face an increased likelihood of bullying. This puts them at a higher risk for depression, self-harm, and suicidal ideation, as well as drug and alcohol abuse, and poor academic performance. Among many disheartening statistics, a staggering 40 percent of transgender adults report having made a suicide attempt, 92 percent of which report making the attempt before the age of 25.
Everyday harms result from this exclusion, from micro-aggressions to outright violence, says Lauren Klein, a digital humanities scholar at Emory University. “The solution is not always to make people’s gender or sexual orientation categorizable or otherwise visible.”
Yilun Yang and Michal Kolinski’s controversial paper claiming that neural networks can detect sexual orientation from facial images set off sparks four years ago. While it was derided and debunked, many researchers and activists spoke up about the dangers of technology that is able to detect sexual orientation, especially when it comes to the privacy and safety of queer people. In their book Data Feminism, Klein and her co-author, Catherine D’Ignazio, call this the “paradox of exposure.” As Klein explains, “It’s often the case that those who stand to significantly gain from being counted in one particular dataset or being recognized by one particular algorithm in one particular context are the same who experience the most potential harm from that same dataset or algorithm in another.” Gender recognition software, for example, could ensure trans folks weren’t consistently misgendered, but they could just as easily identify trans people and put them at risk.
AI on social media sites is another prime example. On Facebook, for instance, if someone has multiple profiles—as is common for some queer people who are not out to everyone in their lives—the algorithm will often recommend their alternate profile to people in their public network. Oliver Haimson, a social computing researcher at the University of Michigan who studies the experiences of trans folks on social media, says, “In some of my studies, we’ve found that this is a way that [trans] people are outed.”
Tomasev and colleagues found that there’s a high risk that AI systems will be designed and deployed unfairly for queer people. “Compounding this risk, sensitive information for queer people is usually not available to those developing AI systems, rendering the resulting unfairness unmeasurable from the perspective of standard group fairness metrics,” they write. Their paper also reminds us that some existing algorithms have their roots in the early 20th-century eugenics movement, which was explicitly homophobic and transphobic.
Meanwhile, another recent study highlights the algorithmic exclusion faced by trans and non-binary individuals in personal health tech apps and gadgets. The study’s authors, Maggie Delano and Kendra Albert, used Fitbit and the Withings Smart Scale to show how long-held (and often inaccurate) assumptions about gender inform the decision-making in quasi-medical applications.
“Algorithmic decision-making or regression-based decision-making can make it harder for folks who are outliers,” explains Albert. In any individual data set, non-binary folks are often a small number, making it easier to just write them off as exceptions. “Sometimes it’s so common for non-binary [and trans] folks to not be able to use these technologies as they’re intended, that it doesn’t really occur to anybody to complain,” they add. “We are hoping [in our paper] to illustrate the ways in which [this] happens—in this very specific case, but also [more broadly.]”
Algorithmic decision-making or regression-based decision-making can make it harder for folks who are outliers.
—Kendra Albert, researcher
People talk about algorithm-based tools like they are neutral, adds Delano, but having clearly defined systems of inputs and outputs are fundamental to how a lot of these systems work and a lot of LGBTQ people are resistant to that for good reason. “But it’s the same group of people who get excluded every single time, [and] when you add up all of these [exclusions] the impact is enormous.”
Haimson also blames categorizations for “a lot of trouble,” making AI- and machine learning (ML)-based technologies “more harmful than useful for trans populations.” Uber’s facial recognition-based security features, for instance, have resulted in trans drivers kicked off the platform. “Similar things happen on some online communities that want to be for women-only” and they decide to screen users by policing appearance using AI, says Haimson. “You not only exclude trans and non-binary people but also end up excluding some cisgender women if their appearance ends up being considered ‘too masculine’ for the algorithm.”
Bridging the Algorithmic Gap
An obvious starting point to address the algorithmic gap is more gender-diverse representation among those who design AI systems, and more direct engagement with communities that might be impacted, says Klein. “And this engagement shouldn’t just happen at the very end of the development process but in the very beginning, when you ask the question: Should this system exist?”
Haimson agrees. Inclusion doesn’t work unless you consider marginalized users throughout the design process. “Like when Facebook started having additional gender options, it definitely didn’t feel like it was something that it was intentionally designed for; it was more like, we’ll tack this on, however we can,” Haimson says.
For real change to happen, Delano believes, it needs to come from all levels, which is part of the challenge. “You need conferences or journals to not publish work that [will] further harm and grant agencies to not fund them; [you need to] fund work that will move things in the right direction. [You need] individual researchers to value these things, and reach out to people who are affected by technology and try to work with them.”
[You need] individual researchers to value these things, and reach out to people who are affected by technology and try to work with them.
—Maggie Delano, researcher
Things are changing, Delano continues. “I mean, we’re having this conversation right now, right?” The collective work of people mentioned in this piece, and others like Sasha Constanza-Chock, Virginia Eubanks, Safiya Noble, and Joy Lisi Rankin are already driving that required change forward. Alongside them, organizations like the Algorithmic Justice League, AI Now Institute, Data4BlackLives, National LGBTQ Taskforce, and FAT/ML are committed to unmasking the biases of AI and holding the right people (and algorithms) accountable.
But this has to happen on a much larger scale than it is happening now, says Delano. “It’s going to be hard. Do you try to fit queer and trans experiences into this rigid ML system, or do you need to think differently about how we quantify things?” Maybe, Delano adds, “we can think of completely new paradigms.”
Lead image by ThisisEngineering RAEng on Unsplash