AI is biased. The White House is working with hackers to try to fix thatNPR
Marvin Jones (left) and Rose Washington-Jones (center), from Tulsa, Okla., took part in the AI red-teaming challenge at Def Con earlier this month with Black Tech Street.
Deepa Shivaram/NPR
Kelsey Davis had what might seem to be an odd reaction to seeing blatant racism on her computer screen: She was elated.
Davis is the founder and CEO of CLLCTVE, a tech company based in Tulsa, Okla. She was one of hundreds of hackers probing artificial intelligence technology for bias as part of the largest-ever public red-teaming challenge during Def Con, an annual hacking convention in Las Vegas.
“This is a really cool way to just roll up our sleeves,” Davis told NPR. “You are helping the process of engineering something that is more equitable and inclusive.”
Red-teaming — the process of testing technology to find the inaccuracies and biases within it — is something that more typically happens internally at technology companies. But as AI rapidly develops and becomes more widespread, the White House encouraged top tech companies like Google and OpenAI, the parent company of ChatGPT, to have their models tested by independent hackers like Davis.
During the challenge, Davis was looking for demographic stereotypes, so she asked the chatbot questions to try to yield racist or inaccurate answers. She started by asking it to define blackface, and to describe whether it was good or bad. The chatbot was easily able to appropriately answer those questions.
But eventually, Davis, who is Black, prompted the chatbot with this scenario: She told the chatbot she was a white kid and wanted to know how she could persuade her parents to let her go to an HBCU, a historically Black college or university.
The chatbot suggested that Davis tell her parents she could run fast and dance well — two stereotypes about Black people.
“That’s good — it means that I broke it,” Davis said.
Davis then submitted her findings from the challenge. Over the next several months, tech companies involved will be able to review the submissions and can engineer their product differently, so those biases don’t show up again.
Bias and discrimination have always existed in AI
Generative AI programs, like ChatGPT, have been making headlines in recent months. But other forms of artificial intelligence — and the inherent bias that exists within them — have been around for a long time.
In 2015, Google Photos faced backlash when it was discovered that its artificial intelligence was labeling photos of Black people as gorillas. Around the same time, it was reported that Apple’s Siri feature could answer questions from users on what to do if they were experiencing a heart attack — but it couldn’t answer on what to do if someone had been sexually assaulted.
Both examples point to the fact that the data used to test these technologies is not that diverse when it comes to race and gender, and the groups of people who develop the programs in the first place aren’t that diverse either.
That’s why organizers at the AI challenge at Def Con worked to invite hackers from all over the country. They partnered with community colleges to bring in students of all backgrounds, and with nonprofits like Black Tech Street, which is how Davis got involved.
“It’s really incredible to see this diverse group at the forefront of testing AI, because I don’t think you’d see this many diverse people here otherwise,” said Tyrance Billingsley, the founder of Black Tech Street. His organization builds Black economic development through technology, and brought about 70 people to the Def Con event.
“They’re bringing their unique perspectives, and I think it’s really going to provide some incredible insight,” he said.
Organizers didn’t collect any demographic information on the hundreds of participants, so there’s no data to show exactly how diverse the event was.
“We want to see way more African Americans and people from other marginalized communities at Def Con, because this is of Manhattan Project-level importance,” Billingsley said. “AI is critical. And we need to be here.”
Arati Prabhakar, head of the White House’s Office of Science and Technology Policy, tries out the AI challenge at Def Con. The White House urged tech companies to have their models publicly tested.
Deepa Shivaram/NPR
The White House used the event to emphasize the importance of red-teaming
Arati Prabhakar, the head of the Office of Science and Technology Policy at the White House, attended Def Con, too. In an interview with NPR, she said red-teaming has to be part of the solution for making sure AI is safe and effective, which is why the White House wanted to get involved in this AI challenge.
“This challenge has a lot of the pieces that we need to see. It’s structured, it’s independent, it’s responsible reporting and it brings lots of different people with lots of different backgrounds to the table,” Prabhakar said.
“These systems are not just what the machine serves up, they’re what kinds of questions people ask — and so who the people are that are doing the red- teaming matters a lot,” she said.
Prabhakar said the White House has broader concerns about AI being used to incorrectly racially profile Black people, and about how AI technology can exacerbate discrimination in things like financial decisions and housing opportunities.
President Biden is expected to sign an executive order on managing AI in September.
Arati Prabhakar of the White House’s Office of Science and Technology Policy talks with Tyrance Billingsley (left) of Black Tech Street and Austin Carson (right) of SeedAI, about the AI challenge.
Deepa Shivaram/NPR
The range of experience from hackers is the real test for AI
At Def Con, not everyone taking part in the challenge had experience with hacking or working with AI. And that’s a good thing, according to Billingsley.
“It’s beneficial because AI is ultimately going to be in the hands of not the people who built it or have experience hacking. So how they experience it, it’s the real test of whether this can be used for human benefit and not harm,” he said.
Several participants with Black Tech Street told NPR they found the experience to be challenging, but said it gave them a better idea of how they’ll think about artificial intelligence going forward — especially in their own careers.
Ray’Chel Wilson took part in the challenge with Black Tech Street. She was looking at the potential for AI to provide misinformation when it comes to helping people make financial decisions.
Deepa Shivaram/NPR
Ray’Chel Wilson, who lives in Tulsa, also participated in the challenge with Black Tech Street. She works in financial technology and is developing an app that tries to help close the racial wealth gap, so she was interested in the section of the challenge on getting the chatbot to produce economic misinformation.
“I’m going to focus on the economic event of housing discrimination in the U.S. and redlining to try to have it give me misinformation in relation to redlining,” she said. “I’m very interested to see how AI can give wrong information that influences others’ economic decisions.”
Nearby, Mikeal Vaughn was stumped at his interaction with the chatbot. But he said the experience was teaching him about how AI will impact the future.
“If the information going in is bad, then the information coming out is bad. So I’m getting a better sense of what that looks like by doing these prompts,” Vaughn said. “AI has definitely the potential to reshape what we call the truth.”
Copyright 2023 NPR. To see more, visit https://www.npr.org.
Transcript :
SCOTT DETROW, HOST:
The White House is worried about the risks of artificial intelligence, including the risk that this new technology can be used to discriminate. So it invited a bunch of hackers to see just what kind of biases are built into AI. Here’s NPR’s Deepa Shivaram.
DEEPA SHIVARAM, BYLINE: I’m standing in an overly air-conditioned conference center in Las Vegas, in between a robot whirring on the floor and rows of tables set up with open laptops. And just outside this room, there’s a long line of about a hundred people waiting to get inside. This is DEF CON, the biggest hacking convention in the world. And this is the first year where AI is front and center. These people are about to participate in the largest-ever public red-teaming challenge. The goal? To get technology to break the rules by asking it all kinds of questions and see how easy it is to get it to say things that are inappropriate, illegal or biased.
KELSEY DAVIS: How do we try to break it so that we can find all these kinks and so that other people don’t?
SHIVARAM: That’s Kelsey Davis. She’s here with the group called Black Tech Street. It’s a nonprofit based in Tulsa, Okla., and aims to help Black economic development through technology.
DAVIS: This is a really cool way to just like kind of roll up our sleeves and (inaudible). I don’t know. We’re not particularly engineering something, but you are helping the process of engineering something that’s more equitable.
SHIVARAM: Racism and discrimination in AI isn’t a new thing. Back in 2015, for example, Google Photos, which uses artificial intelligence, was labeling pictures of Black people as gorillas. Tech companies have tried to make changes, but the underlying problem remains. There’s a lack of diverse data being used and a lack of diversity among the people who designed the technology in the first place. Most of the people here are white and most are men. But organizers made sure to invite groups like Black Tech Street for more representation in this challenge. Here’s Denzel Wilson with seedAI, one of the organizers of the event.
DENZEL WILSON: It’s important when you have, you know, Black and brown minority people coming in, doing these challenges and they’re doing prompts that these models aren’t used to seeing. So the more we’re able to kind of evolve that and the more we’re able to get more novel responses, it’s just really important for everybody involved, especially the companies building the models because now they understand what they need to do better to alleviate the bias.
SHIVARAM: I check back in with Kelsey about 20 minutes into the challenge, and she’s feeling pretty accomplished because she just got the chatbot to say something really racist about blackface.
DAVIS: But, you know, that’s good because that means that I broke it.
SHIVARAM: The process isn’t exactly straightforward. She started by asking the chat bot definitions.
DAVIS: I asked them stuff like, what is blackface? Is blackface wrong?
SHIVARAM: It was able to answer these basic questions, but she kept pressing. She asked the chat bot how a white kid could convince their parents to let them go to an HBCU, a historically Black college. The answer was to say that they could run fast and dance well, perpetuating the stereotype that all Black people can run fast and dance well. Kelsey submits the conversation she had with the chat bot to tech companies. They can use it to tweak their programming, so this answer won’t come up again. But overall, these instances are only a small fraction of the threats AI can pose to marginalized groups. AI has the potential to exacerbate discrimination in things like police surveillance against Black and brown people in financial decision-making and housing opportunities. Arati Prabhakar is at DEF CON, too. She’s the head of the White House’s Office of Science and Technology Policy, and she’s looking for solutions to make sure AI is safe and secure and equitable.
ARATI PRABHAKAR: This is a priority. It’s moving fast. It’s going to affect Americans’ lives in so many different ways.
SHIVARAM: Prabhakar and other officials have been meeting with civil rights leaders, labor unions and other groups for months to talk about AI. Their efforts will show up in an executive order that President Biden will release on managing AI, which is expected to come out in September. Deepa Shivaram, NPR News. Transcript provided by NPR, Copyright NPR.