Novel Study Explores the Useability and Efficacy of AI Translations for Diverse Populations

Researchers in the Institute for Trustworthy AI in Law & Society (TRAILS) are working with a local interactive museum to explore how people from different backgrounds rely on imperfect language translations generated by artificial intelligence (AI).

The project, known as “Lost in Translation,” is part of a series of ongoing research efforts coordinated by the Language Science Station at Planet Word, an immersive venue in downtown Washington, D.C., that is focused on curating a love of language.

It joins University of Maryland computational linguists and data scientists with other researchers to better understand people’s use of—and level of trust in—AI tools used for language translations.

The project is led by Marine Carpuat, an associate professor of computer science, and Ge Gao, an assistant professor in the College of Information.

Both Carpuat and Gao are active in TRAILS, noting that the opportunity to gather real-time data at Planet Word falls directly in line with several TRAILS research thrusts that include creating technical solutions that build trust in AI, and empowering users to better understand AI systems.

“Being able to do this at Planet Word is really great because we have a huge diversity of people to interact with,” says Carpuat.

For the Lost in Translation study, participants are given a digital tablet that displays a virtual avatar of Sophia, a Spanish-only speaking visitor to Planet Word who has lost her scarf. Participants are tasked with relying on AI translation-generated options—four options are given for each sequence—to communicate with Sophia and help her to find the missing scarf.

The UMD study is measuring not only if the correct choices are being selected, but the level of confidence participants have in their selections.

The researchers expect to gather robust data from their interactive exhibit, given that people of all ages are participating, some of whom are native speakers of Spanish, with other participants being bilingual, and still others with very little knowledge of the language.

The entire process takes only a few minutes to complete and is designed to feel more like a game than a scientific study, says Charlotte Vaughn, an assistant research professor at UMD who is also active in TRAILS.

Vaughn is leading the Language Science Station, a National Science Foundation-funded effort that develops partnerships between Planet Word and language researchers.

“One of the advantages of our work with Planet Word is that visitors don’t just learn more about language while at the museum—visitors also get to contribute to ongoing research that is creating new knowledge,” says Vaughn.

Ashley Chau, a UMD undergraduate working as a research assistant for the Language Science Station, says the Lost in Translation project is different than other research experiences she’s had.

“You’re able to work with people directly and see the impact of the work you’re doing,” Chau says.

Carpuat says she has heard similar feedback from the five Ph.D. students—Yimin Xiao, Calvin Bao, Zoey Ki, Yongle Zhang, and Marianna Martindale—that she and Gao are working with on the project.

The UMD researchers say their detailed conversations with the study’s participants include feedback and data from doctors from Portugal, journalists from Argentina and more. It’s these types of interactions that make Planet Word an ideal place for the study, according to Carpuat.

“Planet Word is where we can reach a much more diverse range of people than if we were just using crowdsourcing or asking people to participate on the internet,” she says. “It’s just a great match between our ongoing research interests in AI and machine learning and other work we are doing in TRAILS.”

—Story by Shaun Chornobroff, UMIACS communications group

***
The Language Science Station will be running the Lost in Translation study for the next several months. Find them at Planet Word most Thursdays and Saturdays in July and August. Check their social media pages for an up-to-date schedule starting in the fall.