In another display that AI can see things that humans inherently can’t, researchers have discovered that AI may be able to identify race from X-ray images, despite there being no clear difference to human experts. Based on X-ray and CT images alone, the AI was able to identify race with around 90 percent accuracy and the scientists are unable to understand just how it can do this. 

The results suggest extreme caution needs to be taken when deploying AI in medical technologies, as even when provided images thought to be anonymous, blurry, or otherwise corrupted, AI may still be able to characterize them. 

The research was published in the Lancet Digital Health.

“When my graduate students showed me some of the results that were in this paper, I actually thought it must be a mistake,” said Marzyeh Ghassemi, MIT assistant professor of electrical engineering and computer science, and coauthor of the paper, in a statement to the Boston Globe. 

“I honestly thought my students were crazy when they told me.” 

To test AI’s ability to identify images, a deep-learning model was first fed CT and X-ray images of many different areas of the body on various groups of people, with each image labeled with the person's race. Then, the tags were removed, along with any possible identifying features – such as skin color, and hair color. For each image, the AI was asked to identify the race of the individual. 

For all areas of the body, the AI was able to identify race with around a 90 percent accuracy, even without any identifying features. 

Covering all bases, the researchers wondered if the AI was being sneaky and using statistics to make the guess based on covariables, such as body-mass index (BMI) or breast density, which could suggest one race over another. The researchers took the possible covariables out and only showed the AI datasets of people with similar BMI and body types, but still the AI was able to identify race. 

How can it do so? Frankly, the researchers are unsure.

This isn’t the first time AI has surprised scientists by identifying race through almost impossible conditions – previous studies have shown it can do so when the image is heavily corrupted or altered. However, the removal of covariables makes the results puzzling. The researchers hypothesize that the AI can identify variations of melanin between the skin of Black and white people, which may show up in CT and X-ray scans and humans have just never noticed. That is just one idea, though, and the true work now begins to understand the results. 

Regardless, the information should spark serious concern about the use of AI in hospitals and how they may react differently to different races, something that has been seen many times in the deployment of AI. 

“We need to take a pause,” said Leo Anthony Celi, co-author and associate professor at Harvard Medical School, to the Boston Globe.  

“We cannot rush bringing the algorithms to hospitals and clinics until we’re sure they’re not making racist decisions or sexist decisions.” 

[H/T: The Boston Globe]