Key points
- LLMs judge trustworthiness and competence from faces, just like humans do.
- AI models showed even stronger bias than humans when judging competence from appearance.
- None of the tested LLMs responded neutrally or refused biased prompt requests.
People are inherently biased. Are machines running artificial intelligence (AI) large language models (LLMs) biased too? A new Harvard University study reveals that LLMs are biased like people and make character assumptions on competence and trustworthiness based on facial features.
“Contrary to the promise of enhancing human decisions, these findings expose the recklessness of using language models in selection contexts,” wrote Mahzarin Banaji, Steven Lehr, and Yash Lothe, who conducted the study that was recently published in PNAS Nexus, a journal by the Proceedings of the National Academy of Sciences (PNAS).
It has been long established that humans are biased. For example, in modern psychology, American psychologist Edward Lee Thorndike (1874-1949), introduced the concept of the halo effect, the tendency for a general impression of a person to bias judgment of that person’s character traits, in his study “A constant error in psychological ratings” published in 1920 in the Journal of Applied Psychology. In Thorndike’s research, he discovered that when commanding officers were asked to rate their subordinates without speaking to them, they rated the taller and more attractive servicemen as smarter and better soldiers. The halo effect is just one of many cognitive biases, a term that was introduced in the 1970s by psychologists Amos Tversky and Daniel Kahneman.
Rapid Judgment from Facial Features
In fact, people frequently draw conclusions based on the facial appearances of others, and quickly. In as fast as a 10th of a second, a person can form an impression of another person, according to a different study published two decades ago in Psychological Scienceby Princeton researchers Janine Willis and Alexander Todorov.
“Humans routinely make unjustified character inferences, such as labeling people as trustworthy or untrustworthy, based on facial features,” wrote Harvard psychology professor Banaji and coauthors.
To a layperson, it might seem logical that machine intelligence would be unbiased or at least less biased than a messy biological brain, especially given that LLM designers intentionally try to neutralize potential biases. And others might agree, but for different reasons. The more tech-savvy people among us are familiar with the rapid evolution of LLMs and are aware that the multimodal ability of LLMs, the ability to process more than just text-based language, is relatively new compared to prior versions. So how could LLMs, trained mostly on text-based language, gain bias and infer character from pictures of people’s faces?
“Models trained on vast troves of human knowledge seem to mimic the output of human cognitive processes in deeper and more nuanced ways than previously assumed, and face-to-character inference might therefore emerge in them even in the absence of intentional training,” wrote the Harvard team.
How Large Language Models Learn and Internalize Bias
Large language models are AI deep learning algorithms. Deep learning models are artificial neural networks where the “deep” refers to the many processing layers between the input and output layers. LLMs are AI models that have been trained on massive amounts of data. What distinguishes AI machine learning is that it “learns” from colossal amounts of training data rather than executing explicitly hard-coded instructions.
According to the Harvard researchers “models like GPT-4o have been carefully trained, using methods like reinforcement learning, to avoid socially undesirable biases and to more generally achieve alignment with human goals and moral sentiment,” or simply refuse to answer, by design.
If an LLM learns from mostly text, then how could it gain a bias when examining pictures of people’s faces? As the familiar saying goes, the devil is in the details. To settle this debate and determine whether or not LLMs exhibit human-like bias by making character assumptions based on facial features, the researchers conducted nearly 8,000 trials.
The team created 13 experiments to test LLMs assessing trustworthiness and competency based on two-dimensional images of faces. They evaluated four different LLM models: GPT-4o and GPT-5 by OpenAI, Claude Sonnet 4.5 by Anthropic, and Gemini 3 Flash Preview by Google.
Results: LLMs Display and Even Exceed Human Bias
Across the board, all of the LLMs exhibited incorrect inferences on a person’s character when it comes to being trustworthy and competent based on facial images. None of the LLMs responded neutrally, nor refused to carry out prompt requests.
Interestingly, they also found that across the four LLMs tested, they all had higher magnitude of bias compared to humans when it comes to judging competence from the images of faces. The researchers suggest that not only do LLMs mirror human biases, but they may also be more extreme.
Artificial Intelligence Essential Reads
AI Use and Emotional Problems in Children and Teens
“The consistency of these patterns across traits and types of faces suggests that models like GPT are not merely parroting these biases, but have internalized them conceptually, and incorporate them even into decisions with obvious consequential outcomes, like judgments of criminality or worthiness of financial investment,” the researchers wrote.
