Virginia Tech® home

‘Don’t You Dare Judge Me’: Judgments by AI Tools and their Impact on Minorities

Researchers will study the impact of  judgmental or “Judgy AI” on vulnerable populations such as minorities and the underprivileged. Understanding of the effects of Judgy AI systems can have design, regulatory, and  legal ramifications for the appropriate use of such tools in business.

Funded by the CCI Southwest Virginia Node

Project Investigators

Rationale and Background

Researchers use the term Judgy AI to describe AI systems that evaluate a personal  aspect of a person (e.g., looks, competence), providing feedback that can be used as input into decisions that might have an impact on behaviors. 

For example, a  Judgy AI system might evaluate a person’s skin (e.g., wrinkles) as input into what skin care product someone should purchase or might evaluate a person’s personality (e.g., extraverted) as input into whether that person is the right fit for a job. 

These systems use unstructured data (e.g., photos, audio) to provide results that categorize them based on some physical feature or trait. Little is known about how people respond to algorithmic  feedback, especially in cases of evaluative feedback on highly personal characteristics.

Such judgments from sophisticated AI tools could also exacerbate a troubling phenomenon of young people falling prey to “toxic beauty standards” and purchasing “preventive treatments.”  

Methodology

Researchers will examine both cognitive and  emotional trust, as well as how types of algorithmic transparency can influence trust in AI. They will:

  • Develop a Judgy AI system to provide feedback on the severity of a person’s facial wrinkles, an application of AI that cosmetic companies are using to sell skin care products.
  • Examine the emotional and cognitive responses to both the algorithm and the feedback and study how such a Judgy AI intervention can result in purchases for people with different traits. 

Projected Outcomes

The project’s focus extends to the impact of Judgy AI on vulnerable populations such as minorities and the underprivileged. 

Results will help researchers understand:

  • The effectiveness of Judgy AI as a sales tool, and if it can lead to undesirable spending, especially among those with low self-esteem and minorities by making people feel poorly about themselves. 
  • What types of interventions might help companies use Judgy AI in limiting the potential damage of the tool while preserving the benefits in  customized service that companies hope such tools might be able to provide.