Can AI Protect Employee Health? Understanding Remote Workers' Emotional Changes with Facial Expression Analysis AI. How "INNER FACE™" Will Change How People Engage with Business (Part 1)

Since the pandemic, remote work has become widespread, and many companies continue to operate under a hybrid model combining office and remote work. However, in remote work environments, it is often difficult to detect subtle changes in employees, making mental health care a pressing challenge for many organizations.

Amidst this, Dentsu Digital Inc. is collaborating with academia to develop "INNER FACE™," a solution that analyzes employees' facial expressions during remote work using AI. How will this solution impact the way people work?

We spoke with Dentsu Digital Inc.'s Yosuke Mizumachi, Yuki Hata, and Akino Takasaka, who are involved in the project, about the groundbreaking nature of "INNER FACE™". In this first part of the interview, we cover the solution's overview and the background of its development.

Measures facial expressions every second and graphs emotional transitions

Q. First, could you tell us about your roles in this project and your usual responsibilities?

Mizumachi: I serve as the project leader for "INNER FACE™." My regular responsibilities include creative work, planning, and developing advertising communication strategies.

Hata: I'm the development lead. My regular role is as an engineer in the UI Design Group, working on designs for websites and other platforms.

Takasaka: I handle account and PM duties. My usual work involves creative projects, managing website production and operations, and overseeing SNS project management.

Q. Thank you. Now, could you tell us about the background behind launching the "INNER FACE™" project?

Mizumachi: At Dentsu Digital Inc., we continuously run "Social Projects" that aim to solve social issues by leveraging digital technology and creativity. "INNER FACE™" is a project we initiated independently as part of this effort.

About two years ago, when remote work became widespread, I experienced days where I didn't speak to anyone all day. This made me concerned about my own mental health. Having always been interested in psychology, I wanted to understand the mental health state of people in this unique situation, which led me to conceive this project.

Q. So the project originated from your personal experience and interest in psychology. What kind of solution is "INNER FACE™"? Could you explain specifically what it does?

Mizumachi: Simply put, it's a "solution that uses facial expression analysis AI to monitor the mental health status of remote workers." It captures the expressions of employees working remotely via webcams on their computers and quantifies them once per second, enabling objective tracking of emotional changes. Furthermore, by regularly administering mental health care tests, individuals can also monitor their own condition. Emotional trends can be compared by time of day or day of the week. The more measurements are taken, the better the system understands the individual's emotional patterns, increasing its accuracy.

Q. We understand that Mr. Mizumachi's initial concept idea led to the project's development. As a project member, what were your thoughts when you first heard the outline of the plan, Mr. Takasaka?

Takasaka: At the time, the pandemic had reduced opportunities to meet people, and I saw more people around me experiencing physical ailments or mental health issues. I thought "INNER FACE™," which can track emotional shifts from employees' facial expressions, was truly groundbreaking. I decided to join the project immediately after being approached.

Industry-Academia Collaboration Project Revealing the Correlation Between Facial Expressions and Mental Health

Q.I understand you arecurrentlyresearching the correlation between remote workers' facial expressions and mental health,collaborating with psychology and human sciences experts from Fukushima Gakuin University and Waseda University. Could you tell us how this industry-academia collaboration came about?

Mizumachi: "INNER FACE™" aims to detect the mental state of remote workers through changes in their facial expressions. However, clarifying the link between facial expressions and mental health is a sensitive area, and we felt academic perspectives were essential. Therefore, while seeking input from experts, we developed the research framework to determine what kind of data we needed to collect to verify the correlation between facial expressions and mental health.

Q. Were there any technical challenges during development?

Hata: I joined the project midway through. When I took over the work, I was shocked by the sheer volume of communication data. Since we convert facial expression data into numerical values and store it on the server once per second, the communication volume inevitably becomes substantial. Later, there was a phase where university professors would review the data, so we took great care to prevent data corruption due to any issues. Even after the proof-of-concept experiments began, there were program modifications. Since any data problems would be irreversible, we proceeded cautiously to avoid negatively impacting the data at all costs.

Furthermore, since my usual work involves UI design, I believe I was able to leverage that expertise with "INNER FACE™" as well. Even in high-load environments, I aimed for a clear UI using graphs and other elements, while balancing the communication data volume.

Q. What stage is the development currently at?

Mizumachi: We're currently in the research phase. Participants are using the system continuously for one month, providing facial expression data for researchers to analyze. This phase will reveal correlations, such as "people with these facial expression tendencies are more likely to experience mental health impacts."

Additionally, alongside facial expression analysis, we collect subjective questionnaires approximately every two hours to verify whether the analysis results align with the subjects' self-reported feelings. Furthermore, we administer mental health tests every two weeks and are compiling data on the correlations between mental state, facial expressions, and subjective reports.

It is said that the number of people experiencing mental health issues has been increasing since the spread of COVID-19 (※1). Furthermore, surveys indicate that mental health care for remote workers remains underdeveloped, with over 70% of companies reporting challenges in this area (※2). If we could detect the mental state of remote workers and provide care before situations become severe, wouldn't this create positive effects for both companies and employees, enabling more flexible working styles?

In Part 2, we will delve deeper into the response following the announcement of "INNER FACE™" and explore positive applications beyond just predicting mental health issues.

※1 Reference: Tackling the mental health impact of the COVID-19 crisis: An integrated, whole-of-society response (OECD)
※2 Reference: Over 70% find mental care harder for employees working remotely. Many feel stress increased with telework promotion ( Monthly General Affairs)