Can AI protect employee health? Understanding remote workers' emotional shifts with facial expression analysis AI. How will "INNER FACE™" change the relationship between people and business? (Part 2)

Dentsu Digital Inc. is developing " INNER FACE™," an expression analysis AI designed to protect the mental health of remote workers. By capturing facial expressions via a computer's webcam and objectively tracking emotional shifts, INNER FACE™, if implemented, could help mitigate mental health risks for remote workers.

In the second part of our interview with Dentsu Digital Inc.'s Yosuke Mizumachi, Yuki Hata, and Akino Takasaka, involved in the project, we hear about the response to "INNER FACE™" and potential use cases.

Beyond preventive approaches, it can also identify individuals who positively influence their surroundings

Q.The press release forINNER FACE™was announcedin March 2023. What kind of response did you receive?

Mizumachi: For companies implementing remote work, employee mental health care is a major challenge. After the announcement, many companies expressed interest, and several have requested to implement it.

Takasaka: That said, we are still verifying the academic basis for how facial expression analysis can predict depression. At this stage, we are inviting participation in the form of proof-of-concept experiments. Companies actively using the system have requested additional accounts, showing they are utilizing it proactively.

Q. How do you think utilizing "INNER FACE™" will impact the work styles of remote workers? Could you also share some potential use cases?

Mizumachi: Integration with business software that centralizes schedules and emails could further expand possibilities. For instance, quantifying facial expressions during specific tasks or conversations could help gauge individual conditions per project. Beyond detecting mental health issues like "this person looks gloomy during this project," it could also identify positive impacts: "When this person joins meetings, everyone's expressions brighten. They might be a motivational asset."

Q. It's interesting that it can be used not only for preventive approaches like detecting mental states but also for positive applications like identifying talent who positively influence others. For example, can facial expression analysis detect emotional changes even during a single meeting?

Mizumachi: Yes. I've participated in the experiments myself, and even in a single meeting, emotional shifts were quite clearly discernible. In online meetings, you often see situations where the project leader speaks actively, while new hires and junior employees keep their screens off, mute their microphones, and just take notes... In such cases, it's hard for emotions to become positive, and I suspected participants wouldn't find the meeting enjoyable. This feeling is now being substantiated by data. Initially, we saw value in collecting data over the long term. However, by gathering data from a single meeting and generating a meeting score—such as whether everyone participated enthusiastically—we might create opportunities for change.

Q. Can facial expression analysis AI detect positive emotions as well as negative ones?

Mizumachi: We can discern various emotional shifts from facial changes. Expressions fluctuate most dramatically when people feel happiness or surprise. Analyzing what's happening during those moments could reveal employee situations more clearly.

Negative expressions also have various patterns, and "expressionless" faces are often perceived as negative states. We've observed a trend where people with a high ratio of expressionless faces also tend to give more negative responses in surveys. By combining facial analysis data with survey results, we aim to further analyze correlations—what expressions and emotions people display during specific activities.

DX accelerated during the pandemic. Mental health care for remote workers remains a challenge going forward.

Q. What discoveries did you make by participating in the "INNER FACE™" project?

Takasaka: This project made me more conscious of my own expressions. The busier I am, the more overwhelmed I feel by tasks, the more I notice I become expressionless. While still in the proof-of-concept phase, I feel the correlation between facial expressions and mental state is undeniable.

Simultaneously, I realized I was so focused on my own tasks that I wasn't truly seeing those around me. Having joined the company during the pandemic, my work was primarily remote, and I rarely saw my colleagues' faces. I believe "INNER FACE™" holds significant value as a tool for observing our surroundings and supporting each other.

Hata: The engineers developing "INNER FACE™" are members who typically work in creative fields. In that sense, I think it's an ambitious project. Previously, we were in a position of solving assigned problems, but I felt we needed a more proactive approach—considering how we, as engineers, could contribute to creative ideas and branding from our technical perspective.

Mizumachi: This project has made me keenly aware of the difficulty in building a solution. We aim for a solution that visualizes one's entire psychological state and enables more proactive utilization. However, reaching that level requires academic validation and a significant allocation of development resources. While we're streamlining the project team, we will continue to actively push the project forward toward practical implementation.

Q. Based on the proof-of-concept experiment, what are the current challenges as you advance development toward full-scale service launch?

Mizumachi: The primary challenge is establishing high medical reliability to prove that "the emergence of certain facial expression patterns indicates potential mental health impact." Clearing this hurdle would enable us to set baseline values based on data collected by INNER FACE™ and trigger alerts when those values are exceeded, detecting mental distress.

To achieve this, we plan to utilize the vast facial expression analysis data accumulated during the proof-of-concept phase. We will have researchers verify the correlation with mental health, aiming to continue high-precision research with the goal of publishing papers.

Q. Finally, could you share your future goals?

Mizumachi: Even as the pandemic subsides, employee mental health care during remote work will remain essential as hybrid work becomes widespread. We want to serve as a tool for many companies, helping employees check their daily condition and confirm they are maintaining sound mental health. If mental health concerns are detected, the system can notify employees to "consult with the company doctor" or "schedule a 1-on-1 meeting with your manager or HR representative." Granting viewing permissions to managers or project leads could also encourage them to reach out with messages like, "You haven't spoken a word this week—are you okay?"

I've experienced this myself—it's not uncommon to go a whole week without speaking to anyone when working remotely. This is especially true for roles like programmers, engineers, and designers, who tend to become deeply absorbed in their work. Mental health can deteriorate significantly without anyone noticing. We'd be delighted if INNER FACE™ could help support and maintain healthy mental well-being.

Though still in the pilot testing phase, "INNER FACE™" is generating high expectations for mitigating mental health risks among remote workers. Companies that currently "recommend office work because they can't detect subtle changes in employees" may find that practical implementation of "INNER FACE™" enables them to support more diverse work styles.