Phantom Snack / Feels like you're eating even when you're not

SXSW 2023 (South by Southwest 2023), the world's largest festival of technology, music, and film gathering companies and creators from various industries, was held in March in Texas, USA. Dentsu Inc. and Dentsu Creative X participated in the Interactive division's "Creative Industries Expo". They exhibited three prototype works showcasing cutting-edge experiences created under the concept "Unnamed Sensations." This three-part report explores the expansion of senses like "touch," "texture," and "smell" made possible by the fusion of cutting-edge technology and creativity.

Part 1: The new chewing experience system, " Phantom Snack." What is the sensation of "eating without actually eating"? We asked Ko Izumi of Dentsu Inc., representing the project team.

 

The Body Craves Texture

──What is Phantom Snack?

Phantom Snack is a completely new guilt-free chewing experience. You can enjoy the sensation of crunching and munching, along with flavors, without putting anything in your mouth. Since you're not eating anything, you consume zero calories. In fact, the more you chew, the more calories you burn through exercise.

The idea came from personal experience. I often find myself snacking late at night, but I realized it wasn't so much about hunger as it was about wanting to fill the emptiness in my mouth. If what the body craves isn't calories but texture, I thought, finding a way to satisfy that craving without consuming calories could ease the minds of many people facing the same struggle.

I came up with the concept about 20 years ago, but couldn't get started on development. I thought if someone else came up with the same idea and made it, I'd buy it, but it never appeared. Then, when an internal development opportunity opened up, I applied and made it happen.

Phantom Snack booth. Austin Convention Center, Texas, USA, March 12-15 (local time)

──What kind of team did you assemble for SXSW?

We assembled a team with staff from Dentsu Inc., Dentsu Creative X, Dentsu Digital Inc., and partner companies. This included a Creative Director handling planning/coding/hardware production/development production; a Technologist responsible for naming/planning/visual engineering; and an Art Director overseeing the overall booth experience design/logo/animation. Additionally, we had personnel responsible for AI development for image recognition, development and exhibition production, motion graphics production, audio production, signage art, and explanatory video direction. Furthermore, we received academic information support from research institutions specializing in chewing and the brain.

All three staff members handling visitor interactions at the on-site booth were first-time SXSW attendees and not fluent in English. While the first day of the four-day exhibition brought anxiety, they gained experience quickly through repeated interactions. They reflected on and improved their approach daily. As a result, by the final public viewing period, they were handling customer service quite smoothly.

Facial image recognition linked with bone conduction earphones

──What technology enabled this new chewing experience?

The basic mechanism is this: when you perform the action of chewing food without anything in your mouth, facial image recognition detects your jaw movement. In sync with that, the bone conduction earphones provide feedback in the form of crunching chewing sounds and vibrations. Furthermore, by linking this with visuals and an aroma diffuser, you get a chewing experience that feels just like you're actually eating a snack. At our exhibition booth, we prepared various "chewing experiences" including chocolate chip cookies, potato chips, salad, and even never-before-tasted, otherworldly sweets.

Chewing sounds and vibrations are fed back through bone conduction earphones in sync with jaw movements detected by facial image recognition. Ms. Izumi (left) interacts with visitors.

──Were there any challenges in realizing this idea?

Even if it worked in theory, we couldn't be certain without actually trying it. There were unknown factors like the required man-hours, so we started prototyping before the internal proposal. This involved detecting the 'munching' motion of moving the jaw while keeping the mouth closed. It required ingenuity and trial and error. While the initial calculation principle was relatively simple, coding to process facial feature points obtained from image recognition into a computable format proved extremely challenging. There were almost no precedents to reference, making it a tough development.

Once we had a working model, we migrated the system from rule-based to machine learning to improve accuracy. To succeed at SXSW, it needed to handle facial and jaw movements across diverse ethnicities. We enlisted the cooperation of many Dentsu Group members as data samples.

We also focused heavily on sound and motion graphics to create a pleasant experience. This is our area of expertise. By collaborating with professionals with proven track records in advertising creative production, we believe we achieved a high level of entertainment quality.

The sound and motion graphics synchronize with the participant's jaw movements, allowing them to visually experience the texture.

──How was the response locally?

While it was well-received by our domestic team and stakeholders from the development stage, we were unsure if the experience itself would be accepted overseas. However, I think people really enjoyed it, partly due to its novelty.

Many people laughed and called it crazy during the experience. Those who identified as "kindred spirits" with me—people who tend to overeat snacks—said they wanted one immediately. We also got reactions from people tackling sustainability challenges and those researching related fields, expanding our network.

Our booth consistently had lines, and over 1,000 people actually experienced Phantom Snack. We felt we truly met the expectations of those who came to SXSW seeking unknown experiences.

Above all, having so many people experience something we developed and designed ourselves, and seeing their reactions firsthand, was truly a precious and happy experience.

From facial exercises to dementia prevention

──Tell us about the potential ofPhantom Snack.

Chewing itself has been reported to have various positive effects on both physical and mental health. It's said to enhance concentration and reduce stress. Chewing is also expected to be effective in preventing dementia. Further research is needed to determine if Phantom Snack can provide similar chewing benefits. We're eager to collaborate with university labs and others to develop this technology, recognizing its potential to make a significant contribution to society in this field. While not yet tested, it might be possible to add texture to liquid diets, making meals more enjoyable for those undergoing treatment.

On the other hand, we believe utilizing this simulated chewing for entertainment purposes or as a calorie-suppression tool is feasible immediately. Phantom Snack alone can be developed into an enjoyable next-generation indulgence. When used while actually eating food, it could amplify crispy textures without using oil, for example. Combined with VR, it could build healthy eating entertainment in virtual spaces. Existing VR goggles don't cover the mouth, making them suitable for image recognition. The chewing motion could serve as a controller for fitness games that double as facial exercises, and numerous other use cases are conceivable.

I've heard that the number of chews per bite among Japanese people has been steadily declining since recorded history. It would be fantastic if Phantom Snack could contribute to restoring humanity's chewing frequency and promoting health.

 

──A word for those planning to exhibit or attend SXSW in the future.

We exhibited at the Creative Industries Expo, and this year, the XR Experience exhibits particularly stood out. For those planning to attend, I recommend exploring the entire SXSW to catch and experience the trends firsthand.

For exhibitors, I recommend placing as much emphasis on rehearsing the final experience as you would in regular service development, and repeating integration testing and improvements several times. Phantom Snack faced significant development delays, but we managed to squeeze in several rounds of improvements and fixes. I believe this led to a good outcome.

It's incredibly exciting to directly deliver surprises to visitors who come expecting new experiences. I truly hope many creators will experience exhibiting.

 

Staff List:
Creative Director: Ko Izumi (Dentsu Inc.)
Creative Technologist: Ryunosuke Ono (Dentsu Inc.)
Art Director: Hanako Yanagi (Dentsu Inc.)
AI Engineer: Hideki Murata (Dentsu Digital Inc.)
Producers: Takuji Okamoto, Wataru Tamesue (Dentsu Inc. Creative X)
Production Manager: Kiyonari Oshida (Dentsu Inc. Creative X)
Explanatory Video Director: Ryo Matsumoto (Dentsu Inc. Creative X)
Sound Design: Masayuki Noda, Tsuyoshi Odabe (Invis)
Motion Graphics: Arima Shinnosuke (EDP graphic works)
Cooperation: Kinya Kubo (Chewing and Brain Research Institute)

 

Website:
https://phantomsnack.com/jp