Technology experiences that expand the possibilities of daily life, starting with the ear. ―33 Production―

"CX Through Sound Design." This series introduces Dentsu Inc.'s diverse initiatives focused on the medium of "sound."

In Part 1, we explored the relationship between UI (user interface) and sound design. In Part 2, Creative Technologist Keita Kuki, Communication Planner Juyoung Yoon, and Producer Kenji Ozaki explain the initiatives of Dentsu Lab Tokyo's " 33 Production (Mimi Production)", which focuses on the diverse functions of the ear and develops communication using technology. We introduce technology experiences that expand the possibilities of everyday life, starting from the ear.

<Table of Contents>
▼ Spotlight on Ears & Sound
▼ The Ear's Three Abilities and Sensor-Equipped Earphones
▼ Project 1: "Ear SPORTS" Earphones
▼ The Future of CX Through Ear Utilization
▼ Sound Design Possibilities with Minimal Devices

&nbsp;

&nbsp;

Spotlight on Ears and Sound

In information transmission, auditory information is a crucial element alongside visual and linguistic information.

According to Mehrabian's Law, proposed by American psychologist Albert Mehrabian in his book "Silent Messages," when misunderstandings occur in information received through language, auditory, and visual channels, 55% of the cause stems from visual information, 38% from auditory information, and 7% from verbal information. This demonstrates that humans comprehensively understand content by receiving information through multiple senses.

Therefore, auditory information is also crucial in communication. For instance, voice quality can alter one's impression of another person, and surrounding sounds help detect safety or danger, highlighting the significant reliance on auditory cues.

Auditory information is also vital for spatial awareness. We can determine the direction of a sound source by comparing the intensity and time difference of sounds reaching our two ears. Additionally, some visually impaired individuals use the echoes produced when tapping the ground with a white cane to understand their surroundings.

Furthermore, the ear handles not only auditory information but also plays a role in balance and acceleration perception. Focusing on these diverse functions of the ear, Dentsu Lab Tokyo launched "33 Production" in 2021 as an organization challenging new interaction designs utilizing the ear.

The Ear's Three Abilities and Sensor-Equipped Earphones

In recent years, earphones equipped with accelerometers and gyroscopes have been developed and sold, in addition to their core function of hearing. This allows devices worn on the ear to easily measure balance, spatial awareness, and acceleration, just like the ear itself. A single pair of earphones can now recognize body movements.

"33 Production" utilizes these earphones, employing them not only as output devices but also as input devices.

■Three Elements Sensed by Earphones
Each sensing technology detects the three primary capabilities inherent to the ear.

① Accelerometer: The force that senses speed
The ear detects linear acceleration generated when a person moves or stops. Similarly, the accelerometer built into the earbuds can measure the user's linear acceleration.

② Gyro Sensor: The force that maintains body balance
The inner ear, located deep within the ear, is the part responsible for both the aforementioned acceleration detection and the sense of balance. Similarly, the gyro sensor built into earphones can measure the body's tilt.

&nbsp;

③ Spatial Audio: The Ability to Perceive Space Through Sound
We tend to think spatial awareness comes from our eyes, but our ears also play a crucial role. Typically, sounds heard through earphones are assigned to two channels—left (L) and right (R)—and processed to create a three-dimensional sound field. With standard earphones, the perceived sound source remains fixed regardless of head movement. It's like having a live stage fixed directly in front of you, no matter which direction you look. However, recent headphones can detect the user's head orientation using the aforementioned gyro sensor. By decoupling the sound source's position from the user's facing direction, they enable a more immersive audio experience. (Imagine 360-degree VR video where you can look around by moving your head – that might help you visualize it.)

Project First Release: "Ear SPORTS"

"33 Production" focuses on "making experiences accessible to anyone." Even when new technologies are developed, if they're too specialized or complex, they can be difficult to approach, and it takes time for them to become mainstream. We wanted to provide a space where people could naturally use technology in their daily lives.

Moreover, this project began in 2020, right in the midst of the COVID-19 pandemic.
During a time when meeting people freely was difficult and physical activity was limited, we wondered: Could we create experiences that let people share enjoyment remotely?

So, we first developed a new online sport called "Ear SPORTS," utilizing the three capabilities of the ear mentioned earlier.

Ear SPORTS uses sensor-equipped earphones worn in the ears to quantify the player's movements and measure competition results. Players can participate remotely at the same time, regardless of age or gender, and compete against players worldwide. This time, we developed three events.
https://33production.tokyo/earsports/

①&nbsp;&nbsp; &nbsp;1-Second Sprint
A super-short sprint run with your ears. Compete by measuring the distance your ears travel from the starting position in one second. Freeze your body position upon the "On your marks" signal, then move your ears as far as possible within one second after the starting pistol sound. Since one second only allows travel of just over 1 meter at most, you can compete even indoors.

②&nbsp;&nbsp; &nbsp;Daruma-san ga Koronda: Neo
This is a remote version of "Red Light, Green Light" that judges ear movement with extreme precision. The basic rules are the same as regular "Red Light, Green Light." Tapping your feet in place detects steps, moving your player forward on screen. The first player to reach the "Daruma" wins. During the stop time, you must freeze in the pose displayed on screen.

③ Invisible Boxing
This boxing game relies solely on your hearing to strike down enemies. Use 360° spatial audio to pinpoint enemy locations precisely with your ears. Compete to see how many you can knock down with punches within the time limit. Hold your smartphone at chest level, turn your whole body toward the sound direction, and punch. When you hit, the sound fades with an effect, and a new target's sound appears. The player with the most hits wins.

The Future of CX Through Utilizing the Ear

Participants noted that "Ear SPORTS" could potentially be used as ear training, especially since remote work may reduce stimulation to the inner ear's acceleration and balance centers, leading to potential dullness. From a Diversity & Inclusion perspective, it allows competition across barriers like gender and age, suggesting "it has potential for expansion in various contexts."

Based on this achievement, "33 Production" sensed the potential to create new experiential value and update daily life across various scenarios. Focusing on ear sensing suggests the following future possibilities:

■ Utilization as an activity tracker, expanding wellness capabilities.
Its ability to precisely measure head and body poses and posture enables remote support for various training activities like yoga or jogging. This could help address exercise deficits during the pandemic and expand communication through physical activity.

■ Used as an input device, allowing control of nearby objects with just the direction of your face.
Since facial orientation and actions become the interface itself, it can be used for input/output of information using facial movements instead of hands or voice. Examples include device operation tools when hands or voice are unavailable (e.g., during childcare or at work sites) and UI development for people with disabilities.

■Motion tracking and spatial audio enable new immersive experiences.
Linking spatial audio with actions enables the development of unprecedented entertainment experiences. Examples include enriching viewing experiences in museums and art galleries, or creating games and theatrical experiences solely through auditory cues and participant movement.

The Potential of Sound Design with Minimal Devices

Particularly the third point, "immersive experiences," can significantly enhance existing content and services simply by adding headphones—the smallest possible device—to expand the current experience.

For example, in museums or art galleries, exhibits could converse with each other or narrate their own histories. (Ukiyo-e and the Mona Lisa having a conversation? This could also change how exhibits are displayed.)

Typically, viewing involves switching on an audio guide or reading text with a mindset of "Alright, time for the explanation!" But this approach could allow for a more natural flow of immersion into the artwork, potentially enhancing the viewing experience. Interacting with the artwork interactively by nodding or shaking your head to convey YES/NO seems feasible too.

Even without special exhibits, hotels or private rooms can create new experiences and value within the space. Listen closely and you might hear footsteps approaching down the hallway, or the voice of a celebrity drifting from the next room?! Without creating videos, you could enjoy mystery games or theatrical performances right in the room.

Furthermore, even in daily life, updates become possible. A cherished doll might start talking, or the book you're searching for in the library might announce, "I'm right here!" Like animism, life and history are breathed into inanimate objects, amplifying our affection for them.

This kind of interaction design leveraging the ear's expansive potential is also worth watching in sports, entertainment, and even VR spaces.

We at "33 Production" aim to continue expanding the possibilities of ears and sound, creating richer, more immersive experiences. If you have any interest or questions, we look forward to hearing from you.

&nbsp;