Beyond solving problems to creating enjoyable experiences. What is this window shopping app for the visually impaired?

There are various services for the visually impaired in the world. However, the reality is that many of them remain focused on "solving problems" like ensuring information accessibility or safety.

Dentsu Inc. and Dentsu Creative X believe that pursuing "enjoyable experiences" is what society should aim for going forward. They are jointly developing the "Listening Window" app, which allows visually impaired individuals to enjoy window shopping.

This time, we welcomed as guests Mr. Ken Hyuga, a blind soccer player, and his wife, Ms. Maika Hyuga, a former blind soccer player, who collaborated on the development. Joined by Shinpei Murata from Dentsu Inc. and Yasuhiko Nishimura from Dentsu Creative X, we discussed the project's background and future prospects.

Ken Hyuga
Blind Soccer Player
Blind since birth. Currently active as a member of the Japanese national blind soccer team while working as a company employee.
 

Maika Hinata
Former Blind Soccer Player
Gradually lost her sight starting in junior high school, eventually developing low vision where she could only make out human silhouettes under certain conditions. Works as a massage therapist in a corporate massage room. Former athlete who competed for both the Japanese national blind soccer team and the Japanese national para rowing team.

 

I want to enjoy window shopping at my own pace

──First, could you briefly introduce yourselves?

Ken Hyuga: My name is Ken Hyuga. I work as a company employee while also competing as a para-athlete for the Japanese national blind soccer team.

Hyuga (Maika): I'm Maika Hyuga. I work as a massage therapist in a corporate massage room. I'm a former athlete who competed for the Japanese national blind soccer team and the Japanese national para rowing team.

Murata: I'm Murata, a copywriter at Dentsu Inc. I'm involved in this project as the Creative Director.

Nishimura: I'm Nishimura from Dentsu Creative X. I usually work as a Technical Director, primarily involved in developing digital content. For this project, I'm responsible for figuring out what technology can bring Mr. Murata's ideas to life and actually building the app.

──Could you briefly explain the concept behind Listening Window?

Murata: Listening Window is an app we prototyped to let visually impaired people enjoy window shopping. When you launch the app and walk with your iPhone hanging around your neck, AI identifies store logos captured by the camera and tells you the store's category and name via voice. If a store catches your interest, tapping the screen gives you detailed info like recommended products, sale info, and reviews from food sites.

Listening Window

 

──What led to its development?

Murata: Recent iPhones feature "LiDAR" (Light Detection and Ranging), a technology that uses laser light to measure distances from the smartphone to objects and recognize their shapes. Combining this with the camera's image recognition technology means iPhone capabilities are rapidly approaching human vision. When I thought about that, it suddenly occurred to me: "Maybe I could create something that would bring joy to people with visual impairments." That's when I started discussions with Mr. Nishimura and the others.

Nishimura: Initially, we explored services enabling visually impaired children to enjoy tag games alongside sighted children. We created several prototype versions and had the Hyuga couple experience them. Through repeated discussions, I recall the keyword "window shopping" emerging.

Hyuga (Ken): Having worked in computer-related fields for a long time, I understood the technical mechanisms of LiDAR functionality. But the idea of applying it specifically for the visually impaired struck me as an interesting endeavor. While experiencing the prototype and sharing our daily lives, we explored other potential use cases.

Hyuga (Mai): I was the one who suggested wanting to window shop (laughs). I've always loved visiting dessert shops and would use Google Maps to find places beforehand. But I couldn't recognize what other shops were around, which was frustrating since Tokyo has so many appealing stores. Another big keyword for me was "I want to window shop alone."

──So you wanted to go alone.

Hyuga (Mai): Yes. Window shopping isn't impossible with sighted people or a guide helper, but you have to plan ahead, so it's hard to just pop out whenever you feel like it. Plus, I really thought window shopping is more fun when you can do it at your own pace.

Not just solving challenges for the visually impaired, but providing added value

──I understand you and your husband experienced the Listening Window prototype. Could you share your honest impressions?

Hyuga (Mai): We tried it at Venus Fort in Odaiba. We discovered shops we hadn't known about when we visited together before, and we kept getting new information like sale details. It felt like our world expanded through what we learned. It was really fun that it wasn't just confined to our own little world.

Ken Hinata: Even with shops we already knew, walking while getting extra information like what specific items they carry made it more interesting than we imagined.

Murata: Thank you. We've been steadily developing this for about a year, so getting this kind of positive feedback that people see value in the app is a huge encouragement.

Nishimura: As Maika mentioned earlier, the idea that "using Google Maps lets you go to any shop you want" was a real eye-opener for us. We tend to focus on solving problems faced by visually impaired people, but through discussions with the Hinata couple, we learned we should instead provide added value that makes everyday life more enjoyable. So hearing feedback like "fun" and "interesting" is truly rewarding.

Murata: Every conversation with the Hinata couple sparked new ideas, and honestly, we often received ideas from them. I think our team members' perspectives were constantly updated from the very beginning.

──What specific opinions or ideas did the Hinata couple share?

Murata: Functionally, they requested not just store information, but also basic details like escalator locations and restroom access.

Hyuga (Ken): That's right. Regardless of feasibility, we shared many ideas about things that would be convenient in our daily lives. We also gave honest feedback after actually using the app and engaged in repeated discussions.

Hyuga (Mai): We also mentioned wanting to know the name of the building we're currently in, since there are times when you think you're at your destination only to find you're somewhere else.

Murata: We learned a lot from the Hyuga couple—things we hadn't realized before, like wandering around without noticing you've entered the wrong building, or ending up in the office area even though you're in the right building. Of course, we haven't been able to implement every request, so we plan to continue prioritizing and refining the app going forward.

Creating a world where visually impaired people can actively enjoy going out

──Listening Window is still in the prototype stage. What do you want to focus on next?

Murata: Ensuring safety for the visually impaired is an area other tech companies focus on. As Dentsu Inc. creatives, I believe our goal should be to shape ideas that make people think, "Wouldn't this be more fun?" or "Wouldn't this bring more smiles?"

So our primary direction is to create enjoyable experiences for people with visual impairments. Currently, Listening Window is limited to indoor commercial facilities, so we want to implement technology enabling outdoor window shopping while adding enjoyable experiences through audio navigation.

Nishimura: Yes, the current prototype is only usable in limited environments, so we want to realize a system that allows people to enjoy both indoor and outdoor window shopping as a seamless experience.

──What possibilities did you, Mr. and Mrs. Hyuga, perceive through this project?

Hyuga (Mai): My husband and I are probably the type of visually impaired people who actively go out. But we know many visually impaired people who find going out difficult or hesitate to do so. We thought it would be great if the app could create opportunities for them to go out. Many people only go to the same places, so if they could go somewhere different, that would truly add more enjoyment to their daily lives.

Also, if more visually impaired people start going out, I hope it will increase opportunities for sighted people to interact with us. I expect more people would be willing to lend a helping hand. That's why I really want to keep supporting this project going forward.

Ken Hyuga: Having a visual impairment means you get less information, so it's harder to find that spark to think, "I want to go to this shop" or "I want to buy this." If the app can provide access to that kind of information through voice, it could spark interest in many more things. I really hope it expands by collaborating with shops and towns.

Nishimura: Regarding Listening Window, we envision a world where users can use it at any store simply by having the app, without stores or facilities needing to install equipment or handle setup and registration. However, adding features like sale information or extra functions requires cooperation from stores and facilities, so we're seeking partners willing to take on this challenge together.

Murata: Exactly. We'd be thrilled if you could reach out to those who share our vision of "aiming for a world where no one is left behind," not just in solving problems but also in creating "fun."

【Contact Information】
info@listening-window.com
 

＜Listening Window Staff List＞

Dentsu
CD: Shinpei Murata
AD: Yui Takashima
Creative Technologist: Tomoya Morohoshi
CW: Kazuya Miyata
Business Producer: Kumiko Midorikawa
 

Dentsu Creative X
Producer: Yu Oga
Producer: Kikuko Ando
Assistant Producer: Kōta Terayama
Technology Director: Yasuhiko Nishimura
Engineer: Eiki Kurokawa
Film Director: Mika Samukawa
 

Cooperation Partners
Ken Hyuga
Maika Hyuga