VR, AR, BMI, IoA... Exploring the New Relationship Between the Brain and Digital Technology ~ Naotaka Fujii, RIKEN

Virtual Reality (VR) is in the spotlight. With diverse devices and technologies advancing, and major internet companies actively investing in VR-related ventures, it's a hot topic in the IT industry. In Japan, Naotaka Fujii, who serves as the representative director of the VR Consortium, is one of the key figures in this field. Interestingly, he is also a scientist with a background in brain and neural research. In recent years, building on VR, he has proposed a new concept called SR (Substitutional Reality), launched his own VR-related business, and maintains an interest in BMI (Brain-machine Interface) and IoA (Internet of Ability). He continues to explore "new relationships between the brain and digital technology" from multiple perspectives. This time, we asked him to speak on a wide range of topics.
(Interviewer: Yuzo Ono, Director of Planning and Promotion, Digital Business Division, Dentsu Digital Inc.)

Virtual Reality Will Change Human Values and Create Happiness for Each Individual

──You began with research on the brain and nervous system, yet your interests span a broad range of digital technologies like VR and BMI. How do you perceive their appeal and potential?

Fujii: Humans have actually been grappling with the same fundamental issues for 20,000 years—nothing has changed. I find this lack of evolution frustrating, boring, even infuriating. However, when technology alters human cognition, it changes how we interact with the world, leading to unprecedented shifts in values. That's where I see the potential for humans, who haven't evolved until now, to finally evolve. For example, VR allows us to freely manipulate a person's world and create new perceptions. BMI can blur the boundaries between oneself and the external world. By skillfully utilizing such technologies that alter human cognition, changing the brain and how we perceive the world, I believe we can achieve a more peaceful and tolerant world.

──Specifically, how can VR change human values?

Fujii: Consider a steak that costs 100,000 yen per slice. Such an expensive dish is typically only accessible to the wealthy. However, with the advancement of technologies like VR, a world might emerge where you can experience "the sensation of eating a 100,000-yen steak" for just 100 yen. Experiences could become cheaper through digitization. This could transform the value that "having more money makes you happier." As a result, the nature of human happiness might change, potentially ushering in an era where each individual can create their own happiness.

──I believe the concept of VR first gained widespread attention in the 1990s, but it then faded into the background before regaining prominence recently. How do you personally view the history and evolution of VR up to this point?

Fujii: In VR , devices called HMDs (Head Mounted Displays), which you wear on your head, are commonly used. Actually, it's been about 50 years since the first such device was created. So, I feel VR's evolution has taken far too long.

But I can understand why it took 50 years to develop the computers that display the content inside those HMDs. Displaying natural images in an HMD requires enormous computational power, which computers back then simply couldn't handle. Recently, however, computer processing power has finally caught up with the technology and concept of VR, making it possible to generate incredibly detailed images in real time. That's precisely why VR is gaining renewed attention now.

──I see. So the evolution of computer processing power is the backdrop to VR's widespread attention. How do you think VR will be utilized going forward?

Fujii: What we currently call VR refers to systems where you wear an HMD, look inside it, and experience non-everyday spaces. It's an interesting system in itself, but it doesn't enable work that couldn't be done otherwise, nor does it dramatically improve convenience. In other words, it lacks real necessity. So, as things stand now, VR risks ending up as a technology that's interesting but whose practical applications are unclear—a technology with no clear path forward. And above all, having to put an HMD on your head and deliberately seek out information is incredibly cumbersome.

Therefore, the key to VR's development lies not in confining it to the world inside an HMD, but in freely exploring applications that aren't tied to specific devices or spaces. VR has far more potential when it blends into real space, creating a sense of digital spaces seamlessly integrating into daily life, rather than relying on HMDs. For example: - Any physical object in the real world becoming an information gateway, allowing contact with the VR world without a device. - Wearing a see-through HMD that doesn't change perception whether on or off, allowing the real space and VR world to overlap. - Simply holding up a smartphone to connect to VR. Essentially, we need to create a system that layers multiple VR overlays onto the real world. When these connect seamlessly, the real world becomes a multi-layered information platform. That's probably the way forward for VR.

──I see. But the current VR hype is focused on HMDs delivering visually stunning content, right? Are there any initiatives right now that don't use HMDs?

Fujii: Almost none. That's because engineers building VR want to plunge viewers into the virtual world. They've gone to the trouble of creating this amazing virtual world, so they want people to immerse themselves completely. They just can't help but want to show off how much they've invested in it (laughs).

But rather than building an amazing world far away, it's far more efficient and useful to build a world that enriches everyday life right here in reality. That's why I want people to think with reality as the starting point. If current HMDs are like building a virtual world on a distant island, then I'm trying to build a virtual world that's contiguous with reality. I believe that in the future, every part of space will become a window to your information, allowing you to customize the optimal space for yourself. Only when we reach a state where we can't tell if what we see is real or not can we truly say VR has evolved.

Hacosco, the cardboard, assemble-it-yourself VR viewer developed by Mr. Fujii. Anyone can easily experience VR by setting their smartphone inside (Click the image to view a video on another site)

VR and AR will eventually evolve into SR, where the digital seamlessly merges with the real world

──While VR stands for "Virtual Reality," Mr. Fujii is pursuing a very unique research area called SR (Substitutional Reality), essentially "Substitute Reality," as an extension of that.

Fujii: SR is a system that blends the real world with pre-recorded footage, making users perceive people or objects that aren't actually present as if they were. First, users wear a dedicated HMD and view live footage through its attached camera. Then, at a specific moment, the live feed is seamlessly replaced with a panoramic video shot previously at the same location. Users, unable to distinguish between the live feed they were watching moments before and the past video, perceive the archived footage as "happening right now." Therefore, I see SR as a system that seamlessly bridges the real and virtual worlds—essentially an evolved form of VR.

──What makes SR fascinating is that it doesn't aim to increase the information density of the virtual space to make it more realistic. Instead, it seeks to erase the boundary between virtual and reality by reducing the information density of the real space.

Fujii: The conventional approach focused on "somehow improving VR quality to make it more realistic." However, a CG-created VR world inherently has less information than reality. No matter how hard you try, the gap between it and reality remains unbridgeable. That's why you can't truly believe VR is real. So, SR took the opposite approach: let's deliberately lower the quality. By reducing the quality of real-world footage down to video levels, we created a world with no gap from reality, without incurring high costs.

──Do you think VR will gradually shift towards an SR-like approach in the future?

Fujii: Yes. In the near future, cameras will become standard features on HMDs. This will usher in an era where "pass-through" – overlaying video footage onto real-world images – becomes commonplace. When that happens, VR, which was previously confined solely to the virtual world, will begin to overlap with the real world, moving closer to SR.

Recently, HTC has developed a device called "Vive" that also enables SR-like usage. This works by using the HMD's built-in camera to trace only the outlines of objects in the real space and display them within the VR world. Even while wearing the HMD, users can recognize obstacles like chairs and tables in the real space, avoiding the risk of tripping or falling. Since adding this camera doesn't require significant cost, and it prevents danger, the trend of adding cameras will naturally become widespread. Once cameras are added, research into what they enable will begin. As a result, I believe SR will become commonplace.

Click the image to view an SR explanation video on another site

──I see. Separate from VR and SR, AR (Augmented Reality) has also been widely discussed for a long time. How should we understand the relationship here?

Fujii: Currently, AR and VR are distinct. But as I envision it, if we overlay VR onto the real world—essentially projecting everything onto reality—it naturally approaches AR. In the future, VR, SR, and AR will likely merge, and terms like VR or AR might even disappear. Digital elements will blend into the real world in various forms, eventually becoming indistinguishable as digital and simply part of the environment.

Right now, I'm working under the premise of popularizing VR, but fundamentally, I see VR as the platform that will enable everyone to experience SR content in the future. Given that VR itself isn't widely known yet, we need to get at least two steps ahead for the world of SR to arrive. That's why we need to popularize VR first.

What kind of world will BMI create where the boundaries between self and others fade?

──I understand you also have a strong interest in BMI (Brain-Machine Interface). Recently, research has advanced significantly, such as implanting devices in the body to read brainwaves and control prosthetic limbs. What do you see as the future potential of BMI?

Fujii: Research will advance, but I have doubts about practical implementation. The truth is, BMI currently has limited practical applications. For controlling prosthetic limbs, information from muscles via electromyography is sufficient—no brain involvement needed. There are also technologies like eye-tracking for text input. Compared to these methods, BMI doesn't offer enough benefit to justify surgery and implanting foreign objects into the brain. Plus, maintaining devices embedded in the body is difficult. Furthermore, if we're serious about practical BMI implementation, we must approach it with the premise of reliably supporting a person's entire lifetime—which is also a major challenge. Ultimately, the current reality is that even developing sophisticated BMI devices lacks practical applications. My view is that VR and SR hold more potential right now. That said, if we consider a long timeframe—say, 50 or 100 years from now—BMI could certainly have potential.

──If BMI develops over the long term, could we see something like today's search functions, where the brain thinks "I want to look this up" and the answer comes directly back to the brain?

Fujii: Searching alone might be possible, but feeding the results back to the brain is difficult. The challenge lies in how to directly convey the semantic information contained in text to the brain—a method no one has figured out yet. On the other hand, I think it's possible that technology for reading information from the brain, like "mind reading," could evolve. If that happens, we might see the emergence of something like a "BMI prosthetic" with far superior performance, far surpassing myoelectric-controlled prosthetics.

──If, in the distant future, our brains connect to digital networks via BMI, and beyond that, many other brains are connected, some argue this could lead to "externalized thought" or "the dissolution of the self-other boundary." What do you think?

Fujii: We won't know until we get there. But I am interested in what happens when the boundary between self and other disappears. I'd definitely like to experience it (laughs). There's a manga by Daijiro Morohoshi called "Biological City." It's about "metal and living organisms fusing together," where people all over the world gradually merge with inorganic matter, saying "it feels amazing" as they dissolve. I imagine it might feel something like that (laughs).

──I also hear discussions about artificial intelligence eventually gaining consciousness. But that's still just judging by outward appearances like behavior, like the classic "Turing Test." I don't think anyone can truly know if there's anything inside that can genuinely be called consciousness.

Fujii: I think it's entirely possible for us to perceive personality in the behavior of an artificial system created by humans. But whether such AI truly possesses consciousness is a difficult question; it might ultimately be a philosophical zombie. However, even if it's just the outward appearance, if we accept it as OK, then for humans, it's something with consciousness. For example, dogs and cats don't respond when you talk to them, sometimes ignore you, and occasionally roll over and rub against you. Yet we never doubt their consciousness. It's similar with artificial entities. If we start agonizing over whether "this is truly consciousness," there's no end to it. If we can create artificial things we can live with normally, isn't that sufficient?

As more people get "taken over" by IoA,
society becomes more convenient and tolerant

—Professor Junichi Koyama of the University of Tokyo is also drawing attention with his concept of IoA (Internet of Abilities).

Fujii: IoA refers to human consciousness being shared through networks. It means you're here, but you might wonder, "Wait, whose ears are hearing this sound right now?" You don't know whose senses you've acquired, or who's controlling you—it's a sense of time and space becoming ambiguous.

In other words, with IoA, people get "taken over." The expertise of a distant specialist transfers to someone else who then acts. Consider the example of a doctor. Suppose there's an expert in extremely difficult surgeries. But if that expert has to handle all the routine tasks like suturing—things anyone can do—instead of just the critical, complex parts, their valuable expertise gets wasted on trivial tasks. The expert only needs to work at critical points, so they can just take over the surgeon when needed. Conversely, the person being taken over doesn't need to know anything. This is the essence of IoA: dividing an expert's time and space and distributing it across various locations.

──Is there potential for such IoA to connect with artificial intelligence in the future?

Fujii: It will likely connect. In the future, artificial systems may possess higher skills. Systems that embody expert knowledge could emerge, potentially performing more advanced tasks than less skilled individuals. Consequently, scenarios may arise where acting while having one's body "taken over" by such an artificial system yields better results. Take driving, for example. An AI could dictate the exact timing for braking, accelerating, and shifting gears. Even if you don't fully understand why, following its instructions could result in world-class maneuvers. This is essentially AI educating humans. With repeated practice, these actions become ingrained and turn into the person's own skill.

Eventually, people might just become "taken over." For instance, a young resident just starting on the ward could be taken over by an AI possessing world-class suturing skills, and that AI would just keep suturing with incredible technique. Then, when that resident moves to another duty, the now-unoccupied AI could go to another resident and provide its skills there. In this way, a society where AI and humans coexist worldwide might emerge.

──But if being "taken over" means "not knowing who's in control," doesn't that feel a bit unsettling?

Fujii: But it's like this: no matter who grows the vegetables, they're still vegetables, right? It's the same principle. "Who grew these delicious vegetables?" "Oh, they're from Mr. Koyomimoto? No wonder they're so tasty!" Similarly, it doesn't matter if the skills or expertise being taken over are anonymous. The person being taken over doesn't need to worry about that; they just need to do their job well.

──It's like saying, "Thanks for always driving my car so well, even though I don't know who you are" (laughs). By the way, if we start sharing sensory information and experiences through networks like IoA, could a kind of shared consciousness begin to emerge among people?

Fujii: It's possible. Most things look different depending on your perspective. But normally, you can only experience reality once, so if you're seeing it from this angle, you can only imagine what it looks like from the other. From now on, though, we'll probably be able to see the same event from multiple angles. Then it becomes, "What do you think when you see it from that perspective? Is your view different from before?" As a result, we'll have a society built on the premise that things are multifaceted. Since how the world is perceived is ultimately just each person's subjective experience, if society operates on the premise that there are as many facts as there are people, the meaning of judging things will change, and how we treat people with different opinions will change too.

──By sharing perceptions, it becomes easier to understand, "Ah, so this is how the world looks from your perspective." In that sense, IoA has the potential to create a society that's more accommodating or tolerant toward people from diverse standpoints.

Fujii: If that works well, it could lead to peace or tolerance. I don't know if it's achievable, but the potential is there.
Ultimately, what I aim for is building a better society. Through technologies like VR, SR, BMI, and IoA, I hope we can change the brain, alter how we perceive the world, and foster better development for people and society.