A camera that records emotions using biological information: neurocam

This time, we spoke with Kana Nakano, who handled planning and direction for neurowear's new device neurocam, and Yuta Takeuchi, who handled prototyping and technical direction.

neurocam: A Camera That Records Your Emotions

──neurowear's new product, neurocam, has been announced.

Nakano: Yes . neurocam is the system name for a camera that uses biological information to record emotions. The sensor used in the prototype is NeuroSky's EEG sensor, which has been mentioned frequently in this series, and it uses the same latest chip as mico.

──So, Professor Yasue Mitsukura from Keio University's...

Nakano: That's right . Like mico, it was prototyped using Professor Mitsukura's algorithm. We previously discussed how communication might evolve from person-to-person to machine-to-person or content-to-person. neurocam started from the idea that it could enable person-to-city communication.

──That's why it became a camera.

Nakano: At the end of the 20th century , cameras went digital, and shooting information and timestamps started being recorded on photos. Then, in the 21st century, cameras became standard on mobile phones, and embedding location data in photos became commonplace. Before that, no one ever imagined photos would have location data. So, what could be the next thing embedded in photos? We hypothesized that emotion tags might be possible. That led to neurocam, a prototype camera that records feelings.

──So it uses a smartphone.

Nakano: Using a smartphone , which already has both the camera and the analysis capabilities, was the fastest way to prototype, so we settled on this form factor. When you actually wear it, your "level of interest" appears on the smartphone screen positioned on your temple, making it feel like you're peeking inside your own mind.

──Level of interest?

Nakano: Yes . We named the result of multiplying Professor Mitsukura's "liking level" and "interest level" algorithms the "interest level." It's measured every second and displayed in real-time as a value between 0 and 100. When it reaches 60 or above, recording starts automatically, and the recorded footage is saved as a GIF video.

──So it's a short video lasting just a few seconds?

Nakano: We call them micro-videos . They're becoming popular on apps like Vine and Instagram. Neurocam clips these short videos as personal memories that include your feelings. Currently, it saves based on the "Interest Level," but our ideal future is for Neurocam to increase an individual's total happiness. Imagine being able to look back only at scenes that made you happy or things you loved when reflecting on your day. Don't you think that alone would change how you feel about your daily life? The concept movie depicts a day in the life of a girl in that slightly future world.

(neurowear "neurocam" concept movie (A wearable camera that records scenes that catch your attention) http://youtu.be/CDgkX-JY_wM )

 

Prototyping with many manual processes

──When did you start making this prototype?

Nakano: Around June 2013 . After Takeuchi joined.

──How did Takeuchi-san get involved?

Takeuchi: I handled the implementation of the initial prototypes and the technical direction for the app. Before joining the company, I was doing engineering research. I enjoyed building things, whether hardware or software, so I got involved in the prototyping.

──By the way, what kind of research did you do at university?

Takeuchi: It was what you'd call haptic transmission technology. You can't understand texture without touching it, but I wondered if it could be conveyed to someone in a remote location. My master's thesis research involved recording, reproducing, and searching for tactile sensations. For example, if you could link Amazon product images with their tactile sensations, you could check how something feels before buying it.

──Did you encounter any challenges during this prototyping process?

Takeuchi: We had to rebuild the prototype many times, but we don't have much equipment in-house...

──What kind of equipment do you need, for example?

Takeuchi: If we had a 3D printer or laser cutter, we could model, print, and finish with sanding—that's the ideal workflow. Instead, we often had to go through this analog process: building the shape from scratch with putty, sanding it, painting it, and repeating. That part was a bit tough (laughs). But we got it done, so all's well that ends well.

Nakano: Let's use this as an excuse to ask for some equipment (laughs). Thanks to Takeuchi-san making the first and second prototypes, we could do the basic experiments and think, "Okay, this seems like it'll work!" That really gave the project momentum.

Photo of the first test unit. Initially, it was separate from the smartphone, but miniaturization proved difficult, so we made it an integrated unit.

 

At first, I wanted to make a camera that makes cheeks turn red

Nakano: Actually, at first , I wanted to make a camera that makes cheeks turn red.

──Cheeks?

Nakano: Yes . When developing mico, I had the chance to see Professor Mitsukura's various algorithms and learned they could measure "likes" and "interest." Also, video camera recording lights are usually red, right? So I thought it would be great to have a camera that automatically records when you see something you like or find interesting, making your cheeks turn red. If such a camera existed, it would feel like a state where machines and humans are integrated—half cyborg, half human—and I thought it could be a possible future.

So, I tried an experiment attaching red LEDs under glasses to make cheeks redden, but it ended up making me look like a drunkard (laughs), and it was just too unattractive, so I gave up. In neurocam, I only kept the essence as a recording indicator—it turns red when the numerical value reaches a threshold.

──So the reddening effect was achieved?

Nakano: Yes, that's right . I consider turning red a crucial point. When Google Glass came out, there was debate about privacy issues—people not knowing when photos were being taken. It's a major problem, but I felt it might be solved simply by making it obvious to others when they're being filmed. If the camera itself turns red during recording, that's clear. Having others point out that it's red could build up a sense of "embarrassment," leading the brain to learn. I think it could even change the brain circuits themselves—whether you care or not, whether you take photos or not.

Until now, software, hardware, and the operator have existed independently. I believe we're moving towards a future where "information from the operator's brain and body integrates with the system." The first step was necomimi, the second step was mico, and the third step is probably this neurocam.

──Listening to you, Mr. Nakano, there seems to be a concept of wanting to make people feel embarrassed, like "leaking your feelings" or "your cheeks turning red."

Nakano: That's right . Now that you mention it, it might be an underlying theme. When considering communication, I think the emotion of "embarrassment" is indispensable. It's an emotion that requires the presence of others, and its scope and threshold vary across cultures.

Takeuchi: In the lab, we often built things through logical deduction. Having a concept first and then building toward it is a process we rarely do. It's really interesting.

Nakano: I'm glad you find it interesting (laughs). Since I don't have a science background myself, borrowing Arthur C. Clarke's phrase, "Any sufficiently advanced technology is indistinguishable from magic," I think I'm approaching things from the magic side. I grew up on science fiction, juvenile fiction, myths, and fantasy novels.

 

In San Francisco, MIT Media Lab Director Joi Ito...

──The neurocam was first exhibited at "Human SENSing 2013" held in Yokohama from October 23rd to 25th, right?

Nakano: Yes . At "Human SENSing 2013," we conducted an experiment where participants wearing neurocam watched a slideshow to see what images appeared. We surveyed attendees on-site, and 90% found it interesting, while 68% reported feeling convinced. While we still need to refine the accuracy, it was quite a debut.

("neurocam" first demo at Human Sensing Japan 2013 http://youtu.be/7sHoetvYMH0 )

──Next...

Nakano: We held a demo in San Francisco in early November . Digital Garage, which also invests in Dentsu ScienceJam Inc., opened a wonderful incubation center called "DG717" ( http://www.garage.co.jp/ja/dg717/ ) in San Francisco, and this was their opening event. It was very well received locally too. Joichi Ito, Director of the MIT Media Lab, also tried it out. He must have been hungry, because images of food were recorded (laughs).

MIT Media Lab Director Joi Ito trying out neurocam

 

──What direction do you see neurocam taking in the future?

Nakano: For Phase 1 , we envision B2B applications. For example, in-store surveys. Since neurocam can also record location data, we believe it could be useful for urban planning surveys and similar research. Commercial availability is still a bit further off, so we hope it will first be utilized in such research and survey contexts.