MotionScore: Technology that Transcribes Human Movement into Musical Notation

What does "notating human movement" mean?

Currently, motion data used to control CG characters and robots is managed in frames (short units of time like 1/60th of a second), similar to video data.
However, consider dance: real human movement is conceived from tempo, or beat.
Indeed, when idols practice dance, trainers always call out "One, two, three, four!"—no one ever says, "Okay, lift your leg between 3 and 4 seconds!"

This technology focuses on this very point.
When animating CG characters or robots, what was previously described in data as "extend the right hand forward from 3 seconds to 3.5 seconds" is now changed to something like "extend the right hand forward from the second beat to the third beat." In other words, it manages human movement based on beats.

Now, it turns out that a data format using beats already existed in the digital world.
This is the MIDI data format, used for musical scores.
While most modern music data (like mp3) records music as "the shape of sound waves," MIDI data, like sheet music, encodes music as a "collection of sounds changing according to tempo."

Human movement, when broken down to its essence, is the changing angles of each joint. By encoding human movement as a "collection of joint angles changing in time with a tempo," MotionScore has succeeded in recording human movement in a form extremely close to MIDI. (Patent pending for this technology) We call this process "notation."

What does using "MotionScore" do?

By focusing on the unit of the beat, "MotionScore" successfully reduces the volume of motion data. Furthermore, even if the beat changes, the motion automatically synchronizes.
Furthermore, leveraging its MIDI format enables real-time arrangement and motion synthesis.

This makes it easy to achieve performances such as live music paired with CG dance videos or robot collaborations, or even dance jockey-style performances where dance moves are arranged and played back on the spot.

The Future of "MotionScore"

"MotionScore" will make reusing motion data simple. Looking back at the history of DTM (Desktop Music), where existing sounds were edited and combined to create new music, "MotionScore" might give rise to an entirely new type of dance choreography.

Furthermore, beats are hidden not only in dance but also in everyday motions like walking and running. Combining "MotionScore" with human augmentation (technology that extends human capabilities through machines) could enable efficient learning of walking and running techniques by experiencing the ideal movement for each beat. It also holds potential for rehabilitation in medical settings.

 

"MotionScore" Gains Global Attention

Selected as a finalist in the SXSW ReleaseIt category—the first Japanese company to achieve this.
"MotionScore" was selected as a finalist in the international competition "SXSW ReleaseIt," held at the world's largest interactive festival "SXSW." "SXSW ReleaseIt" is a globally influential competition, and reaching the finalist stage marks the first such achievement for a Japanese company.

Joint Demonstration with Yamaha at SXSW Trade Show
A demonstration of "MotionScore" will be held at the "SXSW Trade Show" in Austin, Texas, USA, from March 13 to 16. The demo will capture beats from live drum performances and have CG models dance in sync with those beats. Yamaha's " Audio Beat Sync " technology will be used for beat capture during this demo. This marks the world's first attempt to generate dance motions synchronized with live drum performances that change speed in real time.

Incorporated into the Next-Generation "GO-DJ Plus"
The "MotionScore" technology has been adopted for the "GO-DJ Plus," which will be launched on the crowdfunding site "Indiegogo" in mid-March. The "GO-DJ Plus" is the next-generation model of the globally hit product "GO-DJ." By incorporating "MotionScore," the "GO-DJ Plus" can generate dance motions in real time that match the DJ's performance, potentially sparking a new movement on the dance floor.

The GO-DJ Plus is also available on the crowdfunding site Makuake.

What is Audio Beat Sync?

This is Yamaha's technology that analyzes musical instrument sounds in real time to detect tempo and beats. It can analyze not only audio sources but also live performances.
Utilizing Audio Beat Sync enables various beat-synchronized entertainment features with ease, such as automatically adding accompaniment that matches the audio's beat or automatically adjusting delay length to match the tempo.
This technology is also featured in Yamaha's latest synthesizer, the "MONTAGE".

---

MotionScore Staff List

Executive Creative Director: Yuki Kishi (CDC)

Executive Producer: Akio Iijima (CDC)

Technical Director: Yasuo Higa (ISID)

Director: Tatsuki Tada (CDC)