During the PhD I developed a Salsa dance learning VR app. The idea was to simulate the act of dancing with a partner in VR using current available devices on the market (HTC Vive + controllers) and provide a way to analyse the user’s motion to give feedback on the performance.
The approach I took was from the dancer point of view, meaning to simulate what a dancer should experience with a real partner. I chose to focus on the interactive experience itself, more like in a gamification way, rather than on the esthetics and spectator point of view. The result is building the motion from “inside”, as to design a mechanical system, rather than what it should look like from outside in a more abstract way.
During the dance, there are actually two combined motion happenings:
- Moving the body on the rhythm.
- The leader guiding the follower (partner) on what to do next.
Designing our interactive system based on this two concurrent actions proved to be successful.
This shows how the feet are moving related to the ground and the 8 beat of the salsa music. The green foot is the one supporting the weight and the red arrow shows the motion direction.
Graph of the distance of each foot related to a "static" position in time (quantified per beat). It describes the basic "mambo" motion of salsa.
Moving the body was made by moving the center of inertia of the avatar always centered on both feet, as to keep balance. And the feet themselves had to reach the ground when the music tempo reached the unpair beat and go back immediately.
The guidance part was in fact a timed open window (Between beat 7 and 8) in which we expect a certain gesture from the leader (To make a direction change). The gesture is detected based on kinematic threshold. To ensure a more natural posture, both arms of the avatar were in a “relax” position with the wrist being “snapped” to the user’s hand when holding the trigger button.
In this example, "Mambo" is a backward-forward motion and "Cucaracha" is a side motion. To guide the partner to transition between each other requires to transfer the information using the hand and applying pressure. In our VR game, it is a force computed with IK.
For the gamification/learning part, a countdown is given at the beginning of an exercise upon which the music starts and our “partner” starts to move. The objective is to move similarly, in synchronisation with the music, and make the “guidance” gesture multiple times (validated if the partner is responding to this, otherwise it’s a fail). Multiple exercises include variations of music tempo and wrong reaction of the partner. At the end of exercises, the music stops and a score is given depending on the performance of the user (Ability to keep tempo and practice guidance gestures).
Tools
Unity 3D – C# – Inverse kinematics – Avatar 3D – HTC vive
Methodology
Two body interactive system described in terms of controls and information transmission.
Comparative study with Expert dancer and Beginner dancers.
Outcome
Interactive dance app to practice basic salsa movements and guidance gestures.