To celebrate the year of achievement of women in science, the HNF museum in Paderborn, Germany collaborated with MIRALab at the university of Geneva to produce an interactive installation that involves Lady Ada Lovelace, one of the first scientist lady from the 1800s.
I was missioned to design and implement one part of the exhibition that includes a virtual avatar of the character itself interacting with the public. My setup idea was to have a 6m long dark tunnel with a projected screen on the side. The public would walk through the tunnel and see the avatar of Lady Ada. Using a dark tunnel presents the advantages of having a defined, controlled space and blending more easily the projected scene with surrounding light.
My idea on how to detect people was to exploit the depth sensor of the Kinect that would be placed at the ceiling of the tunnel, looking downwards, to detect the presence and position of people in the interactive area.
To realize that, two Kinects were needed to cover the width of the tunnel. I developed a real time processing code that merges the two Kinect data as one big depth image and extracts the height of the participants. From there we can also deduce the center point of people or multiple people close together. These high level information (height, position and presence) were sent in real-time (60Hz) to the Unity application that drives the avatar interaction.
Portrait of Ada Lovelace by the British painter Margaret Sarah Carpenter (1836).
Example of the depth data acquired by the Kinect. We can then isolate the part that is closest to the device (whitest grey).
Here we have isolated the closest object to the device (from a defined threshold). We can then process the data to extract its center point coordinates.
For the interactive part, an avatar was back projected in a semi-transparent curtain that would fake the depth perception of people, making them think there is a big space besides them, in which Lady Ada is moving around and approaching them from time to time.
I used behavior trees approach to manage the avatar motion, all rendered in Unity. The 3D scene allows the virtual avatar to move to 3 or 4 “stations” (point in space) in which there is an action (Animation like salute).
At all times, the application receives updates from the detection part and changes its state accordingly. To ensure natural looking motion, the avatar would still finish its walking or action before doing anything else (Like reacting to people).
The avatar of Lady Ada was made by a team member, from the different pictures available and technology we had. I rigged the avatar to suit the motion and ensure smooth transitions.
The finished product was tested and designed to be robust as it was played continuously during 12 months, every day for 8h. The overall public passing through was not measured but I estimated 20,000+ people.
Resulting installation in the museum. The presence of children unlocks the avatar coming and saying hello.
Tools
Unity – C# – OpenCV – C++ – Kinect SDK
Methodology
Real time programming – Robust design – Interactive loop design
Outcome
An interactive installation that shows a virtual Lady Lovelace reacting to people presence. The application ran for 12 months and had 20,000+ users.