Storylab was a collaborative project with industry partner Ultraleap. The brief challenged the group to create a VR experience to engage people with, and demonstrate the affordances of the Ultrahaptics STRATOS Explore Development Kit, a haptics kit. It was 2020, in the middle of COVID, and yet we were very lucky to have the chance to experiment with mid-air haptics and include those in our shadow puppetry VR experience. The STRATOS Explore tracks users’ hands using a Leap Motion Controller attached to the front of the unit, and projects tactile effects onto them using ultrasound.
Working at home, since we were on lockdown, I was sent the Development Kit to experiment with as we designed our group project.
Our group considered of 4 of us from many different backgrounds, with a variety of different skill-levels. There are many very visual Trello boards we put together researching, conceptualising, and storyboarding our project, the Art of Shadows.
Synposis
In the Art of Shadows, you, the player, enter a VR theatre where you meet a PuppetMaster called Sam who I had the create fortune of designing, animating, and voicing using Reallusion and Unity. Sam teaches you how to do hand puppetry, a variety of silhouetted hand shapes form as light bounces off your hands. You are taught to make several different hand gestures, all timed to receive small bursts of feedback from the Stratos Explore. Then, you are taught to make a butterfly with your hands, and it triggers a virtual 3D butterfly to “come to life” and fly around you, then land on your finger. The haptic sensations are programmed to coincide with these interactions so you feel a burst of air as its wings flutter. Then environment around you comes to life, a 360° forest emerges and gently rotates around you avoiding any motion sickness. We thoughtfully designer the experience as a subtle environmental messsage. If you engage well with the butterfly, the forest would be luscious, but if you were poorly skilled and unable to learn, you would notice the forest around you deteriorte and perish.
Sam
Sam, the Shadow Master
Ultraleap and LeapMotion
Through RobotShop I bought my own Leap Motion Controllers and duly attached one to the front of my Quest with duct tape!
I signed up as an Ultraleap Developer and proceeded to install the SDK, and play with the demo sample projects . There were two parts to this, one was the hand tracking component, and the second the haptics. A Leap Motion Controller is built into the front of the Stratos, but in our 4 person group, we had 2 Stratos Dev Kits to share, so I invested in my own standalone controllers. Hand tracking was not yet available in the Quest.
I was already registered as a Quest developer and had been using Unity for 2 years by this point. The developer documentation for the Stratos and Leap Motion Controllers was very detailed, but not incredibly user-friendly. We remotely met with some developers at Ultraleap to discuss our project, and elicit their recommendations.
Experimenting with Stratos Explore, Unity, and PCVR
Here are a few videos capturing my experiments with Quest PCVR and Unity:
Sam’s Performance in Unity/PCVR
Addendum:
While presenting the Magicbook at SIGGRAPH 2023, as a volunteer with Mark Billinghurst’s team, I met, and got chatting to a few people, this ranged from creators from Apple Studios, the Head of Learning at Foundry, to one of the UIltraleap team. He showed me the latest Leap Motion Controller which I duly photographed.
