Interacting with Holograms
2012 – 2015 for Microsoft
In 2012, I joined Microsoft to work on what would become Microsoft HoloLens, a head-mounted Mixed Reality device that allows the wearer to
see 3D content (holograms) as if the content is situated in the real world.
As a Designer/Research Lead on the HoloLens Experiences team, I led a multi-disciplinary team to prototype and
evaluate multi-modal interactions, using Gaze/Gesture/Voice in entertainment experiences.
In 2015, I joined a specialized incubation team to work on future HoloLens and IoT concepts, prototyping how different devices can
communicate and share data and content with each other.
This case study covers our exploration of a broad range of gestures and their use cases prior to the choosing of the "airtap" gesture,
then looks at specific use of the Gaze/Gesture/Voice model when interacting with IoT devices.
Results of this work include the following patent filings:
Legal Note: Due to the extremely sensitive nature of any HoloLens-related work, I am unable to go into detail on our
strategy and findings, and use publicly available work to illustrate our concepts.
Gesture Reference, Research, and Inspiration
Our goal was to understand when gesture input would be appropriate, comfortable, and what type of gestures
would work across a range of contexts.
We investigated how gestures are already used in everyday situations and communication, and what inputs were accepted by existing gesture tech. We explored
how gestures reflect the speaker's mental model (e.g. when referring to events in the past or future, people gesture behind or in front of them),
and how gesture size is affected by social acceptability and visual feedback received.
Prototyping Interactions for Entertainment on HoloLens
As we developed the gesture interaction model, we evaluated how these gestures might be used in real-time
entertainment experiences and live events, alongside gaze and voice.
How should the content be presented to the viewer such that it would be both useful and easy to interact with? Our goal was to provide a
balance of active/passive interactions, without overwhelming the viewer or detracting from the broadcast/event.
To test out different scenarios, my team created proof-of-concept interactive prototypes for NFL and major racing events, exploring
holographic 3D broadcast, augmenting on-field/on-track play, and complementing a TV screen at home.
Exploring and Testing Common Gesture Actions and Extents
We then generated a set of gestures for common actions, like content selection, scaling, browsing, movement/placement, sorting, etc.
We evaluated our different gestures using our interactive entertainment prototypes and other scenarios, trying the gestures out on smaller,
table-top holographic content to room/venue-filling content. Playing with
scale and placement of content helped us determine recommendations for hand poses, arm movement, and content placement relative to the user's body.
A single gesture for v1 HoloLens
My team submitted our findings and suggestions to the HoloLens Platform team, who were taking requirements from all the Experiences teams to create a
system that would work for all experiences.
The Platform team built a streamlined set of gestures that would be easy for the user to repeat and learn. They settled on the bloom and airtap
gestures that you see implemented for HoloLens today, and continue to expand the gesture set, as seen in Mixed Reality Toolkit.
Using HoloLens to interact with other devices
The next goal (after a short break working on Interactive TV) was looking at how HoloLens and holographic content would
could work within a "society of devices".
We created a prototype in which the wearer could move content by grabbing and moving a hologram to different devices in the room: mobile devices, a tablet, an Xbox,
and a printer. For each device, HoloLens would display UI appropriate to that device and content, relative to the device's position in the room.
Working on HoloLens to define both the experiences and hardware was one of the best experiences of my career. I loved the team and
our creative energy and drive. I learned so much about working on pre-release hardware, experimentation, and creating with courage.
Working on HoloLens also crystallized for me that I love working on ambiguous, risky projects
where we have the potential to influence the future. I am excited about all the incredible things people will do with augmented and
mixed reality, and how much we still have left to discover in this space.