Interacting with Holograms

2012 – 2015 for Microsoft

In 2012, I joined Microsoft to work on what would become Microsoft HoloLens, a head-mounted Mixed Reality device that allows the wearer to see 3D content (holograms) as if the content is situated in the real world.

As a Designer/Research Lead on the HoloLens Experiences team, I led a multi-disciplinary team to prototype and evaluate multi-modal interactions, using Gaze/Gesture/Voice in entertainment experiences.

In 2015, I joined a specialized incubation team to work on future HoloLens and IoT concepts, prototyping how different devices can communicate and share data and content with each other.

This case study covers our exploration of a broad range of gestures and their use cases prior to the choosing of the "airtap" gesture, then looks at specific use of the Gaze/Gesture/Voice model when interacting with IoT devices.

Results of this work include the following patent filings:

Legal Note: Due to the extremely sensitive nature of any HoloLens-related work, I am unable to go into detail on our strategy and findings, and use publicly available work to illustrate our concepts.

Gesture Reference, Research, and Inspiration

Our goal was to understand when gesture input would be appropriate, comfortable, and what type of gestures would work across a range of contexts.

We investigated how gestures are already used in everyday situations and communication, and what inputs were accepted by existing gesture tech. We explored how gestures reflect the speaker's mental model (e.g. when referring to events in the past or future, people gesture behind or in front of them), and how gesture size is affected by social acceptability and visual feedback received.

Gesture Reference and Inspiration

Prototyping Interactions for Entertainment on HoloLens

As we developed the gesture interaction model, we evaluated how these gestures might be used in real-time entertainment experiences and live events, alongside gaze and voice.

How should the content be presented to the viewer such that it would be both useful and easy to interact with? Our goal was to provide a balance of active/passive interactions, without overwhelming the viewer or detracting from the broadcast/event.

To test out different scenarios, my team created proof-of-concept interactive prototypes for NFL and major racing events, exploring holographic 3D broadcast, augmenting on-field/on-track play, and complementing a TV screen at home.

Interactive Prototypes Sports Experiences

Exploring and Testing Common Gesture Actions and Extents

We then generated a set of gestures for common actions, like content selection, scaling, browsing, movement/placement, sorting, etc.

We evaluated our different gestures using our interactive entertainment prototypes and other scenarios, trying the gestures out on smaller, table-top holographic content to room/venue-filling content. Playing with scale and placement of content helped us determine recommendations for hand poses, arm movement, and content placement relative to the user's body.

Gesture Zones

A single gesture for v1 HoloLens

My team submitted our findings and suggestions to the HoloLens Platform team, who were taking requirements from all the Experiences teams to create a system that would work for all experiences.

The Platform team built a streamlined set of gestures that would be easy for the user to repeat and learn. They settled on the bloom and airtap gestures that you see implemented for HoloLens today, and continue to expand the gesture set, as seen in Mixed Reality Toolkit.

HoloLens Announce Demo

Using HoloLens to interact with other devices

The next goal (after a short break working on Interactive TV) was looking at how HoloLens and holographic content would could work within a "society of devices".

We created a prototype in which the wearer could move content by grabbing and moving a hologram to different devices in the room: mobile devices, a tablet, an Xbox, and a printer. For each device, HoloLens would display UI appropriate to that device and content, relative to the device's position in the room.

HoloLens IoT Prototypes


Working on HoloLens to define both the experiences and hardware was one of the best experiences of my career. I loved the team and our creative energy and drive. I learned so much about working on pre-release hardware, experimentation, and creating with courage.

Working on HoloLens also crystallized for me that I love working on ambiguous, risky projects where we have the potential to influence the future. I am excited about all the incredible things people will do with augmented and mixed reality, and how much we still have left to discover in this space.