Experiment One Body as Mouse
Inspired by products such as Apple’siPhone and Microsoft’s Surface that allow direct interaction through touch that feel physical and natural, we began investigating seamless interaction between the gestures that we intuitively make and human-scale displays. Experiences such as Funky Forest, created by Muonics and the Infinity Interactive Mirrors created by George P. Johnson in Los Angeles, led us to begin our exploration toward one of the many behaviors introduced in these projects: replacing the mouse with a person’s body to interact with a human-scale display. We began by exploring the body as a navigational mechanism that is invisible, feels seamless between the body and the display, and that feels natural and intuitive.
Test 1: Pan Naturally
The interaction model of the first test was to use a single mechanism that would allow your body to effect the panning of a scene left and right. We designed the UI imagery so that it would appear to have dimensional perspective. When you moved to the left, the foreground would shift to the right while the background would shift at a much faster rate. Walking the opposite direction would give you the reverse affect.
This gave the illusion that you were immersed in a dimensional space and that you were using your body to pan the scene in a very natural way. In other words, you panned the scene by simply performing the same action that you would if you were in a real-life scene; walking left or right.
PROTOTYPE
For the sake of this exploration, we decided to limit ourselves to the use of one proximity sensor and a large format screen. We started out testing the properties of a long-range IR sensor and quickly discovered a slew of technical challenges. Their value jumped between 10 and sometimes 250 so it was difficult to get the smooth reading necessary to accurately control the left/right pan. To solve this problem, we wrote a custom flash class that averaged out the high/low values at a constant rate, which created a smoother buffer, therefore a smoother transition in the UI.
 |
 |
 |
OUTCOMES
This test was limited to a single user, therefore it was designed to be experienced by one person only. Multiple users would interrupt the effect of the panning, resulting in an unnatural experience. Our next test took this a step further and explored a multi-user scenario for a more realistic and real-world experience.
Test 2: Pan, Zoom, & Discover
For test two, we set out to add two more behaviors in addition to panning: zoom and the ability to discover content through progressive disclosure. The second behavioral addition provided the ability to zoom into the scene in a very natural way. Not only were you able to naturally pan left/right by walking left/right, but you could “walk into and out of” the scene by walking forward and backwards. The result zoomed the scene in and out much like walking forward and backward in real life. The third behavioral addition allowed for the progressive disclosure of additional information based on your behavior and only when it is necessary. This added an element of discovery to the experiment. When you walked up to the screen and came within arm’s reach of the display, touch areas were displayed.
These touchable hotspots revealed additional information in the scene that provided another layer of content and drew you into the experience even further.
PROTOTYPE
In the working prototype, we added a second proximity sensor that detects a person walking toward and away from the screen. This allowed the scene to zoom in and out. We turned the display into a touch screen that worked only when within arm’s length of the users. This would trigger subtle animations that revealed hotspots in the scene that the user could touch. We adjusted our custom flash class to smooth the jumpy nature of the proximity sensors until the experience resulted in a somewhat natural interaction between us and the display.
 |
 |
 |
Envisioning real-world applications
Throughout this experiment, we envisioned a commercially-based immersive experience that replaces the body as mouse in such a natural way that anyone could interact with and discover content without having to rely on extensive computer know-how. Some realistic applications that we discussed throughout this exploration included human-scale experiences that are seamlessly integrated into retail spaces, museums, and public spaces. These types of experiences could help solve problems such as reducing the amount of physical merchandise required in a store, bridge the gap between dynamic multi-layered content and the delightful experiences to obtain it, and realize a company’s brand by extending it into physical experiences.
Additional questions around body as mouse:
How can the body be used as a mouse in a natural way? What are natural interactions and gestures between a person and a screen? An object and a screen? And between object?