Microsoft PixelSense


Microsoft PixelSense was an interactive surface computing platform that allows one or more people to use and touch real-world objects, and share digital content at the same time. The PixelSense platform consists of software and hardware products that combine vision based multitouch PC hardware, 360-degree multiuser application design, and Windows software to create a natural user interface.

Overview

Microsoft Surface 1.0, the first version of PixelSense, was announced on May 29, 2007, at the D5 Conference. It shipped to customers in 2008 as an end-to-end solution with Microsoft producing and selling the combined hardware/software platform. It is a 30-inch 4:3 rear projection display with integrated PC and five near-infrared cameras that can see fingers and objects placed on the display. The display is placed in a horizontal orientation, giving it a table-like appearance. The product and its applications are designed so that several people can approach the display from all sides to simultaneously share and interact with digital content. The cameras’ vision capabilities enable the product to see a near-IR image of what’s placed on the screen, captured at approximately 60 times per second. The Surface platform processing identifies three types of objects touching the screen: fingers, tags, and blobs. Raw vision data is also available and can be used in applications. The device is optimized to recognize 52 simultaneous multitouch points of contact. Microsoft Corporation produced the hardware and software for the Microsoft Surface 1.0 product. Sales of Microsoft Surface 1.0 were discontinued in 2011 in anticipation of the release of the Samsung SUR40 for Microsoft Surface and the Microsoft Surface 2.0 software platform.
Microsoft and Samsung partnered to announce the current version of PixelSense, the Samsung SUR40 for Microsoft Surface, at the Consumer Electronics Show in 2011. Samsung began shipping the new SUR40 hardware with the Microsoft Surface 2.0 software platform to customers in early 2012.
The Samsung SUR40 is a 40-inch 16:9 LED backlit LCD with integrated PC and PixelSense technology, which replaces the cameras in the previous product. PixelSense technology enables Samsung and Microsoft to reduce the thickness of the product from 22 in to 4 in. The size reduction enables the product to be placed horizontally, and adds the capability to be mounted vertically while retaining the ability to recognize fingers, tags, blobs and utilize raw vision data. Samsung produces the hardware and Microsoft produces the software platform for the SUR40.

Target market

PixelSense is designed primarily for use by commercial customers to use in public settings. People interact with the product using direct touch interactions and by placing objects on the screen. Objects of a specific size and shape, or with tag patterns, can be uniquely identified to initiate a preprogrammed response by the computer. The device does not require the use of a traditional PC mouse or keyboard, and generally does not require training or foreknowledge to operate. Additionally, the system is designed to interact with several people at the same time so that content can be shared without the limitations of a single-user device. These combined characteristics place the Microsoft Surface platform in the category of so-called natural user interface, the apparent successor to the graphical user interface systems popularized in the 1980s and 1990s.
Microsoft states that sales of PixelSense are targeted toward the following industry verticals: retail, media and entertainment, healthcare, financial services, education, and government. PixelSense is available for sale in over 40 countries, including United States, Canada, Austria, Belgium, Denmark, France, Germany, Ireland, Italy, Norway, Netherlands, Qatar, Saudi Arabia, Spain, Sweden, Switzerland, United Arab Emirates, United Kingdom, Australia, Korea, India, Singapore, and Hong Kong.

History

The idea for the product was initially conceptualized in 2001 by Steven Bathiche of Microsoft Hardware and Andy Wilson of Microsoft Research.
In October 2001, DJ Kurlander, , Joel Dehlin, Bathiche and Wilson formed a virtual team to bring the idea to the next stage of development.
In 2003, the team presented the idea to the Microsoft Chairman Bill Gates, in a group review. Later, the virtual team was expanded and a prototype nicknamed T1 was produced within a month. The prototype was based on an IKEA table with a hole cut in the top and a sheet of architect vellum used as a diffuser. The team also developed some applications, including pinball, a photo browser, and a video puzzle. Over the next year, Microsoft built more than 85 prototypes. The final hardware design was completed in 2005.
A similar concept was used in the 2002 science fiction movie Minority Report. As noted in the DVD commentary, the director Steven Spielberg stated the concept of the device came from consultation with Microsoft during the making of the movie. One of the film's technology consultant's associates from MIT later joined Microsoft to work on the project.
The technology was unveiled as under the "Microsoft Surface" name by Microsoft CEO Steve Ballmer on May 30, 2007, at The Wall Street Journal's 'D: All Things Digital' conference in Carlsbad, California. Surface Computing is part of Microsoft's Productivity and Extended Consumer Experiences Group, which is within the Entertainment & Devices division. The first few companies slated to deploy it were Harrah's Entertainment, Starwood Hotels & Resorts Worldwide, T-Mobile and a distributor, International Game Technology.
On April 17, 2008, AT&T became the first retailer to sell the product. In June 2008 Harrah’s Entertainment launched Microsoft Surface at Rio iBar and Disneyland launched it in Tomorrowland, Innoventions Dream Home. On August 13, 2008, Sheraton Hotels introduced it in their hotel lobbies at 5 locations. On September 8, 2008, MSNBC began using it to work with election maps for the 2008 U.S. Presidential Election on air.
On June 18, 2012, the product was re-branded under the name "Microsoft PixelSense" as a result of the company adopting the Surface brand for its newly unveiled series of tablet PCs.

Features

Microsoft notes four main components being important in the PixelSense interface: direct interaction, multi-touch contact, a multi-user experience, and object recognition.
Direct interaction refers to the user's ability to simply reach out and touch the interface of an application in order to interact with it, without the need for a mouse or keyboard. Multi-touch contact refers to the ability to have multiple contact points with an interface, unlike with a mouse, where there is only one cursor. Multi-user experience is a benefit of multi-touch: several people can orient themselves on different sides of the surface to interact with an application simultaneously. Object recognition refers to the device's ability to recognize the presence and orientation of tagged objects placed on top of it.
The technology allows non-digital objects to be used as input devices. In one example, a normal paint brush was used to create a digital painting in the software. This is made possible by the fact that, in using cameras for input, the system does not rely on restrictive properties required of conventional touchscreen or touchpad devices such as the capacitance, electrical resistance, or temperature of the tool used.
In the old technology, the computer's "vision" was created by a near-infrared, 850-nanometer-wavelength LED light source aimed at the surface. When an object touched the tabletop, the light was reflected to multiple infrared cameras with a net resolution of 1024×768, allowing it to sense, and react to items touching the tabletop.
The system ships with basic applications, including photos, music, virtual concierge, and games, that can be customized for the customers.
A feature that comes preinstalled is the "Attract" application, an image of water with leaves and rocks within it. By touching the screen, users can create ripples in the water, much like a real stream. Additionally, the pressure of touch alters the size of the ripple created, and objects placed into the water create a barrier that ripples bounce off, just as they would in a real pond.
The technology used in newer devices allows recognition of fingers, tag, blob, raw data, and objects that are placed on the screen, allowing vision-based interaction without the use of cameras. Sensors in the individual pixels in the display register what is touching the screen.

Hardware specifications

Microsoft Surface 1.0

Microsoft provides the free Microsoft Surface 2.0 Software Development Kit for developers to create NUI touch applications for devices with PixelSense and Windows 7 touch PCs.
Applications for PixelSense can be written in Windows Presentation Foundation or XNA. The development process is much like normal Windows 7 development, but custom WPF controls had to be created due to the unique interface of the system. Developers already proficient in WPF can utilize the SDK to write applications for PixelSense for deployments for the large hotels, casinos, and restaurants.

Related Microsoft research projects

Microsoft Research has published information about a related technology dubbed SecondLight. Still in the research phase, this project augments secondary images onto physical objects on or above the main display.