Projection augmented model


A projection augmented model is an element sometimes employed in virtual reality systems. It consists of a physical three-dimensional model onto which a computer image is projected to create a realistic looking object. Importantly, the physical model is the same geometric shape as the object that the PA model depicts.

Uniting physical and virtual objects

Spatially augmented reality renders virtual objects directly within or on the user's physical space. A key benefit of SAR is that the user does not need to wear a head-mounted display. Instead, with the use of spatial displays, wide field of view and possibly high-resolution images of virtual objects can be integrated directly into the environment. For example, the virtual objects can be realized by using digital light projectors to paint 2D/3D imagery onto real surfaces, or by using built-in flat panel displays.
Real objects can be physically handled and naturally manipulated to be viewed from any direction, which is essential for ergonomic evaluation and provides a strong sense of palpability. Although simulated haptic feedback devices enable some aspects of computer-generated objects to be touched, they can not match this level of functionality. It is, therefore, unsurprising that physical objects are still used for many applications, such as product design. However, computer-generated objects have a key advantage; they provide a level of flexibility that cannot be matched by physical objects. Therefore, a display is needed that somehow joins the real physical world and computer-generated objects together, thus enabling them to be experienced simultaneously.
Tangible user interfaces and augmented reality both aim to address this issue. TUI systems use real physical objects to both represent and also interact with computer-generated information. However, while TUIs create a physical link between real and computer-generated objects, they do not create the illusion that the computer-generated objects are actually in a user's real environment. That is the aim of augmented reality.
Figure 1 Continuum of advanced computer interfaces, based on Milgram and Kishino.
Unlike virtual reality, which immerses a user in a computer-generated environment, augmented reality joins together physical and virtual spaces by creating the illusion that computer-generated objects are actually real objects in a user's environment. Furthermore, head-mounted-display based AR and VR systems can directly incorporate physical objects. Thus, as a user reaches out to a computer-generated object that they can see, they touch an equivalent physical model that is placed at the same spatial location. Such systems enable the computer-generated visual appearance of the object to be dynamically altered, while the physical model provides haptic feedback for the object's underlying form. However, head-mounted-display based systems require users to wear equipment, which limits the number of people who can simultaneously use the display.
A variant of the AR paradigm that does not suffer from these limitations is spatially augmented reality. Spatially augmented reality displays project computer-generated information directly into the user's environment. Although there are several possible display configurations, the most natural type is the projection augmented model.

Projection augmented models

Figure 2 The Projection Augmented model concept
A projection augmented model consists of a physical three-dimensional model, onto which a computer image is projected to create a realistic looking object. Importantly, the physical model is the same geometric shape as the object that the PA model depicts. For example, the image projected onto the objects shown in Figure 3 provides colour and visual texture, which makes them appear to be made from different materials.
Figure 3 An example of a Projection Augmented model.
PA models use a unique combination of physical objects and computer-generated information, and hence they inherit advantages from both. “The human interface to a physical model is the essence of ‘intuitive’. There are no widgets to manipulate, no sliders to move, and no displays to look through. Instead, we walk around objects, moving in and out to zoom, gazing and focusing on interesting components, all at very high visual, spatial, and temporal fidelity”. PA models combine the high level of intuitiveness of physical models with the flexibility and functionality of computer graphics, such as the ability to be quickly altered, animated, saved and updated. Thus, a PA model essentially gives a physical form to a computer-generated object, which a user can touch and grasp with their bare hands. It is therefore unsurprising that user studies, which compared PA models to other Virtual and Augmented Reality displays, found PA models to be a natural and intuitive type of display.
However, the PA model concept is not new. In fact, one of the first PA model type displays was created over twenty years ago when Naimark built the ‘Displacements’ art installation and more recently in the “Haunted Mansion” attraction in Disney World. At the time technology did not exist for a PA model to be much more than an artistic statement. However, given the technology available today and a little “unfettered imagination”, exploring novel projection displays is now “potentially boundless”.
The growth in PA model technology has been marked by the recent recreation of Naimark's ‘Displacements’ installation at SIGGRAPH. Specifically, new technology has been developed that semi-automates the process of both creating and aligning the physical model and projected image. This supports multiple projectors, which enables a PA model to be illuminated from every direction. Furthermore, powerful projectors can be used to allow a PA model to be located in a well-lit room. However, whilst this technology enables a PA model to be a viable and useful type of display, it does not address its main aim.
A PA model aims to create the illusion of actually being the object that it depicts. For example, when used for a product design application, it is important that a PA model provides a convincing perceptual impression of actually being the final product. Similarly, when used for a museum display application to create a replica of an artefact, a PA model aims to create the illusion of being the real artefact.
However, no previous research has specifically considered this illusion. Therefore, this thesis defines the ‘Projection Augmented model illusion’ as the situation in which a PA model is perceived to actually be the object that it depicts. For example, this illusion occurs when a user perceives the PA model in Figure 3 to be real bricks, flower pots, and pieces of wood, as opposed to white models with an image projected onto them. However, the essence of this illusion does not involve deceiving the user. A user can perceive a PA model to be the object that it depicts, whilst knowing that it is actually a white model and a projected image.
Technology has been developed to enhance this illusion by increasing the physical similarity between the PA model and the object that it depicts, or in other words, increasing the fidelity of the PA model. For example, the way in which the specular highlights on an object move as the viewer changes position can be dynamically simulated. This enables a PA model to appear to be made from a wide range of materials. For example, a dull clay vase can appear to be made from a shiny plastic material.
However, whether or not the PA model illusion occurs is entirely dependent on a user's subjective perceptual impression. Therefore, increasing the fidelity of different aspects of a PA model may each have a different effect on the strength of the illusion. This is essentially the same as the way in which increasing the fidelity of different aspects of a computer-generated photorealistic image, may each have a different effect on the degree to which the image is perceived to be a real photograph. For example, increasing the fidelity of the textures in the image may typically be more important than increasing the fidelity of the shadows.
It cannot therefore be assumed that increasing the fidelity of any aspect of a PA model will automatically strengthen the PA model illusion, and similarly it cannot be assumed that decreasing the fidelity of any aspect will automatically weaken it. Therefore, given that no previous research has investigated this illusion, it is difficult to determine the success of the technology that aims to enhance it, and difficult to make informed decisions when developing new technology. The capabilities of the human perceptual system should guide the development of any advanced interface, hence this issue needs to be addressed.
Note: Projection Augmented models are sometimes referred to as 'Shader Lamps'.

Other relevant publications

Bennett, E., & Stevens, B.. . Journal of Presence: Teleoperators and Virtual Environments, special edition of the best papers from the International Presence Conference, 15, 419-437, MIT Press.
Bennett, E., & Stevens, B.. . In Proceedings of SIGGRAPH's Computer Graphics and Applied Perception Symposium.