Augmented reality (AR) is a technology in which a user's view of the real world is enhanced or augmented with additional information which is superposed on images of the real scene using graphical means. The enhancement may consist of virtual artifacts to be fitted into the environment, or a display of non-geometric information about existing real objects. AR allows a user to work with and examine real 3D objects, while receiving additional information about those objects or the task at hand. Using AR technology, users can thus interact with a mixed virtual and real world in a natural way. AR systems bring the computer to the user's real work environment, whereas VR systems try to bring the world into the user's computer. This paradigm for user interaction and information visualization constitutes the core of a very promising new technology for many applications. It integrates the computer vision and interactive 3D computer graphics techniques in a natural way. However, real applications impose very strong demands on AR technology that cannot yet be met. Some of such demands are listed below. In order to combine real and virtual worlds seamlessly so that the virtual objects align well with the real ones, we need very precise models of the user's environment and how it is sensed. It is essential to determine the location and the optical properties of the viewer (or camera) and the display, i.e., we need to calibrate all devices, combine all the local coordinate systems centered on the devices and objects in the scene in a global coordinate system, register models of all 3D objects of interest with their counterparts in the scene, and track them over time when the user moves and interacts with the scene. Realistic merging of virtual objects with a real scene requires that objects behave in physically plausible manners when they are manipulated, i.e., they occlude or are occluded by real objects, they are not able to move through other objects, and they are shadowed or indirectly illuminated by other objects while also casting shadows themselves. To enforce such physical interaction constraints between real and virtual objects, the AR-system needs to have a very detailed description of the physical scene.