Vision in the Future

As the saying goes, “seeing is believing”. This is unfortunately not yet true for Augmented Reality (AR) applications where the goal is to augment the observed world with real-time computer-generated images and interfaces. At the moment, most consumer AR applications simply display an image on a normal LCD screen or a mobile phone’s display. The user then sees these images, but does not believe that they are truly real. One of the reasons for this lack of believability (or to use the correct term, immersiveness) is simply that the observer’s senses are not fooled into believing that the AR is real.

Creating the Illusion of Reality

So how exactly do we fool the visual sense into believing an image is real? Fooling the visual sense requires that the optical cues that our brains use to interpret what we see, are accurately simulated. The cues can be divided into two sets. The first set of optical cues has to do with what we see, and include perspective correctness, textures, lighting, shades, shadowing and occlusions. All of these cues can be simulated in today’s top-quality rendering software.

The second set of cues is more about how we see and these cues are the ones that cannot yet be simulated easily. The cue that most people are familiar with, is that of stereoscopy, which is when each eye observes a slightly different view of the same scene. This is the cue that is used in so-called 3D movies and 3D displays that create the illusion of depth. The second cue is accommodation, which has to do with the focusing of the eye lens to get a clear image of an object at distance. This change in focus gives some information on how far an object is. The third cue is closely related to both stereoscopy and accommodation and is called convergence. Convergence, as the name states, is the convergence of the two eyes on the focus point. The angle of convergence tells the viewer how close or how far away an object is. The last major cue is motion parallax. This cue is based on the relative movement between objects when the observer shifts his view of the scene, or when objects in the scene move. This relative motion allows the observer to judge size and relative position between objects.

Figure 1: Example of convergence and accommodation cues.

Displaying the Illusion of Reality

Now knowing what the display hardware needs to simulate, lets take a look at what is currently available and what may be available in the future. Starting off with the most common new type of display, the stereoscopic display, otherwise known as 3D TV and 3D cinema. One common feature to all these displays is that they are able to display different images to each eye. There are currently a number of different methods to achieve this, but most of these use glasses that the block one eye from seeing the image intended for the other eye. This may be done by using filters (such as polarising or mutual density (MD) filters) or eclipsing glasses using LCDs. An alternative method that does not require glasses, is the Parallax Barrier method. This method uses a simple grating placed in front of the screen. This grating blocks the light from pixels meant to be observed by the one eye from reaching the other eye. Currently the biggest hurdles to overcome with this method is that viewer(s) are required to be seated at certain positions for the display to work. One common problem with stereoscopic displays is that they cause headaches in some users. This is due to the fact that the cues of accommodation and convergence are conflicting with the stereoscopic cue. This in turn means that user’s brain needs to constantly resolve these conflicting cues, which over time (due to the extra effort) can lead to headaches.

Figure 2: Parallax Barrier diagram.

A science fiction classic that is slowly becoming reality is that of holograms, also known as volumetric displays. Over the years a number of different methods have been tested, but few have achieved good results. The advantage of a holographic display is that it inherently incorporates all the optical cues due to how the image is created. Two notable examples of holographic displays are the Holovizio and Light Field displays. The Holovizio display uses an array of projectors to create light voxels which then create the illusion of a physical object. The Light Field Display uses a projector and a spinning mirror to produce a hologram of an object. The big disadvantages of these displays are their size, the low quality of the image versus the display costm and the physical setup required to get them to work properly.

Figure 3: The Holovizio display.

Figure 4: A cross stereo photo of the Light Field Display.

Another science fiction favourite turned science fact is the Head Mounted Display (HMD). This is a display unit that is a attached to the user’s head and then presents the displayed image to the users in his field of view. A number of different concepts and designs exist for the two types of HMDs. The first type is see-through, where the user is still able to see the physical world overlayed with the displayed image, and the second is non-see-through, where the user is only able to see the displayed image. The advantage of HMDs is that they are close to the user”s eyes and attached to head, so the cues of stereoscopy and motion parallax can easily be achieved. Research is currently being done to include the cues of convergence and accommodation by using variable focus lenses and eye tracking in HMD units.

Figure 5: Augmented Reality maintenance using an HMD.

Taking the idea of HMDs one step further is the concept of a Retinal Display (RD). RDs use a laser to directly draw the displayed image onto the user’s retina. This method has the potential for creating very small HMDs that could incorporate all the optical cues needed to create a believable image and comfortable display devices. A number of problems still need to be overcome before RDs become practical; one of the biggest problems is to compensate for eye movement. This is a mayor hurdle as the laser needs to enter the eye through the centre of the lens, which is difficult to achieve when the eye moves around. However we could soon start seeing RDs being sold, as Brother Inc. may soon launch their first commercial fixed eye RD.

Figure 6: The RD display from Brother Inc.

The Future of the Illusion

Looking a bit further into the future there are two new technologies that may one day offer the ultimate in personal display technology: The Contact Lens Display and the Bionic Eye. The Contact Lens Display is an ongoing reseach project to create a display that is integrated into a contact lens. This should allow for users to have instant access to a computer without having to carry bulky display devices around with them. At the moment the research is still far from creating a working prototype, but could one day be the future of displays.

Figure 7: Contract Lens display.

The other new technology is the Bionic Eye. The research started as a way to improve vision of patients with vision loss. The system consist of electrodes implanted into the retina of the patient. These electrodes stimulate the optic nerves with signals received from a processor and camera unit mounted on a pair of glasses. The prototype unit was developed by Bionic Vision Australia, has 98 electrodes that stimulate the optic nerve and human trials could start as early as 2011.

Figure 8: Schematic of the Bionic Eye.

As these technologies improve it may one day be impossible to distinguish between what is real and what is mearly a figment of our digital imaginations.

, , , , , , , ,


  1. 1
    Carel van Wyk on Wednesday 21 April, 14:06 PM #

    My bet is on the Bionic implant. It pretty much solves all problems related to AR displays with the added advantage that you’ll have super eagle-vision your whole life. Also, it doesn’t require nano-scale light-emitting circuitry that RDs would need, I don’t really see retinal displays as a viable future avenue for AR.

  2. 2
    Dirk B on Thursday 06 May, 16:47 PM #

    Hi waldo – awesome post!
    Keep them coming !

    Ps – ek toets gou die ipad !

    The tweet

Leave a comment

Leave a Reply