The Society for Computer Simulation International
Virtual Worlds and Simulations Conference, San Francisco, January 17-20, 1999
Manager, Electronic Publications
Getty Trust Publications
1200 Getty Center Dr.
Los Angeles, CA 90049
Key words: Augmented, Culture, Longevity, Polychronic, Monochronic
In many respects a "virtual world" is a time machine. Not in some science fiction context, but rather as a kind of content-clock that is inhabitable. These worlds are both monochronic and polychronic, that is, they allow the participant to interact with information in motion and that motion can be a single "clockwise" monochronic direction, or multiple layers of information moving at different speeds ó polychronic - as the user calls on them. These aspects of visualization are intoxicating in their possibilities. Add to these prospects "augmented reality" - using computer information as an overlay to the real world rather than immersive graphical environments - and you have even more possibilities for complex clocks. What is important to keep in mind is that these information conditions are based on a complex ephemeral medium, computer data. To maintain models of this nature over time is exceedingly problematic because not only must the data, the formats, and the storage media and display system be preserved, but also the modes of user interaction must somehow be inventoried. The problem space for these "content clocks" as cultural memory repositories is fascinating. The implication that these time machines may not survive their own complexity is equally of concern.
Artists and scientists have always grappled with the dichotomy of being and knowing, experience and our understanding of it. The use of instruments to enhance or immediately inform experience have been under development forcenturies. Time, vision, and sound have always been personal experiences we continue to try to enhance.
Take time for instance.
In 1762, John Harrison designed the marine chronometer that led to the development of the portable clock. In 1907, Louis Cartier was commissioned by French aviator Alberto Santos-Dumont to create a clock with a wristband, the modern wristwatch. And lest we forget, we occasionally look at our watches, but seldom "watch" them. The term watch was used to describe a person who walked the city streets with a portable clock, the night watchman.
The current excitement about "virtual reality" (VR) stems mostly from the notion that the imagination, in all its layered interactivity and monochronic and polychronic time dimensions can be modeled by computers. Instead of viewing the model externally like animation, however, we can be inside the model by using an immersive display system - we can "wear the model". The key ingredients - high quality three dimensional animated computer graphics, real-time response from an interactive system, and immersive display systems make virtual reality a computer controlled experience of different slices of time.
The VR system must be sensitive to the movement of the user in order to "redraw itself" in response to that movement. The use of this technology has created some stunning simulations of flying though the air, through outer space, through the human body. It has brought us in contact with everything from artistís perceptions of imaginary sculpture to the reconstruction of ancient architectures. But in each case we are forced to leave the real world and enter the time dimensions of the computer.
There is, however, a growing field of research that is concerned with something other than artificial "virtual" time and space. This line of inquiry asks -What if there was a composite digital/analog experience available? What if we could add information to the real world in order to maximize performance or enhance perception without leaving the world? And what if we could do this as easily as wearing glasses or wristwatches or headphones?
Augmented reality research strives to keep the user in the real world and add or manipulate information in a seamless way. Paul Milgram (Milgram et al. 1994) at the University of Toronto Mechanical Engineering Department has described a continuum in which the real world and the virtual world are at opposite ends of the spectrum.
Somewhere in the middle is an area of "mixed reality". Augmented reality is closer to the real world end with the dominant interaction being with something tangible that is enhanced with computer information. To further complicate matters, Milgram has coined the term "augmented virtuality" to describe systems that are artificial with some elements like texture mapping to make them more realistic.
There are a number of approaches to augmented reality that require different mechanisms to combine digital information with analog experience. These mechanisms take a variety of approaches to one of the key limitations of augmented reality, registration. To render digital information and fuse it with experience is not an easy technical task. Errors in registration will make the process not only confusing but useless.
Registration, then, can mean anything from making relevant digital information available at key moments of experience (timing), to visually registering computer animation with real events (visioning), to some combination of time and perception. Augmented reality can be facilitated by personal instrumentation, computer-controlled space, or some combination of instruments and spaces. The final result should be an experience that is not usually apparent to the human senses, like looking through your glasses and seeing the time and temperature and your next appointment.
Instead of wearing the instruments that generate digital information what if the place you were in could somehow sense the information you needed at a particular moment? A number of researchers including the MIT Media Lab are doing work on "Smart Rooms". These are spaces equipped with sensors, cameras, and microphones that can interpret gestures, speech, and movement in order to deliver information at the appropriate moment. These technologies are already being tested in areas like security (the room knows who is in it) to use in cars that know when the driver is trying to turn or stop.
Since the late 1960ís museums have made use of "audio tours" to augment the reality of visitors. We have quickly come to take this use of technology for granted as a staple of museums but it wasnít so very long ago that the only "enhancement" in a museum were the labels on the wall. Now, with multi-tracking and digital audio, the museum visitor can visit Tahiti in the headphones in front of that Gaugan painting. The sophistication of smart museums is becoming taken for granted because of their use of new technologies for augmenting experience.
Another very powerful example of smart places are the galleries of the J. Paul Getty Museum in the new Getty Center in Los Angeles. Museum Director John Walsh declares that the mission of any museum is to provide light with the understanding that light always changes. This seemingly simple statement actually represents a very complex concern for the what the experience of art means. Artists work with physical materials that manifest qualities of color and light. When you "freeze" a painting with a particular wavelength of light provided by electric lights you narrow the quality of the experience to a very thin slice. The work was created in sunlight, light that changes throughout the day, thus giving the artist enormous latitude in creative opportunity. The final work is a summary of all the light the artist used in the process of creating the painting. Thus exhibiting the painting in some semblance of that light- process gives the viewer a closer communication with the artistís feelings and intentions. The art is alive in a sense, not deadened by a single kind of light. What the galleries of the Getty represent in both a poetic and technical sense is a time machine ó one aspect of it being to let the light of the present illuminate the work but also to give you sense of what the work might have looked like at the time of its creation as well.
The galleries of the museum are lit by skylights of heavy, UV filtered glass that allow about thirty percent of the external light into the rooms. The shape, size, and wall color of the rooms also affect the light quality. Because of the often clear brilliant light of the hilltop near Los Angeles where the museum is built, the designers wanted visitors to be able to see the sky so the skylight glass is clear. Working above the glass are are computer controlled louvers that orient themselves automatically based on the best possible light for the time of day and time of year. Because they did not want visitors distracted by constantly shifting louvers, they created elaborate look-up tables from data collected by photocells measuring the light every hour of every day from six by ten-foot wall areas in the rooms. These tables tell the louver inclinometers how to orient themselves on a schedule that subtly shifts them at prescribed times during the day.
When light sensors that constantly take readings from the gallery walls detect below 120 lux, quartz halogen lights with "blue" filters come on (they account for half of the lights in the gallery) because that wavelength mixes better with the dwindling daylight. They fade up very slowly, taking thirty seconds to reach full intensity. Visitors looking at the artwork will not notice the shift. As daylight continues to fade and the light levels again begin to reach 120 lux, the remaining red filtered halogen quartz lights come on because so little daylight is left. At this point the quartz lights are simulating a standard measure of artificial daylight.
Visitors have the extraordinary experience of never seeing a painting in the same light twice but they will never notice the technology orchestrating the experience.
This example of augmented reality goes to the heart of the use of technology to seamlessly enhance experience. No one who visits the museum is aware that they are in a "smart room". The vibrancy of the artwork is the only indication that something is special, that experience has been enhanced. The attention to the specificity of light, the use of computational technology to lend clarity and the invisible nature of the situation makes these rooms magical in sense. The experience is a not a technological "gee whiz" experience. It is more akin to a communion with content, in current techno-talk. For artists and historians, however, it is simply the right way to be with art. Visitors may not know how any of this has been done, but they do know that the artwork at the Getty is memorable.
What the use of technology in museums may be telling us is that the memorable experience may not be just about learning, but about how powerfully that learning takes place - aspects of theater may be a key to imparting information.
In some sense it is inevitable that the combination of personal instrumentation and smart spaces will lead to theater. Sponsored in part by the United States Institute for Theater Technology in Syracuse, New York, the University Theater of the University of Kansas has embarked on series of experiments in virtual and augmented reality. These productions are attempting to fuse virtual reality graphical environments with live stage production. This has meant that the audience wears virtual reality headgear that creates graphical environments for live stage action.
Their latest production was "Tesla Electric" that chronicled the life and career of inventor Nikola Tesla. The play combined vaudeville, carnival sideshows, and magic acts into an industrial morality play. This particular work utilized complex computer generated scenography rather than the three dimensional VR systems
Static computer generated scenes were created with lavish detail and texture. Live actors played in front of a panoramic triple screen on which the scenery was rear-projected. Stereo imaging was used so that the audience saw the scenes in 3-D with the aid of special 3-D "sunglasses". Graphical animated characters were also introduced using the projection and 3-D combination.
This hybrid of physical theater and graphical augmentation marks a genuine departure from traditional theater. The meaning of a work no longer resides in the interpretation of an authorís script, but rather meaning may come from the "rendering" of the play in the digital integration of real players, animation, lighting, and scenographics. The only thing missing from the production might be the use of technology in the costumes.
One of the more interesting areas of current research at the MIT Media Lab is in "wearable computers". Starting with the notion that the "personal computer" is a misnomer because computers are much more like furniture than they are anything "personal", this group of researchers has embarked on something more ambitious than putting the furniture in your lap (the lap-top computer). What they are anticipating are faster, cheaper, smaller, more powerful computer chips that will make it possible to wear information processing technology in a completely unobtrusive way. You might want to do this to increase or specialize memory, enhance a situation, or communicate with others who have similar interests.
A fundamental example of this is "smart fabric". Researchers at MIT (Post and Orth. 1997) are investigating adapting ancient techniques of weaving precious metals to modern electronic circuitry. This electronic fabric can distribute data and power, and perform touch sensing. Experimenting with silk organza, the warp is a plain silk thread while the weft is a silk thread wrapped in thin copper foil. The fabric is very conductive and has been fashioned into musical keyboards and graphic input surfaces by embroidering it onto clothes. The spacing of fibers allows them to be addressed separately much like a familiar computer ribbon cable. Circuits like busses can then be created to interact with various digital devices like microcontroler systems that can detect proximity and touch ó keyboards and touchpads. These have been tested with minature MIDI synthesizers thus making a wearable electronic instrument.
The notion of wearable (and washable) electronic circuitry that empowers the user to slip into a computer, a musical instrument, or communication system makes the cell phone seem like cumbersome hardware. The implications of sensory garb for augmented activities is much like wearing scuba gear for the mind.
The ability to use digital rendering technologies in new and exciting ways is without question. The nagging question that is only beginning to be asked is "How do we preserve this new medium?" Almost everyone assumes that digital files and recording media are somehow more archival than paper. The truth is far from that. Strategies of migrating data from one format to another, having it constantly accessible over networks, or keeping CD ROMís in temperature controlled vaults appear to be the easy answers.
To maintain virtual and/or augmented models of this nature over time is exceedingly problematic because not only must the data, the formats, and the storage media and display system be preserved, but also the modes of user interaction must somehow be inventoried. In the spectrum of digital preservation problems that roughly corresponds to static documents at the simple end to moving images in the middle and interactive static and motion images at the far end, virtual and augmented realities are almost out of bounds in terms of knowing how and what to preserve. The problem space for these "content clocks" as cultural memory repositories is fascinating. The implication that these time machines may not survive their own complexity is equally of concern.
Enhancing experience by augmenting reality with digital technology will become as common place as putting on your digital glasses and headphones. It can also be observed, without scientific or engineering expertise, information is in the air all around us. Satellite technology has filled the air with information signals. Not only television and radio, but also geographic positioning systems (GPS) and geographic information systems (GIS) now fill the spectrum of possible uses of the air. The combining of time, place, and image will continue to drive the invention of new technologies that may not only tell us where we are and when, but why.
Augmented reality and virtual reality may prove to be the most useful and most powerful tools of simulation. The notion of "augmented/virtual reality registration" may be the most crucial concept of the twenty first century, if only we can find a way of preserving it.
Milgram, P. and F. Kishino, H. Takemura, A. Utsumi. 1994. Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum
SPIE (International Society for Optical Engineering) -- Vol. 2351, Telemanipulator and Telepresence Technologies, 1994.
Post, E. and M. Orth . 1997. Smart Fabric, or Washable Computing. Digest of Papers, IEEE International Symposium on Wearable Computers, Cambridge, Massachusetts.
URCS Augmented Reality Home Pages - Introduction
3-D Technology Blends Fact and Fantasy
Augmented Reality Page
CNN - Today/Tomorrow - Is it real, or it is augmented reality? - March 20, 1997
Wearable Computing Intro Page
ALIVE -- Artificial Life Interactive Video Environment
University of Kansas
United States Institute for Theater Technology
VR Home Page
Time and Bits: Managing Digital Continuity
BEN H. DAVIS
Davis, B. and Margaret MacLean, 1998. Time and Bits: Managing Digital Continuity. Getty Information Institute, Getty Conservation Institute, LongNow Foundation, Los Angeles, California.
Davis, B. 1997. The Future of the Past. Scientific American Magazine. Vol.277. No.2 (Aug.): 69-72.
Davis, B. 1996. The Culture Machine: Science and Art on the Web, Scientific American Magazine. Vol. 275,. No.2 (Aug.): 84-88.
Davis, B. 1995. The Gallery in the Machine, Scientific American Magazine. Vol 272. No.5. (May): 108-110.
Davis, B. 1996. Teacher of the Future, Journal of the American Society for Information Science Vol 47, No. 11, (Nov): 849-853.
Davis, B. 1994. Digital Museums, August, Aperture Magazine, No. 136. (Summer): 68-70.
Davis, B. 1995. Wheel of Culture in Sociomedia: Multimedia, Hypermedia, and the Social Construction of Knowledge, E. Barrett, ed. MIT Press, Cambridge, MA.
Benjamin H. Davis is the Program Manager for Communications at the Getty Information Institute at the Getty Center in Los Angeles, CA. The Communications Program is concerned with digital publication, digital design, and digital communications. The Communication Program produces all of the print and electronic materials for the Institute as well as designing production and archiving systems.
Davis came to the Information Institute
in 1995 from the Massachusetts Institute of Technology Center for Educational Computing Initiatives (CECI) were he was a Research Associate, Manager of the AthenaMuse Consortium, and Manager of the Project Athena Visual Computing Group (1987-1991). He was also an Instructor at the MIT Media Lab, a Fellow at the MIT Center for Advanced Visual Studies, and a lecturer in the MIT Visual Arts Program. He has lectured and published widely on education, the arts, and technology.