The Society for Computer Simulation
International
Virtual Worlds and Simulations Conference,
San Francisco, January 17-20, 1999
CULTURAL MEMORY
Manager, Electronic Publications
Getty Trust Publications
1200 Getty Center Dr.
Los Angeles, CA 90049
Key words:
Augmented, Culture, Longevity, Polychronic, Monochronic
ABSTRACT
In many
respects a "virtual world" is a time machine. Not in some science
fiction context, but rather as a kind of content-clock that is inhabitable.
These worlds are both monochronic and polychronic, that is, they allow the
participant to interact with information in motion and that motion can be a
single "clockwise" monochronic direction, or multiple layers of
information moving at different speeds — polychronic - as the user calls on
them. These aspects of visualization are intoxicating in their possibilities.
Add to these prospects "augmented reality" - using computer
information as an overlay to the real world rather than immersive graphical
environments - and you have even more possibilities for complex clocks. What is
important to keep in mind is that these information conditions are based on a
complex ephemeral medium, computer data. To maintain models of this nature over
time is exceedingly problematic because not only must the data, the formats,
and the storage media and display system be preserved, but also the modes of
user interaction must somehow be inventoried. The problem space for these
"content clocks" as cultural memory repositories is fascinating. The
implication that these time machines may not survive their own complexity is equally
of concern.
CLOCK-WISE
Artists and
scientists have always grappled with the dichotomy of being and knowing,
experience and our understanding of it. The use of instruments to enhance or
immediately inform experience have been under development forcenturies. Time,
vision, and sound have always been personal experiences we continue to try to
enhance.
Take time for
instance.
In 1762, John
Harrison designed the marine chronometer that led to the development of the
portable clock. In 1907, Louis Cartier was commissioned by French aviator
Alberto Santos-Dumont to create a clock with a wristband, the modern
wristwatch. And lest we forget, we occasionally look at our watches, but seldom
"watch" them. The term watch was used to describe a person who walked
the city streets with a portable clock, the night watchman.
The current
excitement about "virtual reality" (VR) stems mostly from the notion
that the imagination, in all its layered interactivity and monochronic and
polychronic time dimensions can be modeled by computers. Instead of viewing the
model externally like animation, however, we can be inside the model by using
an immersive display system - we can "wear the model". The key
ingredients - high quality three dimensional animated computer graphics, real-time
response from an interactive system, and immersive display systems make virtual
reality a computer controlled experience of different slices of time.
The VR system
must be sensitive to the movement of the user in order to "redraw
itself" in response to that movement. The use of this technology has
created some stunning simulations of flying though the air, through outer
space, through the human body. It has brought us in contact with everything
from artist’s perceptions of imaginary sculpture to the reconstruction of
ancient architectures. But in each case we are forced to leave the real world
and enter the time dimensions of the computer.
There is,
however, a growing field of research that is concerned with something other
than artificial "virtual" time and space. This line of inquiry asks
-What if there was a composite digital/analog experience available? What if we
could add information to the real world in order to maximize performance or
enhance perception without leaving the world? And what if we could do this as
easily as wearing glasses or wristwatches or headphones?
Augmented
reality research strives to keep the user in the real world and add or
manipulate information in a seamless way. Paul Milgram (Milgram et al. 1994) at
the University of Toronto Mechanical Engineering Department has described a
continuum in which the real world and the virtual world are at opposite ends of
the spectrum.
Somewhere in
the middle is an area of "mixed reality". Augmented reality is closer
to the real world end with the dominant interaction being with something
tangible that is enhanced with computer information. To further complicate
matters, Milgram has coined the term "augmented virtuality" to
describe systems that are artificial with some elements like texture mapping to
make them more realistic.
There are a
number of approaches to augmented reality that require different mechanisms to
combine digital information with analog experience. These mechanisms take a
variety of approaches to one of the key limitations of augmented reality,
registration. To render digital information and fuse it with experience is not
an easy technical task. Errors in registration will make the process not only
confusing but useless.
Registration,
then, can mean anything from making relevant digital information available at
key moments of experience (timing), to visually registering computer animation
with real events (visioning), to some combination of time and perception.
Augmented reality can be facilitated by personal instrumentation, computer-controlled
space, or some combination of instruments and spaces. The final result should
be an experience that is not usually apparent to the human senses, like looking
through your glasses and seeing the time and temperature and your next
appointment.
SMART PLACES
Instead of
wearing the instruments that generate digital information what if the place you
were in could somehow sense the information you needed at a particular moment?
A number of researchers including the MIT Media Lab are doing work on "Smart
Rooms". These are spaces equipped with sensors, cameras, and microphones
that can interpret gestures, speech, and movement in order to deliver
information at the appropriate moment. These technologies are already being
tested in areas like security (the room knows who is in it) to use in cars that
know when the driver is trying to turn or stop.
Since the late
1960’s museums have made use of "audio tours" to augment the reality
of visitors. We have quickly come to take this use of technology for granted as
a staple of museums but it wasn’t so very long ago that the only
"enhancement" in a museum were the labels on the wall. Now, with
multi-tracking and digital audio, the museum visitor can visit Tahiti in the
headphones in front of that Gaugan painting. The sophistication of smart
museums is becoming taken for granted because of their use of new technologies
for augmenting experience.
Another very
powerful example of smart places are the galleries of the J. Paul Getty Museum
in the new Getty Center in Los Angeles. Museum Director John Walsh declares
that the mission of any museum is to provide light with the understanding that
light always changes. This seemingly simple statement actually represents a
very complex concern for the what the experience of art means. Artists work
with physical materials that manifest qualities of color and light. When you
"freeze" a painting with a particular wavelength of light provided by
electric lights you narrow the quality of the experience to a very thin slice.
The work was created in sunlight, light that changes throughout the day, thus
giving the artist enormous latitude in creative opportunity. The final work is
a summary of all the light the artist used in the process of creating the
painting. Thus exhibiting the painting in some semblance of that light- process
gives the viewer a closer communication with the artist’s feelings and
intentions. The art is alive in a sense, not deadened by a single kind of
light. What the galleries of the Getty represent in both a poetic and technical
sense is a time machine — one aspect of it being to let the light of the
present illuminate the work but also to give you sense of what the work might
have looked like at the time of its creation as well.
The galleries
of the museum are lit by skylights of heavy, UV filtered glass that allow about
thirty percent of the external light into the rooms. The shape, size, and wall
color of the rooms also affect the light quality. Because of the often clear
brilliant light of the hilltop near Los Angeles where the museum is built, the
designers wanted visitors to be able to see the sky so the skylight glass is
clear. Working above the glass are are computer controlled louvers that orient
themselves automatically based on the best possible light for the time of day
and time of year. Because they did not want visitors distracted by constantly
shifting louvers, they created elaborate look-up tables from data collected by
photocells measuring the light every hour of every day from six by ten-foot
wall areas in the rooms. These tables tell the louver inclinometers how to
orient themselves on a schedule that subtly shifts them at prescribed times
during the day.
When light
sensors that constantly take readings from the gallery walls detect below 120
lux, quartz halogen lights with "blue" filters come on (they account
for half of the lights in the gallery) because that wavelength mixes better
with the dwindling daylight. They fade up very slowly, taking thirty seconds to
reach full intensity. Visitors looking at the artwork will not notice the
shift. As daylight continues to fade and the light levels again begin to reach
120 lux, the remaining red filtered halogen quartz lights come on because so
little daylight is left. At this point the quartz lights are simulating a
standard measure of artificial daylight.
Visitors have
the extraordinary experience of never seeing a painting in the same light twice
but they will never notice the technology orchestrating the experience.
This example of
augmented reality goes to the heart of the use of technology to seamlessly
enhance experience. No one who visits the museum is aware that they are in a
"smart room". The vibrancy of the artwork is the only indication that
something is special, that experience has been enhanced. The attention to the
specificity of light, the use of computational technology to lend clarity and
the invisible nature of the situation makes these rooms magical in sense. The
experience is a not a technological "gee whiz" experience. It is more
akin to a communion with content, in current techno-talk. For artists and
historians, however, it is simply the right way to be with art. Visitors may
not know how any of this has been done, but they do know that the artwork at
the Getty is memorable.
What the use of
technology in museums may be telling us is that the memorable experience may
not be just about learning, but about how powerfully that learning takes place
- aspects of theater may be a key to imparting information.
SMART
THEATER
In some sense
it is inevitable that the combination of personal instrumentation and smart
spaces will lead to theater. Sponsored in part by the United States Institute
for Theater Technology in Syracuse, New York, the University Theater of the
University of Kansas has embarked on series of experiments in virtual and
augmented reality. These productions are attempting to fuse virtual reality
graphical environments with live stage production. This has meant that the
audience wears virtual reality headgear that creates graphical environments for
live stage action.
Their latest
production was "Tesla Electric" that chronicled the life and career
of inventor Nikola Tesla. The play combined vaudeville, carnival sideshows, and
magic acts into an industrial morality play. This particular work utilized
complex computer generated scenography rather than the three dimensional VR
systems
Static computer
generated scenes were created with lavish detail and texture. Live actors
played in front of a panoramic triple screen on which the scenery was
rear-projected. Stereo imaging was used so that the audience saw the scenes in
3-D with the aid of special 3-D "sunglasses". Graphical animated
characters were also introduced using the projection and 3-D combination.
This hybrid of
physical theater and graphical augmentation marks a genuine departure from
traditional theater. The meaning of a work no longer resides in the
interpretation of an author’s script, but rather meaning may come from the
"rendering" of the play in the digital integration of real players,
animation, lighting, and scenographics. The only thing missing from the production
might be the use of technology in the costumes.
SMART
OUTFITS
One of the more
interesting areas of current research at the MIT Media Lab is in "wearable
computers". Starting with the notion that the "personal
computer" is a misnomer because computers are much more like furniture
than they are anything "personal", this group of researchers has
embarked on something more ambitious than putting the furniture in your lap
(the lap-top computer). What they are anticipating are faster, cheaper,
smaller, more powerful computer chips that will make it possible to wear
information processing technology in a completely unobtrusive way. You might
want to do this to increase or specialize memory, enhance a situation, or
communicate with others who have similar interests.
A fundamental
example of this is "smart fabric". Researchers at MIT (Post and Orth.
1997) are investigating adapting ancient techniques of weaving precious metals
to modern electronic circuitry. This electronic fabric can distribute data and
power, and perform touch sensing. Experimenting with silk organza, the warp is
a plain silk thread while the weft is a silk thread wrapped in thin copper
foil. The fabric is very conductive and has been fashioned into musical
keyboards and graphic input surfaces by embroidering it onto clothes. The
spacing of fibers allows them to be addressed separately much like a familiar
computer ribbon cable. Circuits like busses can then be created to interact
with various digital devices like microcontroler systems that can detect proximity
and touch — keyboards and touchpads. These have been tested with minature MIDI
synthesizers thus making a wearable electronic instrument.
The notion of
wearable (and washable) electronic circuitry that empowers the user to slip
into a computer, a musical instrument, or communication system makes the cell
phone seem like cumbersome hardware. The implications of sensory garb for
augmented activities is much like wearing scuba gear for the mind.
VANISHING
VIRTUAL
The ability to
use digital rendering technologies in new and exciting ways is without
question. The nagging question that is only beginning to be asked is "How
do we preserve this new medium?" Almost everyone assumes that digital
files and recording media are somehow more archival than paper. The truth is
far from that. Strategies of migrating data from one format to another, having
it constantly accessible over networks, or keeping CD ROM’s in temperature
controlled vaults appear to be the easy answers.
To maintain
virtual and/or augmented models of this nature over time is exceedingly
problematic because not only must the data, the formats, and the storage media
and display system be preserved, but also the modes of user interaction must
somehow be inventoried. In the spectrum of digital preservation problems that
roughly corresponds to static documents at the simple end to moving images in
the middle and interactive static and motion images at the far end, virtual and
augmented realities are almost out of bounds in terms of knowing how and what
to preserve. The problem space for these "content clocks" as cultural
memory repositories is fascinating. The implication that these time machines
may not survive their own complexity is equally of concern.
CONCLUSION
Enhancing
experience by augmenting reality with digital technology will become as common
place as putting on your digital glasses and headphones. It can also be
observed, without scientific or engineering expertise, information is in the
air all around us. Satellite technology has filled the air with information
signals. Not only television and radio, but also geographic positioning systems
(GPS) and geographic information systems (GIS) now fill the spectrum of
possible uses of the air. The combining of time, place, and image will continue
to drive the invention of new technologies that may not only tell us where we
are and when, but why.
Augmented
reality and virtual reality may prove to be the most useful and most powerful
tools of simulation. The notion of "augmented/virtual reality registration"
may be the most crucial concept of the twenty first century, if only we can
find a way of preserving it.
REFERENCE
Milgram, P. and
F. Kishino, H. Takemura, A. Utsumi. 1994. Augmented Reality: A Class of
Displays on the Reality-Virtuality Continuum
SPIE
(International Society for Optical Engineering) -- Vol. 2351, Telemanipulator
and Telepresence Technologies, 1994.
http://vered.rose.toronto.edu/people/paul_dir/SPIE94/SPIE94.full.html
Post, E. and M.
Orth . 1997. Smart Fabric, or Washable Computing. Digest of Papers, IEEE
International Symposium on Wearable Computers, Cambridge, Massachusetts.
http://www.wearcam.org/smart_clothing/node6.html
GENERAL
URCS Augmented
Reality Home Pages - Introduction
http://www.cs.rochester.edu:80/u/vallino/research/AR/introduction.html#Section1.
3-D Technology
Blends Fact and Fantasy
http://tracker.usc.edu/~ykcho/star-LATimes030397.html
Augmented
Reality Page
http://www.cs.rochester.edu:80/u/vallino/research/AR/
CNN -
Today/Tomorrow - Is it real, or it is augmented reality? - March 20, 1997
http://tracker.usc.edu/~ykcho/AR/Press/CNN/cnn9703.html
MIT
Wearable
Computing Intro Page
http://lcs.www.media.mit.edu/projects/wearables
Smart Rooms
http://vismod.www.media.mit.edu/vismod/demos/smartroom/ive.html
Augmented
Reality
http://lcs.www.media.mit.edu/projects/wearables/augmented-reality.html
Augmented
Memory
http://lcs.www.media.mit.edu/projects/wearables/augmented-memory.html
Augmented
Reality
http://n1nlf1.media.mit.edu/computing/augmented-reality.html
ALIVE --
Artificial Life Interactive Video Environment
http://alive.www.media.mit.edu/projects/alive/
THEATER
University of
Kansas
http://kuhttp.cc.ukans.edu/~mreaney/index.html
http://www.ukans.edu/~mreaney/tesla/
United States
Institute for Theater Technology
http://www.ffa.ucalgary.ca/usitt/
WINGS
http://kuhttp.cc.ukans.edu/~mreaney/wings.html
VR Home Page
http://kuhttp.cc.ukans.edu/~mreaney/index.html
GETTY
Time and Bits:
Managing Digital Continuity
http://www.gii.getty.edu/timeandbits
Getty Center
http://www.getty.edu
BEN H. DAVIS
Davis, B. and
Margaret MacLean, 1998. Time and Bits: Managing Digital Continuity. Getty
Information Institute, Getty Conservation Institute, LongNow Foundation, Los
Angeles, California.
Davis, B. 1997.
The Future of the Past. Scientific American Magazine. Vol.277. No.2 (Aug.): 69-72.
Davis, B. 1996.
The Culture Machine: Science and Art on the Web, Scientific American
Magazine. Vol. 275,. No.2 (Aug.): 84-88.
Davis, B. 1995.
The Gallery in the Machine, Scientific American Magazine. Vol 272. No.5.
(May): 108-110.
Davis, B. 1996.
Teacher of the Future, Journal of the American Society for Information Science
Vol 47, No. 11, (Nov): 849-853.
Davis, B. 1994.
Digital Museums,
August, Aperture Magazine, No. 136. (Summer): 68-70.
Davis, B. 1995.
Wheel of Culture
in Sociomedia: Multimedia, Hypermedia, and the Social Construction of
Knowledge, E. Barrett, ed. MIT Press, Cambridge, MA.
Benjamin H.
Davis
is the Program Manager for Communications at the Getty Information Institute at
the Getty Center in Los Angeles, CA. The Communications Program is concerned
with digital publication, digital design, and digital communications. The
Communication Program produces all of the print and electronic materials for
the Institute as well as designing production and archiving systems.
Davis came to
the Information Institute
in 1995 from
the Massachusetts Institute of Technology Center for Educational Computing
Initiatives (CECI) were he was a Research Associate, Manager of the AthenaMuse
Consortium, and Manager of the Project Athena Visual Computing Group
(1987-1991). He was also an Instructor at the MIT Media Lab, a Fellow at the
MIT Center for Advanced Visual Studies, and a lecturer in the MIT Visual Arts
Program. He has lectured and published widely on education, the arts, and
technology.