Image Learning

Ben Howell Davis

Leave Comments

Technology and Education Conference, Ekpedeftiria Doukas S.A., Athens, Greece, 1991.

Copyright, 1991.

Plato, in all his striving to imagine an ideal training school, failed to notice that Athens was a greater school than any university even he could dream up. In other words, the greatest school had been put out for human use before it had been thought out. Now this is especially true of our media.

Marshall McLuhan, Understanding Media, 1964. McLuhan, Understanding Media, (New York: McGraw-Hill),p. 49.

Ways of Teaching

New technologies very often verify older ideas about knowing and learning, but as McLuhan points out, they are moving so quickly we hardly have time to notice. Today the computer is making it possible to combine all the media types present in our electronic world. Television, radio, animation, and print in the forms of video, audio, graphics, and word processing can be manipulated by software to perform together like a "memory theater". Around 516 BC Simonides invented the "art of memory". He equated the methods of classical poetry and painting and taught that these forms, acted upon by memory, were intense visualization. In order to demonstrate this, spaces were designed with visual details that would elicit lines of poetry to the initiated. Carefully placed windows and small openings would direct light onto these details. The seminar topic of 516 BC was " Visual Education and Memory Theater Technology."

We live in a world of images. The electronic transmission of video (sound and picture) has literally wrapped the planet in an envelope of imagery. To navigate this realm we have relied on the passive medium of television. All of us have become image interpreters. Education is not only an interpretive situation, it is a translation process as well. To understand how we interpret and translate images it becomes necessary to create tools that allow for more than the passive receiving of audio/visual information.

We need composers.

Compositional tools like multimedia technology begin to allow for an active role in image learning. These tools make possible the connection of categories of imagery into a coherent body of audio/visual knowledge. By examining and linking spatial images (maps, charts), graphic images (graphs, diagrams), pictorial images (painting, sound, photograpy), narrative (film, video), and new dimensions like simulation we begin to create a context for the study of multimedia representation. We can begin to discuss "image logic, image syntax, and image retention."

What becomes quickly apparent is that the transitional structures that move an idea through these categories are important to image learning. How could a map using graphics to reveal a process be enhanced by photography and then animated (motion/audio) to approach a simulation? How can the simple act of sketching become a fundamental learning skill just as reading and writing are today? Is sketching done with a pencil, a camera, a computer? When we say that we want to "put something in perspective" do we mean it literally? What do shifts in perspective sound like?

We are learning that any subject is interesting and learnable if the mind's eye/ear is engaged. Multimedia computer technologies simply give us a good "eye for the obvious". The use of imagery as a teaching tool is our oldest educational experience. Imagery and its inherent interpretive nature allows us new "ways of teaching". By encouraging a multimedia literacy we begin to understand that illustration is not a secondary function of facts but is at the very core of understanding.

In a paper by Harvard cognitive psychologists Stephen Kosslyn and Christopher Chabris it is argued that:

After recognizing an observed picture, one has matched it to a stored representation of the pictured object or scene, and knows only that it is familiar. This matching process is exclusively visual, and it does not result in a name or any other information. In contrast, after identifying an object, one has access to the entire range of information associated with it, including sounds it makes, its texture, the categories to which it belongs, some specific exemplars of its type, and so on. When one has identified an object, one has access to a multimodal set of information.

S. Kosslyn and C. Chabris, Naming Pictures, Journal of Visual Languages and Computing, Volume 1, Number 1, 1990, p.77.

In order to approach this notion of "a multimodal set of information" that is employable as a teaching/learning mechanism, it is important to understand the relationship between image types and the concept of visualization.

Visualization is a current buzz word. The number of journals and conferences on the subject multiplies annually. The basic questions of whether the user of visualization can effectively distinguish between model and reality and whether the "mediated experience" actually diminishes the sense of real discovery will never be resolved.

These questions are attended by others: How does spatialization of information relate to visualization? How does image perception work? How do we conceptualize with vision and sound? What elements of vision affect the other senses? What is the role of the illustration? What is a multimedia simulation? Is seeing believing/learning? A good deal of contemporary psychological research is leaving the verbal behavior tradition to investigate the questions of how visualization works.

This ability to model with "whatever sense is in highest definition" for the investigator is key to visualization as a concept. The sightless person uses sound as the visualization sense. Someone who is deaf as well as blind must use touch as the primary instrument of visualization. Because we have used sight as our basic tool of survival it has become the prejudiced sense for visualization. We discuss visualization but what we really mean is whatever triggers imagination, whatever renders understanding . Perhaps a better term for multimedia visualization would be cognitization. The clarity of understanding is dependent on the rendering of pattern that can be dissected and reassembled according to need.

The reasons for visualizing information are to remember it and to be able to use what is remembered. The human memory and its need for pattern recognition is evident in our constructions for "getting there and back".

We moderns who have no memories at all, like the professor, employ from time to time some private mnemotechnic not of vital importance to us in our lives and professions. But in the ancient world, devoid of printing, without paper for note taking or on which to type lectures, the trained memory was of vital importance. And the ancient memories were trained by an art which reflected the art and architecture of the ancient world, which could depend on faculties of intense visual memorisation which we have lost.

Francis Yates, The Art of Memory, (Chicago: University of Chicago Press, 1966), p. 4.

In order to have a structure for trying to understand the ideas of image logic, image syntax, and image retention I would like to categorize audio/visual information. These categories, of course, naturally overlap and intersect. The use of a kind of syntax to connect them when they do not seem to overlap or intersect will be discussed later. First, the categories of imagery:

Spatial

Graphic

Pictorial

Narrative

Simulation

SPATIAL configuration is an element of all representations. The spatial representation is concerned with making space visible or audible. The most common example that each of us encounters is the map. The map is both a visual design of space and an article of faith. We tend to believe that a place actually exists where a map tells us it is.

The origin of the map is lost to history. No one knows when or where or for what purpose someone got the idea to draw a sketch to communicate a sense of place, a sense of "here" in relation to "there". It must have been many millennia ago, probably before written language. It certainly was long before the human mind could conceive of the worlds beyond shore and horizon, beyond Earth itself, that could be embraced through mapping.

J.N. Wilford, The MapMakers, (New York: Vintage Books, 1981), p. 7.

Another example of early spatial visualization that we are all familiar with is the constellations.

Constellations are the invention of human imagination, not of nature. They are an expression of the human desire to impress its own order on the apparent chaos of the night sky. For navigators beyond a sight of land or for travellers in the trackless desert who wanted signposts, for farmers who wanted a calendar, and for shepherds who wanted a nightly clock, the division of the sky into recognizable star groupings had practical purposes. But perhaps the motivation was to humanize the forbidding blackness of night.

Ian Ridpath, StarTales, (New York: Universe Books, 1988), p. 1.

We are reminded also that the "face" of time is the visual representation of duration and that the calendar is the visual representation of the seasons.

The sundial, water clock, and hour glass were all designed to "show" the passing of time, by gradual shadow across a dial, of water from a bowl, of sand through a glass...The needs of mechanical timekeeping, the logic of the machine itself, imposed new feeling. Instead of being synonymous with repeated cycles of the sun, which varied as the cycles of the seasons commanded, or with shorter cycles of other flowing media, time was not measured by the staccato of a machine.

Daniel Boorstin, The Discoverers, (New York: Random House, 1983), p. 38.

The compass also is a visual tool for spatial navigation.

Sailors, leaving behind their crude sketch maps, rough diagrams of familiar places, now could take along true maps, which oriented them to the whole world...the compass provided a worldwide absolute for space comparable to that which the mechanical clock and the uniform hour provided for time...From the very nature of our spherical spinning planet, the marking of time and the marking of distance were inseparable. When you moved great distances from your home out into the uncharted great oceans you could not know precisely "where" you were unless you had a way of finding precisely "when" you were.

Daniel Boorstin, The Discoverers, (New York: Random House, 1983), p. 46.

The history of the map (both for Earth and sky), the clock, the compass and the calendar are all equally important visualizations that continue to be refined to give us a sense of space. Contemporary mapping techniques using satellite imagery and image processing software have given us new visions of the Earth and the heavens. These new devices and techniques have merged the map, the clock, and the compass into digital information that can be processed with a computer to reveal not only a sense of "here and there" but environmental conditions dependent on changes in the season and human interaction.

Acoustic engineering as a locating technology also makes sound an important player in spatial imaging. The inventions of sonar, seismic measuring of earthquakes, ultrasound, and acoustic microscopy are all technologies that exchange the eye for the ear and have made visualizing phenomena possible.

We are able to use our ears to identify the direction from which a particular sound is coming, even if we cannot see the source. From antiquity, fishermen navigating near a coast in dense fog have made loud noises and listened for echoes to help them guess their distance from known points, thus taking advantage of the relatively low velocity of sound.

Seeing with Sound, The Encyclopedia of Modern Technology, (Boston: G.K. Hall, 1987), p.65.

The interplay of spatial representations and the techniques developed for their portrayal have played a central role in our attempts to visualize the world.

GRAPHS are visual representations of abstracted properties and relationships. They allow the eye to quickly grasp the dynamics of change. At its most basic, the simple graph is composed of graphics (points and lines) and geometry (relative positions of the points and lines) that together visualize a relationship. As the summer goes on (x as time axis) the temperature goes up (y as the heat axis). As they become more complex and use greater numbers of variables (heat, time, number of hours of daylight, etc.) their image becomes more articulate. When combined with techniques like color, shading, geometric organization, and animation they become even more powerful visual signals.

Historically this way of visualizing events has been used to explain quite complex situations such as Napolean's march and retreat from Russia, the classic of Charles Joseph Minard (1781-1870), the French engineer, which shows the terrible fate of Napolean's army in Russia. Six variables are plotted: the size of the army, its location on a two-dimensional surface, direction of the army's movement, and temperature on various dates during the retreat from Moscow. It may well be the best statistical graphic ever drawn. (Edward Tufte, The Visual Display of Quantitative Information, (Cheshire, Conn: Graphics Press, 1983), p. 41.

Today, the term "scientific visualization" in computer science has commonly come to mean complex computational models which produce simulation data that require geometric based algorithms for interpretation. What this means is that numbers become pictures and/or sounds. Radio telescopes scanning the heavens produce more numerical data than can ever be examined point by point. If the points are plotted graphically, however, relationships can be understood. The use of very sophisticated three-dimensional moving graphic images produced by supercomputers is allowing scientists to literally see phenomena that without graphic representation would remain masses of numbers. (The Mechanical Universe and Beyond, produced by the California Institute of Technology and the Annenberg/CPB Foundation, is a fine example of this kind of computer graphic visualization.) The Lorenz Transformation program produced for this series allows students to actually see relativity in three-dimensional animation.

PICTORIAL representation concerns expression. Expression may be the artist's attempt to translate impressions of the world into an audio/visual representation. Pictorial representations may also be a depiction of the invisible world such as an expression of emotion, thought, temporal abstractions, or symbolic associations.

Changes in technology have always affected artistic technique and allowed the artist ever increasing possibilities for representing emotions and ideas about the world. From the earliest cave painting to medieval religious subjects, to Renaissance perspective there is a fascinating catalog of ways in which we have refined the experience of understanding more carefully. This revolution in visualization coincided with developments in optics (the telescope, microscope, eye glasses) and in acoustics (musical instruments, opera, etc).

Suddenly viewers were looking, not at flat metaphorical images, but into worlds that opened and vanished into infinity. And because of technologies like printing they were seeing many many images.

The printing of pictures, however, unlike the printing of words from movable types, brought a completely new thing into existence - it made possible for the first time pictorial statements of a kind that could be exactly repeated during the effective life of the printing surface...it becomes obvious that without prints we should have very few of our modern sciences, technologies, archeologies, or ethnographies - for all are dependent, first and last, upon information conveyed by exactly repeatable visual or pictorial statements."

William Ivins, Prints and Visual Comunication},(Cambridge, MA: MIT Press, 1986) p.10.

The invention of modern still photography, from its humble beginnings in shadow play and silhouettes to stereoscopic imagery to today's astonishing holographic images, provides pictorial visualization with powerful tools. The use of the cutaway drawing, the anatomical sketch, the representation of animal and plant species, the newspaper picture, and the varieties of popular advertising imagery are all characterized by the term pictorial.

Sound imagery also plays an important part in pictorial understanding. R. Murray Shafer refers to the study of "acoustic design". He defines this as:

an interdiscipline in which musicians, acousticians, psychologists, sociologists and others would study the world soundscape together in order to make intelligent recommendations for its improvement. This study would consist of documenting important features, of noting differences, parallels and trends, of collecting sounds threatened with extinction, of studying the effects of new sounds before they are released into the environment, of studying the rich symbolism sounds have for man and of studying human behavior patterns in different sonic environments in order to use these insights in planning future environments for man. Cross-cultural evidence from around the world must be carefully assembled and interpreted.

R. Murray Shafer, The Tuning of the World, (Philadelphia: University of Pennsylvania Press, 1980), p. 4-5..

In my own experience with an ornithologist doing a bird count of an endangered wetland, I realized that he was doing fifty percent of his recognition of bird species by listening to bird calls and was fully confident of his identifications without ever actually seeing the birds.

Sound and pictures create powerful images that evoke memories and emotions. Sound adds a new dimension to visual information that can add context, ask questions, or tell stories. This additive process in image making leads us to the multiple image realm.

Putting two images together always creates comparison. The images usually appear to be similar in some way or completely dissimilar. Either way, we attempt to make some sense of the juxtaposition. Sequencing of images in order to understand an idea or phenomena would be characterized as NARRATIVE.

The creator of a narrative visualization relies on our natural tendency to find a connection between images. A movie, for example, depends on a variety of narrative techniques to which the audience will respond. One of these techniques is the use of cuts to indicate a change of place or time. Another is the use of camera angles which point upward to powerful characters and downward towards characters in weak positions. These techniques rely on the director's knowledge that the presentation of about thirty still frames per second will cause the viewer to judge the contents of the images in motion. Narrative visualizations occur throughout history and across media. The panel paintings of 15th century Siena, Greco-Roman mosaics, and the early French tapestries are but a few of the examples. Using knowledge of the chosen medium, its accepted conventions, and human nature, the task of the narrator is to provide the elements of narrative glue through the creation, positioning, and ordering of images.

Narrative techniques for visualizing phenomena are as old as storytelling itself. Hieroglyphs strung together recorded the myths and legends of the Egyptians, the codexes of the Mayan civilizations were pictographic records of events, Chinese characters are line drawings of complex ideas. Though each constellation in the night sky is singular in its visual representation, collectively they are woven into the rich fabric of mythological events. "Perseus flies to the rescue of Andromeda, Orion faces the charge of the snorting bull, Bootes herds bears around the pole, and the ship of the Argonauts sails in search of the golden fleece. (Ian Ridpath, Star Tales, p.10.) Changes in the seasons move the constellations to new positions in the sky, giving legends a dynamic twist which navigators know by sight.

These traditions foreshadowed the great narrative paintings of the Sistine Chapel. Stained glass cathedral windows used light as a vehicle for telling the myths of Christianity. Illuminated manuscripts integrated text and images. The advent of printing brought the picture book to the masses. The fascination with storytelling in pictures has led us technically from sequential still images to motion pictures. Early movies were only still images with slight variations moved mechanically past the eye. Eadweard Muybridge, in the 1880's, made a bet that a horse at full gallop would at sometime have all four hooves off the ground simultaneously. Or so the story goes. To prove it he made sequential photographs every few seconds of the horse galloping. When the resulting images are played in rapid succession the horse actually appears to move! From scientific studies of motion, film has become the major art form and social storyteller of our time. Our most ubiquitous form of communication, television, has taken the magic lantern of film to its extreme as a global network of information conditions. The act of using the air for storytelling has created a terrestrial overbody of electronic excitation. We literally breath images transmitted every moment, continuously, night and day. There is no time, no place, on the planet that is exempt from this information condition. Microwaves create an invisible mesh of pictures, data, and audio that enclose us in an envelope of possible points of view. Still the visual narrative, both as a technique and a technological inspiration, gives us our most plastic mode of comprehension.

Today, computer graphics begin to rival photographically-produced imagery. The possibility of a completely plastic medium capable of creating fantastic as well as realistic images in motion is upon us. This medium has brought us to understand something beyond audio/visual metaphor, the SIMULATION. Simulation techniques offer dynamic models of the world which can be investigated and experienced. Multiple points of view allow the possibility of the user manipulating them to predict the behavior of complex phenomena.

Early travel logs captured the experience of visiting far-away lands with drawings and text. The journeys of naturalists like Audubon in the 18th and 19th centuries made crude surrogate travel experiences with images of wildlife and stories of native peoples. The travel photography industry started by photographers in the late 1800's like Francis Frith, James Valentine, and George Washington Wilson produced volumes of images taken all over the world that gave the average person a small sense of "being there". This fascination stimulated stereoscopic photography and later developments in 3D film. The concept of surrounding oneself with illusion is a powerful perceptive experience. Even further beyond this is the concept of being able to manipulate "virtual reality". The most common example of this today is the flight simulator. Created to give an aviator or astronaut the experience of controlling an aircraft or spacecraft this device allows the participant to make mistakes without fear of crashing. This modern version of the 'memory theater' is interactive and has become popular as a video game entertainment in the form of auto racing simulators as well. Simulation, however, is effectively the idea behind scientific visualization. Weather modeling, for instance, uses satellite imagery of atmospheric conditions that has been digitally enhanced to react to computer programs that change variables like wind velocity and humidity levels. The resulting image of a storm can be predicted visually. Molecular modeling, diagnostic medicine, brain structure and function, geoscience, astrophysics, fluid dynamics, mathematics, robotics all make use of simulation technologies and techniques.

These categories of imagery (spatial, graphic, pictorial, narrative, and simulation) do overlap naturally. Spatial imagery like maps uses graphical representation like latitude and longitude. Pictorial information in sequences creates narrative. And narratives that use spatial constructions like wrap-around movie screens come close to simulation.

What might a set of syntax mechanisms be that could serve as linkage agents for these categories that would include both natural and arbitrary overlaps and intersections:

Appearance: The recognizable attributes of images. (Color, shape, edges, contrast, opacity, texture, etc.)

Measurement: The scale of images. (How large or small something is in relation to something else.)

Dimension: The temporal constructs embedded in images. (Showing process with audio/video, time lapse, slow motion, moving or still images that capture metamorphic processes, temporal axes in graphs. Ideas related to duration.)

Perspective: The perspectival aspects of an image according to the relative position of the viewer. (Vantage point, 1, 2 & 3-point perspective, multiple perspectives, acoustic perspective.)

Using these linkage agents it is possible to imagine using different categories of imagery to reveal complex ideas. A set of disciplines and an "interdisciplinary problem" that imagery might help solve would include:

Humanities

Science

Engineering

I chose Humanities for the obvious breadth of subject matter - everything from language to fine arts. Science is selected because of the range of subject matter and problem sets available - everything from biology to psychology. Engineering was chosen because it combines mathematics with real world problem sets - everything from mechanical design to environmental safety.

A suitable interdisciplinary question that could be asked in any teaching situation from early school through university might be:

What is the nature of human flight?

How can imagery answer this question?

Let us remind ourselves that it is not "natural" for humans to fly. There is a long history of the desire to fly that is reflected in the map - a "bird's eye" view of the landscape. Was the map a "vision" of flying?

What does the first known map look like? How were early maps made without flying over the landscape? What is the character of its "appearance"? How is this spatial image changed as new communication technologies are developed? What other images of flying are related? Is the image of the angel related? What about Da Vinci's drawings of the helicopter? When was the first balloon used to make maps? What scientific knowledge made balloons possible? How were they engineered? When were the clock and the map used together? Why? How do maps "measure"? How did maps change when airplanes were invented? How was the airplane invented? What new "dimensions" were added to maps when airplanes arrived? Do satellite photographs now represent maps? Is this a new "perspective" on the map? Are flying and map-making now inseparable? If we show a film from the space shuttle of the earth are we looking at a "narrative" map that "simulates" flying? Can we navigate using this kind of map? Can we use computers to decode the "appearance" of satellite photo maps to tell where pollution is occurring?

Obviously the look of a map and the view from an airplane have a great deal to do with each other. Was the map an inspiration for humans to fly? Was it a "visualization" of an event that would only become possible centuries after its conception? How do airplanes work? Students presented with this kind of visual problem will have to develop interpretation and evaluation skills that rely upon noticing characteristics of images. Given a variety of maps, which ones are best for understanding distance, demographics, or land use? Which ones could have been made by aerial photography? Which ones are obviously made before the airplane? What kinds of visual materials are used for designing airplanes? Can we think of design drawings as maps?

If the topic begins with human flight, an experience that is derived from an idea, not from actual physical ability, we are at liberty to try to find out where that "vision" came from. Looking at maps is an interesting place to begin. The map is spatial, uses graphical devices for location, has become increasingly pictorial with the use of photography, with added coding becomes narrative (tells us how many people live in Boston, etc.), and approaches simulation as we use film instead of still pictures to render it. Appearances can take us through these categories by simply looking at examples of different types of maps. Measurement can take us through the maps by showing how each kind of map changes scale and distance,etc. Dimension can take us through by showing how something like time lapse photography changes a map. Changes in perspective can reveal how changes in technology (higher and higher flights) change the nature of the maps. All of this, remember, is not to talk about maps but to talk about the human desire to fly and the science, technology, and engineering that have gone into realizing that vision.

The categories of spatial, graphical, pictorial, narrative, and simulation and the linkages of appearance, measurement, dimension, and perspective form an interesting scheme for teaching and understanding how imagery conveys and inspires ideas. As new technologies make the use of audio and visual imagery easy, it will become increasingly necessary to design and implement schemes for composing them. What should those schemes look/sound like?

Like Plato, we are striving to visualize an ideal academy. As educators we now see that ideas have a wide variety of entry points. Unlike Plato, we seek not the best way of teaching. Rather, we must investigate designs that point to the plural, the ever richer ways of teaching.

Bibliography

Daniel Boorstin, The Discoverers, New York: Random House, 1983.

Encyclopedia of Modern Technology, Boston: G.K. Hall, 1987.

William Ivins, Prints and Visual Comunication, Cambridge, MA: MIT Press, 1986.

Stephen Kosslyn and Christopher Chabris, "Naming Pictures", Journal of Visual Languages and Computing, Volume 1, Number 1, 1990.

Marshall McLuhan, Understanding Media, New York: McGraw Hill, 1964.

Ian Ridpath, Star Tales, New York: Universe Books, 1988.

R. Murray Shafer, The Tuning of the World, Philadelphia: University of Pennsylvania, 1980.

Edward Tufte, The Visual Display of Quantitative Information, Cheshire, Conn: Graphics Press, 1983.

Francis Yates, The Art of Memory, Chicago: University of Chicago Press, 1966.