The Nature of Physical Representations and Simulations

Making predictions in real-world situations is difficult, as we must be able to deal with scenes that have a large number and variety of objects in many different configurations. In addition, physics is a chaotic system, where imperceptible details can influence the outcome: a gust of wind can blow a frisbee off course, or a ball hitting a rough wall can take an unpredictable bounce. Despite these challenges, people are able to produce relatively accurate predictions and inferences quickly enough to act on the world. I study the approximations this system makes to be able to work so efficiently and so robustly.

In my research, I have found that the mind can handle these challenges by sampling from a small set of stochastic physical simulations. Adding noise to our simulations can help smooth over the imperceptible features that cause physical outcomes to diverge. Sampling a set of these simulations allows us to understand the range of possible outcomes the world might take. And we often can get away with using just a small number of these samples to efficiently make and update predictions over time.

But the world is also complex, with many more items than we can possibly represent in working memory at one time. I have also studied the ways in which we simplify our representations for physical reasoning, such as by selectively encoding only relevant objects within a scene, or using simple shape representations for objects.

Inference is a more challenging problem to solve than prediction: if we have a mental model of how the world works it is easy to transform the initial state of the world into a hypothesized future state; however, it is much more computationally difficult to reverse the process to infer the initial state from the final state. I have found that we make and update these inferences by keeping track of a small number of hypotheses that are updated based on how well observations make the physical behavior expected under each hypothesis. This inference algorithm can be approximated in a neurally plausible system, providing a testable hypothesis for how the brain performs this information processing.

Related papers:

Avatar
Kevin A Smith
Research Scientist in Brain and Cognitive Sciences