Integrating Prediction and Action
Our knowledge of physics can only get us so far. Because our models of the world are noisy and imperfect, we cannot know for sure how our plans will turn out until we act. But taking an action requires more effort or cost than simply forming a simulation, and so we must decide where to gather information from: inexact but cheap simulations, or precise but costly actions. If we decide to act and do not accomplish our goal, we glean detailed information not just that it failed, but how it failed and whether our plan was reasonable or should be discarded. This capability to augment our beliefs by observing the outcome of our actions allows us to intelligently and efficiently search through an action space to form our plans for interacting with the world. In this work I study the way we integrate information both from internal simulations and external observations, and how we use that information to update our plans and select our actions.
- The Tools challenge: Rapid trial-and-error learning in physical problem solving
- Learning to act by integrating mental simulations and physical experiments
- Differentiable physics and stable modes for tool-use and manipulation planning
- Abstract strategy learning underlies flexible transfer in physical problem solving