What do people learn when they repeatedly try to solve a set of related problems? In a set of three different exploratory physical problem solving experiments, participants consistently learn strategies rather than generically better world models. Participants selectively transferred these strategies when the crucial context and preconditions of the strategy were met, such as needing to “catapult”, “support”, “launch” or “destabilize” an object in the scene to accomplish their goals. We show that these strategies are parameterized: people can adjust their strategies to account for new object weights despite no direct interaction experience with these objects. Taken together, these results suggest that people can make use of limited experience to learn abstract strategies that go beyond simple model-free policies and are instead object-oriented, adaptable, and can be parameterized by model-based variables such as weight.