Our perceptions are rooted in early-developing expectations about the behavior of everyday objects, which are often exposed by examining surprise responses to impossible events. However, the online mental processes that underlie physical expectations remain hidden. In two EEG experiments, we tracked people’s working memory as they watched short animations of objects moving behind an occluder. The animations showed either expected motion, or violations of object permanence. We found that object tracking was disrupted by an object appearing or disappearing in a physically impossible way: Working memory first reset, and then recovered to represent the adjusted number of items. But, when object permanence violations were explained away (by a hole behind the occluder), object tracking remained intact. Working memory even continued representing the object that disappeared in this non-violating way. Our results show how intuitive physical expectations shape online representations, and form the basis of dynamic object tracking.