Visual processing in the human brain provides the data both for perception and for guiding motor actions. It seems natural that our actions would be directed toward perceived locations of their targets, but it has been proposed that action and perception rely on different visual information [1, 2, 3, 4], and this provocative claim has triggered a long-lasting debate [5, 6, 7]. Here, in support of this claim, we report a large, robust dissociation between perception and action. We take advantage of a perceptual illusion in which visual motion signals presented within the boundaries of a peripheral moving object can make the object’s apparent trajectory deviate by 45 degrees or more from its physical trajectory [8, 9, 10], a shift several times larger than the typical discrimination threshold for motion direction [11]. Despite the large perceptual distortion, we found that saccadic eye movements directed to these moving objects clearly targeted locations along their physical rather than apparent trajectories. We show that the perceived trajectory is based on the accumulation of position error determined by prior sensory history - an accumulation of error that is not found for the action toward the same target. We suggest that visual processing for perception and action might diverge in how past information is combined with new visual input, with action relying only on immediate information to track a target, whereas perception builds on previous estimates to construct a conscious representation.