The function of smooth pursuit eye movements is classically thought to be to stabilize the image of a moving object on the retina. Here we ask whether the stabilizing operation carried out by smooth pursuit operates over positions on the retina or over perceived positions. Participants smoothly pursued the invisible midpoint (Steinbach, 1976) between two Gabor patches aligned in the vertical plane and moving on parallel, oblique paths. On some trials, the Gabors’ internal texture drifted orthogonally to their envelopes’ motion direction. These moving Gabors with internal drift, also known as the double-drift stimulus, induce a large offset of the perceived from the physical position of up to several degrees. The results showed that smooth pursuit eye movements follow the illusory trajectory rather than the physical. Thus, smooth pursuit operates on perceptual rather than retinal coordinates.