Trackers and video generators solve closely related problems: the former analyze motion, while the latter synthesize it. We show that this connection enables pretrained video diffusion models to perform zero-shot point tracking by simply prompting them to visually mark points as they move over time. We place a distinctively colored marker at the query point, then regenerate the rest of the video from an intermediate noise level. This propagates the marker across frames, tracing the point's trajectory. To ensure that the marker remains visible in this counterfactual generation, despite such markers being unlikely in natural videos, we use the unedited initial frame as a negative prompt. Through experiments with multiple image-conditioned video diffusion models, we find that these "emergent" tracks outperform those of prior zero-shot methods and persist through occlusions, often obtaining performance that is competitive with specialized self-supervised models.
We mark a query point in the first frame with a colored dot. Then, using SDEdit, we guide the diffusion model to regenerate the video, propagating the dot across subsequent frames, by enhancing the counterfactual signal. The video on the left is the original video, while the video on the right is the video generated by the diffusion model with the propagated point.
Drag the slider on the videos to compare our results with the original video.
@InProceedings{shrivastava2025pointprompting,
title = {Point Prompting: Counterfactual Tracking with Video Diffusion Models},
author = {Shrivastava, Ayush and Mehta, Sanyam and Geng, Daniel and Owens, Andrew},
booktitle = {arXiv},
year = {2025},
}