In video object tracking, you draw a box around something in the first frame and the model follows it for the rest of the video. This becomes difficult when the scene is crowded with things that look alike, when the object moves fast, or when it disappears behind something for a few seconds. SAMURAI, from Cheng-Yen Yang and colleagues at the University of Washington, takes Meta's Segment Anything Model 2 (SAM 2) and improves how it handles those cases without any extra training.

SAMURAI logo

Where SAM 2 goes wrong

SAM 2 is very good at cutting an object out of an image. For video, it keeps a memory of recent frames and uses it to find the same object in the next one. The authors point to three weaknesses in how it does that:

  1. It trusts appearance over motion. In a crowd of similar objects, say one runner among many in matching kit, SAM 2 can jump to whichever candidate looks most like the target, even if that person is on the other side of the frame.
  2. Its memory is a fixed window. It stores the most recent frames whatever their quality. If the target was hidden behind something for those frames, the memory fills up with bad examples and the error carries forward.
  3. It mostly ignores motion, so fast or sudden movement throws it off.

What SAMURAI adds

SAMURAI makes two changes, both based on fairly old ideas in computer vision.

The first is a motion model. SAMURAI runs a Kalman filter on the object's bounding box, tracking its centre, width, height and how fast each is changing. Every frame, SAM 2 proposes several candidate masks with confidence scores. SAMURAI also checks how well each candidate overlaps with where the Kalman filter expected the object to be, and picks the mask with the best combination of the two. A similar-looking object on the other side of the frame now gets a low score, because the target could not have moved that far in one frame.

The second change is to the memory. Instead of always keeping the most recent frames, SAMURAI only stores a frame if the mask confidence, the object score and the motion score all pass a threshold. Frames where the target was partly hidden or the prediction was uncertain are left out, so they do not affect later frames. It still keeps seven memories, the number SAM 2 was trained with, so the rest of the model works as before.

Neither change needs retraining. SAMURAI uses the existing SAM 2 weights, which is what "zero-shot" means here, and the authors say it runs in real time, with speed close to plain SAM 2 on a single RTX 4090.

Results

The paper reports a 7.1% gain in AUC on LaSOT-ext and a 3.5% gain in average overlap on GOT-10k, and results on LaSOT that compete with trackers trained specifically on tracking data. For a sense of scale, LaSOT has 1,400 videos across 70 object categories, averaging around 2,500 frames each. LaSOT-ext adds harder categories. GOT-10k has over 10,000 clips of real moving objects from more than 560 classes. These are long videos with a lot going on, which is where SAM 2's fixed memory window causes the most problems.

This comparison of SAMURAI and SAM 2 was shared on X. Look at what happens to SAM 2 when similar objects cross paths.

Where it could be used

It could help anywhere you need to follow one object through a busy scene, for example a player during a match for sports analysis, a person in crowd footage, or a vehicle or pedestrian that a robot or car needs to react to. Because there is no training step, you can try it on your own videos straight away.

Trying it

The code is on GitHub under the Apache 2.0 licence. You need a GPU machine with Python 3.10+ and PyTorch. Install SAM 2 from the repo, download the SAM 2.1 checkpoints with the included script, then run scripts/demo.py with a video and a text file holding the first frame's bounding box as x,y,w,h.

My main complaint is that there is no online demo. A Hugging Face space where you could upload a clip and draw a box would let many more people test it. For now you need a GPU and some time to set it up.

References

  1. Paper: arXiv:2411.11922
  2. SAMURAI project page
  3. SAMURAI on GitHub