Selective Mask Propagation

In December, SkalskiP published a basketball AI tutorial where the player tracking was done by SAM2. RF-DETR detects the players in the first frame, every box becomes a SAM2 prompt, and SAM2 propagates each player's mask through the rest of the clip.

What struck me was how well SAM worked out of the box: better than any dedicated player tracker I had seen. Players cross and occlude each other constantly in basketball (and sports in general), and that is exactly where trackers normally are somewhat unreliable: identities get swapped, which is bad because then every event you attribute to that trajectory after that point belongs to the wrong player. SAM2 held identities through these situations where I was used to seeing these trackers break. It propagates masks with memory attention: each new frame attends over stored features of the object from earlier frames, so it carries a much richer picture of each player through an occlusion than the box, motion state, and appearance vector that a tracking-by-detection tracker matches against.

The problem is the cost, and the blog post says that: on an NVIDIA T4 the pipeline runs at 1–2 fps, and SAM2 is the bottleneck, with performance dropping as the number of tracked objects grows. The tutorial tackles basketball, so naturally in sports with even more players like soccer, it would be even slower. Segmenting ten or more players on every single frame is a lot of compute, and most of it is spent on frames where nothing difficult is happening. I played around with this setup for a few weeks, and got to the realization that it was a bit too slow to be useful in practice.

This naturally leads to the question: can we get the same identity preservation performance at a fraction of the cost? Specifically, can we run SAM only on the frames where it is needed, and use a lightweight tracker for the rest? Most frames in sports footage are easy: a lightweight tracker gets them right at hundreds of fps. The hard frames, the occlusions and crossings that actually cause identity switches, are a small fraction. If you can tell which frames those are while tracking, you can dispatch SAM only there and keep most of its identity robustness at a fraction of the cost. This post is about a system that does this, which as of July 2026 is #1 on the SportsMOT leaderboard.

To see what SAM is fixing, it helps to know how a "normal" tracker works. Tracking-by-detection (which is the most common tracking paradigm) splits the problem in two: a detector finds every player in every frame, and the tracker connects those detections over time. Each frame, it predicts where every existing track should be, compares the predictions against the new detections using cheap cues (box overlap, motion, and sometimes an appearance embedding), and solves the resulting assignment problem. At 25 fps a player moves a few pixels between frames, so for most of the video this matching is trivial, and it costs almost nothing.

The failure is concentrated in the following situation: two players close together. A screen, a rebound, a tackle where most or all of the cheap cues collapse at the same time: the boxes overlap so position stops discriminating, motion predictions are worthless because both players are changing direction mid-contact, and appearance is weak because teammates wear the same jersey. Both ways of pairing the two boxes now look almost equally right to the tracker, and it still has to pick one. When it picks wrong, the identities swap, and the mistake is permanent and every stat you compute downstream is now attributed to the wrong player.

Here is what that looks like on a real SportsMOT sequence, with plain Deep-EIoU. At frame 240 the number 31 is clearly legible, so at this moment we know exactly who ID 5 is. Then six players collapse into the paint. When they come apart, the tracker has him as ID 4, and ID 5 has landed on a player from the other team. The number never becomes readable again in this clip, so nothing catches the mistake: if this player scores twenty seconds later, the basket is credited to the wrong player, and every pass, shot, and rebound after that follows it.

Frame 240. Number 31 visible, tracked as ID 5

Frames 314–360. Occlusion

Frame 365. ID 4 now on number 31, ID 5 on an opponent

Hard — T9 crosses through traffic

Easy — T1 stands alone

The clip below is six seconds from a SportsMOT test sequence which shows the failure on real footage, run through two different box trackers on identical YOLOX detections: Deep-EIoU, probably the strongest tracking-by-detection algorithm on SportsMOT (expanded box overlap plus appearance embeddings), and ByteTrack, probably the most widely used tracking-by-detection method (box overlap and motion only).

Now the other extreme, the approach from the basketball tutorial: give SAM2 every player's box on the first frame and let it propagate masks forward. Here the association is implicit. Each new frame is matched against the model's stored memory of the player's pixels, so the correspondence is decided pixel by pixel inside the model rather than box by box in an explicit assignment step. Every player is matched against their own memory independently, and nothing enforces that two masks stay on two different people. Watch T10 during the pile-up: its mask degrades, then latches onto T9's player. From that point on, two masks ride the same person and the player T10 was covering has no mask at all, a state an explicit assignment step could never produce, since it gives each detection to at most one track. This is a failure mode that we tackle later as well.

This failure class, masks drifting, collapsing, or converging onto the same player, happens quite a lot when running SAM on longer clips. Luckily it is easily detectable, and handling it is part of what comes later in this post. The cost is the other half: masks for ten players cost about 8 fps on an RTX 5090 on this very clip, with no detector running, and the cost scales with the number of players.

So the same six seconds break three ways. The box trackers are nearly free and right almost everywhere, but they break exactly when players cross. Brute-force SAM2 costs a lot more, spends nearly all of that compute on frames the cheap tracker already had right, and still fails here, because nothing checks its output either. What is missing is a system that knows when the tracker is at risk, brings in SAM briefly, and verifies what it returns before trusting it.

If we go back to the moment the box trackers broke, two players collide and both ways of pairing the boxes look almost equally right. The key observation behind is that the tracker can see this happening because association is not a black box. Every frame, the tracker builds a cost matrix: one row per existing track, one column per new detection, and each cell holds a single number saying how badly that track and that detection fit together, with box overlap, motion, and appearance folded into one score (lower is better). The Hungarian algorithm (an algorithm for exactly this kind of assignment problem), then picks the pairing with the lowest total cost.

That pairing is the tracker's answer, but the matrix holds more information than the answer. For every match, you can look down the detection's column and ask: how much worse would the second-best track have fit this detection? That difference is the margin. A large margin means every alternative is clearly worse, so the assignment is safe. A margin near zero means another track fits this detection almost exactly as well, which is the "both pairings look almost equally right" situation from the previous section, except now it is a number, computed for every player on every frame, essentially for free, because the tracker already built the matrix to do its job.

The demo below shows the idea on mock data. Three tracks on the left, the live cost matrix on the right, margins in the last column. When T1 and T2 cross, their costs converge, their margins collapse toward zero, and their boxes turn red for exactly the frames where the tracker is guessing. T3, away from the action, keeps a comfortable margin the whole time. You can drag the slider to sit on any single frame and hover the matrix to see which cell belongs to which pair.

And this is not just a widget story. Below is the real margin trace for the number-31 player from the previous section, straight from Deep-EIoU's cost matrix. His margin is noisy but comfortably above the threshold while he is trackable, dipping whenever someone passes near him. The moment the pile forms, it collapses to essentially zero and stays pinned there for the entire occlusion, the same occlusion the tracker comes out of with the identities swapped. The signal fires at contact, before the failure is committed.

The main dispatch rule is therefore one line. Whenever any match's margin drops below a threshold (we use 0.05), the tracker is guessing, and we do not let it guess alone: we open a window and bring in SAM. We can call this SAM-mode, so this is the signal that tell us that that a track should go from lightweight tracker mode to SAM-mode.

When the trigger fires, we open a window: a short stretch of the video where the base tracker's identities are treated as at risk, and SAM tracks the pixels through the ambiguity in parallel. A window has three moments that matter: where SAM starts (the seed), what it does through the mess (propagate), and how it hands control back (the exit).

The subtle part is where SAM should start, because the trigger fires in the middle of the collision, and prompting SAM there would be pointless when the pixels are exactly as ambiguous as the boxes. So instead of starting at the entry, we walk backward from it, looking for a recent frame where the track was still confidently assigned (margin comfortably high for five straight frames) and its box clear of everyone else. That frame becomes the seed: SAM gets the player's pixels from a moment when there was no doubt about who they belonged to. The other player in the collision gets a window of its own, since an identity switch is a statement about a pair, and if no clean seed exists in the recent past, the window is simply discarded.

From the seed, the mask propagates forward through the crossing, and the system watches how it relates to the base tracker's boxes. A window is allowed to exit once the situation has genuinely resolved: the mask sits essentially inside a single box (at least 80% of its pixels), that track's margin has recovered above a stricter bar than the entry one, and both the box and the mask are clear of other players, all sustained for five consecutive frames. At that point the verdict is read directly off the geometry. If the mask ended up in a different track's box than the one it was seeded on, SAM is confidently contradicting the tracker: the window exits as SWAP and the identities are renamed from the frame where the switch happened. If the mask came back to its own box, the tracker had it right: the window exits as CLEAN and nothing changes.

This is how the number-31 play from the failure section actually ends. The margin collapse you saw in the trace opens a window, the mask is seeded on him at frame 255 while he is still alone, rides through the pile, and settles into a different track's box at the exit: SWAP, rename, and ID 5 is back on number 31.

Deep-EIoU alone

With selective mask propagation

That last part is the most important design decision in the system: a confident contradiction is the only outcome that modifies the output. Masks fail too, and we saw how in the failure section: they drift, collapse, or converge onto the same player. All of those are detectable, and every one of them ends the window without an identity decision. If the mask's area collapses, if two masks pile onto the same person, if the player leaves the frame, if the propagation reaches the end of the clip inconclusively, the window is thrown away and the base tracker's output passes through untouched. The failure mode that corrupted uniform SAM2 becomes, here, a wasted window rather than a corrupted track. The only way the system can do worse than the base tracker is a false-positive swap, and everything above, the seeding, the exit conditions, the sustained checks, is built to make that event rare.

With all of the machinery in place, we can go back to the six seconds that broke all three systems. Below is the same event through the full pipeline. During the pile-up, three windows open, and every single one of them exits as SWAP: the contact did not just swap one pair, it rotated three identities, and the box tracker came out wrong on all of them. The masks appear only around the contact, which is the whole point, and the identities come out the other side intact.

Everything so far preserves identity while a player stays on screen. Sports footage has a second identity problem: players leave the camera's view and come back, constantly, as the action moves around the playing surface. When a player re-enters after several seconds, there is nothing for a frame-by-frame tracker to match against, and masks do not help either: there are no pixels to propagate while the player is gone. So the two problems get separated. Selective mask propagation does on-screen identity preservation, holding identities through occlusions while players remain visible. A separate module does off-screen re-identification, linking the tracklets a player leaves behind across exits and re-entries.

That module is global track association (GTA), and it runs after everything else, on finished tracklets, using cues that are stable over long horizons instead of frame-to-frame ones. Each tracklet gets three attributes: a jersey number (pose estimation finds the shoulders and hips, the torso crop goes to a scene-text OCR model, and a per-track majority vote with confidence filters decides the number), a team (a vision-language model labels each track as team A, team B, or other, where "other" catches referees and goalkeepers), and an appearance embedding aggregated over the whole tracklet. Merging is two-tier, ordered by how much you can trust the cue: two tracklets with the same confident jersey number on the same team are merged unconditionally, and the rest are merged greedily by appearance similarity, with vetoes for pairs that overlap in time, exit and re-enter on opposite edges of the frame, belong to different teams, or carry conflicting jersey numbers.

GTA is an optional, sports-specific stage (--gta in the code). It never touches the tracking or the masks. The two components also fix disjoint errors: on SportsMOT validation, selective mask propagation adds +1.5 HOTA over Deep-EIoU alone and +1.8 on top of GTA, GTA adds +7.8 and +8.1 in the corresponding comparisons, and the combined gain (+9.6) is almost exactly the sum of the individual contributions, which is what you would expect from two mechanisms that never compete for the same mistake. GTA gets the larger share because field-of-view exits are the dominant association error on SportsMOT, and every one of them is routed to GTA by design. The benchmark numbers that follow use the full pipeline, selective mask propagation plus GTA.

Two questions matter: does it actually track better, and what does it cost. The headline benchmark is the SportsMOT test set, scored by the official remote evaluator, with the full pipeline: Deep-EIoU as the base tracker, selective mask propagation with SAM 3, and GTA for off-screen re-identification.

Two rows deserve attention. The gap between Deep-EIoU (77.2) and SAM3-Deep-EIoU (87.2) is what this post has been building: the same base tracker, plus dispatched masks for on-screen identities, plus GTA for off-screen ones. And the gap between the SAM2 and SAM3 rows (+1.7 HOTA) comes from swapping one component while the signal, windows, and merge logic stayed frozen: when a better mask propagator ships, the system improves for free. Per sport, the full system gains +9.6 HOTA on basketball, +12.0 on football, and +7.2 on volleyball over Deep-EIoU on the validation set, improving every single sequence of every sport.

Nothing in the method is sports-specific except GTA, so it should transfer. On DanceTrack, a dataset built to break association (uniform costumes, erratic motion, no off-screen exits, so no GTA), we applied the identical recipe to three different base trackers, same YOLOX detections, same thresholds, no per-tracker tuning:

Now the cost. The repository bundles four broadcast clips (two NBA, two international football) and a benchmark script that runs the SAM step both ways on identical YOLOX detections and Deep-EIoU tracks, so the only difference is dispatch: selective windows versus a mask for every player on every frame. On an RTX 5090:

The shape of the numbers says more than the aggregate. On soccer, wide camera views mean players rarely get ambiguous, few windows open, and selective runs 6–7× faster than uniform while holding a flat 5 GB, even though the soccer clips have the most players on screen (30 and 42): uniform's memory grows with the player count, selective's only with the handful of ambiguous ones. Basketball is the honest case: contact is so constant that windows are open on most frames, and the saving comes from tracking only the few ambiguous players per pass instead of all of them. On the full SportsMOT test set (94.8k frames), the same pattern holds at scale: the SAM step averages 13 fps, runs on 79% of frames but only ~4.4 objects per pass, with peak VRAM at 5.4 GB median across clips (6.1 GB worst case). And of the 6,320 windows it opened, 93% closed without changing anything: the asymmetric rule spends compute to double-check the tracker far more often than it overrules it, and only 4.1% of windows ended in a SWAP.

What about SAM 3.1? After this project was finished, Meta released SAM 3.1 with Object Multiplex, which tracks up to 16 objects jointly in one forward pass, a direct attack on the cost that motivated this whole design. So we benchmarked its out-of-the-box dense tracking (its own detector, a "player" text prompt) on the same four clips: 7.3 fps at 27.2 GB peak. That is markedly faster than uniform SAM 3 while also doing its own detection, and it still does not close the gap: selective dispatch stays about 2× faster at 5× less memory, because it wins by not tracking everything rather than by tracking everything faster. The two ideas compose, and a multiplexed VOS model behind the same dispatch signal is the obvious next experiment.

Everything is open source at github.com/holma91/selective-mask-propagation: the full pipeline, the three base trackers from the DanceTrack comparison, the GTA module, and the bundled clips and scripts behind every fps table in this post. The paper is on arXiv.

The algorithm is also exposed as a small API. If your tracker solves a Hungarian assignment, extract the per-detection margin (second-best minus best cost in each matched column) and pass it in; everything in this post happens inside one call:

from selective_mask_propagation.augment import augment

# tracks:  {frame: {track_id: [x1, y1, x2, y2]}}
# margins: {frame: {track_id: float}}
tracks, margins = my_tracker.run(detections)

corrected = augment(tracks, margins, "path/to/sequence", sam3=True)

The fastest way in is scripts/show_fps.py, which measures the SAM step on three bundled clips with no dataset and no setup beyond uv sync, and scripts/fps_benchmark.py, which reproduces the selective-versus-uniform table above on your own GPU.