Tracking API

DeepHMSort is the built-in multi-object tracker. It takes an sv.Detections and returns the subset that matched a track, with tracker_id filled in — the same contract as sv.ByteTrack.update_with_detections, so anything already built around supervision keeps working.

See the tracking guide for what the algorithm does, which knobs matter, and what the defaults assume about your footage.

Tracker

modern_yolonas.tracking.DeepHMSort

Online multi-object tracker: detections in, stable ids out.

Usage mirrors sv.ByteTrack, so it is a drop-in for anything already built around supervision::

from modern_yolonas import Task, YoloNASDetector
from modern_yolonas.tracking import DeepHMSort

detector = YoloNASDetector("yolo_nas_s", conf_threshold=0.4)
tracker = DeepHMSort()

for _, frame, _ in detector.detect_video("match.mp4"):
    result = detector.predict(frame, Task.DETECT | Task.EMBED_OBJECTS)
    tracked = tracker.update_with_detections(result.detections)
    tracked.tracker_id  # (N,) stable ids

Or let the detector drive the whole loop, which is the same thing with the thresholds already lined up — see :meth:~modern_yolonas.inference.detect.YoloNASDetector.track_video.

The appearance vectors come from detections.data["embedding"]. Anything (N, D) works, so a purpose-trained re-identification model can be dropped in where the YOLO-NAS features are; without the key the tracker degrades cleanly to motion-only association, which is HM-SORT without the "Deep".

Parameters:

Name	Type	Description	Default
track_high_threshold	float	Detections at or above this score go into the first association round. The paper's 0.6.	0.6
track_low_threshold	float	Detections below this are dropped entirely; between this and track_high_threshold they are kept for the second round, where they can hold an existing track alive but cannot start one. The paper's 0.4 — which means the detector has to run at 0.4 or lower, or the low band never arrives.	0.4
new_track_threshold	float	Lowest score an unmatched detection may have and still become a new track. The paper gives 0.5 as "the threshold below which we discard tracks"; that phrasing also fits proximity_threshold, and since Deep-EIoU's default for that is also 0.5, the two readings agree on the shipped configuration.	0.5
match_threshold	float	Association costs above this are refused in the first round. The paper's 0.8.	0.8
proximity_threshold	float	IoU-distance above which the appearance cost is not trusted. See :func:~modern_yolonas.tracking.matching.fuse_costs.	0.5
appearance_threshold	float	Cosine-distance above which the appearance cost is discarded. The paper's 0.3.	0.3
expansion	float	Box growth for the first association round; see :func:~modern_yolonas.tracking.matching.expand_boxes. The paper's 0.3.	0.3
expansion_step	float	Added to expansion on each further round. The paper's 0.3, giving 0.3 then 0.6.	0.3
expansion_rounds	int	How many scale-up rounds to run. Deep-EIoU's 2.	2
low_score_expansion	float	Box growth for the low-score round, and for the unconfirmed round. Deep-EIoU hardcodes 0.5 in both places.	0.5
low_score_match_threshold	float	Cost ceiling for the low-score round.	0.5
unconfirmed_match_threshold	float	Cost ceiling for the round that gives first-frame tracks a second chance.	0.7
feature_momentum	float	alpha in the paper's Equation 1, the exponential moving average over a track's appearance vectors. Deep-EIoU's 0.9.	0.9
fusion	str	"harmonic" is the paper's contribution; "min" is Deep-EIoU's original, kept so the difference can be measured on your own footage rather than taken on faith.	'harmonic'
max_lost_seconds	float | None	How long a track may stay unmatched before it is discarded, in seconds of video — converted to frames through frame_rate, so the same number means the same thing on a 25 fps clip and a 60 fps one. The paper keeps every tracklet for the whole sequence, and None still does that. It is not the default here because it is a sports assumption: a fixed camera on a closed pitch, where a player who walks off returns to roughly where they left. On open-world footage the pool grows with every object that has ever appeared, memory and the cost matrix grow with it, and old boxes sit around waiting to catch a weak detection. 2 s is Deep-EIoU's own default (track_buffer 60 at 30 fps), and on a 100-frame clip of the Shibuya crossing it reproduces the unlimited result exactly — 17 ids either way — where 1 s costs two extra ids and splits a 53-frame track in half. Raise it, or set None, for a fixed camera on a closed scene.	2.0
frame_rate	float	Frames per second of the video being tracked, used only to turn max_lost_seconds into a frame count. :meth:~modern_yolonas.inference.detect.YoloNASDetector.track_video sets this from the video it opens; set it yourself when driving the loop, or leave it at 30 and accept that the budget is in 30ths of a second. Processing every N-th frame divides the effective rate, so divide this too.	30.0
class_aware	bool	Refuse to associate a detection with a track of a different class. The paper is single-class (athletes) and says nothing about this; off is the faithful reading, and a class-agnostic track simply carries whatever class its last match had.	False

Source code in src/modern_yolonas/tracking/deep_hm_sort.py

class DeepHMSort:
    """Online multi-object tracker: detections in, stable ids out.

    Usage mirrors ``sv.ByteTrack``, so it is a drop-in for anything already built
    around supervision::

        from modern_yolonas import Task, YoloNASDetector
        from modern_yolonas.tracking import DeepHMSort

        detector = YoloNASDetector("yolo_nas_s", conf_threshold=0.4)
        tracker = DeepHMSort()

        for _, frame, _ in detector.detect_video("match.mp4"):
            result = detector.predict(frame, Task.DETECT | Task.EMBED_OBJECTS)
            tracked = tracker.update_with_detections(result.detections)
            tracked.tracker_id  # (N,) stable ids

    Or let the detector drive the whole loop, which is the same thing with the
    thresholds already lined up — see
    :meth:`~modern_yolonas.inference.detect.YoloNASDetector.track_video`.

    The appearance vectors come from ``detections.data["embedding"]``. Anything
    ``(N, D)`` works, so a purpose-trained re-identification model can be dropped
    in where the YOLO-NAS features are; without the key the tracker degrades
    cleanly to motion-only association, which is HM-SORT without the "Deep".

    Args:
        track_high_threshold: Detections at or above this score go into the first
            association round. The paper's 0.6.
        track_low_threshold: Detections below this are dropped entirely; between
            this and ``track_high_threshold`` they are kept for the second round,
            where they can hold an existing track alive but cannot start one. The
            paper's 0.4 — which means the *detector* has to run at 0.4 or lower,
            or the low band never arrives.
        new_track_threshold: Lowest score an unmatched detection may have and
            still become a new track. The paper gives 0.5 as "the threshold below
            which we discard tracks"; that phrasing also fits
            ``proximity_threshold``, and since Deep-EIoU's default for that is
            also 0.5, the two readings agree on the shipped configuration.
        match_threshold: Association costs above this are refused in the first
            round. The paper's 0.8.
        proximity_threshold: IoU-distance above which the appearance cost is not
            trusted. See :func:`~modern_yolonas.tracking.matching.fuse_costs`.
        appearance_threshold: Cosine-distance above which the appearance cost is
            discarded. The paper's 0.3.
        expansion: Box growth for the first association round; see
            :func:`~modern_yolonas.tracking.matching.expand_boxes`. The paper's 0.3.
        expansion_step: Added to ``expansion`` on each further round. The paper's
            0.3, giving 0.3 then 0.6.
        expansion_rounds: How many scale-up rounds to run. Deep-EIoU's 2.
        low_score_expansion: Box growth for the low-score round, and for the
            unconfirmed round. Deep-EIoU hardcodes 0.5 in both places.
        low_score_match_threshold: Cost ceiling for the low-score round.
        unconfirmed_match_threshold: Cost ceiling for the round that gives
            first-frame tracks a second chance.
        feature_momentum: ``alpha`` in the paper's Equation 1, the exponential
            moving average over a track's appearance vectors. Deep-EIoU's 0.9.
        fusion: ``"harmonic"`` is the paper's contribution; ``"min"`` is
            Deep-EIoU's original, kept so the difference can be measured on your
            own footage rather than taken on faith.
        max_lost_seconds: How long a track may stay unmatched before it is
            discarded, in **seconds of video** — converted to frames through
            ``frame_rate``, so the same number means the same thing on a 25 fps
            clip and a 60 fps one.

            The paper keeps every tracklet for the whole sequence, and ``None``
            still does that. It is not the default here because it is a *sports*
            assumption: a fixed camera on a closed pitch, where a player who walks
            off returns to roughly where they left. On open-world footage the pool
            grows with every object that has ever appeared, memory and the cost
            matrix grow with it, and old boxes sit around waiting to catch a weak
            detection.

            2 s is Deep-EIoU's own default (``track_buffer`` 60 at 30 fps), and on
            a 100-frame clip of the Shibuya crossing it reproduces the unlimited
            result exactly — 17 ids either way — where 1 s costs two extra ids and
            splits a 53-frame track in half. Raise it, or set ``None``, for a fixed
            camera on a closed scene.
        frame_rate: Frames per second of the video being tracked, used only to turn
            ``max_lost_seconds`` into a frame count.
            :meth:`~modern_yolonas.inference.detect.YoloNASDetector.track_video`
            sets this from the video it opens; set it yourself when driving the
            loop, or leave it at 30 and accept that the budget is in 30ths of a
            second. Processing every N-th frame divides the effective rate, so
            divide this too.
        class_aware: Refuse to associate a detection with a track of a different
            class. The paper is single-class (athletes) and says nothing about
            this; off is the faithful reading, and a class-agnostic track simply
            carries whatever class its last match had.
    """

    def __init__(
        self,
        track_high_threshold: float = 0.6,
        track_low_threshold: float = 0.4,
        new_track_threshold: float = 0.5,
        match_threshold: float = 0.8,
        proximity_threshold: float = 0.5,
        appearance_threshold: float = 0.3,
        expansion: float = 0.3,
        expansion_step: float = 0.3,
        expansion_rounds: int = 2,
        low_score_expansion: float = 0.5,
        low_score_match_threshold: float = 0.5,
        unconfirmed_match_threshold: float = 0.7,
        feature_momentum: float = 0.9,
        fusion: str = "harmonic",
        max_lost_seconds: float | None = 2.0,
        frame_rate: float = 30.0,
        class_aware: bool = False,
    ):
        if not 0.0 <= track_low_threshold <= track_high_threshold <= 1.0:
            raise ValueError(
                "expected 0 <= track_low_threshold <= track_high_threshold <= 1, got "
                f"{track_low_threshold} and {track_high_threshold}"
            )
        if not 0.0 <= feature_momentum < 1.0:
            raise ValueError(f"feature_momentum must be in [0, 1), got {feature_momentum}")
        if expansion_rounds < 1:
            raise ValueError(f"expansion_rounds must be >= 1, got {expansion_rounds}")
        if fusion not in ("harmonic", "min"):
            raise ValueError(f"fusion must be 'harmonic' or 'min', got {fusion!r}")
        if max_lost_seconds is not None and max_lost_seconds <= 0:
            raise ValueError(f"max_lost_seconds must be > 0 or None, got {max_lost_seconds}")
        if frame_rate <= 0:
            raise ValueError(f"frame_rate must be > 0, got {frame_rate}")

        self.track_high_threshold = track_high_threshold
        self.track_low_threshold = track_low_threshold
        self.new_track_threshold = new_track_threshold
        self.match_threshold = match_threshold
        self.proximity_threshold = proximity_threshold
        self.appearance_threshold = appearance_threshold
        self.expansion = expansion
        self.expansion_step = expansion_step
        self.expansion_rounds = expansion_rounds
        self.low_score_expansion = low_score_expansion
        self.low_score_match_threshold = low_score_match_threshold
        self.unconfirmed_match_threshold = unconfirmed_match_threshold
        self.feature_momentum = feature_momentum
        self.fusion = fusion
        self.max_lost_seconds = max_lost_seconds
        self.frame_rate = frame_rate
        self.class_aware = class_aware

        self.reset()

    def reset(self) -> None:
        """Forget every track and restart ids from 1.

        Call between videos — ids are unique within a tracker instance, not
        globally, so two videos tracked by the same instance share a namespace.
        """
        self._tracked: list[Track] = []
        self._lost: list[Track] = []
        self.frame_id = 0
        self._next_id = 0

    @property
    def max_lost_frames(self) -> int | None:
        """:attr:`max_lost_seconds` in frames at the current :attr:`frame_rate`.

        Never less than one frame: a positive budget that rounds to zero would
        discard every track the instant it is missed, which is not what any number
        the caller wrote could have meant.
        """
        if self.max_lost_seconds is None:
            return None
        return max(1, round(self.max_lost_seconds * self.frame_rate))

    @property
    def tracks(self) -> list[Track]:
        """Every live track, matched this frame or not, for inspection."""
        return list(self._tracked) + list(self._lost)

    def _new_id(self) -> int:
        self._next_id += 1
        return self._next_id

    # ------------------------------------------------------------------ costs

    def _appearance_distance(self, tracks: list[Track], features: np.ndarray | None) -> np.ndarray | None:
        """``(M, N)`` cosine distance, ``NaN`` wherever either side has no vector.

        :func:`~modern_yolonas.tracking.matching.fuse_costs` turns those ``NaN``\\ s
        into a gated cost, so a track or a detection without an embedding still
        associates — on motion alone.
        """
        if features is None or not tracks or len(features) == 0:
            return None

        rows = [i for i, track in enumerate(tracks) if track.feature is not None]
        usable = np.isfinite(features).all(axis=1)
        if not rows or not usable.any():
            return None

        distance = np.full((len(tracks), len(features)), np.nan, dtype=np.float32)
        cols = np.flatnonzero(usable)
        distance[np.ix_(rows, cols)] = cosine_distance(
            np.stack([tracks[i].feature for i in rows]), features[cols]
        )
        return distance

    def _cost(
        self,
        tracks: list[Track],
        boxes: np.ndarray,
        class_ids: np.ndarray,
        features: np.ndarray | None,
        expansion: float,
        with_appearance: bool,
    ) -> np.ndarray:
        """Association cost between ``tracks`` and a set of detections."""
        if not tracks or len(boxes) == 0:
            return np.zeros((len(tracks), len(boxes)), dtype=np.float32)

        track_boxes = np.stack([track.xyxy for track in tracks])
        iou_distance = expansion_iou_distance(track_boxes, boxes, expansion)

        cost = fuse_costs(
            iou_distance,
            self._appearance_distance(tracks, features) if with_appearance else None,
            proximity_threshold=self.proximity_threshold,
            appearance_threshold=self.appearance_threshold,
            fusion=self.fusion,
        )

        if self.class_aware:
            track_classes = np.array([track.class_id for track in tracks])
            cost = np.where(track_classes[:, None] != class_ids[None, :], GATED, cost)

        return cost

    # ------------------------------------------------------------------ update

    def update_with_detections(self, detections: sv.Detections) -> sv.Detections:
        """Associate one frame's detections with the live tracks.

        Args:
            detections: This frame's detections, in source-image pixels. Per-object
                appearance vectors are read from ``detections.data["embedding"]``,
                an ``(N, D)`` float array — which is exactly what
                :meth:`YoloNASDetector.predict <modern_yolonas.inference.detect.YoloNASDetector.predict>`
                puts there for ``Task.EMBED_OBJECTS``.

        Returns:
            The subset of ``detections`` that matched a track, with
            ``tracker_id`` set and every other column — confidence, class,
            ``data["class_name"]``, ``data["embedding"]`` — carried through
            untouched. Rows come back in track-id order.
        """
        self.frame_id += 1

        boxes, scores, class_ids, features = _unpack(detections)

        keep = scores >= self.track_low_threshold
        high = np.flatnonzero(keep & (scores >= self.track_high_threshold))
        low = np.flatnonzero(keep & (scores < self.track_high_threshold))

        # A track created last frame and not yet seen twice is "unconfirmed": it
        # gets its own, stricter round at the end rather than competing with
        # established tracks for this frame's detections.
        # A track born on frame 1 counts as confirmed straight away — there was no
        # earlier frame it could have failed to appear in, and making the whole
        # first frame's worth of objects fight for the unconfirmed round's narrower
        # terms would cost ids on anything moving from the start.
        def is_confirmed(track):
            return track.hits > 1 or track.start_frame == 1

        confirmed = [track for track in self._tracked if is_confirmed(track)]
        unconfirmed = [track for track in self._tracked if not is_confirmed(track)]
        for track in self._tracked + self._lost:
            track.source_index = None

        activated: list[Track] = []
        # Lost tracks join the first round — retention buys nothing if they are
        # not actually candidates for re-association.
        pool = confirmed + self._lost
        remaining = high

        for round_index in range(self.expansion_rounds):
            if not pool or len(remaining) == 0:
                break
            expansion = self.expansion + self.expansion_step * round_index
            cost = self._cost(pool, boxes[remaining], class_ids[remaining], _rows(features, remaining), expansion, True)
            matches, unmatched_tracks, unmatched_dets = linear_assignment(cost, self.match_threshold)

            for track_index, det_index in matches:
                activated.append(self._attach(pool[track_index], remaining[det_index], boxes, scores, class_ids, features))

            # Deep-EIoU drops lost tracks from the pool after each round, so they
            # only ever compete at the smallest expansion. Kept as-is: a wider
            # search radius for a track last seen long ago is how ids wander.
            pool = [pool[i] for i in unmatched_tracks if pool[i].state == TrackState.TRACKED]
            remaining = remaining[unmatched_dets]

        # Only *tracked* tracks reach the low-score round. A lost track revived by a
        # 0.45-score box near where it used to be is an id resurrected on nothing,
        # and with every tracklet kept forever there are a lot of such boxes. The
        # loop above filters as it goes, but it breaks early on a frame with no
        # strong detections at all — which is exactly the frame this protects.
        pool = [track for track in pool if track.state == TrackState.TRACKED]

        # Second round: low-score detections, motion only. A crop this uncertain is
        # not worth learning an identity from, so no appearance cost and no feature
        # update — it exists to keep a track alive through a bad frame.
        if pool and len(low):
            cost = self._cost(pool, boxes[low], class_ids[low], None, self.low_score_expansion, False)
            matches, unmatched_tracks, _ = linear_assignment(cost, self.low_score_match_threshold)
            for track_index, det_index in matches:
                activated.append(
                    self._attach(pool[track_index], low[det_index], boxes, scores, class_ids, features, learn=False)
                )
            pool = [pool[i] for i in unmatched_tracks]

        newly_lost = []
        for track in pool:
            if track.state != TrackState.LOST:
                track.mark_lost()
                newly_lost.append(track)

        # Unconfirmed tracks against whatever high-score detections are left.
        if unconfirmed:
            cost = self._cost(
                unconfirmed, boxes[remaining], class_ids[remaining], _rows(features, remaining),
                self.low_score_expansion, True,
            )
            matches, unmatched_tracks, unmatched_dets = linear_assignment(cost, self.unconfirmed_match_threshold)
            for track_index, det_index in matches:
                activated.append(
                    self._attach(unconfirmed[track_index], remaining[det_index], boxes, scores, class_ids, features)
                )
            for i in unmatched_tracks:
                unconfirmed[i].mark_removed()
            remaining = remaining[unmatched_dets]

        for det_index in remaining:
            if scores[det_index] < self.new_track_threshold:
                continue
            activated.append(
                Track(
                    track_id=self._new_id(),
                    xyxy=boxes[det_index],
                    score=scores[det_index],
                    class_id=class_ids[det_index],
                    feature=features[det_index] if features is not None else None,
                    frame_id=self.frame_id,
                    source_index=int(det_index),
                )
            )

        self._commit(activated, newly_lost)
        return self._output(detections)

    def _attach(
        self,
        track: Track,
        det_index: int,
        boxes: np.ndarray,
        scores: np.ndarray,
        class_ids: np.ndarray,
        features: np.ndarray | None,
        learn: bool = True,
    ) -> Track:
        feature = features[det_index] if learn and features is not None else None
        if feature is not None and not np.isfinite(feature).all():
            feature = None
        track.update(
            xyxy=boxes[det_index],
            score=scores[det_index],
            class_id=class_ids[det_index],
            feature=feature,
            frame_id=self.frame_id,
            source_index=int(det_index),
            momentum=self.feature_momentum,
        )
        return track

    def _commit(self, activated: list[Track], newly_lost: list[Track]) -> None:
        """Fold this frame's outcome back into the track lists.

        Unconfirmed tracks that failed their round are simply not carried over:
        they were in ``_tracked``, never in ``_lost``, and ``_tracked`` is rebuilt
        from ``activated`` alone.
        """
        matched_ids = {track.track_id for track in activated}

        self._tracked = activated
        self._lost = [
            track
            for track in self._lost + newly_lost
            if track.track_id not in matched_ids and track.state == TrackState.LOST
        ]

        budget = self.max_lost_frames
        if budget is not None:
            self._lost = [t for t in self._lost if self.frame_id - t.frame_id <= budget]

        self._drop_duplicates()

    def _drop_duplicates(self) -> None:
        """Merge a lost track that has drifted onto the same box as a live one.

        Two tracks sitting on one object is the failure that follows from keeping
        every tracklet: a lost id that re-attaches next to the live one and starts
        competing for its detections. Where a tracked and a lost box overlap almost
        exactly, the younger of the two goes.
        """
        if not self._tracked or not self._lost:
            return

        overlap = box_iou(
            np.stack([t.xyxy for t in self._tracked]), np.stack([t.xyxy for t in self._lost])
        )
        drop_tracked, drop_lost = set(), set()
        for i, j in zip(*np.nonzero(overlap > 0.85)):
            age_tracked = self._tracked[i].frame_id - self._tracked[i].start_frame
            age_lost = self._lost[j].frame_id - self._lost[j].start_frame
            if age_tracked > age_lost:
                drop_lost.add(int(j))
            else:
                drop_tracked.add(int(i))

        self._tracked = [t for i, t in enumerate(self._tracked) if i not in drop_tracked]
        self._lost = [t for j, t in enumerate(self._lost) if j not in drop_lost]

    def _output(self, detections: sv.Detections) -> sv.Detections:
        """This frame's matched detections, with ``tracker_id`` attached."""
        matched = sorted(
            (t for t in self._tracked if t.source_index is not None), key=lambda t: t.track_id
        )
        if not matched:
            return sv.Detections.empty()

        result = detections[np.array([t.source_index for t in matched], dtype=int)]
        result.tracker_id = np.array([t.track_id for t in matched], dtype=int)
        return result

max_lost_frames property

:attr:max_lost_seconds in frames at the current :attr:frame_rate.

Never less than one frame: a positive budget that rounds to zero would discard every track the instant it is missed, which is not what any number the caller wrote could have meant.

tracks property

Every live track, matched this frame or not, for inspection.

reset()

Forget every track and restart ids from 1.

Call between videos — ids are unique within a tracker instance, not globally, so two videos tracked by the same instance share a namespace.

Source code in src/modern_yolonas/tracking/deep_hm_sort.py

def reset(self) -> None:
    """Forget every track and restart ids from 1.

    Call between videos — ids are unique within a tracker instance, not
    globally, so two videos tracked by the same instance share a namespace.
    """
    self._tracked: list[Track] = []
    self._lost: list[Track] = []
    self.frame_id = 0
    self._next_id = 0

update_with_detections(detections)

Associate one frame's detections with the live tracks.

Parameters:

Name	Type	Description	Default
detections	Detections	This frame's detections, in source-image pixels. Per-object appearance vectors are read from detections.data["embedding"], an (N, D) float array — which is exactly what :meth:YoloNASDetector.predict <modern_yolonas.inference.detect.YoloNASDetector.predict> puts there for Task.EMBED_OBJECTS.	required

Returns:

Type	Description
Detections	The subset of detections that matched a track, with
Detections	tracker_id set and every other column — confidence, class,
Detections	data["class_name"], data["embedding"] — carried through
Detections	untouched. Rows come back in track-id order.

Source code in src/modern_yolonas/tracking/deep_hm_sort.py

def update_with_detections(self, detections: sv.Detections) -> sv.Detections:
    """Associate one frame's detections with the live tracks.

    Args:
        detections: This frame's detections, in source-image pixels. Per-object
            appearance vectors are read from ``detections.data["embedding"]``,
            an ``(N, D)`` float array — which is exactly what
            :meth:`YoloNASDetector.predict <modern_yolonas.inference.detect.YoloNASDetector.predict>`
            puts there for ``Task.EMBED_OBJECTS``.

    Returns:
        The subset of ``detections`` that matched a track, with
        ``tracker_id`` set and every other column — confidence, class,
        ``data["class_name"]``, ``data["embedding"]`` — carried through
        untouched. Rows come back in track-id order.
    """
    self.frame_id += 1

    boxes, scores, class_ids, features = _unpack(detections)

    keep = scores >= self.track_low_threshold
    high = np.flatnonzero(keep & (scores >= self.track_high_threshold))
    low = np.flatnonzero(keep & (scores < self.track_high_threshold))

    # A track created last frame and not yet seen twice is "unconfirmed": it
    # gets its own, stricter round at the end rather than competing with
    # established tracks for this frame's detections.
    # A track born on frame 1 counts as confirmed straight away — there was no
    # earlier frame it could have failed to appear in, and making the whole
    # first frame's worth of objects fight for the unconfirmed round's narrower
    # terms would cost ids on anything moving from the start.
    def is_confirmed(track):
        return track.hits > 1 or track.start_frame == 1

    confirmed = [track for track in self._tracked if is_confirmed(track)]
    unconfirmed = [track for track in self._tracked if not is_confirmed(track)]
    for track in self._tracked + self._lost:
        track.source_index = None

    activated: list[Track] = []
    # Lost tracks join the first round — retention buys nothing if they are
    # not actually candidates for re-association.
    pool = confirmed + self._lost
    remaining = high

    for round_index in range(self.expansion_rounds):
        if not pool or len(remaining) == 0:
            break
        expansion = self.expansion + self.expansion_step * round_index
        cost = self._cost(pool, boxes[remaining], class_ids[remaining], _rows(features, remaining), expansion, True)
        matches, unmatched_tracks, unmatched_dets = linear_assignment(cost, self.match_threshold)

        for track_index, det_index in matches:
            activated.append(self._attach(pool[track_index], remaining[det_index], boxes, scores, class_ids, features))

        # Deep-EIoU drops lost tracks from the pool after each round, so they
        # only ever compete at the smallest expansion. Kept as-is: a wider
        # search radius for a track last seen long ago is how ids wander.
        pool = [pool[i] for i in unmatched_tracks if pool[i].state == TrackState.TRACKED]
        remaining = remaining[unmatched_dets]

    # Only *tracked* tracks reach the low-score round. A lost track revived by a
    # 0.45-score box near where it used to be is an id resurrected on nothing,
    # and with every tracklet kept forever there are a lot of such boxes. The
    # loop above filters as it goes, but it breaks early on a frame with no
    # strong detections at all — which is exactly the frame this protects.
    pool = [track for track in pool if track.state == TrackState.TRACKED]

    # Second round: low-score detections, motion only. A crop this uncertain is
    # not worth learning an identity from, so no appearance cost and no feature
    # update — it exists to keep a track alive through a bad frame.
    if pool and len(low):
        cost = self._cost(pool, boxes[low], class_ids[low], None, self.low_score_expansion, False)
        matches, unmatched_tracks, _ = linear_assignment(cost, self.low_score_match_threshold)
        for track_index, det_index in matches:
            activated.append(
                self._attach(pool[track_index], low[det_index], boxes, scores, class_ids, features, learn=False)
            )
        pool = [pool[i] for i in unmatched_tracks]

    newly_lost = []
    for track in pool:
        if track.state != TrackState.LOST:
            track.mark_lost()
            newly_lost.append(track)

    # Unconfirmed tracks against whatever high-score detections are left.
    if unconfirmed:
        cost = self._cost(
            unconfirmed, boxes[remaining], class_ids[remaining], _rows(features, remaining),
            self.low_score_expansion, True,
        )
        matches, unmatched_tracks, unmatched_dets = linear_assignment(cost, self.unconfirmed_match_threshold)
        for track_index, det_index in matches:
            activated.append(
                self._attach(unconfirmed[track_index], remaining[det_index], boxes, scores, class_ids, features)
            )
        for i in unmatched_tracks:
            unconfirmed[i].mark_removed()
        remaining = remaining[unmatched_dets]

    for det_index in remaining:
        if scores[det_index] < self.new_track_threshold:
            continue
        activated.append(
            Track(
                track_id=self._new_id(),
                xyxy=boxes[det_index],
                score=scores[det_index],
                class_id=class_ids[det_index],
                feature=features[det_index] if features is not None else None,
                frame_id=self.frame_id,
                source_index=int(det_index),
            )
        )

    self._commit(activated, newly_lost)
    return self._output(detections)

Track state

modern_yolonas.tracking.Track

One tracked object.

There is no Kalman filter here, and that is deliberate rather than missing. Deep-EIoU — the tracker Deep HM-SORT extends — abandons it: a constant-velocity Gaussian model is a poor description of an athlete changing direction, and the prediction it contributes is worth less than the expanded box it would be compared against. So :attr:xyxy is the last box this track actually matched, not a prediction of where it will be.

Attributes:

Name	Type	Description
track_id		Stable id, unique within the tracker instance that made it.
xyxy		(4,) last matched box, in source-image pixels.
score		Confidence of the detection that last matched.
class_id		Class of the detection that last matched. Tracks are not pinned to a class unless the tracker was built class_aware.
feature	ndarray | None	(D,) L2-normalized appearance vector, or None when the track has only ever been matched without one.
state		See :class:TrackState.
start_frame		Frame index the track was created on.
frame_id		Frame index it was last matched on.
hits		How many frames it has matched in total.
source_index	int | None	Row of the current frame's detections it matched, or None if it did not match this frame. Used to carry the caller's own columns (class names, embeddings) through to the output.

Source code in src/modern_yolonas/tracking/track.py

class Track:
    """One tracked object.

    There is no Kalman filter here, and that is deliberate rather than missing.
    Deep-EIoU — the tracker Deep HM-SORT extends — abandons it: a constant-velocity
    Gaussian model is a poor description of an athlete changing direction, and the
    prediction it contributes is worth less than the expanded box it would be
    compared against. So :attr:`xyxy` is the last box this track actually matched,
    not a prediction of where it will be.

    Attributes:
        track_id: Stable id, unique within the tracker instance that made it.
        xyxy: ``(4,)`` last matched box, in source-image pixels.
        score: Confidence of the detection that last matched.
        class_id: Class of the detection that last matched. Tracks are not pinned
            to a class unless the tracker was built ``class_aware``.
        feature: ``(D,)`` L2-normalized appearance vector, or ``None`` when the
            track has only ever been matched without one.
        state: See :class:`TrackState`.
        start_frame: Frame index the track was created on.
        frame_id: Frame index it was last matched on.
        hits: How many frames it has matched in total.
        source_index: Row of the current frame's detections it matched, or ``None``
            if it did not match this frame. Used to carry the caller's own columns
            (class names, embeddings) through to the output.
    """

    __slots__ = (
        "track_id",
        "xyxy",
        "score",
        "class_id",
        "feature",
        "state",
        "start_frame",
        "frame_id",
        "hits",
        "source_index",
    )

    def __init__(
        self,
        track_id: int,
        xyxy: np.ndarray,
        score: float,
        class_id: int,
        feature: np.ndarray | None,
        frame_id: int,
        source_index: int,
    ):
        self.track_id = track_id
        self.xyxy = np.asarray(xyxy, dtype=np.float32).reshape(4)
        self.score = float(score)
        self.class_id = int(class_id)
        self.feature: np.ndarray | None = None
        self.state = TrackState.TRACKED
        self.start_frame = frame_id
        self.frame_id = frame_id
        self.hits = 1
        self.source_index: int | None = source_index

        if feature is not None:
            self.feature = _unit(feature)

    def update(
        self,
        xyxy: np.ndarray,
        score: float,
        class_id: int,
        feature: np.ndarray | None,
        frame_id: int,
        source_index: int,
        momentum: float,
    ) -> None:
        """Attach a detection to this track.

        The appearance vector is blended rather than replaced::

            v_k = momentum * v_{k-1} + (1 - momentum) * f_k

        and then re-normalized — without that last step the running vector drifts
        off the unit sphere and a dot product stops being a cosine.

        Args:
            xyxy: ``(4,)`` matched box.
            score: Its confidence.
            class_id: Its class.
            feature: ``(D,)`` appearance vector, or ``None`` to leave the running
                one untouched (a low-score match carries a crop too unreliable to
                learn an identity from).
            frame_id: Current frame index.
            source_index: Row of the current frame's detections.
            momentum: The ``alpha`` above; higher means slower to change.
        """
        self.xyxy = np.asarray(xyxy, dtype=np.float32).reshape(4)
        self.score = float(score)
        self.class_id = int(class_id)
        self.state = TrackState.TRACKED
        self.frame_id = frame_id
        self.source_index = source_index
        self.hits += 1

        if feature is not None:
            unit = _unit(feature)
            if self.feature is None:
                self.feature = unit
            else:
                self.feature = _unit(momentum * self.feature + (1.0 - momentum) * unit)

    def mark_lost(self) -> None:
        self.state = TrackState.LOST
        self.source_index = None

    def mark_removed(self) -> None:
        self.state = TrackState.REMOVED
        self.source_index = None

    def __repr__(self) -> str:
        return f"Track(id={self.track_id}, state={self.state.name}, frames={self.start_frame}-{self.frame_id})"

update(xyxy, score, class_id, feature, frame_id, source_index, momentum)

Attach a detection to this track.

The appearance vector is blended rather than replaced::

v_k = momentum * v_{k-1} + (1 - momentum) * f_k

and then re-normalized — without that last step the running vector drifts off the unit sphere and a dot product stops being a cosine.

Parameters:

Name	Type	Description	Default
xyxy	ndarray	(4,) matched box.	required
score	float	Its confidence.	required
class_id	int	Its class.	required
feature	ndarray | None	(D,) appearance vector, or None to leave the running one untouched (a low-score match carries a crop too unreliable to learn an identity from).	required
frame_id	int	Current frame index.	required
source_index	int	Row of the current frame's detections.	required
momentum	float	The alpha above; higher means slower to change.	required

Source code in src/modern_yolonas/tracking/track.py

def update(
    self,
    xyxy: np.ndarray,
    score: float,
    class_id: int,
    feature: np.ndarray | None,
    frame_id: int,
    source_index: int,
    momentum: float,
) -> None:
    """Attach a detection to this track.

    The appearance vector is blended rather than replaced::

        v_k = momentum * v_{k-1} + (1 - momentum) * f_k

    and then re-normalized — without that last step the running vector drifts
    off the unit sphere and a dot product stops being a cosine.

    Args:
        xyxy: ``(4,)`` matched box.
        score: Its confidence.
        class_id: Its class.
        feature: ``(D,)`` appearance vector, or ``None`` to leave the running
            one untouched (a low-score match carries a crop too unreliable to
            learn an identity from).
        frame_id: Current frame index.
        source_index: Row of the current frame's detections.
        momentum: The ``alpha`` above; higher means slower to change.
    """
    self.xyxy = np.asarray(xyxy, dtype=np.float32).reshape(4)
    self.score = float(score)
    self.class_id = int(class_id)
    self.state = TrackState.TRACKED
    self.frame_id = frame_id
    self.source_index = source_index
    self.hits += 1

    if feature is not None:
        unit = _unit(feature)
        if self.feature is None:
            self.feature = unit
        else:
            self.feature = _unit(momentum * self.feature + (1.0 - momentum) * unit)

modern_yolonas.tracking.TrackState

Bases: Enum

Where a track stands this frame.

Members

TRACKED: Matched in the current frame. LOST: Alive but unmatched — still in the association pool, and still eligible to be found again. REMOVED: Out of the pool for good.

Source code in src/modern_yolonas/tracking/track.py

class TrackState(enum.Enum):
    """Where a track stands this frame.

    Members:
        TRACKED: Matched in the current frame.
        LOST: Alive but unmatched — still in the association pool, and still
            eligible to be found again.
        REMOVED: Out of the pool for good.
    """

    TRACKED = enum.auto()
    LOST = enum.auto()
    REMOVED = enum.auto()

Video helpers

modern_yolonas.inference.detect.YoloNASDetector.track_video(source, tracker=None, conf_threshold=None, iou_threshold=None, appearance=True, skip_frames=0)

Detect and track each frame of a video, in one forward pass per frame.

The appearance vectors Deep HM-SORT associates on are the same per-object embeddings :meth:predict produces, read off the features the detection pass already computed — so tracking with appearance costs one pass per frame, not the two a bolted-on re-identification model would need.

The detector runs at the tracker's track_low_threshold unless told otherwise. That is deliberate: Deep HM-SORT's second association round exists to hold a track through a frame where the detector wavers, and it can only do that if those weak detections reach it. Raising conf_threshold to the tracker's track_high_threshold turns that round off.

Parameters:

Name	Type	Description	Default
source	str | Path | int	Video file path or camera index (0 for webcam).	required
tracker	DeepHMSort | None	A configured :class:~modern_yolonas.tracking.DeepHMSort, or None for a default one. Pass your own to tune it, to keep inspecting tracker.tracks, or to reuse it across calls — it is stateful, so :meth:~modern_yolonas.tracking.DeepHMSort.reset between unrelated videos. Its frame_rate is set from this video, so max_lost_seconds means the same span of time whatever the clip was shot at.	None
conf_threshold	float | None	Override the tracker-derived detection threshold.	None
iou_threshold	float | None	Override the instance default.	None
appearance	bool	Compute per-object embeddings and associate on them. False drops to motion-only association, which is faster by the ROI pooling and markedly worse through occlusions.	True
skip_frames	int	Process every N-th frame (0 = every frame). Note the tracker sees only the processed frames, so objects move further between them and the expansion has more work to do.	0

Yields:

Type	Description
int	(frame_index, frame, detections) for each processed frame, where
ndarray	detections carries tracker_id and holds only the detections
Detections	that matched a track.

Source code in src/modern_yolonas/inference/detect.py

def track_video(
    self,
    source: str | Path | int,
    tracker: DeepHMSort | None = None,
    conf_threshold: float | None = None,
    iou_threshold: float | None = None,
    appearance: bool = True,
    skip_frames: int = 0,
) -> Generator[tuple[int, np.ndarray, sv.Detections], None, None]:
    """Detect and track each frame of a video, in one forward pass per frame.

    The appearance vectors Deep HM-SORT associates on are the same per-object
    embeddings :meth:`predict` produces, read off the features the detection
    pass already computed — so tracking with appearance costs one pass per
    frame, not the two a bolted-on re-identification model would need.

    The detector runs at the tracker's ``track_low_threshold`` unless told
    otherwise. That is deliberate: Deep HM-SORT's second association round
    exists to hold a track through a frame where the detector wavers, and it
    can only do that if those weak detections reach it. Raising
    ``conf_threshold`` to the tracker's ``track_high_threshold`` turns that
    round off.

    Args:
        source: Video file path or camera index (0 for webcam).
        tracker: A configured :class:`~modern_yolonas.tracking.DeepHMSort`, or
            ``None`` for a default one. Pass your own to tune it, to keep
            inspecting ``tracker.tracks``, or to reuse it across calls — it is
            stateful, so :meth:`~modern_yolonas.tracking.DeepHMSort.reset`
            between unrelated videos. Its ``frame_rate`` is set from this
            video, so ``max_lost_seconds`` means the same span of time
            whatever the clip was shot at.
        conf_threshold: Override the tracker-derived detection threshold.
        iou_threshold: Override the instance default.
        appearance: Compute per-object embeddings and associate on them.
            ``False`` drops to motion-only association, which is faster by the
            ROI pooling and markedly worse through occlusions.
        skip_frames: Process every N-th frame (0 = every frame). Note the
            tracker sees only the processed frames, so objects move further
            between them and the expansion has more work to do.

    Yields:
        ``(frame_index, frame, detections)`` for each processed frame, where
        ``detections`` carries ``tracker_id`` and holds only the detections
        that matched a track.
    """
    from modern_yolonas.tracking import DeepHMSort

    if tracker is None:
        tracker = DeepHMSort()
    if conf_threshold is None:
        conf_threshold = tracker.track_low_threshold

    tasks = Task.DETECT | Task.EMBED_OBJECTS if appearance else Task.DETECT

    import cv2

    cap = cv2.VideoCapture(str(source) if isinstance(source, Path) else source)
    if not cap.isOpened():
        raise FileNotFoundError(f"Cannot open video: {source}")

    # The tracker's memory is written in seconds; tell it how long a frame is.
    # Files usually report an honest rate, cameras often report 0 or something
    # absurd, so an implausible value is left alone rather than believed. With
    # `skip_frames` the tracker sees a slower video than the file claims.
    reported = cap.get(cv2.CAP_PROP_FPS)
    if 1.0 <= reported <= 240.0:
        tracker.frame_rate = reported / (skip_frames + 1)

    frame_idx = 0
    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break

            if skip_frames > 0 and frame_idx % (skip_frames + 1) != 0:
                frame_idx += 1
                continue

            t0 = time.perf_counter()
            prediction = self.predict(frame, tasks, conf_threshold, iou_threshold)
            tracked = tracker.update_with_detections(prediction.detections)
            self.last_inference_ms = (time.perf_counter() - t0) * 1000.0

            yield frame_idx, frame, tracked
            frame_idx += 1
    finally:
        cap.release()

modern_yolonas.inference.detect.YoloNASDetector.track_video_to_file(source, output, tracker=None, conf_threshold=None, iou_threshold=None, appearance=True, codec='mp4v', show_fps=False)

Track a video and write it out with ids drawn on.

Parameters:

Name	Type	Description	Default
source	str | Path	Input video path.	required
output	str | Path	Output video path.	required
tracker	DeepHMSort | None	See :meth:track_video.	None
conf_threshold	float | None	See :meth:track_video.	None
iou_threshold	float | None	Override the instance default.	None
appearance	bool	See :meth:track_video.	True
codec	str	FourCC codec string.	'mp4v'
show_fps	bool	Burn the per-frame time and frame rate into the output.	False

Returns:

Type	Description
dict[str, int | float]	Dict with total_frames, total_detections and unique_ids —
dict[str, int | float]	the last being how many distinct objects the tracker believes it saw,
dict[str, int | float]	which is the number to watch when tuning: far above the truth means
dict[str, int | float]	ids are fragmenting.

Source code in src/modern_yolonas/inference/detect.py

def track_video_to_file(
    self,
    source: str | Path,
    output: str | Path,
    tracker: DeepHMSort | None = None,
    conf_threshold: float | None = None,
    iou_threshold: float | None = None,
    appearance: bool = True,
    codec: str = "mp4v",
    show_fps: bool = False,
) -> dict[str, int | float]:
    """Track a video and write it out with ids drawn on.

    Args:
        source: Input video path.
        output: Output video path.
        tracker: See :meth:`track_video`.
        conf_threshold: See :meth:`track_video`.
        iou_threshold: Override the instance default.
        appearance: See :meth:`track_video`.
        codec: FourCC codec string.
        show_fps: Burn the per-frame time and frame rate into the output.

    Returns:
        Dict with ``total_frames``, ``total_detections`` and ``unique_ids`` —
        the last being how many distinct objects the tracker believes it saw,
        which is the number to watch when tuning: far above the truth means
        ids are fragmenting.
    """
    import cv2

    cap = cv2.VideoCapture(str(source))
    if not cap.isOpened():
        raise FileNotFoundError(f"Cannot open video: {source}")
    fps = cap.get(cv2.CAP_PROP_FPS)
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    cap.release()

    writer = cv2.VideoWriter(str(output), cv2.VideoWriter_fourcc(*codec), fps, (width, height))
    if not writer.isOpened():
        raise RuntimeError(f"Failed to create video writer for {output} with codec '{codec}'")

    seen: set[int] = set()
    total_detections = 0
    frames = 0

    try:
        for frame_idx, frame, detections in self.track_video(
            source, tracker, conf_threshold, iou_threshold, appearance
        ):
            writer.write(self.annotate_tracks(frame, detections, show_fps=show_fps))
            frames = frame_idx + 1
            total_detections += len(detections)
            if detections.tracker_id is not None:
                seen.update(int(i) for i in detections.tracker_id)
    finally:
        writer.release()

    return {
        "total_frames": frames,
        "total_detections": total_detections,
        "unique_ids": len(seen),
        "fps": fps,
    }

modern_yolonas.inference.detect.YoloNASDetector.annotate_tracks(image, detections, show_fps=False)

Draw tracked boxes, coloured and labelled by tracker_id.

Colouring by id rather than by class is what makes an ID-swap visible: the box changes colour the instant the tracker changes its mind.

Parameters:

Name	Type	Description	Default
image	ndarray	BGR frame to draw on.	required
detections	Detections	Output of :meth:track_video.	required
show_fps	bool	Overlay the last frame's time and frame rate.	False

Source code in src/modern_yolonas/inference/detect.py

def annotate_tracks(
    self, image: np.ndarray, detections: sv.Detections, show_fps: bool = False
) -> np.ndarray:
    """Draw tracked boxes, coloured and labelled by ``tracker_id``.

    Colouring by id rather than by class is what makes an ID-swap visible: the
    box changes colour the instant the tracker changes its mind.

    Args:
        image: BGR frame to draw on.
        detections: Output of :meth:`track_video`.
        show_fps: Overlay the last frame's time and frame rate.
    """
    if detections.tracker_id is None:
        return self.annotate(image, detections, show_fps=show_fps)

    labels = []
    for i, track_id in enumerate(detections.tracker_id):
        name = detections.data.get("class_name", [None] * len(detections))[i]
        labels.append(f"#{int(track_id)} {name}" if name is not None else f"#{int(track_id)}")

    annotated = self._track_box_annotator.annotate(image.copy(), detections)
    annotated = self._track_label_annotator.annotate(annotated, detections, labels=labels)
    if show_fps and self.last_inference_ms is not None:
        import cv2

        rate = 1000.0 / self.last_inference_ms if self.last_inference_ms > 0 else 0.0
        cv2.putText(
            annotated, f"{self.last_inference_ms:.1f}ms ({rate:.1f} FPS)",
            (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 0), 2,
        )
    return annotated

Matching primitives

The cost functions are plain numpy and are usable on their own — to build a different tracker, or to check what the tracker is seeing on a frame you care about.

modern_yolonas.tracking.matching.expand_boxes(boxes, expansion)

Grow each box outwards by expansion times its own width and height.

Each side moves out by expansion * w (or * h), so the expanded box is (1 + 2 * expansion) times as wide and as tall, centred on the original. expansion=0 returns the box unchanged.

This is ExpansionIoU's remedy for fast motion: two boxes of the same object in consecutive frames can have no overlap at all, which a plain IoU cost reads as "different objects". Expanding both before intersecting restores a usable signal without expanding the detector's own boxes.

.. note:: The reference Deep-EIoU implementation's expand() adds half the expanded width to each side rather than half the increase, so for the same e its boxes end up 2 * (1 + e) times as wide — larger than the (1 + 2e) the name implies. This uses the plain reading. It matters when porting a tuned expansion value across the two.

Parameters:

Name	Type	Description	Default
boxes	ndarray	(N, 4) in (x1, y1, x2, y2).	required
expansion	float	Non-negative growth factor.	required

Returns:

Type	Description
ndarray	(N, 4), a new array.

Source code in src/modern_yolonas/tracking/matching.py

def expand_boxes(boxes: np.ndarray, expansion: float) -> np.ndarray:
    """Grow each box outwards by ``expansion`` times its own width and height.

    Each side moves out by ``expansion * w`` (or ``* h``), so the expanded box is
    ``(1 + 2 * expansion)`` times as wide and as tall, centred on the original.
    ``expansion=0`` returns the box unchanged.

    This is ExpansionIoU's remedy for fast motion: two boxes of the same object in
    consecutive frames can have no overlap at all, which a plain IoU cost reads as
    "different objects". Expanding both before intersecting restores a usable
    signal without expanding the detector's own boxes.

    .. note::
       The reference Deep-EIoU implementation's ``expand()`` adds half the
       *expanded* width to each side rather than half the *increase*, so for the
       same ``e`` its boxes end up ``2 * (1 + e)`` times as wide — larger than the
       ``(1 + 2e)`` the name implies. This uses the plain reading. It matters when
       porting a tuned ``expansion`` value across the two.

    Args:
        boxes: ``(N, 4)`` in ``(x1, y1, x2, y2)``.
        expansion: Non-negative growth factor.

    Returns:
        ``(N, 4)``, a new array.
    """
    if expansion < 0:
        raise ValueError(f"expansion must be >= 0, got {expansion}")
    boxes = np.asarray(boxes, dtype=np.float32)
    if len(boxes) == 0:
        return boxes.reshape(0, 4).copy()

    margin_x = (boxes[:, 2] - boxes[:, 0]) * expansion
    margin_y = (boxes[:, 3] - boxes[:, 1]) * expansion
    grown = boxes.copy()
    grown[:, 0] -= margin_x
    grown[:, 2] += margin_x
    grown[:, 1] -= margin_y
    grown[:, 3] += margin_y
    return grown

modern_yolonas.tracking.matching.expansion_iou_distance(a, b, expansion)

1 - IoU between the two box sets after both are expanded.

Parameters:

Name	Type	Description	Default
a	ndarray	(M, 4) boxes, usually the tracks' last known positions.	required
b	ndarray	(N, 4) boxes, usually this frame's detections.	required
expansion	float	Passed to :func:expand_boxes; both sides are expanded.	required

Returns:

Type	Description
ndarray	(M, N) in [0, 1], 0 for a perfect overlap.

Source code in src/modern_yolonas/tracking/matching.py

def expansion_iou_distance(a: np.ndarray, b: np.ndarray, expansion: float) -> np.ndarray:
    """``1 - IoU`` between the two box sets after both are expanded.

    Args:
        a: ``(M, 4)`` boxes, usually the tracks' last known positions.
        b: ``(N, 4)`` boxes, usually this frame's detections.
        expansion: Passed to :func:`expand_boxes`; both sides are expanded.

    Returns:
        ``(M, N)`` in ``[0, 1]``, 0 for a perfect overlap.
    """
    return (1.0 - box_iou(expand_boxes(a, expansion), expand_boxes(b, expansion))).astype(np.float32)

modern_yolonas.tracking.matching.cosine_distance(a, b)

Pairwise cosine distance, rescaled to [0, 1].

Cosine distance 1 - cos lives in [0, 2] for vectors that may point opposite ways. Halving it puts it on the same scale as the IoU distance it is about to be fused with, which is what makes a single threshold — and a harmonic mean — meaningful across both.

Rows are L2-normalized here, so a caller that skipped normalization, or an EMA that drifted off the unit sphere, still gets a true cosine. A zero row gives distance 0.5 against everything (no information), not a NaN.

Parameters:

Name	Type	Description	Default
a	ndarray	(M, D).	required
b	ndarray	(N, D).	required

Returns:

Type	Description
ndarray	(M, N) in [0, 1].

Source code in src/modern_yolonas/tracking/matching.py

def cosine_distance(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    """Pairwise cosine distance, rescaled to ``[0, 1]``.

    Cosine distance ``1 - cos`` lives in ``[0, 2]`` for vectors that may point
    opposite ways. Halving it puts it on the same scale as the IoU distance it is
    about to be fused with, which is what makes a single threshold — and a
    harmonic mean — meaningful across both.

    Rows are L2-normalized here, so a caller that skipped normalization, or an EMA
    that drifted off the unit sphere, still gets a true cosine. A zero row gives
    distance 0.5 against everything (no information), not a NaN.

    Args:
        a: ``(M, D)``.
        b: ``(N, D)``.

    Returns:
        ``(M, N)`` in ``[0, 1]``.
    """
    a = np.asarray(a, dtype=np.float32)
    b = np.asarray(b, dtype=np.float32)
    if a.size == 0 or b.size == 0:
        return np.zeros((len(a), len(b)), dtype=np.float32)
    a = np.atleast_2d(a)
    b = np.atleast_2d(b)
    if a.shape[1] != b.shape[1]:
        raise ValueError(f"embedding width mismatch: {a.shape[1]} vs {b.shape[1]}")

    a = a / np.maximum(np.linalg.norm(a, axis=1, keepdims=True), 1e-12)
    b = b / np.maximum(np.linalg.norm(b, axis=1, keepdims=True), 1e-12)
    return np.clip((1.0 - a @ b.T) / 2.0, 0.0, 1.0).astype(np.float32)

modern_yolonas.tracking.matching.harmonic_mean(d1, d2)

Element-wise harmonic mean of two cost matrices.

2 * d1 * d2 / (d1 + d2), the form Deep HM-SORT puts in place of Deep-EIoU's min(d1, d2).

The harmonic mean leans towards the smaller of the two, so a strong cue still carries the pair — but unlike min it does not ignore the other one. Two teammates in the same kit have near-identical embeddings; min will happily match the wrong one on appearance alone, while the harmonic mean lets the motion cost veto it. That is the paper's whole argument for the swap, and its ablation is where the ID-switch count drops.

Both inputs are expected in [0, 1]. A zero anywhere makes the result zero, including 0 against 0, which the plain quotient would make a NaN.

Parameters:

Name	Type	Description	Default
d1	ndarray	(M, N) cost matrix, e.g. expansion-IoU distance.	required
d2	ndarray	(M, N) cost matrix, e.g. cosine distance.	required

Returns:

Type	Description
ndarray	(M, N).

Source code in src/modern_yolonas/tracking/matching.py

def harmonic_mean(d1: np.ndarray, d2: np.ndarray) -> np.ndarray:
    """Element-wise harmonic mean of two cost matrices.

    ``2 * d1 * d2 / (d1 + d2)``, the form Deep HM-SORT puts in place of
    Deep-EIoU's ``min(d1, d2)``.

    The harmonic mean leans towards the smaller of the two, so a strong cue still
    carries the pair — but unlike ``min`` it does not *ignore* the other one. Two
    teammates in the same kit have near-identical embeddings; ``min`` will happily
    match the wrong one on appearance alone, while the harmonic mean lets the
    motion cost veto it. That is the paper's whole argument for the swap, and its
    ablation is where the ID-switch count drops.

    Both inputs are expected in ``[0, 1]``. A zero anywhere makes the result zero,
    including ``0`` against ``0``, which the plain quotient would make a NaN.

    Args:
        d1: ``(M, N)`` cost matrix, e.g. expansion-IoU distance.
        d2: ``(M, N)`` cost matrix, e.g. cosine distance.

    Returns:
        ``(M, N)``.
    """
    d1 = np.asarray(d1, dtype=np.float32)
    d2 = np.asarray(d2, dtype=np.float32)
    total = d1 + d2
    return np.where(total > 0, 2.0 * d1 * d2 / np.maximum(total, 1e-12), 0.0).astype(np.float32)

modern_yolonas.tracking.matching.fuse_costs(iou_distance, embedding_distance, *, proximity_threshold, appearance_threshold, fusion='harmonic')

Combine a motion cost and an appearance cost into the association cost.

Two gates run before the fusion, both from BoT-SORT by way of Deep-EIoU:

• an appearance cost above appearance_threshold is not evidence of anything — same-team players sit well inside it — so it is discarded;
• an appearance cost is not trusted for a pair that is nowhere near each other geometrically, so an IoU distance above proximity_threshold discards it too. Without this, a track can be re-attached across the frame by appearance alone.

A pair whose appearance cost is discarded falls back to the motion cost alone, rather than being fused with a sentinel. That is a decision the paper does not make for us, and it is not cosmetic: harmonic_mean(d, 1) is 2d / (d + 1), which is larger than d for every d < 1. Fusing against a placeholder would therefore charge a pair for evidence that was never available — and since the gate fires on exactly the distant pairs the expansion scale-up exists to reach, it would quietly cancel the scale-up (match_threshold 0.8 would start refusing at an IoU distance of 0.67). Deep-EIoU's min has no such effect, so falling back also keeps the two fusions comparable where only one cue exists, which is what makes an ablation between them mean anything.

Parameters:

Name	Type	Description	Default
iou_distance	ndarray	(M, N) expansion-IoU distance.	required
embedding_distance	ndarray | None	(M, N) cosine distance, or None to associate on motion alone. Individual columns may be all-NaN to mark detections that arrived without an embedding; those fall back to motion too.	required
proximity_threshold	float	IoU-distance gate on the appearance cost.	required
appearance_threshold	float	Cosine-distance gate on the appearance cost.	required
fusion	str	"harmonic" for Deep HM-SORT, "min" for Deep-EIoU's original min(d1, d2). The second exists so the difference can be measured rather than asserted.	'harmonic'

Returns:

Type	Description
ndarray	(M, N) association cost.

Source code in src/modern_yolonas/tracking/matching.py

def fuse_costs(
    iou_distance: np.ndarray,
    embedding_distance: np.ndarray | None,
    *,
    proximity_threshold: float,
    appearance_threshold: float,
    fusion: str = "harmonic",
) -> np.ndarray:
    """Combine a motion cost and an appearance cost into the association cost.

    Two gates run before the fusion, both from BoT-SORT by way of Deep-EIoU:

    * an appearance cost above ``appearance_threshold`` is not evidence of
      anything — same-team players sit well inside it — so it is discarded;
    * an appearance cost is not *trusted* for a pair that is nowhere near each
      other geometrically, so an IoU distance above ``proximity_threshold``
      discards it too. Without this, a track can be re-attached across the frame
      by appearance alone.

    A pair whose appearance cost is discarded falls back to the motion cost
    **alone**, rather than being fused with a sentinel. That is a decision the
    paper does not make for us, and it is not cosmetic: ``harmonic_mean(d, 1)`` is
    ``2d / (d + 1)``, which is *larger* than ``d`` for every ``d < 1``. Fusing
    against a placeholder would therefore charge a pair for evidence that was
    never available — and since the gate fires on exactly the distant pairs the
    expansion scale-up exists to reach, it would quietly cancel the scale-up
    (``match_threshold`` 0.8 would start refusing at an IoU distance of 0.67).
    Deep-EIoU's ``min`` has no such effect, so falling back also keeps the two
    fusions comparable where only one cue exists, which is what makes an ablation
    between them mean anything.

    Args:
        iou_distance: ``(M, N)`` expansion-IoU distance.
        embedding_distance: ``(M, N)`` cosine distance, or ``None`` to associate
            on motion alone. Individual columns may be all-``NaN`` to mark
            detections that arrived without an embedding; those fall back to
            motion too.
        proximity_threshold: IoU-distance gate on the appearance cost.
        appearance_threshold: Cosine-distance gate on the appearance cost.
        fusion: ``"harmonic"`` for Deep HM-SORT, ``"min"`` for Deep-EIoU's
            original ``min(d1, d2)``. The second exists so the difference can be
            measured rather than asserted.

    Returns:
        ``(M, N)`` association cost.
    """
    if fusion not in ("harmonic", "min"):
        raise ValueError(f"fusion must be 'harmonic' or 'min', got {fusion!r}")
    if embedding_distance is None or embedding_distance.size == 0:
        return iou_distance.astype(np.float32)

    appearance = np.array(embedding_distance, dtype=np.float32, copy=True)
    usable = (
        np.isfinite(appearance)
        & (appearance <= appearance_threshold)
        & (iou_distance <= proximity_threshold)
    )
    appearance[~usable] = GATED

    fused = (
        np.minimum(iou_distance, appearance)
        if fusion == "min"
        else harmonic_mean(iou_distance, appearance)
    )
    return np.where(usable, fused, iou_distance).astype(np.float32)

modern_yolonas.tracking.matching.linear_assignment(cost, threshold)

Hungarian assignment, with pairs costlier than threshold refused.

Uses scipy.optimize.linear_sum_assignment — an optimal solver over the whole matrix — and then drops the pairs that came back above threshold. Gated entries are raised above every candidate first so the solver routes around them where it can; a gated pair that survives that is still dropped.

Parameters:

Name	Type	Description	Default
cost	ndarray	(M, N).	required
threshold	float	Highest cost still considered a match.	required

Returns:

Type	Description
list[tuple[int, int]]	(matches, unmatched_rows, unmatched_cols), where matches is a list
list[int]	of (row, col).

Source code in src/modern_yolonas/tracking/matching.py

def linear_assignment(
    cost: np.ndarray, threshold: float
) -> tuple[list[tuple[int, int]], list[int], list[int]]:
    """Hungarian assignment, with pairs costlier than ``threshold`` refused.

    Uses ``scipy.optimize.linear_sum_assignment`` — an optimal solver over the
    *whole* matrix — and then drops the pairs that came back above ``threshold``.
    Gated entries are raised above every candidate first so the solver routes
    around them where it can; a gated pair that survives that is still dropped.

    Args:
        cost: ``(M, N)``.
        threshold: Highest cost still considered a match.

    Returns:
        ``(matches, unmatched_rows, unmatched_cols)``, where ``matches`` is a list
        of ``(row, col)``.
    """
    cost = np.asarray(cost, dtype=np.float32)
    rows, cols = cost.shape if cost.ndim == 2 else (0, 0)
    if rows == 0 or cols == 0:
        return [], list(range(rows)), list(range(cols))

    # A cost above the threshold can never become a match, so make it maximally
    # unattractive rather than merely expensive — otherwise the optimal total can
    # be reached by taking one impossible pair to cheapen another.
    solvable = np.where(cost > threshold, threshold + 1.0 + np.max(cost), cost)
    row_idx, col_idx = linear_sum_assignment(solvable)

    matches = [(int(r), int(c)) for r, c in zip(row_idx, col_idx) if cost[r, c] <= threshold]
    matched_rows = {r for r, _ in matches}
    matched_cols = {c for _, c in matches}
    return (
        matches,
        [r for r in range(rows) if r not in matched_rows],
        [c for c in range(cols) if c not in matched_cols],
    )