Zero-Shot Open-Vocabulary Tracking with

Large Pre-Trained Models

Wen-Hsuan Chu1 , Adam W. Harley2 , Pavel Tokmakov3 , Achal Dave3 , Leonidas Guibas2 , Katerina Fragkiadaki1

Abstract— Object tracking is central to robot perception exceeding their accuracy. Recent methods for visual tracking
and scene understanding. Tracking-by-detection has long been build upon transformer architectures, where feature vectors
a dominant paradigm for object tracking of specific object represent tracked objects, and these are re-contextualized in
arXiv:2310.06992v2 [cs.CV] 25 Jan 2024

categories [1, 2]. Recently, large-scale pre-trained models have

shown promising advances in detecting and segmenting objects each frame by attending to the pixel features and are used to
and parts in 2D static images in the wild. This begs the question: predict per-frame bounding boxes [14, 15, 16]. These methods
can we re-purpose these large-scale pre-trained static image are trained on annotated video data and do not capitalize on
models for open-vocabulary video tracking? In this paper, we pre-trained static image detectors beyond the pre-training of
re-purpose an open-vocabulary detector [3], segmenter [4], and their feature backbones [14].
dense optical flow estimator [5], into a model that tracks
and segments objects of any category in 2D videos. Our Meanwhile, 2D image object detection has been recently
method predicts object and part tracks with associated language revolutionized with open-world image detectors, jointly
descriptions in monocular videos, rebuilding the pipeline of trained for referential grounding and category grounding of
Tractor [6] with modern large pre-trained models for static thousands of object categories [3, 17, 18] across millions
image detection and segmentation: we detect open-vocabulary of images. Can we capitalize on this and make practical
object instances and propagate their boxes from frame to frame
using a flow-based motion model, refine the propagated boxes progress for tracking-by-detection, by re-visiting the Tracktor
with the box regression module of the visual detector, and prompt paradigm [6] with these updated components? In other words,
an open-world segmenter with the refined box to segment the can we re-purpose large pre-trained image models into a
objects. We decide the termination of an object track based on zero-shot open-vocabulary tracker, without ever fine-tuning
the objectness score of the propagated boxes, as well as forward- on video data?
backward optical flow consistency. We re-identify objects across
occlusions using deep feature matching. We show that our We propose a simple and extensible framework for ex-
model achieves strong performance on multiple established video ploring this question, which does not introduce significant
object segmentation and tracking benchmarks [7, 8, 9, 10], and advancements or innovative approaches. We use an open-
can produce reasonable tracks in manipulation data [11]. In vocabulary detector to find objects as they appear [3], obtain
particular, our model outperforms previous state-of-the-art in their masks using an off-the-shelf segmenter [4], propagate
UVO and BURST, benchmarks for open-world object tracking
and segmentation, despite never being explicitly trained for the boxes to the next frame using a motion transformation
tracking. We hope that our approach can serve as a simple and computed from optical flow [5], refine the boxes using the
extensible framework for future research. The project page can detector’s bounding box regression module, and segment
be found here. the box interiors using an off-the-shelf segmenter [4]. We
handle ambiguity in per-frame segmentations by selecting the
I. I NTRODUCTION segmentations with the highest temporal consistency. Finally,
We are interested in the problem of tracking arbitrary we revise the bounding boxes using the segmentation results.
objects in video. A reasonable strategy for this task, which We test our method on multiple established video object
has dominated the area for multiple years, is “tracking by segmentation and tracking benchmarks: UVO [9] and BURST
detection” [12]. Tracking by detection splits the task into two [10], as well as traditional VOS benchmarks like DAVIS [7]
independent problems: (1) detect objects frame-by-frame, and and YoutubeVOS [8]. In open-world datasets like UVO and
(2) associate detection responses across frames. Because of BURST, our method outperforms the previous state-of-the-
its two-stage split, tracking-by-detection is mainly propelled art and achieves competitive performance in DAVIS and
forward by advances in detection. Notably, Tracktor [6] used YoutubeVOS when evaluating VOS baselines using detected
person detections from a Faster R-CNN [13], propagated first-frame masks. Our tracker can also provide reasonable
boxes with a simple motion model, and refined these boxes object tracks in RoboTAP [11], a manipulation-based dataset.
using the Faster-RCNN’s box regression head. This yielded Our method also provides a natural-language interface for
a tracker composed entirely of static image neural modules, tracking, where a user may describe the tracking target in
far simpler than its contemporary methods while matching or words, and the model delivers the corresponding frame-by-
frame segmentation. Given that our approach can improve
1 Carnegie Mellon University <wenhsuac,katef>@cs.cmu.edu,
2 Stanford University <aharley,guibas>@cs.stanford.edu,
as new pre-trained models are swapped in, we hope that our
3 Toyota Research Institute <pavel.tokmakov,achal.dave>@tr approach will serve as a simple yet extensible framework for
i.global future work.
 II. R ELATED W ORK Proposal Network (RPN), similar to Faster-RCNN [13]. In the
Tracking by detection. Many modern object tracking second stage, it spatially refines each box using a regression
approaches rely heavily on accurate per-frame detectors [19, module and predicts an objectness score and category label for
6, 20, 21, 22, 23]. These approaches show that simple post- each. In addition, a category-agnostic mask prediction head
processing of per-frame detection can lead to strong tracking is trained to segment the object in each predicted bounding
approaches for a closed (usually small) set of objects. In box. Our model exploits Detic’s ability to detect and label
particular, CenterTrack [19] also showed that a tracker can be object boxes and also re-uses its bounding box refinement
obtained simply from training on (augmented) static images, module during tracking.
by modeling humans and vehicles as points. However, in open- b) Promptable general-purpose segmenter: For seg-
world settings, this is less feasible, as objects may overlap menting masks from object boxes, we rely on SAM [4], a
and have different sizes (e.g. the upper half body of a person recent interactive general-purpose segmenter, which produces
and their shirt), making points an ambiguous descriptor for a segmentation given box or point prompts that indicate
open-world tracking. Our work is most closely related to the object of interest. SAM is a transformer-based [42]
Tracktor [6], which directly uses a Faster R-CNN [13] person model with a large and high-resolution image encoder and
detector to build an accurate person tracker. Our work builds a lightweight prompt-conditioned mask head. For each user
on this method, extending it to any category, using a strong prompt, SAM predicts multiple segmentation hypotheses.
open-vocabulary detector [3] as the backbone. c) Optical flow estimation: We estimate the motion
Open vocabulary detection. Recent advances in open- transformation of an object box to propagate it from frame to
vocabulary classification [24, 25] have significantly improved frame. We use GMFlow [5], which is a state-of-the-art optical
open-vocabulary detectors. Open-vocabulary, or zero-shot, flow method. GMFlow takes two consecutive frames as input,
detectors largely operate by using language models to and produces a 2D pixel displacement map as output, using
generalize to unseen object classes. Early approaches relied an architecture that computes a spatial argmax of feature
on using text embeddings from pre-trained language models, correlations for each pixel, trained on large synthetic datasets.
such as from BERT [26] or GLOVE [27], as classifiers for We use this flow map to estimate the motion of detected
object proposals [28, 29, 30]. More recent work leverages text boxes and also rely on optical forward-backward flow cycle-
embeddings which are pre-trained to be aligned with vision consistency [43] to estimate occlusion, in which case we
embeddings [24, 25], leading to significant improvements terminate the track.
in accuracy [3, 31, 32]. We show that this recent class of B. OVTracktor
approaches can be directly generalized to open-vocabulary
tracking, using Detic [3] as a representative model. Given an RGB video as input, the goal of OVTracktor is
Open world tracking. Object tracking has traditionally to estimate mask trajectories for all objects in the video and
focused on a few categories, such as people and vehicles. Very estimate category labels for those objects. A mask trajectory
recently, the community has seen renewed efforts to generalize for object i is a sequence of image binary masks Mi =
tracking to arbitrary objects. Traditional approaches focused {mti | t ∈ [0, T ]}, where mti ∈ RW ×H , where W × H denote
on motion-based segmentation [33, 34, 35, 36, 37], leveraging the width and height of the image frame, and t is the frame
motion as a cue to segment never-before-seen objects. More index in time. Each object is associated with a category label,
recent approaches use open-world object proposal methods which we denote with ℓi . We denote the set of all binary
to detect objects per frame and link them together using a instance masks in frame t as M t = {mt0 , mt1 , . . .}.
combination of temporal consistency, appearance, and motion a) Detection: We run the detector on every frame. Let
cues [38, 39, 40]. Our work extends this latter class of Dt denote the object detections and segmentations supplied
approaches to open-vocabulary detectors. by the detector at frame t. At t = 0, our tracker initializes
object masks M 0 from the set of Detic object detections D0 ,
III. M ETHOD thresholded at a confidence threshold λc = 0.5.
Our method builds upon existing open-vocabulary detec- b) Motion-driven box propagation: We propagate ob-
tors [3], promptable general-purpose segmenters [4], and ject boxes across consecutive frames using a 4-parameter
dense optical flow estimators [5]. We call our model Open- box motion transformation that includes a box translation
Vocabulary Multi-Object Tracker, or OVTracktor for short. (dx, dy) and width and height scaling (sx , sy ) using motion
Figure 1 shows an illustration of our model architecture. Our information obtained from an optical flow field of [5]. We
model does not require any tracking-specific training. In this filter the pixel displacement vectors that are forward-backward
section, we first introduce the modules that we rely on, then consistent [44]. A lenient criterion is used: we simply check
discuss how we combine them into OVTracktor. if the forward-backward flows have segmentation consistency,
in the sense that tracking forward and backward leads back
A. Building Blocks to the original instance mask, instead of thresholding the
a) Open-vocabulary object detector: We use Detic [3] forward-backward displacement from the origin. We use the
with a Swin-B [41] backbone as our open vocabulary object filtered pixels to compute a box motion transformation using
detector. Detic is a two-stage detector. In the first stage, it least squares and use this to propagate the box forward. After
generates a large number of candidate boxes with a Region this motion warp, the box is still axes-aligned, we do not
 Past boxes Transformed boxes Refined boxes Masks

Transform or
Refine Segment
terminate

Mask to box

Optical flow
Re-ID or
Detect
spawn

Past mask tracks t-1 t Input frame Detections Mask tracks

Fig. 1: Architecture of OVTracktor. An open-vocabulary detector detects objects and an open-world segmenter segments
their masks. We propagate boxes to the next frame or decide their termination using an optical flow-based motion model.
The propagated boxes are refined with the detector’s box regression module. Refined boxes are used to prompt the segmenter
in the next frame. The detections and their associated appearance features from the next frame are used to determine whether
new tracks should be spawned or merged with previously terminated tracks.

consider object rotation or anisotropic scaling. The category f) Spawning new object tracks: At each frame, we need
label and instance ID of the box are maintained. to take into account new objects that enter the scene or
c) Object track termination: We determine if an object reappear after an occlusion. For each detection in Dt+1 , we
track should be terminated due to occlusions by checking if compute its IoU with all the masks in M t . A new track is
the ratio of forward-backward flow consistent pixels is lower spawned if the IoU between the detection and all masks in
than a fixed ratio λf low or if the object-ness score of the box M t is below some specified threshold λspawn .
is too low. g) Track re-identification: We use appearance feature
matching to determine whether to merge a new track with an
d) Object box refinement: We refine the propagated (non-
existing but terminated track. We store a small window of
terminated) boxes at frame t + 1 using Detic’s bounding box
features before a track’s termination and compare them with
regression module. This adjusts the bounding boxes according
the features of the newly spawned tracks. Newly spawned
to objectness cues on that frame and gives us higher-quality
tracks are considered for Re-ID until Treid time-steps have
box estimates in frame t + 1.
passed. We used the box features from Detic (before the box
e) Temporally consistent object segmentation: The regression module) and normalized them along the channel
bounding box estimates at frame t + 1 are used to prompt dimension to obtain a small set of features that represent
SAM to segment the object’s interior. SAM produces multiple each instance. We then compute the inner product between
segmentation mask candidates per box, to handle ambiguity normalized appearance features for any two tracks and merge
regarding what to segment from the box’s interior. Overall, them if their value is above a threshold λreid .
we found a box prompt often unambiguously determines the None of OVTracktor’s described components require any
object to segment (so all resulting masks will be identical), additional training. As detectors and segmenters improve,
in contrast to a center object point prompt, which does not the components can be easily swapped out for potential
have information regarding the object’s extent. To handle the improvements to the tracker.
cases where this ambiguity exists, we implement a form of Implementation details. During inference, we apply test
temporal cycle consistency at the mask level. time augmentation to video frames when running the detector
SAM segments an object via iterative attention between by scaling and horizontally flipping the individual frames
an object query vector and pixel features, and a final inner to obtain better object proposals, which we found to help
product between the contextualized query vector and pixel improve the recall of detections, especially in the harder
feature vectors. The three segmentation hypotheses consider open-world datasets. OVTracktor has the following hyper-
different object query initialization. For each box i, we use parameters λc = 0.5 for thresholding detector’s confidence,
the updated (contextualized) object query vector at frame λf low for deciding track termination due to occlusion,
t + 1 to segment the object at frame t via inner product with λspawn for instantiating new objects non-overlapping with
the pixel features from frame t; this results in a temporally existing objects tracks, and λreid for merging temporally non-
corresponding mask m̂ti . We select the SAM segmentation overlapping tracks during re-identification. We have found
hypothesis at frame t + 1 whose updated query vector-driven the model robust to the choice of these hyper-parameters,
segmentation m̂ti has the highest Intersection over Union due to the nature of videos: an object suffices to be detected
(IoU) with mti . We then update the object boxes to tightly confidently only very sparsely in time, and our propagation
contain the resulting segmentation mask. method will propagate it forward. In evaluations, we used
some videos in the training set to select a good set of TABLE I: Tracking performance in BURST [10]. Higher
hyperparameters for the individual datasets. As the memory is better. Best performing method is bolded.
consumption scales with the number of objects being tracked, Method HOTAall HOTAcom HOTAunc
we also put a hard limit K on the limit of tracks that can
STCN Tracker [10, 48] 5.5 17.5 2.5
co-exist at the same time. Box Tracker [10] 8.2 27.0 3.6

IV. E XPERIMENTS OVTracktor 12.5 27.4 8.8

We test OVTracktor in multi-object tracking and video

object segmentation benchmarks of BURST [10], UVO [9], TABLE II: Tracking performance in UVO [9] Higher is
DAVIS [7], and YoutubeVOS [8], along with some ablation better. Best performing online method is bolded.
studies and analysis on run-time speed. Qualitative tracking
results for the benchmark datasets, as well as a manipulation Method mAR100 AR.5 AR.75
dataset, RoboTAP [11], can be found in Figure 2. We further Mask2Former VIS [49] (Offline) 35.4 - -
show qualitative results on language-guided tracking, where MaskTrack R-CNN [50] 17.2 - -
our model tracks objects of specific object categories that the IDOL [51] 23.9 - -
user specifies in Figure 3. TAM [52] 24.1 - -
OVTracktor 28.1 44.1 31.0
A. Multi-Object Tracking and Video Object Segmentation
In each benchmark, we compare against multiple SOTA
methods to see where our method stands compared to the
specialized models proposed for each task. performance compared to the baselines, with a significant
1) BURST benchmark: BURST [10] extends TAO [45] to improvement when it comes to the tracking quality of the
mask annotations. Mask annotations in BURST come from a “uncommon” classes.
large set of 482 classes, with the videos covering multiple 2) UVO benchmark: UVO [9] is a recently proposed
types of scenes. These classes can be further divided into benchmark for Open-World Tracking. Like BURST, the
two subsets, the “common” set, which contains the 78 object mask annotations in UVO are not restricted to a small class
classes from COCO [46], and the “uncommon” set, which of objects. The dataset includes annotations for anything
contains the remaining infrequently occurring classes from that a human might consider as an “object”. However, the
LVIS [47]. In particular, we are interested in the “long-tail annotations are not (and cannot be) exhaustive of what an
class-guided” evaluation task, which requires us to detect object is, since there is great ambiguity of what is a trackable
and track objects corresponding to all classes, as well as entity. Since OVTracktor is capable of producing tracks for
predict the correct class label corresponding to each of the “anything” that is trackable, the model often produces object
object tracks. This allows us to evaluate the “open-vocabulary- tracks that may have not been annotated. Thus, we simply
ness” of OVTracktor, which is in stark contrast to other report the Average Recall (AR) at 0.5 and 0.75 spatio-temporal
class-agnostic (i.e. does not require predicting object labels) mask IOU as well as the mean Average Recall (mAR) over
existing benchmarks. We conduct our evaluations on the intervals from 0.5 to 0.95 IOU thresholds, similar to the
validation split of BURST, which contains 993 videos. evaluations in the COCO benchmark [46].
We compare the 2 baselines proposed in the BURST paper For baselines, we compare against MaskTrack R-CNN
that also follows the tracking-by-detection paradigm: (1) a box [50], which was proposed in the original UVO paper [9]. We
tracker, which simply links per-frame detections using IoU additionally consider IDOL [51], a transformer-based online
scores followed by Hungarian matching, and (2) an STCN tracker, and TAM [52], a concurrent work that combines
tracker, which uses STCN [48], a SOTA object tracking SAM with a SOTA object tracker XMem [53]. Furthermore,
method in the VOS literature, to propagate masks from frame we also include a SOTA offline tracker, Mask2Former VIS
t to t + 1, then uses the IoU scores of the propagated mask [49], that extends Mask2Former for tracking through spatial-
and the next frame detections to link object tracks. temporal attention to compare against the performance gap
We show quantitative results in Table I, with results between offline and online methods. Note that these baselines
reportedly separately for all classes, the common classes, and all require training on video data, while OVTracktor is strictly
the uncommon classes. Interestingly, despite STCN being zero-shot (i.e. not trained on any video data).
a significantly more advanced method, the tracking quality We show quantitative results in Table II. We can see
falls behind a simple box tracker. This is due to STCN and that OVTracktor outperforms other online-based methods
STCN-like methods assuming ground truth object masks in despite zero-shot. Many failure cases of OVTracktor are due
the initial frame as input, and when STCN receives noisy to ground-truth object definitions and OVTracktor detections
detections, it tends to propagate the masks in an erroneous being misaligned, which we show in Figure 4.
error, with the error compounding as the model tracks into 3) DAVIS and YoutubeVOS benchmark: We additionally
the future. This effect is more noticeable on BURST as the conduct experiments on DAVIS’17 [7] and YoutubeVOS [8],
detection task is significantly more difficult, often with parts 2 commonly used datasets for VOS tasks, where the ground
of the object being mis-detected. OVTracktor achieves higher truth masks are given in the initial frame, and the tracker needs
to track the objects throughout the entire video. The videos are
downsampled to 480p quality as per common practice in VOS
literature. To reduce the amount of computation, we filter the
detections in the first frame using the ground truth to reduce
the amount of object tracks. In subsequent frames, we reduce
the number of detections by only considering detections that
have the same class labels as the detections selected in the
first frame. We compare using the standard metrics: region
similarity J , contour accuracy F, and their average J &F.
For YoutubeVOS, we also report results separately for “seen”
or “unseen” categories, as per conventional practice, even if
OVTracktor does not explicitly distinguish between the two.
We consider two groups of VOS approaches as baselines:
memory-based approaches such as STM [54] and STCN [48],
which use a growing history of frames and segmentation
masks to propagate the segmentation in the new frame, as well
as non-memory-based approaches such as SiamMask [55],
UNICORN [56], Siam R-CNN [57], and UNINEXT [58], that
do not use a growing history to represent object templates.
All our baselines assume ground truth objects masks in
the first frame as input, which is the standard evaluation Fig. 2: Qualitative object tracking results in DAVIS’17,
setup of these benchmarks, while OVTracktor does not use YoutubeVOS, UVO, and RoboTAP, with frames selected
any ground-truth information. Hence, we conduct additional uniformly from the start (left) to the end (right) of a video.
experiments where we run the same detector (Detic [3]) in the OVTracktor can detect and track objects consistently through
initial frame to serve as input to the best-performing baseline, time and can distinguish between similar-looking instances.
STCN [48], for a more apples-to-apples comparison.
We show quantitative results for our model and baselines
on YoutubeVOS and DAVIS in Table III. We can see that
the performance drops considerably when we switch from
GT mask inputs to detection masks for STCN. This is due to
multiple reasons: (1) ambiguities in “what” to track, as seen
in Figure 4, where the tracker is missing the handheld bags
of the ladies, and (2) existing VOS-based approaches starts
to propagate the errors from detection predictions to future
frames, which we also observed in BURST. This difference
is further exaggerated in YoutubeVOS, where the detection
problem is harder: if the detector fails to detect the object
instance on the annotated frame, it’s not possible to associate Fig. 3: Language-guided tracking. Top two rows: The user
the object track with the correct instance ID in the GT, even prompts with “saddle”, and the model can recover the missing
if the detections are perfect in all subsequent frames. Since tracks (colored pink). We also draw a bounding box for clearer
annotations are usually supplied in the first frame an object visibility. Bottom two rows: The same interface can be easily
appears, there are many cases where the annotated frame is applied if the user only wishes to track specific objects in a
not an easy frame to detect the object, as the entities might video like “coffee makers” (colored pink and green).
have just started entering the scene.

B. Promptable Open-Vocabulary Tracking

C. Ablations
The object tracks our method delivers are associated with
corresponding category labels. A user can specify a category We ablate design choices of our method in Table V in the
to track, and then our method can deliver object tracks only DAVIS benchmark. Not using segmentation to adapt the object
of that specific category. We show such an example in Figure boxes gives a much worse performance, which shows that
3. When the human prompts the tracker for a specific object segmentation helps to track by providing higher quality boxes,
category, we use a lower confidence threshold λc for the as opposed to computing it simply as a post-processing step.
detector only for instances of the mentioned category, as the Not using mask cycle consistency hurts by a small amount.
human provides a powerful cue that the mentioned object is in Skipping the bounding box refinement also hurts by a great
the video. We show more examples in the supplementary file, amount, which indicates that an optical flow-based motion
as well as tracking objects based on open-world referential model alone is not sufficient to obtain good box prompts for
utterances by using a referential detector [17]. the segmenter. Skipping the flow-based motion propagation
 TABLE III: Performance on YoutubeVOS 2018 [8] and DAVIS’17 [7] . Higher is better.
YoutubeVOS 2018 Val DAVIS’17 Val
Method
G Js Fs Ju Fu J &F J F
SiamMask [55] 52.8 60.2 58.2 45.1 47.7 56.4 54.3 58.5
Unicorn [56] - - - - - 69.2 65.2 73.2
Siam R-CNN [57] 73.2 73.5 - 66.2 - 70.6 66.1 75.0
Ground-Truth
UNINEXT [58] 78.6 79.9 84.9 70.6 79.2 81.8 77.7 85.8
masks at t = 0
STM [54] 79.4 79.7 84.2 72.8 80.9 81.8 79.2 84.3
STCN [48] 83.0 81.9 86.5 77.9 85.7 85.4 82.2 88.6
Detected masks Detic + STCN 58.8 68.1 71.7 45.3 50.3 76.5 74.5 78.5
at t = 0 OVTracktor 62.2 65.9 69.4 53.7 59.8 74.8 73.2 76.4

TABLE IV: Average running time in milliseconds for the

individual components in OVTracktor.
Component Avg. Run Time (ms)
Detic (with TTA) 1350
Optical Flow 275
Box Warping 150
SAM 550
Spawning/ReID 100

TABLE V: Ablative analysis in DAVIS’17. We show the

relative changes w.r.t. the complete model (top row). Fig. 4: Failure cases of OVTracktor. Top row: The definition
DAVIS’17 Val
of classes is often ambiguous. In this case, the handbags are
Method not considered a part of a human by our tracker. Middle row:
J &F J F
Even in cases without labels as in open-world settings, there
OVTracktor 74.8 73.2 76.4 can be multiple definitions of what an “object” is. Bottom
− Box Adaptation after Segmentation -3.1 -2.9 -3.3 row: Re-ID failures. In this example, we fail to match the
− Motion Based Propagation -4.3 -3.9 -4.5 upper half of a person (after reappearing) to an entire person
− Box Refinement -3.7 -3.4 -3.9
− Mask Cycle Consistency -2.0 -1.4 -2.7 (before occlusion).

also unsurprisingly hurts performance, since the bounding networks as an extension in our future work. Augmenting
box refinement alone cannot recover the correct object box, SAM with extra modules [59] for tighter spatial-temporal
especially for frames where the motion is large. reasoning, where mask query tokens attend to previous frames
D. Running Time Analysis to be better conditioned on a track’s history is another
interesting avenue of future work.
We analyze the running time of the individual components
in OVTracktor in Table IV. The results are reported over
VI. C ONCLUSION
the average of all the frames in the videos. The model runs
at 0.41 FPS on an Nvidia V100 and costs around 18GB of We present OVTracktor, a zero-shot framework for open-
VRAM to run on 480p videos, without caching anything vocabulary visual tracking, that re-purposes modules of
to disk. We can see that most of the running time is spent large-scale pre-trained models for object tracking in videos
in the Detic detector, with the SAM segmenter coming in without any training or finetuning. Our model can be applied
second. Test time augmentations (TTA) incur a big overhead equally well for video object segmentation and multi-object
for Detic, and for scenes where detection is easy, noticeable tracking across various benchmarks and helps unify the video
speedups can be achieved by turning off TTA. tracking and segmentation literature, which has been oven-
fragmented using different evaluation protocols and ground-
V. L IMITATIONS AND F UTURE W ORK truth information at test time. Instead of specifying what
A limitation of OVTracktor is the use of pre-trained to track with ground-truth object masks, which are hard
features for feature matching for re-identifying object tracks. to annotate and provide, our tracking framework offers a
Empirically we observed that these features are not necessarily language interface over what to focus on and track in videos.
temporally consistent when used directly out-of-the-box, Thanks to its simplicity, we hope that our model can serve
leading to many mistakes in the Re-ID process. We will as an upgradeable and extendable baseline for future work in
explore the possibility of training more general-purpose re-id open-world and open-vocabulary video tracking literature.
 R EFERENCES [28] A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang, “Open-vocabulary
object detection using captions,” in CVPR, 2021.
[1] M. Andriluka, S. Roth, and B. Schiele, “People-tracking-by-detection [29] A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran,
and people-detection-by-tracking,” CVPR, 2008. “Zero-shot object detection,” in ECCV, 2018.
[2] X. Weng and K. Kitani, “A baseline for 3d multi-object tracking,” [30] S. Rahman, S. Khan, and N. Barnes, “Improved visual-semantic
arXiV, 2019. alignment for zero-shot object detection,” in AAAI, 2020.
[3] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra, “Detecting [31] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-vocabulary object
twenty-thousand classes using image-level supervision,” in ECCV, 2022. detection via vision and language knowledge distillation,” ICLR, 2022.
[4] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, [32] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn,
T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., “Segment anything,” A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, et al.,
arXiv, 2023. “Simple open-vocabulary object detection with vision transformers,”
[5] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao, “Gmflow: Learning arXiv, 2022.
optical flow via global matching,” in CVPR, 2022. [33] J. Shi and J. Malik, “Motion segmentation and tracking using
[6] P. Bergmann, T. Meinhardt, and L. Leal-Taixe, “Tracking without bells normalized cuts,” in ICCV, 1998.
and whistles,” in CVPR, 2019. [34] M. Grundmann, V. Kwatra, M. Han, and I. Essa, “Efficient hierarchical
[7] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbelaez, A. Sorkine-Hornung, graph-based video segmentation,” in CVPR. IEEE, 2010.
and L. V. Gool, “The 2017 DAVIS challenge on video object [35] P. Ochs, J. Malik, and T. Brox, “Segmentation of moving objects by
segmentation,” arXiV, 2017. long term video analysis,” TPAMI, vol. 36, no. 6, 2013.
[8] N. Xu, L. Yang, Y. Fan, J. Yang, D. Yue, Y. Liang, B. L. Price, [36] K. Fragkiadaki, P. Arbelaez, P. Felsen, and J. Malik, “Learning to
S. Cohen, and T. S. Huang, “Youtube-vos: Sequence-to-sequence video segment moving objects in videos,” in CVPR, 2015.
object segmentation,” in ECCV, 2018. [37] P. Bideau, A. RoyChowdhury, R. R. Menon, and E. Learned-Miller,
[9] W. Wang, M. Feiszli, H. Wang, and D. Tran, “Unidentified video “The best of both worlds: Combining cnns and geometric constraints
objects: A benchmark for dense, open-world segmentation,” CoRR, for hierarchical motion segmentation,” in CVPR, 2018.
2021. [38] Y. Liu, I. E. Zulfikar, J. Luiten, A. Dave, D. Ramanan, B. Leibe,
[10] A. Athar, J. Luiten, P. Voigtlaender, T. Khurana, A. Dave, B. Leibe, A. Ošep, and L. Leal-Taixé, “Opening up open world tracking,” in
and D. Ramanan, “Burst: A benchmark for unifying object recognition, CVPR, 2022.
segmentation and tracking in video,” in WACV, 2023. [39] S. Li, M. Danelljan, H. Ding, T. E. Huang, and F. Yu, “Tracking every
[11] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, thing in the wild,” in ECCV, 2022.
R. Hadsell, L. Agapito, and J. Scholz, “Robotap: Tracking arbitrary
[40] A. Ošep, W. Mehner, P. Voigtlaender, and B. Leibe, “Track, then decide:
points for few-shot visual imitation,” arXiv, 2023.
Category-agnostic vision-based multi-object tracking,” in ICRA, 2018.
[12] M. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier, and L. Van Gool,
[41] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and
“Online multi-person tracking-by-detection from a single, uncalibrated
B. Guo, “Swin transformer: Hierarchical vision transformer using
camera.” IEEE Transactions on Pattern Analysis and Machine Intelli-
shifted windows,” in ICCV, 2021.
gence, 2011.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N.
[13] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R–CNN: Towards
Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in
real-time object detection with region proposal networks,” in NeurIPS,
NeurIPS, 2017.
2015.
[43] P. Pan, F. Porikli, and D. Schonfeld, “Recurrent tracking using multifold
[14] T. Meinhardt, A. Kirillov, L. Leal-Taixé, and C. Feichtenhofer,
consistency,” in Proceedings of the Eleventh IEEE International
“Trackformer: Multi-object tracking with transformers,” CoRR, 2021.
Workshop on Performance Evaluation of Tracking and Surveillance,
[15] A. Athar, J. Luiten, A. Hermans, D. Ramanan, and B. Leibe, “Hodor:
vol. 3, 2009.
High-level object descriptors for object re-segmentation in video learned
from static images,” in CVPR, 2022. [44] N. Sundaram, T. Brox, and K. Keutzer, “Dense point trajectories by
[16] P. Sun, Y. Jiang, R. Zhang, E. Xie, J. Cao, X. Hu, T. Kong, Z. Yuan, gpu-accelerated large displacement optical flow.” in ECCV, 2010.
C. Wang, and P. Luo, “Transtrack: Multiple-object tracking with [45] A. Dave, T. Khurana, P. Tokmakov, C. Schmid, and D. Ramanan,
transformer,” CoRR, 2020. “TAO: A large-scale benchmark for tracking any object,” in ECCV,
[17] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, 2020.
H. Su, J. Zhu, and L. Zhang, “Grounding dino: Marrying dino with [46] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays,
grounded pre-training for open-set object detection,” 2023. P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft
[18] H. Zhang, P. Zhang, X. Hu, Y.-C. Chen, L. H. Li, X. Dai, L. Wang, COCO: common objects in context,” CoRR, vol. abs/1405.0312, 2014.
L. Yuan, J.-N. Hwang, and J. Gao, “Glipv2: Unifying localization and [Online]. Available: http://arxiv.org/abs/1405.0312
vision-language understanding,” 2022. [47] A. Gupta, P. Dollár, and R. B. Girshick, “LVIS: A dataset for large
[19] X. Zhou, V. Koltun, and P. Krähenbühl, “Tracking objects as points,” vocabulary instance segmentation,” in CVPR, 2019.
in ECCV, 2020. [48] H. K. Cheng, Y. Tai, and C. Tang, “Rethinking space-time networks with
[20] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online improved memory coverage for efficient video object segmentation,”
and realtime tracking,” in ICIP, 2016. in NeurIPS, 2021.
[21] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime [49] B. Cheng, A. Choudhuri, I. Misra, A. Kirillov, R. Girdhar, and A. G.
tracking with a deep association metric,” in ICIP, 2017. Schwing, “Mask2former for video instance segmentation,” CoRR, 2021.
[22] S. Tang, M. Andriluka, B. Andres, and B. Schiele, “Multiple people [50] L. Yang, Y. Fan, and N. Xu, “Video instance segmentation,” CoRR,
tracking by lifted multicut and person re-identification,” in CVPR, 2019.
2017. [51] J. Wu, Q. Liu, Y. Jiang, S. Bai, A. L. Yuille, and X. Bai, “In defense
[23] Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu, “Fairmot: On the of online models for video instance segmentation,” in ECCV, 2022.
fairness of detection and re-identification in multiple object tracking,” [52] J. Yang, M. Gao, Z. Li, S. Gao, F. Wang, and F. Zheng, “Track anything:
IJCV, 2021. Segment anything meets videos,” CoRR, 2023.
[24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, [53] H. K. Cheng and A. G. Schwing, “Xmem: Long-term video object
G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable segmentation with an atkinson-shiffrin memory model,” in ECCV,
visual models from natural language supervision,” in International 2022.
conference on machine learning. PMLR, 2021, pp. 8748–8763. [54] S. W. Oh, J. Lee, N. Xu, and S. J. Kim, “Video object segmentation
[25] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. using space-time memory networks,” in ICCV, 2019.
Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language [55] Q. Wang, L. Zhang, L. Bertinetto, W. Hu, and P. H. Torr, “Fast online
representation learning with noisy text supervision,” in ICML, 2021. object tracking and segmentation: A unifying approach,” in CVPR,
[26] J. Devlin, M.-W. Chang, K. Lee, and K. N. Toutanova, “Bert: Pre- 2019.
training of deep bidirectional transformers for language understanding,” [56] B. Yan, Y. Jiang, P. Sun, D. Wang, Z. Yuan, P. Luo, and H. Lu,
in NAACL, 2019. “Towards grand unification of object tracking,” in ECCV, 2022.
[27] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors [57] P. Voigtlaender, J. Luiten, P. H. S. Torr, and B. Leibe, “Siam R-CNN:
for word representation,” in EMNLP, 2014. visual tracking by re-detection,” in CVPR, 2020.
[58] B. Yan, Y. Jiang, J. Wu, D. Wang, Z. Yuan, P. Luo, and H. Lu,
“Universal instance perception as object discovery and retrieval,” in
CVPR, 2023.
[59] L. Ke, M. Ye, M. Danelljan, Y. Liu, Y.-W. Tai, C.-K. Tang, and F. Yu,
“Segment anything in high quality,” arXiv, 2023.