Extending SAM2 Video Tracking to Multiple Objects

[Buckler89]

I'm currently working on extending the SAM2 video tracking (example) to support tracking multiple objects, as currently, only a single object can be tracked.

After diving into the example code (and fixing unrelated issues such as [#656]), I've found that the primary limitation seems to be related to the context parameter passed to the predict method. If I understand correctly, this parameter only contains information about the current label added by the user, and lacks context about other existing labels.

Is there a way to pass the full "context" to the predict method so that all existing labels are accessible during the tracking?

Here is my labeling interface setup:

<View>
   <Labels name="videoLabels" toName="video" allowEmpty="true">
     <Label value="Man" background="blue"/>
     <Label value="Woman" background="red"/>
     <Label value="Other" background="green"/>
   </Labels>
   
   <!-- Please specify FPS carefully, it will be used for all project videos -->
   <Video name="video" value="$video" framerate="25.0"/>
   <VideoRectangle name="box" toName="video" />
</View>

<!--{
 "video": "/static/samples/opossum_snow.mp4"
}-->

[Buckler89]

Closing this issue as it will be resolved once issue #664 is resolved.