Explore Improved Image and Video Segmentation with SAM 2 for Accurate Context-Aware Results

Welcome to our tutorial on harnessing the capabilities of Segment Anything Model 2 (SAM 2) for enhanced image and video segmentation. SAM 2's unified approach offers improved accuracy, speed, and context awareness, making it a formidable tool in the computer vision field. Whether you are new to computer vision or have considerable experience, this tutorial is designed to familiarize you with SAM 2 and guide you through its essential concepts and advanced techniques.

Introducing SAM 2

SAM 2 handles the intricate challenges of video segmentation that arise from object motion, deformation, occlusion, and varying lighting conditions. It is also adept at processing images with significant efficiency gains over its predecessor, SAM.

Setting Up SAM 2

The SAM 2 model is versatile, with various model sizes to accommodate different computational needs. To get started, clone the repository and install dependencies:

git clone https://github.com/facebookresearch/segment-anything-2.git
cd segment-anything-2
pip install -e .
python setup.py build_ext --inplace

Preparing Data for SAM 2

For effective video segmentation, preprocess your data correctly. First, download the weights for the appropriate model size:

wget -q https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt

Next, configure the environment and load the SAM 2 model for video processing:

import torch
from sam2.build_sam import build_sam2_video_predictor
 
CHECKPOINT = "checkpoints/sam2_hiera_large.pt"
CONFIG = "sam2_hiera_l.yaml"
sam2_model = build_sam2_video_predictor(CONFIG, CHECKPOINT)

Segmenting and Tracking Objects

With SAM 2, you can perform segmentation on individual frames and refine predictions using both positive and negative prompts. Start by initialising the model state and annotating the first object:

import numpy as np
 
points = np.array([[703, 303]], dtype=np.float32) 
labels = np.array([1]) 
frame_idx = 0 
tracker_id = 1
 
_, object_ids, mask_logits = sam2_model.add_new_points(
    inference_state=inference_state, 
    frame_idx=frame_idx, 
    obj_id=tracker_id, 
    points=points, 
    labels=labels
)

Propagating Annotations

To extend your prompts across all video frames, you would use the propagate_in_video method:

import cv2
import supervision as sv
 
with sv.VideoSink(output_path, video_info=video_info) as sink:
    for frame_idx, object_ids, mask_logits in sam2_model.propagate_in_video(inference_state):
        frame = cv2.imread(frames_paths[frame_idx])
        masks = (mask_logits > 0.0).cpu().numpy()
        # Annotate the frame with the segmentation masks...
        sink.write_frame(frame)

Advanced Techniques and Limitations

SAM 2 extends its capabilities to tracking multiple objects and across multiple videos. However, be mindful of its limitations, such as difficulty segmenting objects with fine details or in crowded scenes.

Conclusion

Segment Anything Model 2 elevates the potential of computer vision applications by providing faster, more accurate, and contextually aware segmentation. Its unified model streamlines image and video processing tasks across a myriad of uses in various industries.

For further insight into SAM 2 and its implementation, the official Roboflow blog offers a wealth of knowledge and updates:

Read the detailed SAM 2 guide by Piotr Skalski

Piotr Skalski, an expert in computer vision, shares his wealth of knowledge, ensuring that readers at all skill levels can appreciate the advancements SAM 2 brings.

Thank you for spending time with this tutorial. We hope it empowers you to achieve precise and effective segmentation outcomes.