Segment-Anything-2

​

Overview

The segment-anything-2 pipeline provides direct access to the Segment Anything 2 pipeline developed by Meta AI Research. In its current version, it supports only image segmentation, enabling it to segment any object in an image. Future versions will also support direct video input, allowing the object to be consistently tracked across all frames of a video in real-time. This advancement will unlock new possibilities for video editing and enhance experiences in mixed reality. The pipeline is powered by the latest diffusion models from HuggingFace’s facebook/sam2-hiera-large.

​

Models

​

Warm Models

The current warm model requested for the segment-anything-2 pipeline is:

• facebook/sam2-hiera-large: The largest model in the Segment Anything 2 model suite, designed for the most accurate image segmentation.

​

On-Demand Models

The following models have been tested and verified for the segment-anything-2 pipeline:

​

Basic Usage Instructions

To generate an image with the segment-anything-2 pipeline, send a POST request to the Gateway’s segment-anything-2 API endpoint:

curl -X POST http://<GATEWAY_IP>/segment-anything-2 \
    -F model_id="facebook/sam2-hiera-large" \
    -F point_coords="[[120,100],[120,50]]" \
    -F point_labels="[1,0]" \
    -F image=@<PATH_TO_IMAGE>/cool-cat.png

In this command:

• <GATEWAY_IP> should be replaced with your AI Gateway’s IP address.
• model_id is the diffusion model for image segmentation.
• The point_coords field holds the coordinates of the points to be segmented.
• The point_labels field holds the labels for the points to be segmented.
• The image field holds the absolute path to the image file to be transformed.

For additional optional parameters, refer to the Livepeer AI API Reference.

After execution, the Orchestrator processes the request and returns the response to the Gateway:

{
  "masks": "[[[2.84, 2.83, ...], [2.92, 2.91, ...], [3.22, 3.56, ...], ...]]",
  "scores": "[0.50, 0.37, ...]",
  "logits": "[[[2.84, 2.66, ...], [3.59, 5.20, ...], [5.07, 5.68, ...], ...]]"
}

​

Orchestrator Configuration

To configure your Orchestrator to serve the segment-anything-2 pipeline, refer to the Orchestrator Configuration guide.

​

System Requirements

The following system requirements are recommended for optimal performance:

• NVIDIA GPU with at least 6GB of VRAM.

​

Recommended Pipeline Pricing

The pricing for the segment-anything-2 pipeline is based on competitor pricing. However, we strongly encourage orchestrators to set their own pricing based on their costs and requirements. Setting a competitive price will help attract more jobs, as Gateways can set their maximum price for a job. The current recommended pricing for this pipeline is 3.22e-11 USD per input pixel (height * width).

​

Pipeline-Specific Image

To serve the segment-anything-2 pipeline, you must use a pipeline specific AI Runner container. Pull the required container from Docker Hub using the following command:

docker pull livepeer/ai-runner:segment-anything-2

​

API Reference