Prerequisites
Before running a benchmarking test, make sure you have met all the prerequisites for running an Orchestrator node.Benchmarking Steps
1
Pull the AI Runner Docker Image
First, pull the latest AI runner Docker image from Docker Hub. This image contains the necessary tools for benchmarking.
2
Pull Pipeline-Specific Images (optional)
Next, pull any pipeline-specific images if needed. Check the pipelines documentation for more information. For example, to pull the image for the segment-anything-2 pipeline:
3
Download AI Models
Download the AI models you want to benchmark. For more information see the Download AI Models guide.
4
Execute the Benchmark Test
Run the benchmark test using the following command. This will simulate the pipeline/model combination on your GPU(s) and measure performance.Example command:In this command, the following parameters are used:
<GPU_IDs>
: Specify which GPU(s) to use. For example,'"device=0"'
for GPU 0,'"device=0,1"'
for GPU 0 and GPU 1, or'"device=all"'
for all GPUs.<PIPELINE>
: The pipeline to benchmark (e.g.,text-to-image
).<MODEL_ID>
: The model ID to use for benchmarking (e.g.,stabilityai/sd-turbo
).<RUNS>
: The number of benchmark runs to perform.<NUM_INFERENCE_STEPS>
: The number of inference steps to perform (optional).
5
Analyze the Benchmark Results
After running the benchmark, you should see output similar to the following:This output shows the performance of your GPU(s) and helps you determine the best pipeline/model combination to serve on your GPU(s).