Configuring AI Models

Before deploying your AI Orchestrator node on the Livepeer AI network, you must choose the AI models you want to serve for AI inference tasks. This guide assists in configuring these models. The following page, Download AI Models, provides instructions for their download. For details on supported pipelines and models, refer to Pipelines.

​

Configuration File Format

Orchestrators specify supported AI models in an aiModels.json file, typically located in the ~/.lpData directory. Below is an example configuration showing currently recommended models and their respective prices.

[
  {
    "pipeline": "text-to-image",
    "model_id": "SG161222/RealVisXL_V4.0_Lightning",
    "price_per_unit": 4768371,
  },
  {
    "pipeline": "upscale",
    "model_id": "stabilityai/stable-diffusion-x4-upscaler",
    "price_per_unit": "0.5e-2USD",
    "warm": true,
    "optimization_flags": {
      "SFAST": true,
      "DEEPCACHE": false
    }
  },
  {
    "pipeline": "audio-to-text",
    "model_id": "openai/whisper-large-v3",
    "price_per_unit": 12882811,
    "pixels_per_unit": 1,
    "currency": "USD",
    "url": "<CONTAINER_URL>:<PORT>",
    "token": "<OPTIONAL_BEARER_TOKEN>",
    "capacity": 1
  }
]

​

Key Configuration Fields

​

Optimization Flags

The Livepeer AI pipelines offer a suite of optimization flags. These are designed to enhance the performance of warm models by either increasing inference speed or reducing VRAM usage. Currently, the following flags are available:

​

Image-to-video Pipeline Optimization

​

Text-to-image Pipeline Optimization

​

External Containers

External containers can be for one model to stack on top of managed model containers, an auto-scaling GPU cluster behind a load balancer or anything in between. Orchestrators can use external containers to extend the models served or fully replace the AI Worker managed model containers using the Docker client Go library to start and stop containers specified at startup of the AI Worker.

External containers can be used by specifying the url, capacity and token fields in the model configuration. The only requirement is that the url specified responds as expected to the AI Worker same as the managed containers would respond (including http error codes). As long as the container management software acts as a pass through to the model container you can use any container management software to implement the custom management of the runner containers including Kubernetes, Podman, Docker Swarm, Nomad, or custom scripts to manage container lifecycles based on request volume

• The url set will be used to confirm a model container is running at startup of the AI Worker using the /health endpoint. Inference requests will be forwarded to the url same as they are to the managed containers after startup.
• The capacity should be set to the maximum amount of requests that can be processed concurrently for the pipeline/model id (default is 1). If auto scaling containers, take care that the startup time is fast if setting warm: true because slow response time will negatively impact your selection by Gateways for future requests.
• The token field is used to secure the model container url from unauthorized access and is strongly suggested to use if the containers are exposed to external networks.

We welcome feedback to improve this feature, so please reach out to us if you have suggestions to enable better experience running external containers.