LLM

​

Overview

The llm pipeline provides an OpenAI-compatible interface for text generation, designed to integrate seamlessly into media workflows.

​

Models

The llm pipeline supports any Hugging Face-compatible LLM model. Since models evolve quickly, the set of warm (preloaded) models on Orchestrators changes regularly.

To see which models are currently available, check the Network Capabilities dashboard.
At the time of writing, the most commonly available model is meta-llama/Meta-Llama-3.1-8B-Instruct.

​

Basic Usage Instructions

To generate text with the llm pipeline, send a POST request to the Gateway’s llm API endpoint:

curl -X POST "https://<GATEWAY_IP>/llm" \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [
      { "role": "user", "content": "Tell a robot story." }
    ]
  }'

In this command:

• <GATEWAY_IP> should be replaced with your AI Gateway’s IP address.
• <TOKEN> should be replaced with your API token.
• model is the LLM model to use for generation.
• messages is the conversation or prompt input for the model.

For additional optional parameters such as temperature, max_tokens, or stream, refer to the Livepeer AI API Reference.

After execution, the Orchestrator processes the request and returns the response to the Gateway:

{
  "id": "chatcmpl-abc123",
  "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Once upon a time, in a gleaming city of circuits..."
      }
    }
  ]
}

By default, responses are returned as a single JSON object. To stream output token-by-token using Server-Sent Events (SSE), set "stream": true in the request body.

​

Orchestrator Configuration

To configure your Orchestrator to serve the llm pipeline, refer to the Orchestrator Configuration guide.

​

Tuning Environment Variables

The llm pipeline supports several environment variables that can be adjusted to optimize performance based on your hardware and workload. These are particularly helpful for managing memory usage and parallelism when running large models.

​

System Requirements

The following system requirements are recommended for optimal performance:

• NVIDIA GPU with at least 16GB of VRAM.

​

Recommended Pipeline Pricing

The /llm pipeline is currently priced based on the maximum output tokens specified in the request — not actual usage — due to current payment system limitations. We’re actively working to support usage-based pricing to better align with industry standards.

The LLM pricing landscape is highly competitive and rapidly evolving. Orchestrators should set prices based on their infrastructure costs and market positioning. As a reference, inference on llama-3-8b-instruct is currently around 0.08 USD per 1 million output tokens.

​

API Reference