Add AI: BYOC CPU Pipeline

This is Tutorial 2 of 3.

Tutorial 1: (start here if not completed)
Tutorial 3:

This tutorial builds a custom AI pipeline using PyTrickle, packages it as a Docker container, and routes jobs through the Gateway-Orchestrator pipeline from Tutorial 1. No GPU is required. Time: ~30 minutes | Cost: zero | Requirements: Tutorial 1 completed, Docker, Python 3.10+

Architecture

HTTP client (test job)
     |
     v
+--------------------------------------+
|  Gateway  (port 8935)                |  <- AI job API, off-chain mode
|  -network offchain                   |
+----------------+---------------------+
                 |  HTTP BYOC job request
                 v
+--------------------------------------+
|  Orchestrator  (port 8936)           |  <- routes job to BYOC container
|  -network offchain                   |
+----------------+---------------------+
                 |  Trickle HTTP
                 v
+--------------------------------------+
|  BYOC Container  (port 8000)         |  <- Python processor (CPU)
|  PyTrickle frame processor           |
|  e.g. Green-tint | grayscale |       |
|       passthrough | whisper-tiny     |
+--------------------------------------+

BYOC (Bring Your Own Container) attaches any Docker container as a compute pipeline on the Livepeer Network. The container speaks the trickle streaming protocol - PyTrickle provides a Python interface. The Gateway and Orchestrator handle all routing and payment logic; the container only needs to receive frames (or other media) and return processed output.

BYOC vs standard pipelines: Standard Livepeer AI pipelines (text-to-image, image-to-image, etc.) run inside the ai-runner runtime using the Pipeline interface. BYOC is different - the container speaks the trickle HTTP protocol directly, bypassing ai-runner. This makes BYOC simpler for custom processors: only PyTrickle and processing logic are needed, not the full ai-runner stack.

Trickle Protocol

The trickle protocol is a simple HTTP-based streaming convention:

The Orchestrator calls the container’s PUT /live/{job_id}/source - the input stream (frames, audio, or arbitrary bytes)
The container processes the data and writes results to GET /live/{job_id}/output - the output stream the Orchestrator pulls
PyTrickle abstracts both sides: implement a FrameProcessor that receives data and returns data

Pattern

async def process(self, frame: bytes) -> bytes:
    # Transform frame and return result
    return transformed_frame

The container runs as an HTTP server. The Orchestrator connects to it at startup and keeps the connection alive for the job duration.

Prerequisites

From Tutorial 1:

./livepeer binary installed and working
Off-chain Orchestrator + Gateway tested

New for this tutorial:

Docker Engine 24+
Python 3.10+ with pip
Optional: pip install openai-whisper for the Whisper-tiny step

Steps

What Happened

A complete custom AI pipeline was built and deployed on the Livepeer Network without a GPU:

test_byoc_job.py
  +-> POST to Gateway:8935 (model_id=green-tint-cpu)
        +-> Gateway routes to Orchestrator:8936 (BYOC HTTP, not gRPC)
              +-> Orchestrator forwards to container:8000 via trickle PUT
                    +-> GreenTintProcessor.process() called with frame bytes
                          +-> Tinted frame returned via trickle GET
                                +-> Orchestrator returns result to Gateway
                                      +-> Gateway returns result to client

Key takeaways:

BYOC containers use the trickle HTTP protocol - not gRPC, not the ai-runner Pipeline interface
Any Docker container that speaks trickle can be a Livepeer pipeline
The Gateway-Orchestrator routing logic is identical for BYOC and standard pipelines
CPU-based AI inference (Whisper-tiny, scikit-learn, etc.) works without any GPU changes

Troubleshooting

Gateway returns 404 or 'model not found'

Check that -byocModelID on the Orchestrator matches X-Model-Id in the test job. They must be identical strings. Confirm the Orchestrator log shows BYOC capability registered: green-tint-cpu.

Orchestrator cannot reach the BYOC container

Verify the container is running and reachable:

Check Container

docker ps | grep byoc-green-tint
curl http://localhost:8000/health

If --network host was not used, check Docker bridge network connectivity to port 8000.

Container exits immediately

Check Logs

docker logs byoc-green-tint

Common causes: PyTrickle import error (check pip install inside container) or Python syntax error in the processor.

Process receives empty bytes

The Orchestrator may send a keepalive ping before the actual payload:

Handle Empty

async def process(self, frame: bytes) -> bytes:
    if not frame:
        return frame  # Return empty on keepalive
    # ... Processing

Whisper-tiny is very slow

On CPU, Whisper-tiny processes approximately 1 second of audio in ~10 seconds. This is expected. For real-time inference, a GPU is needed (Tutorial 3).

Tutorial 3: Go Production

On-chain registration, GPU acceleration, and the public Orchestrator network.

BYOC Pipelines

Full BYOC reference: discovery, capability advertisement, and container requirements.

ai-runner Pipelines

For GPU models needing the full ai-runner stack, use the Pipeline interface instead of BYOC.

Python Gateway SDK

Send jobs programmatically with session management and remote signer payments.

​Architecture

​Trickle Protocol

​Prerequisites

​Steps

​What Happened

​Troubleshooting

​Related Pages

Tutorial 3: Go Production

BYOC Pipelines

ai-runner Pipelines

Python Gateway SDK

Architecture

Trickle Protocol

Prerequisites

Steps

What Happened

Troubleshooting

Related Pages