} ; }; export const CustomDivider = ({color = "var(--lp-color-border-default)", middleText = "", spacing = "default", style = {}, className = "", ...rest}) => { const spacingPresets = { default: { margin: "24px 0" }, overlap: { margin: "-1rem 0 -1rem 0" }, tight: { margin: "0 0 -1rem 0" }, section: { margin: "0 0 -2rem 0" }, sectionOverlap: { margin: "-1rem 0 -2rem 0" }, deepOverlap: { margin: "-1rem 0 -1.5rem 0" } }; const spacingStyle = spacingPresets[spacing] || spacingPresets.default; return

{middleText && <> {middleText} }

; }; Warm the model Gateways request most often on the pipeline with the highest demand. Check tools.Livepeer.cloud weekly – demand shifts as new models are listed and Gateways update their routing preferences. *** AI model management covers the operational decisions made after models are downloaded: which models to keep warm in VRAM, how to allocate VRAM across multiple pipelines, when to rotate warm models based on demand changes, and which optimisation flags to apply for throughput gains. Model sourcing and downloading is covered separately in . ## Warm vs cold strategy **Warm** means the model weights are loaded into GPU VRAM at container startup. Job requests are served immediately with no loading latency. **Cold** keeps the container available while leaving the weights out of VRAM. The first request triggers a model load – typically 10 to 60 seconds depending on model size and NVMe storage speed – before inference begins. ### Impact on job routing Gateways track first-response latency per Orchestrator. Nodes with fast first responses win more jobs. For latency-sensitive pipelines – particularly `text-to-image` and `image-to-image` – cold loading creates a competitive disadvantage on the first request of each session. The practical rule: warm your primary revenue pipeline. Keep secondary pipelines cold until VRAM capacity allows warming them. ### Beta constraint: one warm model per GPU During the Beta phase, only one warm model per GPU is supported. Setting `"warm": true` on more entries than you have GPUs causes the AI worker to log a conflict at startup and skip the excess entries. Check logs on startup for: ```bash icon="terminal" title="Check warm model startup logs" theme={"theme":{"light":"github-light","dark":"dark-plus"}} docker logs 2>&1 | grep -i "warm\|conflict\|error" ``` The `Error loading warm model` message indicates a warm model conflict. Reduce `"warm": true` entries to match your GPU count. ### VRAM allocation Pipeline Warm VRAM Notes `text-to-image` (SDXL-Lightning) \~12–18 GB Lightning models lean toward 12 GB end `image-to-image` (SDXL) \~12–18 GB Same architecture as text-to-image `image-to-video` (SVD) \~24 GB+ Requires 24 GB minimum; 32 GB preferred for longer clips `audio-to-text` (Whisper large-v3) \~3 GB VRAM-efficient; pairs well with diffusion models on 24 GB `image-to-text` (BLIP) \~1–2 GB Runs warm on 4 GB GPUs `llm` (Llama 3.1 8B q4) \~8 GB Via Ollama runner; suited for 8–12 GB GPUs `upscale` (SD x4) \~8 GB Smaller than generation models; pairs with Whisper on 16 GB `segment-anything-2` (large) \~12–24 GB Variant-dependent; check model card for exact requirement A 24 GB GPU holds one large diffusion model warm, with a small pipeline (Whisper or BLIP) warm on the same card when using a multi-GPU system. Keep multiple large diffusion models off a single 24 GB GPU – the Beta constraint blocks it, and VRAM is insufficient anyway. ## Model rotation by demand Demand on the Livepeer AI network shifts over time. A model leading one week often falls back the next as new models are listed or Gateway preferences change. Warm model selection should track demand and revenue opportunity. ### Checking current demand Visit [tools.Livepeer.cloud/ai/network-capabilities](https://tools.livepeer.cloud/ai/network-capabilities) weekly. Filter by pipeline to see which models active Gateways are requesting. Models with the most Gateway registrations are receiving the most routing traffic. The [Livepeer Explorer AI leaderboard](https://explorer.livepeer.org) shows per-Orchestrator earnings data, which reveals which price tiers and pipelines are earning the most jobs. ### Rotating the warm model To swap which model is warm, update `aiModels.json` and restart the AI worker: ```json icon="code" title="aiModels.json — rotate warm model" theme={"theme":{"light":"github-light","dark":"dark-plus"}} [ { "pipeline": "text-to-image", "model_id": "SG161222/RealVisXL_V4.0_Lightning", "price_per_unit": 4768371, "warm": true }, { "pipeline": "text-to-image", "model_id": "ByteDance/SDXL-Lightning", "price_per_unit": 4768371, "warm": false } ] ``` The model with `"warm": true` loads at startup. The cold entry is available but will incur first-request latency. After restarting the AI worker container, verify the new warm model appears with **Warm** status at [tools.Livepeer.cloud/ai/network-capabilities](https://tools.livepeer.cloud/ai/network-capabilities). Restart the AI worker container and leave the full go-livepeer process running to minimise downtime: ```bash icon="terminal" title="Restart the AI worker" theme={"theme":{"light":"github-light","dark":"dark-plus"}} docker restart ``` ## Optimisation flags Optimisation flags apply to **warm diffusion models only** – `text-to-image`, `image-to-image`, and `upscale` entries with `"warm": true`. They have no effect on cold models or on non-diffusion pipelines. Both flags are experimental. Apply one at a time and verify output quality before serving jobs. **SFAST (Stable Fast)** compiles the diffusion model's compute graph on first run. Subsequent inferences skip redundant operations, delivering up to 25% faster inference with no measurable quality reduction. The first inference after startup is slower than normal – compilation overhead applies once. Subsequent requests are faster. ```json icon="code" title="Enabling SFAST" theme={"theme":{"light":"github-light","dark":"dark-plus"}} { "pipeline": "text-to-image", "model_id": "SG161222/RealVisXL_V4.0_Lightning", "price_per_unit": 4768371, "warm": true, "optimization_flags": { "SFAST": true } } ``` Best for: operators processing frequent repeated requests on the same warm model, where the compilation overhead amortises across many jobs. **DEEPCACHE** caches intermediate diffusion steps to reduce redundant recomputation across inference calls. Delivers up to 50% faster inference with minor quality reduction (slight reduction in fine detail at high step counts). Quality degradation is more visible at low step counts. Reserve DEEPCACHE for standard-step models. Lightning and Turbo variants are already step-optimised for 1 to 4 inference steps, and DEEPCACHE degrades their output without adding speed. ```json icon="code" title="Enabling DEEPCACHE" theme={"theme":{"light":"github-light","dark":"dark-plus"}} { "pipeline": "text-to-image", "model_id": "stabilityai/stable-diffusion-xl-base-1.0", "price_per_unit": 4768371, "warm": true, "optimization_flags": { "DEEPCACHE": true } } ``` Best for: high-quality base SDXL models at normal step counts (20+), where the 50% speed gain justifies the minor quality trade-off. **SFAST and DEEPCACHE cannot be combined.** Set only one, or neither, in `optimization_flags`. Reserve DEEPCACHE for standard-step models. Lightning and Turbo variants (for example `ByteDance/SDXL-Lightning` and `SG161222/RealVisXL_V4.0_Lightning`) operate at 1 to 4 inference steps, so DEEPCACHE degrades output quality without adding speed. ## Monitoring model loading Verify model state after startup by checking the AI Runner container logs: ```bash icon="terminal" title="Check model loading logs" theme={"theme":{"light":"github-light","dark":"dark-plus"}} docker logs 2>&1 | grep -E "warm|loaded|error" | tail -30 ``` Then confirm via the network registry. Your Orchestrator should appear under the correct pipeline with **Warm** status: [tools.Livepeer.cloud/ai/network-capabilities](https://tools.livepeer.cloud/ai/network-capabilities) A warm model missing from the registry after 5 minutes usually indicates one of these causes: * `Error loading warm model` – warm model conflict (too many `"warm": true` entries for available GPUs) * Container restart loops – check `docker ps` for restart counts * Model download still in progress – warm models must finish downloading before the container loads them ## Related pages Download mechanics, storage layout, HuggingFace sourcing, and gated model access. Full aiModels.json reference and pipeline architecture. VRAM requirements and demand context for all supported pipelines. VRAM budgeting and the capacity field for AI pipeline concurrency.