} ; }; export const CustomDivider = ({color = "var(--lp-color-border-default)", middleText = "", spacing = "default", style = {}, className = "", ...rest}) => { const spacingPresets = { default: { margin: "24px 0" }, overlap: { margin: "-1rem 0 -1rem 0" }, tight: { margin: "0 0 -1rem 0" }, section: { margin: "0 0 -2rem 0" }, sectionOverlap: { margin: "-1rem 0 -2rem 0" }, deepOverlap: { margin: "-1rem 0 -1.5rem 0" } }; const spacingStyle = spacingPresets[spacing] || spacingPresets.default; return

{middleText && <> {middleText} }

; }; Session limits and GPU management apply primarily to Orchestrators running compute locally.
A pure routing Gateway (off-chain AI, no local GPU) is limited mainly by network throughput and horizontal instance count.
Jump to [Horizontal Scaling](#horizontal-scaling) if that describes the setup. A Gateway routes jobs to Orchestrators instead of processing them locally, but it still has resource limits: session capacity, network throughput, and (for Dual Gateways) the GPU resources shared between video and AI workloads.
how to measure those limits, decide when they have been reached, and scale in the right direction. ## Scaling Signals Before adjusting anything, confirm there is actually a capacity problem. These are the reliable indicators. The Prometheus metric `livepeer_current_sessions_total` approaching `livepeer_max_sessions_total` means the Gateway is at capacity and will start rejecting new sessions. This is the clearest signal. ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} curl http://localhost:8935/metrics | grep livepeer_current_sessions ``` For Dual Gateways, the `/hardware/stats` endpoint reports GPU utilisation and memory: ```bash icon="terminal" theme={"theme":{"light":"github-light","dark":"dark-plus"}} curl http://localhost:8935/hardware/stats ``` If VRAM usage is consistently above 85%, the Gateway is at risk of OOM failures on new model loads (AI) or performance degradation on concurrent transcoding segments (video). Rising transcoding latency or AI inference latency under load, combined with high Orchestrator swap rates, suggests the Orchestrators being routed to are also under pressure. This is an Orchestrator-side scaling problem. See for expanding the Orchestrator pool. Errors with `OrchestratorCapped` in the Gateway log indicate that a downstream Orchestrator has hit its own session limit and rejected the job. Either expand the Orchestrator pool or negotiate higher capacity with preferred Orchestrators. ### Session Limits Session limits define how many concurrent jobs the Gateway (or its local Orchestrator, for dual setups) accepts. The default is 10. ```bash icon="terminal" -maxSessions theme={"theme":{"light":"github-light","dark":"dark-plus"}} -maxSessions 20 ``` Setting this too high causes performance degradation or OOM failures. Setting it too low wastes available capacity. The right value comes from benchmarking. ### Benchmarking `livepeer_bench` tests transcoding throughput for a range of concurrent session counts. Run this on the machine that will perform local transcoding before setting `-maxSessions` in production. ```bash icon="terminal" livepeer_bench theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Download the standard transcoding options file wget https://raw.githubusercontent.com/livepeer/go-livepeer/master/cmd/livepeer_bench/transcodingOptions.json # Run the benchmark across 1 to 20 concurrent sessions for i in {1..20}; do ./livepeer_bench \ -in bbb/source.m3u8 \ -transcodingOptions transcodingOptions.json \ -nvidia 0 \ -concurrentSessions $i |& grep "Duration Ratio" >> bench.log done cat bench.log ``` Read the output: find the last line where the real-time duration ratio is at or below 0.8. That concurrent session count is the hardware limit. Leave a 20% buffer for upload and download overhead. ```text icon="terminal" Output Example theme={"theme":{"light":"github-light","dark":"dark-plus"}} Concurrent sessions 12: Duration Ratio 0.79 <- last line at or below 0.8 Concurrent sessions 13: Duration Ratio 0.88 Concurrent sessions 14: Duration Ratio 0.97 ``` In this example, set `-maxSessions 12`. For machines with multiple identical GPUs, benchmark one GPU and multiply. For machines with different GPU models, benchmark each separately and use the lower value as the bottleneck. Bandwidth can be a tighter limit than compute. Each 1080p stream requires approximately 4-8 Mbps inbound and 12-24 Mbps outbound (for a 4-rung ABR ladder). If bandwidth is the bottleneck, it sets a lower ceiling than the hardware benchmark. ### GPU Management For Dual Gateways, video transcoding and AI inference compete for the same GPU. ```bash icon="terminal" GPU Stats theme={"theme":{"light":"github-light","dark":"dark-plus"}} # GPU model, VRAM, and system specs curl http://localhost:8935/hardware/info # Live GPU utilisation, memory, temperature curl http://localhost:8935/hardware/stats ``` VRAM allocation between video transcoding and AI inference is managed by the Orchestrator, not the Gateway. Video transcoding (NVENC/NVDEC) uses dedicated hardware encoder/decoder blocks on NVIDIA GPUs, which do not compete directly with CUDA cores used for AI inference. However, VRAM is shared, and large AI models can exhaust available memory. As a rule of thumb: * Reserve at least 8 GB VRAM per concurrent AI inference model * Keep video transcoding sessions below the benchmarked hardware limit even when running AI jobs simultaneously * Monitor temperature: sustained GPU temperature above 85 degrees C indicates thermal throttling ### Split or Share? If a Dual Gateway runs video + AI on one node, the central decision is when to split those workloads onto separate nodes. Factor Stay on single node Split onto separate nodes Load Low to medium, predictable High, variable, or growing SLA requirements Best-effort Committed latency or uptime GPU VRAM Model fits with transcoding headroom Models exhaust VRAM under load Operational complexity Prefer simplicity Can manage two nodes Cost Minimise hardware spend Willing to pay for isolation The practical trigger for splitting is usually VRAM contention (AI models OOM when video load spikes) or SLA pressure (customers require guaranteed latency for one workload that the other interferes with). Once split, run two separate go-livepeer processes - one configured as a Video Gateway, one as an AI Gateway - each with its own Orchestrator list: ```bash icon="terminal" Split Settings theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Video Gateway (on-chain, port 8935, RTMP 1935) livepeer -gateway \ -network arbitrum-one-mainnet \ -rtmpAddr 0.0.0.0:1935 \ -httpAddr 0.0.0.0:8935 \ -orchAddr # AI Gateway (off-chain, port 8935 on separate host) livepeer -gateway \ -httpAddr 0.0.0.0:8935 \ -orchAddr \ -httpIngest ``` The middleware or load balancer routes incoming requests to the correct Gateway based on job type. ### Horizontal Scaling ```mermaid theme={"theme":{"light":"github-light","dark":"dark-plus"}} %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#18794E', 'primaryTextColor': '#fff', 'primaryBorderColor': '#3CB540', 'lineColor': '#3CB540', 'mainBkg': '#18794E', 'nodeBorder': '#3CB540', 'clusterBkg': 'transparent', 'clusterBorder': '#3CB540', 'titleColor': '#3CB540', 'edgeLabelBackground': 'transparent', 'textColor': '#3CB540', 'nodeTextColor': '#fff'}}}%% flowchart LR subgraph Scale["Scaling path"] direction LR S1["1. Vertical\nBigger GPU"] --> S2["2. Split\nSeparate video + AI nodes"] --> S3["3. Horizontal\nMultiple instances per type"] end ``` **Off-chain AI Gateways** scale horizontally without special configuration. They are stateless: no ETH key, no session state. Add instances behind a standard HTTP load balancer and use the `/health` endpoint for health checks. ```yaml icon="code" /health yaml theme={"theme":{"light":"github-light","dark":"dark-plus"}} # Example: three AI Gateway instances behind nginx upstream ai_gateways { server 10.0.0.1:8935; server 10.0.0.2:8935; server 10.0.0.3:8935; } server { location / { proxy_pass http://ai_gateways; } } ``` **Video Gateways** require sticky sessions for streaming continuity. A live RTMP stream cannot be moved between Gateway instances mid-session. Configure the load balancer to use IP-hash or cookie-based session affinity for RTMP connections. HTTP API requests (non-streaming) can be round-robined normally. **Dual Gateways:** Split into separate video and AI instances before scaling horizontally. Trying to horizontally scale a Dual node adds complexity without solving the resource contention problem. ## Capacity Planning Per GPU, the practical limit for 1080p to 720p/480p/360p transcoding is approximately 8-12 concurrent sessions on a modern NVIDIA T4, and 15-25 on an RTX 3080 or equivalent. Run `livepeer_bench` on the specific hardware to get a precise number. Multiply by the number of GPUs in the Orchestrator pool to get total network capacity. The Gateway can route to all of them. For deposit sizing: video transcoding payments are per-pixel. Estimate the expected pixel throughput (resolution x frame rate x concurrent sessions x hours per day) and size the Arbitrum deposit to cover at least 24 hours of expected traffic. AI inference capacity depends entirely on the model and the Orchestrators in the pool. FLUX.1-dev requires approximately 12-16 GB VRAM per concurrent inference; smaller SD 1.5 models can fit in 6 GB. For off-chain AI Gateways, the bottleneck is the number of AI-capable Orchestrators available. Expand the `-orchAddr` list or use a discovery URL that returns a larger pool. Set a monitoring alert at 70% of observed capacity. This gives time to provision additional Orchestrators before hitting the ceiling. Metric Warning Critical Session utilisation 70% of `-maxSessions` 90% of `-maxSessions` GPU memory 75% VRAM used 90% VRAM used GPU temperature 80 degrees C 87 degrees C Orchestrator swap rate 2x baseline 5x baseline ETH deposit balance 20% remaining 5% remaining ## Related Pages Expand the Orchestrator pool to increase available capacity. Route requests across scaled instances with middleware. Tested GPU models and session limit data. Prometheus metrics and dashboards for scaling signals.