Back to Blog
Infrastructure
Jul 28, 2026
6 min read
True Serverless GPU: 3-Second Cold Starts, Hot and Cold Serving
Discover how five compounded optimizations enable 3-second GPU cold starts and true serverless economics for inference workloads.
Inference workloads are spiky. A chatbot sits idle at 3 AM, then 500 users show up for a morning meeting. Training jobs run for hours, then stop. If you provision for peak, your GPUs sit idle 80% of the time. At $2-4 per GPU-hour, that's thousands of dollars a month doing nothing.
Serverless computing was supposed to fix this: spin up capacity when demand hits, spin it down when it drops. But the conventional wisdom is that serverless doesn't work for GPUs, because "cold starts take minutes." Loading a model onto a GPU takes 30-90 seconds. Waiting for a cloud instance takes minutes more. By the time your capacity arrives, the spike is over.
We refused to accept that. Here's how we got GPU cold starts down to 3 seconds, and what that enables.
The Problem: GPU Cold Starts Are Slow
Let's trace what happens when you naively spin up a new GPU inference replica:
| Step | Time |
|---|---|
| Cloud provider allocates instance | 60-180 seconds |
| Container image pulled and started | 30-60 seconds |
Python imports (torch, vllm, transformers) | 10-20 seconds |
| Model weights loaded from disk to GPU VRAM | 20-90 seconds |
| CUDA graph capture, kernel compilation | 10-30 seconds |
| Total | 2-6 minutes |
During those 2-6 minutes, your users are hitting timeouts. The spike passes while capacity is still booting. You end up over-provisioning just to handle the variance — which is exactly what serverless was supposed to eliminate.
Figure 1: GPU Cold Start Latency Breakdown: Naive vs Optimized Pipeline
Our Stack: Five Optimizations That Compound
We didn't solve this with one silver bullet. We solved it with five optimizations that build on each other:
1. GPU Buffering (Removes Instance Allocation)
We maintain a small buffer of idle, healthy GPUs shared across all customers. When your function needs capacity, it gets scheduled onto an already-running GPU — no cloud API call, no instance spin-up. New instances are provisioned into the buffer asynchronously. This removes the 60-180 second instance allocation from the hot path entirely.
2. Lazy Container Filesystem (Removes Image Pull)
Container images contain thousands of files you never read — timezones, locales, man pages. We built a FUSE filesystem that loads only the metadata (a few MB) to start a container, then loads the remaining files lazily as they're accessed. Combined with a content-addressed, multi-tier cache (memory → SSD → network), typical container starts drop from 30 seconds to under 1 second.
3. CPU Memory Snapshotting (Removes Python Startup)
import torch executes tens of thousands of syscalls. Loading transformers, vllm, and your application code takes 10-20 seconds of CPU time before the GPU is even touched. We checkpoint the process state after the first cold start using gVisor's built-in checkpoint/restore. On subsequent starts, we restore directly from the checkpoint — skipping all the imports and initialization. CPU startup drops from 15 seconds to under 1 second.4. GPU Memory Snapshotting (Removes CUDA Initialization)
The real bottleneck: CUDA graph capture, kernel compilation, and warmup passes. vLLM spends 20-30 seconds compiling CUDA kernels and capturing CUDA graphs before serving its first token. NVIDIA's recent drivers support device-side checkpoint/restore — the driver checkpoints GPU memory into host memory, which our host-side snapshots capture. On restore, the GPU state is restored via DMA in ~1.8 seconds.
| Engine | Snapshots OFF | Snapshots ON |
|---|---|---|
| vLLM boot (1B model) | 95 seconds | 14 seconds |
| SGLang boot (1B model) | 84 seconds | 17 seconds |
5. Hot-to-Cold Gradation (Removes the Binary Choice)
Traditional auto-scaling is binary: you're either running (paying full GPU price) or you're not (cold start penalty). We implemented a thermal gradient with three states:
| State | GPU Status | Wake Latency | Cost |
|---|---|---|---|
| Hot | Model in VRAM, serving traffic | 0ms | Full GPU |
| Warm | Model in VRAM, evictable | ~100ms | Full GPU |
| Cold | VRAM freed, state on host NVMe | <3s | $0 / Free |
The thermal controller moves functions between states based on idle time. A function that served traffic 30 seconds ago stays hot. After 5 minutes idle, it migrates to cold state, freeing the GPU for other tenants.
Figure 2: Three-State Thermal Gradient: Hot, Warm, Cold Transition Lifecycle
The Result
A customer running a document processing pipeline with 1B-parameter vision-language models:
- Peak demand: 200 concurrent GPUs during business hours
- Off-peak demand: 5-10 GPUs overnight
- Average GPU utilization: 72% (vs ~30% without serverless)
- Cold start latency: 3 seconds (P50), 14 seconds (P99)
- Monthly GPU cost: 62% lower than static provisioning
The key number isn't the cold start speed — it's the utilization. Fast cold starts mean you don't need to keep GPUs warm "just in case." You scale to zero when idle, and scale up instantly when traffic arrives.
Figure 3: Monthly GPU Cost & Utilization Comparison: Static Provisioning vs True Serverless
Read Next
Deep dives into infrastructure and AI.
Tutorial
Aug 4, 20264 min read
Run OpenClaw with Ollama Serving Qwen3.6 on a Full H100: No Token Limits, No API Bills
Deploy OpenClaw and Ollama as two separate pods on Istidlal, connect them over public URLs, and run Qwen3.6 35B with no per-token costs and no context ceiling.
Read Article
Tutorial
Aug 3, 20262 min read
Run your Local Jupyter Notebooks on ANY remote GPU directly from VSCode
Eliminate the tedious setup and constant context switching required to run Jupyter notebooks on remote GPUs directly from VSCode using Istidlal extension
Read Article
Tutorial
Jul 28, 20265 min read
How to Deploy and Run Hermes on a GPU Slice
Deploy Nous Research's 405B-parameter Hermes 3 model efficiently using a 48GB GPU slice and FP8 quantization.
Read Article