Back to Blog
Infrastructure
Jul 28, 2026
6 min read
True Serverless GPU: 3-Second Cold Starts, Hot and Cold Serving
Discover how five compounded optimizations enable 3-second GPU cold starts and true serverless economics for inference workloads.
Inference workloads are spiky. A chatbot sits idle at 3 AM, then 500 users show up for a morning meeting. Training jobs run for hours, then stop. If you provision for peak, your GPUs sit idle 80% of the time. At $2-4 per GPU-hour, that's thousands of dollars a month doing nothing.
Serverless computing was supposed to fix this: spin up capacity when demand hits, spin it down when it drops. But the conventional wisdom is that serverless doesn't work for GPUs, because "cold starts take minutes." Loading a model onto a GPU takes 30-90 seconds. Waiting for a cloud instance takes minutes more. By the time your capacity arrives, the spike is over.
We refused to accept that. Here's how we got GPU cold starts down to 3 seconds, and what that enables.
Pipeline Optimization Comparison
The Problem: GPU Cold Starts Are Slow
Let's trace what happens when you naively spin up a new GPU inference replica:
| Step | Time |
|---|---|
| Cloud provider allocates instance | 60-180 seconds |
| Container image pulled and started | 30-60 seconds |
Python imports (torch, vllm, transformers) | 10-20 seconds |
| Model weights loaded from disk to GPU VRAM | 20-90 seconds |
| CUDA graph capture, kernel compilation | 10-30 seconds |
| Total | 2-6 minutes |
That's your cold start. During those 2-6 minutes, your users are hitting timeouts. The spike passes while capacity is still booting. You end up over-provisioning just to handle the variance — which is exactly what serverless was supposed to eliminate.
Our Stack: Five Optimizations That Compound
We didn't solve this with one silver bullet. We solved it with five optimizations that build on each other:
1. GPU Buffering (Removes Instance Allocation)
We maintain a small buffer of idle, healthy GPUs shared across all customers. When your function needs capacity, it gets scheduled onto an already-running GPU — no cloud API call, no instance spin-up. New instances are provisioned into the buffer asynchronously.
This removes the 60-180 second instance allocation from the hot path entirely.
2. Lazy Container Filesystem (Removes Image Pull)
Container images contain thousands of files you never read — timezones, locales, man pages. We built a FUSE filesystem that loads only the metadata (a few MB) to start a container, then loads the remaining files lazily as they're accessed.
Combined with a content-addressed, multi-tier cache (memory → SSD → network), typical container starts drop from 30 seconds to under 1 second.
3. CPU Memory Snapshotting (Removes Python Startup)
import torch executes tens of thousands of syscalls. Loading transformers, vllm, and your application code takes 10-20 seconds of CPU time before the GPU is even touched.We checkpoint the process state after the first cold start using gVisor's built-in checkpoint/restore. On subsequent starts, we restore directly from the checkpoint — skipping all the imports and initialization. CPU startup drops from 15 seconds to under 1 second.
4. GPU Memory Snapshotting (Removes CUDA Initialization)
The real bottleneck: CUDA graph capture, kernel compilation, and warmup passes. vLLM spends 20-30 seconds compiling CUDA kernels and capturing execution graphs before it can serve a single request.
NVIDIA's recent drivers support device-side checkpoint/restore — the driver checkpoints GPU memory into host memory, which our host-side snapshotting system can capture. On restore, the driver reloads the GPU memory directly.
| Snapshots OFF | Snapshots ON | |
|---|---|---|
| vLLM boot (1B model) | 95 seconds | 14 seconds |
| SGLang boot (1B model) | 84 seconds | 17 seconds |
5. Hot-to-Cold Gradation (Removes the Binary Choice)
Traditional auto-scaling is binary: you're either running (paying full GPU price) or you're not (cold start penalty). We implemented a thermal lifecycle with five states:
| State | GPU Status | Wake Latency | Cost |
|---|---|---|---|
| Hot | Model in VRAM, serving traffic | 0ms | Full GPU |
| Warm | Model in VRAM, evictable | ~100ms | Full GPU |
| Cold | VRAM in host memory | ~2s | Fractional |
| Slept | Pod stopped, VRAM snapshot | ~10s | Near zero |
| Frozen | Full CRIU checkpoint on disk | ~30s | Zero |
The thermal controller moves functions between states based on idle time. A function that served traffic 30 seconds ago stays hot. After 5 minutes of idle, it goes warm (VRAM still on GPU but marked evictable). After 30 minutes, cold (VRAM moved to host RAM, GPU freed). After an hour, frozen (full checkpoint to disk, zero resource usage).
Function Lifecycle State Diagram
When a request arrives for a frozen function, the activator wakes it: restores the checkpoint, reloads VRAM, and the function is serving in ~30 seconds. For a cold function, it's 2 seconds. For a warm function, it's instant.
The Result
A customer running a document processing pipeline with 1B-parameter vision-language models:
- Peak demand: 200 concurrent GPUs during business hours
- Off-peak demand: 5-10 GPUs overnight
- Average GPU utilization: 72% (vs ~30% without serverless)
- Cold start latency: 3 seconds (P50), 14 seconds (P99)
- Monthly GPU cost: 62% lower than static provisioning
The key number isn't the cold start speed — it's the utilization. Fast cold starts mean you don't need to keep GPUs warm "just in case." You can actually run serverless: spin up when demand arrives, spin down when it leaves. The speed is what makes the economics work.
Try It
shell
Your function scales from zero to ten replicas automatically. It costs nothing when idle. Cold starts are measured in seconds, not minutes.
That's serverless GPUs. Not a future promise — it's running in production today.
Read Next
Deep dives into infrastructure and AI.
Tutorial
Aug 4, 20264 minutes
Run OpenClaw with Ollama Serving Qwen3.6 on a Full H100: No Token Limits, No API Bills
Deploy OpenClaw and Ollama as two separate pods on Istidlal, connect them over public URLs, and run Qwen3.6 35B with no per-token costs and no context ceiling.
Read Article
Tutorial
Aug 3, 20262 min
Run your Local Jupyter Notebooks on ANY remote GPU directly from VSCode
Eliminate the tedious setup and constant context switching required to run Jupyter notebooks on remote GPUs directly from VSCode using Istidlal extension
Read Article
Tutorial
Jul 28, 20265 min read
How to Deploy and Run Hermes on a GPU Slice
Deploy Nous Research's 405B-parameter Hermes 3 model efficiently using a 48GB GPU slice and FP8 quantization.
Read Article