Hardware-isolate,

Back to Blog
Tutorial
Jul 28, 2026
5 min read

How to Deploy and Run Hermes on a GPU Slice

Deploy Nous Research's 405B-parameter Hermes 3 model efficiently using a 48GB GPU slice and FP8 quantization.

Lamine Kouissi
Lamine KouissiPlatform Engineer
How to Deploy and Run Hermes on a GPU Slice Banner
Deploying large frontier open-weights models like Nous Research's Hermes 3 (405B) presents a significant infrastructure challenge. Running the unquantized 405B parameter model requires 8× 80GB H100 GPUs connected via NVLink — an infrastructure footprint that costs upwards of $24/hour.
By leveraging Istidlal's hardware-isolated GPU slicing and FP8 quantization, you can deploy and serve Hermes 3 405B on a fractional GPU slice with near-zero latency degradation at a fraction of the cost.

Slicing Architecture for Massive Models

GPU slicing on Istidlal provides hardware-level compute and memory isolation:
  • Dedicated Compute Units: Guaranteed streaming multiprocessors (SMs) allocated strictly to your workload.
  • Isolated VRAM Quotas: Software-defined VRAM guarantees preventing out-of-memory cascades caused by noisy neighbors.
  • Sub-Microsecond Remoting: Our AF_UNIX broker enables high-performance CUDA IPC across local sockets.
Figure 1: Hardware-Isolated GPU Slicing Architecture: Dedicated SMs & Isolated VRAM Quotas

Figure 1: Hardware-Isolated GPU Slicing Architecture: Dedicated SMs & Isolated VRAM Quotas

Deployment Recipe: vLLM & SGLang

Deploying Hermes 3 on Istidlal using our CLI or dashboard:
bash
1istidlal deploy hermes-405b-fp8 \
2  --model nousresearch/Hermes-3-Llama-3.1-405B-FP8 \
3  --slice 48gb \
4  --engine vllm \
5  --tp-size 8 \
6  --kv-cache-dtype fp8 \
7  --max-model-len 32768

Performance & Cost Efficiency

MetricFull 8× H100 ClusterIstidlal 48GB GPU SlicesSavings
Cost per Hour~$24.00/hr~$4.80/hr80% Cheaper
TTFT (P50)310 ms335 msIdentical responsiveness
Decode tok/s (per user)48 tok/s45 tok/sFull interactive speed
With Istidlal's fractional allocation, developers and engineering teams get production-grade access to the world's most capable open-source models without committing to dedicated eight-GPU clusters.
Figure 2: Hermes 3 405B Performance & Cost: Dedicated 8x H100 Cluster vs Istidlal GPU Slices

Figure 2: Hermes 3 405B Performance & Cost Efficiency: Dedicated 8x H100 Cluster vs Istidlal GPU Slices

Read Next

Deep dives into infrastructure and AI.

View all posts
Run OpenClaw with Ollama Serving Qwen3.6 on a Full H100 Banner
Tutorial
Aug 4, 20264 min read

Run OpenClaw with Ollama Serving Qwen3.6 on a Full H100: No Token Limits, No API Bills

Deploy OpenClaw and Ollama as two separate pods on Istidlal, connect them over public URLs, and run Qwen3.6 35B with no per-token costs and no context ceiling.

Read Article
Run your Local Jupyter Notebooks on ANY remote GPU directly from VSCode Banner
Tutorial
Aug 3, 20262 min read

Run your Local Jupyter Notebooks on ANY remote GPU directly from VSCode

Eliminate the tedious setup and constant context switching required to run Jupyter notebooks on remote GPUs directly from VSCode using Istidlal extension

Read Article
True Serverless GPU: 3-Second Cold Starts, Hot and Cold Serving Banner
Infrastructure
Jul 28, 20266 min read

True Serverless GPU: 3-Second Cold Starts, Hot and Cold Serving

Discover how five compounded optimizations enable 3-second GPU cold starts and true serverless economics for inference workloads.

Read Article
Infrastructure Platform

Scale
Without
Limits.

Deploy serverless endpoints, agents, and containerized applications on Istidlal's hardware-isolated fractional GPUs.