A serverless GPU is GPU compute you rent by the second, with the provider handling provisioning and scaling, and billing that stops the moment your workload finishes. The core tradeoff against a regular GPU instance comes down to three things:
- Pricing — pay only for active compute time, instead of paying for the whole time the instance is running.
- Scaling — automatic, driven by request volume, instead of manual or scripted.
- Operations — no server to patch or babysit, instead of full control over the box.
That tradeoff makes serverless GPUs a strong fit for spiky, unpredictable workloads and a weak fit for long, continuous ones — the rest of this guide walks through when each side wins.
Serverless GPU vs. GPU Instance at a Glance
| Dimension | Serverless GPU | GPU Instance |
|---|---|---|
| Billing unit | Per-second, only while a request is running | Per-second, but the meter runs the whole time the instance is up |
| Cold start | Seconds to tens of seconds when scaling from zero, depending on container size and provider optimizations | None after boot — the GPU is already warm and idle-ready |
| Scaling | Automatic, driven by request volume or queue depth | Manual, or scripted through your own autoscaling group |
| Best fit | Bursty inference, batch jobs, unpredictable traffic | Training runs, always-on services, latency-sensitive production traffic |
| Container limits | Image size and cold-start time are directly linked — bloated images slow every scale-from-zero event | No comparable constraint; you manage the filesystem yourself |
| Ops overhead | Provider manages the host, scaling, and health checks | You manage the OS, drivers, and any orchestration |
The row that decides most real cases is cold start. If your traffic never truly drops to zero, cold start barely matters and a GPU instance is simpler to reason about. If your traffic is spiky or your service can tolerate cold, occasional requests, serverless usually wins on cost because you stop paying for idle GPU-seconds.
What Cold Start Actually Costs You
Cold start is the time between a request arriving and a GPU replica being ready to serve it. It is the single biggest source of confusion in serverless GPU marketing, because “serverless” implies instant, and naive implementations are not.
Modal’s engineering team published a detailed breakdown of what a naive cold start involves: spinning up a new instance and health-checking it, loading the application and filesystem state, then initializing the program on both the host and the GPU. Done without optimization, that sequence can take “tens of minutes.” With engineering investment in cloud buffers, lazy container filesystems, and CUDA checkpoint/restore, Modal says they cut cold start “from many tens of minutes down to a few seconds or tens of seconds,” citing a 40x improvement — roughly 2,000 seconds down to about 50 seconds for a representative inference server boot.
The practical takeaway: cold start numbers vary wildly by provider and by how much engineering work went into container caching and process checkpointing. When you evaluate a serverless GPU provider, ask for their cold-start number under your actual container image size and GPU type, not a marketing average — a 5GB PyTorch image with custom CUDA kernels will cold-start slower than a 500MB inference-only image on the same platform.
When Serverless GPU Is the Right Call
- Inference with unpredictable or spiky traffic. A chatbot or API that gets hit in bursts, then sits idle, wastes money on an always-on GPU instance. Serverless scales to zero between bursts and back up when traffic returns.
- Batch jobs that run occasionally. Nightly embedding jobs, scheduled image processing, or periodic model evaluation runs don’t need a GPU sitting idle 23 hours a day.
- Early-stage products with unknown load. When you don’t yet know your traffic pattern, per-second billing avoids over-provisioning a fixed instance size you’ll have to resize later.
When a GPU Instance Is the Better Fit
- Model training. Training runs saturate the GPU continuously for hours or days — there’s no idle time for a pay-per-request model to save you money on, and you want direct control over checkpointing and multi-GPU setups.
- Latency-sensitive production traffic with steady volume. If requests arrive continuously, a GPU instance that’s already warm avoids paying the cold-start tax on every request, and gives you a predictable cost floor.
- Workloads needing custom drivers, kernels, or long-lived state. Serverless containers are typically stateless and rebuilt on each cold start; if your workload needs a persistent in-memory cache or non-standard system configuration, a dedicated instance is simpler to reason about.
Decision Framework: Which One Should You Actually Pick?
Answer these three questions in order:
- Is your GPU idle more than it’s busy? If yes, serverless wins on cost. If your utilization is consistently high (say, above 60-70% of the time), a GPU instance’s flat per-second rate while running will usually beat paying serverless overhead on every cold start.
- Can your workload tolerate a cold-start delay on the first request after idle? If a few seconds to a couple of minutes of added latency on an occasional request is acceptable, serverless works. If every request has a strict latency SLA, you need either a GPU instance or a serverless setup with a warm-pool/minimum-replica configuration that keeps at least one worker hot — which erodes some of the cost benefit.
- Do you need full control over the runtime environment? If you need specific driver versions, persistent local state, or non-container-friendly setups, a GPU instance gives you that; serverless abstracts it away in exchange for convenience.
If you answered “serverless” to all three, start there. If you answered “instance” to even one, especially question 3, a GPU instance is the safer default — you can always add serverless endpoints later for the bursty parts of your workload.
Serverless GPU Pricing on Novita AI
Novita AI’s Serverless GPU product bills by the formula Worker Cost = Worker Running Duration (seconds, only while the worker is in the running state) × Worker Unit Price ($/second), and the total endpoint cost is the sum of that across all workers on the endpoint. Unit price depends on the GPU type assigned to the worker, and current per-GPU rates are published on the Serverless console pricing page, since rates are subject to change and vary by GPU type and region.
For workloads that need at least one warm replica so cold start never hits production traffic, Novita’s Dedicated Endpoint product runs the same per-second billing model but on replicas you keep active, with published starting rates around $0.61/hour for entry-tier GPUs. That sits between pure serverless and a full GPU instance: you avoid managing the host, but you also avoid the cold-start tax on every request.
If your workload is training-shaped rather than inference-shaped, Novita’s GPU Instance product gives you full VM control with per-second billing while the instance runs, and templates for common frameworks like PyTorch and Ollama. See the full GPU instance vs. dedicated endpoint comparison for a deeper breakdown of when each fits.
FAQ
Is serverless GPU cheaper than a GPU instance?
It depends entirely on utilization. If your workload is idle most of the time, serverless is cheaper because you stop paying when there’s no request to serve. If your workload keeps a GPU busy most of the day, a GPU instance’s flat per-second rate is usually cheaper, since serverless pricing has to cover the provider’s cost of maintaining scale-up capacity and cold-start engineering.
What causes a serverless GPU cold start, and can I avoid it?
Cold start comes from provisioning a new GPU worker, pulling and unpacking your container image, and initializing your model and CUDA context. You can reduce it by shrinking your container image, using a framework the provider has already optimized for fast loading, or by configuring a minimum number of always-warm replicas — which trades some of the cost savings for lower latency on the first request.
Can I run model training on a serverless GPU?
Serverless GPU platforms are built for short, bursty, request-driven workloads like inference, not continuous multi-hour or multi-day training runs. For training, a GPU instance or a dedicated multi-GPU cluster gives you the sustained, predictable capacity training needs, and generally works out cheaper per GPU-hour for continuous use.
Do serverless GPUs support any container image, or only specific frameworks?
Most serverless GPU platforms, including Novita AI, run standard Docker container images, so you can bring your own model and dependencies. The tradeoff is that larger, more complex images take longer to cold-start, so providers optimized for AI inference often recommend keeping images lean and avoiding unnecessary layers.
Recommended Articles
What Is the Best AI Cloud Platform for Serverless Model Inference?
LLM Dedicated Endpoint on Novita AI: Custom Models, Usage-Based Pricing, and DevOps-Free Scaling
A100 vs H100: Making the Right Choice for Your AI Infrastructure
