AI and cloud services are powerful but resource-intensive. Without proper controls, a single user or process can overload systems, disrupt service, or create unfair access. Rate limiting is essential for keeping AI models and APIs reliable, secure, and available to everyone.
In this article, we’ll explain what rate limiting is, how to understand its key concepts, and how different applications use different rate limiting strategies.
Rate Limiting Fundamentals
Rate limiting is a technique that restricts the number of requests a client can make within a certain period, preventing resource exhaustion and ensuring service availability and performance.
- Protect servers from being overloaded
- Prevent abuse or spam
- Ensure fair access for all users
- Improve security by stopping attacks (like DDoS)
Different Types of Rate Limiting
- User-based: Limits applied to individual users or IP addresses.
- Server-based: Restrictions imposed on each server instance or node.
- Geographic: Traffic limits based on geographic regions.
- Concurrency: Restricts the number of simultaneous requests.
Main Rate Limiting Algorithms
| Algorithm | How It Works | Pros | Cons |
|---|---|---|---|
| Token Bucket | Tokens are added to a bucket at a fixed rate. Each request takes a token. | Handles sudden traffic bursts, uses little memory. | Tokens might clash in high traffic. |
| Leaky Bucket | Requests enter a bucket and leave at a steady rate. | Smooths out traffic, easy to set up. | Sudden bursts may get dropped. |
| Fixed Window | Counts requests in set time blocks (like every minute). | Very simple to build. | Can be unfair at time edges. |
| Sliding Window | Remembers recent request times in a moving window. | Smooth and precise control. | More complex and uses more memory. |
Rate LImiting vs Throttling vs Traffic Shaping
| Concept | What It Does | How It Works | Typical Use Case |
|---|---|---|---|
| Rate Limiting | Sets a hard cap on actions in a time frame | Blocks or rejects excess requests | Preventing API abuse (e.g., max 100 requests/min) |
| Throttling | Slows down requests after a limit | Delays or spaces out extra requests | Smoothing out traffic without blocking (e.g., slow down after 100 requests) |
| Traffic Shaping | Smooths and controls overall traffic flow | Queues, schedules, or paces requests | Managing network bandwidth or API usage for fairness and stability |
Rate limiting best practices for API, Web and Cloud services
API Layer
- Who is it for?
Primarily for developers and third-party applications integrating with your system. - How should rate limiting be set?
- Granularity: Different limits for different developers, endpoints, or business scenarios.
- Transparency: Always inform developers about their current quota usage, remaining requests, and reset time, so they can handle limits gracefully.
- Customizability: Allow flexible adjustments, such as increasing limits for paying customers or special partners.
Web Applications
- Who is it for?
Directly for end-users interacting with your website or system. - How should rate limiting be set?
- Protect key operations: Apply strict limits to sensitive actions like login, registration, or posting to prevent abuse.
- User differentiation: Set different thresholds for free, paid, guest, or member users to ensure fairness and differentiated service.
- Resource control: Limit access to static or valuable resources to prevent scraping or excessive bandwidth consumption.
Cloud Services
- Who is it for?
For a large number of tenants—enterprises, developers, and teams—across a wide range of use cases. - How should rate limiting be set?
- Automated elasticity: Dynamically adjust limits based on real-time traffic and backend resource availability.
- Multi-layer protection: Apply limits globally, per tenant, and per API to prevent a single tenant from overwhelming the platform.
- Handle high concurrency: Smooth out traffic spikes to maintain stability during traffic surges.
- Billing integration: Align rate limits with usage plans and billing models.
Rate Limiting Metrics
Rate limiting metrics are the specific numbers used to define how much activity is allowed in a certain period. Common examples include:
- Requests per minute (RPM): How many requests a user or system can make each minute.
- Requests per second (RPS): How many requests are allowed per second.
- Images per minute (IPM): How many images can be generated or processed per minute.
- Concurrent requests: How many requests can happen at the same time.
- TPM (tokens per model per minute) means the number of tokens that a single AI model can process in one minute.
These metrics set the actual limits for each user, IP, or system.
Why Rate LImiting Matters for AI？
Rate limiting solves these problems by:
- Preventing overload: AI models and cloud APIs can be expensive to run and scale. Rate limiting ensures no single user or project can consume too much, which helps keep services stable and responsive for everyone.
- Ensuring fair access: Many users—often from different teams or even different companies—rely on the same resources. Rate limiting helps guarantee that everyone gets a fair share, no matter how big or small.
- Protecting against abuse: In the cloud, automated scripts or bad actors might try to flood your AI models or APIs with requests. With proper limits in place, you can stop these attacks before they cause real harm.
- Supporting business growth: By introducing tiered limits, platforms can serve both hobbyists and enterprises effectively—offering more capacity to those who need it, while still maintaining stability for all.
In short, smart rate limiting is essential for keeping AI and cloud services reliable, secure, and scalable. Modern platforms need to go beyond basic limits, offering dynamic, transparent, and flexible controls that grow with user needs.
Novita AI: Reliable, Developer-Friendly Rate Limiting
To ensure both stability and a great user experience, advanced API and AI service providers must go beyond basic rate limiting, offering multi-tiered, dynamic, and developer-friendly solutions.
By leveraging comprehensive monitoring, transparent usage feedback, and tiered access, Novita AI ensures that both individual developers and large-scale enterprises enjoy fair, reliable, and predictable access to powerful AI models.
LLM Rate Limiting
To better serve users with higher demands, Novita AI provides a tiered service structure. Setting up tiers helps balance fair access, system security, and business sustainability, while providing a clear path for users to grow with the platform.
| Tier | Criteria (Monthly Top-ups in Any of Last 3 Months) |
|---|---|
| T1 | ≤ $50 |
| T2 | > $50 & ≤ $500 |
| T3 | > $500 & ≤ $3,000 |
| T4 | > $3,000 & ≤ $10,000 |
| T5 | > $10,000 |
You can check details of each LLM Model in Novita AI Docs!
| Model | T1 RPM | T2 RPM | T3 RPM | T4 RPM | T5 RPM | TPM (All Tiers) |
|---|---|---|---|---|---|---|
| deepseek/deepseek-v3-0324 | 10 | 100 | 1,000 | 3,000 | 6,000 | 50,000,000 |
| qwen/qwen3-235b-a22b-thinking-2507 | 300 | 300 | 300 | 300 | 300 | 50,000,000 |
| moonshotai/kimi-k2-instruct | 10 | 100 | 300 | 300 | 300 | 50,000,000 |
| deepseek/deepseek-r1-0528 | 10 | 100 | 1,000 | 3,000 | 6,000 | 50,000,000 |
| qwen/qwen3-30b-a3b-fp8 | 20 | 100 | 1,000 | 3,000 | 6,000 | 50,000,000 |
Image & Video Rate Limiting
- IPM (Images Per Minute): Number of images a model can generate per minute.
- RPM (Requests Per Minute): Number of API requests a video model can handle per minute.
Default Image Model Rate Limits (IPM)
| Resource/Service | Model API | Default IPM |
|---|---|---|
| Text to Image | txt2img_v3 | 20 |
| Image to Image | img2img_v3 | 10 |
| Remove Background | remove_background | 10 |
| Replace Background | replace_background | 10 |
| Remove Text | remove_text | 10 |
| Inpainting | inpainting | 10 |
| Cleanup | cleanup | 10 |
| Merge Face | merge_face | 10 |
| FLUX.1 Text to Image | flux-1-schnell | 10 |
| Upscale | upscale_v3 | 20 |
Default Video Model Rate Limits (RPM)
| Resource/Service | Model API | Default RPM |
|---|---|---|
| Video Merge Face | video_merge_face | 10 |
| Text to Video | txt2video | 2 |
| Image to Video | img2video | 2 |
| Wan 2.1 Text to Video | wan_txt_to_video | 20 |
| Wan 2.1 Image to Video | wan i2v | 20 |
| Hunyuan Video Fast | hunyuan_video_fast | 20 |
| KLING V1.6 Image2Vid | Kling i2v | 20 |
| KLING V1.6 Text2Vid | Kling t2v | 20 |
| Minimax Video-01 | Minimax | 20 |
What are the Consequences of Exceeding Rate Limits?
1. Throttle Requests on the Client Side
- Control the speed of your application’s requests.
- Prevent sending too many requests in a short time.
2. Use Exponential Backoff for Retries
- When you get a rate-limit error (like HTTP 429),
wait longer after each retry attempt. - This reduces the load on the service and increases your success chances.
3. Monitor Your API Usage
- Track request counts, frequency, and error responses.
- Log this data to understand your usage patterns and adjust proactively.
What to Do If You Hit a Rate Limit
If you receive an HTTP 429 (“Too Many Requests”) response:
Retry Later
- Wait a short time before trying again.
Optimize Your Requests
- Reduce how often you make requests.
- Batch or combine calls where possible.
Request a Higher Rate Limit
- If you need more capacity, contact us through Discord or book a call with our sales team.
Smart rate limiting protects AI and cloud services from overload, abuse, and unfair use. Advanced solutions—like those from Novita AI—go further, offering dynamic, transparent, and developer-friendly controls to support both growth and stability.
Frequently Asked Questions
It prevents overload, ensures fair access, stops abuse, and keeps services stable for all users.
Rate limiting sets hard caps, throttling slows down excess requests, and traffic shaping smooths out overall traffic flow.
Novita AI uses tiered and transparent rate limits, with real-time feedback and flexible quotas for different user needs.
Novita AI is the All-in-one cloud platform that empowers your AI ambitions. Integrated APIs, serverless, GPU Instance — the cost-effective tools you need. Eliminate infrastructure, start free, and make your AI vision a reality.
Recommend Reading
- How many H100 GPUs are needed to Fine-tune DeepSeek R1?
- Gemma 3 27B vs Llama 3.3 70B: Which Model for Which Task?
- DeepSeek R1 7B vs 8B: The Smarter Choice for Lightweight Deployment
Discover more from Novita
Subscribe to get the latest posts sent to your email.
