LLM API Streaming Error: Causes and Fixes

An error: llm api error: an error occurred during streaming message almost always comes from one of six sources: a mid-stream provider failure sent as an SSE event after the connection already returned 200, a client- or gateway-side read timeout, a 429 rate limit hit partway through generation, a malformed or unexpected chunk that breaks your SSE parser, a dropped network connection, or an auth/quota problem that only surfaces once the stream opens. Check them in that order — most streaming failures fall into the first three.

Match what you’re seeing to a cause before you read further:

What you observe	Go to
Stream starts, some tokens arrive, then it cuts off with no HTTP status change	Cause 1
Stream stalls for 10s+ with no new tokens, then your client raises a timeout	Cause 2
Error happens most often during traffic spikes or right after a burst of requests	Cause 3
Exception mentions KeyError, JSONDecodeError, or a parsing failure, not a network term	Cause 4
Error is generic (APIConnectionError, ECONNRESET) and happens inconsistently, including on stable networks	Cause 5
Error happens on the very first request after rotating a key, hitting a spend cap, or switching environments	Cause 6

The full diagnostic checklist below repeats this with the likely cause spelled out per row.

Key Takeaways

• A streaming error can arrive as an SSE error event after the HTTP status already returned 200, so status-code-only retry logic misses it.
• Anthropic’s Messages API reports mid-stream overloads as event: error with "type": "overloaded_error" inside the stream body, not as a fresh HTTP 529.
• OpenRouter documents the same pattern across providers: once the first token ships, a failure arrives as a chat.completion.chunk with a top-level error field and finish_reason: "error".
• Client libraries that assume every streamed chunk is success-shaped will crash with a misleading error (an APIConnectionError instead of the real cause) when a provider sends a structured error payload mid-stream.
• Most fixes are the same regardless of provider: parse the stream body for error events, set a token-level stall timeout separate from the connection timeout, and use exponential backoff keyed to the error type, not just the HTTP status.

Diagnostic Checklist: Find Your Root Cause First

Run through this before changing any code. Each row maps a symptom you can observe to the section that fixes it.

What you observe	Likely cause	Go to
Stream starts, some tokens arrive, then it cuts off with no HTTP status change	Mid-stream provider error (overload, content filter, provider crash)	Cause 1
Stream stalls for 10s+ with no new tokens, then your client raises a timeout	Read timeout or idle stall	Cause 2
Error happens most often during traffic spikes or right after a burst of requests	Rate limit hit mid-stream	Cause 3
Exception mentions KeyError, JSONDecodeError, or a parsing failure, not a network term	Malformed or unexpected chunk	Cause 4
Error is generic (APIConnectionError, ECONNRESET) and happens inconsistently, including on stable networks	Dropped connection or masked provider error	Cause 5 and Cause 4
Error happens on the very first request after rotating a key, hitting a spend cap, or switching environments	Auth or quota failure	Cause 6

If your logs only show a generic exception name and no upstream message, that itself is a symptom — see Cause 4 and Cause 5 for why generic wrappers hide the real cause.

Cause 1: Mid-Stream Provider Error (Overload, Content Filter, Provider Crash)

This is the cause that confuses the most people, because the request looked successful. Once a streaming response starts, the HTTP status code and headers are already committed to the client. If the provider then hits a failure — capacity exhaustion, an internal error, a content filter triggering after partial output, or the model process crashing — it can’t switch the HTTP status to an error code. The failure has to travel inside the stream itself as a special event.

Anthropic’s Messages API documents this directly: the API “may occasionally send [errors] in the event stream,” and gives this exact example for an overload condition arriving mid-stream:

event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

This is a real gap in a lot of retry logic. If your code only inspects response.status_code, it already saw 200 before the failure happened, so a retry never triggers. A third-party incident writeup describing this exact pattern puts it plainly: “Retry logic that keys on the HTTP status code never triggers because the status was already 200. The result is a silently truncated response” — and recommends parsing the stream body for error events rather than trusting the status code alone.

OpenRouter’s documentation describes the same structural problem across the providers it routes to, not just one vendor. Once the first token is written, “the HTTP 200 OK status and headers are already committed — they can’t be changed,” so a provider failure “must arrive in-band as an SSE event.” Its documented causes for this class of error:

• Provider disconnect — the upstream connection drops after partial output (network issue, provider crash, load balancer timeout)
• Provider timeout — the model stops responding mid-generation and the read deadline expires
• Token limit hit during generation — the model reaches max_tokens or the context window fills up while producing output
• Output content filter — a moderation system flags generated text after some of it was already streamed
• Provider overload — the upstream returns a rate-limit or capacity error after beginning to stream

OpenRouter’s mid-stream error payload carries the error inside a normal-looking chunk, with a finish_reason that tells you the stream ended abnormally:

{"id":"gen-abc123","object":"chat.completion.chunk","created":1234567890,"model":"openai/gpt-4o","provider":"OpenAI","error":{"code":429,"message":"Rate limit exceeded","metadata":{"error_type":"rate_limit_exceeded"}},"choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}

How to confirm this is your cause: log the raw SSE stream (not just the parsed text output) for a failing request. If you see an error event or a chunk with a top-level error field anywhere in the stream before the terminal event, this is it.

Fix: treat finish_reason: "error" and event: error payloads as retryable failures, not as successful completions with empty content. Retry with backoff on the specific error type inside the payload, not on the HTTP status, since the HTTP status will already read 200.

Cause 2: Read Timeout or Idle Stream Stall

A stall differs from a hard error: the connection stays open, but no new tokens arrive for an extended period. Most HTTP clients conflate two timeout concepts — a total request timeout, and a per-read (idle) timeout. A long stream can legitimately outrun a short total timeout on a large response, while a genuinely stuck stream can sit well within a total timeout window if no idle-read check exists.

The OpenAI Python SDK sets a default total request timeout of 10 minutes: “By default requests time out after 10 minutes. You can configure this with a timeout option, which accepts a float or an httpx.Timeout object… On timeout, an APITimeoutError is thrown.” It also retries certain failures automatically: “Connection errors (for example, due to a network connectivity problem), 408 Request Timeout, 409 Conflict, 429 Rate Limit, and >=500 Internal errors are all retried by default,” twice, configurable via max_retries.

Confirm it: measure the gap between the last received token and the error. A fixed duration matching your configured timeout value points to a timeout, not a provider-side failure.

Fix: set two timeouts, not one — a connect/total timeout for the request lifecycle, and a separate idle-read timeout that fires if no chunk arrives within N seconds. That lets you distinguish “the model is slow” from “the stream died.” 15-30 seconds is a reasonable idle-read timeout for chat completions; raise it if your model has long silent “thinking” phases before the first token.

Cause 3: Rate Limit (429) Mid-Stream

Rate limits usually reject a request before it starts. But some gateways apply limits per-token or per-window, so a request can start streaming and get cut off once it crosses a budget mid-generation — the same in-band error shape as Cause 1, with a 429 code arriving inside the stream instead of as the initial response status.

Confirm it: check whether failures cluster during traffic bursts or at a consistent request-per-minute threshold. A Retry-After header, or an equivalent field in the error payload, confirms it.

Fix: respect Retry-After when present, reduce concurrency before retrying, and back off exponentially. Frequent hits are a signal to lower parallel stream count or upgrade account tier, not to retry more aggressively.

Cause 4: Malformed or Unexpected Chunk Breaking Your Parser

Not every “streaming error” is the model’s fault. Some are entirely client-side: your SSE parser assumes every chunk has a fixed shape, and a differently-shaped payload — including a legitimate error payload from the provider — breaks that assumption and throws an unrelated-looking exception.

A documented case: LiteLLM’s ollama_chat streaming handler expected every chunk to contain a message field. When Ollama instead returned a structured error such as {"error": "error parsing tool call: ..."}, the handler did an unguarded chunk["message"] lookup, raising KeyError: 'message'. A broader exception handler caught it and re-raised as APIConnectionError — a network-sounding error for what was actually a JSON parsing failure inside the model’s tool-call output. The bug report is explicit about the cost: engineers debugging it “logged reason=timeout / LLM request timed out for what was actually malformed tool-call JSON,” costing “two days of misdiagnosis” chasing network and infrastructure causes that weren’t the problem.

The general pattern: a provider sends an error in a shape your parsing code doesn’t expect, a low-level exception fires (KeyError, TypeError, a JSON decode error), and a catch-all except block wraps it in a generic connection or streaming error that erases the real cause.

Confirm it: log the raw chunk before your exception-wrapping code runs. If the raw payload has an error field with a real message, your client discarded useful information on the way to the generic error you’re seeing.

Fix: check for an error key before accessing expected fields like message or delta, and branch on it explicitly. Preserve the upstream error message and status code when re-raising, instead of collapsing every failure into one generic exception type.

Cause 5: Dropped Network Connection

Sometimes the cause really is the network: a proxy or load balancer closes a long-lived connection after an idle period, or a client’s network changes mid-request. This resembles Cause 2 but the fix differs — the connection itself is gone, not just idle within your own timeout window.

Confirm it: look for connection-reset-style errors (ECONNRESET, Broken pipe, Connection reset by peer) rather than an application-level timeout exception, and check whether failures correlate with a known intermediary’s idle-connection timeout (many load balancers default to 60s of inactivity).

Fix: if you control the proxy or load balancer, raise its idle-connection timeout above your expected max stream duration, or add keep-alive pings. If the drop is outside your control, treat it as retryable — most streaming APIs don’t support resuming a partial stream, so a retry means starting the generation over.

Cause 6: Auth or Quota Failure Surfacing Mid-Stream

Some gateways validate authentication and billing lazily — the connection opens, then the auth or balance check happens while generating the first chunk, and a failure there gets reported as a streaming error rather than a clean 401/402 at request time.

Confirm it: check whether the error happens on every request from a given key, rather than intermittently, and whether it started right after a key rotation, a billing event, or an environment change (dev key against a production base URL, or vice versa).

Fix: verify the key format (Authorization: Bearer <key>, not the raw key), confirm the key is active, and check account balance separately from request-level debugging. Every request failing identically regardless of prompt or model points here rather than to the previous five causes.

Retry and Backoff Pattern That Handles All Six

A single retry strategy can cover Causes 1 through 5 if it checks the stream body, not just the HTTP status:

import time
import openai

def stream_with_recovery(client, **kwargs):
    max_attempts = 3
    for attempt in range(max_attempts):
        try:
            collected = ""
            stream = client.chat.completions.create(stream=True, **kwargs)
            for chunk in stream:
                choice = chunk.choices[0] if chunk.choices else None
                if choice and getattr(choice, "finish_reason", None) == "error":
                    raise RuntimeError(f"mid-stream error: {chunk}")
                if choice and choice.delta.content:
                    collected += choice.delta.content
            return collected
        except (openai.APITimeoutError, openai.APIConnectionError, openai.RateLimitError) as e:
            if attempt == max_attempts - 1:
                raise
            time.sleep(2 ** attempt)
    return collected

This example uses the OpenAI-compatible client shape that also works against Novita AI’s OpenAI-compatible chat completions endpoint by setting base_url="https://api.novita.ai/openai". Three things matter beyond the code above:

• Check the chunk contents for an in-band error before assuming a normal completion, per Cause 1.
• Use exponential backoff (2 ** attempt, capped) rather than immediate retries, especially for rate limits and overload errors.
• Log the raw upstream error message before wrapping it in your own exception type, so future debugging doesn’t repeat the two-day misdiagnosis in Cause 4.

If a specific provider or model is failing more often than others, routing that request class to a different model is a mitigation worth having in place before you need it — see operating a multi-provider LLM service to a defined uptime target for how to define that fallback policy rather than improvising it during an incident.

Conclusion

Diagnosing this error is a process of elimination, not guesswork: use the checklist at the top to match your symptom to one of six causes, then confirm it with the specific signal each section calls out — an in-band error event, a stall duration matching your timeout, a burst-correlated 429, a raw payload your parser choked on, a connection-reset string, or a failure that repeats on every request from one key. Causes 1 through 3 are the most common, and Causes 1, 3, and 5 share the same underlying fix: stop trusting the HTTP status code once a stream has started, and instead retry with backoff based on what the stream itself reports.

Fixing the retry logic once, using the pattern above, closes the first five causes at the same time. Cause 6 is the exception — no amount of retrying fixes an invalid key or an empty balance, so treat identical failures on every request as a configuration check, not a network problem.

FAQ

Why does my LLM API request work sometimes and fail with a streaming error other times?

Intermittent failures point to a mid-stream cause rather than a configuration problem — configuration errors like a bad key or wrong endpoint fail every time. Check provider overload and rate limits first, since both are traffic-dependent.

Is a streaming error the same as a timeout?

Not always. A timeout means no response arrived within your configured window. A mid-stream error means a response started, some tokens arrived, and a failure event was then sent inside the stream. Error handling should distinguish the two, since the fixes differ.

Why does my error message say “connection error” when the real problem was something else?

Many client libraries wrap unexpected exceptions in a generic connection-error type when a chunk doesn’t match the shape the parser expects — see Cause 4 for a documented case where a JSON parsing error was reported as a connection timeout for two days before the real cause was found.

Can I resume a stream after a mid-stream error instead of starting over?

Most OpenAI-compatible and Anthropic-style streaming APIs don’t support resuming a partial stream at the point of failure. Retry the full request, discarding the partial output rather than appending to it, to avoid duplicated content.

Should I always use streaming for LLM API calls?

Streaming avoids a single large request timing out, but it introduces the partial-output failure mode covered throughout this guide. For short responses where you don’t need to show partial output, a non-streaming request is simpler to error-handle.

What does finish_reason: error mean in a streamed response?

It’s a terminal signal some gateways attach to the final chunk of a stream that failed partway through, distinct from normal values like stop or length. Treat it as a failed generation, even though the HTTP status for the request was 200.

Recommended Articles

• Best Multi-Provider LLM Service for Lower Cost and Higher Uptime? — SLO design, provider health monitoring, and incident playbooks for teams running LLM traffic across multiple providers.
• OpenAI Python SDK: Installation, Setup, and Practical Integration — Client configuration, streaming, retries, and timeout options for the SDK used in the code example above.
• What is Rate Limiting? A Practical Guide for AI Services — How rate limits are enforced and how to design retry behavior around them.