
FastAPI-stream-lease – Limit concurrent SSE/LLM streams instead of req/min
Limit simultaneously active SSE, LLM token streams, and WebSocket sessions across FastAPI/Starlette workers with Redis. Set a per-user limit, a global limit, or both. An extra connection receives HTTP 429 before streami…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Limit simultaneously active SSE, LLM token streams, and WebSocket sessions across FastAPI/Starlette workers with Redis. Set a per-user limit, a global limit, or both. An extra connection receives HTTP 429 before streaming begins.
Request rate limiters answer “how many requests arrived this minute?” This package answers “how many streams are open right now?” It is useful when a connection can remain open for seconds or minutes. For a general-purpose distributed semaphore, consider py-redis-limiters or py-redis-semaphore.
sequenceDiagram autonumber actor C1 as Client (Tab 1) actor C2 as Client (Tab 2 / Scraper) participant API as FastAPI + StreamLease participant Redis as Redis (Atomic Lua ZSET) participant LLM as OpenAI / Anthropic Stream
Note over API,Redis: LeaseConfig(max_per_user=1, lease_seconds=30, acquire_timeout=0.5)
rect rgba(46, 160, 67, 0.12) C1->>API: GET /v1/chat/stream API->>Redis: EVAL ACQUIRE_SCRIPT (O(1)) Redis-->>API: {1, 0} -> Lease Granted (Active: 1/1) API->>LLM: Start token generation LLM-->>C1: HTTP 200 (X-Stream-Lease-Id: 8f2a...) + SSE chunks end
rect rgba(235, 87, 87, 0.15) Note over C2,LLM: Parallel tab or abuse while Stream #1 is in-flight C2->>API: GET /v1/chat/stream API->>Redis: EVAL ACQUIRE_SCRIPT (waits up to acquire_timeout) Redis-->>API: {2, 24100} -> User limit reached API-->>C2: HTTP 429 Too Many Requests (Retry-After: 25) Note over API,LLM: 0 LLM tokens billed! Blocked before calling upstream end
rect rgba(56, 139, 253, 0.12) Note over C1,LLM: User closes browser tab or hits Stop mid-generation C1--xAPI: ASGI http.disconnect API--xLLM: ProtectedStreamingResponse invokes aclose() (stops token billing!) API->>Redis: Shielded EVAL RELEASE_SCRIPT (<1ms) Redis-->>API: Slot freed immediately (Active: 0/1) end
Loading
Live Terminal Preview
┌─ Terminal 1: Client Stream #1 (Allowed) ──────────┐ ┌─ Terminal 2: Concurrent Stream #2 (Blocked) ─────────┐ │ $ curl -i -N http://localhost:8000/v1/chat/stream │ │ $ curl -i http://localhost:8000/v1/chat/stream │ │ HTTP/1.1 200 OK │ │ HTTP/1.1 429 Too Many Requests │ │ content-type: text/event-stream; charset=utf-8 │ │ content-type: application/json │ │ x-stream-lease-id: 8f2a9c41d0e74b12a5c3... │ │ retry-after: 28 │ │ x-stream-lease-user-limit: 1 │ │ │ │ x-stream-lease-global-limit: 100 │ │ { │ │ │ │ "code": "stream_connection_limit", │ │ data: Hello │ │ "reason": "user_limit", │ │ data: world │ │ "detail": "User stream limit reached" │ │ ^C (Client disconnects -> slot freed in <1ms) │ │ } │ └───────────────────────────────────────────────────┘ └──────────────────────────────────────────────────────┘
Install
Requires Python 3.10+, a Redis server 5.0+, and the Python client redis-py 4.6.0+. The server and Python client have independent version numbers.
The core accepts a caller-owned async Redis client; it does not configure or replace your connection pool. With redis-py 4.6, RESP2 is implicit and protocol is not a supported constructor argument.
With Redis 5 and redis-py 8+, construct your client with redis.from_url(url, protocol=2): redis-py 8 defaults to RESP3, which Redis 5 does not support. See the redis-py protocol documentation.
pip install 'fastapi-stream-lease[fastapi]'
The fastapi extra supplies FastAPI for the HTTP helpers; the core package only requires redis.
The runnable SSE example accepts one API key from STREAM_DEMO_TOKEN and uses it as a demo identity. Start Redis, clone this repository, then run:
git clone https://github.com/agustin18/fastapi-stream-lease.git cd fastapi-stream-lease pip install -e '.[fastapi]' uvicorn export STREAM_DEMO_TOKEN=local-secret export REDIS_URL=redis://localhost:6379/0 uvicorn examples.sse_demo:app
In another terminal, open two streams, then try a third with the same key:
curl -N -H 'X-API-Key: local-secret' http://localhost:8000/stream
The first two requests stream events; the third receives 429 with a Retry-After header until a slot is freed. Replace the demo API key with your application's authenticated user or account ID before production use. Never use an unverified request parameter as the user ID.
from contextlib import asynccontextmanager
from fastapi import Depends, FastAPI, Request from fastapi.responses import StreamingResponse import redis.asyncio as redis
from fastapi_stream_lease import ( LeaseConfig, StreamLeaseManager, StreamLeaseRejected, StreamLeaseUnavailable, )
Set explicit timeouts so slow Redis calls do not block worker threads
redis_client = redis.from_url( "redis://localhost:6379", socket_timeout=1.0, socket_connect_timeout=1.0, ) manager = StreamLeaseManager( redis_client, LeaseConfig(max_per_user=2, max_global=500, lease_seconds=30), )
@asynccontextmanager async def lifespan(app: FastAPI):
Verify cluster configuration consistency on worker startup (fail-fast on mismatch or outage)
await manager.verify_cluster_config(strict=True) yield
Clean teardown: drain telemetry callbacks and close resources
await manager.close(drain=True, timeout=5.0) if hasattr(redis_client, "aclose"): await redis_client.aclose() else: await redis_client.close()
app = FastAPI(lifespan=lifespan)
async def authenticated_user_id() -> str:
Illustrative placeholder: replace with your auth dependency (e.g. API key or JWT sub).
For fully runnable code, see examples/sse_demo.py and examples/websocket_demo.py.
return "user-123"
Register standard 429 (StreamLeaseRejected) and 503 (StreamLeaseUnavailable) handlers:
manager.register_exception_handlers(app)
@app.get("/stream") async def stream(user_id: str = Depends(authenticated_user_id)): async def events(): yield "data: token or event\n\n"
1-Line Protected StreamingResponse (auto-acquires, wraps, and cleans up on errors)
return await manager.stream(user_id, events())
Or manually acquire and convert:
lease = await manager.acquire(user_id)
return lease.as_streaming_response(events())
Close the Redis client in your application's lifespan shutdown handler. Reuse the same StreamLeaseManager and key_prefix across workers that share limits. max_per_user=0 or max_global=0 disables that limit; the global count is unavailable when global tracking is disabled.
For WebSockets, keep the context open for the whole session:
async with manager.lease(user_id) as lease: while True: message = await websocket.receive_text() await websocket.send_text(message)
The manager context and async with lease both renew while open. Handle normal WebSocket disconnects in your route as usual. If renewal fails or the lease expires, StreamLeaseLost interrupts the stream or context. Catch it at the application boundary if you want to record a metric or send an application-specific WebSocket close code. wrap(auto_renew=False) disables automatic renewal; use it only if you renew the lease yourself.
Acquisition, renewal, expiration cleanup, and release use atomic Redis Lua scripts. The keys share a Redis Cluster hash tag, so a user and global limit can be checked in one script.
Every lease expires after lease_seconds without a successful renewal. wrap() and manager.lease() renew every half interval by default. Transient Redis connection errors trigger fast retries across the remaining lease TTL (Adaptive Grace Period), preventing temporary hiccups from dropping active streams.
If Redis is unavailable on initial acquisition, StreamLeaseUnavailable (HTTP 503) is raised by default (fail_open=False). Set fail_open=True in LeaseConfig if your application prefers allowing streams during Redis outages (graceful degradation).
Local pool exhaustion rejects acquisition with StreamLeaseUnavailable, even with fail_open=True, and does not increment circuit failure counters. Older clients signal it with exact ConnectionError messages (Too many connections or No connection available.); newer clients may use MaxConnectionsError. Renewal retries remain bounded by the lease TTL. An already-open circuit retains its configured fallback policy.
Dynamic Retry-After Calculation in Redis Lua: When acquisition is rejected because user or global limits are reached, the atomic Lua script queries the Sorted Set in $O(1)$ (ZRANGE ... 0 0 WITHSCORES) to find the earliest expiring active lease. The dynamic remaining time until capacity is freed is returned directly and exposed via StreamLeaseRejected.retry_after (and populated in the standard HTTP Retry-After response header).
Acquisition Timeout Grace Window (acquire_timeout): Set acquire_timeout in LeaseConfig (e.g. acquire_timeout=0.5) or pass timeout=... directly to manager.acquire(..., timeout=...), manager.lease(..., timeout=...), or manager.stream(..., timeout=...). A non-blocking cooperative retry loop absorbs transient concurrency spikes, client reconnection storms, and connection races without returning premature HTTP 429 rejections. The local lease TTL (expires_at) is calibrated from the exact moment acquisition succeeds.
Graceful Reactive Teardown (manager.close(drain=True)): During worker termination or SIGTERM, await manager.close(drain=True, timeout=5.0) reactively awaits the natural completion of active local streams via an event-driven mechanism without polling. Lingering active leases past the timeout are force-released concurrently in a single round-trip with cancellation shielding (anyio.CancelScope(shield=True)), preventing ghost leases in Redis during rolling deploys.
Normal completion or cancellation attempts immediate release. If Redis is unavailable during release, the lease is removed after expiration; cleanup of the key itself uses a longer TTL. An async iterator abandoned without being closed may also hold its slot until expiration. Use contextlib.aclosing() if your own consumer stops iteration early.
get_active_count(user_id) counts active leases for one identity; get_active_count() counts globally when max_global is enabled. Neither is a historical usage metric.
All workers sharing limits must use the same key prefix and compatible limit settings. Lease expiration is measured by Redis, avoiding clock differences among application workers.
Distributed Guarantees & Failure Model
Failure Mode System Behavior Guarantee Level Operational Trade-off
Worker Process Crash (SIGKILL) Lease expires in Redis after lease_seconds via Redis TIME score. Subsequent acquisitions automatically sweep expired members. Strong (Self-healing within $TTL$) Slot remains held until lease_seconds elapses; no zombie leases persist permanently.
Transient Redis Disconnect (Renewal) Renewal worker enters Adaptive Grace Period, retrying across the remaining TTL. If connection recovers before deadline, stream proceeds normally. High Availability If outage exceeds remaining TTL, lease is revoked, cancelling stream immediately.
Event Loop Stalled / Process Paused > TTL Redis lease expires while local worker is stalled (e.g. extreme GC pause, VM suspension, CPU starvation). When execution resumes, the next renewal detects expiration and terminates the stream immediately with StreamLeaseLost. Eventual Safety Slots self-clean in Redis, but a local worker stalled past TTL cannot observe cancellation until its event loop resumes execution.
Redis Outage on Acquire (fail_open=False) Immediate fail-closed rejection raising StreamLeaseUnavailable (HTTP 503 Service Unavailable). Strict Safety Limits strictly enforced; incoming streams rejected until Redis is reachable.
Redis Outage on Acquire (fail_open=True) Grants an uncoordinated in-memory fallback lease using worker monotonic clock (time.monotonic()). Graceful Degradation Fallback acquisitions are intentionally unthrottled across and within worker processes during outage; fallback leases do not retroactively register upon Redis recovery.
Worker Clock Drift All lease evaluations and expiration purges use redis.call('TIME'). Absolute Worker system clock or NTP skew cannot cause premature expiration or lingering leases.
Slow Observability / Metric Hoo
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。