Engineering / Architecture
Scaling Websockets for Real-Time LLM Streaming
The Latency Problem
When building real-time LLM applications, time-to-first-token (TTFT) and the subsequent streaming latency dictate the user experience. HTTP polling is inefficient, and Server-Sent Events (SSE) often struggle with bidirectional communication and complex state management across distributed load balancers.
At Syntave, we needed a way to stream thousands of concurrent LLM responses directly to clients with sub-50ms latency, while also allowing clients to seamlessly interrupt generation, update parameters mid-flight, and maintain persistent connection state.
A Concurrent Architecture
We rebuilt our streaming layer using a highly concurrent Websocket architecture in Go. By leveraging goroutines and channels, we map each client connection to a lightweight process that independently manages its own stream state.
Instead of holding open long-lived HTTP requests that tie up thread pools, our websocket implementation requires mere kilobytes of memory per connection. This allows a single Syntave node to comfortably handle 100,000+ concurrent streaming LLM sessions without breaking a sweat.
Conclusion
Switching to a fully bidirectional websocket model reduced our P99 streaming latency by over 300ms and completely eliminated head-of-line blocking issues associated with HTTP/1.1 proxies. By taking control of the transport layer, we ensured that the only bottleneck in our pipeline is the underlying LLM inference speed itself.