The LLM Wasn't Hallucinating: The Subtle SSE Streaming Bug That Broke Our AI Planner

Overview

We all love blaming the model.

When your AI web application suddenly starts dropping numbers, truncating timestamps, or outputting malformed JSON, the immediate instinct is usually: "The model hallucinated again. We need a stricter system prompt, lower temperature, or few-shot examples."

Earlier today, while building the real-time AI Planning Coach for Bearing, I fell into that exact trap.

Our AI was supposed to output a focused work schedule:

• Standup: 2:30 PM — 3:00 PM
• Feature QA: 3:15 PM — 5:15 PM (2 hours)

In the conversational text, the model wrote: Feature QA: 3:15 PM — 5:15 PM. Yet in the interactive UI card right below it, the block rendered as 15:15–16:15 (1h 0m).

One whole hour was missing. I tweaked the prompt, added strict arithmetic rules, tuned temperature down to 0.1, and built regex fallbacks. Nothing worked. Every few requests, a timestamp would silently mutate or truncate.

Then I inspected the raw network packets. What I found wasn't an AI failure at all. It was a classic, insidious networking bug in how frontend applications consume Server-Sent Events (SSE).

The Illusion: What the Stream Looked Like

Our backend streams completions using the OpenAI-compatible SSE format (data: {"choices":[{"delta":{"content":"..."}}]}).

On the client side, we were reading the stream using the standard ReadableStreamDefaultReader pattern you see in virtually every tutorial online:

// ❌ The Naive Implementation (Don't do this!)
const reader = res.body.getReader();
const decoder = new TextDecoder();
let fullAssistantText = '';

while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  // Decode the current network chunk
  const chunk = decoder.decode(value, { stream: true });

  // Split lines by newline
  const lines = chunk.split('\n');

  for (const line of lines) {
    const trimmed = line.trim();
    if (!trimmed.startsWith('data:')) continue;

    const dataStr = trimmed.replace(/^data:\s*/, '');
    if (dataStr === '[DONE]') continue;

    try {
      const parsed = JSON.parse(dataStr);
      const delta = parsed.choices[0]?.delta?.content || '';
      fullAssistantText += delta;
    } catch {
      // "Ignore malformed chunks" - the silent killer!
    }
  }
}

It looks completely reasonable. It handles data:, it parses JSON, it extracts the delta token, and it ignores non-JSON lines. So why was 17:15:00Z turning into 16:15?

The Culprit: TCP Packet Fragmentation

reader.read() does not yield complete SSE lines. It yields raw bytes as they arrive over the TCP connection. If a single SSE event line happens to cross a TCP buffer boundary, the payload gets sliced right in the middle:

// --- Packet 1 arrives ---
data: {"choices":[{"delta":{"content":":00Z\",\n \"endTime\": \"2026-08-14T17

// --- Packet 2 arrives (10ms later) ---
:15:00Z\",\n \"priority\": \"HIGH\",\n \"isFixed\": false\n

Look closely at what the naive loop did to these two packets:

• Packet 1 arrives: chunk.split('\n') takes the last line. It runs JSON.parse(). Because the JSON has no closing quote or curly braces, it throws SyntaxError. The try/catch silently discards the entire line.
• Packet 2 arrives: the line now starts with :15:00Z", .... It checks trimmed.startsWith('data:'). Because this is just the tail of the previous event, it doesn't start with data: — so it gets discarded too.

The token :15:00Z vanished into thin air. The accumulated text was left with only 2026-08-14T17. Our downstream time parser saw an incomplete timestamp without minutes, defaulted to startTime + 1 hour, and produced 16:15.

A silent try/catch block around JSON.parse in a stream reader isn't error handling — it's an invisible data shredder.

The Fix: Persistent Stream Buffer Accumulation

The solution is straightforward once you understand that network streams are streams of characters, not structured records. You must maintain a line buffer (sseBuffer) that holds incomplete strings across iterations:

// ✅ The Bulletproof SSE Reader
const reader = res.body.getReader();
const decoder = new TextDecoder();
let fullAssistantText = '';
let sseBuffer = ''; // Persistent line accumulator

while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  // 1. Append new raw text to whatever was left over from the previous read
  sseBuffer += decoder.decode(value, { stream: true });

  // 2. Split into lines
  const lines = sseBuffer.split('\n');

  // 3. Pop the last element - it is either empty (if ended in \n) or incomplete!
  sseBuffer = lines.pop() ?? '';

  // 4. Process only guaranteed complete lines
  for (const line of lines) {
    const trimmed = line.trim();
    if (!trimmed.startsWith('data:')) continue;

    const dataStr = trimmed.replace(/^data:\s*/, '');
    if (dataStr === '[DONE]') continue;

    try {
      const parsed = JSON.parse(dataStr);
      const deltaContent = parsed?.choices?.[0]?.delta?.content || '';
      if (deltaContent) {
        fullAssistantText += deltaContent;
      }
    } catch (err) {
      console.warn('Failed to parse complete SSE line:', dataStr, err);
    }
  }
}

// 5. Handle any final trailing data when the stream closes
if (sseBuffer.trim().startsWith('data:')) {
  const dataStr = sseBuffer.trim().replace(/^data:\s*/, '');
  if (dataStr !== '[DONE]') {
    try {
      const parsed = JSON.parse(dataStr);
      fullAssistantText += parsed?.choices?.[0]?.delta?.content || '';
    } catch {
      // Safe to ignore at EOF
    }
  }
}

What Happens With the Buffer in Place

Let's replay the exact same network split with sseBuffer:

• Packet 1 arrives: sseBuffer contains the incomplete line. lines.pop() removes it and stores it back in sseBuffer. Zero JSON parse errors.
• Packet 2 arrives: sseBuffer += Packet 2. Now sseBuffer contains the complete line, and the loop successfully parses the full event.

fullAssistantText receives 17:15:00Z in its entirety. The schedule displays 15:15 – 17:15 (2h 0m), flawlessly.

Key Takeaways for AI Engineers

• Don't immediately blame the model. When output seems truncated or subtly corrupted, log the raw byte stream before accusing the LLM of hallucination.
• Streams have no message boundaries. TCP packets and HTTP chunk streams don't care about JSON syntax or SSE delimiters. Always accumulate until a boundary is reached.
• Watch out for copy-pasted tutorial code. Most quickstart guides for fetch-based SSE omit the line buffer for brevity, leaving production apps vulnerable to random dropped tokens under fluctuating network conditions.

Have you ever caught a frontend stream reader dropping tokens in your AI apps? How are you handling stream accumulation in production?