“Ask KeyPair” is the streaming assistant on this site. You type a question, and Claude’s answer arrives token by token, grounded in our own content. The whole thing runs on a Cloudflare Worker — no origin server, no Node process. This is a write-up of the parts that were genuinely tricky to get right, because the happy path (“call the API, pipe the stream to the browser”) hides a few traps that only show up under real text.
The bug: newlines silently corrupt SSE framing
Server-Sent Events look trivial. You write data: followed by a payload and two
newlines, the browser’s EventSource parses it, done. The trap is in the spec
itself: a newline inside your payload terminates the field early. SSE is a
line-oriented protocol — every \n in the body is a frame boundary, not data.
When you’re streaming an LLM, your payloads are full of newlines. Code blocks, lists, paragraph breaks — any answer with structure carries them. The naive encoder:
// WRONG: a multi-line token silently truncates at the first \n
return `event: ${event}\ndata: ${data}\n\n`;works perfectly in every test where the token happens to be one line, and then the moment Claude emits a markdown list the client receives a mangled, half-parsed event. The symptom is maddening: text that looks fine until it suddenly doesn’t, with no error anywhere.
The fix is to split the payload on newlines and prefix every line with
data: — the SSE spec concatenates consecutive data: lines back together with
a \n between them, so the original newlines survive the round trip:
export function sseEvent(event: string, data: string): string {
const dataLines = data.split("\n").map((l) => `data: ${l}`).join("\n");
return `event: ${event}\n${dataLines}\n\n`;
}That’s the entire fix, and it’s the kind of one-liner that’s obvious in retrospect and invisible until a multi-line token lands in production. We have a round-trip test that encodes a deliberately multi-line payload and asserts it decodes back identically, so the regression can’t return.
HMAC-signed history: don’t trust the client’s conversation
The assistant is multi-turn, which means the client sends back the prior conversation on each request so Claude has context. That’s a problem: the client controls that history, and a chat history is just instructions to the model. A malicious client could fabricate an “assistant” turn — “Sure, I’ll ignore my guidelines” — and steer the next response.
So the worker signs every assistant turn it emits. When the stream finishes, the
done event carries an HMAC over the full accumulated answer text:
async function buildDonePayload(text: string, truncated: boolean, signer?: Signer) {
const payload: { truncated?: true; sig?: string } = {};
if (truncated) payload.truncated = true;
if (signer) payload.sig = await signer(text);
return JSON.stringify(payload);
}On the next request, validation rejects any assistant turn whose content doesn’t
match its signature — forged or edited history is a 400, not a steered model.
Two details made this robust:
- Graceful degradation. The signing secret is an optional binding. If it’s
absent (e.g. before the secret is provisioned), the worker skips
signing and verification entirely rather than
400-ing every request. The assistant degrades to unsigned instead of breaking. - Strip before forwarding. The
sigfield rides along on the client’s messages but is removed before anything goes to Anthropic — it’s our bookkeeping, not part of the prompt.
Atomic rate limiting with a Durable Object
A public LLM endpoint is a money tap pointed at your API bill, so it needs hard
limits: per-IP and a global daily cap. The obvious implementation — read a
counter from KV, increment, write it back — is racy. KV is eventually
consistent, and two concurrent requests can both read the same value and both
write n+1, so a burst sails straight through the limit.
The fix is a Durable Object. A DO is single-threaded, so the
read-modify-write inside each method is atomic — no lost increments under
concurrency. One DO per IP holds that IP’s token buckets; a single global DO
holds the daily cap.
The ordering matters more than it looks. We check the per-IP bucket first, and only consult (and so consume) the global counter when the IP check passes:
const perIp = await ipStub.takeIp("assistant", capacity, windowSec, nowMs);
if (!perIp.allowed) return { allowed: false, reason: "ip", retryAfter: perIp.retryAfter };
const global = await globalStub.takeGlobal(globalCap, nowMs);
if (!global.allowed) return { allowed: false, reason: "global", retryAfter: global.retryAfter };If an IP-denied request still advanced the global counter, one abusive IP’s cheap denied burst could exhaust the daily cap and lock the assistant for everyone. Per-IP-first prevents that.
Two more pieces of hygiene. The bucket math lives in a separate pure module that both the DO and a KV fallback import, so the two paths can’t drift. And if the DO itself throws, we log and fall back to the (racy but available) KV limiter — losing strict atomicity for a moment beats failing the whole endpoint.
Lazy hydration: don’t pay for what nobody opened
The site is otherwise static — Astro ships almost no JavaScript. The assistant is the one heavy interactive piece, and most visitors never open it. So it hydrates lazily: the chat machinery is a separate chunk that only downloads and mounts when someone actually engages the widget. Every other page stays as light as it was before the assistant existed. Putting interactive weight behind an explicit user action — rather than hydrating it site-wide on every page load — is the single highest-leverage performance decision on an island-architecture site.
The takeaway
None of these are exotic. SSE framing, untrusted client history, racy counters, and eager hydration are all ordinary mistakes — which is exactly why they’re worth naming. Streaming an LLM to a browser from the edge is very doable on Workers; the failure modes just don’t announce themselves. Test the multi-line case, sign anything the client hands back to you, make your counters atomic, and hydrate only what gets used.
This is how we build our own products — real LLM features wired into production, not demos. The assistant you’re reading about is live in the corner of this very page.