A streaming LLM chat endpoint with the AI SDK and Bedrock
Our AI assistant said it needed to ask a clarifying question. Then the question never appeared. The UI showed no error, and CloudWatch had nothing.
The AI SDK’s default error handler hid it from us. To find it, I followed one chat message through the whole endpoint, and this post walks the same path. It’s the first of four posts about our AI chat.
Where it runs
The assistant runs in our page service, a Node service that also owns pages and comments. The
/api/chat route uses the Vercel AI SDK with its Amazon Bedrock provider and streams UI messages to
the browser. A colleague wrote the first AI endpoints in mid 2025, and several people have worked on
the file since. This post describes how it works today.
Before any model call, the endpoint authenticates the caller and checks their AI budget. It also checks that the caller can access the workspace used for knowledge base lookups. Thread ids come from the client, so the endpoint checks that the thread belongs to the caller before it writes to it.
Picking a path with Haiku
A chat message can mean three things: generate new content, edit the user’s selection, or leave a
comment on the page. A small Haiku call decides which. It gets the user message, a short system
prompt and a limit of ten output tokens, and it must answer with one of the path names. The edit
path is only offered when the user has text selected. If the answer isn’t an exact match, or the call
throws, the endpoint logs a warning and uses generate.
One rule overrides the model. A selected passage can make Haiku read a question as a review request, so the endpoint only takes the comment path when the user explicitly asks for a comment.
The system prompt is a stack
The system prompt goes to the model as separate blocks, in this order:
- The static instructions, the same for most requests. In chat mode they go in whole, because an earlier length cap on this block silently dropped instructions.
- Instructions for tools that only exist in some sessions, such as the ones that talk to software open on the user’s desktop.
- Excerpts from the workspace’s Bedrock knowledge base.
- A structured analysis of the current page: its variables, values, dependencies and errors.
Each optional block has a character budget and is truncated to fit.
The prompt and the tool set have to agree. If the prompt describes a tool the model doesn’t have, the model tries to call it. So the prompt template and the tool registry read the same boolean, computed once per request. The block order also matters for prompt caching, which is the topic of the next post.
The tool loop
The model gets 15 tools on every request, plus template tools when there’s a workspace and connection tools when the request carries their context. The tools generate calculations and Python, edit and delete statements, navigate the page, search the docs and the web, and ask the user a clarifying question.
streamText runs the loop and stops when any of these happens:
- 14 steps have run, whether or not the task is finished.
- The clarifying question tool was called, so no stray text appears after the option buttons.
- A tool that runs on the client was called. These tools have no server-side
execute. Their result arrives with the browser’s next request.
The generation and edit tools validate their own output. When validation fails, prepareStep
narrows activeTools for the next step to the tool that failed. Without that, the model can “repair”
a failed edit by calling a generation tool. The page then has a duplicate block, and the original is
still broken.
After the template tool succeeds, prepareStep sets toolChoice to none, because the UI already
shows the template.
Two Bedrock quirks
Opus 5.5 uses adaptive thinking. When its first thinking block carries only a signature, the AI SDK’s
Bedrock adapter emits a reasoning-delta without a reasoning-start, and the stream dies with
reasoning part 0 not found. A language model middleware fixes it by injecting the missing start:
const startedReasoning = new Set<string>()
transform(chunk, controller) {
if (chunk.type === 'reasoning-start') {
startedReasoning.add(chunk.id)
} else if (
chunk.type === 'reasoning-delta' &&
!startedReasoning.has(chunk.id)
) {
startedReasoning.add(chunk.id)
controller.enqueue({ type: 'reasoning-start', id: chunk.id })
}
controller.enqueue(chunk)
}
The real version also handles a late start and a signature-only block that never ends. It’s a workaround for a bug in the adapter.
The second quirk is in the loop. A step can end with reasoning, or contain only reasoning, and
Bedrock rejects that assistant message on the next request of the same loop. So prepareStep strips
trailing reasoning on every step and drops any message left empty.
The silent failure
I added the clarifying question tool in June. Its schema requires at least two options, and the
model sometimes sent fewer. The AI
SDK then marks the tool call as failed, and the default onError on toUIMessageStream returns the
string An error occurred. and logs nothing. So the user saw no question, and we saw no error.
The fix was an onError on both chat streams, generate and edit. It logs through our normal logger
with the workspace, thread and model, and it tags the entry with the failing tool’s name. Datadog can
filter and alert on that tag. A frontend change shipped with it, because the UI dropped the
output-error state completely.
The first version logged the raw error object. Review on the PR pointed out that the error carries
the tool input, which is model-generated and can contain user data. Now the raw error is never
logged. The message and stack are forwarded only for the two error
types whose message describes the violated constraint, such as options: Array must contain at least 2 element(s). Every other error comes from a tool’s own execute, and its message can echo the
request, so only the error name is logged. A value that isn’t an Error logs its typeof.
I also found that the system prompt asked for 2 to 4 options, while the schema allowed up to 5. The prompt now says 2 to 5.
The PR also started with account-wide Bedrock invocation logging, which stores raw prompts and completions in S3 for every call in the account. Review flagged it as a privacy change, so I moved it to a separate branch for its own review.
My PR didn’t stop the model from sending fewer than two options. It only made the failure visible. Within three days a colleague changed the tool itself. Claude models sometimes leaked their native XML tool-call syntax into the nested option objects, and that destroyed all but one option. The options are now a flat list of strings, and the tool recovers labels from leaked tags. That is one way to end up with one option where the schema needs two.