I failed my first LLM system design round. Here's what fixed it

First, a plain note. The candidate in this story is an illustrative composite, not a real person, and the interview described here is not a report of any real company's process or outcome. It is a realistic pattern, written in first person so the mistakes feel concrete.

The prompt was short. Design a support assistant that answers customer questions from a company's help center and past tickets. I had forty-five minutes and a shared whiteboard. I was confident. I had built two chatbots and read plenty about retrieval. I left knowing I had failed, and it took me a few days to understand why.

Mistake one was starting with the model. Within two minutes I had picked a large model, written a system prompt on the board, and started talking about temperature. The interviewer asked who the users were and how many questions a day we expected. I did not know, because I had not asked. A system design round is still a system design round. The fix was a habit I now use every time: spend the first five minutes on users, traffic, data sources, freshness, and what a wrong answer costs. Those answers decide almost everything else.

Mistake two was treating retrieval as a box. I drew an arrow labeled vector database and moved on. Then came the follow-ups. How do you chunk the help articles? What happens when a ticket and an article disagree? How do you keep one customer's tickets away from another? I gave vague answers. The fix was learning retrieval as its own design problem: ingestion, chunking, metadata, access control at query time, and how to check whether the right passage actually came back. The RAG interview guide at /blog/rag-interview-guide is close to the checklist I wish I had carried in.

Mistake three was having no answer for how I would know it works. The interviewer asked how I would decide the assistant was ready to ship. I said we would test it with some users. That is not a plan. The fix was a small evaluation story I could tell in under two minutes: a fixed set of real questions with expected answers, separate checks for retrieval quality, faithfulness to the retrieved text, and whether the answer resolved the question, plus a review of failures before every change goes out. The LLM evaluation guide at /blog/llm-evaluation-interview-guide covers how to phrase that.

Mistake four was ignoring failure paths. I drew the happy path only. No fallback when retrieval returned nothing useful, no hand-off to a human, no limit on how many steps the system could take if I added tools later. When the interviewer asked what happens when the model is unsure, I froze. The fix was drawing failure paths on purpose: a confidence or coverage check, a clear hand-off, and a step limit for anything agent-like. If the design includes tools or multi-step reasoning, the LangGraph agent system design guide at /blog/langgraph-agent-system-design-interview-guide shows how to put state and stopping rules on the board.

Mistake five was cost and latency as an afterthought. Near the end, the interviewer asked what happens when traffic grows. I said we would scale the servers. The better answer talks about where time and money actually go: retrieval, the model call, and output length. Then it names levers such as caching repeated questions, trimming context, streaming the reply, and routing simple questions to a smaller model. The LLM cost and latency guide at /blog/llm-cost-latency-interview-guide lays out those trade-offs.

What fixed it was not memorizing more architectures. It was a repeatable shape for the forty-five minutes. Clarify requirements first. Sketch the data flow end to end. Go deep on retrieval. Explain how you will evaluate it. Draw the failure paths. Close with cost, latency, and what you would build next. I practiced that shape out loud on three different prompts, a support assistant, an internal document search tool, and a meeting summarizer, until I could move through it without notes.

The next design round I sat felt different. I still did not know every answer. But when the interviewer pushed on something, I knew which part of the system the question belonged to, and I could say what I would measure to find out. That turned out to be what the round was testing.