Deep explanation
Design OpenAI architecture for a coding copilot for a large engineering org
Junior system design: scale an OpenAI-based coding copilot with an API gateway, permission-aware code retrieval, streaming, prompt caching, batch embeddings, secret filtering, and per-team rate limiting.
Quick answer
I would put a backend gateway between the IDE plugin and OpenAI so we control auth, logging, rate limits, and model choice in one place. Chat about the codebase uses a code index: we chunk code by functions and files, embed it with an OpenAI embedding model, and only retrieve chunks from repos the engineer can access. Prompts are built with a stable prefix so prompt caching helps, answers stream to the IDE, secrets are stripped before anything leaves our network, and we track tokens and cost per team.
The interview question
A coding copilot prototype built on the OpenAI API works for one team. Now it must serve thousands of engineers across many repositories, with a code index of tens of millions of chunks. What architecture would you propose for completions, chat about the codebase, and the code index, and how would you handle permissions, secrets, cost, and rate limits?
Deep explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 21 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions