AI Latency Interview Questions

Write buffering, mutable segments, async merge, backpressure, and consistency windows for hot ingestion paths.

efSearch, nprobe, over-fetch for filters, offline benchmarks, and continuous monitoring of retrieval quality.

ANN parameter tuning, rebuild windows, incremental indexing, and serving fresh vectors without missing SLOs.

Medium AI Agents interview question on parallel tool calls and fan-out — architecture, trade-offs, eval, and production patterns.

Hard RAG interview question on rag latency budget: sub-second design — architecture, trade-offs, eval, and production patterns.

Scenario question on LLM latency — prefill vs decode, retrieval overhead, streaming, caching, and model routing fixes.

Chain-of-Thought prompting remains one of the most frequently asked prompt engineering questions, yet candidates often give surface-level answers. Learn when CoT helps vs hurts, production token costs, and advanced variants like self-consistency and tree-of-thoughts that impress senior interviewers.

OpenAI's model lineup changes fast. GPT-4 vs GPT-4o is a model selection question that tests whether you understand latency, cost, multimodal capabilities, and when reasoning depth matters. Critical for any role touching OpenAI APIs in production.

500M vectors at sub-100ms p99 is a staff-level vector search design question from Uber, Airbnb, and large-scale ML platform teams. Learn sharding strategies, index tuning, and the operational trade-offs that separate senior from principal engineers.

AGENTS.md is becoming the de facto 'README for agents.'

AI observability extends classic APM.

Medium Gemini/Vertex question on context caching vs RAG trade-offs.

Medium Claude cost question on prompt caching breakpoints and hit rates.

Scenario interview on safe GenAI rollouts — flags, shadow eval, canaries, and kill switches.

LLM interview topic on distillation — teacher/student training, latency/cost wins, and quality regression risks.

**vLLM** High throughput continuous batching; popular for multi-tenant APIs.

**Session model** WebSocket/WebRTC streaming multimodal input/output.

**Model choice** Flash for simple; Pro for hard multimodal only when needed.

**Gemini** Long video understanding, Google ecosystem integration.

**Use cases** Full codebase questions, hours-long media, multi-doc merger analysis.

**Behavior** More deliberation tokens internally — improved accuracy on complex code/math.

**Eval dimensions** SWE-bench-style tasks on private repo sample; CI fix rate; hallucinated APIs.

**Mechanism** Server caches designated prompt prefixes; subsequent requests reuse if prefix identical.

**Session** Persistent realtime connection with session.update for instructions/tools; handle reconnect.