AI Evaluation Interview Questions for 10 Years Experience

Hard project question on use-case benchmarking — golden sets, task metrics, regression gates, and leaderboard limitations.

Hard AI Agents interview question on agent evals with private task suites — architecture, trade-offs, eval, and production patterns.

Hard AI Agents interview question on evaluating agent task success rates — architecture, trade-offs, eval, and production patterns.

Hard RAG interview question on evaluating rag: faithfulness, context precision, recall — architecture, trade-offs, eval, and production patterns.

Treat agent models/prompts/tools like dependencies.

Part 2 of the LLM hallucination handbook — factual, citation, reasoning, math, code, temporal, context, and tool-use hallucinations; decoding risks; RAG failure modes; production case study.

Part 3 of the hallucination handbook — production defense layers from retrieval and prompting through guardrails, citations, confidence scoring, eval frameworks, and enterprise architecture. Asked at OpenAI, Google, Meta, Anthropic, Microsoft, and Amazon.

Hard production LLM question on routing, cascading, and router evaluation.

**Metrics** Requests queued, p95 latency, GPU memory headroom.

**vLLM** High throughput continuous batching; popular for multi-tenant APIs.

**Structure** Eval templates + data JSONL + scoring functions; CI integration via API.

Company-based MCP questions assess platform leadership, not wire protocols alone.

Write buffering, mutable segments, async merge, backpressure, and consistency windows for hot ingestion paths.

Learned sparse representations, inverted index integration, lexical expansion, and fusion with dense ANN.

Shared embedding spaces, separate indexes, CLIP-style models, metadata routing, and fusion strategies.

Programmatic prompt tuning with labeled datasets, composable modules, and optimizers that search instructions and demonstrations.

Hard reasoning prompt architectures that explore, evaluate, and merge intermediate thought states beyond linear chain-of-thought.

Hard AI Agents interview question on long-running agents and checkpoints — architecture, trade-offs, eval, and production patterns.

Hard AI Agents interview question on multi-user concurrent agents — architecture, trade-offs, eval, and production patterns.

Hard AI Agents interview question on building an internal agent marketplace — architecture, trade-offs, eval, and production patterns.

Hard AI Agents interview question on a2a / agent-to-agent protocols — architecture, trade-offs, eval, and production patterns.

Hard AI Agents interview question on agent permissions and least privilege — architecture, trade-offs, eval, and production patterns.

Hard AI Agents interview question on customer support agent system design — architecture, trade-offs, eval, and production patterns.

Hard AI Agents interview question on designing an on-call ai sre agent — architecture, trade-offs, eval, and production patterns.