AI Evaluation Interview Questions for 3 Years Experience

AI observability extends classic APM.

efSearch, nprobe, over-fetch for filters, offline benchmarks, and continuous monitoring of retrieval quality.

Build golden sets, error taxonomy, automated graders, and human review loops to iterate prompts like production software.

Scenario question on offline golden eval vs online A/B, feedback, and guardrail metrics in production.

Scenario question on classical NLP metrics — what they measure, where they break on paraphrase and factuality, and what to use instead.

Hard project question on use-case benchmarking — golden sets, task metrics, regression gates, and leaderboard limitations.

Hallucination in RAG systems is the #1 production failure mode cited in AI engineering interviews. Your interviewer wants a systematic debugging framework — not a list of buzzwords. Learn how to measure faithfulness, fix retrieval precision, and layer mitigations the way senior engineers at Databricks and Meta actually ship RAG.

OpenAI's model lineup changes fast. GPT-4 vs GPT-4o is a model selection question that tests whether you understand latency, cost, multimodal capabilities, and when reasoning depth matters. Critical for any role touching OpenAI APIs in production.

Scenario interview on defining and communicating hallucination metrics to executives.

Medium vector DB / embeddings trade-off question on dimensions vs cost/quality.

Medium prompt engineering question on prompt unit tests and CI gates.

**Gemini** Long video understanding, Google ecosystem integration.

**Eval dimensions** SWE-bench-style tasks on private repo sample; CI fix rate; hallucinated APIs.

**Mechanism** Server caches designated prompt prefixes; subsequent requests reuse if prefix identical.

**Tracing** Span per node: inputs hash, duration, LLM/tool child spans. thread_id correlates user session.

Agents fail opaquely without MCP observability. Treat every tool call as a microservice span.

Latency, recall proxies, ingest lag, index size, filter rates, error budgets, and RAG downstream signals.

MLOps for prompts: semantic versioning, eval gates, traffic splitting, and rollback when prompt changes regress quality or cost.

Medium AI Agents interview question on agent ux: streaming plans and progress — architecture, trade-offs, eval, and production patterns.

Medium AI Agents interview question on tool schema design and versioning — architecture, trade-offs, eval, and production patterns.

Medium AI Agents interview question on agent tracing and replay debugging — architecture, trade-offs, eval, and production patterns.

Medium AI Agents interview question on agent cost caps and budget controllers — architecture, trade-offs, eval, and production patterns.

Medium AI Agents interview question on coding agent diff review policies — architecture, trade-offs, eval, and production patterns.

Medium AI Agents interview question on stateful agents across sessions — architecture, trade-offs, eval, and production patterns.