AI Cost Optimization Interview Questions

Cursor rules are context engineering in disguise.

**Tracing** Span per node: inputs hash, duration, LLM/tool child spans. thread_id correlates user session.

Embedding similarity, k-NN example banks, MMR diversity, and eval-driven example curation for few-shot prompts.

Hard AI Agents interview question on multi-agent debate and critique patterns — architecture, trade-offs, eval, and production patterns.

Medium RAG interview question on compression and contextual distillation — architecture, trade-offs, eval, and production patterns.

Medium RAG interview question on agentic rag vs naive rag — architecture, trade-offs, eval, and production patterns.

Company/scenario question on open vs closed LLMs — self-hosting economics, compliance, upgrade velocity, and hybrid strategies.

Self-hosting Llama 3 is a infrastructure-heavy question for ML platform and AI engineer roles at Meta-adjacent companies. Expect deep dives on quantization, vLLM, GPU sizing, and the TCO math that determines build vs buy decisions.

Cost optimization for coding agents is context engineering + runtime policy.

This is an enterprise platform design question spanning agents, MCP, governance, and observability.

AGENTS.md is becoming the de facto 'README for agents.'

Product teams shouldn't each reinvent agent security.

AI observability extends classic APM.

Treat agent models/prompts/tools like dependencies.

Hard production LLM question on routing, cascading, and router evaluation.

Medium Gemini/Vertex question on context caching vs RAG trade-offs.

Medium Claude cost question on prompt caching breakpoints and hit rates.

Medium vector DB / embeddings trade-off question on dimensions vs cost/quality.

Scenario interview on safe GenAI rollouts — flags, shadow eval, canaries, and kill switches.

LLM interview topic on distillation — teacher/student training, latency/cost wins, and quality regression risks.

**Dimensions** Quality, cost curve, latency, compliance, ops burden, roadmap velocity.

**Metrics** Requests queued, p95 latency, GPU memory headroom.

**Model choice** Flash for simple; Pro for hard multimodal only when needed.

**Ingestion** Transcode standards; chunk very long media if API limits apply.