AI Interview Questions

Self-hosting Llama 3 is a infrastructure-heavy question for ML platform and AI engineer roles at Meta-adjacent companies. Expect deep dives on quantization, vLLM, GPU sizing, and the TCO math that determines build vs buy decisions.

Hard Llama/inference question on continuous batching and GPU efficiency.

**Dimensions** Quality, cost curve, latency, compliance, ops burden, roadmap velocity.

**Analysis** Rank features by volume, quality bar, data sensitivity benefiting on-prem.

**Key clauses** 700M MAU threshold triggers extra terms; AUP prohibits harmful uses.

**Usage** Separate small model for moderation latency.

**Metrics** Requests queued, p95 latency, GPU memory headroom.

**Constraints** No phone-home telemetry; offline license compliance documentation.

**Training** Refusal datasets, toxicity filters on training data.

**Training** Rank selection, learning rate, epochs; validation loss + task evals.

**GPTQ/AWQ** GPU inference friendly; integrate with vLLM/TGI depending on support.

**vLLM** High throughput continuous batching; popular for multi-tenant APIs.

**3.1** 128k context, stronger instruction following — update inference stack for long context KV cache.