Small Language Models Interview Guide: Edge Deployment, Distillation, and Cost Control

Small language model interviews center on economic and latency constraints, not model size bragging. Interviewers ask when a 3B–8B model routed correctly beats a frontier API, how distillation preserves task-specific capability, and what quality floor is acceptable for each product surface.

Deployment targets—mobile, browser, on-prem CPU—drive architecture. Discuss quantization formats (INT4, GPTQ, AWQ), batch size effects on throughput, and memory ceilings that force context truncation. Candidates should quantify trade-offs with real latency and dollar-per-request estimates.

Routing layers make SLM strategies work. A small model handles classification, extraction, and triage; a larger model handles complex generation. Interviewers probe how you measure routing accuracy, prevent error cascades, and eval the full cascade—not isolated models.

Privacy and compliance often motivate SLMs. Articulate air-gapped deployment patterns, update mechanisms without cloud dependencies, and how you validate that distilled models do not leak training data patterns in outputs. Operational maturity includes model signing, rollback, and telemetry without exfiltrating user content.