Transformer Architecture Interview Guide: Attention, Scale, and Serving Implications

Transformer architecture questions in production interviews connect math to serving economics. Interviewers ask why attention is quadratic in sequence length during prefill, how KV caching changes memory footprint during decode, and why FlashAttention matters more at scale than in a notebook benchmark.

Positional encoding choices—absolute, RoPE, ALiBi—affect extrapolation beyond training context and compatibility with certain optimizations. Candidates should explain trade-offs when extending context windows, fine-tuning with rope scaling, and mixing models with different positional schemes in a routing layer.

Multi-head attention is not just a diagram label. Discuss what heads specialize in across layers, how GQA and MQA reduce KV memory for inference, and when mixture-of-experts routing introduces new failure modes like expert collapse or latency spikes on hot experts.

Encoder-decoder versus decoder-only architectures still matter for product design. Seq2seq models suit certain translation and summarization pipelines; decoder-only models dominate agentic workflows. Interviewers want architectural reasoning about where cross-attention adds value and where it adds unnecessary complexity.

Finally, connect transformer internals to debugging. When outputs degrade after a context length increase, is the issue positional extrapolation, attention sink behavior, or insufficient training on long sequences? Production engineers articulate hypotheses testable with ablations, not folklore about model magic.