Deep explanation
Production incident: Google Gemini outage in a compliance review automation system
Junior incident question: handle Gemini 429/503 errors and a cost spike in a compliance review pipeline with triage, retry control, queueing, fallback, and quota planning.
Quick answer
First I stop the damage: turn off aggressive retries so we stop paying for failed attempts and stop making the overload worse, and pause or queue non-urgent reviews. Then I check whether the errors are our quota (429) or the service being unavailable (503), and look at token usage to see why cost doubled. After recovery I add proper backoff, a queue, quota alerts, and a fallback region or model, and reprocess the stuck documents safely.
The interview question
At peak traffic, your compliance review system starts failing requests to Google Gemini on Vertex AI: you see many 429 RESOURCE_EXHAUSTED and some 503 errors, the bill for the day is already double normal, and reviewers are filing tickets about stuck documents. What do you do in the first 30 minutes, and how do you recover and prevent it next time?
Deep explanation
Simple explanation
Sign in to unlock the full answer
Free accounts include 5 full deep answers. Sign in to start unlocking.
- 16 more sections of deep explanation
- Real-world examples
- Common mistakes
- Interviewer expectations
- Follow-up questions
Model ProvidersCommercial ProvidersGoogle GeminiJuniorProduction Incident