DEEP EXPLANATION

LLM Evaluation Metrics: BLEU, ROUGE, BERTScore, and Why They Fail (ANSWERED)

Scenario BasedLLMsMedium15 min read

Scenario question on classical NLP metrics — what they measure, where they break on paraphrase and factuality, and what to use instead.

TL;DR — Quick Answer

BLEU measures n-gram overlap with references (translation legacy); ROUGE measures recall of reference n-grams (summarization); BERTScore uses contextual embedding similarity. They correlate poorly with human judgment on open-ended generation because valid paraphrases score low, factual errors with fluent text score high, and single-reference datasets are brittle. Modern eval uses human rubrics, LLM-as-judge, task-specific checks (exact match, JSON schema), and faithfulness metrics for RAG — not BLEU alone.

The Interview Question

Explain BLEU, ROUGE, and BERTScore for LLM evaluation. Why do these metrics fail for modern generative AI applications?

Deep Explanation

Get deep explanations, PDF export & all LLMs questions

10 more sections of deep explanation
Real-world examples
Common mistakes
Interviewer expectations
Follow-up questions

EvaluationBLEUROUGEBERTScoreMetricsGoogleMetaMicrosoft

Up next

Next Question

LLM-as-Judge: Designing Reliable Automated Evaluation (EXPLAINED)

Hard project question on LLM-as-judge — rubric design, position bias, human calibration, and production eval pipelines.

Continue