“I think A is better” is not reproducible evaluation. Even a simple test can be rigorous: define the task in advance, standardize inputs, record results, and explain limitations.
Build test cases
For document summarization, prepare 12 de-identified texts: four ordinary documents, four with numbers or dates, and four with conflicting information. For each, list facts that must not be omitted and claims that must not be added.
Write the scorecard first
| Item | How to check | What to record |
|---|---|---|
| Factual accuracy | Compare each item with the source | Error count and severity |
| Coverage | Check the required-facts list | Covered / total items |
| Format compliance | Check fields, length, and structure | Pass or fail |
| Latency and cost | Record client time and billing | Values, time, and billing channel |
These columns are a recording template, not actual scores for any model. Give factual errors appropriate weight for important tasks; polished wording should not hide a serious mistake.
Keep conditions consistent
Use the same prompt and input, recording the exact model identifier, parameters, channel, and test date. State any parameters that cannot be aligned. Results can be random; repeat important cases and note any caching.
Run a blind review
Label answers A and B anonymously and have someone who does not know the model names score them. Read the source before scoring to avoid being influenced by length, tone, or brand. Resolve disagreements by discussing the criteria, not by simply averaging.
How should results be reported?
Publish the method, failure cases, and sample count, distinguishing measured data from judgment. A small sample supports a claim about performance on those tasks, not a claim of being “best overall.” Never invent a test or let affiliate revenue determine rankings.