AI lifecycle · rule ai-llm-without-evaluation
LLM apps with no evaluation suite
A Metacenta review checks this under the rule Model output is evaluated, not just generated. Everything below applies whether or not you ever commission one.
What this rule checks
Where a project declares a language-model SDK, this rule checks that an evaluation framework such as promptfoo, Ragas or DeepEval is declared too. It reports once for the project.
Why it matters
A prompt change is a code change whose effect is invisible to every test you have. Without an evaluation suite, the first sign that answers got worse is a customer noticing.
How to fix it
Evaluate model output, not only generate it. Start with a few dozen recorded inputs, each with the properties a good answer must have, and run them on every change. Having a baseline matters more than which framework runs it.
When it is fine to leave
An evaluation suite in a separate repository, or a harness written in-house, does the job without appearing in this manifest. We report this at medium confidence for that reason.
What we need to check it
The dependency manifests in your repository. We reason from what they declare, so the finding is medium confidence at most.