Promptfoo gives AI engineers a practical way to turn prompt testing into a repeatable deployment check. Start with a YAML file, run a sample of support tickets through a prompt or model, and flag behavior that changed before it reaches production.
A sample Promptfoo configuration tests 50 support tickets. Related analysis That is enough to expose the kind of regression that slips through when a team tests one friendly example, sees a sensible answer, and ships.
Prompt changes are code changes when prompts control customer-facing behavior. A revised instruction can make an assistant more concise and less helpful. A model swap can change refusals, formatting, tool selection, or the way it handles an angry customer. Promptfoo gives you a test harness for finding those differences without turning every release into a manual review session.
The core idea is familiar to anyone who has worked with software tests. Define expected behavior, run the system against representative inputs, and treat unexpected output as a failure worth investigating. The difference is that LLM output is variable, so the expected behavior often needs to be expressed as a rule rather than one exact string.
TLDR
Promptfoo puts LLM evaluations into a YAML workflow that fits prompt changes, provider comparisons, CI checks, and red-team testing. Use representative inputs and assertions to catch regressions before deployment. The self-hosted core uses the MIT License, while hosted plans add collaboration.
Key Takeaways
- Promptfoo turns prompt regression testing into a repeatable evaluation workflow.
- YAML configurations keep test cases, providers, prompts, and assertions in one reviewable file.
- Assertions should reflect the failure that would hurt users, not cosmetic wording differences.
- CI runs make prompt changes easier to review before they affect production behavior.
- The open-source core can be self-hosted under the MIT License.
What Promptfoo Tests
Promptfoo tests the behavior of an LLM application against inputs you choose. That can mean support messages, retrieval queries, structured extraction tasks, agent instructions, or red-team prompts designed to expose unsafe behavior.
The most useful tests begin with a real failure mode. If your support assistant must avoid inventing refund policies, write an assertion around unsupported policy claims. If an extraction prompt must return valid fields, test the fields. If a routing prompt must hand billing questions to the billing workflow, test the route.
That is prompt regression testing in plain language: you save a set of inputs that represent behavior you care about, run them before and after a prompt, model, or configuration change, then inspect whether the behavior got worse.
A regression does not need to look dramatic. It may be a previously correct response that starts omitting a required detail. It may be a model that follows a prompt well on ordinary requests but becomes less reliable when a user adds irrelevant instructions. It may be an answer that still sounds polished while quietly violating a policy.
Promptfoo helps because it makes the test set durable. The inputs do not live in a spreadsheet someone forgets to open before a release. The expectations do not live in a reviewerâs head. They sit beside the prompt configuration, where the next person changing the system can see what must remain true.
The tool also works well when you need to compare providers. You may be deciding whether a model from the OpenAI API performs differently from one served through the Anthropic API across 100 GPT-4.1 test cases, which is listed at roughly $0.50 to $2.00. Related analysis The point is not to crown a universal winner. It is to see which option behaves best on your workload.
| Evaluation option | Best use | What it exposes |
|---|---|---|
| Single prompt review | Early exploration | Obvious wording problems |
| YAML evaluation suite | Repeatable prompt testing | Regressions across representative inputs |
| Provider comparison | Model selection | Differences in quality, cost, and behavior |
| CI evaluation run | Pre-deployment review | Changes that should block a release |
| Red-team suite | Safety testing | Prompt injection and policy failures |
Manual prompt review still has a place. It is useful when you are exploring a new interaction or deciding what good looks like. It becomes fragile once the system has users, recurring releases, and multiple people editing prompts. At that point, âI tried it and it looked fineâ is a weak release process.
Create a YAML Evaluation
A Promptfoo evaluation usually begins with a YAML configuration. YAML is useful here because it is readable in code review and keeps the moving parts together: providers, prompts, test variables, and assertions.
A simple support evaluation might look like this:
```yaml description: Support response evaluation
providers:
- openai:chat
prompts:
- |
You are a support assistant. Answer the customer clearly. Do not invent policy details.
tests:
- vars:
ticket: I need help with a refund. assert:
- type: not-contains
value: guaranteed refund
- vars:
ticket: Where is my order? assert:
- type: contains
value: order ```
The configuration is intentionally plain. A test provides variables, those variables flow into the prompt, and assertions check the response. You can add more cases as your product reveals edge conditions, customer language, and failure patterns.
The important work happens before you write YAML. Choose inputs that resemble the requests your system receives when the stakes are highest. A cheerful test prompt rarely tells you much. An ambiguous request, an incomplete request, a hostile request, or an instruction designed to derail the model tells you more.
Promptfooâs appeal is that teams can define hundreds of test cases in YAML. Related analysis That scale changes the workflow. You can preserve a compact test set for quick pull-request checks while keeping a broader suite for model upgrades, major prompt rewrites, or policy changes.
Keep each case focused on one behavior. If a support response fails, you should be able to tell whether it misunderstood the customer, missed a required fact, used a forbidden claim, or failed a formatting requirement. Tests that try to judge every quality dimension at once create arguments instead of useful failures.
Use names and descriptions that explain the business risk. âRefund policy hallucinationâ tells a reviewer more than âtest case alpha.â The configuration becomes part of the productâs institutional memory. Months later, it should still be obvious why a strange-looking input belongs in the suite.
A good evaluation file also makes prompt changes easier to discuss. The author can point to the behavior they intended to improve. The reviewer can see which cases guard against collateral damage. That is a better conversation than debating a handful of screenshots in a pull request.
Assertion Types and Model Comparisons
Assertions should match the kind of failure you want to catch. Exact matches are useful when an application must produce a specific token, label, or structured value. They are usually too brittle for open-ended assistant replies.
Containment assertions work for required language. You might require a response to mention a handoff channel, a policy limitation, or a specific status. Negative containment checks are useful when certain claims should never appear, such as an invented promise or restricted instruction.
Pattern checks help when output has a defined shape. A response may need to include a field name, follow a format, or return something that downstream code can parse. These tests are often more valuable than judging whether prose sounds elegant. If the application cannot consume the output, the rest of the answer does not matter.
Semantic and rubric-style checks fit behavior that cannot be reduced to one phrase. A support assistant may need to be helpful, avoid unsupported claims, and answer the customerâs question. For those cases, define the criteria in language a reviewer would accept. Vague assertions produce vague results.
Model comparisons are a separate decision. Run the same prompt and test set against each candidate provider, then look for the tradeoffs that affect your product. One model may produce stronger answers on complex requests. Another may follow formatting instructions more consistently. A local model may fit a deployment constraint that matters more than a slight quality difference.
Do not make the comparison a beauty contest. A polished response can still fail your product. If the application needs a tool call, evaluate tool selection. If it needs a safe refusal, evaluate the refusal. If it processes support tickets, use support tickets.
Promptfoo is also useful for evaluating systems that use best AI agents patterns, where a failure can happen in routing, tool use, retrieval, or the final response. Running 100 GPT-4.1 test cases is listed at roughly $0.50 to $2.00. Related analysis That makes it practical to rerun the suite when a meaningful change lands.
The review interface matters here. Side-by-side outputs make model behavior visible, but the assertions should decide what counts as passing. Humans are good at spotting odd examples. They are less good at remembering every edge case from the prior release.
Promptfoo Pricing and License
The self-hosted core is available under the MIT License. That makes Promptfoo a sensible option for teams that want evaluation infrastructure in their own environment and prefer to manage the workflow themselves.
Self-hosting is attractive when prompts, test inputs, or outputs contain sensitive material. It also gives engineering teams control over how the tool fits their existing development setup. The tradeoff is operational ownership. Someone still needs to maintain the environment, update the tool, and decide how results are stored and shared.
Promptfoo also has a cloud offering for teams that want collaboration and a managed interface. The Team plan is listed at $50 per month. Related analysis Treat that figure as one part of the evaluation cost, alongside model usage and the time required to maintain a useful test suite.
The test suite is where most of the long-term value lives. A tool can run only the cases you give it. If the cases never include the strange customer request that caused last monthâs incident, the workflow will not protect you from repeating it.
Promptfooâs 2026 page shows a 4.4/5 rating. Related analysis Ratings are a weak substitute for trying the workflow with your own prompts, but they can help frame whether the tool has enough traction to justify a closer look.
The pricing question should be simple: do you need a managed place for people to collaborate on evaluations, or do you need a local, code-owned harness that fits the tooling you already run? The license gives you the second path. The hosted plan gives you the first.
Add Evals to CI Before Deployment
A prompt change should enter CI the same way a code change does. The evaluation configuration runs against the candidate change, results are attached to the review, and failures get investigated before deployment.
This is especially valuable when the change looks harmless. Rewording an instruction, changing a system message, adjusting retrieval context, or switching a provider can all alter output behavior. The diff may be small. The product effect may not be.
Start by running a focused suite on every proposed change. Keep it fast enough that developers will use it. Reserve the broader and more expensive evaluations for changes that affect model selection, system behavior, safety rules, or core customer workflows.
Failures need a policy. Some should block deployment because they violate a hard product requirement. Others should create a review task because the output is ambiguous. If every failed assertion blocks a release, people will learn to weaken assertions. If nothing blocks a release, the test suite becomes theater.
Red teaming belongs in this workflow too. Add prompts that try to override instructions, extract hidden context, manipulate tool use, or push the model toward unsafe behavior. Red-team cases are not a one-time security exercise. They are regression tests for defenses that can weaken as prompts and models change.
The strongest teams add cases after incidents. When a user finds a bad answer, preserve the input, define what should have happened, and add it to the suite. The next release then has to prove it did not reintroduce the same mistake.
That is the practical case for Promptfoo. It gives prompt engineering a place to accumulate discipline. The hard part is deciding which behaviors deserve protection before the next change breaks them.
