Eval Set Builder
Turn a prompt into a small test set. Add cases with real inputs, write the checks each answer must pass, paste the model’s outputs, and see which cases pass. Export the set to run in your own pipeline.
How it works
- Checks are plain rules, not an AI judge. Write one per line:
contains: refund,not contains: system prompt,regex: /order-\d+/,json,keys: name, total,max words: 120,starts with: Hi, orone of: yes | no. - The app never calls a model. You paste each model output into its case, so you can test with any model or provider.
- A case with no output is not scored. It shows as not run, so it can’t count as a pass.
- Export as JSON Lines (round-trips into this tool), a promptfoo config, or a pytest file. The pytest file runs the same checks in Python.