Prompt Evaluation Console

The need

Prompt changes were being reviewed by intuition. The team needed a shared way to run the same cases, preserve results, and inspect regressions.

• Versioned datasets
• Comparable model settings
• Human review alongside automated scores

Implementation

Runs are immutable records tied to dataset and prompt versions. A worker pool executes cases and streams partial results to the comparison interface.

• Provider-neutral adapters
• Retry-aware execution
• Segment-level score breakdowns

Result

Prompt review became an engineering workflow with evidence instead of a collection of screenshots and anecdotes.