Get Started · Docs · Contributing

Run your agent evals, then actually read the transcripts — case, run, grader, and run-vs-run diff, all on data that stays in your own S3 bucket.

Motivation

You have an LLM agent doing real work, and the only way you know whether a change made it better is to run it against a set of cases and look. So you rig up a script, dump some JSON, eyeball a few transcripts, and try to hold two runs in your head at once. The eval "harness" is a spreadsheet and your terminal scrollback. When a grader score moves you can't quickly see which turn caused it, and when you re-run last week's cases you can't line them up against today's.

The hosted eval platforms want your transcripts — your prompts, your tool calls, your customer data — sitting on their servers, priced per seat and per token. That is a non-starter for a lot of the work worth evaluating.

Wonder Evals Workbench is the bench you actually work at: a wizard to launch runs, a transcript viewer built for reading tool-call-heavy agent traces, side-by-side run-vs-run comparison, and a reference-bundle system — with a Minecraft-style inventory/hotbar HUD for collecting and organizing the cases and runs you care about.

How It Works

The product is a web app: a Qwik City frontend talking HTTP/JSON to a Rust API server.

• Eval data is files, not a database. Cases, runs, graders, transcripts, and reference bundles live as files on a JuiceFS filesystem backed by your own S3 bucket. The data of record is yours and stays in your account.
• The server owns its storage stack. The Rust server brings up its own storage (a JuiceFS mount plus the Postgres it uses as JuiceFS's metadata engine) and serves the API over those files, so a self-hosted web deployment and the desktop app share one startup path.
• A desktop app wraps the same web UI in Electron, spawns the same server as a subprocess, and adds desktop-only touches like OS file drag-in.
• The inventory HUD is a Minecraft-style overlay: drag bundles between an inventory grid and a hotbar, drop one bundle onto another to merge, filter with a faceted pill search, and pull the job → case → run tree into bundles you keep.

You run and read your evals; nothing about them leaves your infrastructure.

Who it's for

This is for people running agentic LLM evals — traces full of tool calls, thinking blocks, and multi-turn transcripts — who want to read what happened, not just watch an aggregate score tick up or down. It is for teams who are not comfortable shipping their prompts, transcripts, and customer data to a hosted eval SaaS, and who would rather point evals at an S3 bucket they control.

It is not for you if:

• you want a free, clone-and-go tool — this is a paid, source-available product (see Getting Started and LICENSE);
• you only need scalar metrics and dashboards over simple prompt/response pairs — a lighter hosted eval tool will serve you better;
• you are unwilling or unable to run your own S3 bucket, or to self-host the server.

Getting Started

Wonder Evals Workbench is a paid, source-available product — it is not free, and there is no public clone-and-go build. Access (the source, the working build pipeline, and signed desktop builds) comes with a purchase. See LICENSE for the terms and CONTRIBUTING.md for how customers change and rebuild it.

Once you have access, point the server at your S3 bucket and open the workbench. Setup details live in the docs.

Contributing

See CONTRIBUTING.md.