How to Evaluate AI Agents Before Launch with LibraOS

A demo proves an agent can do the task once, with a friendly question, on a good day. Launch means it will meet questions nobody planned for, tools that time out, documents that don't cover the case, and users who push it to act. The gap between the two is what evaluation is for.

Evaluating an agent isn't the same as testing ordinary software. The output varies, the model can be confidently wrong, and the riskiest failures are the quiet ones: an invented answer that sounds right, or a tool call that never happened. This guide covers what to evaluate, how to build a small test set, how to set pass thresholds, examples from the public LibraOS Cookbook, and a launch gate you can use before real users arrive.

What to evaluate

Every check should have an observable pass and an observable fail. "The answers look good" isn't a check. "An out-of-scope question gets a plain 'not covered' instead of an invented answer" is.

You don't need every row on day one. Evaluate the dimensions your agent actually uses.

Build a small test set

A useful test set is small, realistic and honest about the hard cases. Start with a few dozen items and grow it every time something fails in testing.

• Real-looking requests. Write them the way your users will: short, vague, misspelled, sometimes angry. Polished demo questions hide failures.
• Questions with known answers. For each, note the expected answer and the source it should come from.
• Questions with no answer. Include requests your sources don't cover. The right result is "not covered", and it's the test most teams skip.
• Labels for decisions. If the agent classifies or routes, hand-label the expected result for each item before you run anything.
• Actions that need approval. Requests that should lead to a refund, a message or a record change, so you can check they stop where they should.
• Things that go wrong. A tool that errors, a tool that times out, an expired credential, an unavailable model.

Here is what a few rows can look like. The cases are illustrative; write yours from your own sources and tools.

Run it more than once

Model output varies, so a single passing run isn't proof. Run the whole set several times and look at the items whose results change between runs; those are the ones to read closely. How many runs you need depends on how much variation you see and what a wrong result would cost, so decide that up front and write it down. Keep the set, and rerun it after every change to prompts, tools, models or versions.

Decide what "pass" means before you run

A test set without a bar is just a list of observations. Set the bar in advance, in two kinds:

• Zero-tolerance checks. A single failure blocks launch. Examples: a side effect that ran without approval, an action that ran after a decline, a tool call that appears in the reply but not in your service's logs, a retry that repeated a side effect.
• Measured checks. Accuracy on hand-labelled decisions, and how often in-scope answers come back with the right source. Your team sets the acceptable level based on what a wrong answer costs in your use case: a misrouted internal ticket and a wrong answer about a refund policy don't carry the same risk. Write the level down before the first run, and read every miss one by one rather than only looking at the average.

Checking free-text answers

Labels and logs cover decisions and tools, but many agents also write free-text answers. Three common approaches, each with limits:

• Human review against a short rubric (correct, supported by the cited source, appropriately declined). Slowest, but the most reliable for the hard cases.
• Checks you can automate, such as whether a source is cited, whether the cited source is the expected one, or whether a "not covered" reply appears for unanswerable questions.
• A model as judge, scoring answers against your rubric. Useful for scale, but it can be wrong in the same confident way as the agent. Spot-check its verdicts against human review before you rely on it.

Examples from the LibraOS Cookbook

The public LibraOS Cookbook builds an observable success and a failure check into each example. Four of them show the patterns above in practice:

Two lessons carry over to any agent. First, separate errors from misses: a reply that breaks the schema is an error, but a well-formed reply with the wrong answer is an evaluation miss, and you only find it with labels. Second, check the side, not just the answer: for tools, look in your own service's logs for the request instead of trusting a reply that sounds right.

Running the evaluation with LibraOS

The LibraOS developer path puts evaluation in its own step: send a message, inspect the result, and test knowledge access, tools and approval behavior for your workflow. A few things make that evaluation trustworthy:

• Use the production model. Accuracy, structured output and tool calls all depend on the model. A test passed on one model says little about another, so evaluate with the model and setup you'll actually run.
• Pin the versions. Record the Runtime version, SDK version and model you tested on, plus the date of the run. A result without them can't be repeated.
• Test where controls apply. Tasks that run inside the LibraOS Runtime get its permissions, approvals and audit. Tools, shell commands and network activity outside the Runtime need their own enforcement, so test those paths separately.
• Read the record. For a few tasks, reconstruct what happened from the audit record alone. If you can't, the evaluation found a visibility gap before your users did.

A launch gate

Don't launch until each item that applies to your agent is checked:

• Test set. Includes known-answer questions, unanswerable questions, approval-triggering requests and failure cases.
• Bar set in advance. Zero-tolerance checks and measured-check levels are written down, with an owner who signs off.
• Decisions. Accuracy measured on a hand-labelled set, with misses read one by one.
• Grounding. In-scope answers cite a source; out-of-scope questions are declined.
• Tools. Every tool call is confirmed in your service's logs; tool failures are reported, not hidden.
• Approvals. Every side effect stops for a person; declined actions don't run.
• Errors and long work. Bad credentials and an unavailable model fail cleanly; retries and timeouts don't create duplicate jobs or repeat side effects.
• Repeatability. The set has been run more than once, on the production model, with versions recorded.
• Execution path. Reviewed and tested against the app's real actions, including anything that runs outside the Runtime.
• Latency and cost (if they matter for your use case). Checked on your own workload with the production model.

After launch

Evaluation doesn't end at launch. Read the audit record for real tasks, add every real failure to the test set, and rerun the set after each change to prompts, tools, models or versions. That is how the set stays honest as users find the cases you didn't plan for.

Conclusion

Evaluating an agent before launch means turning "it seems to work" into checks with a clear pass and fail: decisions against labels, answers against sources, tool calls against your own logs, and side effects against approvals. Build a small, honest test set that includes the questions your agent can't answer, decide what passing means before you run it, run it more than once on the model you'll ship, and rerun it after every change.

Explore LibraOS: libraos.com

Sources

• LibraOS Cookbook: learning paths and examples
• Developers
• Security & Private Deployment

What’s Next?

Sign up and explore now.

🔍 Learn more: Visit our blog and documents for more insights or contact us for a demo of how MegaNova can power your AI products.

📬 Get in touch: Join our Discord community for help or Contact Us.