LibraOS Benchmark Results: How to Read Them

Tracy Giang

27 Oct 2026 • 3 min read

An AI runtime sits in the path of every request your app sends, so its own overhead matters. LibraOS publishes a controlled benchmark on its homepage: 244 runs per second at 256 concurrent runs, about 3.8 times Agno in the Minimal Runtime test, with 52 MB of idle memory and 10.9 ms of CPU per run.

Those numbers are easy to quote and easy to misread. This post explains what was measured, the conditions that come with the results, what they tell you about LibraOS, and how to test it on your own workload.

What was measured

The benchmark on the LibraOS homepage uses a mock model, so it measures the runtime itself rather than any model's speed.

Measure	LibraOS result	Test setup
Throughput	244 runs/s	256 concurrent runs; about 3.8× Agno in the Minimal Runtime test
Idle memory	52 MB RSS	LibraOS v0.2.2, fresh database
CPU per run	10.9 ms	Sequential mock-model test

In the same mock-model test at 256 concurrency, the three runtimes compared like this:

Runtime	Runs per second
LibraOS	244
Agno	64
LangGraph	8

Three terms are worth defining. Throughput is how many complete runs the runtime finishes per second. RSS (resident set size) is the memory the process actually holds. CPU per run is the processor time the runtime spends on one run, measured here one run at a time.

The conditions that come with the numbers

The homepage states its conditions next to the results, and they belong with the numbers every time they're quoted:

It's a controlled mock-model test. No real inference runs in the loop.
LangGraph was run with langgraph dev, and polling affects its throughput.
The results are not a production claim. In the page's own words: "These results do not establish production deployment superiority."

The homepage also compares Minimal and Production configurations, and shows peak memory and the comparator measurements. Read those alongside the headline figures before drawing conclusions.

What the numbers tell you

The layer itself is light. Low CPU per run means the runtime adds little processing to each request.
It holds up under concurrency. Throughput at 256 concurrent runs shows how the runtime handles many agents working at once, which is common when an app serves many users.
It fits on modest hardware. 52 MB of idle memory leaves room for the rest of your stack, which matters when you self-host or run next to local models.

What they don't tell you

How fast your app will feel. With a mock model, there's no inference in the loop. In production, the model you choose, knowledge retrieval and the tools an agent calls all add time this test leaves out.
Which runtime will win in your deployment. A controlled test isolates one factor. Your setup, configuration and workload decide the real result.
How it behaves with your features turned on. Memory, knowledge, approvals and screening each do work. The Minimal and Production comparison on the homepage is the place to see how configuration changes the picture.

How to run your own test

The published benchmark tells you where to start. Your own test tells you what to expect.

Use your real model endpoint, not a mock: your provider key, your local model server, or hosted inference.
Send your real traffic: the prompts, knowledge lookups and tool calls your app actually makes.
Turn on the features you'll use: memory, knowledge, approval points and screening.
Match your expected peak concurrency, then go a little beyond it.
Compare against your current setup, on the same hardware and with the same model.

What to measure:

Measure	Why it matters
End-to-end latency (median and slowest 5%)	What users actually wait for
Throughput at peak	Whether you can serve everyone at once
Memory at idle and at peak	How big a server you need
Error and timeout rate	Whether runs finish under load

Run each test more than once and keep the conditions the same, so you compare like with like.

Conclusion

LibraOS's published benchmark shows a light runtime: 244 runs per second at 256 concurrent runs, 52 MB idle memory and 10.9 ms CPU per run, in a controlled mock-model test. It measures the runtime's own overhead, not model speed, and in the page's words it does not establish production deployment superiority. Use it as a starting point, then test with your own models, traffic and features.