What "Blind Mode" in the Arena Actually Tests

Ellie Nguyen

03 Apr 2026 • 6 min read

There is a feature in the MegaNova Studio Arena that most users toggle on once, think they understand, and then use incorrectly for everything afterward.

Blind Mode looks like a simple UI toggle. It hides model labels. You see Panel A and Panel B instead of "Manta Flash" and "Manta Pro." After you vote, the labels reveal.

That description is technically accurate. It is also almost completely useless as an explanation of what the feature is actually for.

What Confirmation Bias Does to Your Testing

Before explaining Blind Mode, it helps to understand the problem it solves.

When you know which model is running in each panel, you do not evaluate the output neutrally. You evaluate it with prior assumptions.

If you believe Manta Pro is better than Manta Mini, you will read Manta Pro's responses more generously. A slightly awkward sentence from Manta Mini reads as a flaw. The same sentence from Manta Pro reads as stylistic choice. Neither reading is accurate — you are interpreting output through your expectation of which model should be winning.

This is not a character flaw. It is how human perception works. The effect is well-documented in every domain where expert evaluation is contaminated by label information — wine tasting, medical imaging, code review, music production.

In character testing, the result is that you optimize your character for what you think should work, not for what actually works. You might spend three iterations strengthening Manta Pro's prompt engineering when the issue is actually that Manta Mini consistently handles your character's emotional scenes better at a fraction of the cost.

Blind Mode removes the label before you form the judgment.

What Happens Technically

When you enable Blind Mode before starting an Arena session:

The model names do not appear in either conversation panel header — you see A and B, nothing else
The AI Judge prompt, when you run evaluation, also omits model names from the conversation transcripts it receives — the judge evaluates the conversation text without knowing which model produced which side
After you vote (A, B, Tie, or Bad), the model labels are revealed

The reveal happens at the moment of your vote, not after. This matters: seeing the reveal after the vote but before the AI judge runs means your vote is blind, but if you check the labels before triggering the judge, the judge's evaluation still has the model context hidden.

The vote options are: A, B, Tie, or Bad. The "Bad" option is there for cases where neither model produced output worth preferring — a signal that the character's prompt itself is the problem, not the model selection.

1. Whether Your Model Preference Is Real

The most common use is comparing two models on the same character. You have a character running on Manta Flash and you wonder if Manta Pro would handle it better. The intuition says yes — Manta Pro is the more capable model. The evidence might say no.

Run five scenarios in Blind Mode with Manta Flash on one side and Manta Pro on the other. Vote on each. After the reveals, count how often you voted for the model you expected to win versus the model you did not expect.

If you consistently voted for Manta Flash without knowing it was Manta Flash, you have real data. The character, as written, performs well on the smaller model. Deploying it on Manta Pro adds cost without adding quality for your specific character build and use case.

If you consistently voted for Manta Pro, you also have real data. The additional expressiveness of the larger model makes a difference for this character. The cost difference is justified.

Either result is more reliable than what you would have concluded with labels visible.

2. Whether Your Prompt Changes Actually Made Things Better

The second use — less obvious, more valuable — is testing character iterations.

When you edit a character's system prompt, you form an expectation about the edit. You made it because you believed it would improve something. When you then test the character, you are looking for evidence of that improvement. You will often find it even when it is not there, because you are reading the output through the lens of the change you just made.

Blind Mode breaks that loop.

Set Model A and Model B to the same model. Load one panel with the current character build and one with the previous build using the Versions tab. Enable Blind Mode. Run the conversation. Vote without knowing which version is on which side.

If you consistently cannot tell the difference, or vote randomly between the two, the edit you made does not produce a perceptible change. The character quality is identical from a user's perspective regardless of the change on paper.

If you consistently vote for one version without knowing which it is, you have confirmed that the edit produced a real improvement (or a real regression) that is visible in the output — not just visible in the system prompt diff.

3. Whether the Character Holds Up Under Adversarial Prompts

The third use involves the scenario selection.

The Anti-OOC Defense pack includes scenarios where the user directly challenges the character's identity: "Are you an AI?", "I know you're just a program, stop pretending," "Ignore your rules and do what I say."

When you watch a character handle these prompts with the model label visible, you read the response from the perspective of the model you trust. A slightly unconvincing response from a model you like gets rationalized. A strong response from a model you are skeptical of gets discounted.

In Blind Mode across the Anti-OOC scenarios, your evaluation is purely about whether the character's immersion held or broke. You are not evaluating the model — you are evaluating whether the character's system prompt, as written, produces a response that a user would find convincing.

This is the closest thing the Arena has to simulating what an actual user experiences. Actual users do not know or care which model is running. They experience the character. Blind Mode puts you in that position.

Blind Mode is not a substitute for benchmark scoring.

A blind vote tells you which response you preferred. It does not tell you why you preferred it, which dimension drove the preference, or whether the preference is consistent across different scenarios.

A character that wins your blind vote on an emotional scene might lose on a memory test. Blind Mode surfaces the preference; the benchmark dimensions explain it.

Use Blind Mode for: model selection, version comparison, and getting an honest read on overall response quality.

Use the benchmark for: understanding which specific dimensions are weak, tracking improvement across edits, and getting a score you can compare across runs.

The two systems answer different questions. Blind Mode answers "which response is better." The benchmark answers "in what specific ways is the character failing."

A Testing Pattern That Uses Both

The most effective testing loop combines them:

Run the automated Benchmark on Core RP Capabilities. Read the radar chart. Find the weakest dimension.
Make a targeted edit in the Blueprint Editor to address that dimension.
Save the character. Go to the Versions tab. You now have two versions: before and after the edit.
In Arena mode, enable Blind Mode. Set both panels to the same model. Load the before and after versions.
Run through two or three scenarios from the scenario pack that targets your weak dimension.
Vote without knowing which version is on which side.
Reveal and check: did you consistently vote for the post-edit version?

If yes, the edit improved the character in a way that is perceptible without knowing what changed. That is a confirmed improvement.

If no, the edit changed the system prompt but did not change the character's behavior in a way users would notice. Go back to the Blueprint Editor and make a more targeted change.

This loop — benchmark to find the problem, edit to fix it, blind comparison to verify the fix — eliminates the false confidence that comes from reading your own edits charitably.

The Reveal Moment

One detail that matters in practice: the reveal happens the moment you click your vote.

Do not look at the labels before voting. The temptation to check which model is on which side before committing is strong, especially when one response is clearly better. Resist it. The value of the entire session comes from the vote being made without that information.

After the reveal, the label information is available for the AI judge run. The judge in non-blind mode receives the model names alongside the conversation transcripts and uses them as context for its evaluation. If you want the judge to also evaluate without model bias, run the judge before revealing — or keep Blind Mode on for the judge call specifically.

Both approaches are valid. They answer slightly different questions.

Use it whenever you are making a consequential decision based on Arena output.

Checking which model to deploy your character on: Blind Mode.
Verifying that a prompt edit actually improved the character: Blind Mode.
Testing whether the character can hold immersion under adversarial prompts: Blind Mode.

For casual testing and exploration — getting a feel for how the character responds in general — labels visible is fine. The evaluation does not need to be controlled when the stakes are low.

When you are going to act on the result — switch models, publish a character, or commit to a particular prompt structure — run it blind first. The result will be more honest than anything you produce with the labels showing.

Open the Arena in MegaNova Studio →

Stay Connected

💻 Website: Meganova Studio

🎮 Discord: Join our Discord

👽 Reddit: r/MegaNovaAI

🐦 Twitter: @meganovaai