Methodology
State-of-the-art research in AI evaluation says: replace the single judge with a jury of diverse models, and don't let the AI set the final score. The Tribunal was born this way.
Tribunal of Ideas uses the architecture that the AI-evaluation literature calls LLM-as-a-Jury (or PoLL, Panel of LLM evaluators): instead of a single model having the final word, a panel of diverse models evaluates criterion by criterion, and the final score is computed by a formula, outside any model's mouth.
This page explains how the trial works under the hood and why each piece of the design answers a documented bias of AI judges.
The rite
How the trial works
- 1
Admissibility
Before anything is spent, a step checks whether the submitted text is actually a judgeable idea. If the petition is inadmissible, the credit is refunded.
- 2
Real evidence
Web searches derived from the idea feed an evidence analyst, which structures real market pain points (each with a source) and named competitors. Every seat cites URLs from that list; none may invent a source.
- 3
Prosecution and defense
Three prosecution models attack the idea and three defense models respond, all required to argue over the collected evidence. The adversarial format is not just theater: without a side tasked with attacking, AI evaluators drift toward praise.
- 4
Witnesses
Six witnesses (three for, three against), each a different model embodying a persona (customer, investor, skeptical expert), testify citing the pain points and competitors found.
- 5
Multi-model jury
Twelve jurors, each a different AI model, vote on nine criteria: relevance, solution, differentiation, market, traction, monetization, timing, execution, and risk handling. Scores must anchor on the real evidence, not on the pitch's enthusiasm.
- 6
Anonymous cross-review
Each juror revisits their own vote in light of the other evaluations, presented anonymously (without knowing which model produced each one). They may keep or revise it, recording what changed and why.
- 7
Calculated verdict
The final score does not come from any model: it is the normalized average of the nine criteria, equally weighted, converted to a percentage. 80 and above, Innocent; 60 to 79, Innocent with concerns; below 60, Guilty.
- 8
Sentence and counsel brief
The judge writes the sentence with an evidence-anchored action plan, and the counsel delivers a private brief with risks and assumptions to validate. A full trial takes around 44 calls to AI models.
Second instance
The appeal: the right to respond
The first verdict measures the idea as it was presented. The appeal measures something else, more valuable: how the idea holds up after you answer the tribunal's hardest questions. That is why every trial includes one appeal, at no extra cost.
The second instance inherits the full case from the first (the evidence, prosecution, defense, and testimonies do not change) and opens with the defendant's interrogation: prosecution and defense pose questions aimed at the case's most fragile points, flagged in the counsel's brief, and you answer what you choose, with the option to skip any question. Your answers enter the record, both sides deliver rebuttals, and the same jury re-votes the nine criteria in light of the entire proceeding.
Methodologically, this treats mitigation as evidence: the risk-handling criterion, for instance, gives credit to plausible answers coming from the defendant, not only to the original pitch text. And since the score still comes from the same formula, with the same jury, the appeal is not a raise-my-score button: the verdict can go up, down, or hold, and the page shows the change.
In practice, the appeal turns the trial into an iteration loop: the brief points to where the prosecution will strike, you refine the idea or answer the tribunal, and the jury tells you whether the answer was convincing.
Why this design
Documented bias, design antidote
Research on AI judges has cataloged recurring biases. Each one gets a structural answer in the Tribunal, not a promise.
Documented bias
Self-enhancement
A model tends to rate text generated by itself, or similar to its own style, more favorably.
Design antidote
Multi-model jury: no model judges alone. Twelve jurors run on different models, and prosecution, defense, and witnesses come from yet other models.
Documented bias
Sycophancy
AI evaluators tend to agree and praise, especially when the text asks for approval.
Design antidote
Adversarial roles (a prosecution tasked with attacking) and scores anchored on real web evidence, with linked sources.
Documented bias
Position bias
The order in which answers appear changes the judgment: the first or the last tends to win.
Design antidote
Anonymous cross-review: jurors revisit their votes against the other evaluations without knowing which model produced each one.
Documented bias
Score instability
The same judge, given the same text, returns different scores across runs; a single holistic score is subjective.
Design antidote
Deterministic verdict: models vote criterion by criterion and the final score is a formula (normalized average of 9 equally weighted criteria). The AI does not set the final score.
Documented bias
One giant judge
Relying on a single large model inherits all of that model's biases, and is expensive.
Design antidote
PoLL: Cohere's research showed that a panel of smaller, diverse models outperforms a single large judge, with less intra-model bias at over 7 times lower cost.
Why trust the verdict
We don't display self-declared accuracy badges. What backs the method are the works that founded and refined the field:
- Zheng et al. (NeurIPS 2023): Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
The paper that founded the field: a strong AI judge agrees with human preference in over 80% of cases, the same agreement level as between two humans. It also cataloged the biases (position, verbosity, self-enhancement).
- Verga et al., Cohere (2024): Replacing Judges with Juries (PoLL)
Proved that a jury of several smaller, diverse models outperforms a single large judge, exhibits less intra-model bias, and costs over 7 times less.
- Arize AI: LLM-as-a-Jury
The jury pattern applied in the practice of evaluating AI systems.
- Evidently AI: LLM-as-a-Judge guide
A practical guide to the concept, its uses, and its limits.
- University of Notre Dame: Can we trust AI to judge?
Research on the 12 cataloged bias types in AI judges and what they mean for trusting the method.
Methodological honesty
These works back the method; they do not validate this specific product. A diverse jury and a deterministic verdict reduce the documented biases; they do not eliminate them. The result is an AI-generated analysis, auditable and anchored in evidence, not a certainty about your idea's future.
Disambiguation
Frequently asked questions
- Is this an AI model evaluator?
- No. LLM eval tools (DeepEval, RAGAS, LangSmith, Braintrust, Arize) use an LLM to score another LLM's output inside a testing pipeline, for AI engineers. The Tribunal uses the same technique underneath (a jury of LLMs voting on a rubric), but the object is different: your business idea, judged in a live spectacle.
- What is LLM-as-a-Judge? And LLM-as-a-Jury?
- LLM-as-a-Judge means using an AI model as an evaluator. Research showed that a single judge carries biases (position, verbosity, self-enhancement), and the field's answer was the jury: a panel of diverse models voting on a rubric, called LLM-as-a-Jury or PoLL. The Tribunal implements the jury version, with 12 models voting on 9 criteria.
- Does the final score come from an AI model?
- No. Each juror votes criterion by criterion, with a justification anchored in the collected evidence. The final score is a deterministic formula: the normalized average of the 9 criteria, equally weighted. You can audit where every point came from.
- Is the verdict infallible?
- No. The design reduces documented biases and makes the result auditable, but AI models remain subject to error and variation. Treat the trial as an informative analysis (a tough mirror for your idea), not as professional advice or a guarantee of success or failure.