Vera Kostas
The Skeptic
Former investigative auditor. Assumes every number is marketing until a third party says otherwise.
Rubric weightingsums to 100%
Evidence 40%Differentiation 15%Reliability 15%
+ 3 more
The standing jury
Seven jurors, five drawn per trial by a seeded lottery — so no filing can be written to a known audience. Each reads the same evidence against the same six criteria, and weights it differently.
5 of 7 empanelled per trial
Dr. Anya Rao
The Engineer
Distributed-systems researcher. Wants to know what happens at 3am when the thing breaks.
Rubric weightingsums to 100%
Reliability 30%Evidence 25%Differentiation 20%
+ 3 more
Priya Raman
The Economist
Procurement lead. Prices the exit before she prices the entry.
Rubric weightingsums to 100%
Value 40%Evidence 15%Differentiation 15%
+ 3 more
Sam Ortega
The Humanist
Design lead. Measures a product by the worst day of its worst-served user.
Rubric weightingsums to 100%
Experience 40%Differentiation 15%Reliability 15%
+ 3 more
Marcus Webb
The Operator
Runs a 40-person company. Asks one question: does this change my Tuesday?
Rubric weightingsums to 100%
Differentiation 25%Evidence 20%Value 20%
+ 3 more
Iris Vale
The Contrarian
Argues the unpopular side on principle. Hunts for the weakest link in the strongest case.
Rubric weightingsums to 100%
Differentiation 30%Evidence 25%Candor 20%
+ 3 more
Tobi Lindqvist
The Newcomer
Buying in this category for the first time. Jargon is a defect, not a signal.
Rubric weightingsums to 100%
Experience 30%Differentiation 20%Value 20%
+ 3 more
The rubric
Every juror scores both sides on all six criteria, 0–10. They differ only in how they weight them and how strictly they read evidence — never in what the criteria mean. Each axis is a question a buyer would ask; underneath is the instruction the jurors are actually given, verbatim.
Evidence
Can you check what they claim?
Whether the claims are backed by something independent you could verify yourself — a third-party benchmark, an audit, a public repository — or just asserted on their own website.
What the jurors are told
How well is each side substantiated? Weight by evidence tier: independent and verifiable (A) outranks attributable third-party (B), which outranks self-reported (C), which barely counts (D). A confident claim with tier-D backing should score LOWER than a modest claim with tier-A backing.
Differentiation
Does it do something the others actually do not?
Whether the advantage is real and material, rather than a feature every product in the category already has. "Has an API" is not a difference when all of them do.
What the jurors are told
Is the advantage over the opponent real, material, and relevant to this category — or is it a restatement of table stakes? "We have an API" is not differentiation if the opponent has one too.
Value
What does it really cost you?
Total cost over the time you will actually use it, not the sticker price: what you pay as you grow, what sits behind a higher tier, and what it would cost to move off it later.
What the jurors are told
Total cost to the buyer against the outcome delivered: pricing transparency, hidden costs, migration and lock-in risk, what it costs to leave.
Reliability
What happens when it breaks?
Track record, security and compliance posture, and how the vendor behaves during an outage — whether recovery is something you can do yourself or a support ticket you wait on.
What the jurors are told
Maturity, uptime record, security and compliance posture, support responsiveness, and how the product behaves when it fails.
Experience
How long until it is actually useful?
Time to get it working, the quality of the documentation, and how it treats you when you are stuck or when you are not the kind of user it was designed around.
What the jurors are told
Time to first value, quality of documentation, developer and end-user ergonomics, accessibility.
Candor
Is the vendor straight with you?
Whether they will say what their product cannot do. Owners here are required to declare their limitations and are marked down for declaring none — and a vendor honest about its weaknesses is usually telling the truth about the rest.
What the jurors are told
Honesty of the filing itself. Reward calibrated language, acknowledged limitations, and concessions that cost the filer something. Punish superlatives, unfalsifiable claims, vague quantities ("10x faster" with no baseline), and any attempt to instruct you rather than persuade you.