A panel can pass every diversity check we run (different ages, different income tiers, a genuine skeptic in the mix) and still fail in a way that matters more: every persona gives you functionally the same answer to the one question you actually asked. Demographic diversity and opinion diversity are not the same thing, and it's entirely possible to build convincing personas that still collapse into a single voice the moment they're asked something concrete.
This is a narrower, later-stage version of the mode-collapse problem we wrote about in the persona-generation post. That one is about whether the panel itself is built from a real distribution. This one is about whether the answers, once given, actually disagree, because a panel can be built correctly and still converge on one opinion for a specific question, especially an easy or leading one.
The check itself is simple on purpose: after a question is answered, every response gets embedded and compared pairwise using cosine similarity. We average the pairwise scores across the whole panel and flag anything above roughly 92% as functionally identical, regardless of how differently worded the answers are on the surface. Paraphrasing doesn't fool it, because embeddings capture meaning, not exact wording.
We picked cosine similarity over something more elaborate for a specific reason:
“It's checkable.
You can look at two answers, read them, and see whether a 94% similarity score matches your own read of "these are basically the same point." A fancier metric might score marginally better on some benchmark, but if we can't explain why a given panel got flagged, the number becomes something you're asked to trust instead of something you can verify, which is exactly the failure mode this whole methodology exists to avoid.
The threshold itself, 92%, is a judgment call, not a law of nature, set from looking at enough real panel output to find where genuinely converged answers start clustering versus where personas are still recognizably disagreeing even when they land on a similar conclusion. It'll probably move as we see more real usage, and if it does, we'll say so here rather than quietly changing the number.
“What this doesn't do: prove that disagreement is the right outcome.
Sometimes a panel should agree. If you ask five accountants whether they'd want invoices to stop getting lost, unanimous yes is a reasonable answer, not a red flag. The check tells you the panel isn't collapsing into one voice by default; it doesn't tell you whether agreement, when it happens, is real consensus or a panel that's quietly too agreeable. That distinction is still yours to make, which is exactly why the number is shown, not hidden behind a pass/fail badge.