LLM-judged policy debates
In many policy areas, the usual convention wisdom on policy is actually in substantial conflict with many of the specific details that experts know about that policy area. However, experts who point this out are usually accused of being politically biased. Which is why we stay stuck in this situation.
However, while LLMs usually offer the usual conventional wisdom when asked simply about policy, they also know about and understand most of those details which are in conflict with it. Furthermore, at least for now LLMs are much less likely to be accused of political bias for reacting to such conflicts. Yes, they don’t notice such conflicts unless someone points them out, and even then they don’t usually update their main models as a result of noticing conflicts.
Even so, this seems to create an opportunity for LLM-judged policy debates. Let an analysis be a <2000? word text citing expert sources/results, and let a position be a <100? word summary, perhaps including a 0-100 numerical score.
Here is a draft process.
Pick a “resolved” policy claim, two debaters who agree to debate this, and an LLM to judge.
Each of these 3 parties writes & shares an initial position.
Each debater then offers an analysis by a deadline (e.g. one day).
LLM then responds with its analysis of those analyses, and its new position.
Each debater offers a responding analysis and position by a deadline
LLM again responds with an analysis and a new position
Each debater again offers analysis and position by a deadline
LLM gives its final analysis and verdict position.
As debater analyses can just point LLMs to standard expert results, three cycles of responses seem sufficient to force the LLM to see conflicts between its initial conventional wisdom position and detailed expert knowledge that debaters point it to. So final LLM verdicts should depend less on the usual conventional wisdom, and more on the logical relation between expert knowledge and the policy claim.
Maybe also pre-train the LLM on a particular policy analysis framework (e.g., econ efficiency) which all participants agree to use. So they don’t have to relitigate that.
arent they a bit too easy to manipulate, so it would turn to be a game of magic phrases not much interesting for human spectators
Where did you get this idea that LLMs are "much less likely to be accused of political bias"? Not only are they routinely accused by all sides of bias, their inherently untraceable (by watchers) operation invites such claims because they're functionally irrefutable. Besides, their conventional training (on a "filtered internet") has a built-in double bias (on the "internet training set" and the manufacturer's filters and safeguard overrides).