Winning at bullshit

The delightfully named Bullshit Benchmark (really the BS Detection Benchmark) by Peter Gostev is really worth looking at. Yet another benchmark you say? Another benchmark, yes, but one that addresses how LLM models respond to nonsense. Given the cocky and often sycophantic confidence exhibited by your favorite chatbot when indulging your musings, you would like to know that it doesn't say "I Love You" to everyone and everything. You want your trustworthy assistant to be... trustworthy and tells you that nonsense is nonsense without losing face.

And yes, Anthropic made all the top models for detecting BS. Which doesn't prevent some of their humans from producing a fair amount of it. Sadly, the amount of BS produced by the humans of frontier labs seems to be a contest of its own.

Vemödalen