Today I’m talking with Wout Schellaert about assessor models. Wout is a PhD student at the Polytechnic University of Valencia..

We’ll be covering a lot of different topics, such as the distributional hypothesis in machine learning, evaluation criteria, the reductive nature of current evaluation methods, task systems, the desiderata of assessor models, how to build assessor models, when to use them, what happens when our models become increasingly complex, how assessor models can help with AI explainability, whether they can help against adversarial scenarios, the challenges of scoring, bias in human evaluators, how assessor models relate to AI alignment, and other topics.

If you'd like to learn more about assessor models, visit the following pages.

José Hernández-Orallo (Wout's advisor): https://scholar.google.com/citations?user=n9AWbcAAAAAJ&hl=en

Training on the Test Set: Mapping the System-Problem Space in AI paper: https://www.aaai.org/AAAI22Papers/SMT-00432-Hernandez-OralloJ.pdf