Today we’re speaking with José Hernández-Orallo. José is a Professor at the Polytechnic University of València in Spain and a Senior Research Fellow at the Leverhulme Centre for the Future of Intelligence, at Cambridge.

We'll be covering an enormous amount of ground surrounding intelligence and its evaluation.

We’ll touch on topics such as operating conditions in ML, agent characteristic curves,, the challenge with average performance scores, task-oriented evaluation, capability-oriented evaluation - in humans and other animals, animal-AI Olympics, meta-data annotation, performance robustness as a function of latent attributes, the limitations of aggregating behavioral performance into a handful of metrics, whether intelligence is meaningless or all-explanatory, the concept of generality in intelligence, AGI and the problem of distributions of tasks, calibration of task difficulties, whether capability metrics are incommensurable, complexity and compressibility, compression-based algorithmic information theory, the limits of intelligence, the intelligence of hybrid systems, whether we’re ready to talk about consciousness given our limited understanding of intelligence, whether rights should be tethered to pain or to cognition, animal rights, and what the AutoML community can do to help with this line of work.

If you're interested in this line of work, here is a recent paper by José and his team: https://ryanburnell.com/wp-content/uploads/Burnell-et-al-2022-Not-a-Number.pdf

His book, The Measure of All Minds, can be found here: https://www.amazon.com/Measure-All-Minds-Evaluating-Intelligence/dp/1107153018