Algorithms that sort people into categories are plagued by incompatible incentives. While more accurate algorithms may address problems of statistical bias and unfairness, they cannot solve the ethical challenges that arise from incompatible incentives.

Algorithm owners are drawn into taking on the tasks of governance, managing and validating the behaviour of those who interact with their systems. The governance role offers temptations to indulge in regulatory arbitrage. If governance is left to algorithm owners, it may lead to arbitrary and restrictive controls on individual behaviour. The goal of algorithmic governance by automated decision systems, social media recommender systems, and rating systems is a mirage, retreating into the distance whenever we seem to approach it.

Tom Slee

SAP

Toward a Handbook of Ethics of AI

Centre for Ethics, University of Toronto, March 1-2, 2019