A deep dive into GShard, a module for scaling giant neural networks, focusing on its application to multilingual machine translation and its impact on training efficiency and model quality.