The paper proposes a training method for transformers using int4 arithmetic, achieving competitive accuracy on various tasks, and can be implemented on current GPUs, speeding up training by up to 35.1%.