The October 20, 2025 Meta FAIR paper introduces the Free Transformer, an innovative extension of the decoder-only Transformer architecture, which addresses the limitations of purely autoregressive language modeling by integrating random latent variables into the generative process. This new model is structured as a conditional Variational Autoencoder (VAE), where an encoder learns the latent variables unsupervised, and a decoder conditions its token generation on these variables. The implementation requires only a minor computational overhead due to sharing half of the decoder's blocks with the encoder. Experimental results with 1.5B and 8B parameter models demonstrate that this conditioning leads to substantial performance improvements on reasoning and coding benchmarks like HumanEval+ and GSM8K. The authors conclude that the Free Transformer significantly improves the inductive bias of the vanilla Transformer. Source: https://arxiv.org/pdf/2510.17558v1