The 2019 Microsoft paper introduced UNILM, and it never really took off because GPT2 followed through without the need of any encoder and GPT3 pushed forward further showing scale gives generalization. It's still an interesting historical footnote. UNILM is a new unified language model designed by Microsoft researchers to handle both natural language understanding and generation. Unlike previous models like BERT, which focus primarily on understanding, this system employs a shared Transformer network with diverse self-attention masks. These specialized masks allow the model to learn from unidirectional, bidirectional, and sequence-to-sequence perspectives simultaneously. This versatile architecture enables it to excel at various tasks, ranging from abstractive summarization and dialogue response to complex question answering. Experiments demonstrate that UNILM achieves state-of-the-art results across multiple benchmarks, proving its effectiveness as a flexible tool for various linguistic applications. By integrating these different objectives into one framework, the researchers have created a more efficient and generalizable model for the field of artificial intelligence. Source: May 2019 Unified Language Model Pre-training for Natural Language Understanding and Generation Microsoft Research Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon https://arxiv.org/pdf/1905.03197