Sign up to save your podcasts
Or
Share Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Ref: https://arxiv.org/abs/1901.02860
The
paper introduces Transformer-XL, a novel neural architecture for
language modeling that overcomes the limitations of fixed-length
contexts in standard Transformer models. It achieves this through a
segment-level recurrence mechanism and a novel relative positional
encoding scheme, enabling the capture of significantly longer-term
dependencies. The resulting model demonstrates state-of-the-art
performance on various language modeling benchmarks, exhibiting
substantial speed improvements during evaluation and the ability to
generate coherent long-form text. The authors present experimental
results and ablation studies validating the effectiveness of their
proposed techniques. They also offer insights into the attention
mechanisms of the model.
View all episodes
By KnowledgeDBRef: https://arxiv.org/abs/1901.02860
The
paper introduces Transformer-XL, a novel neural architecture for
language modeling that overcomes the limitations of fixed-length
contexts in standard Transformer models. It achieves this through a
segment-level recurrence mechanism and a novel relative positional
encoding scheme, enabling the capture of significantly longer-term
dependencies. The resulting model demonstrates state-of-the-art
performance on various language modeling benchmarks, exhibiting
substantial speed improvements during evaluation and the ability to
generate coherent long-form text. The authors present experimental
results and ablation studies validating the effectiveness of their
proposed techniques. They also offer insights into the attention
mechanisms of the model.