Sign up to save your podcasts
Or
Share NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
Pretrained language models have become the standard approach for many NLP tasks due to strong performance, but they are very expensive to train. We propose a simple and efficient learning framework TLM that does not rely on large-scale pretraining1. Given some labeled task data and a large general corpus, TLM uses task data as queries to retrieve a tiny subset of the general corpus and jointly optimizes the task objective and the language modeling objective from scratch.
2021: Xingcheng Yao, Yanan Zheng, Xiaocong Yang, Zhilin Yang
https://arxiv.org/pdf/2111.04130v1.pdf
View all episodes
By Rob
3.7
33 ratings
Pretrained language models have become the standard approach for many NLP tasks due to strong performance, but they are very expensive to train. We propose a simple and efficient learning framework TLM that does not rely on large-scale pretraining1. Given some labeled task data and a large general corpus, TLM uses task data as queries to retrieve a tiny subset of the general corpus and jointly optimizes the task objective and the language modeling objective from scratch.
2021: Xingcheng Yao, Yanan Zheng, Xiaocong Yang, Zhilin Yang
https://arxiv.org/pdf/2111.04130v1.pdf