"ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators"

Core Concept The paper introduces ELECTRA ("Efficiently Learning an Encoder that Classifies Token Replacements Accurately"), a method designed to improve the computational efficiency of pre-training language models. The authors argue that existing state-of-the-art methods like BERT are inefficient because they use Masked Language Modeling (MLM), where the model only learns from the small subset of tokens (approx. 15%) that are masked out.

Methodology Instead of masking tokens, ELECTRA employs a task called replaced token detection. This approach uses two networks:

1. A Generator: A small network that corrupts the input by replacing some tokens with plausible synthetic alternatives.

2. A Discriminator: The main model is trained to distinguish whether each token in the sequence is an original input or a replacement generated by the first network.

Key Results Because the discriminator learns from all input tokens rather than just a masked subset, ELECTRA is significantly more sample-efficient than its predecessors.

• Performance: ELECTRA substantially outperforms BERT when given the same compute budget, model size, and data.

• Efficiency: An ELECTRA-Small model trained on a single GPU for 4 days outperformed GPT (which required 30x more compute). At a larger scale, ELECTRA-Large performed comparably to RoBERTa and XLNet while using less than 25% of the computing resources.