Pre-training for NLP

ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements)

Two-step pre-training approach where a generator replaces tokens and a discriminator identifies replaced tokens, more efficient than traditional MLM.

← Indietro