Language Models

Edit

Natural Language Processing • 138 methods

Language Models are models for predicting the next word or character in a document. Below you can find a continuously updating list of language models.

Subcategories

1 Transformers

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	9443
Diffusion Denoising Diffusion Probabilistic Models	2020	6339
Focus Focus Your Attention (with Adaptive IIR Filters)	2023	5880
BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	5073
GPT-4 GPT-4 Technical Report	2023	1486
GPT-3 Language Models are Few-Shot Learners	2020	1350
RoBERTa RoBERTa: A Robustly Optimized BERT Pretraining Approach	2019	749
GPT Improving Language Understanding by Generative Pre-Training	2018	671
T5 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	2019	554
GPT-2 Language Models are Unsupervised Multitask Learners	2019	544
LLaMA LLaMA: Open and Efficient Foundation Language Models	2023	460
BART BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	2019	455
ELMo Deep contextualized word representations	2018	227
CAM Is Object Localization for Free? - Weakly-Supervised Learning With Convolutional Neural Networks	2015	207
OPT OPT: Open Pre-trained Transformer Language Models	2022	184
mBERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	182
XLM-R Unsupervised Cross-lingual Representation Learning at Scale	2019	161
RAG Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	2020	156
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	156
ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	2019	154
DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	2019	126
Mamba Mamba: Linear-Time Sequence Modeling with Selective State Spaces	2023	125
Electric Pre-Training Transformers as Energy-Based Cloze Models	2020	119
PaLM PaLM: Scaling Language Modeling with Pathways	2022	112
ELECTRA ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators	2020	107
BLOOM BLOOM: A 176B-Parameter Open-Access Multilingual Language Model	2022	82
mT5 mT5: A massively multilingual pre-trained text-to-text transformer	2020	82
Longformer Longformer: The Long-Document Transformer	2020	78
Flan-T5 Scaling Instruction-Finetuned Language Models	2022	72
Performer Rethinking Attention with Performers	2020	72
DeBERTa DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	65
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	61
mBART Multilingual Denoising Pre-training for Neural Machine Translation	2020	59
XLM Cross-lingual Language Model Pretraining	2019	55
CodeBERT CodeBERT: A Pre-Trained Model for Programming and Natural Languages	2020	49
ERNIE ERNIE: Enhanced Representation through Knowledge Integration	2019	48
PEGASUS PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization	2019	43
ULMFiT Universal Language Model Fine-tuning for Text Classification	2018	36
GLM GLM-130B: An Open Bilingual Pre-trained Model	2022	36
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	35
ViLBERT ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	2019	30
ETC ETC: Encoding Long and Structured Inputs in Transformers	2020	29
GPT-Neo	2000	28
CodeGen CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis	2022	23
BLOOMZ Crosslingual Generalization through Multitask Finetuning	2022	23
CTRL CTRL: A Conditional Transformer Language Model for Controllable Generation	2019	21
Pythia Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling	2023	21
Synthesizer Synthesizer: Rethinking Self-Attention in Transformer Models	2020	18
CodeT5 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation	2021	18
Universal Transformer Universal Transformers	2018	17
Switch Transformer Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	2021	16
Reformer Reformer: The Efficient Transformer	2020	15
BigBird Big Bird: Transformers for Longer Sequences	2020	14
Chinchilla Training Compute-Optimal Large Language Models	2022	14
Linformer Linformer: Self-Attention with Linear Complexity	2020	13
GPT-NeoX GPT-NeoX-20B: An Open-Source Autoregressive Language Model	2022	11
Levenshtein Transformer Levenshtein Transformer	2019	11
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	10
Galactica Galactica: A Large Language Model for Science	2022	10
ProphetNet ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training	2020	9
TNT Transformer in Transformer	2021	8
UL2 UL2: Unifying Language Learning Paradigms	2022	8
CharacterBERT CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters	2020	8
Cross-View Training Semi-Supervised Sequence Modeling with Cross-View Training	2018	8
GEE Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes	2021	7
E-Branchformer E-Branchformer: Branchformer with Enhanced merging for speech recognition	2022	7
mT0 Crosslingual Generalization through Multitask Finetuning	2022	6
MATE MATE: Multi-view Attention for Table Transformer Efficiency	2021	6
PLATO-2 PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning	2020	6
Neural Probabilistic Language Model	2003	5
Bort Optimal Subarchitecture Extraction For BERT	2020	5
Charformer Charformer: Fast Character Transformers via Gradient-based Subword Tokenization	2021	5
MobileBERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	5
EGT Global Self-Attention as a Replacement for Graph Convolution	2021	5
CANINE CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation	2021	4
Cross-encoder Reranking ReRankMatch: Semi-Supervised Learning with Semantics-Oriented Similarity Representation	2021	3
Compressive Transformer Compressive Transformers for Long-Range Sequence Modelling	2019	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
DeeBERT DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	2020	3
Subformer Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers	2021	3
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
GANformer Generative Adversarial Transformers	2021	3
VideoBERT VideoBERT: A Joint Model for Video and Language Representation Learning	2019	3
TAPEX TAPEX: Table Pre-training via Learning a Neural SQL Executor	2021	3
ConvBERT ConvBERT: Improving BERT with Span-based Dynamic Convolution	2020	3
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	2
PMLM Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order	2020	2
CuBERT Learning and Evaluating Contextual Embedding of Source Code	2019	2
I-BERT I-BERT: Integer-only BERT Quantization	2021	2
DynaBERT DynaBERT: Dynamic BERT with Adaptive Width and Depth	2020	2
TernaryBERT TernaryBERT: Distillation-aware Ultra-low Bit BERT	2020	2
Fastformer Fastformer: Additive Attention Can Be All You Need	2021	2
Parallel Layers PaLM: Scaling Language Modeling with Pathways	2022	2
ESACL Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization	2021	2
OPT-IML OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization	2022	2
SHA-RNN Single Headed Attention RNN: Stop Thinking With Your Head	2019	2
Nyströmformer Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention	2021	2
Gated Convolution Network Language Modeling with Gated Convolutional Networks	2016	2
AutoTinyBERT AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	2021	2
PermuteFormer PermuteFormer: Efficient Relative Position Encoding for Long Sequences	2021	1
NormFormer NormFormer: Improved Transformer Pretraining with Extra Normalization	2021	1
BP-Transformer BP-Transformer: Modelling Long-Range Context via Binary Partitioning	2019	1
IB-BERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	1
MacBERT Revisiting Pre-Trained Models for Chinese Natural Language Processing	2020	1
CPM-2 CPM-2: Large-scale Cost-effective Pre-trained Language Models	2021	1
mBARTHez BARThez: a Skilled Pretrained French Sequence-to-Sequence Model	2020	1
ooJpiued Dynamic Network Model from Partial Observations	2018	1
PanGu-$α$ PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation	2021	1
KE-MLM Knowledge Enhanced Masked Language Model for Stance Detection	2021	1
SMITH Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching	2020	1
ClipBERT Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling	2021	1
SqueezeBERT SqueezeBERT: What can computer vision teach NLP about efficient neural networks?	2020	1
ERNIE-GEN ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
PAR Transformer Pay Attention when Required	2020	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1
RealFormer RealFormer: Transformer Likes Residual Attention	2020	1
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
SongNet SongNet: Rigid Formats Controlled Text Generation	2020	1
Funnel Transformer Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing	2020	1
T-D Generating Wikipedia by Summarizing Long Sequences	2018	1
SC-GPT Few-shot Natural Language Generation for Task-Oriented Dialog	2020	1
Chinese Pre-trained Unbalanced Transformer CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation	2021	1
BinaryBERT BinaryBERT: Pushing the Limit of BERT Quantization	2020	1
Adaptively Sparse Transformer Adaptively Sparse Transformers	2019	1

Language Models Edit

Methods Add a Method

Language Models

Edit

Methods

Add a Method