AI Rank
← Open Source
abacaj

awesome-transformers

A curated list of awesome transformer models.

ListsModel collections
Open on GitHub
Momentum
+0stars in 24 hours0.0%
691
Stars
49
Forks
+0
This week
3
Contributors
Created 2023-02-25 · Updated 2026-09-28 · #14645 today
Top developers
abacajCoFormerevison
README

Awesome Transformers

Transformers

A curated list of awesome transformer models.

If you want to contribute to this list, send a pull request or reach out to me on twitter: @abacaj. Let's make this list useful.

There are a number of models available that are not entirely open source (non-commercial, etc), this repository should serve to also make you aware of that. Tracking the original source/company of the model will help.

I would also eventually like to add model use cases. So it is easier for others to find the right one to fine-tune.

Format:

  • Model name: short description, usually from paper
    • Model link (usually huggingface or github)
    • Paper link
    • Source as company or group
    • Model license

Table of Contents

Encoder models

  • ALBERT: "A Lite" version of BERT

  • BERT: Bidirectional Encoder Representations from Transformers

  • DistilBERT: Distilled version of BERT smaller, faster, cheaper and lighter

  • DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

  • Electra: Pre-training Text Encoders as Discriminators Rather Than Generators

  • RoBERTa: Robustly Optimized BERT Pretraining Approach

Decoder models

  • BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

  • CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

  • LLaMa: Open and Efficient Foundation Language Models

  • GPT: Improving Language Understanding by Generative Pre-Training

  • GPT-2: Language Models are Unsupervised Multitask Learners

  • GPT-J: A 6 Billion Parameter Autoregressive Language Model

  • GPT-NEO: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

  • GPT-NEOX-20B: An Open-Source Autoregressive Language Model

  • NeMo Megatron-GPT: Megatron-GPT 20B is a transformer-based language model.

  • OPT: Open Pre-trained Transformer Language Models

  • BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

  • GLM: An Open Bilingual Pre-Trained Model

  • YaLM: Pretrained language model with 100B parameters

Encoder+decoder (seq2seq) models

  • T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

  • FLAN-T5: Scaling Instruction-Finetuned Language Models

  • Code-T5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

  • Bart: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

  • Pegasus: Pre-training with Extracted Gap-sentences for Abstractive Summarization

  • MT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

  • UL2: Unifying Language Learning Paradigms

  • FLAN-UL2: A New Open Source Flan 20B with UL2

  • EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

Multimodal models

  • Donut: OCR-free Document Understanding Transformer

  • LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

    • Model
    • Paper
    • Microsoft
    • CC BY-NC-SA 4.0 (non-commercial)

  • TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

  • CLIP: Learning Transferable Visual Models From Natural Language Supervision

  • Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

Vision models

  • DiT: Self-supervised Pre-training for Document Image Transformer

  • DETR: End-to-End Object Detection with Transformers

  • EfficientFormer: Vision Transformers at MobileNet Speed

Audio models

  • Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

  • VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Recommendation models

  • Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)

Grounded Situation Recognition models

  • Grounded Situation Recognition with Transformers

  • Collaborative Transformers for Grounded Situation Recognition