awesome-fast-attention

A curated list of efficient attention modules (last update: Wed, 10 Mar 2021 23:52:22 +0000)

Table of Contents

• Efficient Attention
• Articles/Surveys/Benchmarks

Efficient Attention

Paper (citations)	Implementation	Computational Complexity	AutoRegressive	Main Idea
Generating Wikipedia by Summarizing Long Sequences (282)	memory-compressed-attention		:heavy_check_mark:

EXPAND

compresses key and value + blocked attention

| |CBAM: Convolutional Block Attention Module (999+)|attention-module ||:x:|

EXPAND

combines the SE attention with a per pixel(local) weight

| |Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks (16)|set_transformer ||:x:|

EXPAND

uses K relay nodes

| |CCNet: Criss-Cross Attention for Semantic Segmentation (296)|CCNet ||:x:|

EXPAND

each pixel attends to its row and column simultaneously

| |Efficient Attention: Attention with Linear Complexities (16)|efficient-attention ||:x:|

EXPAND

Softmax(Q)*(Softmax(K^T)*V)

| |Star-Transformer (40)|fastNLP ||:x:|

EXPAND

uses a relay(global) node and attends to/from that node

| |GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond (199)|GCNet ||:x:|

EXPAND

squeeze and excitation with an attention pooling (instead of a GAP)

| |Generating Long Sequences with Sparse Transformers (257)|DeepSpeed ||:heavy_check_mark:|

EXPAND

sparse block based attention

| |SCRAM: Spatially Coherent Randomized Attention Maps (1)|-||:heavy_check_mark:|

EXPAND

uses PatchMatch to find close keys

| |Interlaced Sparse Self-Attention for Semantic Segmentation (24)|IN_PAPER||:heavy_check_mark:|

EXPAND

combination of a short length and then long range(dilated) attention

| |Permutohedral Attention Module for Efficient Non-Local Neural Networks (3)|Permutohedral_attention_module ||:x:|

EXPAND

uses permutohedral lattice approximation algorithm to approximate the attention output

| |Large Memory Layers with Product Keys (43)|XLM ||:heavy_check_mark:|

EXPAND

search for nearest neighbor keys

| |Expectation-Maximization Attention Networks for Semantic Segmentation (79)|EMANet ||:x:|

EXPAND

applys expectation maximization to cluster keys into k clusters

| |BP-Transformer: Modelling Long-Range Context via Binary Partitioning (15)|BPT ||:heavy_check_mark:|

EXPAND

attends to distant tokens coarsely and attends to close tokens in a more fine-grained manner

| |Compressive Transformers for Long-Range Sequence Modelling (48)|compressive-transformer-pytorch ||:heavy_check_mark:|

EXPAND

compresses distant tokens instead of just stop_grad() ing them, more efficient version of transformerXL

| |Axial Attention in Multidimensional Transformers (36)|axial-attention ||:heavy_check_mark:|

EXPAND

apply attention on each axis separately

| |Reformer: The Efficient Transformer (216)|trax ||:heavy_check_mark:|

EXPAND

uses LSH to find close keys

| |Sparse Sinkhorn Attention (16)|sinkhorn-transformer ||:heavy_check_mark:|

EXPAND

uses a cost matrix to limit attention between buckets

| |Transformer on a Diet (2)|transformer-on-diet ||:heavy_check_mark:|

EXPAND

dilated transformer like wavenet

| |Time-aware Large Kernel Convolutions (9)|TaLKConvolutions ||:heavy_check_mark:|

EXPAND

calculate mean over a dynamic subsequence around each token with the help of summed-area table

| |SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection (2)|-||:heavy_check_mark:|

EXPAND

learns the q, k connections == dynamically creates a sparse attention matrix

| |Efficient Content-Based Sparse Attention with Routing Transformers (38)|routing-transformer ||:heavy_check_mark:|

EXPAND

computes attention with same-cluster tokens (computed by online k-means)

| |Neural Architecture Search for Lightweight Non-Local Networks (11)|AutoNL ||:x:|

EXPAND

computes Q(KV) and also down samples q, k, v both in spatial and channel dimensions

| |Longformer: The Long-Document Transformer (159)|longformer ||:heavy_check_mark:|

EXPAND

global + blocked attention

| |ETC: Encoding Long and Structured Inputs in Transformers (16)|-||:x:|

EXPAND

combines global attention (star transformer with multiple global tokens) with local attention

| |Multi-scale Transformer Language Models (2)|IN_PAPER||:heavy_check_mark:|

EXPAND

UNet like + retina attetion is something close to BP-Transformer

| |Synthesizer: Rethinking Self-Attention in Transformer Models (26)|Synthesizer-Rethinking-Self-Attention-Transformer-Models ||:heavy_check_mark:|

EXPAND

does not compute pairwise interactions

| |Jukebox: A Generative Model for Music (45)|jukebox ||:heavy_check_mark:|

EXPAND

better attention patterns from Sparse Transformer

| |Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers (0)|-||:heavy_check_mark:|

EXPAND

does not compute pairwise interactions and uses fixed mask patters

| |GMAT: Global Memory Augmentation for Transformers (2)|gmat ||:x:|

EXPAND

adds global tokens

| |Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (45)|fast-transformers ||:heavy_check_mark:|

EXPAND

uses phi(q)(phi(k)v) and also improves the sequential sampling step

| |Linformer: Self-Attention with Linear Complexity (47)|linformer-pytorch ||:x:|

EXPAND

project key and value from nd to kd

| |Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (8)|google-research ||:heavy_check_mark:|

EXPAND

calculate an unbiased stochastic approximation of the attention matrix

| |Kronecker Attention Networks (1)|kronecker-attention-pytorch ||:x:|

EXPAND

uses horizontal and lateral average matrices

| |Real-time Semantic Segmentation with Fast Attention (5)|-||:x:|

EXPAND

l2_norm(q)*(l2_norm(k)*v)

| |Fast Transformers with Clustered Attention (6)|fast-transformers ||:x:|

EXPAND

groups queries together with LSH

| |Big Bird: Transformers for Longer Sequences (60)|DeepSpeed ||:x:|

EXPAND

ETC with random connections

| |Tensor Low-Rank Reconstruction for Semantic Segmentation (3)|-||:x:|

EXPAND

decompose the full attention tensor into rank one tensors (CP decomposition)

| |Looking for change? Roll the Dice and demand Attention (0)|IN_PAPER||:x:|

EXPAND

uses the fractal tanimoto similarity to compare queries with keys inside the attention module

| |Rethinking Attention with Performers (30)|google-research ||:heavy_check_mark:|

EXPAND

unbiased approximation of the attention matrix with softmax kernel

| |Memformer: The Memory-Augmented Transformer (0)|memformer ||:heavy_check_mark:|

EXPAND

attend to memory slots + Memory-Replay BackPropagation

| |SMYRF: Efficient Attention using Asymmetric Clustering (1)|smyrf ||:x:|

EXPAND

LSH with balanced clusters

| |Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (0)|Informer2020 ||:heavy_check_mark:|

EXPAND

sparse attention + funnel like encoder

| |Sub-Linear Memory: How to Make Performers SLiM (0)|google-research ||:heavy_check_mark:|

EXPAND

Performer but with sublinear Memory usage

| |Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention (0)|Nystromformer ||:x:|

EXPAND

uses Nystrom method to approximate the attention matrix

| |Linear Transformers Are Secretly Fast Weight Memory Systems (0)|fast-weight-transformers ||:heavy_check_mark:|

EXPAND

show that linear transformers are basically fast weight networks + propose a new kernel function to linearise attention, balancing simplicity and effectiveness

| |LambdaNetworks: Modeling Long-Range Interactions Without Attention (6)|lambda-networks ||:heavy_check_mark:|

EXPAND

generates a linear layer based on context + decouple pos/context

| |Random Feature Attention (2)|-||:heavy_check_mark:|

EXPAND

kernel approximation and also transformers are rnn

|

Articles/Surveys/Benchmarks

• A Survey of Long-Term Context in Transformers

• Efficient Transformers: A Survey

• Long Range Arena: A Benchmark for Efficient Transformers