MOA is an open source framework for Big Data stream mining. It includes a collection of machine learning algorithms (classification, regression, clustering, outlier detection, concept drift detection and recommender systems) and tools for evaluation.
Key parameters and specifications
Key parameters and specifications
Parameter
Value
Language
Python
Category
Model Development
Subcategory
Architecture
Keywords
Context engineering
Stars
669
Forks
45
Contributors
2
Created date
2023-12-12
Updated date
2026-10-14
Stars in 1 day
0
Stars in 7 days
-1
Rank
15,228
Previous rank
15,052
Rank change
-176
7-day stars rank
17,745
Previous 7-day stars rank
17,701
7-day stars rank change
-44
Total stars rank
15,401
Previous total stars rank
15,380
Total stars rank change
-21
Country
China
README summary
Agent Attention This repo contains the official PyTorch code and pre-trained models for **Agent Attention**. Agent Attention: On the Integration of Softmax and Linear Attention Introduction The attention module is the key component in Transformers. While the global attention mechanism offers robust expressiveness, its excessive computational cost constrains its applicability in various scenarios. In this paper, we propose a novel attention paradigm, **Agent Attention**, to strike a favorable balance between computational efficiency and representation power. Specifically, the Agent Attention
, denoted as a quadruple $(Q, A, K, V)$, introduces an additional set of agent tokens $A$ into the conventional attention module. The agent tokens first act as the agent for the query tokens $Q$ to aggregate information from $K$ and $V$, and then broadcast the information back to $Q$. Given the number of agent tokens can be designed to be much smaller than the number of query tokens, the agent attention is significantly more efficient than the widely adopted Softmax
attention, while preserving global context modelling capability. Interestingly, we show that the proposed agent attention is equivalent to a generalized form of linear attention. Therefore, agent attention seamlessly integrates the powerful Softmax attention and the highly efficient linear attention. Motivation (a) In Softmax attention, each query aggregates information from all features, incurring quadratic complexity. (b) Leveraging the redundancy between attention weights, agent attention uses a small number of agent tokens to act as the "agent'' for queries, capturing diverse semantic
information from all features, and then presenting it to each query. Method **An illustration of our agent attention and agent attention module. ** (a) Agent attention uses agent tokens to aggregate global information and distribute it to individual image tokens, resulting in a practical integration of Softmax and linear attention. $\rm{\sigma}(\cdot)$ represents Softmax function. In (b), we depict the information flow of agent attention module. As a showcase, we acquire agent tokens through pooling. Subsequently, agent tokens are utilized
to aggregate information from $V$, and $Q$ queries features from the agent features. In addition, agent bias and DWC are adopted to add positional information and maintain feature diversity. Results Classification Please go to the folder agent_transformer for specific document. Comparison of different models on ImageNet-1K. Increasing resolution to $\{256^2, 288^2, 320^2, 352^2, 384^2\}$. Downstream tasks Please go to the folder detection, segmentation for specific documents. AgentSD When applied to Stable Diffusion, our
agent attention accelerates generation and substantially enhances image generation quality **without any additional training**. Please go to the folder agentsd for specific document. Quantitative Results of Stable Diffusion, ToMeSD and our AgentSD. Samples generated by Stable Diffusion, ToMeSD ($r=0. Citation If you find this repo helpful, please consider citing us. Contact If you have any questions, please feel free to contact the authors.
Frequently asked questions
How many stars and forks are recorded?
LeapLabTHU/Agent-Attention has 669 stars and 45 forks.
What is the repository rank?
LeapLabTHU/Agent-Attention is ranked 15,228.
What language and category are recorded?
LeapLabTHU/Agent-Attention has 669 stars, uses Python and belongs to Model Development.
Which repository is closest in rank?
LeapLabTHU/Agent-Attention is compared with PaddlePaddle/Knover, ranked 15,230 with 669 stars.
README
Agent Attention
This repo contains the official PyTorch code and pre-trained models for Agent Attention.
The attention module is the key component in Transformers. While the global attention mechanism offers robust expressiveness, its excessive computational cost constrains its applicability in various scenarios. In this paper, we propose a novel attention paradigm, Agent Attention, to strike a favorable balance between computational efficiency and representation power. Specifically, the Agent Attention, denoted as a quadruple $(Q, A, K, V)$, introduces an additional set of agent tokens $A$ into the conventional attention module. The agent tokens first act as the agent for the query tokens $Q$ to aggregate information from $K$ and $V$, and then broadcast the information back to $Q$. Given the number of agent tokens can be designed to be much smaller than the number of query tokens, the agent attention is significantly more efficient than the widely adopted Softmax attention, while preserving global context modelling capability. Interestingly, we show that the proposed agent attention is equivalent to a generalized form of linear attention. Therefore, agent attention seamlessly integrates the powerful Softmax attention and the highly efficient linear attention.
Motivation
(a) In Softmax attention, each query aggregates information from all features, incurring quadratic complexity. (b) Leveraging the redundancy between attention weights, agent attention uses a small number of agent tokens to act as the "agent'' for queries, capturing diverse semantic information from all features, and then presenting it to each query.
Method
An illustration of our agent attention and agent attention module. (a) Agent attention uses agent tokens to aggregate global information and distribute it to individual image tokens, resulting in a practical integration of Softmax and linear attention. $\rm{\sigma}(\cdot)$ represents Softmax function. In (b), we depict the information flow of agent attention module. As a showcase, we acquire agent tokens through pooling. Subsequently, agent tokens are utilized to aggregate information from $V$, and $Q$ queries features from the agent features. In addition, agent bias and DWC are adopted to add positional information and maintain feature diversity.
When applied to Stable Diffusion, our agent attention accelerates generation and substantially enhances image generation quality without any additional training. Please go to the folder agentsd for specific document.
Quantitative Results of Stable Diffusion, ToMeSD and our AgentSD.
Samples generated by Stable Diffusion, ToMeSD ($r=0.4$) and AgentSD ($r=0.4$).
If you find this repo helpful, please consider citing us.
@inproceedings{han2024agent,
title={Agent attention: On the integration of softmax and linear attention},
author={Han, Dongchen and Ye, Tianzhu and Han, Yizeng and Xia, Zhuofan and Pan, Siyuan and Wan, Pengfei and Song, Shiji and Huang, Gao},
booktitle={European Conference on Computer Vision},
year={2024},
}
Contact
If you have any questions, please feel free to contact the authors.
