Step-Audio 2 is an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation.

同类别比较
名称	排名	指标
dandelin/ViLT	10,060	语言: Python 星标数: 1,537 分叉数: 228 简介: Code for the ICML 2021 (long talk) paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"
allenzren/open-pi-zero	10,069	语言: Python 星标数: 1,534 分叉数: 110 简介: Re-implementation of pi0 vision-language-action (VLA) model from Physical Intelligence
ATH-MaaS/Ovis	10,073	语言: Python 星标数: 1,532 分叉数: 92 简介: A novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings.
kyutai-labs/hibiki	10,090	语言: Rust 星标数: 1,526 分叉数: 119
yu4u/age-gender-estimation	10,105	语言: Jupyter Notebook 星标数: 1,521 分叉数: 500 简介: Keras implementation of a CNN network for age and gender estimation
NVlabs/describe-anything	10,107	语言: Python 星标数: 1,519 分叉数: 95 简介: [ICCV 2025] Implementation for Describe Anything: Detailed Localized Image and Video Captioning
sigsep/open-unmix-pytorch	10,116	语言: Python 星标数: 1,515 分叉数: 206 简介: Open-Unmix - Music Source Separation for PyTorch
mbzuai-oryx/Video-ChatGPT	10,123	语言: Python 星标数: 1,513 分叉数: 130 简介: [ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

关键参数与规格
参数	值
语言	Python
类别	Models
子类别	Audio/Speech models
关键词	Speech-to-text, Text-to-speech
星标数	1,525
分叉数	114
贡献者	5
创建日期	2025-07-15
更新日期	2026-10-16
1 天星标数	0
7 天星标数	-1
排名	10,096
上次排名	10,113
排名变化	17
7 天星标排名	17,117
上次 7 天星标排名	12,550
7 天星标排名变化	-4,567
总星标排名	7,997
上次总星标排名	7,986
总星标排名变化	-11
国家	China

Step-Audio 2

🔥🔥🔥 News!!

Sep 15, 2025: 👋 We release Step-Audio 2 mini Think and its corresponding examples.
Sep 3, 2025: 👋 We release our vLLM backend and corresponding examples.
Aug 29, 2025: 👋 We are pleased to open-source Step-Audio 2 mini, Step-Audio 2 mini Base and their corresponding inference examples. Technical report is also updated.
Jul 24, 2025: 👋 We release demonstration videos for Step-Audio 2.
Jul 23, 2025: 👋 We release our benchmark for paralinguistic information understanding, StepEval-Audio-Paralinguistic.
Jul 23, 2025: 👋 We release our benchmark for tool calling, StepEval-Audio-Toolcall.
Jul 23, 2025: 👋 We release the technical report of Step-Audio 2.

WeChat Developer Group

Introduction

Step-Audio 2 is an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation.

Advanced Speech and Audio Understanding: Promising performance in ASR and audio understanding by comprehending and reasoning semantic information, para-linguistic and non-vocal information.
Intelligent Speech Conversation: Achieving natural and intelligent interactions that are contextually appropriate for various conversational scenarios and paralinguistic information.
Emotional Reasoning: Analyzing user's paralinguistic information such as age and emotion, leading to more accurate and intelligent interpretation of the audio context.
Tool Calling and Multimodal RAG: By leveraging tool calling and RAG to access real-world knowledge (both textual and acoustic), Step-Audio 2 can generate responses with fewer hallucinations for diverse scenarios, while also having the ability to switch timbres based on retrieved speech.
State-of-the-Art Performance: Achieving state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. (See Evaluation and Technical Report).

Open-source: Step-Audio 2 mini, Step-Audio 2 mini Base and Step-Audio 2 mini Think are released under Apache 2.0 license.

Model Download

Models	🤗 Hugging Face	ModelScope
Step-Audio 2 mini	stepfun-ai/Step-Audio-2-mini	stepfun-ai/Step-Audio-2-mini
Step-Audio 2 mini Base	stepfun-ai/Step-Audio-2-mini-Base	stepfun-ai/Step-Audio-2-mini-Base
Step-Audio 2 mini Think	stepfun-ai/Step-Audio-2-mini-Think	stepfun-ai/Step-Audio-2-mini-Think

Model Usage

🔧 Dependencies and Installation

conda create -n stepaudio2 python=3.10
conda activate stepaudio2
pip install transformers==4.49.0 torchaudio librosa onnxruntime s3tokenizer diffusers hyperpyyaml

git clone https://github.com/stepfun-ai/Step-Audio2.git
cd Step-Audio2
git lfs install
git clone https://huggingface.co/stepfun-ai/Step-Audio-2-mini
# git clone https://huggingface.co/stepfun-ai/Step-Audio-2-mini-Base

🔧 vLLM docker image

We highly recommend using our vLLM backend for faster and streaming inference, also deploying across multiple GPUs.

# (Optional) build the docker image yourself (very slow and requires 32GiB of memory)
# docker build -t stepfun2025/vllm:step-audio-2-v20250909 .

# run vLLM docker
docker run --rm -ti --gpus all \
    -v Step-Audio-2-mini:/Step-Audio-2-mini \
    -p 8000:8000 \
    stepfun2025/vllm:step-audio-2-v20250909 \
    -- vllm serve /Step-Audio-2-mini \
    --served-model-name step-audio-2-mini \
    --port 8000 \
    --max-model-len 16384 \
    --max-num-seqs 32 \
    --tensor-parallel-size 1 \
    --enable-auto-tool-choice \
    --tool-call-parser step_audio_2 \
    --tokenizer-mode step_audio_2 \
    --chat_template_content_format string \
    --audio-parser step_audio_2_tts_ta4 \
    --trust-remote-code

🚀 Inference Scripts

python examples.py
# python examples-base.py
# python examples-vllm.py
# python examples-think.py

🚀 Local web demonstration

pip install gradio
python web_demo.py
# python web_demo_vllm.py

Online demonstration

StepFun realtime console

Both Step-Audio 2 and Step-Audio 2 mini are available in our StepFun realtime console with web search tool enabled.
You will need an API key from the StepFun Open Platform.

StepFun AI Assistant

Step-Audio 2 is also available in our StepFun AI Assistant mobile App with both web and audio search tools enabled.
Please scan the following QR code to download it from your app store then tap the phone icon in the top-right corner.

WeChat group

You can scan the following QR code to join our WeChat group for communication and discussion.

Evaluation

Automatic speech recognition

CER for Chinese, Cantonese and Japanese and WER for Arabian and English. N/A indicates that the language is not supported.

Paralinguistic information understanding

StepEval-Audio-Paralinguistic

Model

Avg.

Gender

Age

Timbre

Scenario

Event

Emotion

Pitch

Rhythm

Speed

Style

Vocal

GPT-4o Audio

43.45

Kimi-Audio

49.64

Qwen-Omni

44.18

Step-Audio-AQAA

36.91

Step-Audio 2

83.09

100

Step-Audio 2 mini

80.00

100

Audio understanding and reasoning

MMAU

Model

Avg.

Sound

Speech

Music

Audio Flamingo 3

73.1

76.9

66.1

73.9

Gemini 2.5 Pro

71.6

75.1

71.5

68.3

GPT-4o Audio

58.1

58.0

64.6

51.8

Kimi-Audio

69.6

79.0

65.5

64.4

Omni-R1

77.0

81.7

76.0

73.4

Qwen2.5-Omni

71.5

78.1

70.6

65.9

Step-Audio-AQAA

49.7

50.5

51.4

47.3

Step-Audio 2

78.0

83.5

76.9

73.7

Step-Audio 2 mini

73.2

76.6

71.5

71.6

Big Bench Audio

Model

Accuracy

GPT-Realtime

82.8

Step-Audio 2 mini

54.8

Step-Audio 2

80.0

Step-Audio 2 Think

90.7

Speech translation

Model

CoVoST 2 (S2TT)

Avg.

English-to-Chinese

Chinese-to-English

GPT-4o Audio

29.61

40.20

19.01

Qwen2.5-Omni

35.40

41.40

29.40

Step-Audio-AQAA

28.57

37.71

19.43

Step-Audio 2

39.26

49.01

29.51

Step-Audio 2 mini

39.29

49.12

29.47

Model

CVSS (S2ST)

Avg.

English-to-Chinese

Chinese-to-English

GPT-4o Audio

23.68

20.07

27.29

Qwen-Omni

15.35

8.04

22.66

Step-Audio-AQAA

27.36

30.74

23.98

Step-Audio 2

30.87

34.83

26.92

Step-Audio 2 mini

29.08

32.81

25.35

Tool calling

StepEval-Audio-Toolcall. Date and time tools have no parameter.

Model

Objective

Metric

Audio search

Date & Time

Weather

Web search

Qwen3-32B†

Trigger

Precision / Recall

67.5 / 98.5

98.4 / 100.0

90.1 / 100.0

86.8 / 98.5

Type

Accuracy

100.0

98.5

Parameter

Accuracy

100.0

N/A

100.0

Step-Audio 2

Trigger

Precision / Recall

86.8 / 99.5

96.9 / 98.4

92.2 / 100.0

88.4 / 95.5

Type

Accuracy

100.0

90.5

98.4

Parameter

Accuracy

100.0

N/A

100.0

Speech-to-speech conversation

URO-Bench. U. R. O. stands for understanding, reasoning, and oral conversation, respectively.

Model

Language

Basic

Pro

Avg.

GPT-4o Audio

Chinese

78.59

89.40

65.48

85.24

67.10

70.60

57.22

70.20

Kimi-Audio

73.59

79.34

64.66

79.75

66.07

60.44

59.29

76.21

Qwen-Omni

68.98

59.66

69.74

77.27

59.11

59.01

59.82

58.74

Step-Audio-AQAA

74.71

87.61

59.63

81.93

65.61

74.76

47.29

68.97

Step-Audio 2

83.32

91.05

75.45

86.08

68.25

74.78

63.18

65.10

Step-Audio 2 mini

77.81

89.19

64.53

84.12

69.57

76.84

58.90

69.42

GPT-4o Audio

English

84.54

90.18

75.90

90.41

67.51

60.65

64.36

78.46

Kimi-Audio

60.04

83.36

42.31

60.36

49.79

50.32

40.59

56.04

Qwen-Omni

70.58

66.29

69.62

76.16

50.99

44.51

63.88

49.41

Step-Audio-AQAA

71.11

90.15

56.12

72.06

52.01

44.25

54.54

59.81

Step-Audio 2

83.90

92.72

76.51

84.92

66.07

64.86

67.75

66.33

Step-Audio 2 mini

74.36

90.07

60.12

77.65

61.25

58.79

61.94

63.80

Acknowledgements

Part of the code for this project comes from:

Step-Audio 2 mini and Step-Audio 2 mini Base are initialized with the weights from:

Thank you to all the open-source projects for their contributions to this project!

License

The code in the repository is licensed under Apache 2.0 License.

Citation

@misc{wu2025stepaudio2technicalreport,
      title={Step-Audio 2 Technical Report},
      author={Boyong Wu and Chao Yan and Chen Hu and Cheng Yi and Chengli Feng and Fei Tian and Feiyu Shen and Gang Yu and Haoyang Zhang and Jingbei Li and Mingrui Chen and Peng Liu and Wang You and Xiangyu Tony Zhang and Xingyuan Li and Xuerui Yang and Yayue Deng and Yechang Huang and Yuxin Li and Yuxin Zhang and Zhao You and Brian Li and Changyi Wan and Hanpeng Hu and Jiangjie Zhen and Siyu Chen and Song Yuan and Xuelin Zhang and Yimin Jiang and Yu Zhou and Yuxiang Yang and Bingxin Li and Buyun Ma and Changhe Song and Dongqing Pang and Guoqiang Hu and Haiyang Sun and Kang An and Na Wang and Shuli Gao and Wei Ji and Wen Li and Wen Sun and Xuan Wen and Yong Ren and Yuankai Ma and Yufan Lu and Bin Wang and Bo Li and Changxin Miao and Che Liu and Chen Xu and Dapeng Shi and Dingyuan Hu and Donghang Wu and Enle Liu and Guanzhe Huang and Gulin Yan and Han Zhang and Hao Nie and Haonan Jia and Hongyu Zhou and Jianjian Sun and Jiaoren Wu and Jie Wu and Jie Yang and Jin Yang and Junzhe Lin and Kaixiang Li and Lei Yang and Liying Shi and Li Zhou and Longlong Gu and Ming Li and Mingliang Li and Mingxiao Li and Nan Wu and Qi Han and Qinyuan Tan and Shaoliang Pang and Shengjie Fan and Siqi Liu and Tiancheng Cao and Wanying Lu and Wenqing He and Wuxun Xie and Xu Zhao and Xueqi Li and Yanbo Yu and Yang Yang and Yi Liu and Yifan Lu and Yilei Wang and Yuanhao Ding and Yuanwei Liang and Yuanwei Lu and Yuchu Luo and Yuhe Yin and Yumeng Zhan and Yuxiang Zhang and Zidong Yang and Zixin Zhang and Binxing Jiao and Daxin Jiang and Heung-Yeung Shum and Jiansheng Chen and Jing Li and Xiangyu Zhang and Yibo Zhu},
      year={2025},
      eprint={2507.16632},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.16632},
}

Step-Audio2

记录了多少星标和分叉？

仓库排名是多少？

记录的语言和类别是什么？

哪个仓库排名最接近？

Step-Audio 2

🔥🔥🔥 News!!

Introduction

Model Download

Model Usage

🔧 Dependencies and Installation

🔧 vLLM docker image

🚀 Inference Scripts

🚀 Local web demonstration

Online demonstration

StepFun realtime console

StepFun AI Assistant

WeChat group

Evaluation

Automatic speech recognition

Paralinguistic information understanding

Audio understanding and reasoning

Speech translation

Tool calling

Speech-to-speech conversation

Acknowledgements

License

Citation

Star History