中文

An Unreal Engine plugin that helps you integrate artificial intelligence and machine learning technologies.

 

 =8.0 (Arm64)-brightgreen.svg"> =14.38.33130-aff.svg"> =10.0.22621-orange.svg"> =2022.17.6.3-blue">

Latest News | Documentation | Download | M4U Remoting(Android App) | Speech Model Packages

Free vs. Paid Edition

Demo Projects: Full Demo (Windows) | Android Demo

Introduction

MediaPipe4U provides a suite of components and tools that let you quickly apply artificial intelligence (AI) and machine learning (ML) technologies in Unreal Engine projects. You can immediately integrate these solutions into your UE projects and customize them to your needs. They include motion capture, facial expression capture, text-to-speech (TTS), automatic speech recognition (ASR), and more. All features are real-time, offline, low-latency, and easy to use.

Latest News

20260811

• [new] :rainbow: Added Unreal Engine 5.7 and 5.8 support
• [new] :rainbow: Added VideoPlayerImageSource, which supports opening video files from the Android gallery for motion and expression capture
• [fix] Bug fixes: #243, #262
• [fix] Fixed a crash when stopping motion capture

20250818

• [new] Added Unreal Engine 5.6 support.
• [fix] Incorrect negative-angle handling when using NvAR to capture head rotation. #122
• [fix] Packaging with UE 5.3 or earlier may fail. #237

20250506

• [fix] :rainbow: Motion capture did not work on the Android platform (#221, #190).

20250408

• [new] :rainbow: The free edition can now package all features, including speech and expression capture.
• [new] :rainbow: Ollama support: integrate Ollama for large language model inference, supporting many LLMs such as DeepSeek, LLaMA, Phi, Qwen, and QWQ.
• [new] :rainbow: Dialogue component LLMSpeechChatRuntime: integrates LLM, TTS, and ASR, making it easy to implement chatbots in Blueprints.
• [new] :rainbow: Added new TTS support: Kokoro, Melo.
• [new] :rainbow: Added new ASR support: FunASR (Chinese/English hotword edition), FireRedASR (Chinese/English/dialects), MoonShine (English), SenseVoice (multilingual: Chinese/English/Japanese/Korean/Cantonese).
• [new] :rainbow: Added a Transformer-based TTS model: F5-TTS, with zero-shot voice cloning capabilities (DirectML/CUDA inference; AMD and Nvidia GPUs supported)
• [new] :rainbow: Voice wake-up: small-model inference-based voice wake-up, supporting custom wake words to activate ASR and standalone voice-command wake words.
• [new] :fire: Updated Google MediaPipe to the latest version.
• [new] :fire: Added Unreal Engine 5.5 support.
• [new] :rainbow: Added the NvAR pose capture algorithm, allowing switching between MediaPipe and Nvidia Maxine algorithms.
• [new] :rainbow: Open-sourced MediaPipe4U Remoting (an Android facial-capture app for MediaPipe4U).
• [new] :dizzy: Custom Mediapipe Connector (C++): a custom connector that can completely replace google mediapipe to implement custom logic (#195, #204).
• [new] :dizzy: Custom Mediapipe Feature (C++): a custom feature that can partially replace google mediapipe to implement custom logic (#195, #204).
• [new] :rainbow: Added the Android Demo project (Gitlab)
• [improve]:footprints: Upgraded the Demo project to UE5.5 and added Fake Demo, a C++ extension example that reads local files instead of MediaPipe.
• [improve]:footprints: Added voice wake-up and speaker-selection examples to the Demo project
• [improve]:footprints: Added LLM examples to the Demo project
• [improve]:footprints: Added voice chatbot examples to the Demo project
• [improve]:footprints: The Demo project now supports packaging
• [break change] :collision: Motion capture has been split into a standalone MediaPipe4UMotion plugin. This may break existing Blueprints; update your Blueprints after upgrading.
• [break change] :collision: Due to changes in the license-file format, existing licenses will no longer work. You can obtain free licenses here going forward.

Speech package downloads have moved to huggingface.

Only the Unreal Engine 5.5 version is currently available; other versions will be packaged and released subsequently ...

Known Issues

• The new Google Holistic Task API does not support GPU, so Android uses CPU inference; as before, Windows also uses CPU inference.
• Starting with Unreal Engine 5.4, the built-in OpenCV plugin no longer includes precompiled libraries (DLL files). On the first launch of the UE Editor, because M4U depends on the OpenCV plugin, the UE Editor will attempt to download the OpenCV source code and compile it on your computer. This may take a long time and make it appear that the engine is stuck at 75% loading. Please wait patiently and check the logs in the Saved directory under the project root to confirm that the process has completed. Users in China may also need a proxy to access the Internet. You can also follow the steps in #166 to solve this manually.

Release Notes

For more updates, see the update log:

• Update Log

:green_heart:All features are pure C++, with no Python or external programs required.

Features

• Motion Capture
  • Body motion capture
  • Finger motion capture
  • Translation capture
  • 3D model driving
  • Real-time driving
  • Supports standard cameras
  • ControlRig support
• Expression Capture
  • Facial expressions
  • Arkit Blendshape compatible (52 expressions)
  • Live Link compatible
  • Real-time driving
  • Supports standard cameras
• Multiple Capture Sources
  • Standard cameras
  • Video files
  • Images
  • Streaming media (RTMP/SMTP)
  • Android devices (phones and tablets) (M4U Remoting)
• Large Language Models (LLM)
  • Ollama support
• Text-to-Speech (TTS)
  • Offline synthesis
  • Real-time synthesis
  • 3D model lip-sync
  • Multi-model support (Browse models)
• Automatic Speech Recognition (ASR)
  • Offline recognition
  • Real-time recognition
  • Multi-model support (Browse models)
  • Voice wake-up (similar to Xiao Ai Tong Xue and Xiao Du Xiao Du)
• Animation Export
  • BVH data export
• Plugin Only
  • No external programs required
  • Everything is done in Unreal Engine

Documentation

• Simplified Chinese

• English

• 日本語 (申し訳ありませんが、それは単なるGoogle翻訳です)

• 한국어 (죄송합니다, 이것은 단지 구글 번역입니다)

Download

UnrealEngine 5.1 - 5.8

• Google Drive
• Baidu Netdisk

To learn what was included in each release, visit the Update Log page

Because the plugin uses precompiled binaries and includes a large number of C++ linker symbols and debug symbols, it occupies 5G of disk space after extraction (mostly UE-generated code in Intermediate)
You do not need to worry about disk usage. This is only disk usage during development; after packaging the project, the plugin uses approximately 200M of disk space (mainly GStreamer dynamic libraries)

Cross-platform

M4U now supports Android and Windows (Linux support is coming soon), but platform support differs slightly.

Free License Files

License files are periodically published in the Discussions, and are also automatically included in new plugin packages.

M4U Remoting App (Android)

Get the source code

About M4U Remoting

This is an open-source Android application that can send expression data to the MediaPipe4U Windows application.
With it, you can use an Android mobile device for expression capture and display expressions on Unreal Engine characters.

Demo Projects

Clone or download the source code for the Demo projects from the following GitHub repositories:

• Windows Demo: https://gitlab.com/endink/mediapipe4u-demo
• Android Demo: https://gitlab.com/endink/mediapipe4u-android-demo

Use a Git client to obtain the Demo projects (you must install git and git lfs) :

Windows Demo is a full-featured demo. If you are learning how to use MediaPipe4U, the Windows Demo is recommended.

Windows Demo:

git lfs clone https://gitlab.com/endink/mediapipe4u-demo.git

Android Demo:

git lfs clone https://gitlab.com/endink/mediapipe4u-android-demo.git

The Demo projects do not include the plugin. Download the plugin yourself and place it in the project's Plugins directory.

Plugin Videos

English Video Tutorial (YouTube)

Chinese Video Tutorial (bilibili)

bilibili

FAQ

If you encounter a problem, first check the FAQ, which collects issues encountered by many users and may include yours. If you cannot find it in the FAQ, submit an issue on GitHub first; direct messages may cause me to forget the problem.

Performance

Because the Windows version of MediaPipe does not support GPU inference, Windows relies on the CPU for human-pose prediction (see MediaPipe for details):

Test Environment

CPU: AMD 3600 CPU
Memory: 32GB
GPU: 1660s (this GPU does not support ray tracing)

Performance:

Frame rate (fps): 18-24

CPU utilization: 20% (the test DEMO contains only the MetahumanLight scene and VRM model)

Follow Me

Bilibili

YouTube

Users in China who cannot view the video can visit the Bilibili video

Acknowledgements

We thank the following open-source projects and frameworks for their contributions, which have had an important impact on the development of M4U:

1. M4U uses MediaPipe for motion capture and face capture.
2. M4U uses NVIDIA Maxine AR SDK for advanced face tracking and capture.
3. M4U uses PaddleSpeech for text-to-speech (TTS) synthesis.
4. M4U uses FunASR for automatic speech recognition (ASR).
5. M4U uses whisper.cpp as an ASR solution.
6. M4U uses Sherpa Onnx to enhance ASR capabilities.
7. M4U uses F5-TTS-ONNX to export F5-TTS models.
8. M4U uses GStreamer for video processing and real-time streaming-media decoding.
9. M4U uses code from PowerIK to implement inverse kinematics (IK) and ground adaptation.
10. M4U uses concepts from Kalidokit in the motion-capture domain.
11. M4U uses code from wongfei to enhance interoperability between GStreamer and MediaPipe.

We extend our sincere thanks to the developers and contributors of these projects. Their innovation and open-source contributions have greatly advanced M4U.