ai-avatar-system

🎭 AvatarAI — Real-Time AI Avatar Platform

Upload a photo · Clone a voice · Talk to any face in real time

Quick Start · Features · Architecture · GPU / AWS Deploy · API · Roadmap

The most complete open-source AI avatar / digital human system. Real-time talking-head lip-sync · Zero-shot voice cloning · Multi-LLM · Runs 100% locally or on AWS.

🎬 What is AvatarAI?

AvatarAI is an open-source, production-ready platform for building photorealistic AI avatar conversations. Upload any face photo, clone a voice from a 5-second audio clip, and have a real-time conversation — with lip-sync video generated on every single response.

[mic] → Whisper STT → Claude / GPT / Ollama (streaming) → Chatterbox TTS → MuseTalk lip-sync → [video]
                              < 2–4 s to first video chunk on AWS GPU >

What makes AvatarAI different:

• 🎤 Zero-shot voice cloning — 10 seconds of audio is all you need (Chatterbox Multilingual)
• 🎭 Any face, any language — upload a JPEG, pick from 23 languages, start talking
• ⚡ Token-streaming pipeline — the LLM streams live tokens while TTS + lip-sync run per sentence; the first video chunk plays before the model finishes its reply
• ✋ Barge-in — speak (or hit stop) mid-reply and the avatar yields instantly, like a real conversation
• 🔒 100% local mode — local storage, local Whisper, local LLM via Ollama: nothing leaves your machine
• 🔌 Multi-LLM — Claude (with prompt caching), GPT-4o, or any local model via Ollama / vLLM / LM Studio
• 🚀 AWS GPU deployment — one-command deploy to g5.xlarge for true real-time (~30 FPS)
• 🏗️ Production-grade — JWT + httpOnly-cookie auth, per-user rate limiting, Postgres + Alembic, S3/CloudFront, Prometheus, CI, a real test suite — the only project in this niche you can ship as a product, not just a demo

⚔️ How AvatarAI compares

Toolkits like Linly-Talker are great research playgrounds; Duix ships a Windows product. AvatarAI is the one you can deploy as a real multi-user web service.

✨ Features

🏗️ Architecture

┌──────────────────────────────────────────────────────────────┐
│                        Browser / Client                       │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────────┐ │
│  │Avatar Studio│  │ Voice Studio │  │   Chat Interface     │ │
│  │  (upload)   │  │  (cloning)   │  │ Idle anim + chunks   │ │
│  └──────┬──────┘  └──────┬───────┘  └──────────┬───────────┘ │
└─────────┼───────────────┼─────────────────────┼─────────────┘
          │ REST           │ REST                │ WebSocket
          ▼                ▼                     ▼
┌──────────────────────────────────────────────────────────────┐
│                       FastAPI Backend                         │
│  ┌──────────────────────────────────────────────────────┐    │
│  │                  WebSocket Manager                    │    │
│  │  split sentences → TTS → MuseTalk → stream chunks    │    │
│  └──────────────────────────────────────────────────────┘    │
│  ┌──────────┐ ┌───────────┐ ┌──────────┐ ┌───────────────┐  │
│  │  Whisper │ │Claude/GPT │ │ XTTS v2  │ │  MuseTalk     │  │
│  │   STT    │ │  / Llama  │ │   TTS    │ │  (GPU/CPU)    │  │
│  └──────────┘ └───────────┘ └──────────┘ └───────────────┘  │
│  ┌──────────┐ ┌──────────┐  ┌──────────┐ ┌───────────────┐  │
│  │PostgreSQL│ │  Redis   │  │  Celery  │ │ Local FS / S3 │  │
│  └──────────┘ └──────────┘  └──────────┘ └───────────────┘  │
└──────────────────────────────────────────────────────────────┘

Real-Time Data Flow (one conversation turn)

[User types / speaks]
        │
        ▼
  Whisper STT ─────────────────► transcript
        │
        ▼
  Claude / GPT / Llama ────────► full response text
        │
        ▼
  Split into sentences ────────► ["Hello!", "How are you?", ...]
        │
        ├── sentence 1 → XTTS → MuseTalk → video_chunk WS → browser plays
        ├── sentence 2 → XTTS → MuseTalk → video_chunk WS → queued
        └── sentence N → XTTS → MuseTalk → video_chunk WS → queued

📁 Project Structure

ai-avatar-system/
├── backend/                    # FastAPI application
│   ├── app/
│   │   ├── api/v1/             # REST endpoints (users, avatars, sessions, messages)
│   │   ├── services/           # Core services (LLM, TTS, STT, animator, storage)
│   │   ├── models/             # SQLAlchemy DB models
│   │   └── websocket.py        # Real-time WebSocket handler + sentence streaming
│   ├── alembic/                # Database migrations
│   ├── models/MuseTalk/        # MuseTalk V1.5 (lip-sync engine)
│   │   └── scripts/
│   │       └── musetalk_worker.py  # Persistent worker (models loaded once)
│   ├── tests/                  # pytest suite
│   ├── Dockerfile              # CUDA 11.8 base image
│   └── requirements.txt
├── frontend/                   # Next.js 16 application
│   ├── app/                    # App Router pages
│   ├── components/             # React components (ChatInterface, IdleAvatar, etc.)
│   ├── lib/api.ts              # Axios API client
│   └── store/                  # Zustand global state
├── nginx/
│   └── nginx.conf              # Reverse proxy (HTTP → backend/frontend, WebSocket)
├── infrastructure/
│   ├── main.tf                 # AWS Terraform (ECS, RDS, ElastiCache, S3, CloudFront)
│   └── variables.tf
├── scripts/
│   ├── setup_musetalk.sh       # Download MuseTalk models (~9 GB)
│   └── deploy-aws.sh           # One-command EC2 GPU deployment
├── docker-compose.yml          # Development (CPU) — all services
├── docker-compose.prod.yml     # Production overrides (GPU, no bind mounts, logging)
├── deploy.sh                   # ECR push + Terraform deploy (ECS path)
├── .env.example                # Development env template
└── .env.prod.example           # Production env template

🚀 Quick Start

Prerequisites

• Docker & Docker Compose v2+ (recommended)
• OR: Python 3.10+, Node.js 18+, FFmpeg, PostgreSQL, Redis

Option A — Docker / CPU (development)

git clone https://github.com/PunithVT/ai-avatar-system.git
cd ai-avatar-system
cp .env.example .env          # add your ANTHROPIC_API_KEY (or OPENAI_API_KEY)
docker compose up -d

No AWS required. Set USE_LOCAL_STORAGE=true (default) — uploads saved to backend/uploads/.

Want something to talk to immediately? Seed three ready-made demo avatars (AI-generated faces + personalities):

backend/venv/bin/python scripts/seed_demo.py            # or any python with `requests`
backend/venv/bin/python scripts/seed_demo.py --with-voices   # + cloned demo voices

Prebuilt images are also published on every release — ghcr.io/punithvt/ai-avatar-system-backend and …-frontend.

Option B — Manual (development)

# Backend
cd backend
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp ../.env.example ../.env
alembic upgrade head
uvicorn main:app --reload --port 8000

# Frontend (new terminal)
cd frontend
npm install
npm run dev

Option C — Enable MuseTalk Lip-Sync

# Download models (~9 GB, one-time)
bash scripts/setup_musetalk.sh

# Set in .env
AVATAR_ENGINE=musetalk

# Restart
docker compose restart backend

🚀 GPU & AWS Deployment

MuseTalk achieves 30 FPS at 256×256 on a V100-class GPU (source: MuseTalk paper). On CPU it is 30–50× slower. Deploying on AWS gets you genuine real-time performance.

Recommended Instance

Recommended: g5.xlarge Spot (~$72/mo at 8 hrs/day).

One-Command EC2 Deploy

# 1. Launch g5.xlarge with Ubuntu 22.04 LTS, SSH in, then:
bash <(curl -fsSL https://raw.githubusercontent.com/PunithVT/ai-avatar-system/main/scripts/deploy-aws.sh)

# 2. Fill in API keys:
nano /opt/ai-avatar-system/.env.prod

# 3. Redeploy with your keys:
bash /opt/ai-avatar-system/scripts/deploy-aws.sh --update

The script automatically:

• Installs Docker + nvidia-docker2
• Verifies GPU is accessible
• Downloads MuseTalk models (~9 GB)
• Starts all services with GPU passthrough + float16 (2× faster via Tensor Cores)

Manual Production Docker

cp .env.prod.example .env.prod   # fill in your values
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

What docker-compose.prod.yml adds over development:

• GPU reservation (nvidia driver, count=1) for backend + celery-worker
• float16 inference enabled automatically on CUDA → ~2× speedup
• Persistent musetalk_models volume (survive container restarts)
• No source-code bind mounts (runs from built image)
• Log rotation (100 MB max, 5 files)
• Flower disabled (security)

Verify GPU is Working

# Check GPU is visible in container
docker exec avatar-backend python -c "
import torch
print('CUDA:', torch.cuda.is_available())
print('GPU:', torch.cuda.get_device_name(0))
print('VRAM:', round(torch.cuda.get_device_properties(0).total_memory/1024**3,1), 'GB')
"

# Expected on g5.xlarge:
# CUDA: True
# GPU: NVIDIA A10G
# VRAM: 24.0 GB

# Live GPU utilisation
docker exec avatar-backend nvidia-smi

AWS Terraform (ECS Path)

For a fully managed ECS deployment with RDS + ElastiCache + CloudFront:

cd infrastructure
terraform init
terraform apply -var="environment=production"
bash deploy.sh production

🎛️ Avatar Engines

AVATAR_ENGINE selects how video is generated. MuseTalk is the default and stays installed whatever else you add; any engine failure falls back to simple rather than failing the turn.

LiveAvatar (optional)

Alibaba LiveAvatar (Wan2.2-S2V-14B

• LoRA, Apache 2.0) produces higher fidelity than MuseTalk and supports 10,000+ second continuous generation from a single reference image.

bash scripts/setup_liveavatar.sh    # checks VRAM first, then ~60 GB of weights
# then in .env:
AVATAR_ENGINE=liveavatar

It is not a MuseTalk replacement, and MuseTalk remains the default. Two reasons, both structural:

• 48 GB VRAM minimum with FP8, 80 GB without. That is a different hardware tier from the g5.xlarge this project targets.
• It loads the 14B model on every invocation. Upstream's entry point (minimal_inference/s2v_streaming_interact.py) runs once and exits — there is no persistent-worker mode, which is exactly what MuseTalk uses to keep per-turn cost to inference alone. So a turn costs minutes, not milliseconds.

That makes LiveAvatar a fit for the offline Celery render path, where a slow better-looking result is the point, and a poor fit for live conversation. Switching back is one setting: AVATAR_ENGINE=musetalk.

Giving LiveAvatar a persistent worker would mean patching upstream. That is the maintenance debt that got SadTalker removed from this project, so it has deliberately not been done.

🔍 Face Restoration (optional)

MuseTalk regenerates the mouth region at 256×256 and the worker scales it back up to the crop size, while the avatar itself is stored at AVATAR_RESOLUTION (512). The mouth therefore renders at roughly half the resolution of the face around it — and that gap, not the model, is the pipeline's visible quality ceiling. Swapping lip-sync engines doesn't close it; a face restorer does.

bash scripts/setup_face_restore.sh   # installs gfpgan + ~520 MB of weights
# then in .env:
FACE_RESTORE=gfpgan                  # off | gfpgan

gfpgan is deliberately not in requirements.txt. It depends on basicsr, which ships no wheel and builds from source, and whose setup.py collides with the CUDA base image's preinstalled cuda-toolkit — listing it there failed the backend image build for everyone, including people who never enable restoration. It lives in backend/requirements-face-restore.txt and is installed by the setup script, the same way MuseTalk and Chatterbox are.

For the same reason, restoration is a bare-metal / GPU-host feature today: inside the CUDA container the basicsr build needs work that hasn't been done yet.

GFPGAN runs over each composited frame after MuseTalk has blended its generated mouth back in, so it sees a complete face to detect.

It is off by default, deliberately. Restoration runs per frame, so it trades FPS for sharpness, and whether that trade is worth it depends on your GPU and how close to real-time you need to stay. A/B it against FACE_RESTORE=off on the same avatar and audio before leaving it on.

Everything about it degrades rather than breaks: missing weights, a broken gfpgan install, CPU-only hardware, or a single frame that fails to restore all fall back to plain MuseTalk output. Lip-sync is the feature; sharpness is the nicety.

🧠 Conversation Memory

The LLM context is a fixed window of the most recent turns (MAX_CONTEXT_MESSAGES, 60). Anything older used to be dropped outright, so a long conversation quietly forgot its own beginning — the avatar would lose your name twenty minutes in.

Turns leaving the window are now folded into a running summary carried in the system prompt, and persisted to Conversation.summary so a browser refresh doesn't lose it.

CONVERSATION_MEMORY=true          # default
MEMORY_SUMMARY_MAX_CHARS=1500

A summary rather than embeddings, deliberately. It costs one LLM call when the window overflows — roughly once every 30 turns, not once per reply — needs no vector store or embedding model, and works with every provider the project supports. Requiring an embeddings API would have broken the "runs fully local and free with Ollama" path, since Anthropic has no embeddings endpoint and a local embedding model is another dependency to install and keep working.

The tradeoff is honest: a summary is lossy. It carries the thread of a conversation, not verbatim recall of a detail from an hour ago. Fact-level retrieval is what RAG would add, and that remains open on the roadmap.

Failure never costs a turn — if summarisation fails, the window truncates exactly as it did before and the reply proceeds.

🙌 Hands-Free Mode

Tap the Radio button next to the mic and just talk — no tap-to-record. The avatar detects when you start and stop speaking, and speaking over a reply interrupts it.

How turns are decided (frontend/lib/vad.ts):

• Adaptive threshold. The detector measures the room's ambient level for ~700 ms, then listens for speech a margin above it, and keeps tracking the floor while nobody is talking. A fixed threshold works at a desk and fails in a cafe.
• Pause tolerance. A turn ends after ~900 ms of continuous silence, so "I think… maybe we should" stays one turn instead of three.
• Noise rejection. Anything shorter than ~300 ms is discarded rather than sent, so a cough or a door doesn't cost an STT call and an LLM turn.
• Echo cancellation is requested on the mic. Without it the open mic hears the avatar through the speakers and the session talks to itself.

The detector is pure — it takes (level, timestamp) samples and returns turn boundaries — so its timing logic can be exercised against a synthetic trace without a browser or a microphone.

🎤 Voice Cloning

Powered by Chatterbox Multilingual (Resemble AI) — zero-shot voice cloning from a 10-second sample, in 23 languages.

1. Go to Voice tab → Clone Voice
2. Record 10–60 s of clear speech (or upload a WAV/MP3/WebM)
3. Name it → Clone → select it for your session

Every TTS response then uses your cloned voice.

Chatterbox needs its own environment. It is not in requirements.txt, because it pins torch==2.6.0 / transformers==5.2.0 / librosa==0.11.0 while the MuseTalk lip-sync pipeline needs torch 2.2 / transformers 4.37 — the two cannot share one virtualenv. Install it separately:

python -m venv venv-tts
./venv-tts/bin/pip install -r backend/requirements-tts.txt

Without it nothing breaks: TTS falls through to edge-tts (free Microsoft neural voices, CPU-only) and then gTTS. Voice cloning is the only feature that requires Chatterbox, so set TTS_PROVIDER=chatterbox only once it is actually installed.

# REST API
curl -X POST http://localhost:8000/api/v1/voices/clone \
  -F "audio=@my_voice.wav" -F "name=My Voice" -F "language=en"

📡 API Reference

Authentication

POST /api/v1/users/register   { "email": "...", "username": "...", "password": "..." }
POST /api/v1/users/login      form: username=... password=...   → { "access_token": "..." }
POST /api/v1/users/guest      (no body)                         → { "access_token": "..." }
POST /api/v1/users/logout     clears the httpOnly auth cookie

# Every resource route requires an identity — pass the bearer token:
Authorization: Bearer 
# ...or rely on the httpOnly cookie that /login and /guest set for browsers.

/guest backs the UI's Continue as Guest button. It creates a real but anonymous account, so a guest's avatars and transcripts are scoped to them alone — guests never share data with each other. Guest accounts have no usable password (they are reachable only via the token above) and are deleted after GUEST_RETENTION_HOURS of inactivity by the cleanup_guest_accounts task. Set GUEST_ACCOUNTS_ENABLED=false to require sign-up.

Registration rules: passwords are 8–128 characters, usernames are 3–39 characters of [A-Za-z0-9_-] (must start and end alphanumeric), and email addresses are normalised to lower case so User@x.com and user@x.com are the same account.

Avatars

POST   /api/v1/avatars/upload        Upload photo (multipart: file + name)
                                     limits: MAX_UPLOAD_SIZE (413 if over),
                                     ALLOWED_EXTENSIONS, name ≤ 200 chars
GET    /api/v1/avatars/              List avatars
DELETE /api/v1/avatars/{id}          Delete avatar
PUT    /api/v1/avatars/{id}/voice    Assign voice to avatar

Sessions & Messages

POST   /api/v1/sessions/create       { "avatar_id": "..." }
POST   /api/v1/sessions/{id}/end
GET    /api/v1/messages/session/{id}

WebSocket

WS  /ws/session/{session_id}

Client → Server:

{ "type": "text",         "text": "Hello!" }
{ "type": "audio",        "audio": "" }
{ "type": "stop" }                                  // barge-in: cancel the in-flight reply
{ "type": "set_voice",    "voice_id": "" }    // attach a cloned voice (owner-checked)
{ "type": "set_language", "language": "es" }
{ "type": "ping" }

Server → Client:

{ "type": "token",            "token": "Hel" }       // live LLM stream
{ "type": "transcription",    "text": "Hello!" }
{ "type": "message",          "content": "Hi!", "role": "assistant" }
{ "type": "video_chunk_start","total_chunks": -1 }   // -1 = streaming, total unknown
{ "type": "video_chunk",      "chunk_index": 0, "video_url": "...", "text": "Hi!" }
{ "type": "video_chunk_end",  "sent_chunks": 3 }
{ "type": "status",           "message": "Animating…", "stage": "animation" }
{ "type": "tts_fallback",     "engine": "edge-tts", "voice_cloned": false, "message": "…" }
{ "type": "interrupted",      "message": "Previous response interrupted" }
{ "type": "error",            "message": "Something went wrong" }

⚙️ Configuration

Key .env variables:

# LLM
LLM_PROVIDER=anthropic            # anthropic | openai | ollama (local & free)
LLM_MODEL=claude-sonnet-4-6       # or gpt-4o · llama3.1 · qwen2.5 …
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_BASE_URL=                  # e.g. http://localhost:11434/v1 for Ollama / vLLM / LM Studio

# Avatar engine
AVATAR_ENGINE=musetalk            # musetalk (default) | liveavatar (48GB+) | simple (CPU)
MUSETALK_PATH=models/MuseTalk

# TTS — automatic fallback chain: chatterbox → edge-tts → gtts
# chatterbox requires backend/requirements-tts.txt in its own venv (see
# "Voice Cloning"); without it this transparently uses edge-tts.
TTS_PROVIDER=chatterbox

# STT
WHISPER_MODEL=large-v3-turbo      # tiny | base | small | medium | large-v3 | large-v3-turbo

# Storage
USE_LOCAL_STORAGE=true            # false → AWS S3 (+ presigned URLs / CloudFront)
S3_BUCKET_NAME=...

# Auth (≥32 chars enforced at boot)
SECRET_KEY=$(python -c "import secrets; print(secrets.token_hex(32))")
JWT_SECRET_KEY=$(python -c "import secrets; print(secrets.token_hex(32))")
JWT_EXPIRATION_HOURS=24

🛠️ Tech Stack

Frontend

Backend

AI / ML

🧪 Running Tests

cd backend
pytest -v                           # all tests
pytest tests/test_health.py         # single module
pytest --cov=app --cov-report=html  # HTML coverage

📰 What's New

• 2026-06 — Edge-TTS neural fallback chain · local LLMs via Ollama/vLLM · demo avatar seeding · prebuilt GHCR images · cascade-delete + WebM-STT + 429 fixes · SEO/metadata pass
• 2026-05 — httpOnly-cookie auth (XSS-safe) · conversation resume from history · end-to-end WebSocket tests · perf indexes (migration 0002)
• 2026-03 — Chatterbox Multilingual replaces XTTS v2 (23 languages) · MuseTalk persistent worker (models load once) · barge-in interruption · live token streaming

🗺️ Roadmap

• [x] Streaming LLM — TTS + lip-sync start before the LLM finishes (token-by-token) ✅
• [x] Barge-in — interrupt the avatar mid-reply by speaking ✅
• [x] Local LLMs — Ollama / vLLM / LM Studio via OpenAI-compatible API ✅
• [x] Hands-free mode — VAD-driven always-listening with auto end-of-turn (no tap-to-record) ✅
• [ ] WebRTC streaming — sub-second full-duplex audio/video instead of chunked MP4
• [ ] Wav2Lip engine — lighter lip-sync option for weaker GPUs
• [ ] Emotion-driven animation — detected emotion changes facial expression
• [ ] Embeddable widget — drop a talking avatar into any website with 3 lines of JS
• [x] Long-term memory — rolling summary carries context past the window ✅ (RAG + vector DB still open for fact-level recall)
• [ ] UI i18n — the pipeline speaks 23 languages; the UI should too

❓ FAQ

Do I need a GPU?

No — everything runs on CPU. MuseTalk takes 30–90 s/sentence on CPU (the simple engine is instant). For real-time lip-sync, use an AWS g5.xlarge (~$0.30/hr spot) or any 16 GB+ NVIDIA card.

Can I run it with no API key, fully offline?

Yes — set LLM_PROVIDER=ollama, run Ollama (ollama run llama3.1), and you have a fully local, free conversation stack: Whisper STT, local LLM, Chatterbox TTS, MuseTalk video.

How do I get something to talk to quickly?

Run python scripts/seed_demo.py — it creates three demo avatars (AI-generated faces, distinct personalities) and optionally cloned demo voices with --with-voices.

How do I get MuseTalk models?

Run bash scripts/setup_musetalk.sh — downloads ~9 GB of models automatically.

Why does the first response take longer?

The MuseTalk persistent worker loads all models into GPU VRAM on the first request (~60 s on GPU, ~5 min on CPU). Subsequent requests reuse the loaded models.

What happens if the TTS model can't load?

The pipeline degrades gracefully: chatterbox → edge-tts (free Microsoft neural voices) → gTTS. The UI shows a one-time notice when a cloned voice couldn't be applied.

What avatar photo works best?

A clear, well-lit frontal face photo (JPEG/PNG/WebP). Avoid sunglasses or heavy occlusion.

🤝 Contributing

Contributions welcome! Read CONTRIBUTING.md before opening a PR.

git clone https://github.com/PunithVT/ai-avatar-system.git
git checkout -b feat/my-feature
# make changes + tests
git commit -m "feat(backend): add my feature"
git push origin feat/my-feature

💜 Sponsor

AvatarAI is MIT-licensed and developed in the open. If it saves you time, or your company is running it in production, sponsoring directly funds the work that is expensive to do for free:

• GPU time for testing the MuseTalk lip-sync pipeline across resolutions and hardware
• Model evaluation as new TTS and STT releases land, so the defaults stay current instead of drifting
• Maintenance — dependency and security upkeep, issue triage, and review

Not in a position to sponsor? Starring the repo, filing a good bug report, or writing about what you built with it all genuinely help.

📄 License

MIT © 2026 — see LICENSE for details.

If AvatarAI saves you time or inspires your project, please ⭐ star the repo.

Built with FastAPI · Next.js · MuseTalk V1.5 · Chatterbox · Whisper · Claude AI