🧠 Avatoon

Avatoon is a lightweight React Three Fiber component for rendering animated 3D avatars with real-time viseme-driven lip-sync. It supports lifelike head movements, morph target control, and optional goal-based gestures (e.g., flexing, sleeping), making it ideal for voice assistants, interactive characters, or storytelling apps.

🎮 Try the live demo →

demo video

✨ Features

• 🎤 Real-time lip-sync using phoneme-viseme mapping
• 🧍 Subtle head motion animation while talking
• 👁️ Automatic eye-blinking (when the model exposes blink morph targets)
• 🎛️ Imperative play() / stop() control via a ref
• 🌐 GLTF model support via useGLTF
• ⚛️ Plug-and-play with React Three Fiber + Drei
• 🎯 Goal-based gestures like "Muscle" or "Sleep"

📦 Installation

npm install avatoon

Requirements (peer dependencies)

Avatoon builds on React and the React Three Fiber ecosystem. Install these alongside it if your project doesn't already have them:

npm install react react-dom three @react-three/fiber @react-three/drei

Peer dependency	Supported version
react / react-dom	>=18
three	>=0.153.0
@react-three/fiber	>=8.0.0
@react-three/drei	>=9.0.0

Runs in any browser with WebGL support. Written in TypeScript — type definitions ship with the package.

🚀 Usage

Avatoon — full avatar with audio-synced lip-sync

import { Avatoon } from "avatoon";

const visemeJson = {
  visemes: [
    { time: 0, viseme: "X" },
    { time: 1.3, viseme: "A" },
    { time: 1.367, viseme: "C" },
  ],
  audio_base64: "", // base64-encoded WAV (optional)
};

export default function App() {
  return (
    <div style={{ width: "100%", height: 400 }}>
      <Avatoon
        glbUrl="https://raw.githubusercontent.com/khaledalam/avatoon/main/test/assets/placeholder-avatar.glb"
        goal="Normal"
        visemeJson={visemeJson}
        showPlayVoiceButton
        onRenderComplete={() => console.log("Render Completed!")}
      />
    </div>
  );
}

LipSyncAvatoon — standalone lip-sync (no audio required)

A self-contained component with a built-in Start/Stop button that animates procedural mouth movement. Great for quick demos or "talking head" idle states.

import { LipSyncAvatoon } from "avatoon";

export default function App() {
  return (
    <div style={{ width: "100%", height: 400 }}>
      <LipSyncAvatoon glbUrl="/avatar.glb" />
    </div>
  );
}

Imperative control (ref)

Drive playback from your own UI instead of the built-in button by passing a ref — it exposes play(), stop(), and toggle():

import { useRef } from "react";
import { Avatoon, type AvatoonHandle } from "avatoon";

function App() {
  const avatar = useRef<AvatoonHandle>(null);

  return (
    <>
      <button onClick={() => avatar.current?.play()}>Speak</button>
      <button onClick={() => avatar.current?.stop()}>Stop</button>
      <div style={{ width: "100%", height: 400 }}>
        <Avatoon ref={avatar} glbUrl="/avatar.glb" visemeJson={visemeJson} />
      </div>
    </>
  );
}

To run the example app locally:

npm run example

🧩 API

Avatoon(props)

Prop	Type	Default	Description
glbUrl	string	(required)	URL to the .glb avatar file (T1 or T2)
goal	AvatoonGoal	"Normal"	Motion preset: "Normal", "Muscle", or "Sleep"
onRenderComplete	() => void	undefined	Callback fired when avatar finishes rendering
onError	(error: Error) => void	undefined	Fired if the model fails to load (bad glbUrl) instead of crashing
visemeJson	VisemeData	undefined	JSON structure for syncing visemes with audio playback
showPlayVoiceButton	boolean	false	If true, renders a play/stop voice button in the scene
fov	number	24	Camera vertical field-of-view
cameraPosition	[number, number, number]	[0, 1.45, 2.3]	Camera position [x, y, z]
cameraTarget	[number, number, number]	[0, 1.35, 0]	OrbitControls look-at target [x, y, z]
environmentPreset	EnvironmentPreset	"sunset"	drei lighting preset (city, dawn, night, …)
environmentFiles	string | string[]	undefined	Custom HDR/EXR file(s); overrides the preset
environmentBackground	boolean	false	Render the environment as the scene background

Tune fov / cameraPosition / cameraTarget to frame your own avatar model. You can also drive playback imperatively via a ref.

LipSyncAvatoon(props)

A lightweight, self-contained variant that drives procedural mouth movement (no visemeJson or audio needed) and renders its own Start/Stop button.

Prop	Type	Default	Description
glbUrl	string	"/avatar.glb"	URL to the .glb avatar file (T2)
fov	number	24	Camera vertical field-of-view
cameraPosition	[number, number, number]	[0, 1.45, 2.3]	Camera position [x, y, z]
cameraTarget	[number, number, number]	[0, 1.35, 0]	OrbitControls look-at target
onError	(error: Error) => void	undefined	Fired if the model fails to load
environmentPreset / environmentFiles / environmentBackground	—	—	Same environment options as Avatoon

Exposed building blocks

For advanced composition you can also import the lower-level pieces: AvatoonModel, CameraFovAnimator, SceneEnvironment, and AvatoonErrorBoundary.

👤 Avatar Types

• T1 (Static Face - Realistic)
  • Most photorealistic
  • No facial morphing
  • Lightweight
  • 
• T2 (Blendshape Face - Expressive)
  • Separate eyeballs and mouth
  • Supports morph targets / ARKit visemes
  • Slightly less realistic but animatable
  • 

📘 Types

interface VisemeData {
  visemes: Array<{ time: number; viseme: string | null }>;
  audio_base64?: string;
}

🎚️ Generating viseme data

visemeJson is what drives the mouth animation. Each entry pairs a time (in seconds, from the start of the audio) with a single-letter viseme code. Codes are mapped onto the model's morph targets; unknown or silent codes simply rest the mouth.

Code	Mouth shape / example sound
A	open — "a" as in apple
B	lips together — p, b, m
C	ch, sh, j
D	d, t, th
E	"eh", "ae"
F	f, v
G	g, k
I	"ee"
J	r, l, y
K	"oo", "u"
H / X	silence / rest

The standard Oculus / Ready Player Me viseme names are also accepted as codes directly — sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, I, O, U — which is what the converter helpers below emit.

Entries must be ordered by ascending time. Optionally provide the spoken audio as a base64-encoded WAV via audio_base64 to play it in sync (used when showPlayVoiceButton is enabled).

{
  "visemes": [
    { "time": 0.00, "viseme": "X" },
    { "time": 0.12, "viseme": "B" },
    { "time": 0.20, "viseme": "A" },
    { "time": 0.35, "viseme": "I" }
  ],
  "audio_base64": "UklGR... (optional WAV)"
}

From a speech engine

Rather than hand-authoring the timeline, generate it from a TTS/lip-sync engine. Avatoon ships converters for the most common sources:

import {
  Avatoon,
  fromAzureVisemes,   // Azure Speech SDK viseme events
  fromPollySpeechMarks, // AWS Polly speech marks
  fromRhubarb,        // Rhubarb Lip Sync JSON
} from "avatoon";

// Azure: collect events from the Speech SDK's `visemeReceived` callback
const visemeJson = fromAzureVisemes(azureEvents); // [{ visemeId, audioOffset }]

// AWS Polly: pass the `viseme` speech marks
const visemeJson = fromPollySpeechMarks(pollyMarks); // [{ time, value, type }]

// Rhubarb: pass its JSON output (or just the mouthCues array)
const visemeJson = fromRhubarb(rhubarbOutput); // { mouthCues: [{ start, end, value }] }

Each returns a VisemeData object ready to hand to <Avatoon visemeJson={...} />. Times are normalized to seconds. Add audio_base64 yourself if you want synced playback.

Model requirement (T2): audio-synced lip-sync needs a .glb whose mesh exposes viseme_* morph targets (ARKit / Oculus / Ready Player Me naming — e.g. viseme_aa, viseme_PP, viseme_CH). T1 models have no morph targets, so use LipSyncAvatoon or head-motion only.

🤝 Contribution

Pull requests are welcome! See CONTRIBUTING.md for local setup and development workflow, and please review our Code of Conduct. To report a security issue, see SECURITY.md.

💖 Support

If Avatoon is useful to you, consider supporting its development — it helps a lot and keeps the project maintained:

• ⭐ Star the repo — the easiest way to help
• 💛 Sponsor on GitHub
• ☕ Buy Me a Coffee
• 💵 PayPal

📄 License

MIT © Khaled Alam

🛡️ Author

Khaled Alam

📧 khaledalam.net@gmail.com
🌍 Website | LinkedIn | X(Twitter)