Bagaimana AI Lip Sync Sebenarnya Bekerja
7 Juni 2026 · 8 min read
Indonesia produces some of Southeast Asia's most compelling drama — sinetron series with tens of millions of viewers, web dramas on YouTube and Netflix, action films that fill cinemas from Medan to Makassar. Yet for most of the world, this content might as well not exist. The language barrier is total.
Traditional dubbing solves the audio half of that problem — a French voice actor records over the original Bahasa Indonesia track — but it ignores the visual half entirely. The actors' mouths still move to Indonesian words. To a French viewer, the disconnect is jarring. They feel it even if they can't articulate why.
AI lip sync exists to close that gap. It regenerates an actor's mouth movements to match a new audio track in a different language — so a viewer in Lyon watching a sinetron sees an Indonesian actor whose lips actually match the French they're hearing.
Humans have an extraordinary sensitivity to lip-audio synchronisation. It is not a learned skill — it is neurological. When the visual signal (mouth shape) conflicts with the audio signal (phoneme heard), the brain does not simply ignore one input. It fuses them into a third, erroneous perception. This is called the McGurk Effect.
Picture a tense scene from a popular Indonesian romance drama — the kind of close-up shot where the camera holds on an actor's face as they say, "Aku tidak bisa melupakan kamu." Now imagine the dubbed French audio says, "Je ne peux pas t'oublier." The lip movements for those two phrases are entirely different. Even a French viewer who speaks no Indonesian will register the mismatch unconsciously — and trust the content less.
Even a single frame of drift — about 40 milliseconds — is perceptible to most viewers. AI lip sync must therefore operate at frame-level precision.
The state-of-the-art approach uses two parallel encoders that process audio and video independently, then combines them in a generator to produce new mouth movements.
The audio track is broken into approximately 200-millisecond chunks. Each chunk is converted into a vector that represents not just the raw sound, but the phoneme being spoken — and crucially, the visemethat corresponds to it. A viseme is the visual mouth position that a phoneme maps to. Multiple phonemes can share the same viseme: the "p" in Paris and the "b" in bonjour look almost identical on the lips.
A separate encoder processes the video frames, but with the original mouth region deliberately masked. This prevents the model from cheating by copying the source mouth. What the visual encoder captures instead is everything else that defines this specific person: facial structure, skin tone, head pose, lighting conditions, and emotional expression. For Indonesian drama, this means preserving the exact look of an actor — even across different lighting rigs on a studio set in Jakarta versus an outdoor shoot in Bali.
Cross-lingual dubbing from Bahasa Indonesia to a European language is among the most demanding cases. Bahasa Indonesia is phonetically regular — almost every letter maps to one sound. French is decidedly not. The mouth shapes required are often entirely different.
The table below shows real phrases from Indonesian romance drama alongside their French dubbed equivalents, and why the visual transformation is non-trivial:
| Bahasa Indonesia | French Equivalent | Mouth Shape Challenge |
|---|---|---|
| "Aku cinta kamu" | "Je t'aime" | "aku" = wide-open A; "je" = pursed, forward lips — opposite shapes |
| "Sudah terlambat" | "C'est trop tard" | "sudah" = soft open SU; "c'est" = almost no lip movement, near-closed |
| "Kenapa kamu pergi?" | "Pourquoi tu pars?" | "kenapa" = 3 open syllables; "pourquoi" = tight rounded U followed by silence |
| "Jangan tinggalkan aku" | "Ne me quitte pas" | "jangan" = jaw drop; "ne me" = barely open, nasal — nearly invisible movement |
A naive dubbing system that voice-swaps without re-lipsyncing will produce all four of these mismatches at once. In a 45-minute sinetron episode, that is thousands of misaligned frames.
The generator receives the audio embedding (what should be said in French) and the masked visual embedding (who is saying it) and synthesizes a new mouth region that is coherent with both. It does not warp or stretch existing pixels — it generates new mouth imagery that matches the target phonemes while respecting the actor's face shape, skin tone, and the current lighting conditions of that shot.
The result is composited back onto the original frame. The rest of the actor's face — eyes, nose, hair, neck — is untouched.
Traditional generative models train a quality judge and a generator simultaneously. The problem is that both co-evolve — and early in training, neither is strong enough to force real quality. The generator learns shortcuts.
The breakthrough was using a pre-trained, frozen lip sync expert as the sole quality authority. This expert had already been trained on enormous volumes of real, correctly-synced video. It arrived fully formed — it could immediately and accurately judge whether a generated mouth matched the audio. The generator had no choice but to genuinely satisfy this expert from the first iteration, rather than fooling a weak co-trained judge.
Freezing the quality authority also side-stepped the instability that makes generative model training notoriously difficult. One network was always the authority; the other always the student.
Most dubbing requests involve actors the model has never seen during training — a new sinetron lead, a web drama cast from a small production house in Surabaya. The model must generalize perfectly to a face it has never encountered.
The risk is that models tend to "average" faces from their training data, subtly eroding the distinctiveness of a specific actor. A viewer who has watched 40 episodes of a drama series will notice if the protagonist's face looks slightly different. Solutions include strong identity branches trained with face-recognition networks, and diverse training data that reduces demographic gaps.
Korean drama's global rise was built on subtitle fandom — dedicated communities who translated episodes within hours of broadcast. AI dubbing offers a different path: one where Indonesian drama is not just subtitled but localized — where a viewer in Dakar, São Paulo, or Marseille watches an Indonesian actor whose lips match the language they speak.
That is a fundamentally different viewing experience. It is what transforms a subtitled curiosity into a mainstream hit.
The science covered here is what DubSync is built on — and why the results go meaningfully beyond a simple voice swap.