This article has been accepted for publication in IEEE Computer Graphics and Applications but has not

yet been fully edited.

IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication. 1

A Practical Model for Live Speech Driven

Lip-sync
Li Wei, and Zhigang Deng, Senior Member, IEEE

Abstract—In this article we introduce a simple, efficient, yet practical
phoneme-based approach to generate realistic speech animation in
real-time based on live speech input. Specifically, we first decom-
pose lower-face movements into low-dimensional principal component
spaces. Then, in each of the retained principal component spaces, we
select the AnimPho with the highest priority value and the minimum
smoothness energy. Finally, we apply motion blending and interpolation
techniques to compute final animation frames for the currently inputted
phoneme. Through many experiments and comparisons, we demon-
strate the realism of synthesized speech animation by our approach as
well as its real-time efficiency on an off-the-shelf computer.
Index Terms—Facial animation, speech animation, live speech driven,
talking avatars, virtual humans, data-driven
1 I NTRODUCTION
In signal processing and speech understanding communities,
several approaches have been proposed to generate speech
animation based on live acoustic speech input. For exam-
ple, based on a real-time recognized phoneme sequence,
researchers use simple linear smoothing functions to produce
corresponding speech animation [1], [2]. Meanwhile, a number
of approaches train statistical models (e.g., neural networks)
to encode the mapping between acoustic speech features and
facial movements [3], [4]. These approaches demonstrated
their real-time runtime efficiency on an off-the-shelf computer;
however, their performance is highly speaker-dependent due
to the individual-specific nature of the chosen acoustic speech
features. Furthermore, due to the insufficient visual realism of
these approaches, practically they are less suitable for graphics
and animation applications.
Challenges of live speech driven lip-sync. First, live
speech driven lip-sync imposes additional technical challenges
than the off-line case where expensive global optimization
techniques can be employed to solve the most plausible
speech motion corresponding to novel spoken or typed input.
In contrast, it is extremely difficult, if not impossible, to
directly apply such global optimization techniques to live
speech driven lip-sync applications, since the forthcoming
(unavailable yet) speech content cannot be exploited during
• L. Wei and Z. Deng are with the Computer Graphics and Interactive Media
Lab and the Department of Computer Science, University of Houston,
Houston, TX 77204-3010.
• E-mail: zdeng4@uh.edu.
Digital Object Indentifier 10.1109/MCG.2014.105
the synthesis process. Second, live speech driven lip-sync al-
gorithms need to be highly efficient to ensure a real-time speed
of generating speech animation on an off-the-shelf computer,
while off-line speech animation synthesis algorithms do not
need to meet such a tight constraint. Compared with the case
of the forced phoneme alignment for pre-recorded speech,
the last challenge comes from the low accuracy of state-of-
the-art live speech phoneme recognition systems (e.g., the
Julius system (http://julius.sourceforge.jp) or the HTK toolkit
(http://htk.eng.cam.ac.uk)).
In order to quantify the phoneme recognition accuracy dif-
ference between the pre-recorded speech and live speech cases,
we performed an empirical study as follows. We randomly se-
lected 10 pre-recorded sentences and extracted their phoneme
sequences using the following two different approaches: (1)
The Julius system was used to do forced phoneme-alignment
on the pre-recorded 10 speech clips (called offline phoneme-
alignment), and (2) The same Julius system was used as a
real-time phoneme recognition engine. In other words, by
simulating the same pre-recorded speech clip as live speech, it
outputted phonemes sequentially while the speech was being
fed into the system. Then, by taking the offline phoneme-
alignment results as the ground-truth, we can compute the
accuracies of the live speech phoneme recognition in our
experiment. As illustrated in Figure 1, the live speech phoneme
recognition accuracy of the same Julius system varies from
45% to 80%. Further empirical analysis did not show any
patterns of incorrectly recognized phonemes (that is, which
phonemes are often recognized incorrectly in live speech
case). This empirical finding implies that in order to pro-
duce satisfactory live speech driven speech animation results,
any phoneme-based algorithm must take the relatively low
phoneme recognition accuracy (in the case of live speech)
into design consideration, and it should be able to perform
certain self-correction at runtime, since some phonemes could
be incorrectly recognized and inputted into the algorithm, in
a less predictable manner.
Inspired by the above research challenges, in this paper we
propose a practical phoneme-based approach for live speech
driven lip-sync. Besides generating realistic speech animation
in real-time, our phoneme-based approach can straightfor-
wardly handle speech input from different speakers, which
is one of the major advantages of phoneme-based approaches
over acoustic speech feature driven approaches [3], [4]. Specif-
ically, we introduce an efficient, simple algorithm to compute
0272-1716/$26.00 2014 IEEE
 This article has been accepted for publication in IEEE Computer Graphics and Applications but has not yet been fully edited.
IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication.
1
Live Phoneme Recognition Accuracy
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1 2 3 4 5 6
Sentence Number
Fig. 1. Illustration of the calculated live speech phoneme
recognition accuracies of the Julius system, compared to
its offline phoneme-alignment results.
the priority of each motion segment and select the plausible
segment based on the phoneme information that is sequentially
recognized at runtime. Through our experiments and various
comparisons, we evaluate the accuracy and efficiency of our
live speech driven lip-sync approach. Compared to existing lip-
sync approaches (reviewed in Section 2), the main advantages
of our approach are its efficiency, simplicity practicalness, and
capability of handling live speech input in real-time.
2 R ELATED W ORK
Visual speech animation synthesis. The essential part of
visual speech animation synthesis is how to model the speech
co-articulation effect. Conventional viseme driven approaches
need users to first carefully design a set of visemes and then
employ interpolation functions [5] or co-articulation rules to
compute in-between frames for speech animation synthesis [6].
However, a fixed phoneme-viseme mapping scheme is often
insufficient to model the co-articulation phenomenon in human
speech production. As such, the resulting speech animations
often lack variance and realistic articulation.
Instead of interpolating a set of pre-designed visemes, one
category of data-driven approaches generate speech animations
by optimally selecting and concatenating motion units from
a pre-collected database based on various cost functions [7],
[8], [9], [10], [11]. The second category of data-driven speech
animation approaches learn statistical models from data [12],
[13]. All these data-driven approaches have demonstrated
noticeable successes for offline pre-recorded speech animation
synthesis. Unfortunately, they cannot be straightforwardly ex-
tended for live speech driven lip-sync. The main reason is that
they typically utilize some forms of global optimization tech-
niques to synthesize the most optimal speech-synchronized
facial motion. In contrast, in the case of live speech driven
lip-sync, forthcoming speech (or phoneme) information is
simply unavailable; as such, directly applying such global
optimization based algorithms would be technically infeasible.
Live speech driven facial and gesture animation. A num-
ber of algorithms have been proposed to generate live speech
driven speech animations. Besides pre-designed phoneme-
viseme mapping [1], [2], various statistical models (e.g., neural
networks [3], nearest-neighbor search [4], and others) have
been trained to learn audio-visual mapping for this purpose.
The common disadvantages of these approaches is that the
Digital Object Indentifier 10.1109/MCG.2014.105
2
acoustics features used for training these statistical audio-to-
visual mapping models are highly speaker-specific, as pointed
out by Taylor et al. [10].
3 DATA ACQUISITION AND P REPROCESSING
We acquired a training facial motion dataset for this work
7 8 9 10
using an optical motion capture system. We attached over
100 markers on the face of a female native English speaker.
3D rigid head motion was eliminated using a SVD-based
statistical shape analysis method [9]. The captured subject
was guided to speak a phoneme-balanced corpus consisting of
166 sentences with neutral expression. The obtained dataset
contains 72,871 motion frames (about 10 minutes recording,
120 frames per second). Phoneme labels and the durations of
the phonemes were automatically extracted from the simulta-
neously recorded speech data.
As illustrated in Figure 3(a), 39 markers in the lower
face region are used in this work, which results in a 117-
dimensional feature vector for each motion frame. We apply
Principal Component Analysis (PCA) to reduce the dimension
of the motion feature vectors, which allows us to obtain
a compact representation by only retaining a small number
of principal components. In this work, we keep the five
most significant principal components to cover 96.6% of the
variance of the motion dataset. It is noteworthy that all the
other processing steps in this writing are in parallel performed
in each of the retained five principal component spaces.
Motion segmentation: For each recorded sentence, based
on its phoneme alignment result, we segment its motion
sequence accordingly and extract a motion segment for each
phoneme occurrence. For the purpose of motion blending later,
we keep an overlapping region between two neighboring mo-
tion segments. Currently, we set the length of the overlapping
region to one frame, as illustrated in Figure 2.
s i -1 s i+1
… …
si
Fig. 2. Illustration of motion segmentation. The grids
represent motion frames. si denotes the motion segment
corresponding to phoneme pi . When we segment a mo-
tion sequence based on its phoneme timing information,
we keep one overlapping frame (grids with slash filled)
between two neighboring segments. We evenly sample
five frames (grids with red color) to represent a motion
segment.
Motion segment normalization: Because the duration of a
phoneme is typically short (in our dataset, the average duration
of a phoneme is 109 milliseconds), we can use a small number
of evenly sampled frames to represent the original motion. In
our approach, we down-sample a motion segment by evenly
selecting five representative frames (see Figure 2). To the
end, we represent each motion segment si as a vector vi that
concatenates the PCA coefficients of the five sampled frames.
0272-1716/$26.00 2014 IEEE
 This article has been accepted for publication in IEEE Computer Graphics and Applications but has not yet been fully edited.
IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication.
Clustering: We apply K-Means clustering algorithm to the
motion vector dataset {vi : i = 1...n}, where n is the total
number of motion vectors. Euclidean distance is used to
compute the distance between two motion vectors. The center
of each obtained motion cluster is called an AnimPho in this
writing. Finally, we obtain a set of AnimPhos {ai : i = 1...m},
where m is the number of AnimPhos and ai is the i-th
AnimPho.
After the above clustering step, we essentially convert the
motion dataset to a set of AnimPho sequences. Then, we
further pre-compute the following two data structures:
• AnimPho transition table, T . It is a m × m table, where
m is the number of AnimPhos. T (ai , a j ) = 1 if the An-
imPho transition < ai , a j > exists in the motion dataset,
otherwise T (ai , a j ) = 0. In the remainder of this paper, we
call two AnimPhos ai and a j are connected if and only if
T (ai , a j ) = 1. We also define T (ai ) = {a j |T (ai , a j ) = 1}
as the set of AnimPhos that are connected to ai .
• Phoneme-AnimPho mapping table, L. It is a h × m table,
where h is the number of used phonemes (43 English
phonemes are used in this work). L(pi , a j ) = 1 if the
phoneme label of a j is phoneme pi in the dataset,
otherwise L(pi , a j ) = 0. Similarly, we define L(pi ) =
{a j |L(pi , a j ) = 1} as the set of AnimPhos that have the
phoneme label pi .
(a) (b)
Fig. 3. Illustration of the used facial motion dataset. (a)
Among the 102 markers, the 39 green markers are used
in this work. (b) Illustration of the average face markers.
4 L IVE S PEECH D RIVEN L IP -S YNC
In this work, two steps are used to generate lip motion based
on live speech input: AnimPho selection, and motion blending.
We will describe two steps in detail in this section.
4.1 AnimPho Selection
Assuming phoneme pt arrives at the lip-sync system at time t,
our algorithm needs to find its corresponding AnimPho, at , to
animate pt . A naive solution would be to take the AnimPho
set, φ (pt )=L(pt ) ∧ T (at−1 ), which is the set of AnimPhos that
have the phoneme label pt and have a natural connection with
at−1 , and choose one of them as at by minimizing a cost
function (e.g., the smoothness energy defined in Eq. 1). If
Digital Object Indentifier 10.1109/MCG.2014.105
3
φ (pt ) is empty (called a miss in this paper), then we reassign
all the AnimPhos (regardless their phoneme labels) as φ (pt ).
at = argmin dist(E(at−1 ), B(η)), (1)
η∈φ (pt )
where dist(E(at−1 ), B(η)) returns the Euclidean distance
between the ending frame of at−1 and the starting frame of
η. If L(pt ) ∧ T (at−1 ) is not empty, this method can ensure
the co-articulation by finding the AnimPho at that is naturally
connected to at−1 in the captured dataset. However, it does
not work well in practical applications. The main reason is
that, the missing rate of this method is typically very high
due to the limited size of the captured dataset; as a result, the
synthesized animations are often incorrectly articulated and
over-smoothed (i.e., only minimizing the smoothness energy
defined in Eq. 1).
p1 p2 p3
1 4 7
2 5 8
3 6 9
miss miss
Fig. 4. An illustrative example of the naive AnimPho
selection strategy: p1 to p3 on the top is the phonemes
that sequentially come to the system. Squares below
each phoneme pi is L(pi ), and the number in the squares
denotes the index of AnimPho. Dotted directional lines
denote existing AnimPho transitions in the dataset. The
AnimPhos with green color are the AnimPhos that are
selected based on the naive AnimPho selection strategy.
A simple example of this naive AnimPho selection strategy
is illustrated in Figure 4. Assuming we initially select the
AnimPho #3 for phoneme p1 . When p2 comes, we find that
none of AnimPhos in L(p2 ) can connect to the AnimPho #3 in
the dataset, so we select the one with a minimum smoothness
energy (assuming it is the AnimPho #6). When p3 comes,
again we find that none of AnimPhos in L(p3 ) can connect to
the AnimPho #6, so we select an AnimPho (assuming it is the
AnimPho #7) based on the smoothness energy. The missing
rate of this simple example would be 100%, and the natural
AnimPho connection information (dotted directional lines in
Figure 4) is not exploited in this example.
To solve both the over-smoothness and inaccurate artic-
ulation issues, we need to properly utilize the AnimPho
connection information especially when the missing rate is
high. In our approach, we compute the priority value, vi , for
each AnimPho ai ∈ L(pt ) based on the pre-computed transition
information. Specifically, the priority value of an AnimPho ai
is defined as the length of the longest connected AnimPho
0272-1716/$26.00 2014 IEEE
 This article has been accepted for publication in IEEE Computer Graphics and Applications but has not yet been fully edited.
IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication.
sequence corresponding to the observed phoneme sequence
(up to pt ). The priority value vi of ai is calculated as follows:
( 0.6
max(v j + 1) ∃a j ∈ L(pt−1 ), ai ∈ T (a j )
vi = (2)
1 Otherwise
Then, we select AnimPho at by taking both the priority
value and the smoothness term into consideration. In our
approach, we prefer to select the AnimPho with the highest
priority value; if multiple AnimPhos have the same high pri-
ority values, we select the smoothest one when it is connected
with at−1 .
p1 p2 p3
1 1 4 2 7 1
2 1 5 2 8 3
3 1 6 1 9 1
miss
Fig. 5. An illustrative example of our priority-incorporated
AnimPho selection strategy: the number in a red box is
the priority value of an AnimPho.
We use the same illustrative example in Figure 5 to explain
our algorithm. We initially select the AnimPho #3 for p1 .
When p2 comes, we first compute the priority value for each
AnimPho in L(p2 ) using AnimPho transition information.
The priority value of the AnimPho #4, v4 , is 2, since the
AnimPho sequence < 1, 4 > corresponds to the observed
phoneme sequence < p1 , p2 >. Similarly, we can compute the
priority value of the AnimPho #5 as 2. After we compute
priority values, we select one AnimPho for p2 . Although
we find that none of AnimPhos in L(p2 ) is connected to
the AnimPho #3, instead of selecting the AnimPho #6 that
minimizes the smoothness term, we choose the AnimPho #4
or the AnimPho #5 (both have the same higher priority value).
Assuming the AnimPho #5 has a smaller smoothness energy
than the AnimPho #4, we select the AnimPho #5 for p2 . When
phoneme p3 comes, AnimPho #8 has the highest priority
value, so we simply select the AnimPho #8 for p3 .
As shown in Figure 6, compared with the naive AnimPho
selection strategy, our approach can significantly reduce the
missing rate. Still, the missing rate of our approach is sub-
stantial (e.g., around 50%), due to the following two reasons:
• Unpredictability of the next phoneme. At the AnimPho
selection step of live speech driven lip-sync, the next
(forthcoming) phoneme information is unavailable, so we
cannot predict which AnimPho, as the best selection for
the current step, that would provide the optimal AnimPho
transition to the next phoneme.
Digital Object Indentifier 10.1109/MCG.2014.105
4
Our method
0.9 The naive method
0.8
0.7
Missing Rate
0.5
0.4
0.3
0.2
0.1
0
1 2 3 4 5 6 7 8
Sentence#
Fig. 6. The missing rate comparison of 8 sentences
between the naive AnimPho selection strategy and our
approach.
The limited size of the motion dataset. At most practical
•
applications, the acquired facial motion dataset typically
has a limited size due to data acquisition cost and other
reasons. For example, the average number of outgoing
transitions from an AnimPho in our dataset is 3.45. There-
fore, once we pick an AnimPho for pt−1 , the forward
AnimPho transitions for pt are limited (i.e., on average
3.45).
In theory, in order to dramatically reduce the missing rate
(i.e., close to zero), the above two issues need to be well
taken care of. In this work, rather than capturing a much larger
facial motion dataset to reduce the missing rate, we focus on
how to utilize a limited motion dataset to produce acceptable
live speech driven lip-sync results. In follow-up section, we
will introduce a motion blending step to handle the motion
discontinuity incurred by the missing event.
4.2 Motion Blending
In this section, we describe how to generate the final facial ani-
mation using motion blending technique, based on the selected
AnimPho at (refer to Section 4.1) and the duration of phoneme
pt (outputted from a live speech phoneme recognition system
such as Julius). Specifically, this technique consists of three
steps: (1) The first step is to select an AnimPho bt that both
connects to at−1 and has the most similar motion trajectory
with at ; (2) the second step is to generate an intermediate
motion segment mt based on at−1 , at , and bt ; (3) the final
step is to do motion interpolation on mt to produce the desired
number of animation frames for pt , based on the inputted
duration of pt .
Selection of bt . As mentioned previously, at runtime our
live speech driven lip-sync approach still has a non-trivial
missing rate (e.g., on average about 50%) at the AnimPho
selection step. To solve the possible motion discontinuity issue
between at−1 and at , we need to identify and utilize an existing
AnimPho transition < at−1 , bt > in the dataset such that bt has
the most similar motion trajectory with at . Our rationale is that,
the starting frame of bt can ensure a smooth connection with
the ending frame of at−1 , and bt can also have a similar motion
trajectory with at at most places. We identify the AnimPho bt
by minimizing the following cost function:
bt = argmin dist(η, at ), (3)
η∈T (at−1 )
0272-1716/$26.00 2014 IEEE
 This article has been accepted for publication in IEEE Computer Graphics and Applications but has not yet been fully edited.
IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication. 5

where dist(η, at ) returns the Euclidean distance between at

at
and η. Note that if at ∈ T (at−1 ), which means there exists
a natural AnimPho transition from at−1 to at in the dataset, bt

Motion
Eq. 3 is guaranteed to return bt = at .
Generation of an intermediate motion segment, mt .
After bt is identified, we first need to modify bt to ensure the
motion continuity between the previous animation segment,
pt-1 pt Time
at−1 , and the animation for pt that we are going to synthesize.
In this writing, we use bt0 to denote the modified version of
at
bt . Then, we need to generate an intermediate motion segment
b t’
mt by linearly blending bt0 and at .

Motion
All the AnimPhos (including bt , at−1 , and at ) store the
center of corresponding motion segment clusters, as described
in Section 3; they do not retain the original motion trajectory
in the dataset. As a result, the starting frame of bt and the pt-1 pt Time
ending frame of at−1 may be slightly mismatched although the
AnimPho transition < at−1 , bt > indeed exists in the captured at mt
motion dataset (refer to the top panel of Figure 7). To make the b t’ dt

starting frame of bt perfectly matched with the ending frame
of at−1 , we need to adjust (via a translation transformation)
bt by setting the starting frame of bt equivalent to the ending
frame of at−1 (refer to the middle panel of Figure 7).
Then, we linearly blend two motion segments, bt0 and at ,
to obtain an intermediate motion segment mt , as illustrated in
the bottom panel of Figure 7. As described in Section 3, we
use 5 evenly sampled frames to represent a motion segment.
Therefore, this linear blending can be described using the
following Equation 4.
i−1 i−1
mt [i] = (1 − ) × bt0 [i] + × at [i], (1 ≤ i ≤ 5),
4 4
where mt [i] is the i-th frame of the motion segment mt .
Motion interpolation. By taking the down-sampled
motion segment mt as input, we use an interpolation method to
compute the final animation frames for phoneme pt based on
its duration. We calculate the required number of frames N in
final animation as N = duration(pt )×FPS, where duration(pt )
denotes the duration of phoneme pt , and FPS is the frames
per second in the resulting animation (e.g., 30 FPS).
Motion
pt-1 pt Time
Fig. 7. Illustration of the motion blending and interpolation
process. Top: After the AnimPho selection step, we can
see that either at or bt has a smooth transition from the
previous animation segment (denoted as the black solid
line). Middle: We set the starting frame of bt equivalent to
the ending frame of the previous animation segment (for
phoneme pt−1 ). Bottom: We linearly blend at and bt0 to
(4)
obtain a new motion segment mt and further compute the
derivatives of mt . Finally, we apply Hermit interpolation to
obtain in-between frames.
Algorithm 1 Motion Interpolation
Input: Sample frames, mt ; the derivatives of sample frames,
dt ; the target number of frames, N.
Output: The interpolated sequence, St .
1: for i = 1 → N do

In this work, we use Hermit interpolation to compute in-
between frames between two sample frames. The derivatives
of the five sample frames are computed as follows:

dt−1 [5]
 if i = 1;
dt [i] = (mt [i + 1] − mt [i − 1])/2 if 1 < i < 5; (5)

mt [i] − mt [i − 1] if i = 5,

where dt−1 is the derivatives of the previous down-sampled
motion segment. As illustrated in the bottom of Figure 7,
Hermit interpolation can guarantee the C2 smoothness of the
resulting animation. Finally, we use mt and dt to interpolate in-
between animation frames. The motion interpolation algorithm
can be described using the following pseudo code Algorithm 1.
In Algorithm 1, function p=HermitInterpolation(upper, lower,
α) takes the upper and lower sample frames as input and
computes the in-between frame corresponding to α ∈ [0, 1].
Digital Object Indentifier 10.1109/MCG.2014.105
2: Find the upper and lower sample frame [lower, up-
per]= Bound(i)
3: α = (i-lower)/(upper-lower)
4: St [i] = HermitInterpolation(upper, lower, α)
5: end for
5 R ESULTS AND E VALUATIONS
We implemented our approach using C++, without GPU
acceleration (Figure 8). The test computer used in all our
experiments is an off-the-shelf computer with the following
hardware configuration: Intel core I7 2.80GHz CPU and 3GB
memory. The used real-time phoneme recognition system is
the Julius system that is trained based on the open-source
VoxForge acoustic model. In our system, the used 3D face
model consists of 30K triangles, and thin-shell deformation
algorithm is used to deform the rest 3D face model based
0272-1716/$26.00 2014 IEEE
 This article has been accepted for publication in IEEE Computer Graphics and Applications but has not yet been fully edited.
IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication.
on synthesized facial marker displacements. To improve the
realism of synthesized facial animation, we also synthesize
corresponding head and eye movements simultaneously by
implementing the live speech driven head-and-eye motion
generator [14].
Fig. 8. A snapshot of our experiment scenario and several
selected frames synthesized by our approach
5.1 Quantitative Evaluation
In order to quantitatively evaluate the quality of our approach,
we randomly selected three speech clips with associated mo-
tion from the captured dataset as the test data. We compute the
Root Mean Squared Error (RMSE) between the synthesized
motion and the captured motion (only taking markers in the
lower face into account, as illustrated in Figure 3). To better
understand the accuracies of our approach under different
situations, we synthesized motion in three cases, as shown in
Table 1. Here, live speech phoneme recognition means that the
phoneme sequence is sequentially outputted from the JULIUS
by simulating the pre-recorded speech as live speech input.
Real-time facial motion synthesis means the facial motion is
synthesized phoneme by phoneme, as described in this paper.
The synthesis algorithm used in both case 1 and case 2 is the
live speech driven lip-sync approach presented in this paper.
Live speech Real-time facial
Approach phoneme recog. motion synthesis
Case 1 - Y
Case 2 Y Y
Baseline - Y
TABLE 1
Three cases/configurations of our lip-sync experiments
Comparison with baseline approach. We also compared
our approach with a baseline approach [1]. Essentially, this
viseme-based baseline approach adopts the classical Cohen-
Massaro co-articulation model [5] to generate speech anima-
tion frames via interpolation. This method also needs users
Digital Object Indentifier 10.1109/MCG.2014.105
6
to manually specify (or design) visemes. To ensure a fair
comparison between our approach and the chosen baseline
approach, the visemes used in the implementation of [1] were
automatically extracted as the center (mean) of the cluster that
encloses all the representative values of the corresponding
phonemes (refer to Section 3). The configuration of the
baseline approach is also shown in Table 1.
The comparison results are shown in Table 2. Case 2 has
the largest RMSE errors in the three test cases due to the
inaccuracy of live speech phoneme recognition. The RMSE
error of Case 1 is smaller than that of the baseline approach
in all the three test cases; however, we observe that the
animation results in case 1 are clearly better than those by the
baseline approach in terms of motion smoothness and visual
articulation.
Approach Audio 1 Audio 2 Audio 3
Case 1 18.81 19.80 25.77
Case 2 24.16 25.79 27.81
Baseline 22.34 22.40 26.17
TABLE 2
RMSE of the three test speech clips by different
approaches
5.2 Runtime Performance Analysis
In order to analyze the runtime efficiency of our approach, we
used three pre-recorded speech clips, each of which is about 2
minutes in duration. We sequentially fed the speech clips into
our system by simulating them as live speech input. During the
live speech driven lip-sync process, the runtime performance
of our approach was recorded simultaneously. Table 3 shows
the breakdown of the per-phoneme computing time of our
approach. From Table 3, we can observe the following:
• The real-time phoneme recognition step consumes about
60% of the total computing time of our approach on aver-
age. The average motion synthesis time of our approach
is about 40% (around 40.85ms) of the total computing
time, which indicates that our lip-sync algorithm itself is
highly efficient.
• We also calculated the average phoneme number per
second in the test data (shown in the second rightmost
column) and the average FPS (shown in the rightmost
column). The average FPS is computed as follows:
NumberO f Frames(pi )
∑Ni=1 TotalComputTime(pi )
AverageFPS = , (6)
N
where N is the total number of phonemes in the test data;
NumberO f Frames(pi ) denotes the number of animation
frames needed for visually representing pi , which can
be simply calculated as its duration (in second) mul-
tiplied by 30 frames (per second) in our experiment;
and TotalComputTime(pi ) denotes the total computing
time used by our approach for phoneme pi including the
real-time phoneme recognition time and motion synthesis
time. As shown in Table 3, the calculated average FPS
of our approach is around 27, which indicates that our
0272-1716/$26.00 2014 IEEE
 This article has been accepted for publication in IEEE Computer Graphics and Applications but has not yet been fully edited.
IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication.
Phoneme
Test audio recog. time Motion synthesis Motion blending
number (ms) (ms) (ms)
1 76.73 38.56 0.16
2 90.13 39.10 0.14
3 83.80 44.89 0.15
TABLE 3
Breakdown of the per-phoneme computing time of our approach
approach is able to approximately achieve the real-time
speed on an off-the-shelf computer, without GPU accel-
eration.
6 C ONCLUSION AND D ISCUSSION
In this paper, we introduce a practical phoneme-based ap-
proach to real-time generate realistic speech animation based
on live speech input. On the one hand, our approach is
very simple and efficient, and can be straightforwardly imple-
mented. On the other hand, it can work surprisingly well for
most practical applications, achieving a good balance between
animation realism and run-time efficiency, as demonstrated
in our results. Because our approach is phoneme-based, it
can naturally handle speech input from different speakers,
assuming the employed state-of-the-art speech recognition
engine can reasonably handle the speaker-independence issue.
Despite its effectiveness and efficiency, our current approach
has two limitations, described below.
• The quality of visual speech animation synthesized by
our approach substantially depends on the efficiency and
accuracy of the used real-time speech (or phoneme)
recognition engine. As described in our methodology, if
the speech recognition engine cannot recognize phonemes
in real-time or with a low accuracy, then such a delay
or inaccuracy will be unavoidably propagated to or even
exaggerated in the employed AnimPho selection strategy
in our approach. On the other hand, with the continuous
accuracy improvement of state-of-the-art real-time speech
recognition techniques in the future, we anticipate that our
approach can generate more realistic live speech driven
lip-sync without modifications.
• Our current approach does not consider affective state
enclosed in live speech and thus cannot automatically
synthesize corresponding facial expressions. One of plau-
sible solutions to this problem would be to utilize state-
of-the-art techniques to real-time recognize the change of
the affective state of a subject based on his/her live speech
alone, and then such information can be continuously fed
into an advanced version of our current approach that
can dynamically incorporate the emotion factor into the
expressive facial motion synthesis process.
As the future work, besides extending the current approach
to various mobile platforms, we also plan to conduct compre-
hensive user studies to evaluate its effectiveness and usability.
Also, we would like to investigate better methods to increase
the phoneme recognition accuracy from live speech and to
Digital Object Indentifier 10.1109/MCG.2014.105
7
Average Average phoneme
total time number per second Average FPS
(ms)
128.28 8.84 25.62
144.40 7.29 30.03
142.23 7.43 26.77
fiducially retarget 3D facial mocap data to any static 3D face
models.
ACKNOWLEDGEMENTS
This work was supported in part by NSF IIS-0914965,
NIH 1R21HD075048-01A1, and NSFC Overseas and Hong
Kong/Macau Young Scholars Collaborative Research Award
(project number: 61328204). The authors would like to thank
Bingfeng Li and Lei Xie for helping with the Julius system
and numerous insightful technical discussion. Any opinions,
findings, and conclusions or recommendations expressed in
this material are those of the authors and do not necessarily
reflect the views of the agencies.
R EFERENCES
[1] B. Li, L. Xie, X. Zhou, Z. Fu, and Y. Zhang, “Real-time speech driven
talking avatar,” J Tinghua Univ. Scicnec and Technology, vol. 51, no. 9,
pp. 1180–1186, 2011.
[2] Y. Xu, A. W. Feng, S. Marsella, and A. Shapiro, “A practical and
configurable lip sync method for games,” in Proceedings of the Motion
In Games (MIG) 2013. ACM, 2013, pp. 109–118.
[3] P. Hong, Z. Wen, and T. S. Huang, “Real-time speech-driven face
animation with expressions using neural networks,” IEEE Transactions
on Neural Networks, vol. 13, no. 4, pp. 916–927, 2002.
[4] R. Gutierrez-Osuna, P. Kakumanu, A. Esposito, O. Garcia, A. Bojorquez,
J. L. Castillo, and I. Rudomin, “Speech-driven facial animation with
realistic dynamics,” Multimedia, IEEE Transactions on, vol. 7, no. 1,
pp. 33–42, 2005.
[5] M. M. Cohen and D. W. Massaro, “Modeling coarticulation in synthetic
visual speech,” in Models and techniques in computer animation.
Springer, 1993, pp. 139–156.
[6] A. Wang, M. Emmi, and P. Faloutsos, “Assembling an expressive
facial animation system,” in Proceedings of the 2007 ACM SIGGRAPH
symposium on Video games. ACM, 2007, pp. 21–26.
[7] C. Bregler, M. Covell, and M. Slaney, “Video rewrite: Driving visual
speech with audio,” in Proceedings of SIGGRAPH’97, ser. SIGGRAPH
’97, ACM. ACM, 1997, pp. 353–360.
[8] Y. Cao, W. C. Tien, P. Faloutsos, and F. Pighin, “Expressive speech-
driven facial animation,” ACM Transactions on Graphics (TOG), vol. 24,
no. 4, pp. 1283–1302, 2005.
[9] Z. Deng and U. Neumann, “eFASE: Expressive facial animation syn-
thesis and editing with phoneme-level controls,” in In Proc. of ACM
SIGGGRAPH/Eurographics Symposium on Computer Animation. Eu-
rographics Association, 2006, pp. 251–259.
[10] S. L. Taylor, M. Mahler, B.-J. Theobald, and I. Matthews, “Dy-
namic units of visual speech,” in Proceedings of the 11th ACM SIG-
GRAPH/Eurographics conference on Computer Animation. Eurograph-
ics Association, 2012, pp. 275–284.
[11] X. Ma and Z. Deng, “A statistical quality model for data-driven
speech animation,” IEEE Transactions on Visualization and Computer
Graphics, vol. 18, no. 11, pp. 1915–1927, 2012.
[12] M. Brand, “Voice puppetry,” in Proceedings of the 26th annual confer-
ence on Computer graphics and interactive techniques, ser. SIGGRAPH
’99. ACM, 1999, pp. 21–28.
[13] T. Ezzat, G. Geiger, and T. Poggio, “Trainable videorealistic speech
animation,” ACM Trans. Graph., vol. 21, no. 3, pp. 388–398, Jul. 2002.
0272-1716/$26.00 2014 IEEE
 This article has been accepted for publication in IEEE Computer Graphics and Applications but has not yet been fully edited.
IEEE COMPUTER GRAPHICS AND APPLICATIONS,Some content
VOL. XX, may
NO. XX, change
JULY 2014 prior to final publication. 8

[14] B. Le, X. Ma, and Z. Deng, “Live speech driven head-and-eye motion
generators,” IEEE Transactions on Visualization and Computer Graph-
ics, vol. 18, no. 11, pp. 1902–1914, 2012.

AUTHORS ’ B IOGRAPHIES
Li Wei is a PhD student in the Department of Computer
Science at University of Houston. His research interests
include computer graphics and animation. He had received
his B.S. in Automation and M.S. in Automation from Xiamen
University, China, in 2009 and 2012, respectively.

Zhigang Deng is an Associate Professor of Computer

Science at the University of Houston. His research interests
include computer graphics, computer animation, and human
computer interaction. He earned his Ph.D. from University of
Southern California in 2006, M.S. from Peking University in
2000, and B.S. from Xiamen University in 1997, respectively.

Digital Object Indentifier 10.1109/MCG.2014.105 0272-1716/$26.00 2014 IEEE