Capturing and Analyzing Stability of Human Body Motions Using Video Cameras

Yoshihisa Shinagawa and Jun-ichi Nakajinia

T h e Department of Information Science

Tosiyasu L. Kunii
T h e University of A i m

Graduate School of Science The University of Tokyo { sinagawa,juno}Ois.s.u-tokyo.ac.jp

kunii@u-aizu.ac.jp

Kazuhiro Hara
Architecture Laboratories SONY Corporation gen@net.arch.sony.co.jp

Abstract
The need f o r capturing human b o d y motions has been increasing recently for making movies, sports instruction systems and robots that can simulate human motions. This paper proposes a method to facrlitate motion capturing using inexpensive vadeo cameras. I n our system, a fe,w cameras are used t o obtain multiple views of a human b o d y and a three-dimensional (30) volume consistent with the views is created. A m o d e l of the human b o d y is then fitted to the volume to obtazn the configuration of the human body. We also propose a method to analyze the stability of human postures. W e have analyzed a technique of the traditional Chinese martial art Shorinji Kempo based o n the stability t o show the effectiveness of our method.

This paper proposes a method to facilitate motion capturing. In our system, three cameras are used t o obtain three views of a human body. A three-dimensional (3D) volume consistent with the three views is then created. A model of a human body is then fitted t o the volume to obtain the configuration (joint angles) of the human body. We also propose a method to analyze the stability of human postures. We define a subspace of configuration space called the recoverable space. Using the method, we have analyzed a technique of the traditional Chinese martial art, Shoranjz Kempo, where a person pulls a towel grabbed by his opponent and makes him fall down.

2 1

Introduction

The Proposed Motion Capturing Method

T h e need for capturing human body motions has been increasing recently for making movies, sports instruction systems and robots that can simulate human motions. Commercial systems use sensors to detect the motions: either optically or magnetically. They are, however, expensive and they require a studio t o settle the special devices so that the sensors will not be disturbed by the terrestrial magnetism or the ambient infra-red rays.

First of all, we have used three video cameras t o obtain three different views of the human body. T h e three cameras are calibrated beforehand so that their positions in the world coordinate system are known. T h e cameras have been synchronized by flashing a light at the outset and detecting the video frames with the flash. For each scan line of each view image, we generate a frustrum (see Figure 1). For each view, the frustrums are then put together to form a

48
1087-4844/97 $10.00 0 1997 IEEE

Authorized licensed use limited to: University of Saskatchewan. Downloaded on November 12, 2009 at 16:26 from IEEE Xplore. Restrictions apply.

volume that we call the vaewang volume. T h e intersection of the three viewing volumes is then computed t o obtain the 3D volume (denoted by B in what follows) that encloses the human body (see Figure 2). T h e actual computation of the intersection is carried out by detecting the voxels that belong t o the three frustrums. We have used 256 x 256 x 256 voxels for the computation.

2.1

Recovering human postures

Q Viewp

Leung and Yang [ll] proposed a method t o label the regions of human images the identifiers of body parts. Such inforination is not sufficient in our case because we need the configurations of human postures. Kameda, Minoh and Ikeda [6] estimated a human posture from a silhouette. However, there have been ambiguities that cannot be solved from a single silhouette. Laba1li.e and Szeliski [lo] proposed a method t o recover the position and orientation of an object from a number of its image contours. In our case, however, 1 here are only a few views of objects and hence their method cannot be applied. For these reasons, we fit the model to the data by minimizing the value of the fitting criteria by the steepest descent method. Our human body model has seventeen segments (see Figure 3). T h e fingers and toes are omitted for simplicity. As the fitting criteria, the volume E of the set difference of B and the model is used. T h e model is first fitted roughly t o the coordinates of several sampled points of the body. We first fit the chest t o the volume and then go t o the extremities. Using the obtained configuration as the initial value, we compute
___

Figure 1. view

A Frustrum generated for each

dE

where xk ( k = 1,2,..) is the variable corresponding to each degree of freedom of the model. Each xk is varied by the steepest descent method until E converges or the number of iteration steps exceeds a certain threshold.

Figure 2. Using multiple views t o generate the volume

Experimental Results

We have captured three example postures as shown in Figure 4. T h e three views of Posture

49

Right Shank 3

Left Shank

Right Foot

Left Foot

Figure 3. T h e human body model

2 in Figure 4(b) are shown in Figure 5. The initi a1 and the final configuration of the model fitted to Posture 2 are shown in Figure 6. Those of P osture 3 are shown in Figure 7.

Analyzing t h e Stability of Human Body Motions

There are many researches regarding the synthesis of human body motions [l, 2, 4, 51. In this paper, we focus on the analysis of human body motions [ 3 , 7, 8, 91. We have analyzed the stability of the human body motions using the captured data. At this time, we have used five cameras and the body model has been fitted t o the locations of the markers attached t o the human bodies. We define the instability of a posture as follows. First of all, the center of gravity of the body is projected vertically onto the ground. Let us denote the projected point on the ground by P . When the person stands on a single foot, the instability is defined as the distance between P and the center of the foot (see Figure S(a)). When one stands on both feet, it is defined as the distance between P and the line segment connecting the centers of the feet (see Figure 8(b)).

(c)
Figure 4. Example postures: (a) Posture 1,

(b) Posture 2 and (c) Posture 3

50

Authorized licensed use limited to: University of Saskatchewan. Downloaded on November 12, 2009 at 16:26 from IEEE Xplore. Restrictions apply.

(b)
Figure 6. F i t t i n g t h e model t o Posture 2 w i t h (a) t h e initial configuration and (b) t h e final configuration

Figure

5. Three

views of Posture 2

51

Authorized licensed use limited to: University of Saskatchewan. Downloaded on November 12, 2009 at 16:26 from IEEE Xplore. Restrictions apply.

set of configurations whose instability is smaller than t,he current configuration; i.e., if one moves in tzherecoveriiig space, he/she can recover the b a1ance . T h e recoverable space of the opponent in the successful and unsuccessful cases is represented in Figure 13 (a) and ( b ) , respectively, by the white area. It,s complementary area is paint,ed gray and the region out,side the configuration space dark gray. The current configuration is represented by a circle. To reduce the complexit,y of the problem, only the angles of the knee joint and the hip joint are shown. z-coordinat,e in the image corresponds to the angle of the knee joint and y-coordinate represents that of the hip joint. The right image shows the configuration space of the right leg, and the left image shows that of the left leg. From the images, we notice that the recoverable space in the unsuccessful case is much wider and has t,wo-dimensional degrees of freedom. Therefore, the opponent can move one of the legs to balance. In the successful case, it has only onedimensioiml degree of freedom (a curved line). Therefore, the movement of the lower extremities is limited to tjhe movement along the line and the joint angles become bigger and bigger, which results in falling down. T h e difference in the degree of freedom is caused by the difference in the direction in which the expert pulls the towel grabbed by the opponent. In the successful case, it is the equidistant line of the centers of the feet of the opponent, while in the unsuccessful case, it is not. We can see this in Figure 12 where the loci of the center of gravity are plotted.

Figure 7. F i t t i n g t h e model t o Posture 3 w i t h (a) t h e initial configuration and (b) t h e final configuration

We have captured human body motions and have measured the time-varying instability. As the examples, we have analyzed a technique of the traditional Chinese martial art, Shorznjz liempo, where a person pulls a towel grabbed by his opponent and tries to make him fall down. We have compared the changes of the instability of a successful case (see Figure 9) and an unsuccessful case (see Figure 10). In the successful case, the opponent cannot move his feet from the ground and falls down to the ground being out of balance. In the unsuccessful case, the opponent can move his feet and manages to balance. In this case, the opponent can try t o prepare his next attack. The change of the instability of the opponent is shown in Figure 11. In the successful case, the instability continues t o increase while in the unsuccessful case, the opponent recovers the stability. T h e motions of both cases are very similar although one of them is a success and the other is a failure. T o see the difference between the t w o cases, we introduce the notion of recoverable space. The recoverable space is a

Conclusions and Future Work

We have proposed a method to capture human body motions using a few video cameras. We have also defined the instability of human postures and have analyzed the technique of Shorinji Ir'empo. We have fitted the body model to volumes using the steepest descent method. There is a possibility that it converges to a local minimum instead of the optimal value. T h e development

52

Authorized licensed use limited to: University of Saskatchewan. Downloaded on November 12, 2009 at 16:26 from IEEE Xplore. Restrictions apply.

yes of Stability

Stable Unstable

:.I Q - -1......................-I ...J ..c ..... .........- - - . ~ . . . . .,.. ............ .. ;.; :* ...... .......................... .......; ji ; . * ............................................... 3 :.'i ................................................/.** : . +.. .......... .... .............. . ............................................
i! [ / ; ;

i:.. . *:

* : .

.. -. e. s

........................................... :::....*...AI .....*;: .................................... ..::y/../ .... f l ...................... ................ ..

........................................

Isoplethic Curves of Stability

n..

v
Feet

Stable

Unstable

Figure 8. Isoplethic curves o f instability for (a) a single f o o t and (b) t w o feet

Figure 9. T h e successful case where the opponent falls down

53

55
'10 00

-~
~

. _~ ~
~

-__-

~~~

L_

ThC succeddful case ............................................ The unrueaisful case

45

,.. ............. .
I

FrW
20 W
40 W

ow

60W

Figure 11. The change of t h e instability

of more accurate computation is left as future work. Using the color information of the images t o construct the volumes is now in progress.

References
[l] W. W. Armstrong, M . Green, and R. Lake. Near-real-time control of human figure models. IEEE Computer Graphics and Applzcatzons, pages 52-61, June 1987.

[a]N .

I. Badler, E;. H. Manoochehri, and G. Walters. Articulated figure positioning

by multiple constraints. IEEE Computer Graphzcs and Applzcatzons, pages 28-38, June 1987.

[3] V. V. Beletzky. Nonlinear effects in dynamics of controlled two-legged walking. In W. Schiehlen, editor, Nonlznear Dynamacs an Engzneerzng Systems, pages 17-26. IUTAM Symposium Stuttgart/Germany 1989, Springer-Verlag, 1990.

Figure 10 The unsuccessful case where the opponent steps forward

[4] L. Bezault, R. Boulic, N . M. Thalmann, and D. Thalmann. An interactive tool for the design of human free-walking trajectories. In N . M. Thalmann and D. Thalmann, editors, Computer Anamatzon Seraes: Creatang and Anzmatang the Vzr-

54

Authorized licensed use limited to: University of Saskatchewan. Downloaded on November 12, 2009 at 16:26 from IEEE Xplore. Restrictions apply.

tual World, pages 87-103. Springer-Verlag, 1992.

[5] R. Boulic, N. M. Thalmann, and D. Thalniann. Human free-walking model for a real-time interactive design of gaits. In N. M . Thalmann and D. Thalmann, editors, Computer Animation '90, pages 6179. Springer-Verlag, 1990.

[6] Y. Kameda, M . Minoh, and K. Ikeda. Three dimensional pose estimation of an

articulated object from its silhouette image. In Asian Conference on Computer Vision '93, pages 612-615. 1993.

[7] T. L. Kunii and L. Sun. Dynamic analysisbased human animation. In T. S. Chua and T. L. Kunii, editors, Computer Graphics Around the World (Proc. CG International 'go), pages 3-15. Springer, Tokyo Berlin Heidelberg New York London Paris Hong Kong, 1990.
[8] T. L. Kunii, Y. Tsuchida, Y . Arai, H. Mat#suda, M. Shirahama, and S. Miura. A model of hands and arms based on manifold mappings. In N . M. Thalmann and D. Thalmann, editors, Communicating with Virtual Worlds (Proc. CG International '93), pages 381--398. SpringerVerlag, Tokyo Berlin Heidelberg New York London Paris Hong Kong Barcelona Budapest, 1993.

[9] T. L. Kunii, K.Minamida Y. Shinagawa,

( and I . Hara. Analyzing human body motions by extracting regions in the configuration paths: with an application t o analyzing skills of shorinji kempo. In Proc. Third World Congress on Computataonal Mechanics, pages 442-447. International Association for Computational Mechanics, 1994.

[lo] S. LaballCe and R. Szeliski. Recovering the position and orientation of free-form objects from image contours using 3d distance maps. 17(4):378-390, 1995.

[11] M. K . Leung and Y-H. Yang. First sight: A human body outline labeling system. IEEE Trans. Pattern Analysis an.d Machine Intelligence, 17(4):359-377, 1995.

55

Figure 12. The locus o f the center o f gravity in (a) t h e successful case and (b) t h e unsuccessful case

56

Authorized licensed use limited to: University of Saskatchewan. Downloaded on November 12, 2009 at 16:26 from IEEE Xplore. Restrictions apply.

(b)
Figure 13. T h e recoverable space of (a) t h e successful case and (b) t h e unsuccessful case

57