Proceeding of the 3rd International Conference on Informatics and Technology, 2009

A Conceptual Design for Non-verbal Message Transmission

Ryoichi Komiya
Faculty of Engineering and Science
Universiti Tunku Abdul Rahman, Jalan Genting Kelang Setapak 5330 KL, Malaysia
ryoichi@utar.edu.my
ABTRACT

This article discusses the limitations of the today’s telecommunication systems in terms of non-verbal message
transmission. When it comes to transmitting direct personal messages of an emotional nature such as trust, enthusiasm
or sincerity via current communication systems, it is presently virtually impossible. Human beings normally gauge another
person’s overall personality, character, disposition, education level or other such traits when they physically meet.
People sense and understand these personal attributes very quickly and emotive decisions that are subsequently made,
such as; this person seems trustworthy and sincere so perhaps it would be well for me to enter into a prospective business
transaction with him or her. However, if there is a telecommunication system between the two individuals, then what
happens to these intuitive judgments? This non-face-to-face communication limitation is the motivating force for writing
this paper.

Keywords: non-verbal messages, limitations of current telecommunication systems, sensors and actuators for
non verbal messages

1. Introduction

Multimedia has without doubt enormously contributed to enhance business efficiency. We have been able to send a wide
variety of media over existing networks such as voice, text, image and video via today’s personal computers. Personal
computers nowadays are one of our most important multimedia telecommunication tools. In fact many of today’s cellular
phones can be referred to as mobile multimedia terminals because they function as telephones, but with e-mail, still
picture or video transmission/reception and the Internet access. By exchanging information via multimedia channels it is
now possible to proceed business transactions far more quickly, efficiently and accurately than ever before.
However, present multimedia communications content has not been able to contribute significantly to the
person-to-person communications interface in terms of transmitting trust and other such emotions [1].
This paper firstly discusses the importance of non-verbal messages in our daily lives and then describes the possibility of
implementing sensors and actuators to pick up, transmit and regenerate non-verbal messages between the calling and
called party sites. At the end of the discussion, a conceptual design for a non-verbal telecommunication terminal is
proposed.

2. Human to human understanding components in face to face conversation

In daily life humans exchange large amounts of non-verbal information during face-to-face contact. However, when we
exchange ideas and opinions by making use of telecommunication systems or the Internet, it is oftentimes very difficult to
communicate precisely or thoroughly. Why does this happen? According to the data, we get many different types of
information during face-to-face conversations, typically; 7% of the information is imparted by direct speech, 38% from
speech tone and the remaining 55% comes from non-verbal messages such as gestures, posture, facial expression,
accessories and clothing and other such personality indicators [2].
These emotive messages are referred to as non-verbal messages and are the principal communication mechanisms we
all use to understand, evaluate, judge and ultimately get to know a person’s unique and individual underlying personality
and character.
Humans understand each other most precisely through these non-verbal messages. We use them to understand each
other and even after just a few short word exchanges during a first meeting we use them to decide, for instance, whether
there is the possibility or not or working well with someone in the business arena. In relationships between males and
females, the same can be applied. Videophones or multimedia terminals such as personal computers were developed
with the expectation of better sending and receiving these non-verbal messages, but they have not been successful.
Of course for the direct exchange of information the telephone and Internet have contributed greatly to the benefit of
business globally and it could be said that for data, graphic and pictorial transmission, provide almost perfect
telecommunication tools.
However, current multimedia terminals such as network connected personal computer have not been able to contribute to
non-verbal message exchange at all.
The importance of non-verbal message exchange not only makes it an essential part of personal discussions, but also in
commercial activities. Because, as in all other spheres of human activity, in business we need to establish trust as well as
other emotional (controlled) transmissions, for example; talks between a medical practitioner and a patient, conversations
between a banker and his client or price negotiations between a trader and a consumer.

3. Non verbal messages and their significances

Non-verbal behavior has been well studied and findings published [3]. In this section the high level presentation of
non-verbal messages are summarised in Table l encompassing; physical appearance, clothing, gesture/posture, facial
expressions to vocal tone.
From Table 1 we can see that human beings assimilate a wide variety of human character traits and attributes through
these non-verbal messages when they meet. It is clear that even following the invention and widespread usage of the

©Informatics '09, UM 2009 RDT3 - 63

 Proceeding of the 3rd International Conference on Informatics and Technology, 2009

telephone, facsimile and Internet, the need for and number of face-to-face meetings has not been significantly reduced in
the areas such as business, science and technology or international diplomatic negotiations. Due to the increasing need
for these face-to-face meetings and negotiations, business trips have tremendously increased nationally as well as
internationally [4]. This increase in travel of course works diametrically against initiatives such as efforts to reduce global
warming, to reduce in energy consumption and other such similar environmental issues facing the world.
If we accept that non-verbal transmission in telecommunication systems would contribute to reducing the necessity of
travel for face-to-face meetings, then an individual’s work mode and lifestyle would change significantly making the
prospect of a telecommunications system capable of exchanging both verbal and non-verbal messages a very desirable
future tool for mankind.
Based on Table I, further discussion will be conducted in the following sections.

Table l Non-verbal messages and their significances

Non-verbal Non-verbal message Significances
messages components
Physical Body type Overweight: Slow, sociable, emotional, forgiving, relaxed
appearance Medium sized: Confident, energetic, dominant,
enterprising, hot-tempered
Slim: Tense, awkward, meticulous, tactful, detached
Height Majority of executives in Fortune 500 companies were more
than 180 cm
Skin color No data
Hair Color Gold: fun
Brown: wholesome
Red: tempestuous
Black: sultry
Hair length Long: less serious (sexy, rebellious)
Short: serious
Clothing When we know someone His/her clothing has little influence on our perceptions
When we do not know the real The clothes of strangers are a rich source of information about
person them when no other source exists.
Gesture and Various gestures and movement Together with facial expressions, a person’s posture, gait, limb
movement movement and other behavior provides information about his
or her emotional state or mood.
Facial behavior 1.Eye brows and forehead Sadness and fear: 1
2.Eyes and eyelids Anger: 1&3
3.Lower face (cheeks, nose and Disgust often blends with surprise: 1,2,&3
mouth) Fear: 2
Happiness: 2&3
Surprise: 1,2 & 3
Eye behavior Salience Eliciting the attention of others
Stimulate arousal Like/dislike
Involvement Eye contact
Vocal behavior Vocal qualities Tempo, resonance, rhythm control, articulation control
pitch control, glottis control, vocal lip control, pitch range
Vocalization -non-language sound: laughing, crying, whimpering, giggling,
-vocal characterizer snickering, sobbing
-vocal qualifier -intensity, pitch height, extent
Vocal segregate “uh-huh”, “er”, “ah”, “and-ah”, “you know”

3.1 Sensors required to pick up non-verbal messages

We understand that almost all, except vocal behavior, of the non-verbal messages emitted by humans as in the Table 1
can be picked up by visual sensors. The most prospective sensor would be the video camera. However, it needs to be
ascertained which standard video cameras offer the optimum ability to pick up visual non-verbal messages accurately and
naturally.
For vocal non-verbal messages it is comparatively easy to collect and transmit by microphone(s).
Therefore, essentially all the necessary sensors to pick up non-verbal messages may not be different from the
conventional videophone. However, videophones have not to date been generally well accepted by the public and have a
history of successive commercial catastrophes all over the world.
Let us then review the history of the videophone a little in the section below.

4 History of videophone R&D- Its successive failures-

4.1 AT&T [5]

1927: A one-way system was demonstrated in a conversation between Secretary of commerce Herbert Hoover, and
senior officials of AT&T.
1930: A two-way video link was set up between the headquarters in New York of Bell laboratories and AT&T using the

©Informatics '09, UM 2009 RDT3 - 64

 Proceeding of the 3rd International Conference on Informatics and Technology, 2009

then latest TV equipment within a closed circuit.

1956: Bell laboratories develop a videophone system which could work over the standard telephone network.
1971: Bell laboratories completed the R&D of the Picturephone.

4.2 The R&D outcomes of the Picturephone

The Picturephone was a dream of Bell laboratories’ R&D group because at that time (1971) it was able to handle video
image as well as voice message exchanges leading them to believe that the Picturephone would become the ultimate
solution to human to human network based contact, the ultimate telecommunication tool.
However, because of the restrictions of the existing network infrastructure; such as subscriber loops, transmission and
switching systems, the picture size could only be very small (less than 13 cm by 13 cm) and only in monochrome, and the
overall quality was such that it was never really accepted by the general public at that time.
Since that time relevant technologies have improved, especially in terms of video signal compression.
However, basic features of the videophone in general have not changed from those of Picturephone, even though the
function has been commonly implemented in the modern laptop computers. The usual setup is for a video camera to be
located above computer video screen, but it only caters for the person’s head and shoulders image and if a calling party
moves out of the shooting range of the video camera, the called party can not see the calling party any more. This has
been a specifically disadvantageous aspect of the videophone because the verbal telephone conversation can be
maintained so long as both parties hold the handsets, effectively combining the incompatible elements of one stationary
and one mobile sensor. Hence it should be stressed that one of the most important advantages of the videophone should
be the ability to maintain eye contact between the talking parties. The limitation arises because of the physical alignment
between the camera and the screen. There are, however, on the market nowadays videophone systems essentially
solving the eye contact issue [6].
Today, videophone capability in cellular phones has contributed significantly to on site video news gathering by the
general public [7].
Regardless of these developments, current videophone features do not offer enough flexibility to adequately exchange
human non-verbal messages.

4.3 Technology assessment of the Picturephone from the view point of non-verbal message exchange

Now, let us think about why the present Picturephone design philosophy is unable to match the features required to
adequately communicate non-verbal messages. In relation to the technology assessment of the Picturephone, which was
conducted by Cornell University some 35 years ago [8], it was their view from their analysis results that limitations in
technology were why the Picturephone had not been well-accepted by consumers. However, their assessment method
can be adapted and applied to the modern technology case.
Table ll indicates the results of the assessment.
Based on the assessment below we can clearly understand why the lack of adequate functions of the Picturephone
rendered it unsuitable from the perspective of non-verbal message transmission.

5.0 The intended terminal configuration of non-verbal message transmission

5.1Use cases

The use cases of non-verbal message transmission are as follows

5.1.1Business scenario

(1)Where trust transmission is required: ie- conversations between a MD and a patient, conversations between lawyers
and clients, conversation between bank officers and customers, recruitment interviews, etc.
(2) Where enthusiasm transmission is necessary: ie- presentation of a sales talk/pitch (real world/cyber world) on specific
items, recruitment interviews, etc.
(3) Where entire human character transmission is necessary: ie- any negotiations; including diplomatic commercial, or
similar issues, recruitment interviews, etc.
(4) Where confidence, precision and accuracy transmission is necessary: ie- teaching activities, political campaigns,
technical briefings, recruitment interviews, etc.

5.1.2 Personal/private scenario

(1) Talks between family members

(2) Chats between boyfriend and girlfriend
(3) Guidance from boss to subordinate (Small Office Home Office application)

5.2 Functional design

5.2.1 Necessary functions for non-verbal message transmission

(1) Video Sensing/Actuating: to shoot/regenerate visual non-verbal messages

(2) Audio Sensing/actuating: to pick up/regenerate audible non-verbal messages

©Informatics '09, UM 2009 RDT3 - 65

 Proceeding of the 3rd International Conference on Informatics and Technology, 2009

(3) Video/Audio sensing control: to control video/Audio sensing to and from both calling and called parties to maximize
sensitivity of shoot/pick up visual/audio non-verbal messages.
(4) Human position detection: to detect the human position in a room to operate Video/Audio control accurately
(5) CODEC with bandwidth reduction: to digitize sensed video/audio signals together with appropriate bandwidth
reduction algorithm
(6) MUX/DEMUX: to multiplex/de-multiplex digital data stream
(7) TRANSMIT/RECEIVE: to transmit/receive to/from line interface

Table II Technology assessment

Non-verbal Non-verbal message components Basic features of Picturephone
message
Physical Body type (overweight, medium, Body type and height can not be captured by the
appearance slim) including height Picturephone because its video camera has been
designed to shoot only a person’s head and shoulders
image.
Skin color and hair color/length Depends on the performance and lighting conditions of the
video camera
Clothing Clothing including accessories The video camera of Picturephone has no capabilities to
capture. Only limited image can be shot.
Gesture and Head, torso movement Can be captured with limitation (head and shoulders)
movement Hands and arm movement Only the limited portion can be shot
Legs position on chair Impossible
Gaits Impossible
Facial behavior 1.Eye brows and forehead Distinctive facial behavior can be captured
2.Eyes and eyelids
3.Lower face (cheeks, nose and mouth)
Eye behavior Eye contact Impossible because of the video camera and screen
alignment, One manufacturer currently offers a videophone
which can maintain eye contact.
We still do not know the actual resolution required for
accurately capturing eye behavior.
Vocal behavior Vocal qualities This can be captured by microphone with the band
width of 0.3 Hz-3.4 kHz (telephone audio
characteristics)

Vocalization We have to know the frequency range of the vocalization

-vocal characterizer
-vocal qualifier
Vocal segregate As above

5.2.2 A functional configuration of non-verbal message transmission

The functional configuration of the non-verbal message transmission is illustrated in Fig. 1.

This figure only shows the calling party’s side configuration. The called party’s side functional configuration is identical to
this figure. Functional interface reference points are defined by arrows between each functional block.
Detailed functional blocks are presented in Table III and functional interface characteristics are indicated in Table IV.

A
Video
Sensing/Actuating
E
C
Codec with band
width reduction G
Video/audio sensing control Human position detection
MUX/DEMUX

Transmit/Receive

D
F

Audio
Sensing/Actuating B

Fig. 1 Functional configuration of non-verbal message transmission system (calling party’s side)

©Informatics '09, UM 2009 RDT3 - 66

 Proceeding of the 3rd International Conference on Informatics and Technology, 2009

Table III Description of functional blocks

Functional block Description Candidate hardware

Video Sensing/Actuating
Audio Sensing/Actuating
Video/Audio sensing control
Human position detection
CODEC with bandwidth reduction
MUX/DEMUX
TRANSMIT/RECEIVE
implications
Video sensing: to shoot visual non-verbal messages Video camera/Display
Video actuating: to regenerate visual non-verbal (scanning line resolution to
messages be studied)
Audio sensing: to pick up audible non-verbal Microphone/Receiver
messages (audio bandwidth to be
Audio actuating: to regenerate audible studied)
non-verbal messages
Video sensing control: to control video sensing to/ Video camera control
from both called/calling parties to maximize (zoom in/off, pan, tilt)
sensitivity to shoot visual non-verbal messages
Audio sensing control: to control audio sensing Microphone sensitivity
from both calling/called parties to maximize control
sensitivity to pick up audio non-verbal messages
To detect the human position in a room to operate Floor pressure sensor
Video/Audio control accurately Video image sensor
To digitize sensed video/audio signals together MPEG 4 CODEC
with appropriate bandwidth reduction algorithm
To multiplex/de-multiplex digital data stream Packetizer/De-packetizer
To transmit/receive to/from line interface LAN interface card

Table IV Interface characteristics of the Functional Interface Reference Point (FIRP)

FIRP Direction of the Information to be exchanged on the FIRP

information flow at
FIRP
A bidirectional
B bidirectional
C unidirectional
D unidirectional
E unidirectional
F bidirectional
G bidirectional
Downstream direction: video signal from the called party
Upstream direction: video signal form the calling party
Downstream direction: audio signal from the called party
Upstream direction: audio signal form the calling party
Camera control signal from the called party
Microphone control signal from the called party
Human position indicating signal to control video/audio sensing control
Audio/video sensing control from both calling and called party
Downstream/Upstream multiplexed digital data

5.3 Basic design philosophy of an experimental non-verbal message transmission terminal

Based on the functional configuration of the non-verbal message transmission system, we are now trying to design an
experimental non-verbal message transmission terminal.
Basic design philosophy of the terminal is as below;
(1) The terminal should be considered as the next generation POTS (Plain Ordinary Telephone Service)
(2) The terminal should firstly start as a fixed line services (office room, living room applications)
(3) The terminal is to be designed to meet universal requirements for various business and personal communications
(4) The terminal should be implemented by combining, where possible, existing consumer products
(5) The video image and audio quality should be adequate to transmit non-verbal messages accurately and naturally.
(6) The terminal should contribute to overall telecommunication services innovation by making use of NGN [9].
(7) The terminal should contribute to a decrease in business trips reducing carbon dioxide emissions.

The actual implementation will be preceded by testing and evaluating the quality of non-verbal message transmission.

6. Conclusions and future works

Non-verbal messages such as physical appearance, body movement, clothing (including accessories), facial
expressions are all very important to fully ascertain a person’s emotional state and even his/her background. In the case
of face-to-face conversations, we collect these non-verbal messages intentionally or non-intentionally, but based on these
messages, we understand or determine almost instantaneously whether we are going to get on well with another
individual or not. Because human beings can understand much of another person’s background by directly conversing
with them they instinctively receive and accept non-verbal messages of trust, enthusiasm, sincerity and other emotions in
the people they are confronting. However, where there is a telecommunication system (telephone, Internet or even video
telephony) of current technological capability in between the two persons it is difficult if not impossible to receive these
non-verbal signals. This paper proposes the possibility of accurate non-verbal transmission over the existing network by
combining currently available technology and products.

©Informatics '09, UM 2009 RDT3 - 67

 Proceeding of the 3rd International Conference on Informatics and Technology, 2009

REFERENCES

[1] Ryoichi Komiya, “How multimedia/multimodal information Can Contribute to the Change of Telecommunication?”,
In Proceedings of Eighth IEEE International Symposium on Multimedia 2006, San Diego, pp.774-776.
[2] Albert Merabian,
http://en.wikipedia.org/wiki/Albert_Mehrabian.
[3] Virginia P. Richmond, James C. McCROSKEY, Nonverbal Behavior in International Relations, Fourth Edition, Allyn and
Bacon, 2000.
[4] Research: U.S. Business Trips To Fall Through 2012 While Spending Rises
http://www.management.travel/news.php?cid=Global-Insight-TIA-forecast.Dec-08.04.
[5] Videophone
http://en.wikipedia.org/wiki/Videophone.
[6] THE EYE CATCHER
http://greeniii.com/productContent.php
[7] Internet plays key role in Uighur unrest
http://www.taipeitimes.com/News/world/archives/2009/07/09/2003448178
[8] Edward M. Dickson, “Potential Impacts on the Video Telephone”, IEEE TRANSACTIONS ON COMUNICATIONS,
VOL.COM-23, NO. 10, OCTOBER 1975, pp. 1172-1176
[9] Next Generation Networks Global Standards Initiative
http://www.itu.int/ITU-T/ngn/index.html

BIOGRAPHY

Ryoichi KOMIYA was born in Tokyo, Japan on March 16, 1945. He received his B.E. and Ph.D. degrees from Waseda
University, Tokyo, Japan, in 1967 and 1986 respectively. Joining the Electrical Communication Labs of NTT in 1967, he
was engaged in the development of the PCM repeatered line, digital data terminal equipment, video coder/decoders, stuff
multiplexers, ISDN subscriber loop transmission systems and fiber optic remote multiplexer systems.
After leaving NTT in 1992, he worker with Siemens, Nippon Telecommunication Consulting Co. Ltd., NTT Advanced
Technology and the Distribution and Economics University of Japan.
He lectured at the Faculty of Engineering and Faculty of Information Technology at Multimedia University, Malaysia from
1998 to 2009, where he was responsible for research and development of next generation telecommunication systems,
services, terminals, IP networks, virtual education environment, e-commerce terminal, Intelligent Transportation
SystVirem, Fiber to The Home and Smart Home programmes.
He also worked at the National Institute of Communications Technology, Japan from 2006 to 2008, where he studied
Connected Health Systems using Body Area Network (BAN) systems.
From April, 2008 he worked with the Regional Disaster Information System Labs at Iwate Prefectural Univesity in Japan.
In June 2009, he joined University Tunku Abdul Raman, Malaysia, where he is conducting Research & Development on
ICT applications to transport system, medicine and non-verbal communication systems.
Professor Komiya is a member of IEEE, ACM and IEICE.

©Informatics '09, UM 2009 RDT3 - 68