Emotion Recognition and Affective Computing on Vocal Social Media

Page 2 of 18

Accepted Manuscript

Emotion Recognition and Affective Computing on Vocal

Social Media

Weihui Dai



, Dongmei Han

b,c

, Yonghui Dai

, Dongrong Xu

Department of Information Management and Information Systems, School of Management,

Fudan University, Shanghai 200433, China

School of Information Management and Engineering, Shanghai University of Finance and

Economics, Shanghai 200433, China

Shanghai Financial Information Technology Key Research Laboratory, Shanghai 200433, China

Psychiatry Department, Columbia University/ New York State Psychiatric Institute, New York

City, NY 10032, United States

ABSTRACT: V

ocal media has become a popular way of communication in today’s social networks.

In  the  meantime  of  conveying  semantic  information,  vocal  message  usually  also  contains  abundant
emotional  information  which  has  been  the  new  focus  of  attention  in  the  data-mining  of  social  media
analytics.  This  paper  proposes  a  computational  method  for  emotion  recognition  and  affective
computing on vocal social media to estimate the complex emotion as well as its dynamic changes in a
three dimensional PAD(Position-Arousal-Dominance) space, and furthermore analyzes the propagation
characteristics of emotions on the vocal social media of Wechat.

Keywords:  Social  media,  Social  network,  Voice  instant  messaging,  Vocal  data-mining,  Emotion
recognition, Affective computing

1. Introduction

In today’s social networks, the way of communication is undergoing a new change due to the

emerging  vocal  social  media  such  as  Wechat,  QQ(China),  ICQ,  WhatsApp(U.S.),  Line(Japan)  and
various  tools  of  instant  voice  messaging.  While  facilitating  conveying  semantic  information,  vocal
social  media  can  also  transmit  abundant  emotional  information.  This  variation  has  resulted  in
significant influence on not only improving the users’ experiences and senses of belonging to particular
social  groups  and  therefore  enhancing  their  continuance  intentions  to  these  groups  [31][60],  but  also
strengthening the interpersonal  relationships between the  members  within these  groups as  well as  the
community’s cohesion and cognitive consistence in the social network [23, 57, 58].

From the relevant literatures, we can find a wide application of social networks due to the rapid

development  of  social  media  analytics  [1,  2,  53],  which  provides  the  effective  methodology  for
unearthing more business value from social networks [3, 13, 51]. Recently, the great influence of social
networks on psychological cognition and  social behaviors  has caused  the new  attention  [5, 9, 22, 27,
48].  In  our  previous  research  findings  [23],  the  propagation  effects  of  a  social  network  on  emergent
events affect the community through an approach that contains five interactional layers and this mostly
depends  on  its  group  cognitions:  information,  emotion,  attitude,  behavior  and  culture.  Among  which,
emotion plays an inducing role on the group’s primary recognitions and easily  leads to the consistent
attitude  and  behavioral  reactions  in  the  “small  world”  because  of  the  member’s  close  social
relationships,  trusts  and  the  empathic  effects.  Recent  neural  and  behavioral  research  work  has  also
indicated  the  theoretical  basis  for  this  phenomenon  [32].  The  effects  of  vocal  social  media  on  the
interpersonal  relationships,  group  cognitions  and  especially  the  emotion  propagation,  will  endow  the
social network with some new prominent features and social functions worthy of further research.

In recent years, the emotional impact of social media on the society has been confirmed by more and

more research findings and empirical cases, and thus has drawn great attention to it from a variety of
areas such as Internet marketing research, service comments analysis, social mood monitoring, and



Corresponding author. Tel.: +86 2125011241.

E-mail addresses:

whdai@fudan.edu.cn

(W.H. Dai),

handongmei19610320@gmail.com

(D. M.

Han),

dyh822@163.com

(Y. H. Dai),

dx2013@columbia.edu

(D. R. Xu)

*Manuscript

Page 3 of 18

Accepted Manuscript

emergent event management [5, 8, 9, 22, 27]. Social media has been considered as a sensor to perceive
and  predict  society’s  behaviors  in  the  real  world  through  the  data-mining  techniques  on  emotional
information [5, 27]. Since Professor R. W. Picard at Massachusetts Institute of Technology proposed in
his  book  “Affective  Computing”  in  1997  that  computer  can  capture,  process  and  reproduce  human
emotions  [40],  this  issue  of  human’s  emotion  recognition  and  computing  has  been  explored  by  the
technology  of  machine  intelligence.  Among  these  efforts,  emotion  recognition  is  to  identify  the
possible types of emotions from the signals, and this can be regarded as a task for pattern recognition,
while  the  affective  computing  usually  requires,  furthermore,  a  quantitative  measurement  on  that
emotions. It is commonly related to the issue of value estimation based on a trained model.

So far, researchers have proposed a series of computational models for analyzing emotional

information from the data of social media [14, 19, 26, 48]. Due to the widespread application of voice
instant  messaging  tools,  emotion  recognition  and  computing  on  emerging  vocal  media  has  become  a
new  concerning  hot  point  for  research  in  data-mining  of  social  media  analytics.  Although  the  vocal
emotion  recognition  has  made  great  progress  in  the  past  decades,  from  the  speaker-dependent  and
template  matching  recognition  based  on  simple  vocabulary  to  today’s  speaker-independent  and
statistical  model  based  recognition  that  can  process  a  continuous  speech  [19,  29],  but  there  are  still
barriers towards dealing with the vocal social media. The speech signal in vocal social media appears
as  the  human’s  conversation  using  natural  language,  and  in  most  cases  contains  the  mixed  emotions
embedded with dynamic changes. This signal can’t be recognized simply as one of the typical emotions
by  the  existing  methods.  Computing  such  complex  and  dynamic  emotions  precisely  is  actually  more
technically  difficult.  Therefore,  this  issue  had  to  be  studied  in  more  depth.  This  paper  aims  at
developing an effective computational method for processing the complex and dynamic emotions from
the speech signals of vocal social media, so that the propagation effects of emotions may be analyzed in
a meticulous and deep-going way.

2. Literature review

2.1. Emotion recognition of vocal signals

Vocal emotion recognition involves the issues of emotion classification, signal pre-processing,

feature  extraction,  and  pattern  recognition.  How  to  classify  and  describe  human’s  emotions  has
remained  to  be  a  controversial  issue.  The  classification  of  emotions  follows  into  the  two  categories:
discrete form and continuous  form. Discrete  form only gives the emotion kinds such as the  “six big”:
Anger, Disgust, Fear, Joy, Sadness, and Surprise [12]. Continuous form describes the emotion state in a
continuous  space  with  different  dimensions.  Among  which,  the  one-dimension  only  classifies  the
positive or negative emotions and determines their strength; the 2-D space are usually based on the H.
Hidenori and T. Fukuda’s Emotional Space

[20], where the emotion state is represented in a unit circle

with two opposite coordinates: Peace vs. Excitement, Happiness vs. Sadness; The 3-D space has
different models presented by W. M. Wundt

[54], H. H. Schlosberg [42], C. E. Izard [24], and C. E.

Osgood [39] respectively. Based on a comprehensive psychological research work, A. Mehrabian
demonstrated that any kind of an emotion state can be well described by the three nearly independent
continuous

dimensions:

Pleasure-Displeasure

(P),

Arousal-Nonarousal

(A),

Dominance-Submissiveness (D), and therefore proposed the famous PAD model [35, 36]. This model
provides an effective means for evaluating the complex emotion, and has been successfully applied to
the subjective measurement by manual manner in a variety of areas [25, 33, 49]. Fig.1. shows the
continuous form of emotions in different dimensions.

Fig.1. Continuous form of emotions in different dimensions

Page 4 of 18

Accepted Manuscript

In the procedure of signal pre-processing, the initial speech signal will be dealt and transformed so

as to be suitable for the extraction of its acoustic feature parameters. Generally speaking, this procedure
includes three steps: signal sampling and quantizing; pre-emphasis, framing and windowing [50]. In
this regard, one of the most important contributions is that Dynamic Time Wap (DTW) and Vector
Quantification (VQ) were presented in 1970s to handle the problems arising from the different lengths
of speech signals

[41]. The feature extraction is to find the appropriate and effective parameters which

can be used to identify the emotions from the vocal signal. The commonly used acoustic parameters are
divided into three categories [19]: (1) Prosody parameters such as the duration, pitch and energy of a
vocal  signal;  (2)  Spectral  parameters  such  as  the  LPC  (Linear  Predictor  Coefficient),  OSALPC
(One-Sided  Autocorrelation  Linear  Predictor  Coefficient),  LFPC(Log-Frequency  Power  Coefficient),
LPCC  (Linear  Predictor  Cepstral  Coefficient),  and  MFCC  (Mel-Frequency  Cepstral  Coefficient);  (3)
Sound  quality  parameters  such  as  Format  Frequency,  Bandwidth,  Jitter,  Shimmer,  and  Glottal
Parameter.  In  the  above  categories,  prosody  parameters  are  the  basic  parameters  for  vocal  emotion
recognition.  Human  auditory  system  is  a  special  nonlinear  system  so  as  to  respond  selectively  to  the
different frequency signals. MFCC is based on the known variation of the human ear’s bandwidths. It
has  the  frequency  characteristics  linearly  below  1000  Hz  and  logarithmically  above  1000  Hz,  which
match  well  with  the  auditory  characteristics  of  human  speech  signals.  Experiences  show  that  the
performance of MFCC parameters is usually better than the other spectral parameters

[17, 37]. Recent

experiments have found that the sound quality parameters play an important role in differentiating the
emotions associated with attitudes and intentions, therefore the combined parameters with all three
categories to be applied to the feature extraction may be the new trend [10, 56].

In the pattern recognition, methods are usually based on Hidden Markov Model (HMM), Artificial

Neural Network (ANN), Gauss Mixture Model (GMM), Support Vector Machine (SVM) and Bayesian
Classification. T. L. Nwe et al. reported that the six typical emotions including anger, distaste, fear, joy,
sadness and surprise can be recognized at the accuracy rate of 78% in their paper Speech Emotion
Recognition Using Hidden Markov Models [38]. In Toward Detecting Emotions in Spoken Dialogs, C.
M Lee and S. S. Narayanan recognized correctly the “positive” and the “negative” emotions from the
dialogues of call voice by using the combined information from the speech and its converted texts

[28]

By HMM and GMM, B. Schuller et al. studied the recognition of seven emotional states and obtained
the correct rate of 86.8% [43]. M. W. Bhatti et al. developed a modular neural network to identify the
six typical emotions and reached the rate of 83% [4]. Through the combination of acoustic features and
linguistic information, B. Schuller et al. explored three different methods based on SVM, and achieved
the accuracy rates of 93% (speaker-dependent) and 81% (speaker-independent) [29, 44].    The research
progress  on  speech  emotions  were  overall  summarized  by  D.Ververidis  and  C.  Kotropoulos  in
Emotional Speech Recognition: Resources  Features and Methods [50]. Actually,  the  accuracy  rate of
emotion  recognition  depends  mostly  on  the  training  samples.  Generally  speaking,  we  think  that  the
reliable  rate  may  be  70%-80%  based  on  the  sufficient  training  samples.  Another  problem  in  existing
recognition  methods  is  that  the  reference  emotion  states  to  be  taken  as  the  target  of  the  machine
training  are  all  given  by  people’s  subjective  evaluation,  and  thus  the  evaluation  error  will  obviously
lead to the deviation while compared with the emotion state in a real world.

In recent years, researchers have a more profound understanding on the neural mechanism of

human’s  emotions  due  to  the  developing  experimental  technology  of  fMRI  (functional  Magnetic
Resonance  Imaging),  ERPs,  (Event-related  Potentials)  and  DTI  (Diffusion  Tensor  Imaging).  In
particular, the blood oxygenation level dependent functional magnetic resonance imaging (Bold-fMRI),
with  such  advantages  as  being  non-invasive,  non-traumatic  and  capable  of  locating  accurately  the
activated brain areas, has been applied to the studies the emotions and achieved a number of significant
findings  [21].  Based  on  the  fMRI  technology,  we  have  studied  the  emotional  cognitions  on  the
information of emergent events [23]. The brain activated characteristics in that cognition are shown as
in Fig.2 by our experimental observation.

Fig. 2. Brain activated characteristics of emotional cognitions on the information of emergent events

Page 5 of 18

Accepted Manuscript

From Fig. 2, we can find that some brain areas are activated when the emotion takes place, and this

will  help  to  establish  an  objective  method  for  the  evaluation  of  emotion  states.  Through  a  further
research work, we have analyzed the brain mechanism of vocal emotion and suggested this mechanism
to be applied to the emotional intelligence design of a humanoid robot [52]. The emotions between the
speaker and listener are different, so we  should determine that our recognition target is to identify the
speaker’s  emotions  from  his  speech,  or  to judge  the  activated  emotions  of  the  listener  when  he  hears
the  speech.  From  the  view  of  emotion  propagation,  the  target  is  usually  set  up  to  the  later,  and  thus
we’ll  recognize  the  activated  emotions  according  to  the  statistical  significance  of  a  group  of  the
possible listeners, and the result may be a statistical distribution of the typical emotions. If the target is
to identify the speaker’s emotion, the more objective judgment may be made with the help of machine
detection due to the limitation and differences in the expressive abilities of  the speakers, especially in
the complex emotion states. When the judgment is only possibly made through a subjective evaluation,
our research finding indicates that the speaker’s familiar people can make more accurate judgment.  It
seems  that  this  judgment  is  made  based  on  some  more  information  beyond  the  speaker’s  speeches.
Overall,  there  are  a  lot  of  issues  on  the  recognition  of  vocal  emotions,  and  this  needs  the
comprehensive  and  interdisciplinary  research  work  from  the  neuroscience,  psychology  and  computer
engineering.  At  least,  the  existing  method  should  be  improved  to  recognize  the  mixed  emotions  as  a
statistical distribution of the typical emotions.

2.2. Affective computing on vocal emotions

Affective computing requires the emotion to be measured quantitatively. This involves the issues of

the descriptive model and computing model of emotions. The descriptive model should not only reflect
the  continuous  variations  in  an  emotional  space  with  some  certain  dimensions,  but  also  can  calculate
the  distance  between  the  different  emotions.  As  previously  described  in  this  paper,  A.  Mehrabian
proposed  the  famous  PAD  model  [35,  36].  In  his  model,  the  emotion  state  can  be  described  in  a
continuous 3-D space. Experiments and statistical studies have shown that all the known emotion states
can be almost described in this space very well [6, 30]. The prominent superiority of this model is that
the complex and mixed emotions as well as their dynamic changes can be described in the three nearly
independent

continuous

dimensions:

Pleasure-Displeasure

(P),

Arousal-Nonarousal

(A),

Dominance-Submissiveness (D). This means that the values in different dimensions may be evaluated
by subjective evaluation or calculated by machine respectively.

In order to achieve a precise and consistent result in subjective evaluation, A. Mehrabian designed

the  initial  34-item  test  questionnaire  to  conduct  a  reliable  and  valid  evaluation  [35].  But  the  later
researches  and  tests  showed  that  the  questionnaire  should  be  designed  according  to  the  specific
language due to the differences in language understanding and cultural backgrounds. Therefore,  Y. Lu
et  al.  from  Psychological  Institute,  Chinese  Academy  of  Sciences  put  forward  a  simplified  12-item
Chinese questionnaire [33]. This questionnaire passed the reliability and validity test [30] and has been
widely  accepted  to  evaluate  the  emotions  in  Chinese  language.  In  the  measurement  of  the  distance
between  two  different  emotional  states,  P.  H.  Sun  and  L.  M.  Tao  carried  out  a  series  of  psychology
experiments and found that PAD space is not an isotropic Euclidean space. To solve this problem, they
presented  a  conversion  metric  function  for  calculating  the  Euclidean  distance  of  emotional  states

PAD space [46]. So far, PAD model has been successful applied in a variety of areas such as
audio-visual speech synthesis [25], micro-blog sentiment analysis [6], and music emotion comparison
[34].

The research work of affective computing on continuous vocal signal has just got started in the very

recent years. It requests to establish the quantitative relationship between the emotions and the acoustic
feature  parameters  of  a  continuous  vocal  signal,  rather  than  only  recognizing  one  of  the  typical
emotions from the above acoustic parameters. H. Zhou tested the correlations of PAD values with the
prosody parameters and MFCC, and found no significant correlations between the values of D with the
above  parameters  [60].  Therefore,  she  presented  a  SVR  (Support  Vector  Regression)  model  for
estimating  the  values  of  P  and  A  from  a  continuous  vocal  signal  based  on  the  parameters  of
Hilbert-Huang  transformation.  Her  research  findings  indicated  that  more  acoustic  feature  parameters
would be considered in the affective computing on vocal emotions to obtain effective PAD values. In
the speech conversion and synthesis, PAD model has been very successfully used to adjust and produce
the different affective speeches from a neutral speech or text sentence. J. Jia, et al. proposed a unified
model for emotional speech conversion and audio-visual speech synthesis using Boosting-GMM [25].
In their model, the target PAD values were employed as part of the input variables. Y. X. Chen and R.
T. Long discussed the synthesis of emotional speech by taking the PAD values to adjust the parameters

Page 6 of 18

Accepted Manuscript

of a HMM speech synthesis system [7]. S.W. Gilroy, et al. presented a framework for the multimodal
affective fusion in human-machine interface [15]. This framework provided an effective method for the
fusion computing on affective speech, spontaneous behavior, multi-keyword spotting, and interest
aggregation based on PAD model.

From the existing research work, we can conclude that the affective characteristics in a continuous

vocal signal may be determined and adjusted by its PAD values. Therefore, affective computing on
vocal emotions can be explored as the problem of PAD value estimation from the affective acoustic
feature parameters of a vocal signal.

3. Proposed method and research schema

3.1. Proposed method

Based on the comprehensive analysis of the previous research work, we hereby turn the emotion

recognition and affective computing on vocal social media into the PAD value estimation from the
extracted acoustic feature parameters of its speech signal, and propose the computational method as
shown in Fig.3.

Fig.3. Proposed computational method

Our proposed computational method includes following 5 steps:
(1)Pre-processing: The speech signal of vocal social media should be firstly dealt by a

pre-processing  to  satisfy  for  the  acoustic  feature  parameter  extraction.  The  standard  procedure  of
pre-processing will experience signal sampling and quantizing, pre-emphasis, framing and windowing
[28, 50, 56]. Speech signal in vocal social media usually exhibits as a sequence of  short chats, so the
emotion  recognition  and  affective  computing  can  be  estimated  for  each  chat  on  the  average  level.  In
order  to  reflect  the  dynamic  changes  of  emotions  in  a  long  chat,  a  fixed  time  interval,  for  example  6
seconds, may be set as the calculation period.

(2)Acoustic feature parameter extraction: The next step is to extract the suitable acoustic feature

parameters from the processed signal data. The acoustic feature parameters should not only represent
the affective characteristics in the vocal signal as precisely as possible, but also be computed effectively.
In our method, we choose 25 parameters from all the three

categories: prosody parameters, spectral

parameters and sound quality parameters. This issue will be discussed in the later.

(3)PAD value estimation: The PAD values will be thereafter estimated from the above acoustic

feature parameters by machine learning. Some commonly used non-linear estimators in the machine
learning, such as HMM, ANN, GMM, and SVR, can be considered in this issue. Among which, the
Least Squares SVR (LS-SVR), presented by J. A. K. Suykens and J. Vandewalle in 1999

[47], has the

advantages of superior stability, good generalization ability, and high efficiency, and is therefore
adopted as the estimator in our method.

(4)Affective computing: Based on the trained LV-SVR estimator by machine learning, the PAD

values of speech signal can be estimated as the affective computing results. The dynamic changes of

Page 7 of 18

Accepted Manuscript

PAD values may be also illustrated according to the sequence of short chats or depending on the fixed
time interval orders in a long chat.

(5)Emotion recognition: Furthermore, the above PAD values in each period can be expressed as the

percentage distributions of the typical emotions based on their converted Euclidean distances in PAD
space [46], and thus get the recognition results of the mixed emotions.

3.2. Research schema

In order to achieve the goal of the proposed method, the following issues need further research

according to the characteristics of vocal social media:

(1) The choice of acoustic feature parameters. Because the emotions may be stimulated either by the

voice or by the semantic information in a speech of vocal social media, so the above parameters require
to be tested to show that they are only related to the vocal emotions and independent of the semantic
information. Besides, those parameters should carry the emotional information sufficiently but
necessarily, and can be computed effectively.

(2) The training samples for LV-SVR estimator. Although researchers have developed various vocal

emotion corpora, 80% of their samples are usually used for the training with the left of 20% for the test,
but so far there has been no one can be suitable for the vocal social media which samples need to be
refined from the real environment at representative significance.

(3) The generalization ability in real application. As the interactive activities on vocal social media

usually take place in the “small world” of a group with familiar members, so the proposed method
should be generally applicable for different groups.

Our researches are focused on the choice of acoustic feature parameters and the experimental studies

of the proposed method associated with the above issues. We hereby draw up our research schema in
accordance with the research paradigm on vocal emotion as Fig.4:

Fig.4. Research schema

Different from the speaker’s recognition and semantic recognition which try to reduce the emotional

effects from the vocal signal [29, 50, 59], emotion recognition on the traditional paradigm will utilize
as  much  as  possible  information  which  is  related  to  emotion  and  independent  of  semantic  context.
From the previous research findings, the affective characteristics on vocal signal are relevant to all the
three  categories  of  acoustic  feature  parameters:  prosody  parameters,  spectral  parameters  and  sound
quality  parameters.  Based  on  our  test  experiences,  we  choose  the  25  parameters  from  the  above
categories which can be better to reflect the PAD values in the speech of vocal social media: Short-time
Energy (Max, Min, Mean), Pitch (Max, Min, Mean), Short-time zero crossing rate (Max, Min, Mean),
First  Formant,  Second  Formant,  Voice  speed,  Number  of  voice  breaks,  and  the  12-order  MFCC  (12
coefficients).

In the experimental studies, the computational accuracy of proposed method as well as its chosen

acoustic feature parameters should be tested to show that it is only related to the vocal emotions and
independent of the semantic information in a speech. CASIA is the widely used standard testing corpus
in Chinese [18]. There are total 1200 speeches in this corpus which have almost the same structure and
length in each speech. Every speech with the same semantic texts is spoken by 2 men and 2 women in
the six typical emotional tones: happy, sad, angry, surprise, fear, and neutral. So the recognition rates of
the above six emotions can be used to evaluate the reliability and validity which is only related to the
emotions in this proposed method.

Page 8 of 18

Accepted Manuscript

After having passed the test by standard corpus, an experiment with the real data from the vocal

social media will be conducted to verify the

effectiveness of the proposed method in application. The

speech signal on vocal social media  is characterized by a series of conversational  chats with different
structures and lengths. It is very different from the speech signal in the existing standard testing corpora
such  as  CASIA.  So  the  training  samples  for  LV-SVR  estimator  in  real  application  can’t  be  from  the
existing standard testing corpora. The conversational chats on vocal social media usually take place in
the  “small  world”  of  a  group  with  familiar  members.  As  discussed  before  in  this  paper,  the  familiar
people  can  make  more  accurate  judgment  on  speaker’s  emotion  information,  therefore  the  training
samples  would  be  better  to  acquire  from  the  historical  data  of  the  members  in  the  same  group  if
possible.  In  order  to  extract  the  personalized  features  of  the  participators  for  studying  the  emotion
propagation in the “small world” more precisely, we take 180 real chats from the historical data in the
same  group  on  Wechat  as  the  training  samples  for  machine  learning.  To  ensure  the  representative
significance,  the  above  chats  are  from  9  of  the  most  active  members  with  equal  20  chats  for  each
member.  After  trained  by  those  samples,  the  experiment  of  a  discussion  is  conducted  in  this  group
based on a serious actual incident in food safety to verify the effectiveness of LV-SVR estimator on the
dynamic analysis of emotion propagation.

Generalization ability refers to the machine learning algorithms for the adaptability of new samples.

We here understand it as the applicability for different groups on vocal social media. As discussed
above, the accuracy of LV-SVR estimator in real application is dependent on its trained samples. So the
samples for training and the samples to be estimated in the future should belong to the same statistical
collection in the regular generalization ability examination of a computational method. If the groups on
vocal social media have significant statistical differences in the factors which will affect the estimated
result, the samples for training would be preferably from the historical data of the same group as to be
estimated. However, the historical data are sometimes difficult to obtain when we process with a new
group on vocal social media. In this case, we had to extract the training samples from the historical data
which is close to the new group to be processed. In the generalization ability examination of our
proposed method, we try to use the LV-SVR estimator which has been trained by the historical data of
the group on Wechat to the application of a new group on QQ, and evaluate the influence caused by the
trained samples on the generalization ability.

4. Model and training

4.1. Vector model of acoustic feature parameters

In our proposed computational method, the following 25 parameters are chosen for PAD value

estimation: Short-time Energy (Max, Min, Mean), Pitch (Max, Min, Mean), Short-time zero crossing
rate (Max, Min, Mean), First Formant, Second Formant, Voice speed, Number of voice breaks, and the
12-order MFCC (12 coefficients). Among which, the Short-time Energy, Short-time zero crossing rate,
Pitch, First Formant, Second Formant, Voice speed, Number of voice breaks can be simple calculated
directly in the time domain of the vocal signal [50]. We hereby just discuss the calculation of MFCC.

Let

)

represent the original speech signal, and

represent the signal’s frequency, therefore the

calculation of MFCC is based on the Mel-frequency scale as the Formula (1):

2595*lg 1

700

Mel

















(1)

After processing by a window function, the speech signal

)

will be sampled and turned into the

short time signal

)

. Through the FFT (Fast Fourier Transform),

)

which be converted from the

time domain into the frequency domain and thus we get the power spectrum

( )

S n

Prior to this, it needs to set up a number of band pass filters, such as the Hamming filters described

in Formula (2)

,...,

;

,...,

)

(









(2)

Here,

stands for the number of filters over the frequency spectrum range, which is included in

dynamic Mel filter bank, and is set typically as 24;

is the point number of one frame, which is set

Page 9 of 18

Accepted Manuscript

as  256  to  be  suitable  for  the  speech  signal.  In  frequency  domain,  the  center  frequency  of  filter  is
uniformly  distributed  throughout  Mel  frequency  axis.  The  extraction  algorithm  of  MFCC  parameters
includes the steps as in Fig.5 [17, 37]. Fig. 6 demonstrates the waves of the original speech signal on
Wechat and its MFCC coefficients.

Fig.5. Calculation process of MFCC coefficients

Fig.6. Original speech signal on Wechat and its MFCC coefficients

We hereby take the chosen 25 acoustic parameters to compose a feature vector model as the Formula

(3):





MFCC

NVB

SZC

)

(



(3)

Where, SE represents the collection of Max, Min, and Mean values of the Short-time Energy; P

represents the same collection of Pitch; SZC represents the same collection of Short-time zero crossing
rate; FF represents the value of First Formant, Second Formant; SF represents the value of Voice speed;
NVB represents the value of Number of voice breaks; and MFCC represents the collection of the values
of its 12 coefficients. This vector will be used as the input of the LS-SVR for estimating its PAD
values.

4.2. Emotion recognition and affective computing

-1

-0.5

0.5

Sample of Wechat voice wave

Time (second)

500

1000

1500

2000

2500

-40

-20

MFCC

Frame No

Page 10 of 18

Accepted Manuscript

The emotion recognition and affective computing are based on the trained LV-SVR model as

follows [47]:

Set







as the collection of the training samples, where the input



and the

output



, so the LS-SVR regression model in a high dimensional space can be described as:





)

(

)

(





(4)

Here,



is the vector of the weights, and

)



is the non-linear function for mapping the input

to that high dimensional space,

is the error constant. Therefore the estimation of

)

can be

transformed into the following optimization problem:









)

(

min





(5)

Subject to:



)

(

)

(











(6)

Where,



is the normalized constant,

is the error variable of

. Set the Lagrangian function as:

















)

(

)

(

)

(













(7)

Where,



is the Lagrangian multiplier satisfying





solving extreme value point of

)

(





, we get the matrix of the linear equation:















































)

(



(8)

Where,

)

(





)

(

is the core function which must satisfy the Merce condition.

Here, we choose RBF (

Radical Basis Function

) as the core function:

)

exp(

)

(







(9)

Where,



is the width of RBF,





. Calculating the Lagrangian multiplier



and

the constant

, we get the LS-SVR estimation model:









)

(

)

(



(10)

Combined with the Formula (9), we get the final LS-SVR model:















)

exp(

)

(





(11)

In the computation of LS-SVR model, the normalized constant



and the width of RBF



have

major influence on the accuracy of estimated result. To avoid the fitting problem on the training sample,
we adopt the cross validation method [45, 55] to choose the most suitable values of



and



in our

method.

Due to the nearly independent characteristics, the P, A, D values can be estimated by the trained

LV-SVR  model  respectively.  Based on estimated PAD values, the emotion state  may be converted to
the  percentage  distributions  of  some  typical  emotions  which  can  describe  mixed  emotions  as  well  as
their  changes.  In  that  conversion,  because  the  PAD  space  is  not  an  isotropic  Euclidean  space,  so  a
conversion metric function presented by P. H. Sun and L. M. Tao [46] should be used to calculate the
converted Euclidean distance to the typical emotions:

Page 11 of 18

Accepted Manuscript

)

(











(12)

Where,



and



are the squared variances of the two emotion states to be calculated. The

percentage of each typical emotion is a based on its converted Euclidean distance to the estimated
emotion state.

4.3. PAD annotation of training samples

In order to provide the reference for machine training, the PAD values of training samples should be

evaluated  as  the  annotation  by  a  manual  manner.  Psychologists  have  design  a  strict  method  for
ensuring  the  reliability  and  validity  of  this  evaluation  [30].  Here,  we  used  the  Chinese  version  of
simplified 12-item questionnaire as in Table 1[33].

Table 1. Chinese version of the simplified 12-item questionnaire

Question

Emotion

-4

-3

-2

-1

Emotion

Angry

Activated

Q2:Wide-awake - Sleepy; Q3:Controlled - Controlling; Q4:Friendly - Scornful;

Q5:Calm - Excited; Q6:Dominant - Submissive; Q7:Cruel - Joyful;

Q8:Interested - Relaxed; Q9:Guided - Autonomous; Q10:Excited - Enraged;

Q11:Relaxed - Hopeful; Q12:Influential - Influenced

The scores are assessed based on what kind of feelings is more intense in each item. From the left to

the right, the calibration for scoring records is scaled“- 4” to “4”, and in the middle is “0”.
Finally, the scores will be converted as the normalized values of P, A, D [25]:









(13)











(14)









(15)

As previously described in this paper, our research found that the familiar people can make more

accurate judgment on this evaluation, so the scores are all assessed by the people in the same group on
vocal social media.

5. Test and application

5.1. Test of acoustic feature parameters

The acoustic feature parameters in the vector of Formula (3) require to be tested to show that they

are only related to the vocal  emotions and independent of  the  semantic information in a speech. This
test is carried out with the CASIA, a widely used standard corpus for Chinese language test [18]. In this
corpus,  each  speech  with  the  same  semantic  texts  is  spoken  by2  men  and  2  women  in  six  different
emotional tones: happy, sad, angry, surprise, fear, and neutral, and therefore the recognition rates of the
above  six  emotions  can  be  used  to  evaluate  the  reliability  and  validity  which  is  only  related  to  the

Page 12 of 18

Accepted Manuscript

emotions in this proposed method.

Based on the chosen parameters in the vector of Formula (3), we apply our LV-SVR model to

estimate the PAD values of each emotional speech and convert the values into the most possible typical
emotion state by Formula (12). Table 2 shows the test result of the recognition rates.

Table 2. Test result of recognition rates

Emotion type

Recognition rate

happy

81.32%

sad

85.27%

angry

87.72%

surprise

77.69%

fear

79.37%

neutral

83.23%

Average

82.43%

The test result shows that recognition rates of happy, sad, angry, surprise, fear, and neutral are

81.32%, 85.27%,87.72%, 77.69%, 79.37%, 83.23% respectively, and the average rate reaches 82.43%,
which are higher than the existing results reported by the similar tests.

5.2. Training from the historical data

Due to the special characteristics of speech signal on vocal social media, the samples in existing

standard testing corpora such as CASIA are not suitable for the machine training in real application. We
get the training samples of 180 chats from the historical data on Wechat. The above chats are from 9
members in the same group with equal 20 chats for each member.

Table 3 shows the PAD values of the six typical emotions from the 180 chat samples and reflects the

personalized features of the members in this group.

Table 3. PAD values of the six typical emotions

Emotion

Category

Value

Neutral

0.05

-0.22

-0.02

Angry

-0.53

0.43

0.64

Fear

-0.35

0.52

-0.68

Happy

0.49

0.31

0.29

Sad

-0.26

-0.34

-0.52

Surprise

0.21

0.57

0.13

5.3. Experiment and application

The experiment aims to verify the

effectiveness of the proposed method in real application. It is

carried out on the same group with the trained samples. We conduct the discussion about a serious
actual incident in food safety reported by the media on July 20, 2014, Shanghai China. In this
experiment, we collect 52 chat speeches talking about this incident on Wechat from 37 members with
the total duration of 35

minutes and 11seconds.

Table 4 shows the information of the chats as well as their estimated PAD values by the trained

LV-SVR model with





. Compared with the subjective evaluation, the averaged relative error

of estimated PAD values is 13.76%, but the converted typical emotions match with the results of
subjective evaluation very well.

Table 4. Chats on Wechat and the estimated PAD values

Chat No.

Start time

End time

Speaker

Referred

Estimated PAD values

Page 13 of 18

Accepted Manuscript

listeners’ ID

by the LV-SRV model

00:00:00

00:00:07

NO.001

All

(-0.692, 0.617, 0.891)

00:00:11

00:00:29

NO.002

All

(-0.712, 0.721, 0.913)

00:00:33

00:00:39

NO.003

All

(-0.156, 0.525, -0.192)

00:00:42

00:00:56

NO.004

NO.003

(-0.101, -0.311, -0.114)

00:01:00

00:01:18

NO.005

All

(-0.697, 0.633, -0.907)

00:01:23

00:01:33

NO.003

All

(0.105, -0.251, 0.002)

00:01:45

00:02:02

NO.002

NO.003

(-0.655, 0.626, 0.866)

00:02:17

00:02:30

NO.006

All

(-0.803, 0.609, 0.798)

00:02:39

00:02:56

NO.007

All

(-0.653, 0.707, 0.863)

00:03:10

00:03:45

NO.003

All

(-0.682, 0.644, 0.743)

00:03:52

00:04:14

NO.004

All

(-0.131, -0.267, -0.225)

00:04:21

00:04:48

NO.004

NO.001

(0.102, -0.312, -0.123)

00:04:59

00:05:16

NO.008

All

(0.104, -0.261, -0.002)

00:05:26

00:05:46

NO.004

All

(0.111, -0.107, -0.211)

00:05:56

00:06:12

NO.003

All

(0.108, -0.119, -0.111)

…

00:35:11

00:35:27

NO.001

All

(0.107, -0.351, 0.022)

We illustrate the chat activities and their PAD changes in the group ordered by the chat number as

Fig.7. It can demonstrate the dynamic process of emotion propagation in this group as well as the
reactions caused by each chat activity.

Fig.7. Dynamic process of emotion propagation in the group

For the purpose of the intuitive observation, we convert the PAD values of chat speeches into the

continuous one-dimension with the positive and negative coordinates as shown in Fig.8:

Fig.8. Dynamic process of emotion propagation in positive and negative coordinates

...

-1

-0.8

-0.6

-0.4

-0.2

0.2

0.4

0.6

0.8

Chat No

ale

f e

ion

lar

ity

Emotion of wechat voice

ID:001

ID:002

ID:003

ID:004

ID:005

ID:003

ID:002

ID:006

ID:001

ID:003

ID:004

ID:008

ID:004

ID:003

ID:001

Page 14 of 18

Accepted Manuscript

From Fig.8, we can find that this propagation started from No.001 with a strong negative emotion to

this group, hereafter negatively enlarged by No.002, and finally stopped at No.001 in the nearly neutral
emotion. In whole process, No.002 contributed the most negative emotions, No.004 acted as the most
active participant, and No.003 had the most impacts on the group who looked like the opinion leader.

From Fig.7 and Fig.8, we can furthermore study more precisely on the attentions and the roles of

each participant as well as their relationships in the social network, and therefore establish the dynamic
model to describe the emotion propagation quantitatively through a further analysis of the dominant
factors on which. In order to examine the generalization ability of our proposed method, we apply the
LV-SVR estimator which has been trained by the historical data of the group on Wechat to calculate the
following 28 second duration of vocal message from another group on QQ:

“Wow! Even houses can be printed! Really cool! But 3D technology is not fully developed yet, so I

guess people are still playing with the concept. The 3D printing business will be in big trouble if it fails
to meet the increased demands and upgrade the technology, especially when it turns out that the printed
houses are actually not livable.”

This message contains the dynamics changes of different emotions, so we divide it into 5 segments

by the sampling period of 6 seconds. Fig.9 shows the waves of the 5 segments with the duration of 6 s,
6s, 6s, 6s, and the rest 4s

respectively. The acoustic feature parameters and estimated PAD values by

the LV-SVR estimator are listed on Table 5.

Fig.9. Segmental waves of the vocal message

Table 5. Acoustic feature parameters and PAD values of vocal message in 5 segments

Acoustic feature

parameters

Duration time (s)

Short-time Energy

(Max/Mean /Min)

85.525
77.147
22.003

84.904
74.365
35.990

83.423
74.920
37.884

84.691
75.416
38.181

88.263
81.924
44.101

Pitch

(Max/Mean /Min)

279.762
119.909

90.602

452.043
130.802

95.713

445.153
134.964

47.163

252.689
115.927

49.030

468.043
169.083

98.563

Short-time Zero

Crossing Rate

(Max/Mean /Min)

282

222.627

112

304

231.951

124

302

232.26

122

298

235.42

122

278

208.125

12-order MFCC

Coefficients

4.8790

-3.3281
-2.0294

0.5616

4.5146

-1.9265
-1.8245
-0.1242

3.9430

-0.0019
-2.1559
-0.0149

3.7301
0.4819

-2.2706

0.5528

3.4227

-0.3393
-2.8358
-0.7166

-1

wave of state 1

-1

wave of state 2

-1

wave of state 3

-1

wave of state 4

0.5

1.5

2.5

3.5

-1

wave of state 5

Time(second)

Page 15 of 18

Accepted Manuscript

-1.3772
-0.0616

0.0176

-0.0196
-0.0002
-0.0249
-0.0094

0.0006

-1.2839
-0.1120
-0.0509

0.0324
0.0369

-0.0043
-0.0139

0.0033

-1.2215
-0.0650

0.0240
0.0976
0.0319

-0.0277
-0.0047
-0.0002

-1.6424
-0.0866
-0.0412
-0.0146
-0.0332
-0.0316

0.0365

-0.0044

-1.7581
-0.0686

0.0205
0.0317
0.0345

-0.0073
-0.0281
-0.0033

First Formant

609.567

600.916

569.877

570.547

533.855

Second Formant

1347.997

1462.478

1567.844

1528.431

1523.003

Voice Speed

0.196/s

0.187/s

0.186/s

0.414/s

Number of Voice

Breaks

0.503

0.109

-0.159

-0.255

-0.744

0.316

-0.318

-0.338

-0.518

0.434

0.312

-0.112

-0.211

-0.213

0.532

From the estimated PAD values, the dynamic changes of emotion states in each segments of the

vocal message follow the process: State 1（0.503, 0.316, 0.312）-> State 2（0.109, -0.318, -0.112）->
State 3（-0.159, -0.338, -0.211）-> State 4（-0.255, -0.518, -0.213）-> State 5（-0.744, 0.434, 0.532）,
as shown in Fig.10.

Fig.10. Dynamic changes of emotion states in vocal message

We hereafter conduct a subjective evaluation on the vocal message by 61 persons of this QQ group

and find the dynamic changes of emotion states can match the evaluation results correctly. It shows that
our proposed method can also be  applicable for this group on QQ. However the error of  PAD values
between the machine’s estimation and the subjective evaluation has been enlarged by 11.24% than that
of the former group on Wechat. It indicates that the personalized features in the members of different
groups on social media have significant impacts on the estimation of PAD values, so the choice of the
samples  for  training  should  consider  the  similar  degree  with  the  group  to  be  applied  as  much  as
possible.

6. Conclusion and discussion

The widespread use of emerging vocal social media has facilitated people’s communication as well

as the emotion propagation on social networks greatly, and is therefore bringing the enormous impacts
on social psychological cognition and group behaviors than ever before. This issue of emotion
recognition and affective computing on emerging vocal media has become a new concerning hot point
in social media analytics. The speech signal in vocal social media appears as the human’s conversation
using natural language, and in most cases contains the mixed emotions embedded with dynamic
changes. Its complexity needs further research on the emotion recognition and computation.

-1

-0.5

0.5

-1

-0.5

0.5

-1

-0.5

0.5

Pleasure Value

Dynamic changes of emotion states

Arousal Value

State 1

State 2

State 3

State 4

State 5

Page 16 of 18

Accepted Manuscript

Through the comprehensive analysis of previous research work, this paper proposed a computational

method by turning the issue of emotion recognition and affective computing on vocal social media into
the PAD value estimation from the extracted 25 acoustic feature parameters of speech signal based on a
trained LV-SVR model. The choice of acoustic feature parameters, acquisition of training samples, and
the  generalization  ability  in  real  application  were  discussed  according  to  the  characteristics  of  vocal
social  media  under  a  research  schema.  Test  result  by  the  standard  corpus  and  experiment  in  real
application  showed  that  the  proposed  method  can  reach  a  high  accuracy  independent  of  the  semantic
information,  and  has  the  good  generalization  ability  for  different  social  media  groups.  It  provides  an
effective approach for computing and analyzing the dynamic propagation of mixed emotions on vocal
social  media.  However,  research  findings  in  this  paper  indicated  that  trained  samples  associated  with
the  personalized  features  in  the  members  of  social  media  groups  have  significant  impacts  on  the
accuracy  of  computational  results,  so  the  performance  of  the  proposed  method  has  to  be  tested  and
verified  through  the  large  samples  and  in  different  social  media  groups.  Besides,  the  precise
relationship  between  acoustic  feature  parameters  and  PAD  values  as  well  as  the  optimization  on  the
vector model of acoustic feature parameters are worthy of exploration in the further studies.

Acknowledgements

This research was supported by National Natural Science Foundation of China (No. 91324010,

No.41174007), Shanghai Philosophy and Social Sciences Plan, China (No. 2014BGL022）, and
Graduate Innovation Fund Program (No.CXJJ-2013-445) of Shanghai University of Finance and
Economics, China.

References

[1]A. S. Abrahams, J. Jiao, G. A. Wang, W. Fan, Vehicle defect discovery from social media, Decision

Support Systems. 54(1), 2012, pp.87-97.

[2]A. S. Abrahams, J. Jiao, W. Fan, G. A. Wang, Z. Zhang, What is buzzing in the blizzard of buzz:

automotive component isolation in social media postings, Decision Support Systems. 55(4), 2013,
pp.871-882.

[3]A. S. Abrahams, W. Fan, J. Jiao, G. A. Wang, Z. Zhang, An integrated text analytic framework for

product defect discovery, Production and Operations Management. forthcoming, 2014.

[4]M. W. Bhatti, Y. Wang , L. Guan, A neural network approach for human emotion recognition in

speech, Proceedings of ISCAS 2004, 2004, pp..81-184.

[5]J. Bollen, H. Mao, X..J. Zeng, Twitter Mood Predicts the Stock Market, Journal of Computational

Science. (2)1, 2011, pp. 1-8.

[6]H. T. Cao, Chinese Microblog Sentiment Analysis Based on the PAD Model, Dalian, China: Dalian

University of Technology, 2013.

[7]Y. X. Chen, R. T. Long, Trainable emotional speech synthesis based on PAD, Pattern Recognition

& Artificial Intelligence. 26(11), 2013, pp.1019-1025.

[8]W. H. Dai, H. Z. Hu, T. Wu, Y. H. Dai, Information spread of emergency events: Path searching on

social networks. Scientific World Journal. 2014(Article ID 179620), pp. 1-7.

[9]W. H. Dai, X.Q. Wan, X. Y. Liu, Emergency event: Internet spread, psychological impacts and

emergency management, Journal of Computers. 6(8), 2011, pp. 1748-1755.

[10]A. R. Damasio, Descartes Error: Emotion, Reason and the Human Brain. New York, USA:

Gosset/Putnam Press, 1994.

[11]R. J. Davidson, D. C. Jackson, N. H. Kalin, Emotion, plasticity, context, and regulations:

perspectives from affective neuroscience, Psychological Bulletin. (126), 2000, pp. 890–909.

[12]P. Ekman, M. J. Power, Handbook of Cognition and Emotion. Sussex: John Wiley & Sons, 1999.
[13]W. Fan, M. D. Gordon, The power of social media analytic, Communications of the ACM. 57(6),

2014, pp.74-81.

[14]X. L. Fu, L. H. Cai, Y. Liu,J. Jia,W.F. Chen, Z. Yi,G. Z. Zhao, Y. J. Liu, C. X. Wu, A computational

cognition model of perception, memory, and judgment. Science in China Series F: Information
Sciences, 57(3), 2013, pp. 1-15.

[15]S.W. Gilroy, M. Cavazza, M. Niiranen, E. Andre, T. Vogt, J. Urbain, M. Benayoun, H. Seichter, M.

Billinghurst, PAD-based multimodal affective fusion, Proceedings of Affective Computing and
Intelligent Interaction and Workshops, 2009, pp.1-8.

Page 17 of 18

Accepted Manuscript

[16]C. Gobl, A. N. Chasaide, The role of voice quality in communication emotion, mood, and attitude,

Speech Communication, 40, 2003, pp.189-212.

[17]W. Guo, R. H. Wang, L. R. Dai, Mel-Cepstrum integrated with pitch and information of

voiced/unvoiced, Journal of Data Acquisition and Processing. 22(2), 2007, pp. 229–233.

[18]W. J. Han, H. F. Li, A brief review on emotional speech databases, Intelligent Computer and

Applications. 3(1), 2013, pp.5-7.

[19]W. J. Han, H. F. Li, H. B. Ruan, L. Ma, Review on speech emotion recognition, Journal of

Software, 25(1), 2014,pp.37-50.

[20]I. Hidenori, T. Fukuda, Individuality of agent with emotional algorithm, Proceedings of 2001

IEEE/RSJ International Conference on Intelligent Robots and System. 2001, pp.1195-1200.

[21]R. Horlings, Emotion Recognition Using Brain Activity. Delft, Nederland: Delft University of

Technology, 2008.

[22]H. Z. Hu, D. Wang, W.H. Dai, L. H. Huang, Psychology and behavior mechanism of micro-blog

information spreading, African Journal of Business Management . 6(35), 2012, pp. 9797-9807.

[23]L. H. Huang, W. H. Dai, C. Zhou, Z. C. Xu, S. L. Li, Cognitive Model, Propagation Rules and

Early Warning Mechanism of Network Information in Unconventional Emergency Events, National
Natural Science Foundation of China, 2013.

[24]C. E. Izard, The Psychology of Emotions. New York: Plenum press, 1991.
[25]J. Jia, S. Zhang, F. B. Meng, Y. X. Wang, L. H. Cai, Emotional audio-visual speech synthesis

based on PAD, IEEE Transactions on Audio, Speech, and Language Processing. 19( 3), 2011,
pp.570-582.

[26]J. Jia, X. H. Wang, Z. Ren, L. H. Cai, Emotional cognitive computing on the Internet data,

Communications of The CCF . 10(2), 2014, pp.38-43.

[27]I. Kompatsiaris, D. Gatica-Perez, X. Xie, Special section on social media as sensors, IEEE

Transactions on Multimedia. 15(6), 2013, pp.1229-1230.

[28]C. M. Lee, S. S. Narayanan, Toward detecting emotions in spoken dialogs, IEEE Transactions on

Speech and Processing. 13(2), 2005, pp.293-303.

[29]D. D. Li, Y. C. Yang, W. H Dai. 2014. Cost-sensitive learning for emotion robust speaker

recognition, Scientific World Journal. 2014 (Article ID 628516), pp.1-9.

[30]X. M. Li, H. T. Zhou, S. Z. Song, T. Ran, X. L. Fu, The reliability and validity of the Chinese

version of abbreviated PAD emotion scales, Lecture Notes in Computer Science, 3784, 2005,
pp.513-518.

[31] H. Lin, W. G. Fan, Patrick Y. K. Chau, Determinants of users’ continuance of social networking

sites: A self-regulation perspective, Information & Management. 51, 2014, pp.595-603.

[32]Y. Liu, Q. F. Fu, X. L. Fu, The interaction between cognition and emotion, Chinese Sci Bull.

54(18), 2009, pp.2783-2796.

[33]Y. Lu, X. L. Fu, L. M. Tao, Emotion measurement based on PAD 3-D space, Communications of

The CCF 6(5), 2010, pp.9-13.

[34]K. F. MacDorman, S. Ough, C. G. Ho, Automatic emotion prediction of song excerpts: index

construction, algorithm design, and empirical comparison, Journal of New Music Research. 36(4),
2007, pp.283–301.

[35]A. Mehrabian, Framework for a comprehensive description and measurement of emotional states,

Genetic, Social, and General Psychology Monographs, 121, 1995, pp.339-361.

[36]A. Mehrabian, Pleasure-arousal-dominance: A general framework for describing and measuring

individual differences in temperament, Current Psychology: Developmental, Learning, Personality,
Social, 14, 1996, pp.261-292.

[37]L. Muda, M. Begam and I. Elamvazuthi, Voice recognition algorithms using mel frequency cepstral

coefficient (MFCC) and dynamic time warping (DTW) techniques, Journal of computing. 2(3),
2010, pp.138-143.

[38]T. L. Nwe, S.W. Foo, L.C. De Silva, Speech emotion recognition using hidden Markov models,

Speech Communication, 41(4), 2003, pp.603–623.

[39]C. E. Osgood, Dimensionality of the semantic space for communication via facial expressions,

Scandinavian Journal of Psychology. 17(1), 1996, pp.1-30.

[40]R. W. Picard, Affective Computing. Cambridge, USA: MIT Press, 1997.
[41]S. C. Sajjan, C. Vijaya, Comparison of DTW and HMM for isolated word recognition,

Proceedings of the International Conference on Pattern Recognition, Informatics and Medical
Engineering of the IEEE, 2012, pp.466-470.

[42]H. H. Schlosberg, Three dimensions of emotion, Psychological Review. 61(2), 1954, pp.81-88.
[43]B. Schuller, G. Rigoll, M. Lang, Hidden Markov Model-Based Speech Emotion Recognition,

Proceedings of ISCAS 2003, 2003, pp.1-4.

Page 18 of 18

Accepted Manuscript

[44]B. Schuller, G. Rigoll, M. Lang, Speech emotion recognition combining acoustic features and

linguistic information in a hybrid support vector machine-belief network architecture, Proceedings
of ICASSP 2004. 2004, pp. 577-580.

[45]H. Shi, N. Li, Z. X. Tian, W. L. Zhang, N. Zhu, Multivariate wear trend prediction based on cross

validation of optimization SVR, Lubrication Engineering. 38(11), 2013, pp. 22-25.

[46]P. H. Sun, L. M. Tao, Emotion measuring method in PAD emotional space, Proceedings of The

Fourth Harmonious Man-machine Environment Joint Academic Conference, 2008, pp.638-645.

[47]J. A. K. Suykens, J. Vandewalle, Least squares support vector machine classifiers, Neural Process

Letters. 9(3), 1999, pp.293–300.

[48]J. Tang, Y. Zhang, J. M. Sun, J. H. Rao, W. J. Yu, Y. R. Chen, A. C. M. Fong, Quantitative study

of individual emotional states in social networks, IEEE Transactions on Affective computing. 3(2),
2012, pp. 132-144.

[49]L. M. Tao, Y. Liu, X.L. Fu, L.H. Cai, A computational study on PAD based emotional state model.

Proceedings of the 26th Computer-Human Interaction Conference, 2008.

[50]D.Ververidis, C. Kotropoulos, Emotional speech recognition: resources, features, and methods,

Speech Communication. 48(9), 2006, pp.1162–1181.

[51]G. A. Wang, J. Jiao, A. S. Abrahams, W. Fan, Z. Zhang, ExpertRank: A topic-aware expert finding

algorithm for online knowledge communities, Decision Support Systems. 54(3), 2013,
pp.1442-1451.

[52]Y. H. Wang, X. H. Hu, W. H. Dai, J. Zhou, T. Z. Kuo, Vocal emotion of humanoid robots: A study

from brain mechanism, Scientific World Journal. 2014(Article ID 216341), pp.1-7.

[53]Y. W. Wang, W. H. Dai, Y. F. Yuan. Website Browsing Aid: A Navigation Graph-Based

Recommendation System, Decision Support Systems. 45(3), 2008, pp.387-400.

[54]W. M. Wundt, Outlines of Psychology. Oxford, England: Engelmann, 1897.
[55]Q. Zhang, Y. Q. Yang, Research on the kernel function of support vector machine, Electric Power

Science and Engineering. 28( 5), 2012, pp.42-46.．

[56]S. Q. Zhang, Z. J. Zhao, B. C. Lei, G. Y. Yang, Speech emotion recognition by combining voice

quality and prosody features, Journal of Circuits and Systems. 14(4), 2009, pp.120-123.

[57]Y. Zhang, Exploration of the new interpersonal communication pattern by Wechat in the mobile

Internet era, News Research, (2), 2014, pp.19-21.

[58]Q. Zhao, A survey of group spread effects on vocal social networks: the example of Wechat,

Science & Technology for China’s Mass Media, (6b), 2013, pp.155-156.

[59]H. Zhou, Emotional Speech Conversion and Recognition Based on the Three-dimensional PAD

Model, Lanzhou, China: Northwest Normal University, 2009.

[60]M. Zhou, L. Lei, J. Wang, A. G. A. Wang, W. Fan, Social media adoption and corporate disclosure,

Journal of Information Systems. forthcoming, 2014.