Least Squares Support Kernel Machines for

identification of NARMA model

(LS SKM )NARMA
M. TARHOUNI, S. ZIDI , K. LAABIDI, M. LAHMARI-KSOURI
Laboratory of Research Analysis, Conception and Control of Systems (ACCS, ENIT)
BP 37, LE BELVEDERE 1002, Tunis, Tunisia

LAGIS (USTL, Lille)

Villeneuve dAscq 59650 Lille-France
mouniratarhouni@yahoo.fr, salah zidi@yahoo.fr {kaouther.laabidi,moufida.ksouri }@enit.rnu.tn

Abstract This paper deals with the identification of nonlinear systems using the
Nonlinear Auto Regressive Moving Average (NARMA) model. This type of model
can characterize several classes of nonlinear systems. In this paper, an approach
to model complex systems using a procedure based on Multiple Kernel Learning
is suggested. A novel approach based on Least Squares Support Vector Machines
(LS-SVM) with combined kernel was proposed for NARMA system identification.
The basic idea is to consider the terms of auto-correlation and cross-correlation
of the non linearity of input output discrete time processes, and for every term a
kernel function is used. This proposed approach is called Least Squares Support
Kernel Machines for NARMA model (LS SKM)N ARM A . The choice of the kernel function and the corresponding parameters is an important task because of its
dependency on nonlinear system degree. Hence, the Particle Swarm Optimization
(PSO) is used to find the optimal kernel parameters. Two examples are presented
for qualitative comparison with the classical LS-SVM approach based on a single
kernel function. The results reveal the accuracy of the obtained model based on our
(LS SKM)N ARM A proposed approach.
Keywords : NARMA model, Least Squares Support Vector Machines, Least Squares
Support Kernel Machines for NARMA model (LSSKMN ARM A ), non linear system
identification, Particle Swarm Optimization (PSO), MIMO system.

Introduction

This paper deals with the identification of the NARMA type models for the monovariable and multivariable systems. NARMA model requires a reduced number of
parameters since it uses a limited number of terms in the output expression [1].
Several approaches have been proposed to determine the NARMA type models. In
[2], the binary genetic algorithm was used for the identification of NARMA model.

In [3] and [4], the authors used affine geometry and hyper surface distance minimization for the estimation of the NARMA model parameters. This approach is
based on the combination of the Optimal Parameter Search (OPS) algorithm and
the method of Total Least Squares (TLS). The approach makes it possible to estimate the parameters of the NARMA model even in the presence of a significant
noise. However, it is computationally intensive and the analytical solution cannot
be computed for systems of order higher than three [3, 4].
Recently, nonlinear techniques using Support Vector Machines (SVMs) have received
a great attention to deal with this problem [5], [6], [7]. These approaches obtain
system models based on intelligent behavior and learn automatically from previous
experience. The SVM was first proposed by Vapnik [8] in order to obtain maximum
margin separating hyperplanes in classification problems but this technique has become a general learning theory and is applied in a large field of applications [9], [10],
[11], [12], [13], [14], . A comprehensive description of this method for classification and regression problems can be found in [15] and [16], respectively. The basic
idea is to map linear inseparable input data into a high dimensional linear separable
feature space via a nonlinear mapping technique (kernel function) and to carry out
linear classification or regression in feature space.
As an interesting variant of the standard SVM, LS-SVM has been proposed by
Suykens and Vandewalle [17] as an attractive tool for identification of complex non
linear systems. This method is a widely known simplification of the SVM. It allows to obtain the model by solving a system of linear equations instead of a hard
quadratic programming problem involved by the standard SVM [18].
The Nonlinear Auto Regressive Moving Average (NARMA) model requires a low
number of parameters since it involves a limited number among the old measurements and the cross couples (input-input, input-output and output-output). For
that, we suggest a new approach capable to improve model flexibility by emphasizing these input-output cross terms. A new identification method called Least
Squares Support Kernel Machines for NARMA model (LS SKM)N ARM A has
been investigated for this purpose. The basic idea is to use a function kernel to
estimate the influence of the input, another kernel to estimate the influence of the
previous output and a third one to estimate the cross term input-output.
In previous studies, the rbf kernel and the linear kernel were the most two useful kernels. For that, we make the concentration on these two kernels and therefore several methods have been investigated such as: LS SKMN ARM A (rbf-lin),
LS SKMN ARM A (rbf-rbf) and LS SKMN ARM A (3rbf). The NARMA LS-SKM
(LS SKMN ARM A ) based method formulation requires the adjustment of several
parameters compared to the LS-SVM formulation. Hence, It is important to explore
a model selection methods that allow kernels to be chosen in an automatic way based
on datasets. In order to reach this objective, a Particle Swarm Optimization (PSO)
is adopted to select kernel parameters. Generally, PSO is characterized as a simple
concept, easy to implement, and computationally efficient [19], [20]. Unlike the other

heuristic techniques, PSO has a flexible and well-balanced mechanism to enhance

local exploration abilities [21].
This hybrid approach succeeds to improve the identification performance such as
the accuracy of the elaborated model and the simplicity of implementation.
The effectiveness of the proposed method is demonstrated by numerical examples.
The outline of this paper is as follows. In Section 2, the related work are given.
In Section 3, the Non linear ARMA model is presented. In section 4, we present
a new method for Non linear ARMA System Identification . The section 5 of this
paper deals with an extension of the LS SKMN ARM A method for multi-variable
systems (MIMO). In Section 6, the Particle Swarm Optimization is reviewed and a
PSO-based Hyper-parameters selection for LSSKMN ARM A algorithm is described.
Some illustrative simulation results are shown in section 7 in order to demonstrate
the applicability of our proposed identification strategy.

Non linear ARMA System Identification

We consider the nonlinear plants that can be described by the following relation:
y(k) = g(X(k)) + e(k)

(1)

where g is an unknown nonlinear function and X(k) = [y(k 1), y(k n), u(k
1), u(k m). The integers m and n represent the input u(k) and the output y(k)
regression orders, respectively.
The dynamic structure of an operative NARMA model in a deterministic environment is given by the following discrete mathematical equation:
m
1
1
1
bij u(k i)u(k j) + + m
b u(k i) + m
y(k) = y + m
i=1
i=1
j=1 P
Pn1 Pn
Pm1 Pm1 i=1 i
n1
v=p l=v bil u(k i) u(k l) + i=1 ai y(k i) + i=1 j=1 aij
P 1
P 1 P n1
P 1
ail y(k i) y(k l) + m
nv=p
y(k i)y(k j) + + ni=1
i=1 (2)
l=v
Pm1 Pn1
Pm1
P n1
j) + + i=1 v=1 l=1 cil u(k i) u(k v)
j=1 cij y(k i)y(k
P n1
P 1 Pn1
y(k l) + + m
j=1
i=1
l=1 cijl u(k i)y(k j) y(k l)

where y is the average value. The NARMA model is characterized by a huge number
of terms intervening in the expression of the model output. Thus, it is necessary to
use a procedure allowing the selection of a model among the total possible models
representing the behavior of real system as well as possible. The LS-SVM based
approach is improved in order to overcome these drawbacks.

Least Squares Support Kernel Machines for NARMA model

(LS SKMN ARM A)

Suykens and Vandewalle [17] have presented the LS-SVM approach, in which the
following function is used to approximate the output system as follows:
y =< w, (X(k)) > +b

(3)

where X Rl are the input data, y R are the output data and (.) : Rl 7 Rp
is the nonlinear function that maps the input space into a higher dimension feature
space.
According to the NARMA model given in equation (2), we note that the output
system is expressed by cross couples (input-input, input-output, and output-output).
Therefore, we propose that the output will be given by the following formulation:
y(Xi ) = a1 < wu , u (Xu,i ) > +a2 < wy , y (Xy,i )+a3 < wuy , uy (Xuy,i ) > +b,(4)
where xi = (u(i 1), , u(i m), y(i 1), , y(i n)) is the regressor in different iteration and y(i) is the output system that correspond on xi . Xu,i =
(u(i 1), , u(i m)), Xy,i = (y(i 1), , y(i n)) and Xuy,i = xi presents
the global regressor vector.

Xu

u(k-1)
u(k-n1)
xi =
y(k-1)

Xuy

y(k-m1 )

Xy

Fig. 1: Regression technique.

The fondamental idea consists on assuming that the NARMA model is a non
linear model which has mainly three types of crossing input-input, input-output and
output-output as well as the delayed inputs and outputs. As demonstrated in the
equation 4, the three key bloc are:
a1 < wu , u (Xu,i ) > represents the terms of the auto-correlation input-input,
a2 < wy , y (Xy,i ) represents the terms of the auto-correlation output-output,
a3 < wuy , uy (Xuy,i ) > represents the terms of the cross-correlation inputoutput.

This idea is explained more clearly by the figure 1:

{Xi , yi }N
i=1 constitute the learning data. Thus, rather than considering only
a single kernel function, three different kernel functions are adopted to estimate,
respectively, the influence of Xu,i , Xy,i and Xuy,i on the model.
Therefore, we suggest a new approach called Least Squares Support Kernel Machines
for NARMA model (LS SKMN ARM A ). This approach is defined in the following
constrained optimization problem:

minwu ,wy ,wuy ,b,e JP (wu , wy , wuy , e) = a21 kwu k22 + a22 kwy k22 + a23 kwuy k22 + C 12
subject to : yk = a1 < wu , (Xu (k)) > +a2 < wy , (Xy (k)) > +
a3 < wuy , (Xuy (k)) > +b + ek
k = 1, 2, , N

PN

2
i=1 ei

where C is a regularization factor and ei are slack variables.

To solve this optimization problem, we define the following Lagrange function:
L(wu , wy , wuy , b, e, ) = Jp (wu , wy , wuy , e) N
k=1 k (a1 < wu , u (Xuk ) > +
(6)
a2 < wy , y (Xyk ) > +a3 < wuy , uy (Xuyk ) > +b + ek yk )
P

where k are lagrange multipliers.

The optimality conditions are:

P
l
= 0 7 wu = N
k=1 k u (xuk )
wu
P
N
l
=
0

7
w
=
y
k=1 k u (xyk )
wy
PN
l
= 0 7 wuy = k=1 k uy (xuyk )
wuy
P
L
= 0 7 N
i=1 i = 0
b
l
=
0

i = Cei , i = 1, , N
ei

(7)

By replacing the expression of wu , wy , wuy and ei in the optimization problem

constraints given by equation (5), we obtain:
yi = N
j=1 j (a1 < u (xui ), 1 (xuj ) > +a2 < y (xyi ), y (xyj ) > +
a3 < uy (xuyi ), uy (xuyj ) >) + b + C1 i
P

(8)

The equations above can be rewritten in an equivalent matrix form:

"
#"
#
"
#
T

b
0
0
1
=
(9)

y
1 a1 u + a2 y + a3 uy + C 1 I

where y = [y1 , , yN ]T , 1 = [1, , 1]T , = [1 , , N ] and k (k u, y, uy)

is a square matrix in which the element located in ith column and j th row is
ki,j = k (xki )T k (xkj ) = Kk (xki , xkj ) and Kk are kernel functions, which can be
any symmetric function satisfying Mercers condition.
and b are solved directly from equation (9). Thus, the LS SKMN ARM A output
model y(x) is obtained.
y(x) =

N
X
i=1

i (a1 Ku (xu , xui ) + a2 Ky (xy , xyi ) + a3 Kuy (xuy , xuyi )) + b

(10)

(5)

where i , b R are the solution of equation (9)

In our proposed LS SKMN ARM A method, a new form of output is proposed which
is written as the sum of three terms. This form presents the advantage of having
more adjustment of parameters which ameliorate the precision of the elaborated
model.
Therefore, several kernel configuration were suggested. First, we have considered
that a3 = 0 and in this case we have proposed LS SKMN ARM A (rbf lin) and LS
SKMN ARM A (rbf rbf ) based methods. For example, the LS SKMN ARM A (rbf
lin) based method consists on using a gaussian kernel and a linear kernel in order to estimate the influence of auto-correlation terms: input-input and outputoutput, respectively. In the case of a3 6= 0, the LS SKMN ARM A (3rbf ) based
method is proposed, it consists on making use of three gaussian kernels in order to
estimate the influence of auto-correlation terms (input-input, output-output) and
cross-correlation (input-output) of the non linearity of input output discrete time
processes.
The accuracy of the LS SKMN ARM A model is largely dependent on a good
setting of meta-parameters C, on the kernel parameters and on the three weights
a1 , a2 , a3 . This task of regression usually requires training the model several times
with different hyper-parameter values and then the problem of optimal parameter
selection is further complicated. In the next part, we propose an approach which
allows optimizing automatically the kernel parameters and to adapt the regularization kernel parameter C by evolutionary means. In order to achieve this purpose, we
suggest a hybrid method that combines a constrained Particle Swarm Optimization
(PSO) algorithm and the proposed LS SKMN ARM A based technique.

4
4.1

Adjustment parameters based on Particle Swarm Optimization

Particle Swarm Optimization

Particle Swarm Optimization (PSO), first introduced by Kennedy and Eberhart

[28], is one of the new heuristic algorithms. It was first intended for simulating
social behavior, as a stylized representation of the movement of organisms in a bird
flock or fish school. Moreover, it has been demonstrated that this method is robust
in solving continuous and discret nonlinear optimization problems [29]. The PSO
technique can generate a high-quality solution within shorter calculation time and
stable convergence characteristic compared to other stochastic methods [29, 30]. It
provides a population-based search procedure in which individuals, called particles,
fly in a q dimensional search space, where each dimension corresponds to a parameter
in a function being optimized.
A basic PSO algorithm is as follows:

Algorithm 1 PSO
1: Initialize
2: for i = 1 to nSwarm do
3:
Initial positions and velocities of particles are generated by using random
values
4: end for
5: for k = 1 to iterations do
6:
for i = 1 to nSwarm do
7:
Calculate fitness value
8:
if the fitness value is better than the best fitness value (pbest) in history
then
9:
Set current value as the new pbest
10:
end if
11:
end for
12:
Choose the particle with the best fitness value of all the particles as the gbest
13:
for i = 0 to nSwarm do
14:
Calculate particle velocity according to the equation given in 11:
vk+1 = avk + b1 r1 (pBest zk ) + b2 r2 (gBest zk ).
(11)
15:
Update particle position according to the following equation:
zk+1 = czk + dvk+1 .
(12)
16:
end for
17: end for
The first step consists of the initialization which is composed of next sub-steps:
Set the size of swarm population (nSwarm)
Set iteration number (iterations)
Set constants a, b1 and b2
After that, fitness evaluation is conducted by supplying the candidate solution to
the objective function. Individual and global best fitness and positions are updated
by comparing the newly evaluated fitness against the previous individual and global
best fitness, and replacing the best fitness and positions as necessary.

swarm
best position
Pbest

best position
of its neighbors
Gbest

current
position

next position
current velocity
Fig. 2: Principle scheme of swarm displacement.
The process is repeated until the best bird reaches certain desired location. It is
worth noting here that according to the description, the process involves not only
intelligent behavior but also social interaction. This way, birds learn both from
their own experience (local search) and from the group experience (global search)
[29, 31, 32].
The index of the particle is represented by k. v is the velocity of particle. z is
the position of particle. r1 and r2 are random values regenerated for each velocity
update (0 < r1 < 1, 0 < r2 < 1). The value pBest is the individual best candidate
solution and gBest is the swarms global best candidate solution. The parameter a
is the inertia component. b1 , b2 are the attraction coefficients.

4.2

Adjustment parameters

The accuracy of the LS SKMN ARM A model is largely dependent on a good setting
of meta-parameters C, on the kernel parameters and on the three weights a1 , a2 , a3 .
This task of regression usually requires training the model several times with different hyper-parameter values and then the problem of optimal parameter selection is
further complicated. Our main purpose is to select these parameters automatically,
then we propose an hybrid method that combines a Particle Swarm Optimization
(PSO) algorithm and the proposed training algorithm.
The diagram of the PSO-based hyper-parameter selection algorithm for the LS
SKMN ARM A is shown in Figure 3.

Generate the
initial swarm

Move each particle from the

current position to the next one
No
Parameters
satisfy the constraints?
Yes
Train LS-SKM

NARMA

based

algorithm

Calculate fitness (100-Rtot 2)

Update the best search position

in its track for each particle

Update the best search

position of the swarm

No

Maximum number
of iteration?
Yes
Obtain optimal model
(optimal kernel parameters)

Fig. 3: Diagram of the PSO-based hyper-parameters selection algorithm for LS

SKMN ARM A .
The principal idea is to represent all the parameters as the particles position
z(a1 , a2 , a3 , p1 , p2 , p3 , C). With respect to the parameter constraints (ai > 0, pi >
0, C > 5). pi represents the parameter of the kernel function ki and i 1, 2, 3.
The coefficient :
2
Rtot
= (1

k
y yk
) 100%
kyk

(13)

This coefficient is used to measure the similarity between the real output system
y = [y1 yN ]T and the estimated output y = [
y1 yN ]T .
As it is represented in the flowchart 3: First, we take the parameters z(a1 , a2 , a3 , p1 , p2 , p3 , C)
into the swarms in the PSO algorithm then we start the PSO algorithm.
Once the LS SKMN ARM A parameters are arranged in the form of a parameter
vector which is called a particle in the terminology of PSO. The proposed PSO algorithm applies the velocity updating and position updating formulas (11) to the
population composed of many particles such that better particles are generated. At

each iteration, the constrained parameters are verified. If one constraint is violated,
we return to the first step. If not, we train the LS SKMN ARM A algorithm and we
2
calculate the objective function (100 Rtot
). At last, once the maximum number
of iterations is reached, we obtain the LS SKMN ARM A model ( and b) at the
best optimal parameters and get the output data corresponding to the test input
data. For each of treated methods (Gaussian LS-SVM, LS SKMN ARM A (rbf-lin),
LS SKMN ARM A (rbf-rbf)) and LS SKMN ARM A (3rbf) and for the three simulation examples, we have applied the PSO algorithm with the same parameter settings
( nSwarm = 49, a = 1 and b1 = b2 = 2). In fact, some rules were proposed in the
literature to select these parameters [28, 33]. In fact, the population size (nSwarm)
is crucial for PSO algorithm, a small population does not create enough interaction
for the emergent behavior to PSO to occur. However, the population size is not a
specific problem, a value ranging from 20 to 50 can guarantee the exploring capability of PSO. The inertia coefficient (a) controls the influence of previous velocity
on the current one. Generally speaking, large a facilitates global exploration, while
low a facilitates local exploration. According to the suggestion of Parsopoulos and
Varhatis [33], an initial value around 1.2 and gradual decline towards 0 can be taken
as a good choice of a. And finally, the attraction coefficient (b1 and b2 ) can improve
the probability of finding global optimum and the speed of convergence. The default
value is b1 = b2 = 2.

5
5.1

Simulations and Results

Example 1: SISO system

The test system to be identified is:

u(i)+x2 (i)
x1 (i + 1) = 0.1x1 (i) + 2 1+(u(i)+x
2
2 (i))

x2 (i + 1) = 0.1x2 (i) + u(i)(1 +

y(i) = 0.4x1 (i) + 1.5x1 (i).

u(i)2
)
1+x1 (i)2 +x2 (i)2

(14)

with an input excitation signal u(i) chosen as a random value in [1, 1]. The model
input vector is formed using Xi = [u(i 1) u(i 2), , u(i m), y(i 1) y(i
2), , y(in)]T and as output yi = y(i). This vector will be divided into two groups
Xui = [u(i 1) u(i 2), , u(i m)]T and Xyi = [y(i 1) y(i 2), , y(i n)]T .
During this training phase, 200 samples are applied.
As we are mentioned, for the LS SKMN ARM A (rbf lin), the gaussian kernel
function is used to estimate Xui and the linear one is used to estimate Xyi . For the
LS SKMN ARM A (rbf rbf ), we have used two gaussian kernels where the first one
is used to estimate Xui and the second is used to estimate Xyi . Therefore, in the case
of LS SKMN ARM A (3rbf ) proposed method three gaussian kernels were used to
estimate Xui , Xyi and Xuyi . Xuyi represents the global regressor vector (Xuyi = Xi ).

Table 1 lists the optimal parameters selected by PSO.

Table 1: Optimal parameters, example

Kernels
a1
a2
a3
p1
rbf
74.36

4.91
rbf-lin 12.05 0.53

1.46
LS SKMN ARM A rbf-rbf
9.2
2.3

1.2
3rbf
1.79 0.44 7.85 8.06
Method
LS-SVM

1.
p2

1.39
12.7
7.85

p3

4.54

C
953.63
995.42
1003.5
994.32

2
2
The total correlation coefficient Rtot
, the multiple correlation coefficient Rmult
and, the mean squares error (MSE) of the training data are calculated in order to
2
decide if the obtained model is accepted or not. The model is acceptable if Rtot
,
2
2
Rmult is close to one and the MSE is close to zeros [34]. The Rmult is defined as:
2
Rmult

PN

(
y (k) y)2
100%.
)2
k=1 (y(k) y

k=1
= PN

(15)

The mean squares error (MSE) is given by:

MSE =

PN

k=1 (y(k)

y(k))2

(16)

2
2
Table 2: Values of, Rtot
, Rmult
,
example 1.
Method
Kernels
LS-SVM
rbf
rbf-lin
(LS SKM)N ARM A rbf-rbf
3rbf

MSE and computing time for different methods,

2
Rtot
90.1687
94.4101
94.5879
94.7284

2
Rmult
99.3357
100.9350
101.0441
100.9552

MSE
0.0092
0.0030
0.0028
0.0027

Computing T ime
2.3556
2.3088
2.5428
2.7612

It is noticed, according to Table 2, that the proposed methods LSSKMN ARM A (rbf
lin), LSSKMN ARM A (rbf rbf ) and LSSKMN ARM A (3rbf ) have the best performance compared to the traditional LS-SVM (rbf) based on a single kernel. Indeed,
2
the values of Rtot
converges to 94.7% and the RMSE values is close to 0 and it is
equal to 0.0027 adopting the LS SKMN ARM A (3rbf ) based method. Nevertheless,
2
the Rtot
value is equal to 90.1% based on the LS-SVM(rbf) that uses to train a single
kernel function. Therefore, our methods succeeds to ameliorate the accuracy of a
model which has a significant effect on the performance of the closed loop system.

2.5

y(k)
y(k)

2
1.5
1
0.5
0
0.5
1
1.5
2
2.5

20

40

60

80

100

120

140

160

180

200

Iterations

Fig. 4: Evolution of the output system y and the output model y based on LS
SKMN ARM A (3rbf ), example 1.

0.2

y(k) y(k)

0.15
0.1
0.05
0
0.05
0.1
0.15
0.2
0.25
0

20

40

60

80

100

120

140

160

180

200

Iterations

Fig. 5: Evolution of the modeling error based on LS SKMN ARM A (3rbf ), example
1.
The obtained identification results based on LS SKMN ARM A (3rbf ) are given
by the figure 4. We notice that the real system output and the model output are
overcome and the identification error is acceptable as shown in figure 5. posed
method, we have succeeded to obtain a global and precise model.

5.2

Example 2: MIMO system

To demonstrate the effectiveness of the suggested methods and justify their interest, we consider an example of a MIMO system given by the following recurrent
expression:
u1 (k)+y2 (k)
y1 (k + 1) = 0.1y1(k) + 2 1+(u
2,
2 (k)+y2 (k))

y2(k + 1) = 0.1y2 (k) + u1 (k)(1 +

(17)

u2 (k)2
).
1+y1 (k)2 +y2 (k)2

u1 and u2 are assumed random belonging to the interval [1 1].

In this case, the regressor denoted Xuy,i is formed by [u1 (i 1) u1 (i 2) u2 (i
1) u2 (i 2) y1 (i 1) y2 (i 1) y1 (i 2) y2 (i 2)]. This vector is split into two blocks
Xu,i = [u1 (i1)u1 (i2)u2 (i1)u2 (i2)] and Xy,i = y1 (i1)y2 (i1)y1 (i2)y2 (i2)].
Xu,i combines the delayed control signals and Xy,i combines the delayed outputs of
the system. Thereafter, for each group a kernel will be used to estimate the influence
of these terms on the model and a third kernel will be used in order to estimate the
influence of the global regressor on the model.
First, the PSO is applied in order to reach the optimal parameters. Table 3 lists the
parameters of all taken methods.
The table 4 given below summarizes the overall results obtained adopting the methods (LS SKM)N ARM A (rbf lin), (LS SKM)N ARM A (rbf rbf ) and (LS
SKM)N ARM A (3rbf ) that we have proposed in this work. These results will be compared with those obtained using the LS-SVM method.

Table 3: Optimal parameters, example 2.

Kernels
a1
a2
a3
p1
rbf
5.60

4.20
rbf-lin
4.57 5.03

1.33
(LS SKM)N ARM A rbf-rbf 2.60 2.71

2.81
3rbf
6.5
8.2
10.3
3.2
LS-SVM
rbf
35.6

2.7
rbf-lin 10.41 0.60

3.18
(LS SKM)N ARM A rbf-rbf 9.74 0.64

3.14
3rbf
10.03 7.84 10.43 4.74
Method
LS-SVM

y1

y2

p2

4.85
3.83
13.3

2.42
6.83
8.81

p3

9.4

3.15

C
1001.9
998.64
995.13
1002.7
1001.8
997.82
995.74
999.52

2
2
Table 4: Values of, Rtot
, Rmult
, MSE and computing time for different methods,
example 2.
2
2
Method
Kernels
Rtot
Rmult
MSE Computing T ime
LS-SVM
rbf
80.4043 101.0541 0.0566
2.3556
rbf-lin 86.7834 99.5278 0.0258
2.3244
y1 (LS SKM)N ARM A rbf-rbf 83.3436 100.6182 0.0409
2.6208
3rbf
85.2543 99.5384 0.0321
2.6676
LS-SVM
rbf
84.5615 97.4360 0.0110
2.1528
rbf-lin 90.2741 99.6016 0.0044
2.3868
y2 (LS SKM)N ARM A rbf-rbf 89.0978 100.0633 0.0055
2.6052
3rbf
87.9134 97.6695 0.0068
2.7768

We notice that all proposed methods improve the identification task compared
with the results obtaining based on the classic LS-SVM based method that uses to
train a unique kernel. In fact, in the case of identifying the first output y1, the
2
coefficient Rtot
converge to 86.78% when we adopt the (LS SKM)N ARM A (rbf
lin), to 83.34% when we adopt (LS SKM)N ARM A (rbf rbf ), and to 85.25%
when we adopt (LS SKM)N ARM A (3rbf ). But, this criterion is equal to 80.4%
based on LS-SVM. In the case of the second output y2 , this coefficient converge to
90.27%, to 89.09% and to 87.91% using the (LS SKM)N ARM A (rbf lin), (LS
SKM)N ARM A (rbf rbf ) and (LS SKM)N ARM A (3rbf ), respectively. However, it
is equal to 84.56% adopting the LS-SVM(rbf) based technique.
4

y1 (k)
y1 (k)

3
2
1
0
1
2
3
4

50

100

150

200

250

300

350

400

Iterations

Fig. 6: Evolution of y1 and y1 based on LS SKMN ARM A (rbf lin), example 2.

y2 (k)
y2 (k)

1.5
1
0.5
0
0.5
1
1.5
2

50

100

150

200

250

300

350

400

Iterations

Fig. 7: Evolution of y2 and y2 based on LS SKMN ARM A (rbf lin), example 2.

Figures 6 and 7 present the evolutions of the two output model/real of the system
by adopting the technique (LS SKM)N ARM A (rbf lin). These curves show that
the obtained models keep track the actual output(y1 , y2 ) of the system properly
with low identification errors.
We conclude that the simulation results show that the effectiveness of the proposed
method is strongly related to the choice of multi-kernel configuration. A study
of the types of kernel combination and these parameters is therefore to consider.
Nevertheless, adopting the multi-kernel approach, it exist certainly a multi-kernel
configuration which improves the accuracy of model significantly.

Conclusions

The work presented in this paper dealt with the identification of monovariable and
multivariable nonlinear systems by exploiting NARMA type models. A new approach called Least Squares Support Kernel Machines for NARMA model (LS
SKMN ARM A ) is suggested. The basic idea consists on considering the two terms of
auto- correlation and the terms of cross-correlation and for every block a kernel function is used. Several kernel configurations were considered and consequently several
methods were suggested: (LS SKM)N ARM A (rbf lin), (LS SKM)N ARM A (rbf
rbf ) and (LS SKM)N ARM A (3rbf ). Due to the importance of parameter optimization in our proposed(LS SKMN ARM A ) based method, a Particle Swarm Optimization (PSO) algorithm is adopted to obtain the optimal free parameters. This method
does not need to consider the analytic property of the generalization performance
measure and can determine multiple hyper-parameter at the same time.
These blockwise training algorithms contribute to good results compared to performance obtained based on LS-SVM method based on unique kernel function.

References
[1] W. Liu, L. Zhengpei, L. Fu et W. Yaqi (2003). A systematic method to identify
nonlinear dynamics of BWR by using the reactor noise. In Progress in Nuclear
Energy, volume 43, pages 209-216.
[2] B. Tlili, F. Bouani et M. Ksouri (2006). Estimation des paramtres du modle
NARMA laide des rseaux de neurones et des algorithmes gntiques. In IActes de
la 6me confrence francophone de MOdlisation et SIMulation, Rabat, Maroc.
[3] L. Sheng, K. H. Ju et K. H. Chon (2001). A new Algorithm for linear and
Nonlinear ARMA Model parameter estimation using Affine Geometry. In IEEE
Trans. on Biomedical Engineering, volume 48, pages 1116-1124.
[4] L. Sheng et K. H. Chon (2003). Nonlinear Autoregressive and Nonlinear Autoregressive Moving Average Model parameter estimation by Minimising Hyper surface Distance. In IEEE Trans. on Signal processing, volume 51, pages 3020-3026.
[5] Lei, B., Wang, W. and Li, Z. (2008) On Line Predictive Control Based on
LS SVM, Proceedings of the 7th World Congress on Intelligent Control and
Automation.
[6] Liang, X., Yan, G., Li, S. , Long, Li, Z. X. and Long, W. (2009) Model Predictive
Control for MIMO System Based on LSSVM and Mutative Scale Chaos Optimization, Second International Conference on Intelligent Computation Technology and Automation.
[7] Tao, S., Chen, D. and Zhao, W. (2009) Fast pruning algorithm for multi-output
LS-SVM and its application in chemical pattern classification, Chemometrics
and Intelligent Laboratory Systems Vol. 96, Issue 1, 63-69.
[8] V. Vapnik (1995). The nature of statistical learning theory. Springer-Verlag, New
York.
[9] Lee, K. L. and Billings, S. A. (2002)Time series prediction using support vector machines, the orthogonal and the regularized orthogonal least-squares algorithms, International Journal of Systems Science, Volume 33, Issue 10, pages
811821.
[10] Zidi, S., Maouche, S. and Hammadi, S. (2006) Nouvelle approche pour la
regulation des reseaux de transport multimodal, 6eme Conference Francophone
de MOdelisation et SIMulation, MOSIM06, Rabat, Maroc.
[11] Tarhouni, M., Laabidi, K., Zidi, S. and Lahmari-Ksouri, M. (2010a)
Surveillance des systems complexes par les separateurs `a vaste marges, 8me
Conference Internationale de MOdelisation et SIMulation (MOSIM10), Hammamet, Tunisie.
[12] Hacib, T., Acikgoz, H., Le Bihan, Y., Mekideche M.R., Meyer, O. and Pichon,
L., (2010) Support vector machines for measuring dielectric properties of materials, COMPEL: The International Journal for Computation and Mathematics
in Electrical and Electronic Engineering, Vol. 29 Iss: 4, pp.1081 - 1089.

[13] Mi Hye Song, Jeon Lee, Sung Pil Cho, Kyoung Joung Lee, and Sun Kook
Yoo (2005). Support Vector Machine Based Arrhythmia Classification Using
Reduced Features, International Journal of Control, Automation, and Systems,
vol. 3, no. 4, pp.571-579.
[14] Zhi-kun Hu, Wei-hua Gui, Chun-hua Yang, Peng-cheng Deng, and Steven X.
Ding (2011). Fault Classification Method for Inverter Based on Hybrid Support Vector Machines and Wavelet Analysis, International Journal of Control,
Automation, and Systems, vol. 9, no. 4, pp.797-804.
[15] C. Burge (1998). A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery. In vol 2, pp 121-167.
[16] V. Vapnik, S. Golowich et A. Smola (1997). Support Vector Method for Function Approximation, Regression Estimation, and Signal Processing. In M. Jordan
in M. Mozer et T. Petsche (eds.), editeurs, Neural Information Processing Systems, volume 9, Cambridge, MA, MIT Press.
[17] Suykens, J.A.K. and Vandewalle, J. (1999) Least Squares Support Vector Machine Classifier, Neural Processing Letters,Vol. 9, issue 3, pp. 293300.
[18] J. Valyon and G. Horvath, A Robust LSSVM Regression, Proceeding of word
academy of science, engeneering and technology, Volume 7, pp. 14853, 2005.
[19] Z. Du, X. Wang, Hammerstein model identification using quantum deltapotential-well-based particle swarm optimization, International Journal of Modelling, Identification and Control, Vol. 12, issue 4, pp. 421 - 427, 2011.
[20] Tokuda M., Yamamoto T. (2011) A data-driven PID control system using
particle swarm optimization, International Journal of Modelling, Identification
and Control, Vol. 13, issue 1/2, pp. 88 96.
[21] Abido M. A. (2002) Optimal Design of PowerSystem Stabilizers Using Particle
Swarm Optimization, IEEE transactions on energy conversion, Vol. 17, issue 3,
pp. 406413.
[22] S. Amari and S. Wu, Improving Support Vector Machine Classifiers by Modifying Kernel Functions, Neural Networks, Vol. 12, issue 6, pp. 783-789, 1999.
[23] B. Zhejing, P. Daoying and S. Youxian, Multiple Kernel Learning, Conic Duality, and the SMO Algorithm Machine with Multikernel, Proceedings of the 21
st
International Conference on Machine Learning, Banff, Canada, 2004.
[24] M. Tarhouni, K. Laabidi, S. Zidi and M. Lahmari-Ksouri, System Identification
based on Multi-kernel Least Squares Support Vector Machines (Multi-kernel LSSVM), International Conference on Neural Computation (ICNC 2010), Valancia,
Spain, 2010b.

[25] A. Tatarchuk, E. Urlov, V. Mottl, and D. Windridge, A Support Kernel Machine for Supervised Selective Combining of Diverse Pattern-Recognition Modalities, MCS 2010, LNCS 5997, pp. 165-174, Springer-Verlag Berlin Heidelberg,
2010.
[26] Diosan, L., Oltean, M., Rogozan, A., and Pecuchet, J.P. (2007) Improving
SVM Performance Using a Linear Combination of Kernels, B. Beliczynski et
al. (Eds.): ICANNGA 2007, Part II, LNCS 4432, pp. 218-227,. SpringerVerlag
Berlin Heidelberg.
[27] G. R.G. Lanckriet, N. Cristianini, P. Bartlett, L. El Ghaoui, and M. I. Jordan, Learning the Kernel Matrix with Semi-definite Programming, Journal of
Machine Learning Research, Vol. 5, pp. 27-72, 2004.
[28] Kennedy, J. and Eberhart, R. (1995) Particle swarm optimization, in Proc.
IEEE Int. Conf. Neural Networks, vol. IV, Perth, Australia, pp. 19421948.
[29] Fliss, I. and Tagina, M. (2010) Multiple faults fuzzy detection approach improved by particle swarm optimization, 8th International Conference of Modeling
and Simulation (MOSIM10), Hammamet, Tunisia.
[30] Gaing, Z.L, (2004) A Particle Swarm Optimization Approach for Optimum Design of PID Controller in AVR System, IEEE Transaction on energy conversion,
Vol. 19, issue 2, pp. 384391.
[31] Ramanathan, K., Periasamy, V.M., Pushpavanam, M. and Natarajan, U. (2009)
Particle Swarm Optimisation of hardness in nickel diamond electro composites,
International Scientific Journal, Computational Materials Science and Surface
Engineering, Vol. 1, Issue 4, pp. 232236.
[32] Rashtchi, V., Shayeghi, H., Mahdavi, M., Kimiyaghalam, A. and Rahimpour,
E. (2008) Using an Improved PSO Algorithm for Parameter Identification of
Transformer Detailed Model, International Journal of Electrical Power and Energy Systems Engineering,Vol 2, issue 11, pp. 666672.
[33] Parsonpoulos, K. E., Varhatis, M.N. (2002) Recent approaches to global optimization problems through particle swarm optimization, Natural Computing,Vol. 1, issue 2-3, pp. 235-306.
[34] Haber, R. and Unbehauen, H. (1990) Structure identification of Non linear
systemsA survey on Input/output Approaches, in Automatica, (Vol 26, issue
4), pp. 651677.