Signal Processing 84 (2004) 17091718

www.elsevier.com/locate/sigpro
State-space recursive least-squares: Part I
Mohammad Bilal Malik

College of Electrical and Mechanical Engineering, National University of Sciences and Technology, Rawalpindi, Pakistan
Received 12 June 2003; received in revised form 29 April 2004
Abstract
In this paper we present state-space recursive least-squares (SSRLS) lter. This algorithm is a new addition to the family
of RLS lters. We cover core topics like batch processing, recursive updates, initialization and steady state solution, etc.
SSRLS is very well-suited to estimate a wide class of deterministic signals corrupted by observation noise. This new lter
exhibits excellent tracking performance by overcoming some of the limitations of the standard RLS algorithm. With its
state-space formulation and sound mathematical basis, SSRLS is expected to become an important tool in estimation theory,
adaptive ltering and control systems.
? 2004 Elsevier B.V. All rights reserved.
Keywords: RLS; Adaptive ltering; State-space RLS; SSRLS; Tracking
1. Introduction
Recursive least-squares (RLS) and least mean
square (LMS) lters are principal tools in the realm
of adaptive ltering [10]. The philosophy of LMS is
somewhat dierent from RLS but their areas of appli-
cation are common. LMS and its variants have lower
convergence rates than RLS in general. Another lim-
itation of LMS is its higher steady-state mean square
error as compared to RLS. However, LMS oers
higher degree of robustness as it is optimal in the
sense of H

[8].
One of the major interests in literature has been
the tracking performances of LMS and RLS lters
[2,5,6,10]. It is believed [10] that typically LMS ex-
hibits better tracking properties due to its model in-
dependent nature. On the other hand the derivation

Postal address: 26 Race Course Rd, Street 16, Rawalpindi,

Pakistan. Tel.: +92-51-9278045.
E-mail address: mbmalik@ieee.org (M.B. Malik).
of standard RLS assumes multiple linear regression
model. If this model does not closely match the un-
derlying model of the environment, RLS is prone to
inferior tracking performance. In view of these con-
siderations, there is a need to devise a form of RLS
that gives a designer freedom to choose an appro-
priate model. The state-space recursive least-squares
(SSRLS) algorithm derived in this paper serves
this purpose. This concept was rst introduced in
[4,13,14]. An application of SSRLS to track linear
time-varying systems is given in [18]. Sayed and
Kailath [20] have given a state-space model for RLS.
An astute reader would, however, appreciate that their
model is just a representation of the standard RLS.
It is not a dierent algorithm. Based on [20], Haykin
et al. [9] have exploited one-to-one correspondence
between RLS and Kalman lter to devise extended
RLS (ERLS) algorithms. Their approach is problem
specic and dierent from the one presented here,
although the common aim is to improve the tracking
performance.
0165-1684/$ - see front matter ? 2004 Elsevier B.V. All rights reserved.
doi:10.1016/j.sigpro.2004.05.022
1710 M.B. Malik / Signal Processing 84 (2004) 17091718
One of the challenges faced by the family of RLS
lters is to track a time-varying signal whose model
is not exactly known. RLS, SSRLS and ERLS adap-
tive lters face certain limitations in this context. As
a natural extension of SSRLS, we have developed
SSRLS with adaptive memory [15,16]. The work
is then extended to devise computationally ecient
forms of the actual lter. The resultant algorithms
exhibit extraordinary tracking properties for dicult
problems like the one under discussion. We have suc-
cessfully employed SSRLS with adaptive memory in
harder areas like nonlinear stochastic control systems
[16]. RLS with adaptive memory [10] or the cur-
rently available ERLS lters do not reach this level
of versatility and sophistication.
Derivation of SSRLS has a formal mathematical
foundation. This enables us to carry out the stability
and convergence analysis of SSRLS, which follows in
Part II of this work [17].
In this paper we begin with the discrete state-space
model of the signal. The output of the system that
may have been corrupted by observation noise is as-
sumed to be available for measurements. Batch pro-
cessed least-squares state estimation forms the basis
of further development. The work is then extended to
recursive algorithms whilst employing the concept of
exponential forgetting [10]. In order to alleviate matrix
inversion, we derive the Riccati equation for SSRLS
using matrix inversion lemma. Analogous to Kalman
gain [3,7,11,22], we develop expression of observer
gain for SSRLS. The mathematical results and deriva-
tions of SSRLS have a close resemblance with Kalman
lter [3,7,11,22].
SSRLS can be initialized using the regularization
term [10]. An alternate method is to delay the recur-
sion and batch process the initially observed data. This
approach leads to a natural way of initializing SSRLS
and hence avoids problems like peaking [12,23]. It
also oers superior convergence behavior [17].
One of the major contributions in this work is the
development of time-invariant solution of SSRLS. The
resultant lter is numerically ecient as most of the
computations can be done oine. This includes so-
lution of a discrete algebraic Lyapunov equation. We
also discuss special cases of practical importance and
give transfer functions of SSRLS for ready reference.
The development of SSRLS algorithms ends with
a discussion on computational complexity of SSRLS
and steady-state SSRLS. Acomparison of SSRLS with
RLS and LMS lters has also been drawn in this sec-
tion. Finally an example of tracking a single tone con-
cludes the paper. A comparison of RLS, LMS and
SSRLS shows superior convergence and tracking per-
formance of SSRLS.
2. State-space model
Consider the following unforced discrete-time
system.
x[k + 1] = Ax[k],
+[k] = Cx[k] + t[k], (2.1)
where x R
n
are the process states and + R
m
is the
output vector. Observation noise is represented by
t[k]. We make no assumptions about the nature of
t[k] at this stage. The state-transition matrix A is as-
sumed to be invertible. The pair (A, C) is assumed to
be !-step observable [19]. Notice the absence of pro-
cess noise. Under these considerations +[k] can be
viewed as a deterministic signal corrupted by obser-
vation noise. The state-transition matrix A is neutrally
stable for an important class of signals as would be
signied later in the paper. This means that all of the
eigenvalues of A are strictly on the unit circle for such
cases.
3. Batch-processing least-squares approach
Consider a block of r consecutive measurements
with the last one being at time k. Our goal is to t this
batch of observations on the model (2.1) according to
the method of least-squares. From the second equation
of (2.1), we can write the observation vector, y
r
[k] as
follows:
y
r
[k] =

+[kr+1]
+[kr+2]
.
.
.
+[k 2]
+[k 1]
+[k]

Cx[kr+1]
Cx[kr+2]
.
.
.
Cx[k 2]
Cx[k 1]
Cx[k]

+v
r
[k]
M.B. Malik / Signal Processing 84 (2004) 17091718 1711
=

CA
r+1
x[k]
CA
r+2
x[k]
.
.
.
CA
2
x[k]
CA
1
x[k]
Cx[k]

+ v
r
[k]. (3.1)
The observation noise vector is given by
v
r
[k]=[t[kr+1] t[kr+2] t[k1] t[k]]
T
.
Eq. (3.1) may be written as
y
r
[k] = H
r
x[k] + v
r
[k], (3.2)
where H
r
is dened as
H
r
= [CA
r+1
CA
r+2
CA
1
C]
T
. (3.3)
If r! the matrix H
r
is full rank because of the ob-
servability assumption. With r!, (3.2) can be rec-
ognized as an overdetermined linear system of equa-
tions [10]. The solution of system (3.2) in terms of
least-squares is given as follows [10,11,21]:
x[k] = (H
T
r
H
r
)
1
H
T
r
y
r
[k]. (3.4)
Dimension of (H
T
r
H
r
)
1
H
T
r
is n mr. This matrix
can be calculated o-line. The solution (3.4) can be
regarded as the batch-processed least-squares state es-
timate. Optimality properties associated with it can be
found in [10,11,21]. Another variant of (3.4) could be
the weighted least-squares solution. In this case we
dene a weighting matrix W and the corresponding
solution is
x[k] = (H
T
r
WH
r
)
1
H
T
r
Wy
r
[k]. (3.5)
The formulae (3.4) and (3.5) could be computation-
ally intense if r is large. However, they lead not only
to the development of recursive algorithms but also
present a method of initializing the recursion.
4. Recursive algorithm
The essence of recursive algorithm is that given
the state estimate x[k 1], we want to compute the
estimate x[k] on arrival of the observation +[k]. The
prior (or a priori) state estimate, which is our estimate
prior to receiving the observation +[k], is given by
x[k] = A x[k 1]. (4.1)
Similarly a prior estimate of the measurement is
+[k] = C x[k] = CA x[k 1]. (4.2)
The prediction error can be dened as
c[k] = +[k] +[k]. (4.3)
We employ the same form for state-space recursive
least-squares algorithm which is usually used for
Kalman estimator [3,7,11,22].
x[k] = x[k] + K[k]c[k], (4.4)
where the observer gain K[k] is to be determined ac-
cording to the method of least-squares. The predic-
tion error can be referred to as innovations just like in
Kalman lter theory. Example in Section 8 illustrates
the nature of prediction error as white noise under
certain set of conditions.
Suppose that the observations start to appear at time
k =0. Since the current estimate is based on previous
values, we assume that some valid initial estimates
x[0] and K[0] are available. More will be discussed on
initialization in Section 4.6. Let r = k + 1 equations
in (3.1) be considered. This gives
H[k] = [CA
k
CA
k+1
CA
1
C]
T
. (4.5)
The observation vector in this case is
y[k] = [+[0] +[1] +[k 1] +[k]]
T
. (4.6)
We introduce a forgetting or weighting factor [10]
such that observations are given lesser weight as
they become old. In other words we try to forget
the past. Dening z 61 as the forgetting factor, the
corresponding weighting matrix is given as
W[k] =

z
k
I
m
0 0 0
0 z
k1
I
m
0 0
.
.
.
.
.
.
.
.
.
0 0 zI
m
0
0 0 0 I
m

, (4.7)
1712 M.B. Malik / Signal Processing 84 (2004) 17091718
where I
m
is identity matrix of dimension mm. Start-
ing as scalars for k = 0, the matrices H[k], W[k] and
y[k] grow in size with time. Analogous to classical
formulation of RLS [10], we dene the following sym-
bols
1[k] = H
T
[k]W[k]H[k],
.[k] = H
T
[k]W[k]y[k]. (4.8)
From (3.5) we get
1[k] x[k] = .[k],
x[k] = 1
1
[k].[k]. (4.9)
We term (4.9) as direct form of SSRLS. The recursive
nature of the algorithm will become evident in the next
sections.
4.1. Recursive update of 1[k]
We have from (4.5), (4.7) and (4.8)
1[k] = z
k
(A
T
)
k
C
T
CA
k
+z
k1
(A
T
)
k+1
C
T
CA
k+1
+ + zA
T
C
T
CA
1
+ C
T
C. (4.10)
Similarly
1[k 1] = z
k1
(A
T
)
k+1
C
T
CA
k+1
+ + zA
T
C
T
CA
1
+ C
T
C. (4.11)
Comparing (4.10) and (4.11), we get the dierence
Lyapunov equation for SSRLS as follows
1[k] = zA
T
1[k 1]A
1
+ C
T
C. (4.12)
4.2. Recursive update of 1
1
[k]
Making use of matrix inversion lemma [10], (4.12)
gives the Riccati equation for SSRLS after some al-
gebraic manipulations [16]:
1
1
[k] = z
1
A1
1
[k 1]A
T
z
2
A1
1
[k 1]A
T
C
T
[I + z
1
CA1
1
[k 1]A
T
C
T
]
1
CA1
1
[k 1]A
T
. (4.13)
If we are dealing with a single output system i.e. m=
1, then Eq. (4.13) does not require matrix inversion.
Otherwise, matrix of dimension m m is required to
be inverted.
4.3. Recursive computation of .[k]
We now develop recursive formula for .[k]. From
(4.5)(4.8) we have
.[k] = z
k
(A
T
)
k
C
T
+[0] + z
k1
(A
T
)
k+1
C
T
+[1]
+ + zA
T
C
T
+[k 1] + C
T
+[k]. (4.14)
Also
.[k 1] = z
k1
(A
T
)
k+1
C
T
+[0] +
+zA
T
C
T
+[k 2] + C
T
+[k 1],
(4.15)
which gives us
.[k] = zA
T
.[k 1] + C
T
+[k]. (4.16)
Eqs. (4.9), (4.13) and (4.16) constitute SSRLS in its
general form.
4.4. Observer gain
Take observer gain as
K[k] = z
1
A1
1
[k 1]A
T
C
T
[I + z
1
CA1
1
[k 1]A
T
C
T
]
1
. (4.17)
Eq. (4.13) can now be written as
1
1
[k] = z
1
A1
1
[k 1]A
T
z
1
K[k]CA1
1
[k 1]A
T
. (4.18)
Rearranging (4.17)
K[k] = [z
1
A1
1
[k 1]A
T
z
1
K[k]
CA1
1
[k 1]A
T
]C
T
= 1
1
[k]C
T
. (4.19)
It can be shown [16] that with this denition of ob-
server gain and expressions (4.9) and (4.16), we
arrive at (4.4).
4.5. State-space representation of SSRLS
Dene
w[k] = .[k 1]. (4.20)
M.B. Malik / Signal Processing 84 (2004) 17091718 1713
We have from (4.16)
w[k + 1] = zA
T
w[k] + C
T
+[k]. (4.21)
Similarly
x[k] = 1
1
[k].[k]
= z1
1
[k]A
T
w[k] + K[k]+[k]. (4.22)
Assuming w[k] as the states, +[k] as the inputs
and x[k] as the outputs, we can identify (4.21) and
(4.22) as the state-space equations. The quadru-
plet {zA
T
, C
T
, z1
1
[k]A
T
, K[k]} constitutes the
state-space matrices.
4.6. Initializing SSRLS
Proper initialization of any recursive algorithm is an
important phase. There may be certain practical di-
culties involved like the peaking phenomenon [12,23].
In case of SSRLS, we face an additional problem of
1[0], 1[1], . . . , 1[! 2] all being rank-decient. This
fact is a consequence of !-step observability of (2.1).
One way of tackling this situation is to employ the
regularization term [10]. We can either use 1[0] =oI
or more appropriately 1[0] =oI +C
T
C where o 0.
Since under the stated assumptions, 1[k] asymp-
totically settles down to a unique and well-dened
steady-state value [17], the eect of oI dies down with
time. The choice of o depends on SNR [10]. Some
initial (a priori) estimate of the process states x[0] is
required as well to start the recursion. It is customary
to take x[0] = 0 if no other estimates are available.
Batch processing approach of Section 3 oers
another method for proper initialization of SSRLS,
which is a more logical solution as compared to the
one based on regularization term. As the observations
start to appear, we wait for at least !-samples for the
matrix H
r
(refer to Eq. (3.3)) to become full rank.
Without the loss of generality these samples are given
negative time indices so that our recursion always
starts at time k = 0. Using the expressions (3.1) to
(3.5) with W given in (4.7), we can calculate x[ 1]
and hence x[0]. Similarly (4.8) can be used to ob-
tain an estimate of 1[0], which would be invertible.
The choice of the number of samples assimilated
by the initialization phase depends on factors like
observation noise or model uncertainty. The method
of delayed recursion produces unbiased estimates
[16,17]. On the other hand, regularization term only
oers asymptotic convergence in mean [10,16].
5. Steady-state solution of SSRLS
The SSRLS algorithm constitutes a time-varying
lter which is computationally intense. In this section,
we study the asymptotic behavior of SSRLS in order to
explore its steady state solution. Eq. (4.12) transforms
into the following discrete Lyapunov equation as time
approaches innity.
zA
T
1A
1
1 =C
T
C. (5.1)
The unique and well-behaved solution of the matrix
equation (5.1) [16,19]
lim
k
1[k] = 1 =

i=0
z
i
(A
T
)
i
C
T
C(A
1
)
i
(5.2)
exists if

z min | Eigenvalues(A)|. (5.3)

For a neutrally stable system the condition (5.3) trans-
lates to
z 1, (5.4)
which in turn prohibits the use of innite memory. The
signicance of this development is that 1 and hence
1
1
can be calculated oine.
5.1. Direct form
The direct form is given as
x[k] = 1
1
.[k] (5.5)
with .[k] given by (4.16).
5.2. Observer gain
Observer gain K is no longer a function of time and
is given as
K = 1
1
C
T
. (5.6)
5.3. State-space representation
The state-space representation for the time invariant
case is the same as mentioned in Section 4.5.
1714 M.B. Malik / Signal Processing 84 (2004) 17091718
5.4. Transfer function representation
For the time invariant case we can also represent
the estimator as a transfer function
H(z) = z1
1
A
T
(zI zA
T
)
1
C
T
+ K
= 1
1
[zA
T
(zI zA
T
)
1
+ I ]C
T
. (5.7)
This form is obviously not available for the time vary-
ing solution.
5.5. Initialization
Since the observer gain K is not a function of time,
we only need x[0] to start the recursion. The method of
delayed recursion discussed in Section 4.6 can be used
to obtain a reasonable estimate of x[0]. As steady-state
SSRLS is slow to converge, a good estimate of x[0]
may be important in certain applications. However for
the cases where the rate of convergence is not critical,
we may take x[0] = 0 to simplify the initialization
process. Example of Section 8 illustrates both of these
approaches.
5.6. Memory length
The length of lter memory could be approximated
by the following expression [10]
1
1 z
. (5.8)
The expression uses the fact 1 + z + z
2
+ =
1(1 z). In a time-varying environment, the
steady-state SSRLS will work well if the LTI model
(2.1) is a good temporal approximation of the signal
for at least an interval of duration 1(1 z) samples.
6. Special cases
The discussion in previous sections focuses on
general formulation of the SSRLS theory. We now
discuss some cases of practical signicance that cover
a wide range of problems. It was shown in Sec-
tion 5.4 that steady-state SSRLS can be represented
as transfer function. Computation of transfer func-
tions is algebraically tedious. We derive it for ready
reference as follows.
6.1. Constant
A constant is modeled by [1,16]
A = 1,
C = 1. (6.1)
The transfer function of steady-state SSRLS is
H(z) =
(1 z)z
(z z)
. (6.2)
6.2. Constant velocity model
The constant velocity model of a signal or trajectory
is given as [1,16]
A =

1 1
0 1

,
C = [1 0], (6.3)
where 1 is the sampling period. The transfer function
of steady-state SSRLS is
H(z) =

(1 z)z(z(1 + z) 2z)
(z z)
2
(1 z)
2
z(z 1)
(z z)
2

. (6.4)
6.3. Constant acceleration model
The constant acceleration model of a signal or tra-
jectory is given as [1,16]
A =

1 1 1
2
2
0 1 1
0 0 1

,
C = [1 0 0]. (6.5)
The transfer function of steady-state SSRLS is
H(z)=

z(z
2
(1z
3
)3zz(1z
2
)+3z
2
(1z))
(zz)
3
(1z)
2
z(z1)(3z(1+z)(1+5z))
21(zz)
3
(1 z)
3
z(z 1)
2
1
2
(z z)
3

.
(6.6)
M.B. Malik / Signal Processing 84 (2004) 17091718 1715
6.4. Sinusoidal signal
A sinusoidal signal of frequency c is represented as
A =

cos(c1) sin(c1)
sin(c1) cos(c1)

,
C = [1 0]. (6.7)
The transfer function is
H(z)=

(1 z)z(z(1 + z) 2z cos(c1))
z
2
2zz cos(c1) + z
2
(1z)z(z(1z)cos(c1)+z cos(2c1)1)
z
2
2zz cos(c1)+z
2

.
(6.8)
6.5. Remarks
The transfer functions (6.2), (6.4), (6.6) and (6.8)
are all minimum phase. Therefore, all of these lters
exhibit minimum delay and hence oer better tracking
performance. The relative degree is zero, which im-
plies existence of realizable inverses of the respective
lters. These features may be important for certain ap-
plications. As all the models are neutrally stable, the
corresponding transfer functions have their poles on
a circle of radius z. For the rst three cases the poles
are real.
7. Computational complexity
Computational complexity of an algorithm is usu-
ally of signicant importance particularly in real-time
applications. In this section, we discuss this aspect of
SSRLS. The SSRLS algorithm discussed in this pa-
per can take one of two possible forms. One form
makes use of the Riccati equation. The complete al-
gorithm is given by equations (4.1)(4.4), (4.13) and
(4.19). As evident from (4.13), an m m matrix is
required to be inverted. Another possible form com-
prises of equations (4.1)(4.4), (4.12) and (4.19). In
this case we have to invert an n n matrix. We name
the rst one as SSRLS (Form I) and the later one as
SSRLS (Form II). Although mathematically equiva-
lent, their computational eorts are dierent. The Ric-
cati equation (4.13) is of considerable higher com-
plexity as compared to the matrix dierence equation
(4.12). Therefore, alleviation of matrix inversion in
Form I is noteworthy only if m is small as compared
to n. Eqs. (4.1)(4.4) comprise steady-state SSRLS.
The observer gain can be calculated oine and (4.12),
(4.13) and (4.19) are not required. The result is a much
simpler algorithm. The complexities of Riccati equa-
tion (4.13), Matrix dierence equation (4.12), Matrix
inversion by GaussJordan elimination and variants
of SSRLS lters are listed in Table 1.
SSRLS can handle multiple observations (m1)
simultaneously, whereas RLS and LMS can only be
used to observe scalar random processes. In order to
compare SSRLS with RLS and LMS, we take m = 1
and give the corresponding numerical complexities
in Table 1. It is seen from Table 1 that SSRLS is
O(n
3
), steady-state SSRLS and RLS are O(n
2
) and
LMS or NLMS are O(n) in general. Another factor
that is not evident from Table 1 is that their may be
case-specic simplications in SSRLS. As a particu-
lar example, steady-state SSRLS for the constant ac-
celeration model (Section 6.3) requires a total of only
13 operations. On the other hand, general third order
steady-state SSRLS requires 27 operations (2n
2
+2n+
1 with n =3). The simplication for this specic case
is due to the form of matrices A and C as given in
Section 6.3.
For simplicity sake, we have only made a com-
parison between the basic algorithms. Faster versions
of the both RLS and LMS exist and are well-known
[10]. Certain schemes that reduce the complexity of
RLS may be used to do the same for SSRLS lters.
However, this issue is not within the scope of this
paper.
8. Example
We want to track a sinusoidal wave r(t)=a sin(ct +
m). The amplitude a and phase m are assumed to be
unknown, whereas the frequency c is known. The
discrete state-space model (2.1) with sampling time 1
for this continuous signal is
x[k + 1] =

cos(c1) sin(c1)
sin(c1) cos(c1)

x[k];
x[0] =

sin(m)
cos(m)

,
+[k] = [a 0]x[k] + t[k]. (8.1)
1716 M.B. Malik / Signal Processing 84 (2004) 17091718
Table 1
Numerical complexities of LMS, RLS and SSRLS algorithms
S. No. Filter type Multiplications Divisions
and additions/subtractions
1 SSRLS Riccati equation (4.13) 4n
3
+ 2n
2
m + 6nm
2
+
2
3
m
3
m
2
2 + O(m)
+O(n
2
) + O(m
2
) + O(nm)
2 Matrix dierence equation (4.12) 4n
3
+ O(n
2
) 0
3 GaussJordan Inversion
2
3
n
3
+ O(n
2
) n
2
2 + O(n)
of n n matrix
4 SSRLS (Form I) 4n
3
+ 4n
2
m + 6nm
2
+
2
3
m
3
m
2
2 + O(m)
+O(n
2
) + O(m
2
) + O(nm)
5 SSRLS (Form II) 4
2
3
n
3
+ 2n
2
m + O(n
2
) + O(nm) n
2
2 + O(n)
6 Steady state SSRLS 2n
2
+ 4nm + O(n) + O(m) 0
7 SSRLS (Form I) 4n
3
+ O(n
2
) 1
m = 1
8 SSRLS (Form II) 4
2
3
n
3
+ O(n
2
) n
2
2 + O(n)
m = 1
9 Steady state SSRLS m = 1 2n
2
+ O(n) 0
12 RLS 4n
2
+ O(n) 1
13 LMS 3n + 2 0
14 NLMS 5n + 2 1
The following set of parameters is used through out
this example
c = 0.1,
m = 3,
a = 1,
1 = 0.1. (8.2)
White noise t[k] with zero-mean and variance 0.001
corrupts the observations. We use estimation error
e[k] = +[k] C x[k] as the performance measure to
compare SSRLS lters with the standard RLS and
LMS algorithms. The forgetting factor for RLS and
SSRLS lters is taken to be z =0.98 and the step-size
parameter for LMS is taken as j=1z=0.02. SSRLS
and RLS are initialized using the regularization term
with o = 0.1. In one of the simulations, steady-state
SSRLS is initialized by the method of delayed recur-
sion (Section 4.6) with rst 10 samples reserved for
initialization. All the rest of the lters are taken to be
initially at rest i.e. the initial state-estimates and/or tap
weights are zero. The results of simulations are illus-
trated in Fig. 1. In order to have a clear view of the
steady-state behavior, we have split the plots in two
dierent amplitude scales. The left portion highlights
the transient behavior whereas the right one shows the
steady-state estimation error.
8.1. Remarks
SSRLS and RLS exhibit fast convergence, whereas
steady-state SSRLS (initially at rest) is the slow-
est to converge. We have shown in the companion
paper [17] that if properly initialized, SSRLS be-
comes an unbiased estimator within nite time. In our
simulation the eect of regularization term rapidly
disappears due to small o. This results in the SSRLS
estimates to appear to be unbiased soon after the ini-
tialization. In case of steady-state SSRLS, the lter
asymptotically converges in mean [17]. However,
initialization of steady-state SSRLS according to the
method of delayed recursion signicantly improves its
transient response. Looking at the simulation results,
it appears that properly initialized steady-state SSRLS
is almost as good as the full-edged SSRLS. This is
an important nding since SSRLS is computationally
much more complex (Section 7) than steady-state
SSRLS. After settling down, SSRLS and both the
variants of steady-state SSRLS result in almost the
same estimation error. Their performance exceeds
that of RLS and LMS by about an order of magnitude.
M.B. Malik / Signal Processing 84 (2004) 17091718 1717
Fig. 1. Tracking performance comparison of SSRLS, RLS and
LMS lters.
As expected tracking performance of LMS is better
than RLS. It has also been shown [17] that the predic-
tion error processes (4.3) for SSRLS and steady-state
SSRLS asymptotically approach white noise as z
1. We can approximately observe the same behavior
via simulations. It is generally believed that increas-
ing the order of the standard RLS and LMS partially
compensates for model inconsistency [10]. However
for this particular example, only a slight improvement
in tracking performance is observed by increasing the
order of RLS and LMS.
9. Conclusion
We have seen that the development of SSRLS
has overcome the limitations of the standard RLS.
This new lter exhibits excellent tracking perfor-
mance whilst giving fast convergence rate. It is the
state-space formulation of SSRLS that enables design
of adaptive lters to be carried out in a much intuitive
and direct manner as compared to the approach of ex-
tended RLS lters as suggested in [9]. We have also
seen that steady-state SSRLS is a numerically ecient
alternative of the actual lter. It requires less than half
the number of operations as compared to the standard
RLS. If steady-state SSRLS is properly initialized by
the method of delayed recursion, the problem of slow
rate of convergence is solved. Hence it is expected to
become the lter of choice for many applications.
Our work furnishes sucient details so as to make
the basic theory of SSRLS clear and ready to be used
by a designer. Some remaining relevant details can be
found in the companion paper of this work. The theory
is expected to attract a much wider area of research in
the future. It will give rise to new algorithms and so-
lutions for a diverse range of estimation and adaptive
ltering problems. SSRLS is also likely to become a
tool in stochastic control systems [16]. We leave for-
mal development of this aspect of SSRLS for future
considerations.
References
[1] Y. Bar-Shalom, Xiao-Rong Li, T. Kirubarajan, Estimation
with Applications to Tracking and Navigation, Wiley, New
York, 2001.
[2] A. Benveniste, Design of adaptive algorithms for the tracking
of time-varying systems, Internat. J. Adaptive Control Signal
Process. 1 (1987) 329.
[3] R.G. Brown, P.Y.C. Hwang, Introduction to Random Signals
and Applied Kalman Filtering, Wiley, New York, 1997.
[4] B. Chun, B. Kim, Y.H. Lee, Generalization of exponentially
weighted RLS algorithm based on a state-space model, IEEE
Internat. Symp. Circuits System V (1998) 198201.
[5] E. Eleftheriou, D.D. Falconer, Tracking properties and
steady-state performance of RLS adaptive lter algorithms,
IEEE Trans. Acoust. Speech Signal Process. ASSP-34 (1986)
10971110.
[6] E. Ewada, Comparison of RLS, LMS and sign algorithms for
tracking randomly time-varying channels, IEEE Trans. Signal
Process. 42 (1994) 29372944.
1718 M.B. Malik / Signal Processing 84 (2004) 17091718
[7] G.F. Franklin, J.D. Powell, M.L. Workman, Digital Control of
Dynamical Systems, 2nd Edition, Addison-Wesley, Reading,
MA, 1990.
[8] B. Hassibi, A.H. Sayed, Thomas Kailath, H

optimality of
the LMS algorithm, IEEE Trans. Signal Process. 44 (1996)
267280.
[9] S. Haykin, A.H. Syed, J. Zeidler, P. Yee, P. Wei, Adaptive
tracking of linear time-variant systems by extended RLS
algorithms, IEEE Trans. Signal Process. 45 (6) (May 1997)
11181128.
[10] S. Haykin, Adaptive Filter Theory, 4th Edition, Prentice-Hall,
Englewood Clis, NJ, 2001.
[11] S.M. Kay, Fundamentals of Statistical Signal Processing,
Estimation Theory, Prentice-Hall, Englewood Clis, NJ,
1993.
[12] H.K. Khalil, Nonlinear Systems, 3rd Edition, Prentice-Hall,
Englewood Clis, NJ, 2002.
[13] M.B. Malik, State-space recursive least squares, IEEE
Conf. on Acoust. Speech Signal Process. VI (2000)
645648.
[14] M.B. Malik, State-space recursive least squares, IEEE Conf.
on Multi-Media Expo I (2003) 837840.
[15] M.B. Malik, State-space RLS with adaptive memory, Internat.
Symp. Image Signal Process. Anal. I (2003) 146151.
[16] M.B. Malik, State-Space Recursive Least-Squares, Ph.D.
Dissertation, College of Electrical and Mechanical
Engineering, National University of Sciences and Technology,
Pakistan, 2004.
[17] M.B. Malik, State-space recursive least squares: Part II, Signal
Process. 84 (2004) this issue.
[18] M.B. Malik, H. Qureshi, R.A. Bhatti, Tracking of linear
time-varying systems by state-space recursive least squares,
IEEE International Symposium on Circuits and System, Vol.
III, 2004, pp. 305308.
[19] Wilson J. Rugh, Linear System Theory, Prentice-Hall,
Englewood Clis, NJ, 1996.
[20] A.H. Sayed, Thomas Kailath, A state-space approach to
adaptive RLS ltering, IEEE Signal Process. Mag. 11(3)
(1994) 1860.
[21] M.D. Srinath, P.K. Rajasekaran, R. Viswanathan, Introduction
to Statistical Signal Processing with Applications, Prentice-
Hall, Englewood Clis, NJ, 1996.
[22] H. Stark, J. Woods, Probability, Random Processes, and
Estimation Theory for Engineers, Prentice-Hall, Englewood
Clis, NJ, 1994.
[23] H.J. Sussman, P.V. Kokotovic, The peaking phenomenon and
the global stabilization of nonlinear systems, IEEE Trans.
Automat. Contro. 36 (1991) 424440.