Gorm Andresen

20020491

14. juni 2006

Nonlinear Least-Square fit

Project in Numerical Methods 2006

Introduction

Given a set of N data points (xi , yi ) with errors i and a function f (x; c) which
depend on the vector of coefficients c = (c1 , c2 , . . . , cn ), least-square fitting is a
problem of finding a choice of coefficients c0 which minimizes the distance between
the function and the datapoints. In this context the distance is defined as:
2

N
X

[yi f (xi ; c)]2 /i 2

(1)

i=1

If f (x; c) depends linearly on the ci s the problem can be solved using linear
algabra (see Note 5), but if f (x; c) is nonlinear in some or all of the ci s then
other methods have to be used (see Note 8).
In the present paper a numerical method which can minimize 2 as a function
of c, even if nonlinear ci s are present, is described and an implementation of the
method is presented and tested.

Numerical method

In the most general case 2 has an arbitrary dependence on the ci s, and the problem of finding c0 is thus equivalent to finding a global minimum in n-dimensional
space.
In this paper the downhill simplex method is used to find a lokal minimum of
2 in n-dimensional space. This method is very robust and it requires relatively
few evaluations of the function. On the other hand the method has the drawback,
that an initial simplex has to be specified and that one only finds a local and
thus not necessarily a global minimum.
To mend the problem of finding the global minimum the initial simplex can
be chosen near the global minimum, or in other words one has to be able to guess
the approximate location of the global minimum of 2 in n-dimensional space.
If the user of the method is able to specify such a guess the Downhill simplex

method will return c0 with a given accuracy.

It is often of interest to know the typical error dc of c0 . Since the typical
chance of 2 is unity dc can be estimated from the second order derivatives of
2 with respect to the ci s as shown below.
By definition c0 is a minimum of 2 in n-dimensional space, so the second
order Taylor expansion of 2 can be written as:

X 2 2
1
dci dcj
(2)
2 (c0 + dc) = 2 (c0 ) +

2 i,j ci cj
c=c0

Now if only on of the ci s are varied a typical variation of ci , that is a dci

fulfilling: 1 = 2 (c0i + dci ) 2 (c0 ), will be given as:

!1
2 2

dc2i = 2
(3)

c2i
c=c0

More generally the (i, j)th element in the covariance matrix of the fit can be
estimated as:

!1
2 2
dci dcj = 2
(4)

ci cj
c=c0

Implementation

Source code is found at: http://www.phys.au.dk/~ga/numeric/

The numerical method described above is implmented in the Fortran 95 subroutine nonlinearfit(). The subroutine takes the following inputs:
n_data is the number of datapoints (N).
x, y and are N -dimensional arrays containing the datapoints and errors.
f() is the function to fit to.
cmin and cmax is a guess on the lower and upper bound of the domain in
n-dimensional space where c0 is located.
acc_size, acc_f and max_it determines when the simplex algorithm stops.
acc_size and acc_f are typically in the order of 0.01-0.001 or smaller and
the upper bound on the number of iterations max_it can be set to the order
of 1000 or more.

The subroutine returns a n-dimensional array cfit as an estimate of c0 and a

n-by-n matrix Cov as an estimate of the covariance matrix of the fit. Furthermore
the value of 2 corresponding to cfit and the number of iterations are printet
to the default terminal.
The heart of nonlinearfit() is the downhill simplex method and the implementation of this method is chosen as described on note 8.
To start the Downhill simplex routine an initial simplex must be specified.
The following code builds the initial simplex, represented by a n-by-(n+1) matrix
S, from the upper and lower guess cmin and cmax:
do i=1,n+1
call random_number(S(:,i))
S(:,i)=(cmin+cmax)/2.0 + 0.1*(cmax-cmin)*S(:,i)
end do
With this choice the initial simplex is located in the center of the n-dimensional
box determined by cmin and cmax and it has a volume of approximately 10%
of the box volume. The reason behind this choice is to make the simplex small
enough to roll in the box and to locate it at the expected location of the global
minimum of 2 , but there is no way to ensure that it is an optimal choice for all
problems.
When the estimate of c0 is found the elements of the covariance matrix (see
(4)) are calculated as:
dci dcj =
2 (c0i + ci , c0j + cj ) 2 (c0i + ci , c0j ) 2 (c0i , c0j + cj ) + 2 (c0i , c0j )
(5)
ci cj
where ci = 0.001 ci and cj = 0.001 cj are small variations.

Test of nonlinearfit()

nonlinearfit() has been tested on a harmonic function:

f (x; (c1 , c2 , c3 )) = c1 sin(c2 x + c3 )

(6)

(xi , yi , i ) data were generated using the subroutine generate_data() which for
a value of xi returns:
yi = f (x; (c1 , c2 , c3 )) + rnd i
where rnd is a random number between -0.5 and 0.5.

(7)

With (c1 , c2 , c3 ) = (1.3, 3.4, 0.3) 19 datapoints with x-values randomly distributed between 0.1 and 11.0 and with = 0.2 has been generated. A fit to the
points has then been performed using nonlinearfit(), and the result is shown
below:
Nonlinear fitting routine using a downhill simplex method.
==================================================================
Function: f(x)=c1*sin(c2*x+c3) (x,y) data is generated as
(x,f(x)+noise) with random x and with
(c1,c2,c3):
1.300000
3.400000
0.3000000
-----------------------------------------------------------------Minimum bounds on coefficients:
1.170000
3.060000
0.2700000
Maximum bounds on coefficients:
1.430000
3.740000
0.3300000
-----------------------------------------------------------------Number of itterations done:
33
Best chi**2 found:
1.376749
-----------------------------------------------------------------cfit
1 =
1.288280
+/- 6.4141616E-02
cfit
2 =
3.394368
+/- 8.7871952E-03
cfit
3 = 0.3091391
+/- 5.0209433E-02
-----------------------------------------------------------------Estimated covariance matrix:
4.1141468E-03 9.6444059E-03 2.8554073E-02
9.6444059E-03 7.7214798E-05 5.2758143E-04
2.8554073E-02 5.2758143E-04 2.5209871E-03
A similar fit to the same data has been performed with Gnuplots built in
nonlinear fitting routine. Here the results are:
After 4 iterations the fit converged.
final sum of squares of residuals : 1.22912
rel. change during last iteration : -1.57168e-14
degrees of freedom (ndf) : 16
rms of residuals
(stdfit) = sqrt(WSSR/ndf)
: 0.277165
variance of residuals (reduced chisquare) = WSSR/ndf : 0.0768202
Final set of parameters
=======================

Asymptotic Standard Error

==========================

a
b
c

+/- 0.01803
+/- 0.004638
+/- 0.02694

= 1.29204
= 3.39951
= 0.272031

(1.396%)
(0.1364%)
(9.904%)

Nonlinear fit
2
Data
Nonlinear fit
Gnuplot fit

1.5

0.5

-0.5

-1

-1.5

-2
0

6
x

10

Figur 1: Test of nonlinearfit(). The fit returned from nonlinearfit() to 19 datapoints are compared with the datapoints and with a similar fit by Gnuplot.

When comparing the two results it is seen that Gnuplot and nonlinearfit()
agrees on the coeefficients within the errors. Gnuplot returns a slightly better 2
and the errors on the coefficients are a little smaller, but within the same order
of magnitude as those estimated by nonlinearfit(). Figure 4 shows a plot of
the datapoints and the fits returned by both Gnuplot and nonlinearfit() and
it is seen that both fits agrees very well with the datapoints and with each other.

Conclusion

A numerical method to do a nonlinear least-square fit has been described and

implemented in Fortran 95. The routine has been tested and a test case shows
that the routine agrees well with the slightly better fitting routine build into
Gnuplot.
A comprehensive test has not been performed, but there are no reason to
believe that the the method should be erroneous.

12