Basic Numerical Methods and FreeMat

By Timothy Cyders and Gary Schaefer

Essentially, all models are wrong, but some are useful.
- George Box
We demand guaranteed, rigidly-defined areas of doubt and uncertainty.
- Douglas Adams
About This Document
In a nutshell,
Students are poor. Commercial softare pac!ages such as "AT#AB are expensi$e. %ree"at
is free, as in freedom, and as in &eer.
Students are poor. 'ngineering("athematics texts are expensi$e. This document is free as
in freedom, and as in &eer.
A lot of &asic ideas put forth in most 'ngineering("athematics texts are rather old, and are
generally common !noledge, at least to the educated. 'specially ith respect to explanations of
commercial softare pac!ages such as "AT#AB, and introductions(explanations of numerical
methods, texts tend to under-explain and o$ercharge. "y personal experience ith college texts
has &een that the ones here explanation ould ha$e greatly augmented learning lac!ed such
explanation, &ut certainly didn)t hold &ac! on cost.
This text is a response to that phenomenon. This is a &asic introduction to numerical methods,
shoing ho the commonly used methods or! &y some simple examples. Also, it)s a &it of an
introduction to the language of %ree"at, hich is easily interchangea&le ith commercial
pac!ages such as "AT#AB, or is at the least reada&le pseudo-code for examples to use in other
languages *such as C++ or %,-T-A./. 0ro&a&ly the most significant aspect of this document is
that it is free *as in freedom/. If there are pictures, asides, explanations or examples that can ma!e
learning &etter, you can add them and repu&lish the document *citing the original or!,
o&$iously/1
There are certain things that this text is and isn)t. This text is a &asic introduction. It is by no means
a full-on graduate-le$el text on the intricacies of numerics and solution of differential e2uations.
This text, as you)ll see, doesn)t present much of anything in the ay of analytical solution of
differential e2uations. This text $ery lightly addresses error and uncertainty, &ut su&stantially
addresses examples of &asic pro&lems engineers may face. This text is a &asic introduction to the
idiosyncrasies of the %ree"at programming en$ironment, and does pro$ide donloada&le
examples of e$ery type of pro&lem co$ered.
This text is not for commercial sale, and may not &e printed, copied or modified for the purpose of
selling it. This text is free to copy, distri&ute, donload and disseminate 3 e encourage exactly
this. 4e 5ust as! that you gi$e %ree"at a try, and you might fall in lo$e ith it li!e e ha$e.
Table of Contents
Basic .umerical "ethods and %ree"at...................................................................................................6
A&out This Document...............................................................................................................................7
Basic .umerical "ethods and %ree"at...................................................................................................8
Section 69 -oot %inding.........................................................................................................................8
6.6 - The Bisection "ethod...............................................................................................................8
6.: - The .eton--aphson "ethod................................................................................................;
6.7 - The Secant "ethod.................................................................................................................6<
6.= - .onlinear Systems of '2uations.................................................................................................6>
The .eton--aphson "ethod for Systems of '2uations...........................................................6>
Section :9 .umerical Differentiation .................................................................................................:6
Section 79 .umerical Integration.......................................................................................................:6
:.6 - -ectangular -ule....................................................................................................................:=
:.: - Trape?oid -ule........................................................................................................................76
:.7 - Simpson)s -ule........................................................................................................................76
:.= - Gaussian @uadrature..............................................................................................................76
:.8 - "onte Carlo Integration.........................................................................................................76
Section 79 Initial Aalue 0ro&lems *,rdinary Differential '2uations/...............................................7=
7.6 - 'uler)s "ethod.......................................................................................................................7=
7.: - -unge-Butta "ethods...........................................................................................................7C
7.7 - Adams-Bashforth "ethods...................................................................................................=7
7.= - Systems of Differential '2uations........................................................................................=7
Section =9 Boundary Aalue 0ro&lems................................................................................................==
=.6 - #inear Shooting "ethod........................................................................................................==
=.: - Shooting "ethod for .onlinear 0ro&lems...........................................................................==
=.7 - %inite-Difference "ethods....................................................................................................==
Section 89 0artial Differential '2uations and the %inite 'lement "ethod......................................==
Basic Numerical Methods and FreeMat
.umerical methods pro$ide a ay to sol$e pro&lems 2uic!ly and easily compared to analytic
solutions. 4hether the goal is integration or solution of complex differential e2uations, there are
many tools a$aila&le to reduce the solution of hat can &e sometimes 2uite difficult analytical
math to simple alge&ra and some &asic loop programming. %ree"at has a &ig speed ad$antage in
terms of simple looping than!s to the ne DIT compiler, hich can run loops and other simple
programming as fast if not much faster than e$en most commercial pac!ages such as "AT#AB.
Section 1: Root Finding
,ne of the most &asic applications of numerical methods is to find the roots of a single e2uation.
In cases here an e2uation)s solution cannot &e o&tained &y alge&raic means *as is the case ith
many non-linear e2uations/, there are se$eral methods for finding the roots *solutions/ ith the
poer of a computer and some &asic algorithms, hich ill &e discussed here.
1.1 - The Bisection Method
If you ha$e e$er searched in a phone &oo! for a name, you)$e intuiti$ely performed something li!e
the &isection method. The &isection method is a simple ay to find a single root of an e2uation,
gi$en that the root exists &eteen to &ounds *hich must &e defined at the outset of the
script/, and it is the only root &eteen those &ounds. The method then reduces these &ounds &y
half, alays selecting the half containing the root, until the &ounds are reduced &elo an
accepta&le error limit.
'xample9 #et)s examine the nonlinear e2uation
(
x
2
4
)
sin( x)=0
4e can 2uic!ly notice se$eral things a&out this e2uation. %irst, it can &e descri&ed as the
intersection of to functions,
f
1
( x)=
x
2
4
f
2
( x)=sin( x)
Second, e see &oth &y inspection of either graph that there are to solutions, one at ?ero *a
tri$ial solution/ and one near :. %inally *and most pertinently to the su&5ect under discussion here/,
e see that this e2uation cannot be solved ith con$entional alge&ra, so the only ay to o&tain a
solution ithout using something li!e a Taylor series expansion is either graphically or &y a
numerical method. 4e)ll approach this pro&lem using the &isection method. The &asic operating
principle of this method is as follos9
You are looking for the name Stevens in a honebook. !ick u the honebook, and oen
it to the middle, you will find names beginning with ". Which half of the book is
Stevens in# $s Stevens is in the latter half, we then take that half of the book, and slit
it in two. %he resulting age has the name &euben on it. Stevens comes after &euben
'i.e.( Stevens is in the second half), and so we slit the second half again. *ontinue this
rocess, always selecting the half containing the solution you are looking for, and you
will find the age with Stevens on it in a matter of seconds, even with a large
honebook. You reduce your bounds by a factor of two each iteration, so the method
converges very +uickly,
4e can no apply this method to root finding using a hile loop ith some simple &oolean
operators *if-then or true-false statements/. %irst, e ill determine hich half of our &ounds
contains the solution. Then, e)ll reduce our &ounds &y setting a ne point for either the upper
limit or loer limit to the midpoint of the &ounds, and reiterate the selection process. After 5ust a
fe iterations, e ill narro our &ounds 2uic!ly to the point that e can estimate an anser.
#et)s start ith a set of &ounds that surround the root in 2uestion *and only the root in 2uestion1/,
E6 7F. An example code ould loo! li!e so9
% bisection.m
%% --- Define Bounds and Error --- %%
hi = 3; % Upper bound
low = 1; % Lower bound
epsilon = e-!; % "cceptable error for solution
counter = #; % $ounter %to a&oid infinite loops'
limit = 1###; % Limit number of iterations to a&oid infinite loops
%% --- (tart loop --- %%
while %hi-low' ) epsilon ** counter + limit; % $,cle bound reduction
until solution or ma-its reached
mid = %hi . low' / ; % (et midpoint halfwa, between lo and hi
if %%%hi0hi/1' - sin%hi''0%%mid0mid/1' - sin%mid'' + #' %
e&aluate % 2%-' at mid and hi3 and multipl, the results 4 a ne5ati&e
result means % the function crosses an a-is %a root6'
low = mid; % Eliminate lower half if it doesn7t contain the
%root
else
hi = mid; % Eliminate upper half if it doesn7t contain the
%root
end
counter = counter . 1; % 8ncrease counter b, 1 for each iteration
end
mid = %hi . low' / % (et midpoint halfwa, between lo and hi one last
% time %to output solution'
'xecute this script, and it ill arri$e at the anser 6.C77G8. If e plug this into the original
e2uation, e get a $ery small error on the order of 6<e-G, in only a millisecond and 6G iterations1 To
determine an appropriate num&er for maximum iterations, remem&er that your &ound idth is9
(hi low)
2
n
here n is the num&er of iterations completed. To narro Ayn -and)s &oo! Atlas Shrugged
*>=8,<<< ords, approximately 8C< ords per page 3 the third largest single-$olume or! in the
'nglish language/ to the to ords Hto shrugI *page =::, Centennial 'dition/ ould ta!e only
2=
645000
2
n
n=
ln(322500)
ln (2)
19
iterations to narro don to those ords alone.
There are a fe things to note a&out this method. %irst, there are se$eral ays to select hich half
contains a solution, &ut perhaps the simplest is to multiply %*mid/J%*hi/. If the solution lies in
&eteen mid and hi, the effect of the function crossing an axis ill force the output to &e negati$e
*so e compare the product to </. Second, if there are multiple roots inside the &ounds, the
se2uence ill con$erge on one of the original &ounds *except in the case of an odd num&er of
roots, in hich case it ill con$erge on only one root/. %or these reasons, you must !no
accepta&le &ounds &efore running the script to use this method, so it is somehat limited.
1.2 - The Newton-Rahson Method
The .eton--aphson "ethod *or simply .eton)s "ethod/ is another so-called local method to
determine the root of an e2uation. This method uses a single starting point *as opposed to the
&ounds re2uired &y the &isection method/, and repeatedly uses a deri$ati$e to pro5ect a line to the
axis of the root in 2uestion, as shon &elo. This results in a $ery fast con$ergence in many cases,
&ut does re2uire the solution of a deri$ati$e &efore the method can &e used.
Kere)s ho it or!s9 %irst, you need to e$aluate the function and its deri$ati$e. 4e)ll start again
ith the same e2uations e used in the last section9
f ( x) =
x
2
4
sin( x)
f ' ( x) =
x
2
cos( x)
.o, e ta!e our initial guess, find the tangent line to our function %*x/, and set our ne guess to
the x-intercept of that tangent line. The actual e2uation to calculate this is9
p
i+1
= p
i

f ( p
i
)
f ' ( p
i
)
here i is counting iterations, starting at 6. 'ssentially, the ne guess *p
i+6
/ is displaced along the x-
axis from the old guess *p
i
/ &y the function $alue o$er the slope9
4e continue this process until each ne iteration of p increases &y an inter$al less than our
accepta&le error *i.e. Lp
i+6
- p
i
L M error/. .o, to do this in %ree"at, e)ll use a hile loop again,
&ecause e)re not sure 5ust ho many iterations e ill need. 4e)ll also use a method called
anonymous functions that allos you to pass a function as a $aria&le. This ill help condense our
code, and allo for future modification to our program. %irst, e)ll define our function f, and its
deri$ati$e fp9
f = 9%-' %%-.:/1' 4 sin%-'';
fp = 9%-' %-/ 4 cos%-'';
.otice the decimal after x in the first relation 3 this ensures that the matrix elements are s2uared,
instead of %ree"at trying to s2uare the matrix, hich is a different operation all together re2uiring
the matrix to ha$e specific properties. .o, at the command indo, if you type
--) f%#'
ans = #
--) f%1'
ans = -.#!;1!
4e can see from the plot of our function on page ;> that these $alues are simply the e$aluation of
the function at those to points. So no, e can set our error and initial guess, and form a hile
statement to do our looping for us9
dx
dy
f ( p) = dy
f ' ( p) =
dy
dx
dx =
dy

(
dy
dx
)
=
f ( p)
f ' ( p)
error = e-!; % (et error bound
p = 3; % 8nitial 5uess
i = 1; % (tart counter at 1
p%i.1' = p%i' - f%p%i''/fp%p%i''; % <et initial &alue for p%' for
while loop
while abs%p%i.1' - p%i'' ) error
i = i . 1;
p%i.1' = p%i' - f%p%i''/fp%p%i'';
end
-un this script, and it ill complete after only 8 iterations, much faster than the &isection method1
The ay e did it here as &y simply creating ne elements in a p matrix at each iteration. In the
end, e ha$e9
--) p7
ans =
3.##############
.1!3#!=>;#133;
1.;!1#3?>1##!?#
1.;33;=1!3=!#=
1.;33=!3=??!!=>3
1.;33=!3=>?=#
.otice that each iteration, e dou&le the num&er of accurate digits after our first guess. The last
iteration is accurate to all the decimal places shon in long format. If e plug p*>/ into our original
function, e should get something very close to <9
--) f%p%>''
ans =
!.!!111!131!=?e-#1>
,ur last guess as actually accurate on the le$el of 6e-6>1 .eton)s method is $ery poerful, &ut
you ha$e to !no the deri$ati$e &eforehand. Also, if you pass a minimum or maximum in the
function *or a spot here the deri$ati$e approaches </, this method ill 2uic!ly di$erge ithout
arning. To get around our deri$ati$e pro&lems, e can estimate a deri$ati$e numerically and
include some or!arounds 3 con$ergence on)t &e as fast, &ut it ill &e easier in some cases, and
a &it more relia&le. This is called the Secant "ethod.
1.! - The "ecant Method
The Secant "ethod performs the same tas! and or!s much in the same ay as the .eton-
-aphson "ethod, &ut ithout needing to !no the deri$ati$e *and also ithout &eing su&5ect to
singularities ith said analytical function/. It differs on se$eral !ey points9 first, e start ith to
initial guesses at the root. It is important that these guesses &e close to each other and near the
root, otherise e *as alays/ ris! di$ergence. Second, instead of using the analytical deri$ati$e,
e numerically calculate a secant line, saying that it)s close to the deri$ati$e. By draing this line
&eteen f*x/ at our to guesses, the root of that line gi$es us our next guess for the function
root. As the method con$erges, our guess gets closer and closer to ?ero. Kere are se$eral images
depicting hat happens - gi$en again our test function9
f ( x)=
(
x
2
4
)
sin( x)
4e pic! to pointsN in this case e)ll use 7.8 and :.8. .o, e)ll dra a line &eteen the to
points and find its root using the folloing formula9
x
n+1
= x
n

x
n
x
n1
f ( x
n
) f ( x
n1
)
f ( x
n
)
The x-intercept of our line turns out to &e :.6<>=. #et)s chec! to see ho close e are to ?ero9
2.1064
2
4
sin (2.1064) = 0.2493
4e)re still a fair &it from the root, so no e)ll use the points :.8 and :.6<>= for our next iteration9
Osing our formula again, e get 6.C>C6. Again, e)ll chec! our anser, and see ho close e are
to the root9

1.9691
2
4
sin(1.9691) = 0.0477
Getting closer1 As e continue this process, e e$entually find our result at 6.C77G, our
approximate root. So, ho do e code thisP Kere)s an example script9
%secantmethod.m script
- = @3.!; .!A; % - arra, of 5uesses
n = ;
while% %-%n':'/1 4 sin%-%n'' ) 1e-!'
-%n.1' = -%n' 4 %%-%n' - -%n-1''/%%%-%n':'/1 4 sin%-%n''' -
%%-%n-1':'/1 4 sin%-%n-1'''''0%%-%n':'/1 4 sin%-%n''';
n = n.1;
end
If e run this from the %ree"at command line, e get9
--) secantmethod
--) -
ans =
3.!#############
.!#############
.1#>3;;>1!!=3=1
1.;>;133!?1!3!#
1.;3>>?#?;=;;>
1.;33?#?>3!;3;1!
1.;33=!3?1;?1?#=
0erhaps a more ro&ust program ould allo us to simply pass our function and to initial guesses
as arguments9
function ret = secant%f-3-13-'
- = @-1; -A; % initial 5uesses
n = ; % initiate counter for loop
while%abs%f-%-%n'' ' ) 1e-!' % precision of 1e-!
-%n.1' = -%n' - %%-%n' - -%n-1''/%f-%-%n'' - f-%-%n-
1''''0f-%-%n'';
n = n.1;
end
ret = -%n'; % return last &alue of - %our root'
.o, e can pass any function e li!e, ith to initial guesses9
--) f- = 9%-' %-.:! - - 4 1'; % we can7t sol&e this one anal,ticall,
--) secant%f-3#.;31' % one of the roots is somewhere near 1
ans =
1.1>=3#3?;1;;?!#
--) f-%ans' % let7s checB our answer
ans =
->.?;>;=?!11>1!#?e-#=
The secant method con$erges a &it sloer than .eton)s "ethod in most cases, &ut tends to &e
somehat more sta&le, and is easier to em&ed into adapti$e code that helps it a$oid di$ergence.
1.4 - Nonlinear Systems of Equations
0erhaps a more interesting *and more useful/ application of root-finding is to sol$e systems of non-
linear e2uations. In many areas of science, e ant to model hole systems, hich can in many
cases pro$e difficult to sol$e analytically, so e can use the method descri&ed here to
approximate some solutions.
The Newton-Rahson Method for "#stems of $%uations
Section 6.7 detailed .eton)s "ethod to sol$e for the roots of a nonlinear e2uation. But hat if
e ha$e a hole system of e2uationsP .eton--aphson also or!s for systems of e2uations, and
is relati$ely simple *especially in %ree"at1/ hen e use a matrix approach. I)ll s!ip some theory
here, &ut &y the same assumptions made in .eton)s "ethod, e can arri$e at the folloing
e2uation9

6 f
1
6 x
1
6 f
1
6 x
2
6 f
2
6 x
1
6 f
2
6 x
2
|

A x
1
A x2
|
=

f
1
f
2
|
The first matrix seen here is called the Daco&ian "atrix, simply a matrix ith $alues for partial
deri$ati$es corresponding to their location inside the matrix. If you loo! at this e2uation, it has the
classic Ax Q & appearance, hich means e can simply use AR& *Gaussian 'limination/ to sol$e for
x. Gi$en that the only un!nons in this e2uation are x
6
and x
:
, e start ith a guess *5ust as in
.eton)s "ethod/, and use Gaussian 'limination to sol$e for these $alues. 4e then get our ne
$alues for x
6
and x
:
from our original x
6
and x
:
as follos9

x
1
x
2
|
new
=

x
1
x
2
|
old
+

A x
1
A x
2
|
4e iterate this idea until our delta matrix *the error/ &ecomes smaller than some threshold that
e set, and e can then say )close enough1). Straight aay, e can ma!e a simple program to
e$aluate this method for a system of to e2uations ith to un!nons. #et)s 5ust put the
e2uations and their deri$ati$es right into the code and ma!e a script .m file9
% newts,s.m script file
%% --- Cwo (imple Cest 2unctions --- %%
f1 = 9%-3,' %-: 4 10%,:3'';
f = 9%-3,' %sin%-' . 30cos%30,'';
%% --- Cheir Deri&ati&es --- %%
df1d- = 9%-3,' %0-';
dfd- = 9%-3,' %cos%-'';
df1d, = 9%-3,' %-10,:';
dfd, = 9%-3,' %-;0sin%30,'';
-, = @1;1A; % 8nitial <uess at (olution
delta- = @1#; 1#A; % starter &alue for delta - matri-
counter = #; % initialiDe counter &ariable
%% --- Eacobian Fatri- --- %%
E = 9%-3,' @df1d-%-3,' df1d,%-3,';dfd-%-3,' dfd,%-3,'A;
%% --- 2unction Fatri- --- %%
2 = 9%-3,' @f1%-3,';f%-3,'A;
%% --- 8terati&e CechniGue --- %%
while abs%ma-%delta-'' ) 1e-!
delta- = E%-,%1'3-,%''H2%-,%1'3-,%''; %<aussian Elim. (tep
-, = -, 4 delta-; % modif, @-;,A per iterati&e techniGue
counter = counter . 1; % count number of iterations
end
printf%7%3d iterationsHn73counter';
printf%7- = %3fHt , = %3fHn73-,%1'3-,%'';
Kere, e)$e set up a matrix D*x,y/ hich contains anonymous functions, each also a function of
*x,y/ and a matrix %*x,y/ ith similar properties. This ay, e can simply put in our guess for x and
y &y e$aluating D*x,y/ and %*x,y/. In this case, x is xy*6/ and y is xy*:/. 4e perform our iterati$e
techni2ue &y Gaussian 'limination using the &ac!slash operator *AR& or in this case, D*x,y/R%*x,y//,
and the resulting output is our delta matrix containing x and y. 4e simply loop these to
commands, and e e$entually reach a solution9
--) newts,s
> iterations
- = 1.33;!>3 , = 1.>=>#13
If e e$aluate f
6
and f
:
at these coordinates, e should get something $ery close to ?ero9
--) f1%1.33;!>331.>=>#13'
ans =
-=.1#!1e-1!
--) f%1.33;!>331.>=>#13'
ans =
.>>1!e-1!
So, it or!ed1 This techni2ue can &e easily extrapolated to systems of many e2uations and many
un!nons, ith to ca$eats9 all partial deri$ati$es ha$e to &e !non *or numerically
approximated/, and in such systems, many times there are multiple solutions. This method can
di$erge, and the solutions reached are usually hea$ily dependent on your initial guesses, so it)s
necessary to ha$e an idea of hat your functions loo! li!e in the first place, and here the
solutions might &e. Sou)$e &een arned.
.o, let)s loo! at a real-orld example, along ith a program that allos us some input and
manipulation. Binematic analysis of the four &ar mechanism shon &elo results in the folloing
system of e2uations9

f
1
(0
3
, 0
4
)
f
2
(0
3
, 0
4
)
|
=

r
4
cos(0
4
) r
3
cos(0
3
) + r
2
cos(0
2
) r
1
r
4
sin (0
4
) r
3
sin (0
3
) + r
2
sin(0
2
)
|
Kere, all radii *lin!age lengths/ are !non, as is
2,
so e really 5ust ha$e to functions, f
6
and f
:
of
to $aria&les,
3
and
4
. %or the Daco&ian, e ta!e the partial deri$ati$es as shon pre$iously9

6 f
1
60
3
6 f
1
60
4
6 f
2
60
3
6 f
2
60
4
|
=

r
3
sin(0
3
) r
4
sin(0
4
)
r
3
cos(0
3
) r
4
cos(0
4
)
|
.o, if e ha$e an initial guess at
3
and
4
, e can plug them into our iterati$e techni2ue, and
e$entually get real, accurate $alues1 #et)s start &y coding our functions and deri$ati$es as
anonymous functions *functions e can enter right from the command console, and pass around
li!e $aria&les/. 4e)ll assume here that you !no the $alues for r6, r:, r7, r= and th:. Instead of
typing their $aria&le names here, type in their actual $alues *e.g.9 f6QT*th7,th=/
*6.:GJcos*th=/...etc./
f1 = 9%th33 th1' %r10cos%th1' - r30cos%th3' . r0cos%th' . r1;
f = 9%th33 th1' %r10sin%th1' - r30sin%th3' . r0sin%th';
df1dth3 = 9%th33 th1' %r30sin%th3'';
dfdth3 = 9%th33 th1' %-r10sin%th1'';
df1dth1 = 9%th33 th1' %-r30cos%th3'';
dfdth1 = 9%th33 th1' %r10cos%th1'';
.ext, e)ll put together a function to hich e can pass our e2uations as matrix &loc!s. 4e)ll
simply define matrices for f and D li!e so9
f = 9%th33th1' @f1%th33th1'; f%th33th1'A;
E = 9%th33th1' @df1dth3%th33th1' df1dth1%th33th1';dfdth3%th33th1'
dfdth1%th33th1'A;
and e)ll design our function so e can pass these &loc!s to it as an argument. Kere)s an example9
function ret = nrs,s%f3E35uess' % Iewton-Japhson for eGuations in
unBnowns
epsilon = #.##1; % acceptable error
counter = 1;
delta = @1;1A
while delta ) epsilon
delta = E%5uess%1'35uess%''Hf%5uess%1'35uess%'';
5uess = 5uess . delta;
counter = counter . 1;
end
fprintf%7(ol&ed in %d iterations73counter'
ret = 5uess;
But ait, there)s more1 4hat if e don)t !no the analytical deri$ati$es of our functionsP 0erhaps
they)re &ehind a &lac! &ox, or 5ust really difficult to differentiate *or perhaps e 5ust don)t ha$e
time/P As e)ll see in the next section, e can calculate the deri$ati$e &y ta!ing hat)s called a
finite difference. Simply stated, a deri$ati$e is the rate of change of a function ith respect to a
change in its input, or rise o$er run. So, e could calculate an approximate partial deri$ati$e for
our four-&ar function a&o$e li!e so9
0
2, guess
= 40 , 0
3, guess
= 65
6 f
1
60
2

f
1
(0
2, guess
+c , 0
3, guess
) f
1
(0
2, guess
c , 0
3, guess
)
2c
here is relati$ely small. This means e can use the .eton--aphson method for systems
*effecti$ely the secant method here/ ithout ha$ing to analytically calculate the deri$ati$es1 4e
simply su&stitute our estimation of each deri$ati$e into the Daco&ian matrix, and sol$e as usual.
4e must &e careful, though. .umerical differentiation can &e su&5ect to large inaccuracy for a lot
of different reasons *hich ill &e discussed a &it in the next section/, so use caution hen you
use this method1
,ne last thing to mention for multiple root finding9 there is no a &uilt-in function in %ree"at
called fsol$e*/ that performs a .eton--aphson $ery much li!e the scripts shon here, &ut ith a
little more teeth and error chec!ing. %or most pro&lems, this &uilt-in su&routine can gi$e you an
out-of-the-&ox solution should you choose to use it. Simply type )help fsol$e) *sans 2uotes/ in your
%ree"at command indo for an explanation of usage.
Section 2: Numerical Differentiation
There are se$eral different methods that can &e used to calculate a numerical deri$ati$e.
Section 3: Numerical ntegration
.umerical integration is something that computers excel at. In fact, numerical integrals are the
&asis for many of the poerful things computers are capa&le of today. 4hether it)s pro5ecting the
position of your mouse cursor on the screen &ased on an infrared diode)s electrical &eha$ior or
calculating the physics effects in your fa$orite computer game, this &asic functionality is integral
*pun intended/ to the usefulness of the machine in front of you. )4hy integrate numericallyP), you
might as!. 4ell, ta!e a 2uic! loo! at hat an integral is doingN it)s the sum of a gi$en function,
iterated an infinite *or 5ust a high num&er/ of times o$er a $ery small inter$al. %or example, loo! at
this integral9

0
2
x
2
dx
4e can sol$e this analytically 2uite easily using an in$erse poer rule. There is no reason to
numerically integrate this formula., &ecause the closed form is easily o&tained. But hat if you
e$er encounter this mess of an e2uation9

1
u
x
.(2)
exp
(
( xx)
2
2u
x
2
)
dx=1
-igure .( /aussian robability density function '!0-) curve with a standard
deviation of ..
This is the calculation for the area from some point ? to infinity under the cur$e of the Gaussian or
normal pro&a&ility density function *0D%/. Such a cur$e is shon in %igure 6. This is actually a $ery
important function for a lot of different applications, and as an engineer or mathematician, you
ill indeed use it, especially its integral. The integral of this cur$e is e2ual to the pro&a&ility that a
normally distri&uted $aria&le)s $alue lies &eteen to &ounds, expressed in standard de$iations
from the mean at ?ero. The total area under the cur$e, from minus infinity to plus infinity, is e2ual
to one *the function as designed ith that specific characteristic in mind 3 there is a 6<<U
pro&a&ility that the $aria&le has some $alue on this inter$al, folloP/. If e change one of the
limits, hoe$er, then e may ha$e something that loo!s li!e this9
Pr (Z>z)=

1
u
x
.(2)
exp
(
( xx)
2
2u
x
2
)
dx=?
This is a rather nasty integral, &ut in the immortal ords of the Kitchhi!er)s Guide to the Galaxy,
don)t panic. All the e2uation a&o$e does is calculate the area under the Gaussian cur$e from some
point ? up to infinity. An example is if ?Q6, then the area to &e calculated ould &e as shon in
%igure :.
This is an example of an e2uation for hich there is no closed form solution *short of an infinite
series expansion/. The only ay to handle this type of e2uation is through numerical integration.
Than!s to computers and softare such as %ree"at, that)s much easier than if you happened to
&e using the solar-poered alternati$e shon in %igure 7.
-igure 1( 2ey, it3s a 2emmi,
-igure 4( /rah of /aussian curve showing e5amle area to
be calculated using numerical integration.
4e)re going to use this e2uation to demonstrate some some of the methods of numerical
integration. ,!ay, &ac! to the e2uation for the Gaussian, also called the normal, pro&a&ility
density function. It is9
PDF =
1
u
x
.(2)
e
(
( xx)
2
2u
x
2
)
here9

x
Q the standard de$iation of the distri&ution. This is an indication of ho ide or fat the
distri&ution is.

x Q the a$erage of the distri&ution. This is here the center of the distri&ution is located.
Osing %reemat, e can create a plot of this function, shon in %igure =.
This as created using this small &it of code9
si5ma=1;
-=-!K#.#1K!;
5aussian=%1/%si5ma0%0pi':#.!''0e-p%--.:.#/%0si5ma:'';
plot%-35aussian'
%rom this, e)ll explain and demonstrate se$eral methods for calculating the area underneath the
cur$e.
2.1 - Rectan&ular Rule
,ne ay to add up the area is ith the a &unch of rectangles. Dust place a &unch of e2ual-idth
rectangles under the area. 'ach rectangle stretches so that one of the top corners touches the
cur$e and the &ottom touches the &ottom, hich in this case is ?ero. This is shon in %igure 8.
-igure 6( /aussian robability density function
By adding up the areas of each rectangle, e can calculate the area under the cur$e. If e loo! at
each rectangle as ha$ing a idth and a height e2ual to the distance from ?ero to here it
touches the graph, e)ll ha$e something as shon in %igure >.
4e)ll no use this this graph and some %ree"at code to add up the area of all of the rectangles.
Ko many rectanglesP Good 2uestion1 The graph goes to infinity. But e don)t ant *or need/ to
go that high. So ho high do e need to goP .ote that the graph falls close to ?ero hen it gets
a&o$e 7. So, e)ll go a &it a&o$e that and go to 6<. And hat !ind of idth do e ant to useP
#et)s 5ust use 6(=, or <.:8.
% (cript to calculate the specific area under a <aussian probabilit,
% densit, function usin5 the Jectan5ular Jule.
w=#.!; % Chis is the width of each rectan5les.
startLpoint=1;
stopLpoint=1#;
-=startLpoint.wKwKstopLpoint.w; % Chis is the &alues of -. Iote that
-igure 7( E5amle of calculating the area from . to infinity
using rectangular areas.
-igure 8( /aussian curve with height and width annotated.
% each rectan5le touches the 5raph on
% the 0ri5ht0 side. Chis side is at
% the point of -.w3 which is wh, the
% counter 5oes from 1.w to 1#.w3 with
% a step siDe of Must w.
si5ma=1; % Che standard de&iation of the cur&e.
% Che ne-t line calculates the hei5ht of each rectan5le.
h=%1/%si5ma0%0pi':#.!''0e-p%-%-.:'/%0si5ma:'';
areaLrect=w0h; % $alculate the area of each rectan5le.
areaLsum=cumsum%areaLrect'; % (um up the areas of each rectan5le.
totalLarea=areaLsum%len5th%areaLsum''; % Chis is the final area.
printf%7Che area under the cur&e from %f to %f is
%f.Hn73startLpoint3stopLpoint3totalLarea';
Sa$ing this script as gaussian_area.m, then running it, e get the folloing result.
--) 5aussianLarea
Che area under the cur&e from 1.###### to 1#.###### is #.1;>=.
The area e calculated is <.6:C>G:. But is this correctP Another good 2uestion1 ,ne ay to find
out is to ma!e the idth of each rectangle smaller, rerunning the routine, and comparing the
anser to this one. #et)s ma!e the idth <.6, hich means our code ill appear as follos9
% (cript to calculate the specific area under a <aussian probabilit,
% densit, function usin5 the Jectan5ular Jule.
w=#.1; % Chis is the width of each rectan5les.
startLpoint=1;
stopLpoint=1#;
-=startLpoint.wKwKstopLpoint.w; % Chis is the &alues of -. Iote that
% each rectan5le touches the 5raph on
% the 0ri5ht0 side. Chis side is at
% the point of -.w3 which is wh, the
% counter 5oes from 1.w to 1#.w3 with
% a step siDe of Must w.
si5ma=1; % Che standard de&iation of the cur&e.
% Che ne-t line calculates the hei5ht of each rectan5le.
h=%1/%si5ma0%0pi':#.!''0e-p%-%-.:'/%0si5ma:'';
areaLrect=w0h; % $alculate the area of each rectan5le.
areaLsum=cumsum%areaLrect'; % (um up the areas of each rectan5le.
totalLarea=areaLsum%len5th%areaLsum''; % Chis is the final area.
printf%7Che area under the cur&e from %f to %f is
%f.Hn73startLpoint3stopLpoint3totalLarea';
-e-running the routine, e get9
--) 5aussianLarea
Che area under the cur&e from 1.###### to 1#.###### is #.11>=!?.
The area has gone up to <.6=>G8;. The difference from the old area is <.6=>G8; - <.6:C>G: Q
<.<6G<;>. The difference is 66.>U of the total area e 5ust calculated. %or most math programs,
that)s not good enough. But hat is good enoughP Typically, for numerical integration, e ant
to ha$e a difference of 6<
->
or less. That means a one in a million difference. The great thing is that
e can set up %ree"at to decide hen it has reached that threshold. 4e can !eep ma!ing the
idth smaller and smaller until it reaches the point here the difference is only 6 in 6 millionth of
the calculated area.
4e)re not going to go that far yet. Instead, e)ll use a less precise le$el of 6<
-:
5ust to demonstrate
the principle. To ma!e the decision hen this has &een reached, e)ll use a while loop, as shon
&elo.
% (cript to calculate the specific area under a <aussian probabilit,
% densit, function usin5 the Jectan5ular Jule.
clear all; % (tart b, clearin5 all &ariables.
w=#.1; % Chis is the width of each rectan5les.
startLpoint=1;
stopLpoint=1#;
-=startLpoint.wKwKstopLpoint.w; % Chis is the &alues of -. Iote that
% each rectan5le touches the 5raph on
% the 0ri5ht0 side. Chis side is at
% the point of -.w3 which is wh, the
% counter 5oes from 1.w to 1#.w3 with
% a step siDe of Must w.
si5ma=1; % Che standard de&iation of the cur&e.
% Che ne-t line calculates the hei5ht of each rectan5le.
h=%1/%si5ma0%0pi':#.!''0e-p%-%-.:'/%0si5ma:'';
areaLrect=w0h; % $alculate the area of each rectan5le.
areaLsum=cumsum%areaLrect'; % (um up the areas of each rectan5le.
totalLarea=areaLsum%len5th%areaLsum''; % Chis is the final area.
% Create a while loop that runs until the precision criteria is met.
diff_area=total_area; % This will be the difference between loops used
% to tell if the precision level has been met.
last_area=total_area; % This will be used to hold the area from the
% previous loop. It will be compared with the
% area of the current loop.
precision=1e-2; % This is the desired precision.
while diff_area!total_area"precision##; % $tart the while loop.
w=w%2; % $tart b& dividin' the width in half.
(=start_point)w*w*stop_point)w; % $et the values for (+ as before.
h=1%si'ma"2"pi#,-..##"e(p-(.,2#%2"si'ma,2##; % Calculate h.
area_rect=w"h; % Calculate the areas of each rectan'le.
area_sum=cumsumarea_rect#; % $um up the areas of each rectan'le.
total_area=area_sumlen'tharea_sum##; % Total area under the curve.
diff_area=abstotal_area-last_area#; % The difference from the previous
% calculation.
last_area=total_area; % /e-set the last area0s calculation before
runnin'
% the ne(t loop.
end
printf%7Che area under the cur&e from %f to %f is
%f.Hn73startLpoint3stopLpoint3totalLarea';
printf0The number of steps re1uired is %i for a precision of
%f.2n0+len'th(#+precision #;
.ext 2uestion9 4e)re adding up our rectangles going from 6 to 6<. In the actual integral, e)re
going from 6 to infinity. So is 6< high enoughP #et)s find out. 4e)ll set the upper limit, defined &y
the $aria&le stopVpoint, to :< and see if it ma!es a difference. 4e on)t sho all of the code here.
Dust change the fifth line a&o$e to read9
stopLpoint=#;
.o e)ll re-run the script. Kere)s the results, along ith the results from the pre$ious run here
the upper limit as 6<.
Che area under the cur&e from 1.###### to 1#.###### is #.1!=11>.
Che number of steps reGuired is =1 for a precision of #.#1####.
--) 5aussianLarea
Che area under the cur&e from 1.###### to #.###### is #.1!=11>.
Che number of steps reGuired is 1!1 for a precision of #.#1####.
The change from 6< to :<, hile it dou&led the num&er of steps, it didn)t change the final anser.
It ould appear that, once the upper limit is near 6<, the amount of additional area a&o$e that is
negligi&le. %or the rest of this exercise, I)m going to lea$e the upper limit at :< 5ust to !eep the
num&er of changes lo.
.ext, e)re going to sloly dial don the precision. Thus far, e)$e used a precision of 6<
-:
. .ext,
e)ll set it to 6<
-7
to see ho &ig of a change it ma!es. Again, I)m not going to display all of the
codeN instead, 5ust change the line that sets the $aria&le precision to9
precision=1e-3; % Chis is the desired precision.
-unning the script, e get9
--) 5aussianLarea
Che area under the cur&e from 1.###### to #.###### is #.1!?!>1.
Che number of steps reGuired is 131 for a precision of #.##1###.
The total area ent up from <.68G6=> to <.68;8>6. That)s a fairly large increase. Also note that the
num&er of steps ent up &y a factor of almost 6>. 4here did this extra area come fromP To
understand that, loo! &ac! at the original graph shoing ho the rectangles fit into the cur$e.
.ote that there is a lot of open space &eteen the tops of the rectangles and the cur$e itself, as
shon in %igure G.
So, ho do e capture that missed areaP The only ay to do so is to decrease the idth of the
rectangle. As e ma!e the idth smaller and smaller, the amount of missed area ill also
decrease. The dra&ac!, for us, is that, as e ma!e the idth smaller, the num&er of steps goes
up. That means that it)s going to re2uire more calculations to get to the le$el of precision desired.
4e ant a le$el of precision of 6<
->
. #et)s set the precision as shon here9
precision=1e->; % Chis is the desired precision.
.o, let)s run the script again.
--) 5aussianLarea
Che area under the cur&e from 1.###### to #.###### is #.1!?>!!.
Che number of steps reGuired is 1;#3>?1 for a precision of #.#####1.
4hoa1 That too! a little hile1 At least, it did on my computer
6
. 0erhaps yours ran faster. Still, it
should ha$e ta!en more time than hen you ran ith the precision set to 6<
-7
. And ho long did it
ta!e on my computerP #ong enough for me to al! to the !itchen, gra& a soda from the fridge,
and al! &ac!. But ho long as that, preciselyP #et)s find out. 4e)ll use the %reemat commands
of tic and toc. The command tic starts a timer runningN toc stops it and outputs the time, in
seconds. The code no loo!s li!e this, ith the ne code set in &oldface9
% (cript to calculate the specific area under a <aussian probabilit,
% densit, function usin5 the Jectan5ular Jule.
tic
clear all; % (tart b, clearin5 all &ariables.
w=#.1; % Chis is the width of each rectan5les.
startLpoint=1;
stopLpoint=#;
-=startLpoint.wKwKstopLpoint.w; % Chis is the &alues of -. Iote that
% each rectan5le touches the 5raph on
% the 0ri5ht0 side. Chis side is at
6 Currently, that computer is a Gateay G<<G- 0entium = ith hyper-threading. Bought it on sale at Best Buy &ac!
in :<<7.
-igure 9( When calculating the area of a curve using
rectangles, there3s a lot of sace between the rectangles
and the curve that is missed.
% the point of -.w3 which is wh, the
% counter 5oes from 1.w to 1#.w3 with
% a step siDe of Must w.
si5ma=1; % Che standard de&iation of the cur&e.
% Che ne-t line calculates the hei5ht of each rectan5le.
h=%1/%si5ma0%0pi':#.!''0e-p%-%-.:'/%0si5ma:'';
areaLrect=w0h; % $alculate the area of each rectan5le.
areaLsum=cumsum%areaLrect'; % (um up the areas of each rectan5le.
totalLarea=areaLsum%len5th%areaLsum''; % Chis is the final area.
% $reate a while loop that runs until the precision criteria is met.
diffLarea=totalLarea; % Chis will be the difference between loops used
% to tell if the precision le&el has been met.
lastLarea=totalLarea; % Chis will be used to hold the area from the
% pre&ious loop. 8t will be compared with the
% area of the current loop.
precision=1e->; % Chis is the desired precision.
while %diffLarea)%totalLarea0precision'';
w=w/; % (tart b, di&idin5 the width in half.
-=startLpoint.wKwKstopLpoint.w; % (et the &alues for -3 as before.
h=%1/%si5ma0%0pi':#.!''0e-p%-%-.:'/%0si5ma:''; % $alculate h.
areaLrect=w0h; % $alculate the areas of each rectan5le.
areaLsum=cumsum%areaLrect'; % (um up the areas of each rectan5le.
totalLarea=areaLsum%len5th%areaLsum''; % Cotal area under the cur&e.
diffLarea=abs%totalLarea-lastLarea'; % Che difference from the pre&ious
% calculation.
lastLarea=totalLarea; % Je-set the last area7s calculation before
% runnin5 the ne-t loop.
end
printf%7Che area under the cur&e from %f to %f is
%f.Hn73startLpoint3stopLpoint3totalLarea';
printf%7Che number of steps reGuired is %i for a precision of
%f.Hn73len5th%-'3precision ';
total_time=toc;
printf0The time re1uired for this runnin' was %f
seconds.2n0+total_time#;
The result is9
--) 5aussianLarea
Che area under the cur&e from 1.###### to #.###### is #.1!?>!!.
Che number of steps reGuired is 1;#3>?1 for a precision of #.#####1.
Che time reGuired for this runnin5 was 1!.=>3### seconds.
Is it time to trade in the old computerP Kardly. This &east is good for another 6<< &illion
computations. At least. Is there a ay to impro$e the code such that it doesn)t ta!e so longP Ses.
That)s here the trape?oid and Simpson)s -ule come into play.
2.2 - Trae'oid Rule
2.! - "imson(s Rule
2.) - *aussian +uadrature
2., - Monte Carlo -nte&ration
"onte Carlo methods are methods that use random num&ers to do different mathematical tas!s.
0ro&a&ly the most common usage is for numerical integration, as e)ll descri&e here. This method
can or! on anything from simple functions to image analysis. Kere)s ho it or!s9
Say e ere to lay out the domain E< F onto a dart &oard, and plot cos*x/ on that domain. If e
thro a random distri&ution of darts at this &oard, it might loo! li!e this9 *red crosses are darts/
.umerically, the proportion of darts Hunder the cur$eI *&eteen the cur$e and the x-axis/ to the
total num&er of darts thron is e2ual to the proportion of area under the cur$e to total area of
the domain. .ote that darts a&o$e the cur$e &ut &elo the x-axis su&tract from this total.
Bnoing that there are 6<< darts in the a&o$e figure, e can manually count and come up ith 6;
darts &elo the cur$e in the left half of the graph, and 6; darts a&o$e the cur$e *&ut &elo the
axis/ in the right half. 4e su&tract this from our first num&er and get <. The total area of our
domain is :, so <J: is the area under the cur$e. 4e !no, of course, that the $alue of this
integral should &e ?ero, and see that e come up ith that here.
0retty simple, rightP #et)s loo! at some simple code then1 #et)s start &y ma!ing a function e can
pass arguments to. 4e need to tell it three things9 our function *hich e can pass as a $aria&le
using an anonymous function/, the minimum x and the maximum x *the extrema of our domain/.
function returnL&alue = montecarlo%f-3 -min3 -ma-'
nLdarts = 1##; % number of darts to throw; more darts = more accurate
%% --- Determine Domain (iDe --- %%
- = linspace%-min3 -ma-3 ##';
for n = 1Klen5th%-'
,%n' = f-%-%n''; % draw cur&e to be inte5rated
end
,min = min%,';
if ,min ) # % if , is all positi&e3 min is #
,min = #;
end
,ma- = ma-%,';
%% --- Chrow Darts6 --- %%
dartcounter = #; % Chis will count the number of darts below the cur&e
for i = 1KnLdarts
r- = rand%'0%-ma---min' - -min; % 5enerates random dart
r, = rand%'0%,ma--,min' - ,min;
%% --- $ount Darts for $alculation --- %%
if r, + f-%r-' ** r, ) #
dartcounter = dartcounter . 1;
else if r, ) f%r-' ** r, + #
dartcounter = dartcounter - 1;
end
end
end
totalarea = abs%-ma---min'0abs%,ma--,min';
returnL&alue = %dartcounter/nLdarts'0totalarea; % area under cur&e
.o, let)s pass an anonymous function *a function stored in a $aria&le/ and a domain to our
program and see hat happens9
--) f = 9%-' %e-p%--.:'; % NEJO hard to inte5rate anal,ticall,6
I)ll ta!e a 2uic! aside and explain hat I 5ust did 3 the name of our function *f/ can &e passed
around 5ust li!e any other $aria&le. 4hen e say f Q T*x/, the T sym&ol denotes that it)s a
function *not a $ector or cell or other entity/, and the *x/ denotes that it)s a function of x. If it ere
a function of x,y and ?, e ould ha$e f Q T*x,y,?/. The rest of the expression is simply the
function to &e e$aluated, in this case9
f ( x)=e
x
2
4hen e call f*7/, for example, it ill return exp*-7W:/, or 6.:7=6x6<
- =
. #et)s continue9
--) -# = #; -1 = ; % e&aluate inte5ral from - = # to
--)montecarlo%f3-#3-1'
ans =
#.??1#
Straight aay e notice to things. %irst, our anser <.;;=< is close to *&ut not exactly/ the
analytical solution to the integral, <.;;:6. The more darts you thro, the more accurate the
anser ill &ecome, at the expense of computational time. Second, if e run the same input
se$eral times, e get different ansers. The random num&er generator o&$iously tries to
generate num&ers randomly, so the distri&ution of darts inside and outside the cur$e in 2uestion
changes slightly from one run to the next. This method is poerful &ecause it can integrate any
function that can &e e$aluated, &ut it)s expensi$e in terms of time, and its precision is sometimes
unrelia&le.
Section 3: nitial !alue "ro#lems $%rdinary Differential Equations&
0ro&a&ly one of the most significant applications of numerical techni2ues, especially in
engineering, is e$aluation of differential e2uations. "any of today)s pursuits in engineering in$ol$e
solution of ordinary differential e2uations *,D')s/ and(or partial differential e2uations *0D')s/.
Anything from dynamics and $i&rations to heat transfer, fluid mechanics and e$en
electromagnetism can &e modeled using differential e2uations. "any times, hoe$er, these
e2uations pro$e either $ery difficult or $ery time consuming to sol$e analytically, so e turn to
numerical methods for 2uic!, approximate solutions.
Application of numerical methods to differential e2uations can $ary in difficulty. Dust as some
differential e2uations are hard to sol$e *and some solutions are hard to compute e$en ith an
analytical solution1/, e often encounter difficulty in numerics as ell. Different criteria such as the
#ipschit? Condition or the Courant-%reidrichs-#ey Condition exist as guides to the using these
methods for solutions, and extensi$e use of numerical methods re2uires at the $ery least a &asic
understanding of these. 4e on)t discuss them in detail here 3 e trust that if you)re using these
methods for really specific or! that you)$e ta!en it upon yourself to learn hen to use hat. It)s
easy to get models to pro$ide an anser. It)s less easy to ma!e sure that anser is accurate.
The most common, simplest form of ,D' e as engineers generally encounter are initial $alue
pro&lems *IA0)s/. They are ,D')s for hich e !no starting $alues. %or example, if e ant to
descri&e the acceleration of a car from rest, e ould li!ely descri&e it as an IA0 &ecause e !no
its initial properties *position, speed and acceleration are at ?ero to start ith/. The methods here
all attempt to approximate a solution to the &asic e2uation
y' (t ) = f (t , y(t )) , f (t
0
) = y
0
As e)ll see later, this single e2uation can translate to higher order e2uations *such as our car)s
acceleration, or current in an -#C circuit, or motion of a mass-spring-damper system, etc./ &y a
simple extension of the or! in this chapter, hich e)ll discuss. #et)s start ith the most &asic
method for differential e2uations, 'uler)s "ethod.
!.1 - $uler(s Method
'uler)s method ta!es a $ery simple approach to approximating solutions for IA0)s. This method is
idely regarded as too simplistic and not accurate enough for most serious uses, &ut it ma!es for
easy practice for some $ery similar functions that are really accurate.
4e)ll s!ip some of the theory, &ut a long time ago, a Siss mathematician #eonhard 'uler
*pronounced )oyler)/ performed a lot of calculation and analysis in numerical integration to
produce hat are no !non as the 'uler Approximations. Among these is one nota&le method
that can &e used for a simple numerical solution of a &asic IA0. 'ssentially, his approximation
or!s as follos9 if e separate the domain e)re interested in into small time steps, using the
information at the first time step and our unsolved differential e2uation, e can ma!e a guess at
hat the $alue ill &e at the next iteration. %or small enough time steps, e can continue this o$er
the hole domain and get a reasona&le approximation of the actual solution.
#et)s start ith a &asic differential e2uation 3 .eton)s #a of Cooling says that my cup of coffee
ill cool more 2uic!ly hen the difference &eteen its temperature and the air is large, and less
2uic!ly as the temperature approaches the am&ient air temperature. In other ords, the change in
temperature per unit time is proportional &y a constant, !, to the difference &eteen its current
temperature and the air temperature. Assuming the air temperature remains constant, e can
translate this into e2uations as follos9
dT
dt
= k (T
air
T )
T
0
=160 F | , T
air
=75 F| , k = 0.1

1
min
|
'uler)s iterati$e function loo!s li!e this9
y
n+1
= y
n
+ hf (t
n
, y
n
)
So, our first couple of iterations using 'uler)s method to pro$ide an estimate ould loo! li!e this9
T
1
= T
0
+ hk (T
air
T
0
)
T
2
= T
1
+ hk (T
air
T
1
)
here h is the length of our time step. 'ssentially, e dra a straight line from our initial point at
*t
<
, T
<
/ to *t
<
+ h, T
6
/ ith the slope implicitly gi$en in the initial condition &y e$aluating our first-
order deri$ati$e *dT(dt/ using our initial condition. 4e can see right here, this loo!s li!e a pretty
simple &it of code, and in reality 3 it is. ,ur script ould loo! li!e this9
temp%1' = 1>#; % initial coffee temp3 C#
tair = =!; % air temp Cair
B = .1; % coolin5 coefficient
h = .1; % time step @1/minA
t = @#KhK>#A; % one hour
for i = 1Klen5th%t'-1 % we7re 5oin5 one e-tra step with temp
temp%i.1' = temp%i' . h0 B0%tair-temp%i'';
end
The next to last line of code *the only thing e loop/ is identical to 'uler)s iterati$e function a&o$e.
#et)s run this and plot the results9
--) plot%t3 temp'
That)s it1 4e ha$e a numerical solution. 4e ha$e a nice smooth cur$e that ma!es sense 3 as the
coffee gets closer and closer to room temperature, it cools more sloly, and our initial condition
of 6>< is met. ,ne pro&lem though 3 run this four or fi$e times ith different $alues for h, and plot
the results. Sou might get something that loo!s li!e this9
As e ma!e our time step &igger, our solution mo$es aay from the actual solution *I)$e plotted
the exact analytical solution here 3 it)s the one most toards the top/. ,ur first guess at h *>
seconds/ is small enough for the time scale e)re loo!ing at 3 this particular differential e2uation
isn)t too sensiti$e. ,thers, though, can &e really sensiti$e, and so 'uler)s method doesn)t or! as
ell. Thus, in practice, e)ll use a &it more sophisticated method, the -unge-Butta.
!.2 - Run&e-.utta Methods
This is the real deal. -unge-Butta "ethods are &y far the most commonly used methods in most
engineering applications today. They ere de$eloped around 6<< years ago *relati$ely ne in
terms of math history 3 .eton as 6G
th
century, 'uler as early 6;
th
century/, and are an
extension of the same math 'uler de$eloped. If you loo! at 'uler)s or!, he seemed to lo$e
applying Taylor series to all sorts of different pro&lems. In fact, that)s ho he came up ith the
method e 5ust discussed 3 it)s a first-order Taylor polynomial expansion, so its accuracy is limited,
and if the deri$ati$es of the function don)t &eha$e nicely, e can get some pretty serious error.
-unge-Butta "ethods include additional calculation of slope in the middle of each time step, and
ta!e a eighted a$erage of the $alues to e$aluate the function. This helps reduce our error as e
go from time step to time step, and can result in some $ery accurate results. These methods are
named &y ho many terms they use9 an -B: *:-term model/ is $ery &asic, and e2ui$alent to the
"idpoint "ethod, hich e didn)t &other to discuss since e)re doing it here. An -B= has four
terms, and is generally the most accurate per computation time. %ree"at)s *and "AT#AB)s/
,D'=8 routine sitches &eteen an -B= and -B8 &ased on hich is pro$iding a &etter result,
hence the name ,D'=8. #et)s start &y loo!ing at the -B:. -unge-Butta methods or! &y se$eral
e$aluations of our ,D' at different points, then a$eraging those $alues9
k
1, n
= h f (t
n
, y
n
)
k
2, n
= h f
(
t
n
+
h
2
, y
n
+
k
1, n
2
)
y
n+1
= y
n
+
(
k
1
+k
2
2
)
!
6
is 5ust 'uler)s "ethod. !
:
is 'uler)s "ethod again, &ut e$aluating our function at the midpoint,
using our pre$ious calculation of y in !
6
. Then, e a$erage these $alues to get our *more accurate/
estimation for y
n+6
. #et)s al! through a 2uic! example. 4e 5ust did a .eton)s #a of Cooling
pro&lem ith my cup of coffee in 'uler)s "ethodN let)s do it again here9
dT
dt
= k (T
air
T )
T
0
=160 F | , T
air
=75 F| , k = 0.1

1
min
|
-emem&er that our function, f*t,y/ is the right hand side of our differential e2uation. So, our first
iteration loo!s li!e this *ta!ing h Q <.6/9
k
1
= hk (T
air
T
0
) = 0.10.1(75 160) = 0.85
k
2
= hk(T
air
T
1
+
k
1
2
|) = 0.841
y
n+1
= y
n
+
(
k
1
+k
2
2
)
= 160 0.846 = 159.154
4e 5ust ta!e our !
6
$alue, plug it into our !
:
formula, and a$erage the to for the total
approximation y
n+6
. .otice that our final temperature change is a little different from hat 'uler)s
"ethod *!
6
/ gi$es us. As e increase h *ma!e our time step &igger/, this difference generally
increases. 4e ould also see &igger differences for more difficult functions, e$en hen using a
small h. Anyay, let)s rite a script1
temp%1' = 1>#; % initial coffee temp3 C#
tair = =!; % air temp Cair
B = .1; % coolin5 coefficient
h = .1; % time step @1/minA
t = @#KhK>#A; % one hour
for i = 1Klen5th%t'-1 % we7re 5oin5 one e-tra step with temp
B1 = h0B0%tair - temp%i'';
B = h0B0%tair - %temp%i'.%B1/''';
temp%i.1' = temp%i' . .!0%B1.B';
end
0lot your results, and you)ll see a cur$e nearly exactly li!e e sa in 'uler)s "ethod. #oo!ing at
our code, e can see that it)s $ery similar to the setup for 'uler)s "ethod, &ut e calculate
temp*i+6/ in a different ay. Also, e ha$e to call our function f*t,y/ se$eral times each iteration.
As e mo$e to higher-order -B methods, it ill &e easier to name a function and call it, instead of
riting it out for each ! $alue. #et)s mo$e on to an -B=, and a more sophisticated example.
The -B= or!s exactly li!e the -B: except for to points 3 there are more ! terms, and e eight
our a$erage, traditionally in the middle. Technically, you can eight these hoe$er you li!e, &ut for
the most part, e use the eighting shon here. Kere)s our iterati$e function9
k
1
= hf (t
n
, y
n
)
k
2
= hf (t
n
+
h
2
, y
n
+
k
1
2
)
k
3
= hf (t
n
+
h
2
, y
n
+
k
2
2
)
k
4
= h f (t
n
+h , y
n
+k
3
)
y
n+1
= y
n
+
(
k
1
+2 k
2
+2 k
3
+k
4
6
)
.otice that the first to terms are exactly the same as the -B:. The third term !
7
is calculated
exactly the same ay as !
:
, &ut ith !
:
as our y-$alue instead of !
6
. This is 5ust a refinement method
for !
:
)s $alue. !
=
e$aluates y at *t+h/ using !
7
)s approximation for y, and then e ta!e a eighted
a$erage here the middle $alues are more eighted than the ends. This is shon in %igure MX.
4e can 2uic!ly implement this again in our script from &efore, 5ust as a demonstration9
-igure :( &; values for guess at single time ste. %he black line is the
actual solution. <ote that a weighted average would be near k
4
and k
1
,
therefore it would have some error. %his is due to a large time ste '=.4
seconds)
temp%1' = 1>#; % initial coffee temp3 C#
tair = =!; % air temp Cair
B = .1; % coolin5 coefficient
h = .1; % time step @1/minA
t = @#KhK>#A; % one hour
for i = 1Klen5th%t'-1 % we7re 5oin5 one e-tra step with temp
B1 = h0B0%tair - temp%i'';
B = h0B0%tair - %temp%i'.%B1/''';
B3 = h0B0%tair - %temp%i'.%B/''';
B1 = h0B0%tair - %temp%i'.B'';
temp%i.1' = temp%i' . %1/>'0%B1.0B.0B3.B1';
end
-eally 2uic!ly, let)s loo! at error. The exact solution of this e2uation is pretty easy to come &y9
T (t ) = 85e
(kt )
+75
4e can run our three scripts, eulerex.m r!:.m and r!=.m, and put their $alues up against the real
solution for a comparison. I 5ust too! h Q 6, and pic!ed a fe points o$er the domain9
.otice 'uler isn)t $ery accurate compared to the -B:, and li!eise on to the -B=. Kere)s a ta&le of
absolute error *error that doesn)t ta!e into consideration the si?e of the $alues it)s comparing9
These num&ers ill shrin! significantly if e ta!e h to &e smaller, &ut e don)t ant to aste time
and memory. The lesson to ta!e aay here is that the more sophisticated model gi$es you *in
most cases/ a more accurate result. In a stiffer or less simple e2uation, this difference ould &e
greatly magnified. .o let)s ta!e a loo! at a more ad$anced program ith some real teeth.
function ret = rB1%f3t3f#';

,%1' = f#;

for i = 1Klen5th%t'-1
$uler R.2 R.) $/act
160.0000 160.0000 160.0000 160.0000
151.5000 151.7125 151.9108 151.9112
107.9307 108.7632 109.5570 109.5584
86.4822 87.1036 87.7122 87.7133
Error [F]
$uler R.2 R.)
0.0000 0.0000 0.0000
0.4112 0.1987 0.0003
1.6277 0.7952 0.0014
1.2311 0.6097 0.0011
h = t%i.1' 4 t%i';

%--- sol&es Jun5e-Putta P &alues ---%
B1 = h0f%t%i'3,%i'';
B = h0f%t%i'..!0h3,%i'..!0B1';
B3 = h0f%t%i'..!0h3,%i'..!0B';
B1 = h0f%t%i'.h3,%i'.B3';

%--- sol&es for , from Jun5e-Putta coefficients ---%
,%i.1' = ,%i' . %1/>'0%B1.0B.0B3.B1';
end

plot%t3,'; % optional plot3 comment out if ,ou don7t want this
ret = @t7 ,7A;

end
.otice that this program is the same length as our pre$ious script, &ut it)s much more poerful.
Osing this program, e can pass any function using an anonymous function *the f argument here/
from the console, along ith a ro $ector of time $alues *t/, and our initial condition *f</. K is
calculated &y loo!ing at the distance &eteen to time $alues at each iteration, hich means e
can alter this &ased on ho our function loo!s 3 our domain doesn)t ha$e to &e e2ually spaced1
4e can certainly pass a monotonic $ector, &ut for functions ith long flat &oring spots, e can
ma!e our time step much &igger there to speed up calculation. If you don)t ant this feature, you
can simply mo$e h outside the loop and fix it at t*:/ 3 t*6/. 4e can $erify that this thing or!s &y
using our test function again9
--) t = @#KhK>#A; % one hour
--) p = 9%t3,' %#.10%=!-,''; %our function 4 t is time3 , is
temperature
--) p# = 1>#; % our initial condition
--) rB1%p3t3p#'
4e get the exact same $alues e had &efore ith our -B= script, and a plot to match1 A couple of
notes here9 hether our function is a function of &oth t and y or not, e need to define it as such.
This test function doesn)t really care a&out t, 5ust y *hich is temperature, T in our case/ 3
remem&er that t is included imlicitly &y using h in the calculation of each $alue for y. 4hen our
r!= program calls the function though, it ants to pass to arguments, time and temperature. 4e
can pass fewer arguments than are defined in a function, &ut not more. Therefore, e defined p Q
T*t,y/ instead of 5ust p Q T*y/. The t $alue doesn)t do anything in our function, &ut r!= doesn)t
really !no or care.
!.! - Adams-Bashforth Methods
!.) - "#stems of Differential $%uations
"ection )0 Boundar# 1alue 2roblems
).1 - 3inear "hootin& Method
The linear shooting method is a relati$ely simple approach to sol$ing a &oundary $alue pro&lem. In
general, e !no conditions at to &oundaries, resulting in a more complex system than a simple
-unge-Butta can sol$e 3 e need to ta!e into account not only the initial $alue of the solution, &ut
$alues at later points as ell. A physical e2ui$alent of a &oundary $alue pro&lem could &e
throing a &ase&all to a catcher, as shon in %igure MX. The throer ad5usts the speed of the &all
and the angle at hich it)s thron to ma!e the &all land here they ant. In fact, the human &rain
is possi&ly one of the fastest sol$ers for &oundary $alue pro&lems, in that it can sol$e these
systems in an analog fashion in milliseconds, hether it)s mo$ing one)s hand to catch a &all, or
manually landing an aircraft on an aircraft carrier dec!.
%or our purposes here, e)ll generally consider functions of the form9
y' ( x) = f ( x , y( x)) , f ( x
0
) = y
0,
f ( x
1
) = y
1
Kere, x
<
is the initial point and x
6
is the final point on the independent axis *usually denoting a
length or displacement, depending on the pro&lem/. Continuing the example of a pitcher throing
a &all, he or she !nos se$eral !ey pieces of information &efore throing the &all9 the initial height
of the &all *the position of the pitcher)s hand at release, x
<
/, the final height of the catcher)s glo$e
*x
6
/, and the general &eha$ior of the &all in flight *f Ex,yExFF/. 4ith this information, the pitcher
ma!es a guess at the initial $alues for slope and &all $elocity needed to hit the catcher)s glo$e.
Shooting methods simply sol$e these pro&lems the same ay a pitcher ould9 guess the initial
conditions you need to hit the final &oundary ith good enough precision. 0racticed pitchers are
5ust good at that initial guess 3 a product of muscle memory and lots of practice. 4ith the shooting
method, e simply ta!e a guess and use that guess to sol$e an IA0 *hich is usually pretty simple,
as e sa in the last chapter/. 4e ta!e a loo! at the $alue on the final &oundary, and ad5ust our
guess accordingly. This process is repeated until e get ithin an accepta&le error at the opposing
&oundary, and e)re left ith an accurate guess at a correct initial condition as ell as the
solution.
).2 - "hootin& Method for Nonlinear 2roblems
).! - Finite-Difference Methods
Section ': "artial Differential Equations and t(e Finite Element )et(od