Linear Algebra

CHAPTER
One
Computational Linear Algebra Using C0 Type Objects
Cn spaces, as in usual mathematical definition, are continuous (linear) vector spaces with derivatives up to order n. This chapter deals with objects in C0 space.
1.1 C0 Type Objects

In VectorSpace C++ Library, objects in C0 space is represented by C0 type. C0 type contains many fundamental objects in numerical computation. We subdivide them into primary and utility objects. The primary objects include Scalar, Vector and Matrix, which are fundamental objects in linear algebra. The utility objects include Subvector, Submatrix and Basis. In engineering practice, Subvector and Submatrix are very popular in giving out explicit formula. In linear algebra, the expression can be written in Basis. The applications in this chapter focus on matrix algebra, while for each application we have the choice of using either (1) Subvector/Submatrix or (2) Basis expression to achieve the equivalent mathematical expressions in C++ program. VectorSpace C++ Library has many extended definitions that are not in ANSI/ISO C++. A simple program using VectorSpace C++ Library is shown in Program Listing 11. The first line is to include a header file vs.h. This file is in the sub-directory vs\include.1 The programs in this chapter are all in project workspace file C0.dsw under directory vs\ex\C0.
1. The include file enables C++ complier to understand the extended definitions in the VectorSpace C++ Library.
You also need to use the VectorSpace C++ Library vs.lib under directory vs\lib, for the linker to resolve the external references of your program to the library in order to construct an executable file.
Workbook of Applications in VectorSpace C++ Library 1
Chapter
int main() { C0 a(0.0);
1 Computational Linear Algebra Using C0 Type Objects
#include include\vs.h
cout << a << endl; C0 b = SCALAR(const double&, 1.0), c =C_0 (const double&, 2.0); cout << b << endl << c << endl; try (C0 d = SCALAR(wrong string, 0.0); } catch(const xmsg& e) { cout << Excetption: << e.what() << at << e.where() << line << e.line() << endl; } }
dedicated constructor; value = 0.0 virtual constructors; the string severs as a memonic for the parameter that is supplied to it. exception handling
Listing 11 Scalar object constructor (project: scalar_examples)
1.1.1 Scalar
The concept of data abstraction in C++ organizes data and the operations on the data in a coherent unit class. The class of a Scalar defines the simplest data abstraction in VectorSpace C++ Library. A Scalar is a class with a number s as its private member data, represented as double type in C++, associated with its constructors, operators and member functions (see Figure 11). The private member data, in this case a double type, is shielded by the operators and the member functions through which the access from the outside world to the private member data is only possible. In mathematics, a group, with a set G and operator , is denoted as (G , ). Here a class in C++ defined with the concept of data abstraction closely resembles the concept of a group in mathematics.
... double + sin() = ...
operator=()
Access from outside world only possible through member functions and operators
Figure 11 Data abstraction organizes data and its operators and member functions in a coherent
Constructors
A scalar object in C++ program is declared as a C0 type with either dedicated constructor or virtual constructor as in Program Listing 11. The advantage of using a dedicated constructor is that it is very concise. It only specifies a double value as the argument for the C0 constructor. C0 constructor knows the result is a Scalar object by identifying that there is only one argument of type double supplied to the C0 constructor. In the Pro2
Workbook of Applications in VectorSpace C++ Library
C0 Type Objects
gram Listing 11 (the project: scalar_examples in porject workspace file C0.dsw), a double value 0.0 is used to initialized the value of the Scalar object of C0 type as C0 a(0.0); Supplying arguments of different types to the dedicated constructor of C0 type may result in other kinds of objects, e.g., a Vector, a Matrix ... etc., in the C0 type family. The philosophy of the dedicated constructor is that it is in accordance with the style of the C language. The syntax of using dedicated constructor saves a few punches on the keyboard for programmers. Especially beneficial to professional programmers who work on computer so often that tedious acts can be very annoying. However, the flexibility of the dedicated constructor is compromised. What would happen, if we want to supply a pointer to a double to initialize the Scalar object? How do we instantiate a reference to a double value, a by-value from a Scalar object, a by-reference from a Scalar object, or a pointer to a Scalar object ... etc. ? To extend such flexibility, virtual constructor1 is used to instruct the C0 type constructor to know exactly what kind of object to generate, a Scalar, a Vector, or a Matrix, ... etc. This kind of constructor is called virtual because it mimics the function dispatching behavior of the virtual functiona salient objected-oriented programming feature in C++. Through the virtual function mechanism in C++, a call on the virtual function (foo() in Figure 12) in the base class can be dispatched to the same function in the derived class. Actually, critics of C++ often say that C++ is not an orthodox object-oriented language. The implementation of the virtual constructor, which is not supported by C++, steps in the direction closer to an orthodox object-oriented language with the use of VectorSpace C++ Library.
C0 call foo() virtual foo(); ...
Base class
inheritance
dispatch route Derived classes Scalar foo(); ... Vector foo(); ... Matrix foo(); ... etc.
Figure 12 Object-oriented class inheritant relationship and virtual function dispatching mechanism. A concrete object, for example a Scalar, is generated at run-time by the virtual constructor of its base class C0 as (see Program Listing 11) C0 b = SCALAR(const double&, 1.0);
1. J.O. Coplien, 1992, Advanced C++ Programming Styles and Idioms, Addison-Wesley, p. 143.
Chapter
The object, b, is of type C0. C0 type constructor generates a Scalar object, by explicitly specifying SCALAR. The first argument supplied is a constant stringconst double&, with & meaning that it is a reference, which in C++ is used to improve the efficiency of argument passing, and const to mean that the content can not be changed for data integrity. The second argument is a double constant1.0, consistent with the semantics of the string in the first argument. The purpose of the string is not only for the constructor to identify the kinds of arguments to be passed following it, but it also serves to increase readability of the program. Many different types of arguments are possible to be supplied to for the Scalar object. This set of virtual constructor strings furnishes the flexibility for the virtual constructor. virtual constructor string by reference C0& C0* double& double* by value const double& const double* const C0& const C0*
Strings in C0 virtual constructor for Scalar object.
VectorSpace C++ library definition
priority
C0 type Scalar object a pointer to C0 type Scalar object double double pointer
1 2 3 4
double double pointer C0 type Scalar object pointer to C0 type Scalar object
5 6 7 8
Some of the string contains more than one word. In VectorSpace C++ library, the string for the constructor are parsed in free-format. Free-format uses one or more spaces or commas to separate words. For example, two words separated by one space const^double& and two words separated by two spaces const^^double& will be recognized as the same by the virtual constructors. Instead of explicitly specifying SCALAR for the C0 type constructor, one can also use C_0 in place of the SCALAR to construct a Scalar object. For example (see also Program Listing 11), C0 b = C_0(const double&, 2.0); In such a case, the C0 type constructor searches for a matching string according to the priority ranking of the strings (see Figure 12, searching in the order of left to right through the branches of object-hierarchy tree). When the constructor finds the first match it generates a corresponding object, a Scalar, a Vector, a Matrix ... etc. A specification of C_0 instead of any specific type of object being committed in the time of program writing, is a late-binding technique used in the VectorSpace C++ Library. The determination of the actual kind of object to generate can be delayed until run-time. You can code logic control statements in your program to determine what kind of object to generate, then pass an appropriate string to the constructor. Therefore, the object and its type are both created on the fly. In VectorSpace C++ Library, this particular kind of virtual constructor is called autonomous virtual constructor1. However, the power of flexibility comes as a frontal assault on the security of
1. see autonomous generic exemplar idiom in J.O. Coplien, 1992, Advanced C++Programming Styles and Idioms, Addison-Wesley, p. 291.
C0 Type Objects
C++ as a type-compiled language. The programmers own discretion is advised for the use of the virtual constructor. We have provided with many kinds of constructorsdedicated constructor, virtual constructor, and the most dynamic autonomous virtual constructor. This does not mean that you have to use them all in order to have your program up and running. You can use dedicated constructor exclusively to maintain the strong type-complied C++ language tradition for the sake of safety. The additional virtual constructors are provided to unleash the power of the object-oriented programming. The limitation is only a programmers imagination not the C++ preference on not fully committed to the object-oriented paradigm. In complement, the exception handling can be used to tame potentially rampant situations. Exception handling can be used as (see Program Listing 11) try (C0 d = SCALAR(wrong string, 0.0); // cause an exception to be thrown catch(const xmsg& e) { cout << Exception: << e.what() << at << e.where() << line << e.line() << endl; } The try-catch statement is standard in C++ language. The xmsg object simulates the standard C++ library exception handling1. The details and location (function name and line) of an error, if any, caused by the C0 constructor enclosed in try clause will be reported to the standard output in the above code segment.
Operators and Functions

There are three kinds of operators in VectorSpace C++ library. The symbolic operator, arithmetic operator, and logic operator. Symbolic operator in Scalar object has two kinds of assignment operators. The assignment by reference operator &= and assignment by value operator = (see Program Listing 12).
#include include/vs.h int main() { C0 a(0.0), b, c(1.0), d; cout << a << endl; b &= a; c =a; a = 10.0; cout << b << endl << c << endl; a = c; d = a; }
a = 0.0; b not initialized c = 1.0; d not initialized 0.0 assignment by reference assignment by value reset to 10.0 10.0 0.0 garbage collection behind the scene make a new Scalar object
Listing 12 Assignment by reference and assignment by value (project: scalar_examples).
1. P.J. Plauger, 1995, The draft standard C++ library, Prentice Hall, Inc., p.53.
Chapter
1 Computational Linear Algebra Using C0 Type Objects a

b &= a;
Scalar label double
a = 10.0;
0.0
10.0
c 1.0 =
c = a;
Figure 13 Assignment by reference (upper part) and by value (lower part). In Program Listing 12 (see also Figure 13 ), variable b is declared as an object of type C0. The variable b actually acts more like a label (or a symbol), because it has no concrete data type, e.g., a Scalar, a Vector or a Matrix, associated with it. The label b is then assigned to share the same concrete data, the Scalar object with a as its label, and 0.0 as its content. This is done by the assignment by reference operator as (illustrated in the upper part of Figure 13) b &= a; We can think this expression as an operation to attach the label b to the Scalar object that has already been labeled as a. Therefore, changes made to the content of a to 10.0, by a = 10.0; later, will also change the content of b, because a and b are referring to the same memory location. The assignment by value (see the lower part of Figure 13) in Program Listing 12 is c = a; In this case, c already has its own copy of a Scalar object with its content initialized as 1.0. The assignment by value operation then reassigns its value to that of a object0.0. Later on, changing a=10.0; does not change the value of c. The value of c remains as 0.0. What then will happen if we write (see also upper part of the Figure 13) a &= c The result is that the label a will be peeled off from the Scalar object that it is attached to. Then, some housecleaning chore needed to be done. The VectorSpace C++ Library will check if the detached Scalar object has any other label refers to it. If there is, in this case the label b is still referring to it, the Scalar object will survive. If there isnt, the Scalar object will be killed. This is done by reference-counting1, a popular garbage collection technique for memory management in C++. The next step, after garbage collection, is that the label a is then
1. J.O. Coplien, 1992, Advanced C++Programming Styles and Idioms, Addison-Wesley, p. 58.
C0 Type Objects
a &= c;
a b
c 0.0
10.0 d ?=
Figure 14 Assignment by reference (upper part) and by value (lower part).
d = a;
reassigned to the Scalar object that label c is pointing to. In short, a is striped off its original associated Scalar object and re-assigned to point to the Scalar object that c is appointed to. A contrary scenario is the case of assigning a label d by value as d = a; In this case (see the lower part of Figure 13), d is only a label, with no concrete object associated with it. Upon the assignment by value, label d will be instantiated with a concrete object with the same type of object as a, a Scalar object in this case. Then this newly instantiated Scalar object will be assigned a value of 0.0. In short, a newly constructed Scalar object (since c pointed to a Scalar object) is created for label d, and its value is also set to the Scalar object associated with a. That is both the object type and its content of label d is determined according to what label a is referring to. Two symbolic operators & and && are column-wise-concatenation operators. Their return values are Vector objects. The operator | and || are row-wise-concatenation operators. They return row-vectors which are represented as Matrix of row-length = 1 in VectorSpace C++ Library. We defer discussion of these operators until Vector and Matrix are introduced. The use of member arithmetic operators, logic operators and functions for the Scalar object are straight-forward. They are defined to be consistent with C++ without much explanations (see box in page 8). The actual set of operators and functions is many times greater than the partial listing here. Many operators and functions not listed here are actually proliferation due to the promotion among different types of a binary operator. For example, (in project: scalar_examples) C0 a(1.0); C0 b = a + 2.0; C0 c = 2.0 + a;
// a a Scalar object of type C0 plus a constant double 2.0 // 2.0 a constant double plus a Scalar object of type C0
Chapter
That is all the double(s) in the above have been promoted, by VectorSpace C++ Library not by C++ per se, to a Scalar object of C0 type before they are added to the other Scalar object of C0 type. Many other transcendental functions can be expressed using this small set of functions. However the function pow(int) takes only integer argument. It is illegal, for example, to write a.pow(-2/3) to express a-2 / 3, because -2/3 is a fractional argument. However, one can express it with the available functions exp() and log(), as exp(log(a)*(-2/3)). operator or function symbolic operators C0& operator &= ( ) C0& operator = ( ) C0 operator & ( ) const C0 operator && () const C0 operator | ( ) const C0 operator || () const arithmetic operators C0 operator + ( ) const C0 operator - ( ) const C0 operator + (const C0&) const C0 operator - (const C0&) const C0 operator * (const C0&) const C0 operator / (const C0&) const C0& operator += (const C0&) C0& operator -= (const C0&) C0& operator *= (const C0&) C0& operator /= (const C0&) logic operators int operator == (const C0&) const int operator != (const C0&) const int operator >= (const C0&) const int operator <= (const C0&) const int operator > (const C0&) const int operator < (const C0&) const functions C0 pow(int) const C0 sqrt(const C0&) const C0 exp(const C0&) const C0 log(const C0&) const C0 sin(const C0&) const C0 cos(const C0&) const VectorSpace C++ library definition remark
assignment by reference assignment by value column concatenation one-by-one column concatenation row concatenation one-by-one row concatenation
positive negative addition subtraction multiplication multiplication replacement addition replacement subtraction replacement multiplication replacement division
unary unary
equal not equal greater or equal less or equal greater less
TRUE == 1 FALSE == 0
power square root exponent log sin cos
Partial listing of scalar object arithmetic operators, logic operators and functions.
C0 Type Objects
1.1.2 Vector
The design of data abstraction for a Vector is represented as two private data members (length, vi) where length is an integer numberlength I, and an array of real number vi , with 0 i < length. In C++ a variable size array can not be declared as an array of double. Therefore, vi is actually represented by an array of pointers to double. Please notice that the differences of array of pointers to double and an array of double. Following traditions of C language, C++ makes pointer syntax looks like array from users perspective through the so-called pointer-arithmetic. There are situations where subtle differences exist, and from compiler writers perspective these two are in fact completely different animals.1 The internal implementation of VectorSpace C++ Library uses array of pointers to double exclusively. In VectorSpace C++ Library a vector is always viewed as a column-wise vector, while the row-wise vector, when the distinction is necessary, is represented as a matrix of row-length = 1. Before we go any further on the introduction of Vector, we re-visit two column-wise-concatenation operators left un-explained on page 7. The column-wise-concatenation operator & is used for two Scalars as (project: vector_examples) C0 a(0.0), b(1.0); C0 c = a & b; cout << c << endl; // two Scalars of type C0 // concatenation of two Scalars return a Vector of length =2. // {0.0, 1.0}T
The return value of c is a Vector object of C0 type with its value as {0.0, 1.0}T, column-vector is denoted with transpose superscript. Using && makes no difference in the case with two Scalar objects as operands. However, if Vector is used as either of the two operands of the binary operators & and &&, we will see different results from these two operators (see Figure 16 in page 14).
Constructors
Now lets get on with Vector in VectorSpace C++ Library. The dedicated constructor for the Vector can be written as (see Program Listing 12)
#include include\vs.h int main() { double a[3] = {0.0, 1.0, 2.0}; C0 b( 3, a); cout << b << endl; C0 c(3, (double*)0); cout << c << endl; return 0; }
array name a is treated as double* {0.0, 1.0, 2.0}T by reference null pointer 0 is cast as double* {0.0, 0.0, 0.0}T, with default value and its own memory.
Listing 13 Dedicated constructor of a Vector object (project: vector_examples).
1. Chapters 4 and 9 in P. V. Linden, 1994, Expert C programming: deep C secrets, Prentice-Hall Inc.
Chapter

// array name a is cast as double* by C++ // {0.0, 1.0, 2.0}T
double a[3] = {0.0, 1.0, 2.0}; C0 b( 3, a); cout << b << endl;
In the second line of the above codes, the array name a is treated as a double*, and passed, as a reference, to the dedicated constructor of C0 type for getting a Vector object. Changes on the values of double array a will change the content of Vector object pointed to by b. From this example, it is clear that the dedicated constructor is obtained by being given two arguments to the C0 type constructor, with the first one as an int type and the second one as a pointer to double type. We can avoid declaring array a at all, if the dedicated constructor is not going to refer to an outside memory space as (see Program Listing 12) C0 c( 3, (double*)0); cout << c << endl; // null pointer 0 is cast as double* // {0.0, 0.0, 0.0}T
In this case, a Vector c will have its own copy of memory space and its values are all initialized to a default value0.0. This is the most concise way of initializing a Vector object. In VectorSpace C++ we call this specific style as default dedicated constructor. The default value can be easily reset to other values with a statement as c = 2.0. In this case all three elements of the Vector c will be set to have the value of 2.0. This says that the assignment by value operator takes a double value as its argument. Such implicit type conversion happens, in VectorSpace C++ Library, only when it makes unambiguous intuitive sense. The dedicated constructor can be used to make a new kind of object that we have not introduced in section 1.1.1 about Scalar. This new kind of object is a subvector which refers to an existing Vector object. The subvector can start from and end at any index within the index range of the referenced Vector object a, provided that Vector a has to be continuous in its physical memory space. For example, double d[8] = {0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0}; C0 a(8, d), b(4, a, 3); cout << (+b) << endl; In Figure 15, Vector a is constructed as a vector of length 8 by C0 a(8, d); where double array d is declared as double d[8] = {0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0};. A reference Vector b is initialized to have its length 4, the referenced object a, and the index to start from a being 3 by writing the following statement. C0 b(4, a, 3); We consider the reference Vector as a special kind of Vector object instead of a Subvector. The word Subvector (with capital S) in VectorSpace C++ Library will be reserved for subvectors that can have its referenced vector equal-partitioned as will be discussed in Section 1.1.5. A reference Vector, as its name implies, always references to another Vector object with continuous physical memory space. It doesnt own the memory space of its data. A common practice in VectorSpace C++ Library is to use the unary positive operator + to cast a reference Vector into a new independent Vector object such as +b. A temporary Vector object will be generated in this case to have its own copy of memory with the same
10
C0 Type Objects
double d[8] = 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 0 1 2 3 4 5 6 7 C0 b(4, a, 3); 0 1 2 3 3.0 4.0 5.0 6.0 +(b) 0 1 2 3 C0 a(8, d);
Figure 15 Referenced Vector b, and primary casting by unary operator +(). length (= 4), and with the contents of the memory to be set to have the same values as b. The use of the unary positive operator + to convert a specialized object, in this case a reference Vector, into a more primitive object will be encountered many times in VectorSpace C++ Library. This operation is defined as primary casting, VectorSpace C++ Library, by the operator +. Similar to the case for Scalar, the constant strings used for the plain virtual constructors (used macro definition VECTOR in place of SCALAR) and the autonomous virtual constructor are shown in the following box Some of the constant strings do not have a priority number. That is because those strings clash with the strings in the Scalar. The C0 type autonomous virtual constructor will first find a match in Scalar object; it will never have a chance to be dispatched up to Vector object virtual constructor to make it. This means the strings in Vector are hidden (or masked) by the same strings in Scalar. Therefore, these strings although useful for the plain virtual constructor will not be useful for the autonomous virtual constructor to make any Vector object of C0 type. virtual constructor string by reference C0& C0* int, double* int, double*, int, int by value int int, double* int, const double* int, const C0* const C0* int, C0&, int
Strings in C0 virtual constructor for Vector object.
VectorSpace C++ library definition
priority
C0 type Vector a pointer to C0 type Vector length, double* != 0 length, double*, m_row_size, m_col_size
10 11
length length, double* = 0 length, double* length, C0* of a Scalar C0* length, C0, starting index (the only one for reference Vector)
9 10 12 13 14
11
Chapter
Priority number 10 appears both in the by reference and by value categories because they can be distinguished by passing a pointer to double or a casted double* of a null pointer. The semantics of constructing an object with default value by the virtual constructors is exactly the same here as that in the case for dedicated constructors. The last string int, C0&, int is for constructing a reference Vector. The argument passing of the virtual constructor for the reference Vector matches exactly with that of the dedicated constructors.

The symbolic operators for Vector have two assignment operators, one selectoroperator [] (int), and four concatenation operators (see box in page 13). The assignment operators, if using only Vector objects as operands, will have exactly the same behavior as we have introduced in Program Listing 12 for the case of two Scalars. The complexity begins to fold on when we assign a Vector object with different type objects as in the following (see Program Listing 12.)
#include include/vs.h int main() { C0 a(1.0), b(3, (double*)0); b = a; cout << b << endl; b = 2.0; cout << b << endl; b &= a; cout << b << endl; return 0; }
a = 0.0; a Scalar b = {0.0, 0.0, 0.0}T; a Vector assignment by value with a Scalar {1.0, 1.0, 1.0}T assignment by value with a double {2.0, 2.0, 2.0}T assignment by reference with a Scalar 1.0; b is now a Scalar!
Listing 14 Assignment operators with different types of operand.
Assigning Vector b by value with either a Scalar object of C0 type or a double type in C++ is defined as setting all of the components of b to have the value of the Scalar or the double. Assignment by reference for a variable b of type C0, in this case a Vector, by a Scalar results in a Scalar (as in b &= a). The type of object associated with label b has been changed from a Vector to a Scalar. The garbage collection mechanism explained in page 6 will be activated to check if the memory space for the Vector object needs to be released. The brand new symbolic operatorselector is the operator [](int). For example, in Program Listing 12, a is a Vector of length = 8, with its value refers to a double array d. b is a reference Vector that has length = 4, with the first index (0) of b pointing toward the fourth index (i.e., off-set3 from the first position) of a. Therefore, the selectors of a[3] and b[0] will both return a Scalar with the value 3.0, and both return Scalar objects pointing toward the same memory position (see Figure 15 in page 11). The remaining symbolic operators for the Vector objects are the column-wise concatenation operators & and &&, and row-wise concatenation operators | and ||. The row-wise concatenations will return a Matrix
12
C0 Type Objects
operator or function symbolic operators C0& operator &= ( ) C0& operator = ( ) C0& operator [] (int) C0 operator & ( ) const C0 operator && () const C0 operator | ( ) const C0 operator || ( ) const arithmetic operators C0 operator ~ ( ) const C0 operator + ( ) const C0 operator - ( ) const C0 operator + (const C0&) const C0 operator - (const C0&) const C0 operator * (const C0&) const C0 operator %(const C0&) const C0 operator / (const C0&) const C0& operator += (const C0&) C0& operator -= (const C0&) C0& operator *= (const C0&) C0& operator /= (const C0&) logic operators int operator == (const C0&) const int operator != (const C0&) const int operator >= (const C0&) const int operator <= (const C0&) const int operator > (const C0&) const int operator < (const C0&) const functions int length() const double norm(int = 2) const double norm(const char*) const C0 pow(int) const C0 sqrt(const C0&) const C0 exp(const C0&) const C0 log(const C0&) const C0 sin(const C0&) const C0 cos(const C0&) const VectorSpace C++ library definition remark
assignment by reference assignment by value selector column concatenation one-by-one column concatenation row concatenation one-by-one row concatenation
return a Scalar
return a Matrix return a Matrix
transposed (into a row vector) return a Matrix positive (primary casting) unary negative unary addition subtraction multiplication by a scalar; scalar product of two Vectors tensor product of two Vectors return a Matrix division (by a Scalar or a Matrix only) return a Vector replacement addition replacement subtraction replacement multiplication (by a Scalar only) replacement division (by a Scalar only)
length of the Vector 1-norm or 2-norm infinite-norm takes strings infinity, or maximum power (applied to each element of the Vector) square root (applied to each element of the Vector) exponent (applied to each element of the Vector) log (applied to each element of the Vector) sin (applied to each element of the Vector) cos (applied to each element of the Vector) Partial listing of Vector object arithmetic operators, logic operators and functions. Workbook of Applications in VectorSpace C++ Library
13
Chapter
#include include/vs.h int main() { double d[8] = {0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0}; C0 a(8, d), b(4, a, 3); cout << a[3] << endl; cout << b[0] << endl; return 0; }
a = {0.0, 1.0, ..., 7.0}T; a Vector b; a reference Vector a[3] = 3.0 b[0] = a[3] = 3.0
Listing 15 Selector for Vector and reference Vector objects (project: vector_examples). object, and we will discuss them in Section 1.1.3. The column-wise concatenation operations for two Scalar objects have been introduced in page 9 of this section. For simplicity we first focus on the case for column-wise concatenations of two Vector objects (see Figure 16). The column-wise-concatenation of two Vectors, by operator &(const C0&) appends the second vector after the first as in the left-hand side of Figure 16. The lengths of the two Vectors (say len1 and len2) do not have to be the same. The return Vector has the length of (len1+len2). The one-by-one column-wise-concatenation, by operator &&(const C0&), of two Vectors requires that the two Vectors to have the same length (= len) as shown in the right-hand-side of Figure 16. The values of the two Vectors are interlaced to form a new Vector of length = 2 len. If the two Vectors do not have the same length, an C++ exception will be thrown. If not handled by a catch clause, the default behavior of C++ exception handling mechanism will cause your program to crash. What if one of the operands for & or && operators is a double or a Scalar object? (see left-hand-side of Figure 17) For the column-concatenation operator &, the double or Scalar will be added in front or appended after the Vector object (with length = len) according to the order of the operands. In Figure 17, a and c are either a double or a Scalar. a & b adds the value 1.0 in front of the values of b to form a new Vector. b & c
1.0 a 2.0 3.0 4.0 b 5.0 6.0 7.0
1.0 2.0 3.0 4.0 5.0 6.0 7.0 a&b d c
1.0 3.0 5.0 7.0 2.0 4.0 6.0 8.0
1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 c && d
Figure 16 Column-concatenation and one-by-one column concatenation of two Vectors.
14
C0 Type Objects
d && e a 1.0 1.0 2.0 3.0 4.0 2.0 b 3.0 4.0 5.0 2.0 3.0 4.0 5.0 6.0 c 6.0 b&c 5.0 1.0 0.0 2.0 0.0 3.0 0.0 4.0 0.0 Figure 17 Column-concatenation & and one-by-one column concatenation && of one Scalar and one Vector. appends the value of 6.0 after the values of b as shown. Both of their return objects are new Vectors with length = len + 1. For the one-by-one column-wise-concatenation operator && with mixed-type operands, VectorSpace C++ library defines the return new Vector object to have the length = 2 len, with the value of the double or the Scalar interlaced in front or after the values of the original Vector object. In the right-hand-side of Figure 17, d and f are either a double or a Scalar. e && f and d && e interlace the value of f after and the value of d in front of the values of the Vector e, according to the order of operands for the && operator. The explanations on two operators, the transpose by ~ and the tensor product by %, will be delayed until the next section on the subject of Matrix, because they both return a Matrix object. The member operator * of a Vector object may take a Scalar or a double as its argument. For example, double a[4] = {1.0, 2.0, 3.0, 4.0}; C0 v(4, a), s(2.0); cout << (3.0*v) << endl; cout << (v*s) << endl; // (project: vector_examples) // {3.0, 6.0, 9.0, 12.0}T // {2.0, 4.0, 6.0, 8.0}T f 0.0 a&b e && f e 1.0 2.0 3.0 4.0 0.0 1.0 0.0 2.0 0.0 3.0 0.0 4.0
d 0.0
15
Chapter
in VectorSpace C++ Library this operation is defined to return a new Vector object with the value of every element of the Vector multiplied by the value of the Scalar or the double. Applying an operator or a function to every component of an object is consistent with the distribution rule. We say that the pre- or post- multiplication of a Vector with either a Scalar or a double obeys the distribution rule in VectorSpace C++ Library. When the operator * of a Vector object takes another Vector object, with the same length, as its argument, this operation is defined as the scalar inner product. double a[4] = {1.0, 2.0, 3.0, 4.0}, b[4] = {2.0, 4.0, 6.0, 8.0}; C0 v(4, a), w(4, b); cout << (v * w) << endl; // (project: vector_examples)
// 60
If the two Vectors do not have the same length, an exception will be thrown. Three different notations in mathematics for scalar inner product are
Indicial Notation
Matrix Algebra
Tensor Algebra
vi wi
vT w
vw
TABLE 11. Three popular mathematical notations for scalar inner product. In the indicial notation the scalar inner product is indicated by using repeated indices i. The repeated indices is defined to imply summationthe summation convention. In the matrix algebra notation, the scalar inner product is achieved by re-orienting the column-vector v into a row-vector vT, and the row vector multiplied by the column vector w of the same length gives a scalar result. This is consistent with a matrix of row-length = 1 multiplied by a column-vector gives a scalar. In these two notations the expressions for the scalar inner product are defined implicitly using the multiplication operation. The operands need to be tempered with by either adding repeated indices or imposing transpose to define the scalar inner product. In the tensor algebra, the two vectors do not need to be manipulated. The operator is defined as scalar inner product per se. The operator, in this case, is defined to have the knowledge of how each component of the two vectors should be multiplied and summed together. For the reason of inclusiveness for all three conventions, VectorSpace C++ Library defines the Vector::operator *(const C0&) as (~v) * w = v * w. Eq. 11
The left-hand-side fits in matrix algebra, and the right-hand-side fits in tensor algebra. The notations in tensor algebra have the most uncluttered expressions, in which physical meaning of an expression is usually less distracted by the trivia. The Vector::operator / (const C0&) accepts either a Scalar or a Matrix as its argument. When the argument is a Scalar, or a double as well, the operator obeys the distribution rule as in the multiplication operator; i.e., the division by the Scalar value is applied to every component of the Vector object. For the division operator to accept an argument of a Matrix, lets first look at the solution of a set of simultaneous equation in Matrix form as
Mv = w
Eq. 12
16
C0 Type Objects
v = M-1 w = w / M; Eq. 13
Where M is a matrix of size m n, v is a vector of size n, and w is a vector of size m. In this case, the division of a vector w by a Matrix M is naturally defined as the solution v of the simultaneous equation M v = w. Replacement multiplication ( *=) and division ( /=) operators have semantic issues to be clarified. The replacement operator means the l-valued object (the object in the left-hand-side) is to be operated on and then reassigned to itself. However, if the argument taken for *= is a Vector, the replacement multiplication will mean that it is a scalar inner product of the two Vectors and it will have to return a Scalar object instead of a Vector. This is inconsistent with the semantics of a replacement operator. You might also want to consider the /= to accept a Matrix object, since it returns a Vector object. However, the original Vector object that calls the /= operator is the right-had-side vector (w) of Eq. 12, and the return Vector is the vector (v) of Eq. 13. Although they are both Vector objects, they are not the same vector as required by the semantics of replacement operation. Exactly the same situation occurs for *= to have a Matrix object as argument. Consequently, in VectorSpace C++ library, we define that both *= and /= can only take a Scalar or a double argument. If other types are used, an exception will be thrown. The logic operator, Vector::operator ==(const C0&), define the condition for equal as two Vector objects have the same length and values for every element. The function Vector::length() returns the size of the Vector. In the box for Scalar on page 8, we did not mention there is Scalar::length() function available, since it doesnt make much sense to ask the length of a Scalar object. Actually it exists. The return value is always 1. Another example is the transpose operator ~. It is also applicable to the Scalar object. It always returns a Scalar of the same value. In VectorSpace C++ library we call this kind of functions backward compatible. The existence of the backward compatible functions enlarges the function set for an object type. It is useful for a more dynamic programming method (see the example discussed on page 41). Functions Vector::norm(int) and Vector::norm(const char*) are defined to take either the values of 1 or 2 with the integer argument version, and to take either infinity, Infinity, maximum or Maximum with the constant string version. In mathematics a p-norm or the Hlder norm of a vector v of length n is defined by1
= ( v1 p + v2 p + + vn p ) 1 / p
Eq. 14
Therefore, for p = 1, the 1-norm (or sum norm) is defined as
= v1 + v2 + + v n
Eq. 15
For p = 2, the 2-norm (or the Euclidean norm) is defined as
2 2 2 v1 + v2 + + vn
Eq. 16
1. B.N. Datta, 1995, Numerical Linear Algebra, and applications, Brooks/Cole Publishing Company, p. 25.
17
Chapter
For, p = , the infinite (or maximum norm), is
= max v i
Eq. 17
Free functions of the forms of C0 norm(const C0&, int = 2) and C0 norm(const C0&, const char*) can be used for retrieving the norms of a vector v. A 2-norm is written as norm(v), or norm(v, 2). The omission of second argument in norm(v) implies that 2 is the default value for the second argument of the function norm( ). A 1-norm is norm(v, 1), and infinite or maximum norm is written as norm(v, infinity) = max(v, Infinity) = max(v, maximum) = norm(v, Maximum). The norm functions are also backward compatible with a Scalar object, which simply returns the absolute value of the Scalar. For the remaining transcendental functions it is suffice to say that these functions perform the distribution rule as discussed in page 16; i.e., applying these functions to a Vector results in returning a new Vector with the values obtained by applying these functions to every element of the Vector. For example, applying a trigonometric function sine to a Vector v of length n is defined as a vector with every element of it the result of applying the trigonometric function sine to that element. sin(v) = {sin(v1), sin(v2), ... , sin(vn) }T
1.1.3 Matrix
The data abstraction for a Matrix is represented as two integer numbers, row-length I and columnlength I, and a value-array mij . Since the row-length and column-length are variables, the memory space for mij has to be managed dynamically (see Figure 18). The value-array of mij is represented by an array of pointers to double (m[0] of type double*) of size = row-length column-length, while an index-array is an array of pointer to pointer to double (m of type double**) to simulate array of double like syntax. The index of the value-array representing mij has the following relation: mij = m[i][j] = value-array[i column-length + j], where 0 i < row-length, and 0 j < column-length. A simple memory management scheme of mij can be implemented for the Matrix object of C0 type. At the time of construction,
18
C0 Type Objects
value-array: m[0] = new double* [row-length * column-length];
m[0][0] m[0][1] . . .
double* m[0]
m00
m01 m02 m03
m04
m10
m11 m12 m13 m14 m20 m21 m22
m23 m24 m30
m31 m32 m33 m34
row-length = 4, column-length = 5
m[0] m[1] m[2] m[3]
m 00 m 01 m 02 m 03 m 04
double** m
m0
m1
m2
m3
index-array: m = new double* [row-length]; for(int i = 1; i < row-length; i++) m[i] =m[i-1] + column_length;
mij
m 10 m 11 m 12 m 13 m 14 m 20 m 21 m 22 m 23 m 24 m 30 m 31 m 32 m 33 m 34
Figure 18 Memory space management of a Matrix.
double** m; m = new double* [row_length]; m[0] = new double [row_lengh * column_length]; for(int i = 1; i < row_length; i++) m[i] = m[i-1] + column_length;
// index-array instantiation // value-array instantiation // setup index-array to point to the // beginning of each row
Notice that according to the pointer arithmetic in C language, the semantics of m[i] can be explained in two steps: (1) the braces after m performs casting as m[] (double*) m, and (2) the index i in the braces indicates the off-set from the first position as m[i] ((double*) m)+i. At the time of destruction, if(m && m[0]) delete [] m[0]; if(m) delete [] m; // release value-array // release index-array
The purpose of this introduction on memory management is not asking you to master the internal working of the VectorSpace C++ Library. The data abstraction has wrapped all these details of how to maintain the valuearray and the index-array in the constructors and destructors of each class. However, from users perspective we need to understand that if we are passing an array by reference to an object, it will always be passed as a double*the value-array, and the value array should always be thought of as an one dimensional array, exactly as how it is organized in memory. (see double* array m[0] in Figure 18). This general concept remains valid for even more complicated classes in VectorSpace C++ Library. We have defined that a row-vector, vT, where v is a (column-) vector, is represented as a Matrix of rowlength = 1, in VectorSpace C++ Library. Now we recall the two row-wise concatenation operators left undefined on page 7. The row-wise-concatenation operator | is used for two Scalars as (project: matrix_examples) C0 a(0.0), b(1.0); C0 c = a | b; cout << c << endl; // two Scalars of type C0 // row-concatenation returns a row-vector // {{0.0, 1.0}}; a Matrix of row-length = 1
19
Chapter
The return value is a Matrix of row-length = 1 with the content as {{0.0, 1.0}}. The double braces are used for expressing the content of a Matrix. Using || instead of | makes no difference in the case with two Scalar objects as operands. The row-wise concatenation operators for two Vectors are defined accordingly, as (project: matrix_examples) double d1[4] = {0.0, 1.0, 2.0, 3.0}, d2[4] = {10.0, 11.0, 12.0, 13.0}; C0 a(4, d1), b(4, d2); C0 c = a | b; cout << c << endl; The output will be a Matrix (expressed in multi-dimensional array format in C language) { {0.0, 10.0}, {1.0, 11.0}, {2.0, 12.0}, {3.0, 13.0} } Again, using || instead of | for two Vectors makes no difference. For mixed type operands with one Scalar and one Vector the row-wise concatenation operator | and the one-by-one row-wise concatenation operator || are defined completely parallel to & and && as illustrated in Figure 17. The only difference is the source and the return (column) Vectors are now a vector represented as a Matrix of row-length = 1. For operator % to take two Vectors, tensor product is written in VectorSpace C++ Library as double a[3] = {1.0, 2.0, 3.0}, b[4] = {4.0, 5.0, 6.0, 7.0}; C0 v(3, a), w(4, b); cout << (v % w) << endl; // (project: matrix_examples)
The result of the tensor product is called a dyad, and is represented as a Matrix object of C0 type as { {4.0, 5.0, 6.0, 7.0 }, {8.0, 10.0, 12.0, 14.0 }, {12.0, 15.0, 18.0, 21.0 } } {{ v0w0, v0w1, v0w2, v0w3}, { v1w0, v1w1, v1w2, v1w3}, { v2w0, v2w1, v2w2, v2w3} }
or,
The two Vector objects in the example above, do not have to have the same length. Three different popular notations for the tensor product are
Indicial Notation
Matrix Algebra
Tensor Algebra
vi wj
v wT
v w
TABLE 12. Three popular mathematical notations for tensor product.
20
C0 Type Objects
Parallel to the definitions for scalar inner product as in Eq. 11 on page 16, the tensor product is written in two ways v * (~w) = v % w Eq. 18
The left-hand-side is consistent with v wT in the matrix algebra, and the right-hand-side is consistent with v w in the tensor algebra.
Constructors
The examples of dedicated constructors for the Matrix can be written as in Program Listing 12. The Matrix object of C0 type is constructed by defining its row-length = 4 and column-length = 8. The double* array m1[0] is the value array passed as a reference to the Matrix a. Notice the semantics of m1[0] = ((double*) m1)+0. Just as in the dedicated constructor for Vector object, the Matrix can be constructed to have its own memory by passing an argument of (double*)0, a null pointer cast to a pointer of double, to the third argument of the dedicated constructor as C0 a(4, 8, (double*)0); However, in doing so all the elements in the Matrix object will be initialized to have the value 0.0. The reference Matrix can be constructed by the dedicated constructor of Matrix class as (see Figure 19) C0 b(2, 3, a, 1, 2); where the first two arguments say that the reference Matrix b has row-length = 2 and column-length = 3. The third argument is the referenced Matrix a, and the last two arguments are the starting indices which are the second row-index 1 and the third column-index 2 in a. Again, the unary positive operator of Matrix + serves the function of primary casting. For example, +b constructs an independent temporary Matrix object of
#include include/vs.h int main() { double m1[4][8] = { { 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0}, {10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0}, {20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0}, {30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0} }; C0 a( 4, 8, m1[0]); cout << a << endl; { { 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0}, {10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0}, {20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0}, {30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0} }; C0 b(2, 3, a, 1, 2); cout << b << endl; return 0; } { {12.0, 13.0, 14.0}, {22.0, 23.0, 24.0} };
Listing 16 Dedicated constructors of Matrix objects (project: matrix_examples).
21
Chapter

a: 4 x 8 a12 a13 a14 a22 a23 a24 +b
b00 b01 b02 b: 2 x 3 b10 b11 b12 Figure 19 Reference Matrix b and referenced matrix a. a12 is the starting element. C0 type. The temporary object +b will be instantiated with the size of b, and its value is initialized to that of b. The constant strings for Matrix virtual constructors (use macro definition MATRIX) and autonomous virtual constructors are shown in the following box virtual constructor string by reference C0& C0* int, int, double* int, int, double*, int, int VectorSpace C++ library definition priority
C0 type Matrix a pointer to C0 type Matrix row-length, column-length, double* != 0, row-length, column-length, double* != 0, memory-row-length, memory-column-length
16 17
by value int, int int, int, double* int, int, const double* int, const C0* const C0* int, int, C0&, int, int
row-length, column-length row-length, column-length, double* = 0, row-length, column-length, double*, length, C0* of a Vector C0* type of a Matrix* row-length, column-length, C0&, starting row-index, starting column-index (the only one for reference Matrix)
15 16 18 19 20
Strings in C0 virtual constructor for Matrix object.

The symbolic operators for Matrix have two assignment operators, three selectors and four concatenation operators (see box in page 23.).
22
C0 Type Objects
operator or function symbolic operators C0& operator &= ( ) C0& operator = ( ) C0& operator [ ] (int) C0& operator( )(int) C0& operator ( )(int, int) C0 operator & ( ) const C0 operator && () const C0 operator | ( ) const C0 operator || ( ) const arithmetic operators C0 operator + ( ) const C0 operator - ( ) const C0 operator + (const C0&) const C0 operator - (const C0&) const C0 operator * (const C0&) const C0 operator / (const C0&) const C0& operator += (const C0&) C0& operator -= (const C0&) C0& operator *= (const C0&) C0& operator /= (const C0&) matrix algebra operators C0 operator ~ ( ) const C0 operator !( ) const logic operators int operator == (const C0&) const int operator != (const C0&) const int operator >= (const C0&) const int operator <= (const C0&) const int operator > (const C0&) const int operator < (const C0&) const functions int row_length() const int col_length() const double norm(in) const double norm(const char*) const VectorSpace C++ library definition remark
assignment by reference assignment by value row selector column selector element selector column concatenation one-by-one column concatenation row concatenation one-by-one row concatenation
return a Vector return a Vector return a Scalar
positive (primary casting) unary negative unary addition subtraction multiplication division (by a Scalar or a Matrix only) replacement addition replacement subtraction replacement multiplication (by a Scalar only) replacement division (by a Scalar only)
transpose matrix decomposition
row-length of the Matrix column-length of the Matrix 1 (maximum column-sum)-norm or 2 (spectral)- norm infinity (max row-sum),Forbenisu(Forbenius-norm) on next page).
Partial listing of Matrix object arithmetic operators, logic operators and functions (continued
23
Chapter

operator or function VectorSpace C++ library definition remark
functions C0 pow(int) const C0 sqrt(const C0&) const C0 exp(const C0&) const C0 log(const C0&) const C0 sin(const C0&) const C0 cos(const C0&) const matrix algebra fucntions int rank() const C0 identity() const C0 cond() const C0 inverse() const C0 det() const
power (applied to each element of the Matrix) square root (applied to each element of the Matrix) exponent (applied to each element of the Matrix) log (applied to each element of the Matrix)) sin (applied to each element of the Matrix) cos (applied to each element of the Matrix
rank of a Matrix identity Matrix condition number of a Matrix inverse of a Matrix determinant of a Matrix
Partial listing of Matrix object arithmetic operators, logic operators and functions.
For the two assignment operators, which are applied with two Matrices, the results are intuitively straightforward and the behavior is similar to that for two Scalars or two Vectors. Again, complexity increases when different types are taken as their arguments (see Program Listing 12). If Matrix::operator = (const C0&) takes a Scalar, it is defined as to have all the elements in the Matrix set to the value of the Scalar. For example, C0 a(2, 3, (double*)0), b(3.0); a = b;
#include include/vs.h int main() { double d[2] = {0.0, 1.0}; C0 a(2, 3, (double*)0), b(3.0); c(2, d); a = b; cout << a << endl; a = c; cout << a << endl; a &= b; cout << a << endl; a &= c; cout << a << endl; return 0; }
b = 3.0 c = {0.0, 1.0} T a= { { 3.0, 3.0, 3.0 }, { 3.0, 3.0, 3.0 } } a= { { 0.0, 0.0, 0.0}, {1.0, 1.0, 1.0} } a = 3.0; a Scalar
a = {0.0, 1.0} T; a Vector
Listing 17 Assignment operators with different types of operand (project: matrix_examples).

24
C0 Type Objects
All the elements in a, totally six of them, will be assigned the value of 3.0 that the Scalar b has. This definition is consistent with that in Vector::operator = (const C0&) taking a Scalar argument. What is new is when the Matrix::operator = (const C0&) takes an argument of type Vector. We define that every column of the Matrix a is assigned with the values of Vector b. Therefore, the row-length of Matrix a and the length of Vector b should be the same. Otherwise an exception will be thrown. The assignment by reference Matrix::operator &=(const C0&) is defined exactly the same as that of Vectors, and is very straight-forward. The Matrix a upon being re-assigning to other types by reference just changes both the type and value of a as in Program Listing 12 for the Vector objects. We say that the label a is peeled off from the original Matrix object and re-attached to whatever kinds of object it is referring to. Of course garbage collection discussed on page 6 will be activated to see if the original Matrix object must go down the drain. The row selector uses the Matrix::operator [](int), and the column selector uses the Matrix::operator ( )(int). Both of these two operators return a Vector object. For example, (see Figure 110) row selector: a[1] 10.0 11.0 20.0 21.0 22.0 23.0 12.0 13.0 2.0 12.0 column selector: a(2) 22.0 a: 3x4 0.0 1.0 2.0 3.0 element selector: 12.0 a(2, 1) = a[1][2] = a(2)[1]
10.0 11.0 12.0 13.0
Figure 110 Row selector, column selector and element selector. double d[3][4] = { { 0.0, 1.0, 2.0, 3.0}, {10.0, 11.0, 12.0, 13.0}, {20.0, 21.0, 22.0, 23.0} }; C0 a(3, 4, d[0]); cout << a[1] << endl; cout << a(2) << endl; cout << a[1][2] << endl; cout << a(2)[1] << endl; cout << a(2, 1) << endl; // (project: matrix_examples)
// row selector; {10.0, 11.0, 12.0, 13.0}T // column selector{ 2.0, 12.0, 22.0} T // 12.0 // 12.0 // element selector; 12.0
The returned Vectors for row and column selectors can again be applying Vector::operator [](int), the selector of the Vector, to access a single element. However, for single element access, we can by-pass the row and column selectors to use element selector, Matrix::operator ( )(int, int), directly with two int arguments for higher efficiency. The element selector has the semantics close to Fortran language. So it is also called the Fortran-style selector. The result of the row-wise-concatenation operators & and &&, when two Matrix objects are taken as arguments, are intuitively comprehensible (see Figure 111). Their definitions are consistent with those for the two Scalars or two Vectors. We notice that, for the regular row-wise-concatenation operator, the columnlengths of the two concatenated Matrices must be the same. For the one-by-one row-wise-concatenation opera-
25
Chapter
1 1 1 4 2 5 3 6 & 4 7 2 5 8 3 6 9 = 4 1 4 7
2 5 2 5 8
3 6 3 6 9 1 4 2 5 3 6 && 1 4 2 5 3 6 = 1 1 4 4
2 2 5 5
3 3 6 6
Figure 111 Row concatenation and one-by-one row concatenation of two Matrices.
1 1 4 2 5 3 6 & 1 = 4 1
2 5 1
3 6 1
1 1 4 2 5 3 6 && 1 = 1 4 1
2 1 5 1
3 1 6 1
1 1 1 4 2 5 3 6 & 2 3 = 4 1 2 3
2 5 1 2 3
3 6 1 2 3 1 4 2 5 3 6 && 1 2 =
1 1 4 2
2 1 5 2
3 1 6 2
Figure 112 Row-wise concatenation and one-by-one row concatenation with Scalar or Vector tor, both row-length and column-length must be the same. We also define the row-wise-concatenation of a matrix with a Scalar or a Vector consistently as those of previous ones (see Figure 111). Row-wise-concatenation with a Scalar is to have every column of the Matrix concatenates row-wisely with the Scalar. One-by-one row-wise-concatenation has every element of the Matrix concatenates row-wisely with the Scalar. Row concatenation of the matrix with a Vector is to have every column of the Matrix concatenates row-wisely with the Vector, and one-by-one row-wise-concatenation with a Vector has every element of the Matrix concatenates rowwisely with the Vector. In the last case the row-length of the Matrix and the length of the Vector should be the same. The behaviors of the column-wise-concatenation operators | and || for two Matrices are also straight-forward (see Figure 111). Similarly, for the column concatenation operators the row-length must be the same. For the one-by-one column-wise-concatenation operator both the row-length and column length of the two matrices must be the same. Column-wise-concatenation of a matrix with a Scalar or a Vector can be defined accordingly (see Figure 111). The column-wise-concatenation with a Scalar is defined as to have every row of
26
C0 Type Objects
1 3 5
2 4 6 |
1 4 7
2 5 8
3 6 9 =
1 3 5
2 4 6
1 4 7
2 5 8
3 6 9
1 3 5
2 4 6 ||
1 3 5
2 4 6 =
1 3 5
1 3 5
2 4 6
2 4 6
Figure 113 Column concatenation and one-by-one column concatenation of two Matrices.
1 3 5
2 4 6 | 1 =
1 3 5
2 4 6
1 1 1
1 3 5
2 4 6 || 1 =
1 3 5
1 1 1
2 4 6
1 1 1
1 3 5
2 4 6 |
1 2 3 =
1 3 5
2 4 6
1 2 3
1 3 5
2 4 6 ||
1 2 3 =
1 3 5
1 2 3
2 4 6
1 2 3
Figure 114 Column concatenation and one-by-one column concatenation of two Matrices. the Matrix concatenates column-wisely with the Scalar. The one-by-one column-wise-concatenation is defined as for every element of the Matrix column-wise concatenates with the Scalar. The column-wise-concatenation and one-by-one column-wise-concatenation with a Vector is also intuitive, but the row-lengths of the Vector and the Matrix must be the same. The positive unary operator for the Matrix can be used to perform the primary casting for a reference Matrix to a Matrix that owns its memory space. The addition and subtraction operators, Matrix::operator +(const C0&) and Matrix::operator -(const C0&), need some explanations. If they take another Matrix as their argument, this Matrix must has the same row-length and column-length with the original Matrix. If the argument taken is a Scalar (or equivalently a double), the addition or substraction is defined, with the distribution rule, to have every element of the Matrix plus or minus the Scalar. If the argument is a Vector, the distribution rule in such a case defines that every column of the Matrix is to plus or minus the Vector. In this case, the row-length of the Matrix and the length of the Vector should be the same, otherwise, an exception will be thrown. Matrix::operator *(const C0&) can take a Scalar, a Vector or another Matrix. For a Scalar (or a double) argument the multiplication obeys the distribution rule. As defined earlier, this means the Scalar value will be multiplied with every element of the Matrix. For a Vector argument, according to usual mathematical definition, the Matrix and its argument Vector should have compatible lengths as Workbook of Applications in VectorSpace C++ Library
27
Chapter

M * v = w,
where dim M = m n, dim v = n ( 1), and dim w = m ( 1) as required in matrix algebra. In Eq. 11 of page 16, the scalar inner product of two vectors (v and w of the same length n) is written with VectorSpace C++ library as (~v) * w = v * w On the left-hand-side, the transpose of a Vector v is a Matrix of row-length = 1. Now, lets see the dimension of this expression is dim vT = {1 n } and dim w = { n 1 }. Therefore it is consistent with the definition of a Matrix multiplies with a Vector. For * operator to take another Matrix object, the length compatible requirement is M*N=L where dim M = m l, dim N = l n, and therefore dim L = m n. In Eq. 18 in page 21, the dyad obtained from the tensor product of two Vectors is written as v * (~w) = v % w the transpose of the Vector w has dim (~w) = 1 m. The Vector v has dim v = n 1. The tensor product definition therefore is consistent with multiplication in matrix algebra where the dyad has dim (v*(~w)) = n m. The division operator / takes either a Scalar or a Matrix. For the case of a Matrix it is defined similar to the process of finding solution of simultaneous equations with multiple right-hand-side vectors. Therefore the rowlength and column-length of the both Matrices must be the same. The replacement division, /=, and replacement multiplication, *=, again only take a Scalar object in order to be consistent with the semantics of the replacement operators. Logic operators ==, is defined to have same row-length, column-length and values for every element. Four norms of a Matrix are defined in VectorSpace C++ library. The maximum column sum matrix norm is denoted with subscript 1 as1
m1
= max
0j<n
i=0
a ij
Eq. 19
The maximum row sum matrix norm is denoted with subscript as

n1
= max
0i<m
j=0
a ij
Eq. 110
1. B.N. Datta, 1995, Numerical Linear Algebra, and applications Brooks/Cole Publishing Company, p. 26-27.
28
C0 Type Objects
The spectral norm is denoted with subscript 2 as
A
2
maximum eigenvalue of A T A
Eq. 111
The Frobenius norm is the one that is most consistent with the 2-norm (Euclidean norm) of a Vector. It is denoted with a subscript F as
n 1m 1
a ij 2
Eq. 112
i= 0j =0
The maximum column norm subscript 1 and spectral norm subscript 2 for a Matrix m are called by functions norm(m, 1) and norm(m, 2), respectively. The maximum row sum norm subscript and Frobenius norm subscript F are called by functions norm(m, infinity) (= norm(m, Infinity)), and norm(m, frobenius) (= norm(m, Frobenius)), respectively. The transcendental functions for the Matrix obey the distribution rule similar to those for the Vector. For example, for a Matrix m of row-length m and column-length n,
sqrt ( m 00 ) sqrt ( m 01 ) sqrt ( m 0n ) sqrt ( m ) = sqrt ( m 10 ) sqrt ( m 20 ) sqrt ( m m0 ) sqrt ( m mn )
Eq. 113
The function, sqrt(), has been applied to every element of the Matrix m.
1.1.4 Matrix Algebra

Many matrix decomposition methods for (1) solution of simultaneous equations, and (2) eigenvalue problems for symmetrical matrix are introduced in this section.
LU Decomposition
For the solutions of simultaneous equations A x = b, a square matrix, A, can be decomposed first into an upper triangular matrix (U) and a lower triangular matrix (L), by the LU decomposition, as A x = (L U) x = b We define Eq. 114
y=Ux
and substitute Eq. 115 into Eq. 114, and we obtain
Eq. 115
29
Chapter

Ly=b Eq. 116
The solutions of triangular matrices in the form of Eq. 115 and Eq. 116 are known to be particularly easy; the solution steps begin either from the first or the last equation that has only one unknown. Then an equation next to the one just solved will have one more new unknown to be solved for. The last step is repeated until all the unknowns are solved. Therefore the solution of the original system can be performed in three steps: (1) perform LU decomposition, (2) solve triangular system for y in Eq. 116. This step is known as forward elimination (or forward substitution), (3) substitute y into Eq. 115 and solve for x from the triangular system. This last step is known as back substitution. In VectorSpace C++ Library, the LU decomposition can be called by using matrix decomposition operator Matrix::operator !(). And, the forward elimination and back substitution are performed by multiplying the decomposed matrix with the right-hand-side vector with the LU::operator *(const C0&). For example, double d[3][3] = { { 0.0, 1.0, 1.0 }, { 1.0, 2.0, 3.0 }, { 1.0, 1.0, 1.0} }, v[3] = { 2.0, 6.0, 3.0 }; C0 A(3, 3, d[0]), b(3, v); C0 x = (!A)*b; cout << x << endl; // (project: matrix_algebra)
// Matrix A and Vector b // decomposed then forward/back substitutions // { 1.0, 1.0, 1.0 }T
You can explicitly form an LU decomposed matrix object by calling the LU constructor as LU a(A); C0 x = a * b; or in short just C0 x = LU(A)*b; The LU decomposition is the default matrix solver in VectorSpace C++ Library. We can achieve a neater expression by considering the mathematical expressions as in Ax=b // LU decomposition // * performs forward and back substitutions
x = A-1 b = b / A
Therefore, the C++ codes can be written as C0 x = b / A; The Vector::operator /(const C0&), taking a Matrix object of C0 type as its argument, is invoked with the definition as in Eq. 13 of page 17. Or equivalently, C0 x = A.inverse() * b; or even // x = A-1 b
30
C0 Type Objects
C0 I = A.indentity(); C0 x = (I / A) * b; // x = (I/A) b
The determinant is computed by the member function call det(), the rank of the matrix by rank(), and the condition number by cond(). For the LU-decomposition without pivoting, the decomposition algorithm is unstable. Some elements in the reduced matrices can grow arbitrarily large that the information in the original data can be corrupted. The default pivoting method for the LU-decomposition is the partial-pivoting (or row-pivoting). The algorithm can be even more stabilized if the complete-pivoting is used. The default behavior can be over-written as Matrix::Pivoting_Method = Matrix::Complete_Pivoting; C0 x = b / A; Matrix::Pivoting_Method = Matrix::Partial_Pivoting;
// reset to default pivoting
The change made to the pivoting method in the above will affect not only the division operator /, but also all invoking methods of the matrix solver used in the above examples.
Cholesky Decomposition
The idiosyncrasy of matrix computation is that there are different methods for specific kinds of matrices. For a (square) symmetric matrix, the Cholesky decomposition is often used. The Cholesky decomposition can be written as A = L D LT Eq. 117
where D is the diagonal matrix. The Cholesky decomposition is two times faster than the LU decomposition. However, the plain Cholesky decomposition works only for a positive definite symmetric matrix ( i > 0 , where i are eigenvalues). For a positive semi-definite symmetric matrix ( i 0 ), the Cholesky decomposition with diagonal-pivoting is necessary to keep the algorithm stable. For an indefinite symmetric matrix, one can just ignore its being symmetric, therefore, the LU-decomposition with complete-pivoting is a must. In the context of the modified Newton method in optimization problem (see page 129 in Chapter 2), the negative curvatures of an indefinite symmetric matrix (Hessian matrix; the second partial derivatives of the objective function) can be modified to have positive curvatures by using the modified Cholesky decomposition. The examples of using the Cholesky decomposition in VectorSpace C++ Library are1 double m[3][3] = { { 16.0,4.0, 8.0}, // (project: matrix_algebra) { 4.0, 5.0, -4.0}, // data for a symmetric matrix { 8.0, -4.0, 22.0} }, v[3] = {5.0, 1.0, 3.0 };// data for the right-hand-side vector C0 A(3, 3, m[0]), b(3, v);
1. example data from A. Jennings and J.J. McKeown, 1992, Matrix computation, 2nd ed., John Wiley & Sons, New York, p.100-101.
31
Chapter

Cholesky a(A); // Cholesky decomposition C0 x = a*b; // forward/back substitutions cout << x << endl;// {0.315972, -0.0416667, 0.0138889}T
Cholesky a(A); calls the Cholesky constructor explicitly. Operators ! or / or function inverse() can be used by setting the matrix solver from default LU decomposition. For example, Matrix::Decomposition_Method = Matrix::Cholesky_Decomposition; C0 x = b/A; Matrix::Decomposition_Method = Matrix::LU_Decomposition; // reset back to default If you are dealing with a positive semi-definite symmetric matrix the diagonal pivoting can be invoked by setting Matrix::Pivoting_Method = Matrix::Diagonal_Pivoting; The modified Cholesky decomposition, for the symmetric indefinite Hessian in optimization, can be invoked with a second argument indicating a small tolerance value, . A critical example is shown in the following1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 double m[3][3] = { { 1.0, 1.0, 2.0}, { 1.0, 1.0+1.e-20, 3.0}, { 2.0, 3.0, 1.0} }, v[3] = {4.0, 5.0+1.e-20, 6.0 }; C0 A(3, 3, m[0]), b(3, v); Cholesky a(A); C0 x = a*b; cout << x << cout; Cholesky a_bar(A, 1.e-20); C0 x_bar = a_bar*b; cout << x_bar << endl; for(int i = 0; i < 3; i++) b[i] +=a_bar.diagonal_increase(i); x = a_bar * b; cout << x << endl; // (project: matrix_algebra) // data for a symmetric matrix // x {1.0, 1.0, 1.0}T for A x = b // Cholesky decomposition // produce garbage when ill-conditioned // {-1.44e82, 1.44e82, -2.4e+61}T // A-1 : modified Cholesky decomposition // x = A-1 b forward/back substitutions // x = {0.0668383, -0.152525, 1.95023}T // add diagonal increase on the left-hand-side // to the right-hand-side(x {1.0, 1.0, 1.0}T) // x = A-1 b, check consistency // x = {1.0, 1.0, 1.0}T
In line 9, the second parameter = 1.e-20 in the constructor Cholesky a_bar(A, 1.e-20); specifies the lower bound (from zero) for the modified eigenvalues. The original matrix has the eigenvalues of 5.1131, -2.2019, and 0.0888, which is clearly indefinite. The Cholesky decomposition of the matrix gives
1. data from P.E. Gill, W. Murray, and M.H. Wright, 1981, Practical optimization, Academic Press Limited, San Diego, pp. 109-111.
32
C0 Type Objects
1.0 0.0 0.0 L = 1.0 1.0 0.0 and, 2.0 10 20 1.0 1.0 0.0 0.0 0.0
20 20
D = 0.0 10
Eq. 118
0.0 0.0 ( 3.0 + 10 )
The modified Cholesky decomposition gives
L = 0.2652
0.0 0.0 1.0 0.0 0.5303 0.4295 1.0
1.0
and, D =
3.771 0.0 0.0 0.0 5.750 0.0 0.0 0.0 1.121
Eq. 119
Now all the diagonal elements have been modified to positive numbers. The difference of the original and the modified matrix has the Frobenius norm of only 6.154. The amount being increased on the diagonals by the modified Cholesky decomposition can be obtained by calling Cholesky::diagonal_increase(int i), the argument specifies the off-set from the first element of the diagonal. In this case, the increased amount of the three diagonals are {2.771, 5.016, 2.243}.
QR Decomposition
Any matrix A can be written as A= QR Eq. 120
where R is a upper triangular matrix and Q is an orthogonal matrix. An orthogonal tensor Q satisfies the necessary and sufficient conditions of QTQ = I, and det Q = 1. Eq. 120 is called the QR decomposition. For a square matrix A, the simultaneous equations A x = b can be solved by the QR decomposition as A x = (QR) x = b And, set y = QT b Then, solve the triangular system of equations Rx = y Eq. 122 Eq. 121
The QR decomposition for a square matrix, if carried out by Householder transformation, is two times more expensive than the LU decomposition. The QR decomposition is always stable. Recall that the LU decomposition is stable only with complete pivoting. Using QR decomposition with VectorSpace C++ Library is simple. For example,
33
Chapter

double d[3][3] = { { 0.0, 1.0, 1.0 }, { 1.0, 2.0, 3.0 }, { 1.0, 1.0, 1.0} }, v[3] = { 2.0, 6.0, 3.0 }; C0 A(3, 3, d[0]), b(3, v); QR a(A); C0 x = a*b; cout << x << endl; // (project: matrix_algebra)
// Matrix A and Vector b // QR decomposition // * is back substitution of Eq. 122 // { 1.0, 1.0, 1.0 }T
As in the case of the Cholesky decomposition, instead of explicitly calling the QR constructor, you can use operators ! and / or function inverse() by setting the default matrix solver to the QR decomposition as Matrix::Decomposition_Method = Matrix::QR_Decomposition; C0 x = b / A; // implicitly call QR decomposition The member functions of the QR class QR::Q() and QR::R() give clearly what they say they are. For a rectangular matrix A of size m n ( m n ) with full rank, the QR decomposition produces R1 0
Q = Q1 Q 2 , and R =
Eq. 123
Q is an m m matrix and R is a m n matrix, where Q1 with n vectors form the orthonormal basis of the range space of A, and Q2 with (m-n) vectors form the orthonormal basis of null space of AT. R1 is a n n matrix and the lower part of the R matrix is a null matrix of size (m-n) n. In the overdetermined full rank least squares problem, the residual of a rectangular matrix A with right-handside vector, b, and the solution, x, is written as r=Ax-b The square of residual norm is
n
Eq. 124
2 2
r r
= (Ax b ) (Ax b ) =
T
i m m
m A x b i ij j j
n
m A x b i ik k k
m n n
xj xk Aij Aik 2 xj Aij b i + b i bi

j k i j i i
Eq. 125
The least squares means to minimize the sum of squares (the residual norm). Taking derivatives with respect to x, for the three terms in the last line, gives first term:
n m m -------- x j x k A ij A ik = x q j k i m m n m n
( jq xk + kq xj ) Aij Aik
j k i
= 2 x k A iq A ik
k i
34
C0 Type Objects
second term: 2 -------- x j A ij b i = 2 jq A ij b i = 2 A iq b i
j i
x q
third term:
n -------- b i b i = 0 x q i
Add three terms together and set the derivatives to zero for the purpose of minimization, we get
2 r 2 ------------ = 2 x k A iq A ik 2 A iq b i = 0 x k i i m n n
Eq. 126
Eq. 126 can be expressed in matrix form as AT A x - AT b = 0 This equation is known as the normal equations. The solution can be obtained from Eq. 127 as Eq. 127
x = [AT A]-1AT b = A-g b
Eq. 128
where A-g = [AT A]-1AT is called the generalized inverse. On the other hand the projection of vector b (of size m) into a lower dimensional range space of A (of size n, with m n ) gives the minimum length of the Euclidean norm of r (= A x - b, see Figure 115)
r
Ax Range(A)
m = 3, n = 2
Figure 115 Projection into range space of A gives the minimum length of r. Since r and the range of A are perpendicular to each other, every column of A is orthogonal to r. Therefore, AT r = 0 Substituting Eq. 124 for r ( = A x - b), we get AT (A x - b) = AT A x - AT b = 0 Workbook of Applications in VectorSpace C++ Library
35
(orthogonal property)
Chapter
This is the geometrical interpretation of the normal equations as obtained from range space projection1. One way to tackle the least squares problem is to obtain first ATA and AT b then solve the system of equations. Since ATA is symmetrical, we can use Cholesky decomposition for efficiency. However, the process to get ATA is sometimes problematic. Round-off errors accumulated in the multiplication of the two matrices, ATA , may corrupt the information in the original A matrix. We can remedy this by using the QR decomposition for the least squares solution of A. Consider the squares of residual norm as r
2 2
= Ax b
2 2
Eq. 129
An orthogonal transformation of Eq. 129 with QT should not change the length of the residual as r where
R1 0
2 2
= Ax b
2 2
= QTAx QTb
2 2
Eq. 130
QTA = R =
and Q T b = b =
b1 b2
Eq. 131
The submatrix R1 and subvector b1 have sizes of n n and n, respectively, and the null matrix and the subvector b2 have sizes of (m-n) n and (m-n), respectively. Therefore Eq. 130 becomes r
2 2
= QTA x QTb
2 2
= R 1 x b1
2 2
+ b2
2 2
Eq. 132
In Eq. 132, the squares of residual norm is minimized with respect to x if we set R 1 x b1 = 0 Eq. 133
Therefore, after we have done the QR decompositionA = Q R, the least squares solution can be found by first obtaining b1 = QT b, then, solving Eq. 133 for x. For example,2
1. D.G. Luenberger, 1969, Optimization by Vector Space Methods, John Wiley & Sons, Inc., p. 55. 2. B.N. Datta, 1995, Numerical Linear Algebra, and applications Brooks/Cole Publishing Company, pp. 333-4, and pp.337-8.
36
C0 Type Objects
double d[3][2] = { {1.0, 1.0}, {1.e-4, 0.0}, {0.0, 1.e-4} }, v[3] = {2.0, 1.e-4, 1.e-4}; C0 A(3, 2, d[0]), b(3, v); QR a(A); C0 x = a * b; cout .precision(12); cout << x << endl; // (project: matrix_algebra)
// QR decomposition // * is back substitution by Eq. 133 // {1.0, 1.0}T
The solution can be checked by computing in your mind, since the right-hand-side vector, b, is just summation of each row of the left-hand-side matrix, A. The QR decomposition therefore yields an exact solution in this numerical case. On the other hand, the normal equation method for this ill-conditioned matrix shows some discrepancies in the following codes (see also project: matrix_algebra) C0 gram_matrix = (~A)*(A); Cholesky c(gram_matrix); C0 x = c * ((~A)*b); cout .precision(12); cout << x << endl; // AT A // Cholesky decomposition // * is forward/back substitutions // {0.99999999875, 1.00000000125}T
For this example the solution has only been mildly corrupted. You may argue that the Cholesky decomposition for normal equation is actually acceptable in practice, if not theoretically sound. However, the flop-count of (ATA) followed by Cholesky decomposition is about two times more expensive than that of QR decomposition alone. QR decomposition can be also used for rank deficient problem to reveal its column rank provided that column-pivoting is used. In VectorSpace C++ Library we reset default of no pivoting for QR decomposition by Matrix::Pivoting_Method = Matrix::Column_Pivoting; before the QR decomposition is called. Then, member function QR::rank() can be called to reveal its column rank. However, the rank revealing QR decomposition is not as reliable as the singular value decomposition (SVD), although the SVD is about one order of magnitude more expensive than the QR decomposition.
Eigenvalue Problem
Before we get to the singular value decomposition, lets first look at the symmetric eigenvalue problem. A symmetric (square) matrix A of size n n have eigenvalue and corresponding eigenvector x if Ax=x Eq. 134
The computation of eigenvalues and eigenvectors of a symmetric matrix can be written in VectorSpace C++ Library as (in project: matrix_algebra)
37
Chapter

double d[4][4] = {{ 1.0, -3.0, -2.0, 1.0}, {-3.0, 10.0, -3.0, 6.0}, {-2.0, -3.0, 3.0, -2.0}, { 1.0, 6.0, -2.0, 1.0}}; C0 A(4, 4, d[0]); Eigen a(A); C0 lambda = a.Eigenvalues(), x = a.Eigenvectors(); // column vectors of x are eigenvectors cout << lambda << endl; // eigenvalues: {14.3295, 4.45696, -0.371375, -3.41509}T cout << x(0) << endl; // { 0.119346, -0.855989, 0.279432, -0.418279 }T cout << x(1) << endl; // { 0.655651, -0.229133, -0.693100, 0.192958}T cout << x(2) << endl; // { 0.505683, -0.00299097, 0.640311, 0.578167}T cout << x(3) << endl; // {-0.547871, -0.463435, -0.177574, 0.673448 }T
Singular Value Decomposition

For a rectangular matrix A of size m n, m n , the singular value decomposition gives A = U VT The singular values of matrix A are the diagonals in the diagonal matrix . Assuming r (r n) is the rank of the matrix, the first r singular values are non-zero in the diagonal submatrix 1. U = [U1, U2], and V = [V1, V2] are subdivided so that the submatrices U1 and V1 to have first r column vectors of U and V, respectively..
mxn
A
mxm
mxn
nxn
U1
U2
1 rxr
V1T rxn
mxr mx(m-r)
V2 T (n-r)xn
The singular values, i, are the non-negative square root of the eigenvalues, i, of the ATA. The column vectors of U and V are the orthonormal bases that span the range and null spaces of A and AT (see TABLE 13.)
Space Range(A) Null(A) Range(AT) Null(A )
T
Orthonormal Bases
column vectors of U1 column vectors of V2 column vectors of V2 column vectors of U1
TABLE 13. Orthogonal projection using singular value decomposition.
38
C0 Type Objects
The singular values in are arranged in non-increasing order. If A is non-singular, the first one 1 is the maximum and the last one n is the minimum. The spectral norm is
A
2
= 1 ,
and the Frobenius norm can be computed by

A
F
2 2 2 ( 1 + 1 + + n )
The condition number of a non-singular matrix is cond = 1 / n The singular value decomposition in VectorSpace C++ library can be called by writing double d[3][2] = { { 1.0, 2.0 }, { 2.0, 3.0 }, { 3.0, 4.0 } }; C0 A(3, 2, d[0]); SVD a(A); C0 sigma = a.Singularvalues(), U = a.U(), V = a.V(); cout << sigma << endl; cout << U(0) << endl; cout << U(1) << endl; cout << U(2) << endl; cout << V(0) << endl; cout << V(1) << endl; cout << a.cond() << endl; cout << a.rank() << endl; // (project: matrix_algebra)
// {6.54676, 0.374153}T // {0.338098, 0.550649, 0.7632}T // {0.847952, 0.173547, -0.500858}T // {0.408248, -0.814697, 0.408248}T // {0.569595, 0.821926}T // {-0.821926, 0.569595}T // 17.4975 // 2
The least square problem can be solved by singular value decomposition. Sometimes the problem can be overdetermined that you make more measurements than the unkown, but it can be still rank deficient because some measurements basically repeat the information of the others, while, at the same time, some vital information has never been obtained. The use of singular value decomposition for the solution of least squares or simultaneous equations is parallel to the use of LU, Cholesky or QR decomposition. For example, we have SVD a(A); C0 x = a * b; or set default matrix solver by Matrix::Decomposition_Method = Matrix::Singular_Value_Decomposition; C0 x = b / A;
39
Chapter
We can also use the decompose operator ! or the member function inverse(). In fact, the function inverse() gives the so-called pseudoinverse (also known as Moore-Penrose generalized inverse). The condition of m n on matrix A is not considered restrictive at all. If we have m < n, we can work on the SVD of AT = U VT and the SVD of A = (U VT)T = V U T.
In summary, we have introduced all the primary objects, Scalar, Vector, and Matrix objects of C0 type, in Sections 1.1.1 to 1.1.4. For a complicated object like Matrix, data abstraction begins to play a more important role compared to simple objects like Scalar or Vector. Data abstraction helps us encapsulate the complexity of memory management of the data array for the Matrix class. The details of how actually these data are stored in the Matrix object is hidden from users. The various constructors and the destructor of the Matrix object help users, behind the scene, handle the resources of the Matrix class. On top of this, because the Matrix class needs an extensive bag of tricks on the subject of matrix algebra, the data abstraction help to organize different kinds of decomposition methods-LU, Cholesky, QR, SVD, and eigen-system solver with the associated matrix data into a coherent module. This integrated module, under the concept of data abstraction, acts like an intelligent entity that has the knowledge to deal with problems of its own. In the section on Scalar, we introduced the object-oriented programming that achieves the flexibility by the use of virtual constructors. In the definition for two assignment operators = and &=, we introduced the C0 type object actually acting more like a label that can be peeled off or attached to a concrete object . This symbolic flavor is also a feature enabled by the object-oriented programming method implemented in VectorSpace C++ library. The C0 class is the base class for the derived concrete classes, Scalar, Vector, Matrix, ..., etc. The base class C0 also serves as a flat-interface1 for all of its derived concrete classes. The C0 class contains all member operators and functions of all its derived concrete classes. In the users program, C0 is the only generic type used. C0 serves as the delegate for all of its constituents. This VectorSpace C++ library generic type feature actually pushes the programming environment of C++ more towards the side of a full-fledged object-oriented language for programming versatility, which complements the strong type-compiled language that C++ is originally designed for safety. We use one simple example to illustrate the advantage of this dynamic, late-binding feature supported by VectorSpace C++ Library. For example, root-finding for a function f(x) is stated as the following Problem: Solve scalar x, for a scalar function f(x) = 0 Approximation of function f(x) by the Taylor expansion to the first order gives f(x0+dx) f(x0) + f (x0) dx = 0 Therefore, the increment of solution dx can be found, from the above equation, by using dx = -f(x0) / f (x0) (known as Newtons formula), and the solution is updated with increment dx by xi+1 = xi + dx, where i is the iterative index. A converged solution is obtained when dx becomes negligible. We can easily extend this prob-
1. Bjarne Stroustrup, 1991, The C++ programming language, 2nd ed., Addison-Wesley Publishing Company, Massachusetts, p. 452.
40
C0 Type Objects
lem to a multi-dimensional case. For a vector function f(x) of size n in n-dimensional space x, the same root-finding problem can be solved by the same Newtons method. However, in the n-dimensional case, both f and x are vectors of size n, and the first derivatives df is a Matrix (of size n n, = grad f ). In VectorSpace C++ Library, we write C0 newton_formula(const C0& f, const C0& df) { return -f / df; } At the time of writing this subroutine, we do not need to distinguish whether it is for a one-dimensional problem or a multi-dimensional problem. When this subroutine is called by the user, if the problem is one-dimensional, the arguments f, df and the return value are all Scalar objects of C0 type. If the problem is multi-dimensional, the user passes a Vector object of C0 type through argument f, and a square Matrix object of C0 type through argument df, and the return value will be a Vector objectthe increment dx from Newtons formula. The division operator /, in the multi-dimensional case, implicitly calls the default matrix solverthe LU decomposition to solve the problem. We discuss one more example of object-oriented function dispatching mechanism. Assuming in the subroutine newton_formula(), we want to display the value of f column-wise and add a line as for(int i = 0; i < f.length(); i++) cout << f[i] << endl; The member function C0::length() makes sense if the variable f is a Vector. We mentioned on page 17 the concept of backward compatibility of a member function. For a Scalar object, a call to length() exists and will always return the default value of 1 . This compatibility of member functions is the result of using the flatinterface provided by the C0 for all its derived concrete classes. Furthermore, whatever f is, a scalar or a vector, we can always write cout << f << endl; cout in VectorSpace C++ Library knows how to handle the output whatever f is! We see that as the result of using C0 instead of concrete types a general form of syntax can be defined. We note by passing that for Newtons method we better use C1 type instead, which will be introduced in the next chapter. With the C1 type, a differentiable object can be easily defined. The above hypothetical example is only used to illustrate, from programming method perspective, the versatility of using the more dynamic, latebinding technique. It is not for the purpose of explaining the implementation of a real world numerical problem.
1.1.5 Subvector and Submatrix

Fortran and C++ languages support multi-dimensional array for representing higher order tensors. It is tempting for us to implement objects with dimension greater than Matrix (dim. > 2). In fact, higher order tensors are needed in practice. For example, the elasticity tensor in solid mechanics is a fourth order tensor. In engineering literatures, however, the fourth order tensor is rearranged in engineering convention and is represented by a 2dimensional matrix. There is an obvious reason for doing that. Lower dimensions are always easier to be written down in paper. Learning from engineering experience, we realize that arrays of too high dimensions should be avoided if possible, because they are always very hard to be understood and handled. Therefore, the implementation of VectorSpace C++ Library put emphasis on the use of Subvector and Submatrix objects instead of objects Workbook of Applications in VectorSpace C++ Library
41
Chapter

9 = 3x3 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 Figure 116 Referenced source Vector and referenced source Matrix are equalpartitioned to Subvectors and Submatrices. 0. 1. 2. 3. 4. 5. 6. 7. 8. 4 x 9 = {2x2} x {3x3}
10. 11. 12. 13. 14. 15. 16. 17. 18. 20. 21. 22. 23. 24. 25. 26. 27. 28. 30. 31. 32. 33. 34. 35. 36. 37. 38.
of arbitrarily higher dimensional array. In many engineering applications, for example in the finite element method, subvectors and submatrices are used as a convention to represent objects that would otherwise be written in a higher dimensional array. The mathematical subvector and submatrix, of course, should include the reference Vector and reference Matrix defined on page 10 and page 21, respectively. The reference Vector or reference Matrix can have arbitrary sizes and starting/ending indices as long as they are within the bounds of the Vector or the Matrix they are referring to. In VectorSpace C++ Library the terms Subvector and Submatrix are reserved for special kind of subvector and submatrix. This special kind of subvector and submatrix should be able to have their referenced source vector and referenced source matrix partitioned into equal-sized blocks. For example,, in Figure 116 the referenced source Vector has length = 9. The Vector can be subdivided into 3 equal sized sub-blocks of length = 3. Similarly, the reference source Matrix has row-length = 4 and column-length = 9. This Matrix can be subdivided into 6 equal sized blocks. Each has row-length = 2 and column-length = 3. We will show you later that the Subvector and Submatrix not only serve the purpose of representing higher dimensional tensors with a lower dimensional ones, but also with the use of the VectorSpace Subvector/Submatrix index scheme, the need for the for control statement in C++ being substantially reduced, leads to a much uncluttered C++ codes that become close to the mathematical expressions.
Constructors and Selectors

To construct a Subvector or Submatrix object of the C0 type, the constructor and selector need to act in concert to make the Subvector and Submatrix objects (see examples in project: subvector_submatrix_examples). The dedicated constructors for Subvector and Submatrix can be written in VectorSpace C++ Library as (see Figure 116) double v[9] = {0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0}, d[4][9] = {{0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0},
42
C0 Type Objects
{10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0}, {20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0}, {30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0}}; C0 a(v, 9, 3), b(d[0], 4, 9, 2, 3); // Nominal_Subvector and Nominal_Submatrix The double array v and d are the physical memory space to be referenced. The declaration by the dedicated constructors of the last line actually gives a Nominal_Subvector and a Nominal_Submatrix. By the word Nominal, we mean that it is the name for a Subvector or a Submatrix. Therefore, a and b are the symbols that will be used to generate Subvector and Submatrix per se. The declaration above by the dedicated constructors of the C0 type only provides information on how to partition a continuous memory space into equal sized sub-blocks. For example, a(v, 9, 3) means that the double array v has length = 9, and it is partitioned into three equal sized sub-blocks, each of which has length = 3, and b(d[0], 4, 9, 2, 3) means that the double array d has rowlength = 4, column-length = 9, and it is partitioned into six equal sized sub-blocks, each of which has rowlength = 2, and column-length = 3. The Subvector and Submatrix are generated by using selectors. For example, (in project: subvector_submatrix_examples, and see also Figure 116) // continuous block selector; operator ( . ) or ( . , . ) // subvectors cout << a(0) << endl; // {0.0, 1.0, 2.0}T cout << a(1) << endl; // {3.0, 4.0, 5.0}T cout << a(2) << endl; // {6.0, 7.0, 8.0}T // submatrices cout << b(0, 0) << endl; // {{ 0.0, 1.0, 2.0}, // {10.0, 11.0, 12.0}} cout << b(0, 1) << endl; // {{ 3.0, 4.0, 5.0}, // {13.0, 14.0, 15.0}} cout << b(0, 2) << endl; // {{ 6.0, 7.0, 8.0}, // {16.0, 17.0, 18.0}} cout << b(1, 0) << endl; // {{20.0, 21.0, 22.0}, // {30.0, 31.0, 32.0}} cout << b(1, 1) << endl; // {{23.0, 24.0, 25.0}, // {33.0, 34.0, 35.0}} cout << b(1, 2) << endl; // {{26.0, 27.0, 28.0}, // {36.0, 37.0, 38.0}} The operator ( ) of the Nominal_Subvector and Nominal_Submatrix selects a continuous block of memory space to form a Subvector or Subvectors (see Figure 116). For example, the index of a(1) is 1, which means the off-set from the first continuous block subvector. The contents of the Subvector a(1) is, therefore, {3.0, 4.0, 5.0}T referring to the second continuous block in the equal-partitioned double array v. The indices of b(1, 2) means the continuous block Submatrix has 1 row-offset and 2 column-offset from the first continuous block. The contents of b(1, 2) is then {{26.0, 27.0, 28.0}, {36.0, 37.0, 38.0}}. The operator [ ] of the Nominal_Subvector and Nominal_Submatrix selects a regular increment memory spots to form a Subvector or a Submatrix . For example, in VectorSpace C++ Library, the Subvectors and Submatrices are generated as (in project: subvector_submatrix_examples) Workbook of Applications in VectorSpace C++ Library
43
Chapter

0.0 1.0 2.0 3.0 0. 1. 2. 3. 4. 5. 6. 7. 8.
10. 11. 12. 13. 14. 15. 16. 17. 18. 20. 21. 22. 23. 24. 25. 26. 27. 28. 30. 31. 32. 33. 34. 35. 36. 37. 38.
a(1)
3.0 4.0 5.0
4.0 5.0 6.0 7.0 8.0
b(1, 2)
26. 27. 28. 36. 37. 38.
Figure 117 Subvectors and Submatrices are generated by calling a continuous block selector with the selector ( ). // subvectors cout << a[0] << endl; cout << a[1] << endl; cout << a[2] << endl; // submatrices // operator [ ] applied twice cout << b[0][0] << endl; cout << b[0][1] << endl; cout << b[0][2] << endl; cout << b[1][0] << endl; cout << b[1][1] << endl; cout << b[1][2] << endl;
// {0.0, 3.0, 6.0}T // {1.0, 4.0, 7.0}T // {2.0, 5.0, 8.0}T
// {{ 0.0, 3.0, 6.0}, // {20.0, 23.0, 216.0}} // {{ 1.0, 4.0, 7.0}, // {21.0, 24.0, 27.0}} // {{ 2.0, 5.0, 8.0}, // {22.0, 25.0, 28.0}} // {{10.0, 13.0, 16.0}, // {30.0, 33.0, 36.0}} // {{11.0, 14.0, 17.0}, // {31.0, 34.0, 37.0}} // {{12.0, 15.0, 18.0}, // {32.0, 35.0, 38.0}}
For example, see Figure 116, a[1] means every element with offset of 1 from every equal-partitioned subblock is selected to form a Subvector. The contents of the Subvector a[1] is therefore {1.0, 4.0, 7.0}T, which is referring to every second element of the sub-blocks in the double array v. The indices of b[0][1] means every element with 0 row offset and 1 column offset from the first elements of the equal-partitioned sub-blocks is selected to form a Submatrix. The contents of the Submatrix b[0][1] is therefore {{1.0, 4.0, 7.0}, {21.0, 24.0, 27.0}}, which is referring to every first row, second column element of each sub-block in the double array d.
44
C0 Type Objects
0.0 1.0 2.0 3.0 a[1] 1.0 4.0 7.0 4.0 5.0 6.0 7.0 8.0 Figure 118 Subvectors and Submatrices are generated by calling a regular increment selectors [ ]. [ ] needs to be applied twice for the regular increment Submatrix. The access to a part or an element of a Subvector or a Submatrix is exactly like the access to that of the Vector and Matrix. We can use operator[ ](int) to select an element of a Subvector. For the Submatrix, operator [ ](int) is the row-selector and operator( )(int) is the column selector, they both return a Vector object, and operator( )(int, int) is the element selector. For example, (project: subvector_submatrix_examples) cout << a[1][1] << endl; cout << a(1)[0] << endl; cout << b(1, 2)[0] << endl; cout << b(1, 2)[0][1] << endl; cout << b[0][1](2) << endl; cout << b[0][1](2, 0) << endl; // 4.0 // 3.0 // {26.0, 27.0, 28.0}T // 27.0 // {7, 27}T // 7 b[0][1] 1. 4. 7. 0. 1. 2. 3. 4. 5. 6. 7. 8.
10. 11. 12. 13. 14. 15. 16. 17. 18. 20. 21. 22. 23. 24. 25. 26. 27. 28. 30. 31. 32. 33. 34. 35. 36. 37. 38.
21. 24. 27.
In the above example, the Subvectors and Submatrices are always referring to memory space directly. In VectorSpace C++ Library, one more layer of indirection is possible. You can define Subvector of Subvector, or Submatrix of Submatrix. For example (see Figure 119, and project: subvector_submatrix_examples), double v[12] = {0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0}, d[8][12] = { { 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0}, {10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0}, {20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0}, {30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0}, {40.0, 41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0}, {50.0, 51.0, 52.0, 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0, 60.0, 61.0}, {60.0, 61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 67.0, 68.0, 69.0, 70.0, 71.0}, {70.0, 71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0, 81.0} }; C0 a(v, 12, 6), b(d[0], 8, 12, 2, 2), c(a(0), 3), // c is a Nominal_Subvector of a Subvector Workbook of Applications in VectorSpace C++ Library
45
Chapter

e(b[0][1], 2, 3); cout << a(0) << endl; cout << c[0] << endl; cout << b[0][1] << endl; // e is a Nominal_Submatrix of a Submatrix // {0.0, 1.0, 2.0, 3.0, 4.0, 5.0}T // {0.0, 3.0}T; Subvector of Subvector // { { 1.0, 3.0, 5.0, 7.0, 9.0, 11.0}, // {21.0, 23.0, 25.0, 27.0, 29.0, 31.0}, // {41.0, 43.0, 45.0, 47.0, 49.0, 51.0}, // {61.0, 63.0, 65.0, 67.0, 69.0, 71.0} } // { {41.0, 43.0, 45.0}, Submatrix of Submatrix // {61.0, 63.0, 65.0} }
cout << e(1, 0) << endl;
In this example, a and b are Nominal_Subvector and Nominal_Submatrix, respectively, which refer to double array v and d. c and e are Nominal_Subvector and Nominal_Submatrix that refer to Subvector a[0] and Submatrix b[0][1]. Upon being applied selectors, c[0] and e(1, 0) are a Subvector of Subvector and a Submatrix of Submatrix, respectively. The virtual constructor (use macro definitions SUBVECTOR and SUBMATRIX) and autonomous virtual constructor serve exactly the same purpose as those of the primary objects of C0 type. The strings that are used for the virtual constructor and autonomous virtual constructor are listed in the following boxes.
d v 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10. 11. a(0) 0.0 1.0 2.0 3.0 4.0 5.0 c[0] 0.0 3.0
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10. 11. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. 60. 61. 60. 61. 62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 70. 71. 72. 73. 74. 75. 76. 77. 78. 79. 80. 81.
b[0][1]
1.0 3.0 5.0 7.0 9.0 11. 21. 23. 25. 27. 29. 31. 41. 43. 45. 47. 49. 51. 61. 63. 65. 67. 69. 71.
41. 43. 45. e(1, 0) 61. 63. 65.
Figure 119 Subvector of Subvector(left-hand-side) and Submatrix of Submatrix (right-hand-side).

46
C0 Type Objects
virtual constructor string nominal Subvector int, int, double* int, C0& nominal Submatrix int, int, int, int, double* int, int, C0& VectorSpace C++ library definition double*, source-length, block-length Vector or Subvector of C0 type, block-length source row-length, source column-length, block row-length, block column-length, double* block row-length, block column-length, Matrix or Submatrix of C0 type priority 21 22
Strings in C0 virtual constructor for generating nominal Subvector and nominal Submatrix.
We emphasize that the C0 constructors play the role of constructing Nominal_Subvectors and Nominal_Submatrices, while the selectors of the Nominal_Subvector and Norminal_Submatrix play the role of constructors to generate Subvectors and Submatrices.
C0 constructor
Nominal_Subvector & Nominal_Submatrix
selectors (of Nominal_Subvector and Nominal_Submatrix)
Subvector & Submatrix
Other Operators and Functions

Arithmetic operators for Nominal_Subvector and Nominal_Submatrix only make sense when we introduce basis expression in the next section. We restrict our discussion here on Subvector and Submatrix. On page 11, we introduce the primary casting for reference Vector by unary operator +. operator +() works for Subvector and Submatrix to convert them into Vector and Matrix, respectively. Actually not just the unary + operator, all arithmetic operators of Subvector and Submatrix are parallel to those of Vector and Matrix. When an arithmetic operator of a Subvector or a Submatrix is called, it converts (primary cast) the operand(s), and the operator is dispatched to that of the Vector or Matrix accordingly. The transcendental functions and matrix algebra functions of Subvector and Submatrix also primary cast them into Vector and Matrix. Continuing from the previous example, we explore the addition operator, (project: subvector_submatrix_examples) cout << a(0) << endl; cout << a(1) << endl; cout << (a(0)+a(1)) << endl; // {0.0, 1.0, 2.0, 3.0, 4.0, 5.0}T // {6.0, 7.0, 8.0, 9.0, 10.0, 11.0}T // {6.0, 8.0, 10.0, 12.0, 14.0, 16.0}T
In the last line, when binary C0::operator +(const C0&) is called, both Subvectors a(0) and a(1) are turned into Vectors, and then the two Vectors are added together. In fact, in the first two lines, upon << being called, the Subvectors are also converted into Vectors for output.
47
Chapter
Subvector and Submatrix may not have been used extensively in physics and mathematics. However, its use in engineering literatures is very extensive. They are used to explicitly lay out formula for engineers to follow easily. Many of them have become quite conventional, for example, the B-matrix in the finite element method. We will demonstrate the power of Subvector and Submatrix when we get to the finite element method in Chapter 4 and Chapter 5. On the contrary, the mathematicians and scientists favor the linear algebraic expression using basis. VectorSpace C++ Library support the best of the both worlds.
1.1.6 Basis
The usage of Basis is a lot like that of Subvector and Submatrix. The constructor and selector are orchestrated with each other to create Basis objects of C0 type. To understand the semantics of constructing the Subvector and Submatrix, we need to understand how the temporary objects like Nominal_Subvector and Nominal_Submatrix are generated in the intermediate step. The semantics of generating Basis object is just the same. The names of those temporary objects are getting quite wordy because of the combinatorial explosion of objects. We can learn how to generate those complicated basis expressions by examples and illustrations in figures, without regard to those meticulous temporary object names. Only in one occasion or two, when we need to get into the semantics of their usage, we need to spell their names all out. In VectorSpace C++ Library, a Basis is created by calling the C0 dedicated constructor and then applying the selector as (in project: basis_examples) C0 r(5.0), e(5); C0 v = r * e[3]; cout << v << endl; C0 w = e[1] + r * e[3] + 6.0 * e[4]; cout << w << endl; // Scalar 5.0 // Nominal_Basis of space dimension = 5 // Vector; r porjected into fourth position of e // {0.0, 0.0, 0.0, 5.0, 0.0}T // {0.0, 1.0, 0.0, 5.0, 6.0}T
The C++ code r * e[3] is completely parallel to the mathematical expression,
v = r e 3,
Eq. 135
where r * e[3] means that a Scalar r is projected, by operator * to the basis e[3]. The result of such projection gives a Vector with the fourth component, with index 3 as its offset number, having the value of r. If one of the terms in the expression does not have a Scalar, like r or 6 in the following, projected to the basis, it assumes the scalar constant is 1. For example,
w=e1+re3+6e4
Eq. 136
where w = {0.0, 1.0, 0.0, 5.0, 6.0}T. The second component of w is 1.0, which has assumed a constant 1.0 projected at e 1. The projection by * can be further visualized as in Figure 120, in which r is projected by using Basis e[3] as its projection map (or filter). In the C++ code, w is a Vector that has the value of 5.0 from projecting the Scalar r into its fourth component. Similarly, the fifth component of w has double 6.0. What if we have mathematical expressions, using bases, that generate second order tensor, such as
48
C0 Type Objects
Basis: e[3] 0 1 2 operator *(const C0&) Scalar: r = 5.0 3 4 Figure 120 Projection using *. Basis e[3] serves as the projection map. Vector: w
5.0
t 00 e 0 e 0 + t01 e 0 e 1 + t 02 e 0 e 2 + t 10 e 1 e 0 + t11 e 1 e 1 + t 12 e 1 e 2 + t 20 e 2 e 0 + t21 e 2 e 1 + t 22 e 2 e 2
Eq. 137
In C++ code, it can be written as (in project: basis_examples) double t[3][3] = { { 0.0, 1.0, 2.0}, {10.0, 11.0, 12.0}, {20.0, 21.0, 22.0} }; C0 e(3), T = t[0][0] * (e[0]%e[0]) + t[0][1] * (e[0]%e[1]) + t[0][2] * (e[0]%e[2]) + t[1][0] * (e[1]%e[0]) + t[1][1] * (e[1]%e[1]) + t[1][2] * (e[1]%e[2]) + t[2][0] * (e[2]%e[0]) + t[2][1] * (e[2]%e[1]) + t[2][2] * (e[2]%e[2]); cout << T << endl; // { { 0.0, 1.0, 2.0}, // {10.0, 11.0, 12.0}, // {20.0, 21.0, 22.0} }; Again the tensor product is represented by % to produce a Basis_Matrix. The operator * still acts as projection operator. Matrix: T
Basis_Matrix: e[2]%e[1] 21.
double t[2][1] = 21.0 Figure 121 Projection using *. Basis_Matrix e[2]%e[1] is used as a projection map.
49
Chapter
In Eq. 137 and the above code segment, every term has been written out explicitly, although missing a few terms or omitting some constant values of tij before a second order basis ei ej still works. We call Eq. 135 to Eq. 137 as primary basis expressions. When we introduce the use of Subvector and Submatrix in the previous section, we argued the disadvantage of higher dimensional objects from the experience of the development of the engineering convention (page 41). We do not promote using tensor of order higher than 2, because they are much harder to comprehend and are almost impossible to be written out neatly. In complement, we provide extended basis expressions, when some higher dimensional tensors are needed in an application. The extended basis expression gives Nominal_Subvector or Nominal_Submatrix as its return value. For example, (in project: basis_examples) double v[6] = {0.0, 1.0, 2.0, 3.0, 4.0, 5.0}, w[6] = {6.0, 7.0, 8.0, 9.0, 10.0, 11.0}; C0 V(6, v), W(6, w), // two Vectors e(6), E(2); // two Basis C0 A = V * (e*E[0]) + W * (e*E[1]); // use Basis_Subvector: e*E[0] and e*E[1] cout << (+A) << endl; // {0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0}T cout << A(0) << endl; // {0.0, 1.0, 2.0, 3.0, 4.0, 5.0}T cout << A(1) << endl; // {6.0, 7.0, 8.0, 9.0, 10.0, 11.0}T cout << A[3] << endl; // {3.0, 9.0}T C0 U = e * E; // Nominal_Basis_Subvector: U C0 A1 = V * U(0) + W * U(1); // A1 == A where A is a Nominal_Subvector generated by projecting two Vectors V and W into two Basis_Subvectors e*E[0] and e*E[1]. Lets first look at the Basis_Subvectors e*E[0] (see Figure 122). The Basis e is projected into the a Basis E[0] to form a Basis_Subvector e*E[0]. Then, a Vector V is projected using e*E[0] as its projection map to form a Nominal_Subvector V*(e*E[0]). In the above code segment, a similar step is done by projecting a Vector W into the lower part of a Nomial_Subvector W*(e*E[1]). Then, these two Nominal_Subvectors are added together and assigned to a C0 variable A, which serves as a label for it. A can be used just as we used a Nominal_Subvector to generate all kinds of Subvectors that we want in the previous section. For the output of A, (+A) is the primary casting using unary positive operator + to transform the Nominal_Subvector into a Vector. A different path to generate the same thing can be done (see Figure 123). The Nominal_Basis e which has length = 6 is projected into another Nominal_Basis E which has length = 2. The result is a Nominal_Basis_Subvector e*E. We then assign the C0 variable U to this Nominal_Basis_Subvector. A Vector V is projected to a Basis_Subvector U(0) to produce a Nominal_Subvector V*U(0). In the above example the Nominal_Basis is projected to a Basis. We now discuss a case with a reverse order of the object occurrences; i.e., with a Basis (E[0]) projecting to a Nominal_Basis (e), such as E[0]*e (in project: basis_examples). C0 B = V * (E[0]*e) + W * (E[1]*e); // use Basis_Subvector: E[0]*e and E[1]*e cout << (+B) << endl; // {0.0, 6.0, 1.0, 7.0, 2.0, 8.0, 3.0, 9.0, 4.0, 10.0, 5.0, 11.0}T cout << B(0) << endl; // {0.0, 6.0}T cout << B[1] << endl; // {6.0, 7.0, 8.0, 9.0, 10.0, 11.0}T
50
C0 Type Objects
e * E[0] e * E[0] 0 1 = V 0.0 1.0 2.0 3.0 4.0 5.0 * e * E[0] = V*(e*E[0]) 0.0 1.0 2.0 3.0 4.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 Figure 122 Projection of a Nominal_Basis e on a Basis E[0] to form a Basis_Subvector e*E[0] and projection of Vector V into the Basis_Subvector e*E[0]. e*E=U e * E = V 0.0 1.0 2.0 3.0 4.0 5.0 * U(0) = V*U(0) 0.0 1.0 2.0 3.0 4.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 Figure 123 Projection of Nominal_Basis e on Nominal_Basis E to form a Nominal_Basis_Subvector U = e*E, and projection of Vector V on a Nominal_Basis_Subvector U(0) to form a Nominal_Subvector V*U(0). Workbook of Applications in VectorSpace C++ Library
51
Chapter

C0 Z = E*e; C0 B1 = V * Z[0] + W * Z[1]; // Nominal_Basis_Subvector: Z // B1 == B
In the above code (see also Figure 122), a Basis E[0] is projected to a Nominal_Basis e to form a Basis_Subvector E[0]*e. A Vector V is then projected to E[0]*e to generate a Nominal_Basis_Subvector V*(e*E[0]). E[0]*e = V*(E[0]*e) E[0]*e * = 0.0 * 0.0 e E[0] 0 1 V 0.0 1.0 2.0 3.0 4.0 5.0 1.0 0.0 2.0 0.0 3.0 0.0 4.0 0.0 5.0 0.0 Figure 124 Projection of a Basis E[0] to a nominal Basis e to form a Basis Subvector E[0]*e. A Vector V is projected to E[0]*e to generate a nominal Basis Subvector V*(E[0]*e). There are cases that the Nominal_Basis_Subvector can be used. (see Figure 122). The Nominal_Basis E which has the length = 2 is projected into another Nominal_Basis e which has the length = 6. The result is a Nominal_Basis_Subvector E*e, then a C0 variable Z is assigned (by copy constructor which is always by reference in VectorSpace C++ Library) to this Nominal_Basis_Subvector. A Vector V is projected to a Basis_Subvector Z[0] to produce a Nominal_Subvector V*Z[0]. If you have mastered the Subvector, Submatrix and the Basis_Subvector in the above, the Basis_Submatrix should become natural according to the rules that you have just learned. The only thing is you will notice a two dimensional Basis_Submatrix is certainly more complicated than the one dimensional Basis_Subvector. We have warned you that the names of the temporary objects generated along the way could get quite wordy. Those names are important for understanding the semantics and therefore the internal working of VectorSpace C++ Library. However, in most cases you can get a practical understanding by just reading the example codes and the Figures that are associated with the codes. For example, (in project: basis_examples) double d1[4][6] = { { 0.0, 1.0, 2.0, 3.0, 4.0, 5.0}, {10.0, 11.0, 12.0, 13.0, 14.0, 15.0}, {20.0, 21.0, 22.0, 23.0, 24.0, 25.0},
52
C0 Type Objects
E*e=Z * e V E 0.0 1.0 2.0 3.0 4.0 5.0 = * Z[0] = V*Z[0] 0.0 0.0 1.0 0.0 2.0 0.0 3.0 0.0 4.0 0.0 5.0 0.0 Figure 125 Projection of nominal Basis E on nominal Basis e to form a nominal Basis Subvector Z = E*e, and projection of Vector V on a nominal Basis Subvector Z[0] to form a nominal Subvector V*Z[0]. {30.0, 31.0, 32.0, 33.0, 34.0, 35.0} }, d2[4][6] = { { 6.0, 7.0, 8.0, 9.0, 10.0, 11.0}, {16.0, 17.0, 18.0, 19.0, 20.0, 21.0}, {26.0, 27.0, 28.0, 29.0, 30.0, 31.0}, {36.0, 37.0, 38.0, 39.0, 40.0, 41.0} }, d3[4][6] = { {40.0, 41.0, 42.0, 43.0, 44.0, 45.0}, {50.0, 51.0, 52.0, 53.0, 54.0, 55.0}, {60.0, 61.0, 62.0, 63.0, 64.0, 65.0}, {70.0, 71.0, 72.0, 73.0, 74.0, 75.0} }, d4[4][6] = { {46.0, 47.0, 48.0, 49.0, 50.0, 51.0}, {56.0, 57.0, 58.0, 59.0, 60.0, 61.0}, {66.0, 67.0, 68.0, 69.0, 70.0, 71.0}, {76.0, 77.0, 78.0, 79.0, 80.0, 81.0} }; C0 a1(4, 6, d1[0]), a2(4, 6, d2[0]), // four matrices a3(4, 6, d3[0]), a4(4, 6, d4[0]), e1(4), e2(6), E(2); // three Bases C0 A = a1 * ((e1%e2)*(E[0]%E[0])) + a2 * ((e1%e2)*(E[0]%E[1])) + a3 * ((e1%e2)*(E[1]%E[0])) + a4 * ((e1%e2)*(E[1]%E[1])); cout << A(0, 0) << endl; // { { 0.0, 1.0, 2.0, 3.0, 4.0, 5.0}, // {10.0, 11.0, 12.0, 13.0, 14.0, 15.0}, // {20.0, 21.0, 22.0, 23.0, 24.0, 25.0}, // {30.0, 31.0, 32.0, 33.0, 34.0, 35.0} } Workbook of Applications in VectorSpace C++ Library
53
Chapter

C0 X = (e1%e2)*(E%E); C0 A1 = a1 * X(0, 0) + a2 * X(0, 1) + a3 * X(1, 0) + a4 * X(1, 1); // Nominal_Basis Submatrix // A1 == A
In the above code segment (see also Figure 126), a Matrix a1 of row-length = 4 and column-length = 6 is projected into a Basis_Submatrix ((e1%e2)*(E[0]%E[0])). A Nominal_Basis_Matrix e1%e2 is projected into a Basis_Matrix E[0]%E[0] to form a Basis_Submatrix ((e1%e2)*(E[0]%E[0])). This Basis_Submatrix is then used to project a Matrix a1 to form a Nominal_Submatrix a1*((e1%e2)*(E[0]%E[0])). The equivalent expression by defining X = (e1%e2)*(E%E) is (see also Figure 126) that a Nominal_Basis_Submatrix X can be defined as projecting a Nominal_Basis_Matrix e1%e2 into another Nominal_Basis_Matrix E%E. Then, a Matrix a1 is projected to a Basis_Submatrix X(0) to form a Nominal_Submatrix a1*X(0).. Just as in the case of Basis_Subvector, we can switch from projecting Nominal_Basis_Matrix into Basis_Matrix to projecting Basis_Matrix into Nominal_Basis_Submatrix. For example, (in project: basis_examples) C0 B = a1 * ((E[0]%E[0])*(e1%e2)) + a2 * ((E[0]%E[1])*(e1%e2)) + a3 * ((E[1]%E[0])*(e1%e2)) + a4 * ((E[1]%E[1])*(e1%e2)); cout << B[0][0]<< endl; // { { 0.0, 1.0, 2.0, 3.0, 4.0, 5.0}, // {10.0, 11.0, 12.0, 13.0, 14.0, 15.0}, // {20.0, 21.0, 22.0, 23.0, 24.0, 25.0}, // {30.0, 31.0, 32.0, 33.0, 34.0, 35.0} } C0 Y = (E%E)*(e1%e2); // nominal Basis Submatrix C0 B1 = a1 * Y[0][0] + a2 * Y[0][1] // B1 == B + a3 * Y[1][0] + a4 * Y[1][1]; In the above (see also Figure 126), a Basis_Matrix E[0]%E[0] is projected into a Nominal_Basis_Submatrix e1%e2 to form a Basis_Submatrix (E[0]%E[0])*(e1%e2). A Matrix a1 is then projected by using the Basis Submatrix as a map to form a Nominal_Submatrix a1*((E[0]%E[0])*(e1%e2)).. The Nominal_Basis_Submatrix can be used in this case to generate an equivalent expression (see. Figure 126). A Nominal_Basis_Matrix E%E is projected into another Nominal_Basis_Matrix e1%e2 to form a Nominal_Basis_Submatrix (E%E)*(e1%e2). The Nominal_Basis_Submatrix is assigned to C0 variable Y. a Matrix a1 is projected into a Basis_Submatrix Y[0][0] to form a Nominal_Submatrix a1*Y[0][0].
54
C0 Type Objects
(e1%e2)*(E[0]%E[0])
E[0]%E[0] * e1%e2 a1*(e1%e2)*(E[0]%E[0]) 0.0 1.0 2.0 3.0 4.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 10. 11. 12. 13. 14. 15. 0.0 0.0 0.0 0.0 0.0 0.0 20. 21. 22. 23. 24. 25. 0.0 0.0 0.0 0.0 0.0 0.0 30. 31. 32. 33. 34. 35. 0.0 0.0 0.0 0.0 0.0 0.0 (e1%e2)*(E[0]%E[0]) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 *
a1
0.0 1.0 2.0 3.0 4.0 5.0 10. 11. 12. 13. 14. 15. 20. 21. 22. 23. 24. 25. 30. 31. 32. 33. 34. 35.
Figure 126 Projection of a Nominal_Basis_Matrix e1%e2 into a Basis_Matrix E[0]%E[0] to form a Basis_Submatrix (e1%e2)*(E[0]%E[0]). The Basis_Submatrix is used to project a Matrix a1 to form a Nominal_Submatrix a1*(e1%e2)*(E[0]%E[0]).
55
Chapter

X = (e1%e2)*(E%E)
* e1%e2
E%E
a1*X(0) 0.0 1.0 2.0 3.0 4.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 10. 11. 12. 13. 14. 15. 0.0 0.0 0.0 0.0 0.0 0.0 20. 21. 22. 23. 24. 25. 0.0 0.0 0.0 0.0 0.0 0.0 30. 31. 32. 33. 34. 35. 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 X(0) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 *
a1
0.0 1.0 2.0 3.0 4.0 5.0 10. 11. 12. 13. 14. 15. 20. 21. 22. 23. 24. 25. 30. 31. 32. 33. 34. 35.
Figure 127 Projection of a Nominal_Basis_Matrix e1%e2 into another Nominal_Basis_Matrix E%E to form a Nominal_Basis_Submatrix (e1%e2)*(E%E) = X. A Basis_Submatrix X(0) is used to project a Matrix a1 to form a Nominal_Submatrix a1*X(0).
56
C0 Type Objects
(E[0]%E[0])*(e1%e2)
e1%e2
* E[0]%E[0] a1*(E[0]%E[0])*(e1%e2) 0.0 0.0 1.0 0.0 2.0 0.0 3.0 0.0 4.0 0.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 10. 0.0 11. 0.0 12. 0.0 13. 0.0 14. 0.0 15. 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 (E[0]%E[0])*(e1%e2) 20. 0.0 21. 0.0 22. 0.0 23. 0.0 24. 0.0 25. 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 33. 0.0 34. 0.0 35. 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 *
a1
0.0 1.0 2.0 3.0 4.0 5.0 10. 11. 12. 13. 14. 15. 20. 21. 22. 23. 24. 25. 30. 31. 32. 33. 34. 35.
Figure 128 Projection of a Basis_Matrix E[0]%E[0] into a Nominal_Basis_Matrix e1%e2 to form a Basis_Submatrix (E[0]%E[0])*(e1%e2). The Basis_Submatrix is used to project a Matrix a1 to form a Nominal_Submatrix a1*((E[0]%E[0])*(e1%e2)).
57
Chapter

(E%E)*(e1%e2) = Y
e1%e2
* E%E a1*Y[0][0] 0.0 0.0 1.0 0.0 2.0 0.0 3.0 0.0 4.0 0.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 10. 0.0 11. 0.0 12. 0.0 13. 0.0 14. 0.0 15. 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Y[0][0] 20. 0.0 21. 0.0 22. 0.0 23. 0.0 24. 0.0 25. 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 33. 0.0 34. 0.0 35. 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 *
a1
0.0 1.0 2.0 3.0 4.0 5.0 10. 11. 12. 13. 14. 15. 20. 21. 22. 23. 24. 25. 30. 31. 32. 33. 34. 35.
Figure 129 Projection of a Nominal_Basis_Matrix E%E into another Nominal_Basis_Matrix e1%e2 to form a Nominal_Basis_Submatrix (E%E)*(e1%e2) = Y. The Basis_Submatrix Y[0][0] is used to project a Matrix a1 to form a Nominal_Submatrix a1*Y[0][0].
58
C0 Type Objects
In the case of the Subvector/Submatrix and the extended Basis the objects and their names become quite complicate. However, the design of these complicated objects are driven entirely by practical need in real-world applications. Chapter 4 and Chapter 5 on the finite element method make extensive use of them. We have introduced the primary objects including Scalar, Vector, and Matrix. These primary objects together with the utility objects make up the Basis expression up to the second order such as e and e e. They are very commonly used in mathematics and physics. Subvector and Submatrix are very popular in engineering literatures. The extended basis expression we just introduced is implemented in accordance with the spirit of the use of Subvector and Submatrix that higher order tensors are tacitly avoided, and the resultant objects are the Subvector and Submatrix.
59
Chapter
1.2 Applications of Computational Matrix Algebra

1.2.1 Solution of Simultaneous Linear Equations: LU Decomposition
The LU decomposition is the most basic matrix solver that has its root in Gauss elimination method. Two examples are shown in this section. The first example is the solution of electric current of an electric circuit. A Wheatstone bridge that can not be solved without resorting to solution of simultaneous equations is used. The second example is a two dimensional heat conduction problem using finite difference method. Compared to the electric circuit problem this example is not that trivial. The mathematical equations and the geometry of the problem have a much more complicated relation. This more complicated relation gives us an opportunity to underline the importance of the object-oriented programming. In many numerical problems, such as finite difference method, we are not just dealing with mathematics. There are a lot of non-numerical algorithms to be dealt with. As the current programming paradigm, object-oriented programming is strongly supported by C++ language.
Electric CircuitWheatstone Bridge

First lets look at total resistance of a parallel circuit as shown in Figure 1301. From Ohms law I1 = V / R1, I2 = V / R2, I3 = V / R3.
V +
I1 R1 I1+ I2 + I3
I2 R2
I3 R3
Figure 130 Parallel circuit with three resistances and a voltage source. The total resistance of this parallel circuit R|| is R|| = V / (I1 + I2 + I3) = V / (V / R1 + V / R2 + V / R3) Therefore, 1 / R|| = 1 / R1 + 1 / R2 + 1 / R3 Eq. 138
1. Examples taken from H.V. Malmstadt, C.G. Enke, and E.C. Toren, Jr., 1963, Eletronics for Scientists, Principles and Experiments for Those who Use instruments., W.A. Benjamin, Inc., New York, p. 536-540.
60
Applications of Computational Matrix Algebra

Step 2: 5 90 40 5 20 10 30 60 Step 3: Step 1: 5 x 5 = 2.5 5+5 Step 4: 40 5 100 20 90 25.4 40 2.5 + 20 + 47.4 = 69.9 40 5 20 90 x 100 90 + 100 = 47.4 2.5
Figure 131 Series and parallel resistances circuit reduction steps. This last formula is applicable for parallel resistances of two or more. A reduction procedure can be applied to a more complicated network of circuit (see Figure 131). In four steps, we can reduce the resistances of the circuit to a total resistance. The total current can be calculated. Then, we can work backwards to resolve the currents in each part of the circuit. However we may not be always very fortunate like in the above case. Some circuits can not be reduced merely by applying series resistances (simple addition) and parallel resistances formula (Eq. 138). For example, a Wheatstone bridge in Figure 132 provides a good example that can only be solved by a linear algebraic approach. The most important law with regarding to the calculation of the electric current in this circuit is the Kirchhoffs law. This law says (1) The sum of all the current flowing toward and out of a node is zero. (2) The sum of all the voltages from the source equals the sum of all the voltages drop in any loop. For solving this problem the direction of current I4 is temporary assumed to go from node 2 to node 3. Applying the first part of the Kirchhoffs law to node 1, node 2, and node 3, respectively I1 = I2 + I3 I2 = I4 + I5 I6 = I3 + I4
Eq. 139
From the second part of the Kirchhoffs law, note that V = IR, loops L1, L2, and L3, have the following relations.
61
Chapter

3 = 30 I3 + 100 I6 0 = -30 I3 +10 I4 +50 I2 0 = -200 I5 + 10 I4 + 100 I6
Eq. 140
Loop L1 has a source voltage of 3 volts. L2 and L3 are both 0. Eq. 139 and Eq. 140 is implemented in C++ as double v[6] = { 0.0, 0.0, 0.0, 3.0, 0.0, 0.0}; // (project: electric_circuit) C0 M(6, 6, (double*)0), V(6, v); Eqn0 = M[0], Eqn1 = M[1], Eqn2 = M[2], // aliases; copy constructor is always Eqn3 = M[3], Eqn4 = M[4], Eqn5 = M[5]; // by reference in VectorSpace Eqn0[0] = 1.0; Eqn0[1] = -1.0; Eqn0[2] = -1.0; // I1 = I2 + I3 Eqn1[1] = 1.0; Eqn1[3] = -1.0; Eqn1[4] = -1.0; // I2 = I4 + I5 Eqn2[5] = 1.0; Eqn2[2] = -1.0; Eqn2[3] = -1.0; // I6 = I3 + I4 Eqn3[2] = 30.0; Eqn3[5] = 100.0; // 3 = 30 I3 + 100 I6 Eqn4[2] = -30.0; Eqn4[3] = 10.0; Eqn4[1] = 50.0; // 0 = -30 I3 +10 I4 +50 I2 Eqn5[4] = -200.0; Eqn5[3] = 10.0; Eqn5[5] = 100.0; // 0 = -200 I5 + 10 I4 + 100 I6 C0 I = V / M; // default solver is the LU decomposition cout << I << endl; // {0.0351158, 0.0130105, 0.0221053, 1.26316e-2 // 0.0117474, 2.33684e-2}T Notice that Eqnn are aliases to the rows of Matrix M. This is in accordance with the implementation of VectorSpace C++ Library where the copy constructor, for example, invoked by the statement C0 Eqn0 = M[0];, is always performed as copy by reference. In other words, if the copy constructor is performed as by value, Eqnn will not be the aliases to the rows of Matrix M. However, assigning the coefficients of the matrix in the above is not only mathematically unsound, but also aesthetically un-pleasant. This can be improved by using the basis expression as (project: electric_circuit) C0 e(6); Eqn0 = 1.0*e[0] - 1.0*e[1] - 1.0*e[2]; Eqn1 = 1.0*e[1] -1.0* e[3] - 1.0*e[4]; Eqn2 = 1.0*e[5] - 1.0*e[2] - 1.0*e[3]; Eqn3 = 30.0* e[2] + 100.0* e[5];
4 100 200
// Basis of dimension = 6 // I1 = I2 + I3 // I2 = I4 + I5 // I6 = I3 + I4 // 3 = 30 I3 + 100 I6
3 Volts + L1
I6
3 30
L3
10
I5
2
I4 L2 I3
1 50
I1
I2
Figure 132 Calculation of electric current in Wheatstone bridge.

62

Eqn4 = -30.0* e[2] + 10.0* e[3] + 50.0* e[1] Eqn5 = -200.0* e[4] + 10.0* e[3] + 100.0*e[5]; // 0 = -30 I3 +10 I4 +50 I2 // 0 = -200 I5 + 10 I4 + 100 I6
These C++ statements are more readily to be compared with Eq. 139 and Eq. 140. Therefore, the programming task should be much transparent and the finishing code is much more readable and maintainable. The linear algebraic approach to the Wheatstone bridge certainly is also applicable to the previous less complicated circuit network. It is a more systematic approach to the general circuit network problem. We note by passing that the same idea on the partial elimination of a part of a circuit network can be studied systematically with graph theory. The graph theory are also used in developing efficient algorithms for the solution of sparse matrix1.
Finite Difference for 2-D Heat Conduction

For a two dimensional heat conduction problem2, the governing equation is
T 2 q T 2 -------- + -------- = --2 K x y 2
Eq. 141
where is the ratio for anisotropic heat conduction in two coordinate directions, q is the intensity of internal heat source, and K is the thermal diffusivity. The boundary conditions of this problem are shown in Figure 133. The geometry of the domain is a square area of 2 x 2 square units, with material properties of = 3, q / K = 16. The temperature condition on both the right-edge and the left-edge is fixed at zero. This kind of boundary condition is the so-called essential (or Dirichlet) boundary condition, also known as boundary condition of the first kind. In particular, the essential condition that vanishes (with zero value) is also known as the homogeneous boundary condition. The upper and lower-edges have boundary conditions of T / y = -T. Due to the symmetry of the problem only a quarter of the area needs to be modeled. It is shown in the right-hand-side of the Figure 133 that X-axis and the Y-axis are also the lines of symmetry. Therefore the lower-edge and the left-edge of this reduced model have the boundary conditions T / y = 0 and T / x = 0, respectively. This kind of boundary condition is the so-called natural (or Neumann) boundary condition, also known as boundary condition of the second kind. In the present case, the zero-flux on these boundaries physically means to impose insulation walls. The boundary condition on the upper-edge is T / y = -T, which mixes the independent variable, T, and the derivative of the independent variable, T / y , in linear combination. This kind of boundary condition is known as the mixed boundary condition or boundary condition of the third kind. The grid size for the above problem is h = 1/4. The finite difference approximation, with the central difference scheme, to the Eq. 141, is
1 q ---- ( T i + 1, j 2T i, j + T i 1, j ) + ---- ( T i, j + 1 2T i, j + T i, j 1 ) = --K h2 h2
Eq. 142
1. For example, Chapter 4 and Chapter 5 in S. Pissanetsky, 1984, Sparse Matrix Technology, Academic Press, New York. 2. Example from G.D. Smith, 1985, Numerical Solution of Partial Differential Equations: Finite Difference Methods, p. 242-245.
63
Chapter

T ------ = -T y
2 6 10 14 18 -14 3 7 11 0 4 1 5 9 13 17 2 6 10 14 18 3 7 11 15 19
B.C. of 3rd kind

T ------ = -T y
-4 -3 -2 -1 -5 -9 0 4 8 12 16 -12
-5 1 5 9 13 17 -13
T=0
-13 -17 -13 15 19 -15
T=0
T ------ = 0 8 x
B.C. of 1st kind T=0
12 16
T ------ = 0 y
B.C. of 2nd kind

T ------ = -T y
Figure 133 Boundary value problem of heat conduction with all three kinds of boundary conditions.
where i and j are row and column indices to the nodal positions. With the grid size (h = 1/4) and material properties ( = 3, and q / K = 16) give in the above, we obtain Ti+1,j + 3 Ti,j+1 + Ti-1,j +3 Ti,j-1 -8 Tij = -1 Eq. 143 leads to a finite difference stencil Eq. 143
0 3T i 1, j 0 0 3 0 = 1 T 1 8 1 T i, j 1 8T i, j T i, j + 1 = 1 0 3 0 0 3T i + 1, j 0
Eq. 144
For any node with Tij , the finite difference stencil in the left-hand-side maps the corresponding coefficients to the node around it. Although the finite difference stencil is extremely simple, which does not involve any highflown mathematics, treatment of the boundary conditions in finite difference method is notoriously non-systematic comparing to that in finite element method. The specification of boundary conditions in finite difference method is full of trivial details pertaining to each problem. We simply cant completely encapsulate these complexities from users.
64

Essential Boundary Conditions : We have homogeneous boundary conditions on the right-edge of the reduced model. Nothing needs to be done for them. If instead of the homogeneous boundary conditions we have essential boundary condition T = T. As a more general essential boundary condition, the left-hand-side vector V needs to be modified. For example, for node number 3 next to the right-edge (corresponding to the fourth row in the left-hand-side matrix, and the fourth element of the right-hand-side vector) we need to modify V[3] -= T where the essential boundary condition is to the right of the center-node(3). The right-node has the coefficient 1 in the finite difference stencil. Therefore, the coefficient 1 multiplies the essential boundary condition T is T. Then, shift to the right-hand-side of the equation number 3 by changing sign. We use replacement subtraction, -=, to subtract this amount out of the corresponding element in the right-hand-side vector, V[3]. Natural Boundary Conditions : The left-edge and the lower-edge of the reduced model has zero natural boundary conditions that can be considered as having the same nodes on the opposite side of the boundaries, considering the boundaries as lines of symmetry. For example, for node number 8, we can just specify both its left-node and right-node as node number 9. Therefore, the corresponding coefficients in the finite difference stencil will be all added to the left-hand-side matrix corresponding to the node number 9. In a more general case, for non-zero flux, the natural boundary condition will be - T / x = q, where q is the outward heat flux. In this more general case (T-9 - T9) / 2 h = 2 (T-9 - T9) = -q; i.e., T-9 = T9 -q / 2 This shows T-9 can be represented by T9 -q / 2. For the first term, T9, we can specify the left-node (-9) as the node number 9. For the second term, -q / 2, the right-hand-side vector corresponding to the centernode (8) needs to be modified as V[8] += q / 2. The quantity -q / 2 has been shifted to the right-hand-side vector V, therefore, a change of sign is necessary. Mixed Boundary Conditions : The boundary condition of third kind on top gives, for example, the current node number 0, (T-4 - T4) / 2h = 2 (T-4 - T4) = -T0 Therefore, T-4 = T4 - 0.5 T0 Eq. 145
with upper-node specified as node number 4. The left-hand-side matrix corresponding to the center-node (0) needs to be modified with M[0][0] -= 3.0*0.5. Notice that the coefficient in the finite difference stencil corresponding to the upper-node (-4) is 3.0. Program Listing 18 is the implementation of the above finite difference problem. The global left-hand-side matrix M and the global right-hand-side vector V are declared static that only one copy of them is allowed in the program for memory space consideration. At the heart of this code, we use object-oriented model to map the finite difference stencil in Eq. 144 to the global matrix and global vector. The class name is FD. Two private
65
Chapter
#include c0_init.h #define ndf 20

global left-hand-side matrix and global right-hand-side vector finite difference stencil mapping class reference to M and V pointers constructor mapping finite difference stencil center node left and bottom nodes right node and top nodes
static C0 M(ndf, ndf, (double*)0); static C0 V(ndf, (double*)0); class FD { // finite difference stencil C0 *the_M_ptr, *the_V_ptr; public: FD(C0* M_ptr, C0* V_ptr) : the_M_ptr(M_ptr), the_V_ptr(V_ptr) {} void add(int, int, int, int, int); }; void FD::add(int i0, int, i1, int i2, int i3, int i4) { (*the_M_ptr)[i0][i0] -= 8.0; (*the_V_ptr)[i0][i0] = -1; if(i1>=0) (*the_M_ptr)[i0][i1] += 1.0; if(i2>=0) (*the_M_ptr)[i0][i2] += 3.0; if(i3>=0) (*the_M_ptr)[i0][i3] += 1.0; if(i4>=0) (*the_M_ptr)[i0][i4] += 3.0; } int main() { FD fd(&M, &V); // I. Problem Definition Phase fd.add(0, 1, 4, 1, 4); fd.add(1, 0, 5, 2, 5); fd.add(2, 1, 6, 3, 6); fd.add(3, 2, 7, -1, 7); fd.add(4, 5, 8, 5, 0); fd.add(5, 4, 9, 6, 1); fd.add(6, 5, 10, 7, 2); fd.add(7, 6, 11, -1, 3); fd.add(8, 9, 12, 9, 4); fd.add(9, 8, 13, 10, 5); d.add(10, 9, 14, 11, 6); fd.add(11, 10, 15, -1, 7); fd.add(12,13,16,13,8); fd.add(13,12,17,14,9); fd.add(14,13,18,15,10); fd.add(15,14,19,-1,11); fd.add(16,17,12,17,12);fd.add(17,16,13,18,13);fd.add(18,17,14,19,14);fd.add(19,18,15,-1,15); for(int i = 0; i < 4; i++) M[i][i] -= 1.5; C0 T = V / M; for(int i = 0; i < 5; i++) { for(int j = 0; j < 4; j++) cout << ( << (i*4+j) << , << T[i*4+j] << ) ; cout << endl; } return 0; } // II. Solution Phase // III. Output Phase
first row second row third row fourth row fifth row B.C. of the third kind modification matrix solver output: (node#, T) (0, 3.06), (1, 2.91), (2, 2.42), (3, 1.50), (4, 3.72), (5, 3.53), (6, 2.92), (7, 1.80), (8, 4.17), (9, 3.95), (10, 3.26), (11, 2.0), (12,4.43),(13,4.20),(14,3.46),(15, 2.11), (16,4.52),(17,4.28),(18,3.52),(19,2.15)
Listing 18 Finite difference method for 2-d heat conduction problem (project: finite_difference). member data are C0 pointers to the global matrix and the global vector. The constructor of FD will establish the link between the class and the global objects by FD fd(&M, &V);. The only public member function in FD is FD::add(int, int, int, int, int). The FD::add(int, int, int, int, int) will assemble and add the corresponding coefficient to the global matrix and the global vector according to the finite difference stencil. The five integer arguments on the function interface represent the node numbers of the center, left, lower, right, and upper nodes. For a typical interior node 5, as shown in Figure 133, it can be written as fd.add(5, 4, 9, 6, 1); The node numbers 4, 9, 6, and 1 are the node numbers relative in their position to the node 5. For a homogeneous boundary condition node such as node 7
66

fd.add(7, 6, 11, -1, 3); We note that -1 is in place of the right-node relative to node 7. Any negative integer value can be used to suppress the corresponding coefficient in finite difference stencil to be added to the left-hand-side matrix. For the natural boundary conditions on the left-edge and lower-edge, considering the symmetry of node 8 for example, we can specify fd.add(8, 9, 12, 9, 4) The right-node (9) on the right-hand-side of the center-node (8) is also used as the left- node. In more general case, the natural boundary condition is - T / x = q, where q is the outward heat flux and (T-9 - T9) / 2 h = 2 (T-9 - T9) = -q; i.e., T-9 = T9 -q / 2. We may write in C++ together with FD::add() as fd.add(8, 9, 12, 9 4); V[8] += q /2; Similarly, for the boundary condition of the third kind, using node 1 as an example is fd.add(1, 0, 5, 2, 5); M[1][1] -= 1.5; Eq. 145 for current node 1 instead of 0 is T-5 = T5 - 0.5 T1 Eq. 147 Eq. 146
To represent the top node, node 5 is used for the upper-node that accounts for the first term in Eq. 147. In fact, the finite difference stencil coefficient of 3 on the top node position will be added to the left-hand-side matrix at the corresponding elements. For the second term, we need to modify with the finite difference stencil coefficient 3, then, multiply by the second term in Eq. 147 as 3 (- 0.5 T1) = -1.5 T1 that has not been accounted for by the class FD. In the second part of the Eq. 146 M[1][1] -= 1.5; performs this modification. The result of this problem is shown in Figure 134. The finite difference grid is drawn on the top of the box. The height of the shaded surface represents the value of the temperature. The basic feature of the solution is a warp-up surface along the center due to internal heat generation of q in Eq. 142. The homogenous boundary condition on the two sides of the problem pin down the surface to zero. The surface has a concave shape along the Y-axis, which means that the heat is lost both from the top and bottom that have the boundary condition of the third kind. From this simple example of implementing finite difference method, we can recognize that VectorSpace C++ Library made programming easy only on some mathematical related aspects. However, the finite difference method is NOT intensively mathematical. The most advantage comes from the data abstraction offered by C++ that we modeled the mapping of finite difference stencil to the global matrix and vector as a classFD. The experience learned from this example tells us to stay with the current industrial flag-ship general purpose language. We dont want to invest our effort on a specialized environment or, even worse, a mixed environment with more than one language to worry about. The applicability of a special purpose language on wide-ranging application areas is most likely to break down somewhere. The advantage of object-oriented programming is where most of the strength of applications using VectorSpace C++ Library comes from. In Chapter 3, examples on con-
67
Chapter

-1 1.0
Y
X 0.0
Y 0.0
1.0 4
-1 1 3 2 1 0 T
Figure 134 Solution of finite difference heat conduction problem, with internal heating, homogeneous boundary conditions on the two sides and boundary conditions of the third kind on the top and bottom. strained optimization and Chapter 4 and 5 on finite element method, we will show you that when an application problem is not trivial, the power from object-oriented programming becomes increasingly critical! Some remarks need to be made for this finite difference problem. Assuming we even use a laptop computer for this problem, we will get the result instantly. We will not want to do anything more. However, what if you want 1000 1000 grid or even greater to increase resolution? Computing time and memory space need to be considered. Firstly, we can use a sparse matrix instead of a full matrix to represent the global matrix M to save memory space. When the size (n n) of the global matrix grows, the required memory space will grow in the order of O(n2), and computing time of the matrix solver, in direct method, will grow in the order of O(n3). Secondly, the global matrix in this case is not only sparse but also diagonally dominant. An iterative method such as sucessive over relaxation (SOR) can be used to search for a feasible solution.1 How to tackle large-size problem in terms of programming method? The idea is simple. Get rid of everything offered by VectorSpace C++ Library! Simply go back to the basics by using plain C exclusively for ultimate speed. However, we can start from using VectorSpace C++ Library as the first step with a smaller number of variables like the above example. Then, in the second step, we can reverse engineer the program segment by segment for efficiency. This two-steps programming method is known as rapid proto-typing. In the first step, programming in VectorSpace C++ Library is much more mathematically friendly comparing to that in Fortran or
1. Introduction on the technical details of sparse matrix representation and sucessive over-relaxation (SOR) can be found in W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, 1992, Numerical Recipes in C, the Art of Scientific Computing, 2nd ed., Cambridge University Press, UK.
68

C. The proto-type program written with VectorSpace C++ Library is fully functional. The reverse engineering step is to reduce memory space usage and computing time. During the second step, the output from the prototype program can be used to debug the optimized production codes.
1.2.2 Linear Least Squares: Cholesky and QR Decompositions Car Sale Market Prediction
A regional sports car dealer would like to analyze their sale for market prediction.1 They first survey the number of sports cars sold and the population in each area of the local dealers (see TABLE 14.).
No. of Carsy 252 261 251 430 258 334 321 409 243 322 Populationx 34,549 53,943 42,983 102,832 12,034 9,023 20,934 73023 23,294 83,543
Location A B C D E F G H I J
TABLE 14. Car sale and population. As a first cut, more people would buy more cars seems to be a good assumption, if no better a priori information is available. We model the relationship of the number of cars sold and the population as a straight line y=a+bx Eq. 148
where y is the number of cars sold, x is the population, and a and b are two coefficients (intercept and slope) for a straight line. This problem is clearly over-determined. Ten data are available for just resolving two model parameters m = {a, b}T : the intercept a" and the slope b. The full rank over-determined least squares problem is explained as projection method in Figure 115. The normal equation of Eq. 127 on page 35 can be used to solve this problem. Program Listing 19 uses normal equations method to solve the least squares problem using Cholesky decomposition. This problem is not too ill-conditioned. In fact, using QR decomposition as discussed on page 36 produces the same result, with intercept a = 268.92 and slope b = 0.0078214. The fitted line is shown in Figure 135. The simple-minded assumption that the area with more population demand more
1. The abstract form of the problem in terms of fitting a straight line is presented in William Menke, 1984, Geophysical Data Analysis: Discrete Inverse Theory, Academic Press, Inc., p.10-11.
69
Chapter
#include include/vs.h int main() { double y[10] = {421, 569, 514, 1139, 287, 543, 615, 934, 327, 918}, a[10][2] = { {1.0, 34549.0}, {1.0, 53943.0}, {1.0, 42983.0}, {1.0, 102832.0}, {1.0, 12034.0}, {1.0, 9023.0}, {1.0, 20934.0}, {1.0, 73023.0}, {1.0, 23294.0}, {1.0, 83543.0}}; C0 A(10, 2, a[0]), Y(10, y); Matrix::Decomposition_Method = Matrix::Cholesky_Decomposition; C0 AtA = ((~A)*A), x = ((~A)*Y) / AtA; cout << x << endl; C0 cov_m = AtA.inverse(), N = A * cov_m *(~A); cout << cov_m << endl; cout << N << endl; return 0; }
y obsv. 10 rows of {1, x}
Cholesky decomposition normal equations mest = [ATA]-1 AT yobsv. = {268.92, 0.0078214}T covu m = ATA = unit covaraince matrix N = data resolution matrix
Listing 19 Least squares solution using normal equations method for fitting a straight line (project: least_squares_line_fitting). sports cars in general seems to be a good prediction model. However, there are two outliers with small population (Locations F and G) having stronger demand relative to the other areas. Some factors other than the population may have been overlooked. We will come back to this later.
y 1200
Location D Location F&G
1000
Number of Cars
800 600 400 200 0 20000 40000 60000 80000 100000 x
Population Figure 135 Least squares solution of fitting a straight line
The purpose of the present analysis is for market prediction. We may ask how well are the observational data y obsv. contribute to the model we built. Since we have built a model to predict the sale of cars y pred. according to
70

y pred. = A m est = A {[ATA]-1AT y obsv.} = N y obsv. N = A [ATA]-1AT is called the data resolution matrix. N maps the observations, y obsv., which we collected for the analysis to the market prediction, y pred., from the model we built. From Figure 136, a plot of the data resolution matrix, the diagonal elements corresponding to the collected data from each area of the local dealers, which show the importance to their own predictions. Each row of the data resolution matrix is a linear combination with other data to make the prediction. Therefore, if the diagonal element is the most dominant (closer to 1), it means that the prediction around this particular data can be more independently resolved. The data resolution matrix for the above problem shows that the most important data is at the location D. This is a location with the largest population and the greatest sports car demand. This particular information is not only consistent with the model assumption in the first place, but also helps spread out the data to cover a wider range. The location D is also expected to be resolved relatively more independently, because along the row and column number 4, this location is much greater than the rest in the row. However, along row number 4, location H and J are also relative large. From Figure 135, Locations H & J are those having large population and relative large car sale. These information are sort of duplicating what Location D stands for. We note that for an overdetermined problem, we have more data than the model parameters. Therefore the model parameters can always be perfectly resolved, while the data resolution matrix shows how well each datum is resolved J H D Location D H J
0.4 4 0.2 2 0
A B 2 C D 4 E I 8 H G 6 F F 6 C H 8 I
10 J
E 4 D Row number G
Column number
2 B
J 10A
Figure 136 Data resolution matrix. Recall probability and distribution from statistics. The peak width of a distribution can be measured by using a quadratic form of (y - y Exp)2, where superscript Exp denotes expectation. The variance 2 of a distribution is defined as
71
Chapter

2 =
( y yExp )2 P ( y ) dy
Eq. 149
where P(y) is the probability function of the distribution. For the correlation of any two data the covariance is defined similarly as cov y = cov ( yi, y j ) =
[ yi y iExp ] [ y j yjExp ]P ( y )d y1 d y N
Eq. 150
Assuming all data are not correlated (off-diagonals are all zeros) and have the same variance 2, the unit covariance matrix of model parameters [covu m] is defined as mapping the uncorrelated data covariance [cov y ] to model covariance [cov m ] with a normalized factor considered [covu m ] -2 A-g [cov y ] [A-g]T = [ATA]-1
g
Eq. 151
Recall from Eq. 128 that A-g = [ATA]-1AT is the generalized inverse for the over-determined problem, where Ais used to map y - y Exp in Eq. 150 to m. Then, [cov y ] is divided by the variance such that -2[cov y ] = I. The unit covarinace matrix [covu m] indicates how much error in the data, y - y Exp, will be amplified into the model parameters, m. The diagonals of the unit covariance matrix for this problem is diag [covu m ] = {0.325285, 1.08269e-10}T. Assuming that all car sales have money back guaranteed if there is a lemon product, and at the time of survey some sales are pending with the contingency of loan to be approved by banks. That is a lot of uncertainty may be contained in the collected data. We test with an extremely large variance of data d = 10 (d2 = 100) for this problem, and also assume that it is the same for every location for simplicity. After the mapping by Eq. 151, the diag [covu m ] remains small with intercept a = 268.92 5.7, slope b = 78.214e4 1.04e-4. It is not uncommon to think the two seaters probably are going to be most popular among wealthy people. The regional car dealer collected the average incomes of all locations as shown in TABLE 15. We assume that the car sale is also proportional to the average income, and build a modified model as y = a + b x1 + c x2
No. of Sale Location A B C D E F Population Average Income
Eq. 152
y
421 569 514 1139 287 543
x1
34,549 53,943 42,983 102,832 12,034 9,023
x2
39,028 40,239 32,984 45,032 56,029 92,034
TABLE 15. Car Sale, population and average income.
72

No. of Sale Location G H I J Population Average Income
y
615 934 327 918
x1
20,934 73,023 23,294 83,543
x2
89,034 59,032 40,932 38,094
TABLE 15. Car Sale, population and average income. Program Listing 110 (project: least_squares_plane_fitting) implemented the fitting of a plane in Eq. 152 with the normal equations method. The Matrix A now contains rows of coefficients of {1, x1, x2} instead of {1, x}. There is almost no difference for the rest of code comparing to Program Listing 19.
#include include/vs.h int main() { double y[10] = {421, 569, 514, 1139, 287, 543, 615, 934, 327, 918}, a[10][3] = { {1.0, 34549.0, 39028.0}, {1.0, 53943.0, 40239.0}, {1.0, 42983.0, 32984.0}, {1.0, 102832.0, 45032.0}, {1.0, 12034.0, 56029.0}, {1.0, 9023.0, 92034.0}, {1.0, 20934.0, 89034.0}, {1.0, 73023.0, 59032.0}, {1.0, 23294.0, 40932.0}, {1.0, 83543.0, 38094.0}}; C0 A(10, 3, a[0]), Y(10, y); Matrix::Decomposition_Method = Matrix::Cholesky_Decomposition; C0 AtA = ((~A)*A), x = ((~A)*V) / AtA; cout << x << endl; C0 cov_m = AtA.inverse(), N = A * cov_m *(~A); cout << cov_m << endl; cout << N << endl; return 0; }
y obsv. 10 rows of {1, x1, x2}
Cholesky decomposition normal equations m est ={-183.61, 9.9271e-3, 6.695e-3}T covu m = unit covaraince matrix N = data resolution matrix
Listing 110 Least squares solution using normal equations method for fitting a plane. The solution of the problem is m est ={-183.61, 9.9271e-3, 6.695e-3}T. The fitted plane is shown in Figure 137. Compared to Figure 135 Location G&F now do not look that much of outliers. With added parameter of average income, every data seems to be fitting into a plane much nicely. We did distill an important factor to this problem. On the other hand, it has been said that Give me five parameters, I can fit an elephant!. The more parameters we use to build a model the more nicely our data will fit the model. The nature of limited information is inherent from the data we collected. If we try to make to many inferences from the data, we may simply have the data beaten to death. The more statements we make the weaker they are. The data resolution matrix of the plane-fitting problem is shown in Figure 138. Now besides Location D the importance of Location F & G increase. They both explore the range of data to cover locations that have small population but strong demand for the sports cars. However F and G are also plagued by the same problem that D has and the problem getting even worse. That is when we examine row 6 and row 7, F and G Workbook of Applications in VectorSpace C++ Library
73
Chapter

Income
100000 80000 60000 40000 20000
0 1000 00 750 50
Number of Cars
500 250 0 0 25000 50000 75000 Population 100000
Figure 137 Least squares solution of fitting a plane. always have about the same height. Both the off-diagonals and the diagonals in data resolution matrix N are strongly coupled. This means that F and G can not be resolved independently. This result actually is not surprising. After all these two locations are small towns with predominantly high income folks living there. The information from these two locations duplicates each other. The diagonal vector of the unit covarinace matrix for this case is {1.77863, 1.39737e-10, 3.18123e-10}T. Assuming the same large variance for every data collected as in the straight-line fitting problem (d = 10; i.e., d2 = 100), the mapping of the variance from data to model parameter gives a = -183.606 13.33, b = 0.00992712 0.0001182, and c = 0.0066951 0.0001784. It seems the model we built is pretty robust even when it is subject to large error or uncertainty in the collected data.
Computer Tomography
Computer Tomography (CT) is commonly used in medical practice today. In this section we study a much simplified hypothetical acoustic tomographic1 problem to show the essence of mathematical treatment that CT uses. Sound devices are used instead of X-ray, and the source and receiver arrangement is very primitive compared to that of medical CT. Define the slowness s as the inverse of the velocity s=1/v Eq. 153
1. problem after William Menke, 1984, Geophysical Data Analysis: Discrete Inverse Theory, Academic Press, Inc., p.1114 and p. 182-186.
74

Location F & G
0.4 4 0.2 .2 0 6 2 4 4 6 8 10 2 8 10
Row number
Column number
Figure 138 Data resolution matrix for fitting a plane. The problem illustrated in Figure 139 has 10x10 blocks of size h (=1.0). 91 unshaded blocks have low velocity (v = 10), and 3 3 shaded blocks have high velocity (v = 12). The measurements are carried out either row-wise or column-wise. The travel time for the i-th measurement is ti = h si0 + h si1 + h si2 + ... + h si9 Eq. 154
Indices 0 to 9 denote the consecutive blocks along a row or a column, and h is the size of the blocks. Therefore, we will solve the slowness s of each block given travel time t from measurements. There are 100 slowness variables associated with each block and we only have twenty measurements (10 row-wise + 10 column-wise measurements). The problem is underdetermined. The 10x10 blocks is completely surrounded and dominated by low velocity blocks (v = 10). We work on a solution that is a perturbation to an entirely low velocity solution
s est = s a priori + A-g [t - A s
a priori]
= s a priori + A-g t
Eq. 155
where t = t - A s a priori. We define s = s est - s a priori, where s a priori is a vector of length = 100 with the slowness of 0.1=1/10 (low velocity is a priori information for all blocks). t is a vector of length = 20 for the travel time measurements. The A-g in Eq. 155 is defined as A-g = AT[AAT]-1 Eq. 156
That is Eq. 156 is the generalized inverse for the underdetermined problem. The derivation of Eq. 156 is discussed in the following. Since the problem is underdetermined, a common a priori assumption is that the solution is simple that its Euclidean norm, ||s||2 = sT s = si, is minimized. The problem solved is a standard constrained optimization problem (detailed in Chapter 2): Minimized ||s||2 subject to the constraint c
= t - A s = 0.
75
Chapter

0 10 1 11 21 31 41 51 61 71 81 91 2 12 22 32 42 52 62 72 82 92 3 13 23 33 43 53 63 73 83 93 4 14 24 34 44 54 64 74 84 94 5 15 25 35 45 55 65 75 85 95 6 16 26 36 46 56 66 76 86 96 7 17 27 37 47 57 67 77 87 97 8 18 28 38 48 58 68 78 88 98 9 19 29 39 49 59 69 79 89 99
Source
20 30 40 50 60 70 80 90
Receiver
Figure 139 10x10 blocks with 91 unshaded blocks as low velocity (10) and 3x3 crossed hatched blocks as high velocity (12). Twenty travel time data are collected from either row-wise or column-wise arrangement of devices of source and receiver.
An objective functional can be constructed using Lagrange multiplier method (with m as number of solution varialbes and n number of observations)
T 2+ c =
f(s) = s
i=1
s i2 +
i=1
i t i
Aij sj
j=1
Eq. 157
where is the Lagrange multiplier. Take the derivative of the objective function f with respect to s and set this derivative to zero for minimization
f ----------- = s k
m
i=1
s i 2 ----------- s i s k
i=1
s j i A ij ----------- = 2s k s k
j=1
i Aik
i=1
= 0
Eq. 158
Written in matrix form Eq. 158 is 2 s = AT Therefore, we are facing solving two equations simultaneously Eq. 159
76

2 s - AT = 0 = t As As in a constrained optimization problem the Lagrange multiplier can be eliminated from the equations. Substituting s in the first equation into the second equation yields A AT / 2 = t Hence, = 2[A AT]-1t, and is eliminated by substituting this back to the first equation s = AT[AAT]-1 t = A-g t This is the proof for Eq. 156 that defines the generalized inverse of an underdetermined problem as A-g = AT[AAT]-1. Program Listing 111 implements the tomographic problem. The kernel of this program is the class Ray. This class maps the blocks, that the sound passes, to a global matrix according to Eq. 154. The first ten arguments are the block numbers. The last argument is the size of the block. The computation of the solution in terms of slowness is straight forward according to Eq. 155. // minimum Euclidean norm solution minimum Euclidean norm solution C0 ds = (~A)*(dT /(A*(~A)); slowness cout.precision(6); output slowness solution
for(int i = 0; i < 10; i++) for(int j = 0; j < 10; j++) cout << (0.1+ds[i*10+j]) << , ; cout << endl; } // model resolution matrix C0 R = (~A)*(A*(~A)).inverse()*A; // output row # 33 of the model resolution matrix in matrix form cout << { << endl; for(int i 0; i < 10; i++) cout << {; for(int j = 0; j < 10; j++) { if(j != 9) cout << (R[33][i*10+j]) << endl << , ; else cout << (R[33][i*10+j]); } if(i != 9) cout << }, << endl; else cout << } << endl; } cout << } << endl; return 0; }
model resolution matrix output row # 33 of model resolution matrix and rearranged the values in the row into a matrix form (10x10).
Listing 111 Acoustic tomography with row and column measurements (project: acoustic_tomography_1).
77
Chapter
For an underdetermined problem, there are more model parameters than data. Therefore, data t can always be fitted to the model perfectly. We may want to know, contrary to the over-determined problem, how well are the model parameters sest been resolved. sest = A-g t =( AT[AAT]-1) Astrue = R strue where R= AT[AAT]-1 A Eq. 160
is the model resolution matrix that indicates the closeness of our solution sest to the true solution strue. As for the data resolution matrix, the diagonals of the model resolution matrix show the importance of a particular model parameter. If the model resolution matrix is close to I, it means the model parameters are nearly perfectly resolved. In Program Listing 111, we chose to print out information corresponding to the center block of the high velocity (block number 33 in Figure 139), the row number 33 of the model resolution matrix.
#include include/vs.h #define ndf 100 #define neqn 20 static C0 A(neqn, ndf, (double*)0); static int eqn = 0; class Ray { C0 *the_A; public: Ray(C0* A) : the_A(A) {} void add(int, int, int, int, int, int, int, int, int, int, double = 1.0); }; void Ray::add(int i0, inti1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, double h) { (*the_A)[eqn][i0] = (*the_A)[eqn][i1] = (*the_A)[eqn][i2] = (*the_A)[eqn][i3] = (*the_A)[eqn][i4] = (*the_A)[eqn][i5] = (*the_A)[eqn][i6] = (*the_A)[eqn][i7] = (*the_A)[eqn][i8] = (*the_A)[eqn][i9] = h; eqn++; }; int main() { C0 dT(neqn, (double*)0); Ray ray(&A); // add rows for(int i = 0; i < 10; i++) ray.add(i*10, // add columns for(int i = 0; i < 10; i++) ray.add(i, i+10, i+20, i+30, i+40, i+50, i+60, i+70, i+80, i+90); for(int i = 0; i < neqn; i++) dT[i] = 0.0; dT[2] = dT[3] = dT[4] = dT[12] = dT[13] = dT[14] = -0.05; i*10+1, i*10+2, i*10+3, i*10+4, i*10+5, i*10+6, i*10+7, i*10+8, i*10+9); // Map blocks that pass by sound to global matrix
100 blocks 20 measurements global matrix global matrix row index class Ray to map blocks in the sound path to the global matrix. 10 block numbers, and grid size add() used to map to global matrix
increase global matrix row index measurements: t // add row path
// add column path
assign measurement values
78

The result is shown in Figure 140. The upper-left part shows the true model of slowness that we used to cook up the row-wise and column-wise travel time data for the Vector dT. The lower-left is the solution using the minimum Euclidean norm solution method from Eq. 155. The right-hand-side shows the resolution of the center block of the 3 3 high velocity blocks. The importance of number 33 model parameter only has a value of 0.19. Two big side-lobes of the left-hand side graphs are parallel to the row-wise and column-wise measurements that pass through block number 33. From Eq. 154, it is mostly natural that the solution of block number 33 is a linear combination of the row-wise and column-wise blocks. Hence, block number 33 can not be resolved independently from these blocks. The solution of slowness compared to the true model of slowness in Figure 140 is not very good, and we simply need more data to resolve the model better. One way to get a better resolution is to include measurements on 45o and 135o angles. 38 more measurements can be added on top of the row-wise and column-wise measurements, and we should have the resolution of the model improve to some extent. Program Listing 112 is the implementation of the tomographic problem using 20 row-wise and column-wise measurements plus 38 measurements at 45o and 135o angles. The added member function of class Ray is Ray::add(), which allows variable length of arguments. Considering the additional measurements taking on NE-SW direction, this may involve minimum 1 to maximum 10 blocks in program specification. The double arguments immediately follow the int arguments of block numbers are the distance of ray that passes through the corresponding blocks. For example at 45o , the ray passes through a single block, e.g., number 0, can be simply written as (*ray).add(0, sqrt2); where sqrt2 has been defined by the macro definition to have the value of NE-SW diagonal has 10 blocks on the way. It can be written as (*ray).add(90, sqrt2, 81, sqrt2, ... , 9, sqrt2); with ten int values and all given the length of 2 . This function is very versatile. In fact, if you begin to consider ray path at different angles other than those we have just taken (0o, 45o, 90o, and 135o), the distances of a ray which passes through each block can be different and the different distances need to be specified with different blocks. The solution is shown in the left-hand-side of Figure 141. Compared to the solution in Figure 140 the model has been improved at least incrementally (since you can obtain even better resolution by adding more data to the solution procedure). The model resolution matrix shown in the right-hand-side of Figure 141 shows that the over-all importance of the data has been significantly improved. The value of the importance increases from 0.19 to 0.44 a more than two times growth. However, the model resolution matrix also shows the number of dependent blocks increases. The new pattern looks like the British national flag with 45o and 135o stripes that corresponding to the newly slanted measurement paths superposed on the previous vertical/horizontal cross pattern
2 . The ray that passes through the
79
Chapter
1.2.3 Eigenvalue Problems Buckling of a Rod

Consider a beam1 distorted from its natural state with the application of a load value P as in Figure 142.
2 2 4 4 6 8 10 0 6 8 10 0 0.11 0 0.1 0. 0.09 0. 0.08
True model of slowness row #33 from Model Resolution Matrix
0.2 2 0.1 1 2 0 -0.1 .1 2 4 6 8 8 10 10 6 4
2 2 4 4 6 8 10 0 6
Solution of slowness
8 10 0 0.11 0 0.1 0. 0.09 0. 0.08
Figure 140 Solution of acoustic tomographic problem using minimum solution Euclidean norm. The high velocity area is smeared out to have two pairs of big side-lobes. The left-hand side is from the 33-rd row of model resolution matrix.
1. For an introduction on Bernoulli-Euler beam theory see J.M. Gere and S.P. Timoshenko, 1984, Mechanics of Materials, 2nd ed., Wadsworth, Inc., California, p. 553-8, and p. 680 for a brief historical account.
80

#include include/vs.h #define ndf 100 #define neqn 58 static C0 A(neqn, ndf, (double*)0); static int eqn = 0; class Ray { // Map block that pass by sound to global matrix C0 *the_A; public: Ray(C0* A) : the_A(A) {} void add(int, int, int, int, int, int, int, int, int, int, double = 1.0); void add(int, double, int = -1, double = 0.0, int = -1, double = 0.0, int = -1, double = 0.0, int = -1, double = 0.0, int = -1, double = 0.0, int = -1, double = 0.0, int = -1, double = 0.0, int = -1, double = 0.0, int = -1, double = 0.0); }; void Ray::add(int i0, inti1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, double h) { (*the_A)[eqn][i0] = (*the_A)[eqn][i1] = (*the_A)[eqn][i2] = (*the_A)[eqn][i3] = (*the_A)[eqn][i4] = (*the_A)[eqn][i5] = (*the_A)[eqn][i6] = (*the_A)[eqn][i7] = (*the_A)[eqn][i8] = (*the_A)[eqn][i9] = h; eqn++; }; void Ray::add(int i0, double h0, int i1, double h1, int i2, double h2, int i3, double h3, int i4, double h4, int i5, double h5, int i6, double h6, int i7, double h7, int i8, double h8, int i9, double h9) { // variable length arguments version (*the_A)[eqn][i0] = h0; if(i1 >= 0) (*the_A)[eqn][i1] = h1; if(i3 >= 0) (*the_A)[eqn][i3] = h3; if(i5 >= 0) (*the_A)[eqn][i5] = h5; if(i7 >= 0) (*the_A)[eqn][i7] = h7; if(i9 >= 0) (*the_A)[eqn][i9] = h9; eqn++; } void add_rowcol(Ray *ray, C0 *dT) { // add blocks along rows and columns; 20 measurements // add rows for(int i = 0; i < 10; i++) (*ray).add(i*10, // add columns for(int i = 0; i < 10; i++) (*ray).add(i, i+10, i+20, i+30, i+40, i+50, i+60, i+70, i+80, i+90); for(int i = 0; i < neqn; i++) (*dT)[i] = 0.0; (*dT)[2] = (*dT)[3] = (*dT)[4] = (*dT)[12] = (*dT)[13] = (*dT)[14] = -0.05; } #define sqr2 1.414213562 void add_diagonals(Ray *ray, C0 *dT) { // add 45o blocks; 38 measurements // add NE-SW ray paths (*dT)[eqn] = 0.0; (*ray).add(0, sqrt2); i*10+1, i*10+2, i*10+3, i*10+4, i*10+5, i*10+6, i*10+7, i*10+8, i*10+9); if(i2 >= 0) (*the_A)[eqn][i2] = h2; if(i4 >= 0) (*the_A)[eqn][i4] = h4; if(i6 >= 0) (*the_A)[eqn][i6] = h6; if(i8 >= 0) (*the_A)[eqn][i8] = h8;
100 blocks 58 measurements global matrix global matrix row index class Ray to map blocks in the sound path to the global matrix. 10 block numbers, and grid size variable length of block numbers and grid size
add() used to map to global matrix
increase global matrix row index variable length of arguments version
measurements: t add row path
add column path
assign measurement values
45o measurements
81
Chapter
(*dT)[eqn] = 0.0; (*ray).add(10, sqrt2, 1, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(20, sqrt2, 11, sqrt2, 2, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(30, sqrt2, 21, sqrt2, 12, sqrt2, 3, sqrt2); (*dT)[eqn] = sqrt2*(1.0/12.0-1.0/10.0); (*ray).add(40, sqrt2, 31, sqrt2, 22, sqrt2, 13, sqrt2, 4, sqrt2); (*dT)[eqn] = 2.0*sqrt2(1.0/12.0-1.0/10.0); (*ray).add(50, sqrt2, 41, sqrt2, 32, sqrt2, 23, sqrt2, 14, sqrt2, 5, sqrt2); (*dT)[eqn] = 3.0*sqrt2(1.0/12.0-1.0/10.0); (*ray).add(60, sqrt2, 51, sqrt2, 42, sqrt2, 33, sqrt2, 24, sqrt2, 13, sqrt2, 6, sqrt2); (*dT)[eqn] = 2.0*sqrt2(1.0/12.0-1.0/10.0); (*ray).add(70, sqrt2, 61, sqrt2, 52, sqrt2, 43, sqrt2, 34, sqrt2, 25, sqrt2, 16, sqrt2, 7, sqrt2); (*dT)[eqn] = sqrt2*(1.0/12.0-1.0/10.0); (*ray).add(80, sqrt2, 71, sqrt2, 62, sqrt2, 53, sqrt2, 44, sqrt2, 35, sqrt2, 26, sqrt2, 17, sqrt2, 8, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(90, sqrt2, 81, sqrt2, 72, sqrt2, 63, sqrt2, 54, sqrt2, 45, sqrt2, 36, sqrt2, 27, sqrt2, 18, sqrt2, 9, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(91, sqrt2, 82, sqrt2, 73, sqrt2, 64, sqrt2, 55, sqrt2, 46, sqrt2, 37, sqrt2, 28, sqrt2, 19, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(92, sqrt2, 83, sqrt2, 74, sqrt2, 65, sqrt2, 56, sqrt2, 47, sqrt2, 38, sqrt2, 29, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(93, sqrt2, 84, sqrt2, 75, sqrt2, 66, sqrt2, 57, sqrt2, 48, sqrt2, 39, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(94, sqrt2, 85, sqrt2, 76, sqrt2, 67, sqrt2, 58, sqrt2, 49, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(95, sqrt2, 86, sqrt2, 77, sqrt2, 68, sqrt2, 59, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(96, sqrt2, 87, sqrt2, 78, sqrt2, 69, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(97, sqrt2, 88, sqrt2, 79, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(98, sqrt2, 89, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(99, sqrt2); // add NW-SE ray paths (*dT)[eqn] = 0.0; (*ray).add(9, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(8, sqrt2, 19, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(7, sqrt2, 18, sqrt2, 29, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(6, sqrt2, 17, sqrt2, 28, sqrt2, 39, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(5, sqrt2, 16, sqrt2, 27, sqrt2, 38, sqrt2, 49, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(4, sqrt2, 15, sqrt2, 26, sqrt2, 37, sqrt2, 48, sqrt2, 59, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(3, sqrt2, 14, sqrt2, 25, sqrt2, 36, sqrt2, 47, sqrt2, 58, sqrt2, 69, sqrt2); (*dT)[eqn] = sqrt2(1.0/12.0-1.0/10.0); (*ray).add(2, sqrt2, 13, sqrt2, 24, sqrt2, 35, sqrt2, 46, sqrt2, 57, sqrt2, 68, sqrt2, 79, sqrt2); (*dT)[eqn] = 2.0*sqrt2*(1.0/12.0-1.0/10.0); (*ray).add(1, sqrt2, 12, sqrt2, 23, sqrt2, 34, sqrt2, 45, sqrt2, 56, sqrt2, 67, sqrt2, 78, sqrt2, 89, sqrt2); (*dT)[eqn] = 3.0*sqrt2*(1.0/12.0-1.0/10.0); (*ray).add(0, sqrt2, 11, sqrt2, 22, sqrt2, 33, sqrt2, 44, sqrt2, 55, sqrt2, 66, sqrt2, 77, sqrt2, 88, sqrt2, 99, sqrt2); (*dT)[eqn] = 2.0*sqrt2*(1.0/12.0-1.0/10.0); (*ray).add(10, sqrt2, 21, sqrt2, 32, sqrt2, 43, sqrt2, 54, sqrt2, 65, sqrt2, 76, sqrt2, 87, sqrt2, 98, sqrt2);
135o measurements
82

(*dT)[eqn] = sqrt2(1.0/12.0-1.0/10.0); (*ray).add(20, sqrt2, 31, sqrt2, 42, sqrt2, 53, sqrt2, 64, sqrt2, 75, sqrt2, 86, sqrt2, 97, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(30, sqrt2, 41, sqrt2, 52, sqrt2, 63, sqrt2, 74, sqrt2, 85, sqrt2, 96, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(40, sqrt2, 51, sqrt2, 62, sqrt2, 73, sqrt2, 84, sqrt2, 95, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(50, sqrt2, 61, sqrt2, 72, sqrt2, 83, sqrt2, 94, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(60, sqrt2, 71, sqrt2, 82, sqrt2, 93, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(70, sqrt2, 81, sqrt2, 92, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(80, sqrt2, 91, sqrt2); (*dT)[eqn] = 0.0; (*ray).add(90, sqrt2); eqn++; } int main() { C0 dT(neqn, (double*)0); Ray ray(&A); add_rowcol(&ray, &dT); add_diagonals(&ray, &dT); // minimum Euclidean norm solution C0 ds = (~A)*(dT /(A*(~A)); cout.precision(6); for(int i = 0; i < 10; i++) for(int j = 0; j < 10; j++) cout << (0.1+ds[i*10+j]) << , ; cout << endl; } // model resolution matrix C0 R = (~A)*(A*(~A)).inverse()*A; // output row # 33 of the model resolution matrix in matrix form cout << { << endl; for(int i 0; i < 10; i++) cout << {; for(int j = 0; j < 10; j++) { if(j != 9) cout << (R[33][i*10+j]) << endl << , ; else cout << (R[33][i*10+j]); } if(i != 9) cout << }, << endl; else cout << } << endl; } cout << } << endl; return 0; } // add row-wise and column-wise blocks // add 45o angle measurements
minimum Euclidean norm solution slowness output slowness solution
model resolution matrix output row # 33 of model resolution matrix and rearranged the values in the row into a matrix form (10x10).
Listing 112 Acoustic tomography with row-wise and column-wise plus 45o and 135o measurements (project: acoustic_tomography_2).
83
Chapter

row #33 from Model Resolution Matrix
2 2 4 6 8 6 10 0 0.11 0 0.1 0 0.09 0. 0.08
0.2 2 0.1 1 0 -0.1 .1 2 4 6 8 10 10 8 6 4 2
Solution of slowness
4
8 10 0
Figure 141 Acoustic tomographic problem with row-wise, column-wise, 45o, and 135o measurements. For a small P the rod deforms as uniform compression. When P is increased to a critical load value PEuler, known as the Euler load, named after the greatest mathematician of all times, the rod will buckle upward or downward if the rod is fixed in plane. M P P v P > PEuler Figure 142 Buckling of a rod. The rod buckled up when the load is increased beyond the Euler load. P < PEuler Let be the angle between tangent of the rod and the horizontal axis, and v the lateral deflection. The curvature of the rod is related to the bending moment M and the flexure rigidity of the beam EI as
d2 v M -------- = ----EI dx 2
Eq. 161
where E is the Youngs modulus and I is the moment of inertia. For static equilibrium, M = Pv. We have
EIv = Pv
Eq. 162
The length of the rod is L, and the boundary conditions at the ends are v(0) = v(L) = 0. We use finite difference to approximate numerically the second order derivative in the left-hand-side of Eq. 162. The rod is subdivided into n segments, and the size h of each segment is h = L / n. The central difference stencil for the second order derivative of v evaluated at node xi is
84

v i + 1 2v i + v i 1 d2 v -------- ( x i ) = -----------------------------------------2 dx h2
Eq. 163
x = xi
Substituting Eq. 163 into Eq. 162, we have the standard matrix form of an eigenvalue problem Av=v This equation has the same form as Eq. 134 on page 37. Program Listing 113 is the implementation of this problem with VectorSpace C++ Library. Class FD is the finite difference stencil for Eq. 163, which is very similar to class FD in Program Listing 18. The computation of eigenvalues and eigenvectors is straight forward as has been discussed on page 37. Only the first three eigenvalues and eigenvectors are reported to a file rod.out. The results of the eigen computation are shown in Figure 143. These are actually eigenvalues ({} = {i22}, i = 1, 2, ...) and eigenfunctions ({sin ix}, i = 1, 2, ...) of the operator -d2/dx2. In the present case, the smallest eigenvalue and its eigenvector is the most stable. The second and third modes on the right-hand-side of Figure 143 are only attainable if the inflection points are supported at the beginning of an incremental loading procedure. The branching diagram in the upper left-hand-side of Figure 143 is actually quite un-realistic that the eigenvalues exist only in some discrete values and the value of v can not be determined. This is caused by using Eq. 162, which is a linearization of a fully non-linear problem of 1
'' + sin = 0
Eq. 164
The fully non-linear version of the branching diagram shows pitch fork bifurcation2. The eigenvalues of the linearized problem (Eq. 162)3 are only the branching points of the fully non-linear problem (Eq. 164). If the resultant matrix is symmetrical, the so-called generalized eigenvalue problem and quadratic eigenvalue problem can be solved by the symmetrical eigenvalue solver provided by VectorSpace C++ Library. In many practical engineering applications, the symmetrical solver is sufficient. An example of how a generalized eigenvalue problem can be solved using the symmetrical eigenvalue solver is shown on page 234.
1. I. Stakgold, 1979, Greens Functions and Boundary Value Problems, John Wiley & Sons, New York, p. 572-576. 2. J.E. Marsden and T. J.R. Hughes, 1983, Mathematical Foundations of Elasticity, Prentice-Hall, Inc., Englewood Cliffs, N.J., p.427-429. 3. I. Stakgold, 1979, Greens Functions and Boundary Value Problems, John Wiley & Sons, New York, p. 584.
85
Chapter
#include "include/vs.h" #include "assert.h" #include <iostream.h> #include <fstream.h> #include <stdlib.h> #include <iomanip.h> static ofstream ofs("rod.out", ios::out | ios::trunc); #define n 40 // number of segments #define P 1.0 // loading #define L 1.0 // length of the rod #define E 1.0 // Youngs modulus #define I 1.0 // moment of inertia static C0 M(n-2, n-2, (double*)0); static const double h = L / n; static const double k = - E * I / (P*h*h); class FD { // finite difference stencil C0 *the_M; public: FD(C0 *M) : the_M(M) {} void add(int, int, int); }; void FD::add(int i_center, int i_left, int i_right) { // finite difference stencil if(i_center >= 0) (*the_M)[i_center][i_center] += 2.0*k; if(i_left >= 0) (*the_M)[i_center][i_left] += -1.0*k; if(i_right >= 0) (*the_M)[i_center][i_right] += -1.0*k; } int main() { FD fd(&M); fd.add(0, -1, 1); // first node for(int i = 1; i < n-3; i++) fd.add(i, i-1, i+1); fd.add(n-3, n-4, -1); // last node Eigen a(M); C0 lambda = a.Eigenvalues(), x = a.Eigenvectors(); ofs << lambda: << lambda[0] << , << lambda[1] << , labmda[2] << endl; for(int i = 0; i < 3; i++) ofs << x(i) << endl; ofs.close(); return 0; }
number of segments loading length of rod Youngs modulus moment of inertia global matrix finite difference grid size class of finite difference stencil
(-1, 2, -1) finite difference stencil
first node middle nodes last node eigenvalues eigenvectors output first three eigenvalues and eigenvectors
Listing 113 Finite difference implementation for buckling of rod eigenvalue analysis (project: rod_buckling)
86

Eigenvalues
Eigenvectors
0.2
v branching diagram of a linearized problem
0.15
0.1
|| 1 2 3
0.05
10
15
20
25
30
35
0.2
0.1
10
15
20
25
30
35
fully non-linear branching diagram
-0.1
v
branching points
-0.2
inflection points
0.2
|| 3
0.1
5 -0.1
10
15
20
25
30
35
-0.2
Figure 143 First three eigenvalues and eigenvectors of the rod buckling problem using finite difference method.
Singular Value Decomposition and the Principal Components Analysis

Three simple examples of principal components analysis using singular value decomposition, namely (1) scholastic achievement analysis, (2) stock market analysis, and (3) geomorphology analysis, are implemented with VectorSpace C++ library in this section. Scholastic Achievements Analysis:1 of 20 students over 6 subjects are given in TABLE 16. The covariance matrix defined in Eq. 150 on page 72 is cov y = cov ( yi, y j ) =
[ yi y iExp ] [ y j yjExp ]P ( y )d y1 d y N
1. A. Jennings and J.J. McKeown, 1992, Matrix Computation, 2nd ed., John Wiley & Sons, Inc., New York, p.181-184.
87
Chapter
In this case, the indices i and j span over six subjects that the twenty students have taken, and N = 20 is the number of the students. We first subtract each column with its average value (in place of yExp in Eq. 150). The results from a matrix X of size 20 6. The covarinace matrix, consistent with Eq. 150, is XTX/(N-1) of size 6 6 .
Student 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Average Math. 50 17 60 49 64 35 51 57 99 36 43 36 37 70 78 66 49 68 43 44 52.6 English 45 29 51 70 63 55 67 69 77 60 33 39 30 48 48 72 63 50 71 34 53.7 Physics 41 40 49 46 52 49 48 60 58 81 40 34 47 58 60 66 55 68 68 33 52.65 Geography 45 43 69 63 53 44 64 52 86 74 52 55 40 55 70 73 69 62 71 51 59.55 French 46 40 58 71 68 66 64 73 81 68 45 49 52 72 61 83 55 55 71 42 61.0 Art 30 22 51 44 61 47 57 60 67 52 43 44 35 60 41 66 57 47 59 53 49.8
TABLE 16. Scores and average of 6 subjects from 20 students. The implementation of this problem is shown in Program Listing 114. The result of the covariance matrix is
336.2 139.8 83.1 TX ) ( N 1) = C = (X 137.8 134.5 127.4
139.8 233.4 111.5 130.5 168.4 135.1
83.1 111.5 154.7 93.9 104.8 71.1
137.8 130.5 93.9 154.1 95.0 91.4
134.5 168.4 104.8 95.0 161.6 114.3
127.4 135.1 71.1 91.4 114.3 141.4
Eq. 165
88

It is argued that the largest diagonal element is in the first row and column which means the mathematics has the largest variance, and all of the off-diagonal elements, Eq. 165, are positive which means that any student who is good at one subject has the tendency to do well in other subjects. The greatest off-diagonal is 168.4 which means that the strongest correlation between any two subjects are French and English. The result of the singular value
#include "include/vs.h" #define N 20 #define sub_no 6 static double s[N][sub_no] = { {50.0, 45.0, 41.0, 45.0, 46.0, 30.0}, {17.0, 29.0, 40.0, 43.0, 40.0, 22.0}, {60.0, 51.0, 49.0, 69.0, 58.0, 51.0}, {49.0, 70.0, 46.0, 63.0, 71.0, 44.0}, {64.0, 63.0, 52.0, 53.0, 68.0, 61.0}, {35.0, 55.0, 49.0, 44.0, 66.0, 47.0}, {51.0, 67.0, 48.0, 64.0, 64.0, 57.0}, {57.0, 69.0, 60.0, 52.0, 73.0, 60.0}, {99.0, 77.0, 58.0, 86.0, 81.0, 67.0}, {36.0, 60.0, 81.0, 74.0, 68.0, 52.0}, {43.0, 33.0, 40.0, 52.0, 45.0, 43.0}, {36.0, 39.0, 34.0, 55.0, 49.0, 44.0}, {37.0, 30.0, 47.0, 40.0, 52.0, 35.0}, {70.0, 48.0, 58.0, 55.0, 72.0, 60.0}, {78.0, 48.0, 60.0, 70.0, 61.0, 41.0}, {66.0, 72.0, 66.0, 73.0, 83.0, 66.0}, {49.0, 63.0, 55.0, 69.0, 55.0, 57.0}, {68.0, 50.0, 68.0, 62.0, 55.0, 47.0}, {43.0, 71.0, 68.0, 71.0, 71.0, 59.0}, {44.0, 34.0, 33.0, 51.0, 42.0, 53.0}}; int main() { C0 X(N, sub_no, s[0]), a(sub_no, (double*)0); for(int i = 0; i < sub_no; i++) { for(int j = 0; j < N; j++) a[i] += X[j][i]; a[i] /= (double)N; for(int j = 0; j < N; j++) X[j][i] -= a[i]; } C0 C = (~X)*X / (N-1.0); cout << C << endl; SVD svd(C); C0 sigma = svd.Singularvalues(), x = svd.U(); cout << sigma << endl; cout << x(0) << endl; cout << x(1) << endl; return 0; }
number of students number of subjects
matrix X, and average a compute average Xj = Xj - a C = ( XTX) / (N-1) singular value decomposition {799.2,178.4,87.4, 58.9,41.6,16.0}T {.518, .474, .305, .360, .401, .351}T {-.815, .389, .350, .027, .235, .077}T
Listing 114 Principal components analysis of scholastic achievement of 20 students on 6 subjects (project: scholastic_achievements).
89
Chapter
decomposition is shown in right-hand-side column of Program Listing 114. The first and second eigenvalues account for 799.2 / 1181.3 = 67.7 %, and 178.4 / 1181.3 = 15.1 % of the total variance, respectively. The all positive values of the first eigenvector reconfirm the tendency that a student who does well in one subject will do very well in the other. The second eigenvector indicates, again, that mathematics is the subject that has the largest variation of student performance. Stock market1 weekly return rates from five companies give a covariance matrix as
Allied Chemical Allied Chemical DuPont Union Carbide Exxon Texaco 1.000 0.577 0.509 0.387 0.462
DuPont 0.577 1.000 0.599 0.398 0.322
Union Carbide 0.509 0.599 1.000 0.436 0.426
Exxon 0.387 0.398 0.436 1.000 0.523
Texaco 0.462 0.322 0.426 0.523 1.000
Program Listing 115 is the implementation of the singular decomposition of this problem.The proportion of total variance in the first two eigenvalues are 2.857 / 5.084 = 56 %, and 0.809 / 5.084 = 16 %. The rest of the eigenvalues sum to only 18% of the total variance. The first two eigenvectors are x1 = {0.464, 0.457, 0.470, 0.421, 0.421}T, and x2 = {-0.240, -0.509, -0.260, 0.526, 0.582}T. The first eigenvector is called the market component which is an equal weighted vector of the five stocks. This component reflects the general condition of the market. The second eigenvector is called the industry component which reflects the contrast between the chemical and the oil industries. Although this problem can be analyzed using eigenvalue solution as well, in general the singular value decomposition is recommended for the principal components analysis in statistics. Mountain Morphology :2 can be analyzed using singular value decomposition. An array of 14 mountain profiles are digitized with 11 points on each profile and the profiles are shown in Figure 144. A matrix S contains this digitized data is of size 14 11. S can be decomposed using singular value decomposition as S= (U ) VT = CF Eq. 166
1. example taken from B.N. Datta, 1995, Numerical Linear Algebra and Applications, Brooks/Cole Publishing Company, p.399-400. 2. example data digitized from William Menke, 1984, Geophysical Data Analysis: Discrete Inverse Theory, Academic Press Inc., Orlando, p.167-170.
90

#include "include/vs.h" int main() { double cv[5][5] = {{1.000, 0.577, 0.509, 0.387, 0.462}, {0.577, 1.000, 0.599, 0.389, 0.322}, {0.509, 0.599, 1.000, 0.436, 0.426}, {0.387, 0.389, 0.436, 1.000, 0.523}, {0.462, 0.322, 0.426, 0.523, 1.000}}; C0 cm(5, 5, cv[0]); SVD a(cm); C0 sigma = a.Singularvalues(), U = a.U(); cout << sigma << endl; cout << U(0) << endl; cout << U(1) << endl; return 0; }
covaraince matrix
singular value decomposition {2.857, 0.809, 0.540, 0.452, 0.343}T {0.464, 0.457, 0.470, 0.421, 0.421}T {-0.240,-0.509,-0.260,0.526,0.582}T
Listing 115 Stock-market analysis by singular value decomposition (project: stock_market).
10
11
12
13
Figure 144
14
14 mountain profiles.
where C = U is called factor loadings, and rows of F are the factors. Program Listing 114 implements
this problem using VectorSpace C++ Library.

The first three singular values in the percentage of the total are 8.56 / 16.4 = 52.2%, 2.41 / 16.4 = 14.7%, and 1.91 / 16.4 = 11.6% These three components together account for 78.5 % of the total. The three dominant factors are shown in Figure 145. The first factor (52% strong) is called the average mountain, while the second and third factors are called skewness and sharpness, respectively, for obvious reasons. The first three components of factor loadings C = U for each row represent the proportion of the first three factors in each mountain profile. These three numbers of each mountain profile are plotted in a 3-D graph as shown in Figure 146. This figure shows a quantitative method to analyze geometrical objects. For example,
91
Chapter
#include "include/vs.h" static double v[14][9] = {{0.29, 0.54, 0.83, 1.07, 1.42, 1.11, 0.82, 0.55, 0.25}, {0.56, 0.82, 0.93, 1.05, 1.23, 1.06, 0.93, 0.82, 0.55}, {0.17, 0.30, 0.87, 1.12, 1.37, 1.08, 0.86, 0.34, 0.14}, {0.23, 0.79, 1.34, 1.35, 1.26, 1.08, 0.93, 0.82, 0.55}, {0.58, 0.83, 0.96, 1.08, 1.22, 1.37, 1.39, 0.86, 0.34}, {1.35, 1.34, 1.35, 1.17, 1.15, 1.19, 0.82, 0.58, 0.25}, {0.23, 0.55, 0.84, 1.27, 1.25, 1.24, 1.39, 1.40, 1.10}, {0.02, 0.04, 0.13, 0.58, 1.36, 0.55, 0.12, 0.03, 0.02}, {0.12, 0.30, 0.58, 0.84, 1.38, 0.83, 0.57, 0.28, 0.13}, {0.09, 0.23, 0.82, 1.37, 0.81, 0.28, 0.11, 0.12, 0.02}, {0.02, 0.16, 0.16, 0.27, 0.85, 1.42, 0.85, 0.28, 0.12}, {0.01, 0.03, 0.02, 0.11, 0.12, 0.28, 0.58, 1.39, 0.58}, {0.59, 1.37, 0.58, 0.28, 0.15, 0.14, 0.03, 0.02, 0.01}, {0.03, 0.03, 0.17, 0.58, 1.38, 1.27, 1.24, 0.96, 0.68}}; int main() { C0 S(14, 9, v[0]); SVD a(S); C0 sigma = a.Singularvalues(), U = a.U(), V = a.V(), C(14, 9, (double*)0); for(int i = 0; i < 14; i++) for(int j = 0; j < 9; j++) C[i][j] = U[i][j] * sigma[j]; C0 Sp(14, 9, (double*)0); for(int i = 0; i < 14; i++) for(int j = 0; j < 3; j++) Sp[i] += C[i][j] * V(j); cout << "sigma: " << endl << sigma << endl; cout << V << endl << C << endl << Sp << endl; return 0; }
singular value decomposition C; factor loadings spectral representation by the first three principal components singular values; = {8.56, 2.41, 1.91, 1.33, 0.74, 0.55, 0.37, 0.29, 0.26}T
Listing 116 Principal components (Factor) analysis of mountain geomorphology (project: mountain_geomorphology).
F1: Average Mountain
F2: Skewness
F3: Sharpness
Figure 145 Three dominant principal components from the 14 mountain profiles.
92
F3(sharpness)
12
7 13 6 2 5 14 11 4 F1(average mountain) 1 9 3
F2(skewness)
10 8
Figure 146 The factor loadings of first three factors (F1, F2, and F3) of the 14 mountain profiles.
you can use this analysis to discriminate different mountain shapes caused by different erosional agents (ice, water, or wind). Alternatively, you may want to differentiate shapes that are caused by the same agent but at different stages (from juvenile to mature stages) of geomorphological evolution. On the other hand, since the first three factors account for most (78.5%) of the information of the 14 mountain profiles. We can use the first three columns of the factor loadings and the first three factors to reproduce the 14 mountain profiles as Si3 = Ci 1 F1 + Ci 2 F2 + Ci 1 F3, (i = 1, ..., N)
where superscript 3 denotes that the profile is only composed of only three most significant factors. The reconstructed mountain profiles from the three dominant factors are plotted against the digitized profiles in Figure 147. A close fit is more likely to happen if the original mountain is close to the average mountain.
93
Chapter
Figure 147 Mountain profiles shown in gray lines are reconstructed from three dominant factors. They are plotted against the solid lines that represents the original mountain profiles.
94

Linear Algebra

Загружено:

Сведения о документе

Исходное описание:

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Linear Algebra

Загружено:

Авторское право:

Доступные форматы

CHAPTER

Computational Linear Algebra Using C0 Type Objects

1.1 C0 Type Objects

Workbook of Applications in VectorSpace C++ Library 1

1 Computational Linear Algebra Using C0 Type Objects

Listing 11 Scalar object constructor (project: scalar_examples)

Workbook of Applications in VectorSpace C++ Library

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

VectorSpace C++ library definition

Workbook of Applications in VectorSpace C++ Library

Operators and Functions

Listing 12 Assignment by reference and assignment by value (project: scalar_examples).

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects a

Scalar label double

Workbook of Applications in VectorSpace C++ Library

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

equal not equal greater or equal less or equal greater less

power square root exponent log sin cos

Workbook of Applications in VectorSpace C++ Library

Listing 13 Dedicated constructor of a Vector object (project: vector_examples).

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

Workbook of Applications in VectorSpace C++ Library

VectorSpace C++ library definition

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

Operators and Functions

Listing 14 Assignment operators with different types of operand.

Workbook of Applications in VectorSpace C++ Library

return a Matrix return a Matrix

equal not equal greater or equal less or equal greater less

1 Computational Linear Algebra Using C0 Type Objects

1.0 a 2.0 3.0 4.0 b 5.0 6.0 7.0

1.0 2.0 3.0 4.0 5.0 6.0 7.0 a&b d c

1.0 3.0 5.0 7.0 2.0 4.0 6.0 8.0

1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 c && d

Figure 16 Column-concatenation and one-by-one column concatenation of two Vectors.

Workbook of Applications in VectorSpace C++ Library

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

Workbook of Applications in VectorSpace C++ Library

Therefore, for p = 1, the 1-norm (or sum norm) is defined as

For p = 2, the 2-norm (or the Euclidean norm) is defined as

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

For, p = , the infinite (or maximum norm), is

Workbook of Applications in VectorSpace C++ Library

m01 m02 m03

m11 m12 m13 m14 m20 m21 m22

m23 m24 m30

m31 m32 m33 m34

Figure 18 Memory space management of a Matrix.

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

TABLE 12. Three popular mathematical notations for tensor product.

Workbook of Applications in VectorSpace C++ Library

Listing 16 Dedicated constructors of Matrix objects (project: matrix_examples).

Workbook of Applications in VectorSpace C++ Library

1 Computational Linear Algebra Using C0 Type Objects

Strings in C0 virtual constructor for Matrix object.