Linear Algebra

0 Axb
1

Content
Linear regression for calibration
Simple linear regression
Weighted linear regression

Error estimation
MATLAB
Solve the linear regression

Multidimensional case

Your future salary..

Years of education, worth doing it?
how much are you going to earn?
6

x 10

income ($)

0
0

10
years of education (y)

15

20

Formulate a model
Salary (mostly) depends on
education
Salary = a * x_education + b
6

x 10

salary 12227+1833 xeducation

income ($)

Cannot explain
single salaries,
but a reasonable
approximation

0
0

10
years of education (y)

15

20

Formalism
Dependent / independent variables,
parameters
Education?
Salary?
a, b?

Independent chosen =
education
Dependent = Salary
Parameters = a, b

How were a and b calculated?

Measurement devices (GC, LC, spectophotometer,

)

A compound of interest produces a

readout
(External) calibration is used to
quantify response (slope & offset)

Linear model
The signal intensity (y) is a linear
function of the concentration (x):

y=ax
+b
Measuring two different
concentrations, a and b can be
determined:
y1 = a x1 + b

y2 = a x2 + b
7

a=

y1 - y2
x1 - x2

, b=

y2x1 - y1x2
x1 - x2

Is that all?
What about random noise (errors)?

y=ax
+b+e
Has a major impact on the slope and
offset:
y1 = ax1 + b + e1
y2 = a x2 + b + e2

a=

y1 + e1 - y2 - e2
x1 - x2

, b=

y2x1 - y1x2 - x1e2 + x2e1

x1 - x2

More points, repetitions

Determine a and b from more
measurements

Normal inverse does not work

anymore
9

Calculate the best estimate

y = b + ax

The best solution for a and b is the one with

minimal residuals between measurements and line
Define a value for the deviations most commonly
sum of squared errors
nobs
SSQ =
10

( yi - ( axi
i =1

+ b) )

Calculate minimum
Mathematical formulation
nobs

= argmin ( yi

a,b
b
i =1

( a xi

+ b) )

Solve?
Determine minimum of the function SSQ ->
derivative = 0
SSQ
SSQ
= 0,
=0
a
b

nobs

b=

i =1

nobs

yi

i =1

xi2

nobs

nobs

i =1

11

nobs

xi2

i =1

nobs

xi

i =1

nobs

-
xi

i =1

nobs

xi yi
2

, a=

nobs xi yi i =1

nobs

nobs

i =1

xi2

nobs

i =1

nobs

xi

yi

i =1
2

nobs

-
xi

i =1

Statistics

Or.. How accurate are your results?

12

Perfect measurements?
What assumptions were implicitly
made?
Accuracy of the measurements?
Each point had the same absolute error

0.1 1, 1 1, 10 1

Mostly, we assume observe a constant

relative
error..
0.1 measurement
0.01, 1 0.1
, 10 1

13

Or a mixture of both = heteroscedastic

Weighing of the measurements

Each e has a normal distribution
e1 : N (0, s12)

y1 = a x1 + b + e1
y2 = ax2 + b + e2
M
ynobs = a xnobs + b + enobs

e2 : N (0, s 22)
M
2
enobs : N (0, s nobs
)

Define residuals normalized to the

standard deviation
ax +b- y
r1 =

r2 =

s1
ax2 + b - y2

M
14

rnobs =

s2
axnobs + b
s nobs

Weighted linear regression

nobs
yi

= argmin

a,b
b

i =1

- ( axi + b)

si

Same procedure as previously

SSQ
SSQ
= 0,
=0
a
b
nobs

b=

i =1

s i- 1yi

nobs

i =1

s i- 1xi2

nobs

i =1

s i- 1xi

nobs

i =1

nobs

- 1
- 1 2
si si xi - si- 1xi
i =1
i =1
i =1

nobs

nobs

s i- 1xi

nobs

yi

15

Good luck

, a=

i =1

si- 1

nobs

i =1

s i- 1xi yi

nobs

i =1

si- 1xi

nobs

i =1

nobs

- 1
- 1 2
si si xi - si- 1xi
i =1
i =1
i =1

nobs

nobs

s i- 1yi
2

Apply linear algebra

Vector and Matrix formalism

Parameter vector (a and b) =

Measurement vector

y1

y= M
y

ynobs

Linear model:

16

x1 1

x
1

y=
b+e

M
M

x
1

nobs

continued
Sum of squared errors (weighted)
x1 1

x
1

A =

M
M

x
1

nobs

r = ( Ab - y) S-

0.5

s12

s 22

S =

SSQ r r r
2

yS y 2 A S y+ A S A
T

17

snobs

Solution
Derivative
SSQ
2 AT S 1y + 2 AT S 1A 0

A S A AT S 1y
T

Same as
nobs

b=

i =1

s i- 1yi

i =1

s i- 1xi2

nobs

i =1

si- 1xi

nobs

i =1

nobs

- 1
- 1 2
si si xi - si- 1xi
i =1
i =1
i =1

nobs

18

nobs

nobs

s i- 1xi
2

nobs

yi
, a=

i =1

s i- 1

nobs

i =1

s i- 1xi yi

nobs

i =1

si- 1xi

nobs

i =1

nobs

- 1
- 1 2
si si xi - si- 1xi
i =1
i =1
i =1

nobs

nobs

s i- 1yi
2

Construct a vector & matrix

Standard concentrations
x = [ 0; 1; 2; 3; 5; 10; 20 ]
Measurements
y = [0.07; 5.30;
10.09;16.29;27.65; 54.97;96.46]
100
90

measurement signal (au)

80
70
60
50
40
30
20
10

19

10
15
standard concentration (mmol/L)

20

MATLAB
Construct matrix A
A = [ x, ones( size(x) ) ]

20

x1 1

x
1

A =

M
M

x
1

nobs

Calculate a & b
A A AT y
T

ab = (A'* A)^-1 * A' * y

a = ab(1)
b = ab(2)
21

Visualization
Plot the measurements & regression line
plot( x, y, sb );
hold on
plot( [0; x(end)], [ b; x(end)*a+b ], r: )
100
90

measurement signal (au)

80
70
60
50
40
30
20
10

22

10
15
standard concentration (mmol/L)

20

But does not tell you

concentration +/- confidence /
standard error

23

Error in slope and offset prediction confidence

The calibration line is based on

errorprone measurements
There is some uncertainty in a and b
120

AT A A T y
1

d d
Sy
dy dy

a 4.874 0.156
b 1.570 1.370
24

measurement signal (au)

100
80
60
40
20
0
-20

6
8
10
12
14
standard concentration (mmol/L)

16

18

20

Weighted regression
Motivation
Measurement error is relative (weight =
1/x)
Measurement error is known
(repetitive measurements)

25

MATLAB
Construct matrix A

x1 1

x
1

A =

M
M

xnobs 1

A = [ x, ones( size(x) ) ]

Construct matrix S

s12

s 22

S=

s nobs

S = diag( [0.5; 0.6; 0.7; 0.8; 1.0; 1.5;

2.5].^2 )
26

Result weighted regression

Different calibration line
Closer to low concentrated observations
Less close for high concentrations

a 5.145
b 0.342

120

measurement signal (au)

100

80

60

Different to:
40

a 4.874
b 1.570

20

27

6
8
10
12
14
standard concentration (mmol/L)

16

18

20

Error in a, b & prediction confidence

120

measurement signal (au)

100
80
60
40

a 5.145 0.158
b 0.342 1.110

20
0
-20

28

6
8
10
12
14
standard concentration (mmol/L)

16

18

20

Multidimensional
More than one observable (y)
More than 1 slope / offset
Examples:
Determine coefficients in chemical
reactions
aC 6H12O6 + bNH 4+ + cH 2O + dH + + e HCO3- + f C 3H 5O3- + CH 1.8O0.5N 0.2 = 0
Determine HP coefficients Black-Box

model

29

aC 6H12O6 + bNH 4+ + cH 2O + dH + + e HCO3- + f C 3H 5O3- + CH 1.8O0.5N 0.2 = 0

Step 1: Define parameter vector

Step 2: Generate matrix A

C

30

6
H
12

O
6
0
N

charge 0

0 0 0
4
0
1
1

2
1
0
0

1 1 5 1.8

0 3 3 0.5
0 0 0 0.2

1 1 1 0

a

b
c

d 0
e

f
1

6
H
12

O
6
0
N

charge 0

0 0 0
4
0
1
1

1 1 5 1.8

0 3 3 0.5
0 0 0 0.2

1 1 1 0

2
1
0
0

a

b
c

d 0
e

f
1

Step 3: Separate known (independent)

and unknown (dependent
aC 6H12O6 + bNH 4+ + cH 2O + dH + + e HCO3- + f C 3H 5O3- + CH1.8O0.5N 0.2 = 0

3
0 1 0 0 0

0 1 0 1 1
E5555555555555
F
A
d
6 0 0 0
12 4 2 1

6 0 1 0

31

b
c

d
e

5
1.8

3 0.5

0
0.2

1 0
E555555
F
A
f
3

f

1

0
0

0
0

1 a
3 1

1 b
5
1.8

6
0
1
0
3
c
3
0.5

1
0 1 0 0 0 d
0
0.2

E5F

1 0 x f
e
0 1 0 1 1 E5
F E555555
F
E5555555555555
F x
d
A
A
f
d
6 0 0 0
12 4 2 1

1 a
3 1

1 b
5
1.8

6 0 1 0 3 c 3 0.5

1
0 1 0 0 0 d
0
0.2

E5F

1 0 x f
e
0 1 0 1 1 E5
F E555555
F
E5555555555555
F x
d
A
A
f
d
6 0 0 0
12 4 2 1

Step 4: Solve

A d xd A f x f 0
1

xd A d A f x f
32

Herbert Pirt

Determined Case 3 experiments

Strain

33

mu
1/h-1

qS
mmol/g/h

qP
mmol/g/h

S1

0.2084

4.9947

1.6525

S2

0.1515

3.7447

1.4274

S3

0.0957

2.495

1.1612

Step 1: Define parameter vector

a
b b

m

Step 2: Generate matrix A

qS ,1 a 1 b qP ,1 m
qS ,2 a 2 b qP ,2 m
qS ,3 a 3 b qP ,3 m

34

A 2

3

qP ,1
qP ,2
qP ,3

1
1

Step 3: Separate known (independent)

and unknown (dependent
Already separated..

xf = qS

Step 4: Solve

A qS
A
35

qS

More experiments
Strain

36

Lac
addition
mM

mu
1/h-1

qS
qP
mmol/g/h mmol/g/h

S1

0.2084

4.9947

1.6525

S2

0.1515

3.7447

1.4274

S3

0.0957

2.495

1.1612

S1

50

0.2098

5.0141

1.388

S2

50

0.1518 Assume
3.7624 a, b1.2008

S3

50

S1

150

0.1982

5.0141

1.0454

S2

150

0.139

3.7619

0.9052

independent
LAC
0.0946
2.511 of0.9796
ms dependent of LAC

qS ,1 a 1 b qP ,1 m0
qS ,2 a 2 b qP,2 m0
qS ,3 a 3 b qP,3 m0
qS ,4 a 4 b qP,4 m50
qS ,5 a 5 b qP,5 m50
M
qS ,12 a 12 b qP ,12 m150

Step 1: Define parameter vector

37

a
b

m0

m50
m150

Step 2: Generate matrix A

a
b

q S A m1

m2
m3

38

1,1

1,2
1,3

2,1

2,2

2,3

3,1
3,2

q p ,1,1
q p ,1,2
q p ,1,3
q p ,2,1
q p ,2,2
q p ,2,3
q p ,3,1
q p ,3,2

1 0 0
1 0 0

1 0 0

0 1 0
0 1 0

0 1 0
0 0 1

0 0 1

Step 3: Separate known (independent)

and unknown (dependent
Already separated..

xf = qS

Step 4: Solve

A qS
A

qS
Inverse cannot be calculated

39