Advanced Techniques for Engineering

Analysis
Forecasting Models
GC University Lahore
2
What is Forecasting?
Forecasting is attempting to predict the future
i.e. expected state of an event, condition or
variable at some future time index
Decision makers want to reduce the risk by
predicting future events or values of related
variables
3
Forecasting Methods
Forecasting methods
Qualitative
intuitive, educated guesses that may or may not depend on
past data
Quantitative
based on mathematical or statistical models
4
Forecasting Methods contd
We will consider two types of forecasts based on
mathematical models:
Regression forecasting
Time Series forecasting
5
Regression Forecasting
We predict by modeling the relationship
between dependent variable and independent
variables
We use some latest observations to fit a least
square regression line to data and then use it for
forecasting purpose.
6
Regression Forecasting contd
Suppose that Y is a variable of interest, and X
1
,
, X
p
are explanatory or predictor variables
such that
Y = f(X
1
, , X
p
; )
f is the mathematical model that determines
the relationship between the variable of
interest and the explanatory variables
| = (
0
, ,
m
)' are the model parameters.
7
Regression Forecasting contd
1. We choose to minimize
2. We use least squares estimation to estimate
3. Then we forecast y as
. )

; , , (
1

p
x x f y =

( )

= =
=
n
i
n
i
i ip i i
e x x f y
1 1
2
2
1
. ) ; , , (
8
Regression Forecasting contd
Examples of Regression Models:
) s t variable independen
with two model linear ( . 5
) model quadratic ( . 4
) model growth l exponentia ( ) exp( . 3
) model regression linear simple ( . 2
) model mean constant ( . 1
2 2 1 1 0
2
2 1 0
1 0
1 0
0
i i i i
i i i i
i i i
i i i
i i
x x y
x x y
x y
x y
y
c | | |
c | | |
c | |
c | |
c |
+ + + =
+ + + =
+ =
+ + =
+ =
9
Constant Mean Regression
Suppose that the y
i
s are a constant value plus noise:
y
i
= |
0
+
i
,
i.e., | = |
0
.
We want to determine the value of |
0
that minimizes
. ) ( ) (
1
2
0 0
=
=
n
i
i
y S | |
) , 0 ( ~
2
o c N
i
10
Constant Mean Regression contd
Taking the derivative of S(|
0
) gives
Finally setting this equal to zero leads to
Hence the sample mean is the least squares
estimator for |
0
.
. ) ( 2 ) (
1
0 0
=
=
'
n
i
i
y S | |
.
1

1
0
y y
n
n
i
i
= =

=
|
y
11
Constant Mean Regression contd
Example: y
i
= |
0
+
i
,
y
98.30963
99.18569
101.2684
97.52997
103.4013
98.84521
111.1842
98.70812
93.08922
5 . 102

0
= = y |
80
85
90
95
100
105
110
115
120
1 2 3 4 5 6 7 8 9
y
12
Simple Linear Regression
Consider the model
y
i
= |
0
+ |
1
x
i
+
i
,
i.e., | = (|
0
, |
1
).
We want to determine the values of |
0
and |
1
that
minimize
. ) ( ) , (
1
2
1 0 1 0
=
=
n
i
i i
x y S | | | |
13
Simple Linear Regression contd
Setting the first partial derivatives equal to zero gives

=
=
= =
c
c
= =
c
c
n
i
i i i
n
i
i i
x x y S
x y S
1
1 0 1 0
1
1
1 0 1 0
0
. 0 ) ( 2 ) , (
0 ) ( 2 ) , (
| | | |
|
| | | |
|
14
Simple Linear Regression contd
Solving for |
0
and |
1
leads to the least squares
estimates
( )( )
( )
.

1 0
2
2
1
x y
x x n
y x y x n
i i
i i i i
| |
|
=

=


15
Simple Linear Regression contd
Example:
x y
1 2.6
2.3 2.8
3.1 3.1
4.8 4.7
5.6 5.1
6.3 5.3
0
1
2
3
4
5
6
0 1 2 3 4 5 6 7
X
Y
16
Simple Linear Regression contd
Example continued:
x y
1 2.6
2.3 2.8
3.1 3.1
4.8 4.7
5.6 5.1
6.3 5.3 93 . 3
85 . 3
99 . 109
16 . 103
6 . 23
1 . 23
6
2
=
=
=
=
=
=
=

y
x
x
y x
y
x
n
i
i i
i
i
68 . 1

58 . 0

0
1
=
=
|
|
17
Simple Linear Regression contd
Example (continued):
Regression equation:
x y 58 . 0 68 . 1 + =
0
1
2
3
4
5
6
0 1 2 3 4 5 6 7
Y
Yhat
18
General Linear Regression
Consider the linear regression model
or
where x
i
= (1, x
i1
, , x
ip
)' and | = (|
0
, , |
p
)'.
i ip p i i i
x x x y c | | | | + + + + + =
2 2 1 1 0
i i i
y c +
'
= x
19
General Linear Regression contd
Suppose that we have n observations of y i.e. y
i ,
i=1,2,..n. We introduce matrix notation and define
y = (y
1
, , y
n
)', = (
1
, ,
n
)',
Note that yis n 1, is n 1, and X is n (p + 1).
.
1
1
1
1
2 21
1 11
(
(
(
(

=
np n
p
p
x x
x x
x x

X
20
General Linear Regression contd
Then we can write the regression model as
y has a mean vector and covariance matrix
given by
where I is the n n identity matrix.
X y + =
, ) var(
] [
2
I y
X y
o =
= E
21
General Linear Regression contd
Note that by var(y), we mean the matrix
This is a symmetric matrix.
|
|
|
|
|
.
|

\
|
) var( ) , cov( ) , cov(
) , cov( ) var( ) , cov(
) , cov( ) , cov( ) var(
2 1
2 2 1 2
1 2 1 1
n n n
n
n
y y y y y
y y y y y
y y y y y

22
General Linear Regression contd
In matrix notation, the least squares criterion can be
expressed as minimizing
Setting , the least squares estimator is given
by
. ) (

1
y X X X
' '
=

. ) ( ) ( ) ( ) (
1
2

=

'
=
'
=
n
i
i i
y S X y X y x

0
) (
=
c
c
|
| S
Prediction for Y is then given by
. ) (

1
y X X X X X y
' '
= =

23
General Linear Regression contd
Example: Simple Linear Regression
. ) (

1
y X X X
' '
=

x y
1 2.6
2.3 2.8
3.1 3.1
4.8 4.7
5.6 5.1
6.3 5.3
0
1
2
3
4
5
6
0 1 2 3 4 5 6 7
X
Y
24
General Linear Regression contd
Example: Simple Linear Regression (p = 1)
x y
1 2.6
2.3 2.8
3.1 3.1
4.8 4.7
5.6 5.1
6.3 5.3
|
|
|
|
|
.
|

\
|
=
3 . 6 1
6 . 5 1
8 . 4 1
1 . 3 1
3 . 2 1
1 1
X
|
|
|
|
|
.
|

\
|
=
3 . 5
1 . 5
7 . 4
1 . 3
8 . 2
6 . 2
y
25
General Linear Regression contd
Example: Simple Linear Regression (p = 1)
|
.
|

\
|
=
'
99 . 109 1 . 23
1 . 23 6
X X
( )
|
.
|

\
|

=
'

047 . 0 18 . 0
18 . 0 87 . 0
1
X X
( )
|
.
|

\
|
=
' '

58 . 0
68 . 1
1
y X X X
26
Homework-1
1. Suppose that the following data represent the total costs and
the number of units produced by a company.
a. Graph the relationship between X and Y.
b. Determine the simple linear regression line relating Y to X.
c. Predict the costs for producing 10 units.
d. Compute the SST, SSR, SSE, R and R
2
. Interpret the value
of R
2
.
Total Cost (Y) 25 11 34 23 32
Units Produced (X) 5 2 8 4 6
27
Homework-1 contd
2. Consider the fuel consumption data on the next slide, and the
following model which relates fuel consumption (Y) to the
average hourly temperature (X
1
) and the chill index (X
2
):
a. Plot Y versus X
1
and Y versus X
2
.
b. Determine the least squares estimates for the model parameters.
c. Predict the fuel consumption when the temperature is 35 and the chill
index is 10.
d. Compute the SST, SSR, SSE and R
2
. Interpret the value of R
2
.
.
2 2 1 1 0
c | | | + + + = X X Y
28
Data for Problem 2
Average Hourly
Temperature, x
i1
Chill Index, x
i2
Weekly Fuel
Consumption, y
i
28 18 12.4
32.5 24 12.3
28 14 11.7
39 22 11.2
57.8 16 9.5
45.9 8 9.4
58.1 1 8.0
62.5 0 7.5
29
Time Series Forecasting
Time series
Sequence of observations of response variable at
regular time intervals
Stochastic or dynamic, it does change over time.
For forecasting
We use past history of response variable to predict
the future.
Predictions exploit correlations between past
history and the future.
30
Time Series Forecasting contd
1
1.5
2
2.5
3
3.5
0 12 24 36 48 60 72 84
Month
S
a
l
e
s
1
1.5
2
2.5
3
0 12 24 36 48 60 72 84
Month
T
r
e
n
d
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0 12 24 36 48 60 72 84
Month
S
e
a
s
o
n
a
l
i
t
y
31
Time Series Forecasting contd
Classical Decomposition Model:
where
m
t
= trend component, the gradual upward or downward movement
of the data over time.
s
t
= seasonal component, pattern of the demand fluctuation above
or below the trend line that repeats at regular intervals
d = seasonal period
Z
t
= random noise component
t t t t
Z s m Y + + =
. 0 and , , ) var( , 0 ] [
1
2
= = = =

=
+
d
j
j t d t t t
s s s Z Z E o
32
Simple Average
Suppose that the time series {Y
t
} is generated by a constant
process subject to random noise,
Let y
1
, , y
n
be observations from the time series. We know
that average of the observations is a least squares estimator for
the mean m. Hence we can use the simple average
as a forecast for time period t + 1, t = 1, , n.
.
t t
Z m Y + =
.
1

1
1
=
+
=
t
i
i t
y
t
y
33
Moving Average
Now suppose that the mean m
t
of {Y
t
} changes slowly
over time. Hence it may be desirable to reduce the
influence of past data on the forecast. A moving
average forecast of order k is given by
The moving average of order k deals only with the latest
k periods of data.
It does not handle trend or seasonality very well,
although it does better than the simple mean.
. 1 , , , ,
1

1
1
= =

+ =
+
n k t y
k
y
t
k t i
i t

34
Exponential Smoothing
An extension to the moving average is forecasting
using a weighted moving average, which gives
more weight to the most recent observations.
Exponential smoothing is a class of methods that
apply exponentially decreasing weights as the
observations get older.
35
Exponential Smoothing contd
Let y
1
, , y
n
be observations of a time series {Y
t
} with
mean m
t
and no seasonality, and o [0, 1]. Define
weighted averages
Application of these equations is called simple
exponential smoothing.
. , , 2 ,

) 1 (

1 1
1 1
n t y y y
y y
t t t
= + =
=

o o
36
Exponential Smoothing contd
Note that
Hence the forecast is a weighted average of past
observations, with exponentially decreasing weights.
1
1
2
2
2
2
1
2 2
2
1
1 1
1
) 1 ( ) 1 ( ) 1 ( ) 1 (
] ) 1 ( [ ) 1 ( ) 1 (
] ) 1 ( )[ 1 (
) 1 (
y y y y y
y y y y
y y y
y y y
t t
t t t
t t t t
t t t
t t t




+
+ + + + + =
+ + + =
+ + =
+ =
o o o o o o o o
o o o o o o
o o o o
o o

Seasonal Adjustment
Compute a seasonal index for each data point
in time series
By dividing the value of each data point by total
average of all data
Divide the time series by seasonal indices
This minimizes the effect of seasonal variations
Fit a suitable model in adjusted data of time
series
Obtain the forecast and then multiply it with
corresponding seasonal index
37
Regression with Seasonal and Trend
Components
38
Multiple regression can be used to forecast both trend and
seasonal components in a time series
One independent variable is time
Dummy independent variables are used to represent the
seasons
The model is an additive decomposition model
4 4 3 3 2 2 1 1
X b X b X b X b a Y + + + + =

where
X
1
= time period
X
2
= 1 if season 2, 0 otherwise
X
3
= 1 if season 3, 0 otherwise
X
4
= 1 if season 4, 0 otherwise
39
Measuring Forecast Accuracy
Mean Squared Error (MSE):
Mean Absolute Error (MAE):
Mean Absolute Percentage Error (MAPE):

=

n
t
t t
y y
n
1
2
) (
1

=

n
t
t t
y y
n
1
| |
1

n
t
t
t t
y
y y
n
1

100
40
Homework-2
1. The Paris Chamber of Commerce and Industry has been
asked to prepare a forecast of the French index of industrial
production (see data on next slide).
a. Compute a forecast using moving averages with 12 observations in
each average.
b. Now compute a series of moving average forecasts using six
observations in each average.
c. Compute a series of exponential smoothing forecasts with o = 0.7.
d. Graph the data and each of your forecasts.
e. Compare your forecasts using the three metrics of forecast accuracy
discussed in class. How accurate would you say the forecasts are?
41
Homework-2 contd
Period French index
of industrial
prod.
Period French index
of industrial
prod.
Period French index
of industrial
prod.
1 108 10 95 19 101
2 108 11 95 20 104
3 110 12 92 21 101
4 106 13 95 22 99
5 108 14 95 23 95
6 108 15 98 24 95
7 105 16 97 25 96
8 100 17 101 26 98
9 97 18 104 27 94
Auto Regressive Models
42
Observation or dependent variable is a function of
itself at the previous moments of period or time.
) ,..., , (
, 2 1 t p t t t t
y y y f y c

=

=

+ + =
p
i
t i t i t
e y b b y
1
0 Linear Model
where:
y
t
the dependent variable values at the moment t,
y
t-i
(i = 1, 2, ..., p) the dependent variable values at the
moment t-i,
bo, bi (i=1,..., p) regression coefficient,
p auto-regression rank,
e
t
disturbance term.
Auto Regressive Models contd
43
A first-order autoregressive model is concerned with
only the correlation between consecutive values in a
series.
A second-order autoregressive model considers the
effect of relationship between consecutive values in a
series as well as the correlation between values two
periods apart.
t t t
e y b b y + + =
1 1 0
t t t t
e y b y b b y + + + =
2 2 1 1 0
i p i p i i i
e Y b Y b Y b b Y + + - - - + + + =
2 2 1 1 0
Auto Regressive Model Order P
44
t
y
t
1 1.89
2 2.46
3 3.23
4 3.95
5 4.56
6 5.07
7 5.62
8 6.16
9 6.26
10 6.56
11 6.98
12 7.36
13 7.53
14 7.84
15 8.09
Example: For following data, find coefficients of AR-1 and
AR-2 models and compare both for prediction accuracy.
Homework-3
Hints Homework-3
45
t
y
t
y
t-1
y
t-2
1 1.89 - -
2 2.46 1.89 -
3 3.23 2.46 1.89
4 3.95 3.23 2.46
5 4.56 3.95 3.23
6 5.07 4.56 3.95
7 5.62 5.07 4.56
8 6.16 5.62 5.07
9 6.26 6.16 5.62
10 6.56 6.26 6.16
11 6.98 6.56 6.26
12 7.36 6.98 6.56
13 7.53 7.36 6.98
14 7.84 7.53 7.36
15 8.09 7.84 7.53
t t t t
e y y y + + + =
2 1
08 . 0 8 . 0 1 . 1
y X X X b
T T 1
) (

=
t t t
e y b b y + + =
1 1 0
t t t t
e y b y b b y + + + =
2 2 1 1 0
Compare two forecasts using
the three metrics of forecast
accuracy