ECON1203 Statistics

Chapter 16 Simple Linear

Regression & Correlation
Contents
1.
2.
3.
4.
5.
6.

Model
Estimating the Coefficients
Error Variable: Required Conditions
Assessing the Model
Using the Regression Equation
Regression Diagnostics (Part 1)

Introduction

Regression analysis predicts one variable based on other variables.

The dependant variable is to be forecast ( Y ).

The statistics practitioner believes that it is related to independent

variables ( X 1 , X 2 , , X k ).

Correlation analysis determines whether a relationship exists:

o Scatter diagram
o Coefficient of correlation
o Covariance

16.1 Model

Deterministic models determine the dependent variable from the

independent variables. They are unrealistic because there may be other
influencing variables.
Probabilistic models include the randomness of real life (e.g. an error
variable).

The error variable ( ) is the difference between the estimated and

actual dependent variable.

The first-order linear model (or simple linear regression model) is a
straight-line model with one independent variable:
o

y= 0 + 1 x +

y=

dependant variable

x=

independent variable

0=

y-intercept

1=

slope of the line

error variable

16.2 Estimating the coefficients

The least squares line ( ^y =b 0+ b1 x ) uses the least squares method

n

to minimise the sum of squared deviations (

i=1

).

The sum of squares for error (SSE) is the minimised sum of squared
deviations.
Residuals are the deviations between the actual data points and the line:
o

( y i ^y i )2

e i= y i ^y i

Least squares line coefficients:

s xy

b1 =

b0 = y b 1 x

s2x

( x ix )( y i y )

s xy = i=1

n1

( x i x )2

s 2x = i=1

n1

xi

x = i=1
n
n

y = i=1
n

Shortcuts:
o

yi

s xy =

1
n1

1
s =
n1
2
x

xi yi

i=1

x i y i i=1

x
i=1

( )
xi

n
2
i

i=1

i=1

Excel:
o Have two columns of data: one for the dependent variable; the
other for the independent variable.
o Click Data, Data Analysis, and Regression.
o

Specify the Input

Range and the Input

Range.

Assumptions of Classical Linear Regression Model

1. The least squares line coefficients ( 0

1 ) are linear.

and

2. The observed variables ( x i , y i ) are randomly sampled.

x i ; they are not all equal.

3. There is sample variation in

4. The mean of

a. Therefore,
5. The variance of

x :

is 0, regardless of

and

E ( i|x i )=0 .

are uncorrelated.

is a constant:

Var ( i ) = 2 .
i

a. But in reality, not necessarily true (e.g. higher income may increase
variance in expenditure because they have a greater range of
choices)
6. The error variables are uncorrelated:

Cov ( i , j )=0 .

7. The error variables are normally distributed:

i N ( 0, 2 ) .

16.4 Assessing the Model

There are three ways of assess how well the linear model fits the data:
1. The standard error of estimate
2. The

t -test of the slope

3. The coefficient of determination

Sum of Squares for Error (SSE)

n

SSE= ( y i ^y i ) =( n1 ) s
i=1

2
y

s2xy
s2x

Standard Error of Estimate ( s )

SSE
( s = n2
) isusually compared with y

Testing the Slope

We can use hypothesis testing to infer the population slope ( 1 ) from

the sample slope ( b1 ).

If

1=0 , there is no linear relationship (but there may be a quadratic

relationship).

The sample slope ( b1 ) is an unbiased estimator of the population

slope ( 1 ) ( E ( b1 )= 1 ) because the estimated standard error of

b1

s
s
=
b
(
( n1 ) s 2x ) decreases as
1

Test statistic for

1=t=

increases.

b1 1
[ where =n2 ]
sb
1

Confidence interval estimator of

1=b1 t 2 s b [ where =n2 ]

1

Coefficient of Determination

Coefficient of Determination:

R 2=

s2xy
2

=1
2

sx s y

( y i y ) SSE = Explained variation

SSE
=
2
Variation y
( y i y )
( yi y )2

( y i y ) =( y i y ) + ^y i^y i

( y i y ) =Unexplained residual ( y i ^y i ) + Explained variation ( ^y y )

( y i y )2= ( yi ^y i )2+ ( ^y i y )2

Variation y=SSE+ SSR

Coefficient of Correlation

We can use hypothesis testing to infer the population coefficient of

correlation ( ) from the sample coefficient of correlation ( r ).

Sample coefficient of correlation:

Test statistic for

t=r

r=

s xy
sx sy

n2
1r 2 [where

=n2 and variables are

bivariate normally distributed]

16.5 Using the Regression Equation

^y i=b 0+b1 x i

is a point estimator.

There are two interval estimators:

1. Prediction interval:

1 ( x gx )
^y t 2,n2 s 1+ +
n ( n1 ) s 2x

2. Confidence interval estimator of the expected value of

y : ^y t 2,n2 s

2
1 ( x gx )
+
n ( n1 ) s 2x

The farther the given value of

x , the greater the estimated error:

is from

( x gx )

( n1 ) s 2x

16.6 Regression Diagnostics (Part 1)

Residual analysis

Standard deviation of the ith residual : s =s 1h i

i

2
1 ( x ix )
Where hi= +
n ( n1 ) s 2x

Normality

The residuals should be normally distributed.

Homoscedasticity

The variance of the error variable should be constant.

Independence of the error variable

The error variable should be independent.

Outliers
Outliers may be:
1. Recording errors
2. Points that should not have been included in the sample
3. Valid and should belong to the sample

Influential observations

Some points are influence in determining a least squares line. Without it,
there would be no least squares line.

Procedure
1. Develop a model that has a theoretical basis; find an independent variable
that you believe is linearly related to the dependent variable.
2. Gather data for the two variables from (preferably) a controlled
experiment, or observational data.
3. Draw a scatter diagram. Determine whether a linear model is appropriate.
Identify outliers and influential observations.
4. Determine the regression equation.
5. Calculate the residuals and check the required conditions:
a. Is the error variable normal?
b. Is the variance constant?
c. Are the errors independent?

6. Assess the models fit:

a. Compute the standard error of estimate.
b. Test
c.
7. If the
a.
b.

1 or

to determine whether there is a linear

relationship.
Compute the coefficient of determination.
model fits the data, use the regression equation to:
Predict a particular value of the dependant variable
Estimate its mean