ASSIGNMENT #1

DECISION AND SUPPORT SYSTEMS

ID: 999991355
Question1:

Score Actual Class

.51 -

.53 -

.55 +

.56 -

.58 +

.59 +

.61 -

.65 +

(i) Determine EER, (ii) compute threshold to maximize accuracy, and (iii) plot ROC for the above data.

Answer:

1) Consider threshold value=.51

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - -

.55 + -

.56 - -

.58 + -
.59 + -

.61 - -

.65 + -

4 of 4 actual + is identified falsely as - , so the FNR is 4/4=1.

1 of 4 actual - is identified falsely as + , so the FPR is 1/4

So for threshold=.51, FNR=1 and FPR=1/4

0 of 4 actual + is correctly identified as +.

3 of 4 actual - is correctly identified as - .

So 3 out of 8 cases are correctly identified.

So the accuracy for threshold= .51 is 3/8

FROM THE CONFUSION MATRIX

TP=0; FN=4; FP=1; TN=3.

FPR=FP/FP+TN =1/4=.25

TPR=TP/TP+FN=0/4=0

2) Consider threshold value=.53

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - +

.55 + -

.56 - -

.58 + -
.59 + -

.61 - -

.65 + -

4 of 4 actual + is identified falsely as -, so the FNR is 4/4=1.

2 of 4 actual - is identified falsely as +, so the FPR is 1/2

So for threshold=.53, FNR=1 and FPR=1/2

0 of 4 actual + is correctly identified as +.

2 of 4 actual - is correctly identified as - .

So 2 out of 8 cases are correctly identified.

So the accuracy for threshold= .53 is 2/8

FROM THE CONFUSION MATRIX

TP=0; FN=4; FP=2; TN=2.

FPR=FP/FP+TN =2/4=0.5

TPR=TP/TP+FN=0/4=0

3) Consider threshold value=.55

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - +

.55 + +

.56 - -

.58 + -
.59 + -

.61 - -

.65 + -

3 of 4 actual + is identified falsely as -, so the FNR is 3/4.

2 of 4 actual - is identified falsely as +, so the FPR is 1/2

So for threshold=.53, FNR=1 and FPR=1/2

1 of 4 actual + is correctly identified as +.

2 of 4 actual - is correctly identified as - .

So 3 out of 8 cases are correctly identified.

So the accuracy for threshold= .55 is 3/8

FROM THE CONFUSION MATRIX

TP=1; FN=3; FP=2; TN=2.

FPR=FP/FP+TN =2/4=0.5

TPR=TP/TP+FN=1/4=0.25

4) Consider threshold value=.56

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - +

.55 + +

.56 - +

.58 + -
.59 + -

.61 - -

.65 + -

3 of 4 actual + is identified falsely as -, so the FNR is 3/4

3 of 4 actual - is identified falsely as +, so the FPR is 3/4

So for threshold=.56, FNR=3/4 and FPR=3/4

1 of 4 actual + is correctly identified as +.

1 of 4 actual - is correctly identified as - .

So 2 out of 8 cases are correctly identified.

So the accuracy for threshold= .56 is 2/8

FROM THE CONFUSION MATRIX

TP=1; FN=3; FP=3; TN=1.

FPR=FP/FP+TN =3/4=0.75

TPR=TP/TP+FN=1/4=0.25

5) Consider threshold value=.58

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - +

.55 + +

.56 - +

.58 + +
.59 + -

.61 - -

.65 + -

2 of 4 actual + is identified falsely as -, so the FNR is 2/4=1/2.

3 of 4 actual - is identified falsely as +, so the FPR is 3/4.

So for threshold=.58, FNR=1/2 and FPR=3/4.

2 of 4 actual + is correctly identified as +.

1 of 4 actual - is correctly identified as - .

So 3 out of 8 cases are correctly identified.

So the accuracy for threshold= .58 is 3/8

FROM THE CONFUSION MATRIX

TP=2; FN=2; FP=3; TN=1.

FPR=FP/FP+TN =3/4=0.75

TPR=TP/TP+FN=2/4=0.5

6) Consider threshold value=.59

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - +

.55 + +

.56 - +

.58 + +
.59 + +

.61 - -

.65 + -

1 of 4 actual + is identified falsely as -, so the FNR is 1/4

3 of 4 actual - is identified falsely as +, so the FPR is 3/4

So for threshold=.59, FNR=1/4 and FPR=3/4

3 of 4 actual + is correctly identified as +.

1 of 4 actual - is correctly identified as - .

So 4 out of 8 cases are correctly identified.

So the accuracy for threshold= .59 is 4/8

FROM THE CONFUSION MATRIX

TP=3; FN=1; FP=3; TN=1.

FPR=FP/FP+TN =3/4=0.75

TPR=TP/TP+FN=3/4=0.75

7) Consider threshold value=.61

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - +

.55 + +

.56 - +

.58 + +
.59 + +

.61 - +

.65 + -

1 of 4 actual + is identified falsely as -, so the FNR is 1/4.

4 of 4 actual - is identified falsely as +, so the FPR is 1

So for threshold=.61, FNR=1/4 and FPR=1

3 of 4 actual + is correctly identified as +.

0 of 4 actual - is correctly identified as - .

So 3 out of 8 cases are correctly identified.

So the accuracy for threshold= .61 is 3/8

FROM THE CONFUSION MATRIX

TP=3; FN=1; FP=4; TN=0.

FPR=FP/FP+TN =4/4=1

TPR=TP/TP+FN=3/4=0.75

8) Consider threshold value=.65

if score threshold, the algorithm decides that class= +

So from the above equation the predicted class is given in the table

score Actual class Predicted class

.51 - +

.53 - +

.55 + +

.56 - +

.58 + +
.59 + +

.61 - +

.65 + +

0 of 4 actual + is identified falsely as -, so the FNR is 0/4=0.

4 of 4 actual - is identified falsely as +, so the FPR is 1

So for threshold=.53, FNR=1 and FPR=1/2

4 of 4 actual + is correctly identified as +.

0 of 4 actual - is correctly identified as - .

So 4 out of 8 cases are correctly identified.

So the accuracy for threshold= .65 is 4/8

FROM THE CONFUSION MATRIX

TP=4; FN=0; FP=4; TN=0.

FPR=FP/FP+TN =4/4=1

TPR=TP/TP+FN=4/4=1

From all the above findings

score FPR FNR accuracy

.51 1 3/8=.375
.53 1 2/8=.25
.55 3/8=.375
.56 2/8=.25
.58 3/8=.375
.59 4/8=.5
.61 1 3/8=.375
.65 1 0 4/8=.5

(i) If we find that in a row FPR=FNR, then that value is called equal error rate
(EER).in the above table FPR=FNR is at .56 therefore the EER IS .56

(ii) From the above findings the maximum accuracy is .5 and is at .59 and .65
therefore the threshold is .59 And .65

(iii) ROC CURVE IS FROM THE BELOW TABLE

CLASS - - + - + + - +
.51 .53 .55 .56 .58 .59 .61 .65
TP 0 0 1 1 2 3 3 4
FN 4 4 3 3 2 1 1 0
FP 1 2 2 3 3 3 4 4
TN 3 2 2 1 1 1 0 0
TPR 0 0 0.25 0.25 0.5 0.75 0.75 1
FPR 0.25 0.5 0.5 0.75 0.75 0.75 1 1

ROC CURVE Characterize the trade-off between TPR and FPR.ROC curve plots
TPR (on the y-axis) against FPR (on the x-axis) .
 ROC CURVE
1

0.75

0.5

0.25

0
0 0.25 0.5 0.75 1

Question 2 (2 Bonus Points)

ROC divides the plot area into two parts. Explain what those parts signify.

ANSWER: It is a plot of the True Positive Rate (on the y-axis) versus the False
Positive Rate (on the x-axis) for every possible classification threshold. Both the
True Positive Rate and the False Positive Rate range from 0 to 1. ROC curve will
trace the 45 degree line either in this plot or on this plot and this is the case that
would call a random or the worst and so what we have concluded here is that
whenever you report a performance of binary classes of binary classifier you can
always represented by an ROC curve and that ROC curve will be in the worst case
in area at a 45-degree line. In the best-case scenario it will occupy Entire Square.
The bisected line is also called as non-discriminant line and the area above the
line is good area and the area below the line is bad area