Академический Документы
Профессиональный Документы
Культура Документы
Week 3 Recap
n If dependent variable is continuous à regression such as linear
regression
Annoucements
Today’s Agenda
n Predicting “Churn”
n ROC Analysis
Q3 2015 2015
Carrier Monthly Churn Annualized Churn
Source: FierceWireless
Customer Retention
n If the customer churns, the company makes $0 from this customer in the future
n If the customer does not churn, the company makes $1000 (under no price
reduction) or $800 (if they give the price reduction)
Churn $0
p/2 Elprofit I 0
Reduce Price
No Churn 80011 E
1- p/2 $800
Don’t Reduce Price
Churn $0
p Edprofit p 0
No Churn
1- p
100011
p
$1000
IEOR 242, Spring 2020 - Week 4
+ 9
n From the decision tree, our expected profit with a price reduction
is 800 x ( 1 - p/2 ) and our expected profit with no price reduction
is 1000 x ( 1 – p ) 800 1
I 100011
p
业
n Using arithmetic, the break-even point is p = 0.333
n Customers with churn probability above 0.333 are dubbed
high churn-risk
n Customers with churn probability below 0.333 are dubbed low
churn-risk
Churn, continued
n Our dataset:
nA telecom working with Watson Analytics
n Data on whether customers churn in a given year
Null
glm.fit <- deviance:
glm(Churn 5699.5 on+ PaymentMethodCredit
~ SeniorCitizen 4921 degrees of freedom
+ …, data = churnData.train, family = ‘binomia
Residual deviance: 4209.0 on 4912 degrees of freedom
summary(glm.fit)
AIC: 4229
n If the customer churns, the company makes $0 from this customer in the future
n If the customer does not churn, the company makes $1000 (under no price
reduction) or $800 (if they give the price reduction)
Churn $0
p/2
Reduce Price
No Churn
1- p/2 $800
Don’t Reduce Price
Churn $0
p
No Churn
1- p
$1000
IEOR 242, Spring 2020 - Week 4
+ 20
Confusion Matrix
1238 + 418
Accuracy = = 0.785
1238 + 311 + 143 + 418
Model
Low Churn-Risk High Churn-Risk
Pr(Yi = 1) < 0.333 Pr(Yi = 1) > 0.333
Reality No Churn (Yi = 0) 1,238 311
Churn (Yi = 1) 143 418
n Let us now see how we can assess how well the model
performs across all break-even thresholds
1.00
TPR = 0.745
FPR = 0.201
0.50
0.25
0.00
0.00 0.25 0.50 0.75 1.00
False Positive Rate
1.00
TPR = 0.504
0.50
FPR = 0.090
0.25
0.00
0.00 0.25 0.50 0.75 1.00
False Positive Rate
1.00
TPR = 0.000
0.50
FPR = 0.000
0.25
0.00
0.00 0.25 0.50 0.75 1.00
False Positive Rate
1.00
TPR = 0.945
0.50
FPR = 0.533
0.25
0.00
0.00 0.25 0.50 0.75 1.00
False Positive Rate
1.00
TPR = 1.00
FPR = 1.00
0.50
0.25
0.00
0.00 0.25 0.50 0.75 1.00
False Positive Rate
ROC Curve
n The ROC curve plots the TPR and FPR for every
break-even threshold p between 0.0 and 1.0
n ROC curve can also be drawn for LDA, and for any
method that predicts probabilities
1.0
0.8
True positive rate
0.6
0.4
0.2
0.0
n TPR = 0.2
1.0
n FPR = 0.2
0.8
True positive rate
0.6
0.4
0.2
●
0.0
n TPR = 0.5
1.0
n FPR = 0.5
0.8
True positive rate
0.6
●
0.4
0.2
●
0.0
n TPR = 0.8
1.0
n FPR = 0.8
0.8
●
0.4
0.2
●
0.0
1.0
0.8
True positive rate
0.6
0.4
0.2
0.0
n Model:
n Assume that we know the prior marginal probability
distribution of Y :
⇡k = Pr(Y = k)
n Given that Y = k, we know the distribution of the feature
vector X
n (Say X has density function fk (x) )
5
4
np =1
3
2
1
0
−4 −2 0 2 4 −3 −2 −1 0 1 2 3 4
np =2
x2
x2
x1 x1
4
2
2
X2
X2
0
0
−2
−2
−4
−4
−4 −2 0 2 4 −4 −2 0 2 4
X1 X1
n Logistic = blue
True positive rate
0.6
n LDA = red
0.4
n Similar performance
like this is typical
0.2
0.0