You are on page 1of 36

Psychometric testing in credit risk

assessment
A digital readiness primer and case study

@eflglobal // Amie.Vaccaro@EFLglobal.com
EFL Intro

Where EFL works best

Digital readiness primer


Agenda HCD: Designing for the feature
phone

Equity Bank case study

Q&A
Traditional credit scoring is blind to billions of people
3 billion are unscorable, another 3 billion will enter the middle class in the next 20 years

Covered by
70%
credit Not covered
bureau,
30%
•  Billions of good applicants rejected
Not
covered, •  Enormous missed opportunity
70%

Source: http://www.ey.com/gl/en/issues/driving-growth/middle-class-growth-in-emerging-markets; http://data.worldbank.org/indicator/IC.CRD.PRVT.ZS

3 Copyright © 2017. All Rights Reserved.


We are entering a new era of lending
And credit scoring needs to adapt to serve more people with end to end digital processes

Loan officers Loan officers Loan officers Digital approvals


with deep + + without loan officers
relationships Traditional Traditional and +
credit scoring alternative scoring Traditional and
alternative scoring

1990’s 2000’s 2010’s Today


Predictive power and availability vary by data source
Psychometric is only data source with 100% availability – everyone has a personality

More Available
(Hit rate on sample of low-income applicants in emerging markets)

100% PSYCHOMETRIC DATA


90%
80%
MOBILE DATA
70%
Availability

60%
50%
40%
30%
ONLINE DATA BUREAU DATA
Less Available

20%
10%
0%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Less Predictive More Predictive

Predictive Power
(Out of sample gini coefficient with comparable model & sample)

6 Copyright © 2017. All Rights Reserved.


EFL is the proven universal score for today’s lenders
Empowering digital and in-person approvals across $1.5B+ and counting in disbursed loans

Alternative scoring leader Universal coverage Third party vetted

•  4 years R&D at Harvard on •  Score anyone, anywhere •  Published 3rd party


psychometric, demographic, with our psychometric and validations by World Bank
social & CDR scoring behavioral science based and others Link
assessment
•  6 years implementing •  Won multiple awards
scores with lenders across •  No client history or 3rd including G-20 SME
Africa, Latin America & Asia party database needed Finance Challenge, African
Business Award for
Innovation, and Mahindra
Finance Challenge

7 Copyright © 2017. All Rights Reserved.


EFL assesses risk with a short online loan application
How we score anyone The character traits our
assessment measures
EFL Algorithms
Analyze the answers and
Unscorable Applicant determine risk level Score Generation
Takes EFL assessment on Create 3-digit EFL Score and
the web, mobile or SMS
2 Score Confidence rating Attitude

1
1 3 Control

Entrepreneurial potential

Interaction metadata

4
Continuous Learning Social Behavior
EFL receives monthly loan
repayment used to constantly
customize and improve
models

8 Copyright © 2017. All Rights Reserved.


EFL Intro

Where EFL works best

Digital readiness primer


Agenda HCD: Designing for the feature
phone

Equity Bank case study

Q&A
EFL works best when there’s a clear need for an
alternative credit score, and the FI is challenged by:

1. 2. 3.

Low Acceptance Rates High Levels Costly & Lengthy Loan


(due to lack of data) of Default Application Processes

High rejection rates due Default rates over target; Loan application process
to lack of data – too hesitant to dramatically takes time and is resource-
many grey zone or thin decrease acceptance intensive for lender
file clients rate

10 Copyright © 2017. All Rights Reserved.


Use the EFL score to…
Our partners typically use our behavioral psychometric score in one of three ways:

1. 2. 3.

Increase Your Decrease Your Increase Your


Acceptance Rate Default Rate Operational Efficiency;
Lower Application Cost

Approve more people, Decrease and control Fast track: Cut pre-filters
without taking on more your default rate, with for high-scoring
risk limited volume impact applications

11 Copyright © 2017. All Rights Reserved.


‘Institutional Readiness’ plays a large role in success
We’ve noticed certain traits within organizations that allow the EFL digital credit assessment to thrive

High Volume,
Credit Score Ready Driven to Push Innovation Mindset

Ability to receive a 3-digit, Strong desire across Willingness to try new solutions
numerical, credit score and use it in management & teams to and consider a departure from
their existing credit decision making implement and scale traditional processes.
process. solution quickly.

Receiving thousands of credit Motivated team, with Willingness to take calculated


applications each month. willingness and motivation to risks and move in digital direction.
change existing process
Extra Plus? Ability to combine EFL
score with other credit scores via
scorecard or ensemble model

12 Copyright © 2017. All Rights Reserved.


EFL Intro

Where EFL works best

Digital readiness primer


Agenda HCD: Designing for the feature
phone

Equity Bank case study

Q&A
Financial institutions are going digital globally
Digital technologies can cut the cost of providing financial services by 80 to 90% and improve scalability

Annual cost to serve one customer in emerging economies,


2014 ($)

Accounts Cash-in, cash-out Transactions Total Traditional


bank branch

Digital

Cost savings
due to digitization

Source: McKinsey Global Payments Map; “Fighting poverty, profitably: Transforming the economics of payments to build sustainable, inclusive financial
systems,” Bill and Melinda Gates Foundation, September 2013; McKinsey Global Institute analysis

14 Copyright © 2017. All Rights Reserved.


Going digital? It’s more than building an app
What does it mean for a bank to be “digital”?

Being digital means being scalable. If you have to add staff to increase lending, you aren’t digital.

Acquisition Affordability Risk KYC / ID Servicing


Online acquisition is Web scrappers are Multiple companies, Proving ID online is Few companies are
possible, though emerging to support including EFL are an emerging field, focused on are looking
usually pricey. digital affordability, proving that risk however, in EM to outsource
however, limited profiles can be built limited databases collections.
coverage and via digital coverage is a factor.
inconsistent data interactions.
remain an issue

15 Copyright © 2017. All Rights Reserved.


But there are challenges to going digital
Moving the credit application process from in person with a loan officer to online comes with its own risks

Increased Risk
Digital applications can come with increased
risk and fraud. Effective credit scoring is key EFL is your digital
to keeping a healthy portfolio.
loan officer.

Efficiently assess risk


Lose Personal Touch on anyone, anywhere,
No human interaction can dampen the bank- online. Build trust with
to-customer relationship. Tools are needed applicants.
to increase human designed interaction.

16 Copyright © 2017. All Rights Reserved.


Going digital is a process
We work with partners at each stage
100%

Fully digital process;


pushing for scale
No existing digital
processes Integrate EFL into a
digital application:

% DIGITAL
Use EFL as a stand alone •  Control risk while
tool, focused on: Hybrid model – maintaining
•  Customer Acquisition some parts digitized efficiency.
•  Reject Recovery •  Increase
Add EFL to partially digital acceptance rates.
process flow, see results in:
•  Operational efficiency

0%

17 Copyright © 2017. All Rights Reserved.


EFL Intro

Where EFL works best

Digital readiness primer


Agenda HCD: Designing for the feature
phone

Equity Bank case study

Q&A
Product Design: Understand Your “Bones”
What framework are you working within?

•  SMS = a necessarily text-heavy product


•  Question length limited to 160 chars –containing both question and answer key
•  Need to battle potential mistrust of the product – concerns of SMS scams
•  Open answer responses – need to validate and send appropriate responses

Understand your customers:


•  Design for the mass-market, and try to incorporate the low-common denominator as
much as possible. People will struggle with:

Literacy Understanding Technology Connectivity

19 Copyright © 2017. All Rights Reserved.


Know What You NEED
Hypothesize:
Why does someone
default?
•  Relatively easy to understand (both
the question and desired answer
Intelligence General Behavior
format)

•  Predictive even within question


number and character limits Honesty Stability

•  Forgiving if users make mistakes


and need to correct answers
Fraud Attitudes
•  Trust building – Reliable product
that is connected to bank and
reward believed
Product design PROCESS

3. Data analysis of
flows and field 1. Field observation
interviews: ensure & interviews: on
which key metrics both question
are improving and framing and content
which aren’t

2. Experimental design:
validate/quantify hypotheses
from field via more real-life
channel using flow split-testing
and within-flow experiments
1 Get OUT! START

Watch! How and when are people


interacting with their phones?

Get to know your competition – what mobile services are people


using? What do they like / dislike about them?

Build: Fast and easy prototypes


Paper print outs of phones with text questions

“Hey! Can I ask your opinion?”

Talk with people- get them to use your product, explain as little as
possible.

“What do you think it means?”

22 Copyright © 2017. All Rights Reserved.


2 Create the RIGHT data
Test many versions of your product, over wide audiences, through various
channels
CHANNEL EXPERIMENTAL DESIGN

Parallel Flows
Test two similar products at the same time, with different
versions of the same content you need to test – isolates
the “treatment effects”
In Person

Single Flows

Intersperse new iterations of product parts within your


current product iteration to see if behavior changes –
controls for individual differences

Distance Testing

Output: Well-structured quantitative and qualitative data to analyze

23 Copyright © 2017. All Rights Reserved.


3 Explore Your DATA
Define metrics for what makes the product “good” – explore the data to see what's
working…
•  Answer distribution (extreme values vs. well-distributed
•  Answer timing (confusing questions take longer)
•  Drop off rates (do certain questions cause people to disengage?)
•  % of unusable answers
…Do we see an improvement in performance
metrics as a result of a new iteration?

Percent of responses with extreme


answers or unusable answers
BEFORE

10 Point Scale
Reply with a NUMBER ONLY on a scale from 0 - 35%

10 (0=never 5=sometimes 10=always). Do you 30%


show up late or miss appointments? 25%

20%

15%
AFTER

Percentages 10%

What PERCENT (%) of the time do you show up 5%

late or miss appointments? 0%


10-point scale Percent scale

24 Copyright © 2017. All Rights Reserved.


Final SMS Content: Three Targeted Core Traits
Objective: Identify a few core traits that are most relevant to the population. Focus a majority of
psychometric content around these to max predictive power.

Understanding the
user’s concept of Understanding the
honesty and fairness user’s sense of
in the context of her personal responsibility
financial life for and control over
her life
Locus of
Integrity Control
% of responses with
no valid answer
25%

Opportunism/
20%
Progress
mindset Understanding the
15%
user’s perspective on
10%
opportunities and
Locus of Confidence, motivations to take risks
Control, Optimism, to improve her life
Integrity, GSE
Opportunism

25 Copyright © 2017. All Rights Reserved.


Final SMS Product

Maximizing predictive power Reply with a


NUMBER in KSH. If
1)  3 core traits focused on
you found 100,000
2)  Granular answer potential (%s) KSH today, how
much would you
… through ensuring understandability spend on leisure?
1)  Answer in percentages
2)  Non-hypotheticals – concrete questions / “mini stories”
3)  Data quality check – smartphone? Help on survey?
… and building trust
1)  Human approach – make people feel comfortable
2)  Start with easy, but non-invasive questions (back load more intrusive
demographic questions)
3)  Trust this product – start with the person’s name

26 Copyright © 2017. All Rights Reserved.


EFL Intro

Where EFL works best

Digital readiness primer


Agenda HCD: Designing for the feature
phone

Equity Bank case study

Q&A
Overview - Scoring on the feature phone
Equity extends credit through chat, proves ability to boost acceptance rates

How does it work?


The Challenge
High volume, feature phone mobile lending comes with Tine, an Equity
significant risk. Equity needed an innovative solution to accept customer, applies
more of their customers without taking on additional risk. for a loan via her
Equitel SIM card.

The Engagement
EFL designed a new product – taking the learnings of our
traditional psychometric content, and transforming them into a 22
She’s new to the
SMS question assessment. Instead of being flat-out rejected, low bank, so she
data customers could take the assessment remotely on their receives the first
feature phones, and gain an opportunity to be approved.
EFL SMS asking
her to start the test.
The Results
Collecting 24k completed SMS tests, EFL built a custom SMS
model, and saw significant predictive power in the data Tine’s survey is
(psychometric & meta). The strong model results indicate analyzed, and her
potential SMS uses across markets / channels. credit score sent back
to Equity in real time.
Tine is disbursed!

28 Copyright © 2017 EFL. All Rights Reserved.


Designing the SMS Pilot – Process & Criteria
FIVE STEPS UNDERTAKEN TO
Equity & EFL partnered in 2013 to develop a custom
COMPLETE THE SMS PILOT…
feature phone credit scoring tool

1 Design SMS
Test
…TO VALIDATE TWO MAIN HYPOTHESIS

2 Build & Integrate


SMS Platform
HYPOTHESIS DESIGN MEASUREMENT

1. The SMS channel Test

3 Run Pilot;
Gather Results
will facilitate a high
conversion rate.
UX of SMS
Test
Completion
Rate (%s in
funnel)

4 Build Custom
Model
2. SMS psychometric
5 Out-of-Sample Predictive
testing will enable Equity Psychometric
Model Power of SMS
Validation to better segment and Assessment Model
predict risk

29 Copyright © 2017. All Rights Reserved.


Assess willingness to repay through a conversation
Through EFL’s chatbot platform, we converse with your clients, one SMS at a
time. Through an automated conversation, we ask our traditional psychometric
questions – straight to their phone. Predictive, real-time, credit scores available
in minutes.
What percent (%)
of businesses
Derives predictive power from: cheat their
clients?
1.  Psychometric answers

2.  Timing data SMS – USSD


3.  Free-text input data ~20 psychometric questions
4.  Self-reported demographic data 4 test sections
Any phone, any location
Only need to score once

30 Copyright © 2017. All Rights Reserved.


Seamlessly assess an applicant’s character
PRODUCT FLOW

CREDIT
SCORE
337

Benson, a new Instead, he immidietely Benson engages, Credit Score


customer, applies for receives an inia9on conversa9on style, in the returned to lender
loan. Normally, he SMS masked from the SMS test.
would be rejected. lender

Behavioral – Text Analysis; Timer Analysis Psychometrics


What time of day are they responding?
What are response answer patterns?
Do they understand the question?
How is their spelling? Grammar?
What words or phrases do they use?
© 2016, EFL Global. All rights reserved. 31
EFL SMS test seamlessly fits in with existing UX mobile
lending product flow
Before rejecting low-data and new applicants, Equity can digitally score and asses risk on borrowers.

CREDIT
SCORE
337

Benson, a customer Immidietely Benson engages,conversa9on Credit Score


of Equity Bank, receives inia9on style, in the SMS test. -
applies for loan via SMS from - Returned in real-
Equitel SIM SDK/ dedicated Equity Answers test in 4 secPons. SMS Pme through API to
USSD menu short code reminders sent to keep Benson Equity’s CU
- - focused.
Benson replies to
the SMS (zero- Total test )me <15 minutes,
rated) and begins taken at once or over hours/
the test days

32 Copyright © 2017. All Rights Reserved.


Results: Predictive Model; Strong Usability

Customized Model Out of


High Sample
With 24k tests completed, EFL built a Default
AUC:
customized model for SMS – achieving a 63.7
predictive power of 63.7% AUC

Default Rate
Default Rate
Usability validated through low drop off
rates.
Average completion rate of 78%, with an
average test time of 33 minutes.
Low EFL Score Quartiles
80% of drop-off occurred within first 4 questions. Default
1 Worst 12 23 34 54Best
56% of tests were completed within 15 minutes. Worst Scoring Best Scoring

bad45_m3 bad30_m3 bad60_m3

33 Copyright © 2017. All Rights Reserved.


Ability to increase acceptance rates by 20% when
combined with internal behavioral score
High
Cumulated Default Rates
Default

Default Rate
EFL & Equity’s Internal
Behavioral Score
Combining the EFL score with an
internal behavioral (transactional) score:
Low
Default
boost acceptance rate by 20% High 1.2x 1x Low
without taking on additional risk Acceptance Rate
20%
Ensemble Experian
Behavioral EFL

34 Copyright © 2017. All Rights Reserved.


EFL finds different bads, and different goods, than
traditional credit bureau scores
Cumulated Default Rates
High
Default

Default Rate
EFL & Traditional Bureau
Score
Combining the EFL score with a
traditional bureau score:
Low
boost acceptance rate by 53% Default

without taking on additional risk


High Low
Acceptance Rate
Acceptance
53% Acceptance

Ensemble Bureu
Bureau EFL

35 Copyright © 2017. All Rights Reserved.


EFL Intro

Where EFL works best

Digital readiness primer


Agenda HCD: Designing for the feature
phone

Equity Bank case study

Q&A