Noise

How to Overcome the High, Hidden Cost of Inconsistent Decision Making

Mir Faisal Talpur

Mar 19, 2018

www.ptcl.com.pk 1
Variability of Judgement

• Human are unreliable decision makers

• Influenced by irrelevant factors
• Mostly invisible factors
• However,
 No impact on noise free jobs
e.g Clerks, job keeper etc.

www.ptcl.com.pk 2
Variability of Judgement

• Decision – A matter of Judgement & prone to

noise
• Noise is far above the limits
• Noise is often deceptive
• Resulting
 Deviates significantly from peers / own decision in past
• Where there is judgment, there is noise―and
usually more of it than you think.

www.ptcl.com.pk 3
What is Noise?

Theoretically

The chance variability of judgement is called

Noise
Can save million by reducing only few percent in noise

www.ptcl.com.pk 4
Replacement of Judgement with Algorithms

• Simple statistical algorithms are more accurate then experts

 Even expert have more info than algorithms
• Algorithms are noise free

Bias
• Estimation is either too high of too low – Standard
 e.g A scale that consistently underestimates true weight by exactly four pounds is
seriously biased but free of noise.
• Many errors of measurement arise from a combination of bias and noise.

www.ptcl.com.pk 5
How noise & bias affect accuracy?

• Free of Noise & Bias (A): All shots are on

the bull’s-eye and close to one another.
• Noisy (B): Shots are centered around the
bull’s-eye but widely scattered.
• Biased (C): Shots missed the bull’s-eye
but cluster together.
• Noisy & Bias (D): Shots are scattered,
even out of external circle.

www.ptcl.com.pk 6
Noise Audit

• To improve the quality of decisions

• Success is conditional to acceptability of unpleasant results and
act on them
• All unit members should participate in the audit
• Social scientist with experience in conducting rigorous
behavioral experiments should supervise
• Professional unit must own the process.

www.ptcl.com.pk 7
Types of Bias & Noise and Corrective Actions
Type of Bias
GENERAL : The average judgment is wrong.
SOCIAL : Discrimination occurs against certain
categories of cases.
COGNITIVE : Decisions are strongly influenced by
irrelevant factors or insensitive to relevant ones.
Type of Noise
VARIABILITY ACROSS OCCASIONS: Same case to
same individual
VARIABILITY ACROSS INDIVIDUALS: Professionals
in the same role make different decisions.
Corrective Actions
Continual monitoring, Guidelines and targets, Eliminating
incentives that favor biases
Monitoring statistics for different groups, Blinding of applications,
Objective and quantifiable metrics, Open channels for complaints,
Guidelines and training
Training employees to detect situations in which biases are likely
to occur, Critiques of important decisions, focused on likely biases
Corrective Actions
Algorithms to replace human judgment, Checklists that encourage
a consistent approach to decisions
Algorithms to replace human judgment, Frequent monitoring of
individuals’ decisions, Roundtables at which differences are
explored and resolved, Checklists that encourage a consistent
approach to decisions
www.ptcl.com.pk 8
Dialing Down the Noise

Studies show that algorithms do better than

humans in the role of decision maker
• Radical solution to the noise problem is to replace human judgment with formal
rules—known as algorithms
• Algorithms are noise free
• Algorithms must be monitored and adjusted for occasional changes in the
population of cases
• Managers must also keep an eye on individual decisions and have the authority
to override the algorithm in clear-cut cases
• Can be used for an intermediate source of information

www.ptcl.com.pk 9
Bringing Discipline to Judgment

• To adopt procedures that promote consistency –

 Similar methods to seek information,
 Integrate it into a view of the case,
 And translate that view into a decision.
• Training is crucial
• Firms sometimes combat drift by organizing roundtables
• Individual should study the case independently before roundtables & Send own
opinions to the group leader before the meeting
• Final - A group discussion in which differences of opinion are explored.
• Professionals should be offered user-friendly tools - checklists & Questionnaire
• Unwanted variability should be reduced by tools

www.ptcl.com.pk 10
Conclusion

Controlling noise is hard, but we expect that an

organization that conducts an audit and evaluates the
cost of noise in dollars will conclude that reducing
random variability is worth the effort.

www.ptcl.com.pk 11
Thanks
www.ptcl.com.pk 12