ALARM

MANAGEMENT
DISTILLED
How a simple tool can change everything.
Insulator String
Insulator String
Dramatic picture
of the damage!
 BRANDON
HARRINGTON
Associated Electric Cooperative
Springfield, MO
Operator • Then a picture of an Operator?

Out
 Picture of Operator at console with arms raised in surprise

Then an arrow points to the alarms, with a *ping*

Or, zoom into the screen

IN THE
CONTROL
ROOM
(Alarms Scrolling Off the Page)
System Operators Plant Operators

Network Operators Smooth Operators

 Flash forward

• Video of same event with

current alarm configuration
A Tool Tips
First, educate yourself
The seven steps 1. Define Philosophy
2. Benchmark System
3. Resolve Bad Actors
4. Document and Rationalize
5. Audit and Enforce
6. Real-Time Management
7. Control and Maintain
 It is… a
spreadsheet!
… a Swiss
Army Knife
 4 sections
Gray Green Blue Orange

Collect Catorganize Rationalize Configure

It has 4, count em’ FOUR color-coded sections!

 4 sections

Gray Green Blue Orange

Collect Catorganize Rationalize Configure

Collect

COLLECT
Collect

First Step: Dump SCADA points database

into your spreadsheet
Export Points!!!
SCADA key
Point Description
Alarm Text
Station
Configuration Data
Trigger, etc…
Collect
Create a second sheet and dump
Second Step:
your alarm history. We’ll need this
Create a next.
Benchmarking
Workbench
Collect

Look for Pick a time frame to benchmark such

as three months
Patterns
Collect
Pattern:
Frequent
alarms
Collect
Pattern:
Frequent
alarms
The Vital Few:
Obviously Broken
 1/2/2016
1/2/2016 0:20
1/2/2016 0:40
1/2/2016 1:00
1/2/2016 1:20
1/2/2016 1:40
1/2/2016 2:00
1/2/2016 2:20
Collect

1/2/2016 2:40
1/2/2016 3:00
1/2/2016 3:20
1/2/2016 3:40
1/2/2016 4:00
1/2/2016 4:20
1/2/2016 4:40
1/2/2016 5:00
1/2/2016 5:20
1/2/2016 5:40
1/2/2016 6:00
1/2/2016 6:20
1/2/2016 6:40
1/2/2016 7:00
1/2/2016 7:20
Alarm
floods
Pattern:

1/2/2016 7:40
1/2/2016 8:00
1/2/2016 8:20
1/2/2016 8:40
1/2/2016 9:00
1/2/2016 9:20
1/2/2016 9:40
1/2/2016 10:00
1/2/2016 10:20
1/2/2016 10:40
1/2/2016 11:00
1/2/2016 11:20
1/2/2016 11:40
1/2/2016 12:00
1/2/2016 12:20
1/2/2016 12:40
1/2/2016 13:00
1/2/2016 13:20
1/2/2016 13:40
1/2/2016 14:00
1/2/2016 14:20
1/2/2016 14:40
1/2/2016 15:00
1/2/2016 15:20
1/2/2016 15:40
1/2/2016 16:00
1/2/2016 16:20
1/2/2016 16:40
1/2/2016 17:00
1/2/2016 17:20
1/2/2016 17:40
1/2/2016 18:00
This ain’t good

1/2/2016 18:20
1/2/2016 18:40
1/2/2016 19:00
1/2/2016 19:20
1/2/2016 19:40
1/2/2016 20:00
1/2/2016 20:20
1/2/2016 20:40
1/2/2016 21:00
1/2/2016 21:20
1/2/2016 21:40
1/2/2016 22:00
1/2/2016 22:20
1/2/2016 22:40
1/2/2016 23:00
1/2/2016 23:20
1/2/2016 23:40
Collect
1. Read it
Operator 2. Acknowledge it
alarm 3. Navigate to the right

response screen
4. Decide if its even real
5. Determine what to do
6. Make it happen
7. Watch and wait

At least a
few minutes
Collect

How many is 1 alarm every 10 minutes

too many? is ideal

1 alarm every 5 minutes

is manageable

No more than
300 alarms per day
Collect
Logged in spreadsheet:
Recapitulation Created master alarm database with
• Data points that are generating alarms
• Any useful metadata

History and benchmark

Created benchmark workbench with
• Used Excel to find patterns
• Found the Vital Few
• Learned how deep the rabbit hole goes
Catorganize

CATORGANIZE
 Catorganize

Categorize Categorize all points into

and buckets of similar points
Organize
The Vital Few:
80% of your points fall to
20% of your buckets
 Catorganize

Category • MVA line loading

• Voltage limits
• EMS system
• Comm. and health
Examples • Tie line MW shift • XFMR and CB Alarms
• Circuit Breakers and • Relaying and
Switches protection
• Load tap changers • Plant cycle

• Duplicates and points

to disable
Collect Catorganize

Find your experts

No time to
go solo
Seek out other stakeholders
Collect Catorganize

Recapitulation
Logged to spreadsheet:
• Contact – Person or team
• Categories – All of em.
• Notes
Collect Rationalize

RATIONALIZE
Collect Rationalize

Determine
Two Things: Should an How important
alarm exist? is an alarm?
Collect Rationalize

An audible and/or visible means of

“Alarm” indicating
Textbook to the operator an …

Definition abnormal condition

requiring a response. (ISA 18.2)
Collect Rationalize

An audible and/or visible means of

indicating
Should an to the operator an …

alarm exist? abnormal condition

requiring a response. (ISA 18.2)
Collect Rationalize

To the Cause of the abnormal condition

spreadsheet!
Action the Operator should take
Collect Rationalize

How Why
important Focus on what needs to be done next

is it?
How
The operator’s ability to quickly tell
Priority things apart
Collect Rationalize

We’re bad
at this.
Collect Rationalize

3 things
is simple.
Collect Rationalize

Color!
Woah!
Collect Rationalize

For some reason

we do this.
Collect Rationalize

Priority – 3 levels
Collect Rationalize

Priority – 3 levels
80%

High
Medium
15%
Low 5%

Low Medium High

Collect Rationalize

Where to
focus?
Collect Rationalize

Sweet relief!
Collect Rationalize

Priority Make Color Count

80%

High
Medium
15%
Low 5%

Low Medium High

 Collect Rationalize

Severity How
Severe?
Priority
High
Consequence Medium
Low
Time to How long
before it
Respond happens?
Collect Rationalize

Severity
Minor Major Severe
Severity Severity Severity
Collect Rationalize

Severity
Minor Major Severe
Area of Impact
Severity Severity Severity

Compliance

Operation

Generation Loss

Gen Impact to
Grid

Cost
Collect Rationalize

Severity
Minor Major Severe
Area of Impact
Severity Severity Severity
Continued
Compliance File report Report and Fines
Violation
Over/under
Operation Warning Line overloading
voltage

Generation Loss <2% Capacity In between Loss of Unit

Gen Impact to Fixed by Auto Re-dispatching or Potential of

Grid Gen Control purchase brown/blackout

Cost <$10k <=$100k >$100k

Collect Rationalize

• Consistency is the goal

• You’ll hear “It depends” a lot.
• Focus on the one alarm at hand and
Tips don’t play what-if.
• The Operator can do the job of
piecing things together.
• They’ll still get the alarm.
• Can make updates later.
Collect Rationalize

Severity Minor Major Severe

+ ≈ 30 minutes

≈
Low Low Medium

≈ 10
10 minutes
Time minutes

< 3 minutes
< 3 minutes
Low

Medium
Medium

High
Medium

High
Collect Rationalize

Severity + Time
= Minor Major Severe

Priority ≈ 30 minutes Low Low

≈ 10 minutes Low

< 3 minutes
Collect Rationalize

Recapitulation Logged to spreadsheet:

• Cause
• Action
• Consequence
• Severity (Minor, Major, Severe)
• Time (3, 10, 30)
• Priority (L, M, H)
Collect Configure

CONFIGURIZE
Collect Configure

Translate Convert rationalize data into your

specific system settings:

• Alarm Group and Classes

• Priority and Color
• Sound
• Email
• Operating Guide or Procedure
• Deadbanding
 FEP
OpenVIEW
OpenNERC
HIGH…
MTR…
N2_3
A4_TNAN
A4_N1AA
Collect

T2_NCAA
R3_1
ICCP
S2_10
V2_10
N1_3
AL_NCAA
Gen_Con…
T1_TNAN
T2_1
T1_NCAA
A4_TNB
AL_10
N3_3
T1_N_A
Deadbanding

AL
A4_N0AA
UNIT…
T1_TNB
T1_1
ALERT2
AL_TNAN
AL_1
T2_TNAN

1/2/2016
1/2/2016 0:40
1/2/2016 1:20
1/2/2016 2:00
1/2/2016 2:40
1/2/2016 3:20
1/2/2016 4:00
1/2/2016 4:40
1/2/2016 5:20
Slow these down!

1/2/2016 6:00
1/2/2016 6:40
1/2/2016 7:20
1/2/2016 8:00
1/2/2016 8:40
1/2/2016 9:20
1/2/2016 10:00
1/2/2016 10:40
1/2/2016 11:20
1/2/2016 12:00
1/2/2016 12:40
1/2/2016 13:20
1/2/2016 14:00
1/2/2016 14:40
1/2/2016 15:20
1/2/2016 16:00
1/2/2016 16:40
1/2/2016 17:20
1/2/2016 18:00
1/2/2016 18:40
1/2/2016 19:20
1/2/2016 20:00
1/2/2016 20:40
1/2/2016 21:20
1/2/2016 22:00
1/2/2016 22:40
1/2/2016 23:20
Configure
Collect Configure

Deadbanding
A little goes a long way.
– 1-2% on the value
– 20 second delay or less

Wait to call it Normal

– Alarms immediately
– Prevents chatter
Collect Configure

Recapitulation
Logged to spreadsheet:
• Configuration
• Alarm Groups and Classes
• Color, Priority, Sound, Email

• Deadbands
All done right?!

Gray Green Blue Orange

Collect Catorganize Rationalize Configure

 One
Last
Thing
Turn this…
Into this!
Goodbye 1. It’s not so hard.
2. You can have my stuff.
 Brandon Harrington
bharrington@aeci.org

QUESTIONS?