ADVISORY STAFFING

2017
&TRAINING

How to Develop a Data Center

Training Program

• Training progression
• Training material
• Who should be trained
• Methods, materials, and evaluations
Uptime Institute Inside Track provides its subscribers with actionable
resources and access to an elite peer community to ensure that IT functions
as a competitive advantage rather than cost center. Uptime Institute is
the IT industry’s most trusted and adopted global standard for the proper
design, build and operation of data centers – the backbone of the digital
economy. For over 20 years, Uptime Institute has been providing customers
with the assurance that their digital infrastructure can perform at a level that
is consistent with their business needs, across a wide array of operating
conditions. With its data center Tier Standard & Certifications, Management
& Operations reviews, Efficient IT Stamp of Approval, and accredited
educational curriculum for data center professionals, Uptime Institute helps
organizations optimize critical IT assets while managing costs, resources
and efficiency. Uptime Institute has become the de facto standard for
data center reliability, sustainability and efficiency. Today, thousands of
companies rely on Uptime Institute to enable their digital-centric business
success. Uptime Institute, a division of The 451 Group, has office locations
throughout the world, including those in North and South America, Europe,
U.A.E., Russia, and Asia.

For more information on Inside Track please contact Matt Stansberry

mstansberry@uptimeinstitute.com

Uptime Institute is an unbiased advisory organization focused on improving the performance, efficiency, and reliability of business
critical infrastructure through innovation, collaboration, and independent certifications. Uptime Institute serves all stakeholders
responsible for IT service availability through industry leading standards, education, peer-to-peer networking, consulting,
and award programs delivered to enterprise organizations and third-party operators, manufacturers, and providers. Uptime
Institute is recognized globally for the creation and administration of the Tier Standards & Certifications for Data Center Design,
Construction, and Operations, along with its Management & Operations (M&O) Stamp of Approval, FORCSS® methodology, and
Efficient IT Stamp of Approval.
 
Uptime Institute – The Global Data Center Authority®, a division of The 451 Group, has office locations in the U.S., Mexico, Costa
Rica, Brazil, U.K., Spain, U.A.E., Russia, Taiwan, Singapore, and Malaysia.
Visit www.uptimeinstitute.com for more information.

All general queries:

Uptime Institute
5470 Shilshole Avenue NW, Suite 500
Seattle, WA 98107
USA
+1 206 783 0510

info@uptimeinstitute.com

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

2
ADVISORY | 2017

How to Develop a Data Center

Training Program
The majority of data center outages are blamed on “human error” stemming from a lack of
knowledge or procedural mistake. Inadequate training can lead to a situation where operators
are unprepared to respond to an incident and mishandle it. A robust training program drives
reliability and predictable outcomes. A highly trained staff will also be equipped to maximize
infrastructure utilization and can extend the life of your existing assets.

A formal data center training program provides staff with site specific knowledge, skills, and
experience to perform regular activities and to respond to abnormal incidents. This guide
will explain the basic progression of the development of a data center training program, offer
recommendations for materials a training program should cover, and provide guidance on
which personnel should be trained.

This guide will also discuss the methods, materials, evaluation, and tracking required to
implement an effective training program. Lastly, keep in mind that training is an iterative and
ongoing process. This guide can supplement your existing training program.

Training Program Development Flowchart:

Here is a straightforward timeline flow diagram that can be used to guide the institution and
management of the training:

ID Training ID Personnel to Determine Training

Topics be Trained Training Method Materials

Determine
Implement Track and
Qualification
Training Record
Method

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

3
ADVISORY | 2017

How to Develop a Data Center

Training Program
Identify Training Topics

What should a data center training program ID Topics ID Personnel Determine Develop

accomplish? Method Materials

Perhaps we can start with a different question:

What should a data center facility operations Track and
Qualification
team know and be able to do? What general
Implement
Record

abilities should it have in order to meet the

typical data center’s mission requirement?
Answering these types of questions results in lists of knowledge and abilities that will correlate
to training topics. So our first step in developing a training program is to compile a list of training
topics. The following category list can be a guide to selecting the training topics that can be used
to begin to formulate a formal data center facility operations training program.

Operational Readiness:
Know how the specific facility functions from an overall, integrated systems perspective.
Know what the normal operating site conditions are. Be able quickly and correctly
identify abnormal conditions and then respond to them correctly using established, well-
rehearsed incident response process and procedure.

If the most basic purpose of a data center facility operations team is to staff a site,
monitor systems, and respond to emergencies, then it follows that the team as a whole
should possess an excellent overall understanding of the site-specific nature of the
design of the facility, its normal operating behavior, and very high familiarity with a set of
prepared responses to the most likely abnormal condition scenarios.

Typical abnormal and emergency operations scenarios include:

• Loss of municipal power and/or water
• Activation of remote emergency power off (EPO) button
• Activation of fire suppression system
• UPS fail to bypass
• Loss of chiller
• And so on…

As the list becomes more specific, the exact scenario list becomes highly site dependent and
must be evaluated systematically.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

4
ADVISORY | 2017

How to Develop a Data Center

Training Program
Training materials and resources in this category would include but not be limited to:
• Basis of Design (BOD) narratives
• Sequence of operations (SOO) narratives
• Building management system (BMS) alarm matrix
• Access to the original design engineers
• Commissioning documents
• Record drawings, including electrical and mechanical one-line diagrams
• Incident response policy and associated procedures
• Emergency response procedures (ERP) or Emergency Operating Procedures (EOP)
• Facility rounds sheets

Policies, Processes, and Procedures:

Understand the site’s access control requirements and what processes, and procedures must
be conformed to for all normal, routine, and recurring types of activities performed at the
data center.

Data center facility operations organizations typically exert control over their environments
through the use procedure based operational methodologies that rely on the use of highly
reviewed, approved procedures, and access control policies. The common expectation from
these methodologies is predictability through adherence to policy and procedure. To realize this
expectation, operations personnel must be indoctrinated into the methodology and the
operational culture that relies on it to prevent operator-induced failures.

Associated materials and resources to utilize for training in policies, processes and procedures:
• Operations policies that articulate the basis of the operational methodology, its importance,
and its adherence requirements
• Critical facility work rules
• Housekeeping policies
• Access policies
• The procedures themselves via the existing procedure library

Site and systems configuration and operations:

Understand standard change-of-state for systems and routine equipment configurations.
Maintain normal desired operating conditions through management of alarm and equipment
set points.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

5
ADVISORY | 2017

How to Develop a Data Center

Training Program
Data center facility operators are usually tasked to perform all routine change of state and
systems or equipment configurations to prepare for maintenance, such as enabling the external
maintenance bypass for a UPS, or to ensure uniform hours usage on major assets, such as
rotating chillers or pumps, or changing preferred sources on devices such as static and
automatic transfer switches to facilitate maintenance or formal site configuration changes.

Associated training materials and resources:

• Standard operating procedures (SOP)
• Maintenance procedures
• Operating procedures
• Operations policies defining alarm and set-point configuration

Vendor Support:
Understand the role of vendors and the processes and procedures used to manage their
activities.

Most data centers rely on original equipment manufacturer (OEM) approved vendors to perform
highly technical or proprietary maintenance on installed assets. Facility operations personnel
need to understand the exact role of vendors and the policies and procedures that govern their
access, escort, and supervision.

Associated training materials and resources:

• Operations policies that detail the role of support vendors
• Critical facility work rules
• Housekeeping policies
• Access policies
• Emergency response plans

Applications:
Understand how to use the computer-based applications the site relies on to schedule,
review, approve, and track all activities as well as facilitate all other aspects of facility
operations job functions.

Operations personnel need excellent fluency with the applications used at the facility. The
typical applications in use at data centers are a computerized maintenance management
system (CMMS) and a change control ticketing system, accompanied by soft copy document
storage and management systems. Additional applications can include contemporary data

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

6
ADVISORY | 2017

How to Develop a Data Center

Training Program
center infrastructure management (DCIM) systems and various web portals.

Associated training materials and resources:

• Vendor provided training, classroom or course based
• OEM provided tutorials
• OEM manuals and Internet-sourced training

Basic MEP Theory:

Understand the basic theory of operation of the data center’s critical mechanical, electrical, and
plumbing (MEP) systems and their subcomponents.

Associated training materials and resources:

• Commodity “data center university” courses and programs
• Professional Engineers
• Trade reference manuals
• OEM sponsored or performed training sessions
• Manuals for site-specific equipment
• Internet resources
• Resident team trade and institutional knowledge

Maintenance:
Understand the maintenance requirements for all the site’s assets, how they are formulated,
and how those requirements are met.

Facility operations personnel usually administer or control all planned and unplanned
maintenance activities, independent of who performs maintenance at the facility. The
overall formalized preventive maintenance program and the specific maintenance
identified and performed on all assets must be well understood by the team.

Associated training materials and resources:

• Formalized program or company policies
• OEM requirements
• Corporate maintenance standards
• Major maintenance plans and calendars
• Standards bodies: ASHRAE, National Electrical Manufacturers Association (NEMA),
International Electrical Testing Association (NETA), Institute of Electrical and Electronics
Engineers (IEEE), American National Standards Institute (ANSI), etc.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

7
ADVISORY | 2017

How to Develop a Data Center

Training Program
• Vendor recommendations
• Institutional knowledge
• Design engineer recommendations

Compliance:
Understand and comply with all corporate and authority having jurisdiction/local authority having
jurisdiction (AHJ/LAHJ) compliance requirements for operating and maintaining the facility
(safety, hazardous materials, spill prevention and control, licenses, etc.).

Personnel must be trained and in compliance with Occupational Safety and Hazards Agency
(OSHA) and state OSHA requirements, to include National Fire Protection Association (NFPA)
70E compliance, the handling and disposal of hazardous materials such as fuel oil, battery
materials and refrigerants, and hold all licenses and certifications required by AHJ. Additionally,
there may be additional corporate compliance required training of the team.

Associated training materials and resources:

• NFPA 70E, OSHA, U.S. Environmental Protection Agency (EPA)
• Arc flash studies and short circuit coordination studies
• Site-specific lock out/tag out program
• Corporate environmental health and safety organization
• State and city websites
• County fire marshall

Information Access:
Understand how to access all the site and corporate informational resources, from BMS/EPMS,
to DCIM systems, to all document and documentation resource location, such as record
drawing, as-built drawings, equipment manuals, design, construction, and operating manuals,
commissioning reports and SOO for mechanical and electrical and integrated systems such as
the fire suppression system and any EPO system. Include everything from policy locations,
procedure library locations, and escalation and vendor contact lists to shift schedules, etc.

Associated training materials and resources:

• Reference library
• BMS, emergency power management system (EPMS), DCIM
• Computer-based file management systems

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

8
ADVISORY | 2017

How to Develop a Data Center

Training Program
Space Power and Cooling Resource Allocation and Tracking:
Understand how to track, provision, and manage the site’s space, power, and cooling (SPC)
resources and all the workflows around IT deployments requiring these resources.

Associated training materials and resources:

• SPC tracking and trending and forecasting tools (spreadsheets, etc.)
• Data modeling from DCIM and BMS/EPMS system consultation
• Any specific resource utilization dashboards
• Formalized equipment deployment workflows and associated company policies

Subject Matter Expert Specialization:

Ultimately acquire expert level or specialized understanding of the technology and systems of
the site. This expertise can then be used to resolve longstanding issues, improve the design,
improve energy efficiency, and identify and reduce risk. This high level of expertise could also
be used for continuous quality improvement of processes, procedures, and, ultimately, training.

Associated training materials and resources:

• Detailed study of design documents, SOO documents, and equipment manuals
• Vendor or OEM-sourced training course
• Independent study
• Constant on-the-job (OTJ) learning

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

9
ADVISORY | 2017

How to Develop a Data Center

Training Program
Identify Personnel to be Trained

Who will receive training?

ID Topics ID Personnel Determine Develop
Method Materials

Once training topics have been

comprehensively identified, the next
step identifies everyone who will
receive training in the facility operations Qualification Implement
Track and
Record
organization. This consists of the
following:

1. Identify all job positions. These could include such positions and functions as:
• Chief engineer/Critical facility manager
• Data center technician/Facility engineer
• Team lead/Assistant manager
• Application administrator

2. Determine training requirements for each position:

• Can include part or all of the identified training topics
• Training within topics should be progressive based on experience and role

3. Determine training requirements for vendors:

• Access control policy
• Critical facility work rules or data center house rules
• Pre-work and post work requirements
• Emergency response

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

10
ADVISORY | 2017

How to Develop a Data Center

Training Program
Determine Training Method

There are a variety of training ID Topics ID Personnel Determine Develop

methodologies that can be employed Method Materials

to conduct actual training. These

methodologies can vary throughout the
training program and can include but are
Track and
not limited to: Qualification Implement
Record

• OJT
• Tabletop discussions and job/task/activity walks
• Classroom sessions
• Outsourced training, conducted on or off site
• Online tutorials and web-based training sessions
• Self-study and learning programs
• Shift drills

OJT

• Typically performed by the most experienced on-site personnel

• Usually focused on new employees
• OJT is site specific training typically involving:
• Access to the site, building, and computer room
• Incident response and event notification policy, process, and procedure
• Critical facility or data center house work rules
• General housekeeping, tools, parts, and material management and storage
• Maintenance activities and the use of associated procedures and applications
• Shift monitoring requirements such as BMS and site rounds

Formal Training

• Training conducted in a classroom setting with an instructor and reference materials

• Can include various types of operational demonstrations
• Can take the form of online courses and independent study

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

11
ADVISORY | 2017

How to Develop a Data Center

Training Program
Training Materials

Now that the training topics, personnel, ID Topics ID Personnel Determine Develop
and training methods have been identified, Method Materials

the overall program content and structure

should be formalized and captured with
two types of training program materials:
Qualification Track and
Implement
Record

Syllabus: The training program syllabus

outlines the entire framework of the formal
training program and any associated qualification program. It lists the training topics, who will
receive the training, the method to be used for the training, and when each trainee should be
trained.

Lesson Plans: A lesson plan provides the scope of each training activity and identifies:
• The instructor
• The specific training activity content
• The goal of the training
• The successful completion of the training requirements

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

12
ADVISORY | 2017

How to Develop a Data Center

Training Program
Determine Qualification Method

Once training has been conducted, ID Topics ID Personnel Determine Develop

there must be some way to measure Method Materials

how well the trainee has mastered the

material presented in the training. This
measurement of the success of the
Track and
training session or course usually takes Qualification Implement
Record

the shape of one of the following:

• Written examinations
• Oral examinations
• Demonstrations

The quality of these measurements is just as important as the quality of the training materials
themselves. Particularly with written examinations, care must be taken not to design questions
that are either too easy (give away multiple choice) or “tricky” (requiring extra effort using
discerning abilities that create unhelpful examination stress and can actually interfere with the
learning measurement that is desired).

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

13
ADVISORY | 2017

How to Develop a Data Center

Training Program
Implement Training

Training Schedule: All training should be ID Topics ID Personnel Determine Develop

planned out well in advance with a master Method Materials

training schedule that identifies when each

training activity will be conducted for every
team member. This schedule may have
Track and
specific dates or generic timelines based Qualification Implement
Record

on the date of hire for all personnel.

Refresher Training: An important consideration in the training schedule is the recurrence

of training. Frequencies should be instituted for specific training, especially that needed for
compliance with formal policies, processes, and procedures.

Start Training! Once the schedule is in place, a commitment has been made to actually
conduct the assigned training in the manner and method prescribed by the program.
Management must ensure the scheduled training occurs as expected.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

14
ADVISORY | 2017

How to Develop a Data Center

Training Program
Track and Record

Now that training is now being conducted ID Topics ID Personnel Determine Develop
in a regular fashion, completed assigned Method Materials

training in any form (classroom, online

courses, etc.) and the success of that
training must now be tracked. Individual
Track and
training records should include: Qualification Implement
Record

• Personal data
• Licensing requirements and status
• Any pre-employment interview assessment or testing results
• Professional growth plan
• Record of training attended
• Examination results (where applicable)
• Any individual certifications obtained
• Any team certifications obtained
• Recertification and refresher training requirements

Simple spreadsheets as shown in the example below can be used to track and record training.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

15
ADVISORY | 2017

How to Develop a Data Center

Training Program
Personnel Consideration

Raw Material
Those considering a career in facilities operations can be a diverse group, with varied
educational, training, and experience backgrounds. Many candidates have a trades
background, having gone through an apprenticeship program and are licensed or otherwise
certified in a relevant trade. Many aspiring team members have experience in the armed
forces, particularly the nuclear Navy. College graduates with project management, engineering,
and other degrees are learning that a career in facilities operations, particularly mission critical
facilities operations, offers interesting challenges, opportunities for professional growth, and
the job security of a growing market. Any and all of these backgrounds produce desirable
candidates, and the best team is often a diverse one, with various backgrounds represented.
These are the raw material from which, with the benefit of an effective and properly applied
training program, an excellent facility operations team can be realized.

Training Program General Misconceptions and Assumptions

A Learning Management System is required.

Many mission critical facilities operations teams determine that a software-based, learning
management tool is a basic necessity before going forward with instituting a formal training
program. This starts a process of trying to determine the best software for your organization
and the formation of committees assigned to gather information for logistical and budgetary
consideration. This can be a lengthy process, and while software-based systems can be
extremely effective at managing the program, they are not a necessity for getting started.
Program requirements can be identified, scheduled, documented, and tracked using familiar,
universal tools such as centrally located and managed spreadsheets and flat files.

Data center subject matter experts are the goal of training programs.
Many begin the process of developing a formal training and qualification program by
identifying the qualifications of the perfect facility technician/operating engineer. This
perception is based on the assumption that it should be a requirement to have staff that
is fully knowledgeable in electrical theory (to include UPS and power distribution), code,
practical wiring, advanced refrigeration theory, and mechanical code and who also possess
electrical and HVAC licenses.Technical writing skills are also viewed as required, as are
specialist level expertise in life safety, CMMS, battery monitoring, energy management, and
power generation. These highly technical requirements often become the general basis for
the training topic in a formal data center operations training program.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

16
ADVISORY | 2017

How to Develop a Data Center

Training Program
The reality is the facility technician or operating engineer needs the site-specific knowledge,
skills, and experience need to perform well-defined, regular activities that take place for his/her
role and responsibility at the data center. These are usually based on site rounds, monitoring
systems, and escorting and supervising vendors, who often have the advanced expertise and
are charged with holding any required advanced licenses. Facility technicians are expected
to understand and adhere to company policy and correctly use pre-formulated process and
procedure, especially in the context of an incident. While it can be very beneficial to acquire or
develop staff who have advanced technical knowledge and licenses, the basis of the training
program should be derived from the actual activities the facility technicians perform at the site
and the operational readiness they are expected to possess in the event of an incident.

Conclusions:

• A formal data center training program consists of detailed identification requirements,

choosing training methodologies for specific training content, then scheduling,
conducting, measuring, and tracking that training using systematic approaches that
don’t require high-tech resources.

• Data center operators need to be trained for operational readiness and all other
planned activities that take place on a regular basis at the site.

• Traditional, intuitive concepts that base required training on in-depth trade and
technology theory and SME levels of understanding, while ultimately beneficial, do not
necessarily reflect the key training requirements for the site.

• Personnel who come from operational cultures of advanced operational readiness

serve the data center facility operations industry well.

• Training is an iterative process that never stops.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

17
ADVISORY | 2017

How to Develop a Data Center

Training Program

Author

Stephen Burgess, Uptime Institute

Uptime Institute Consultant Stephen Burgess is a highly

experienced data center facility technologist, with extensive
experience managing mission critical data center facility
infrastructure design and engineering operations teams.

© COPYRIGHT 2017 UPTIME INSTITUTE. ALL RIGHTS RESERVED

18