Switchgear Application Issues for Mission Critical Power Systems, Part One

ABSTRACT With the advent of high reliability data centers and telecommunication facilities, designers of commercial power face new challenges including local power generation and advanced transfer systems. This article is intended to explore the fundamental design issues pertaining to switchgear equipment used in these applications. Part one will discuss system voltage selection and control system design. Part two is dedicated to the subject of fault tolerance and two fundamental design issues with ground fault protection.

I. INTRODUCTION Fueled by advances in communication technology and the explosive expansion of the national fiber optic backbone, the 1990s decade brought about unprecedented growth in data center construction. This growth presented a special problem to data center designers. The sheer number of facilities being built necessitated a high volume design approach, yet there was no industry consensus on the reliability requirements of data centers. Today we see a variety of design approaches from which three common themes have emerged: 1. Utility source is generally considered to be at 99.9% availability level. This necessitates local power generation in the form of standby or prime power generators. 2. Maintainability is engineered into the data center design through redundant power flow paths such that scheduled maintenance can be performed on any piece of equipment without down time (maintenance bypass). 3. Redundant power paths are designed in such a way that an unscheduled loss of any one path will not cause loss of the load. Figure 1 presents a simple power distribution, which accomplishes most of these tasks (All illustrations are at the end of the document). Circuit breakers within switchgear equipment perform the necessary switching operations. We will use Figure 1 throughout the remainder of this document to illustrate the fundamental design considerations with mission critical power systems. II. MEDIUM VOLTAGE VERSUS LOW VOLTAGE TRANSFER Utility company plays a major part in the voltage selection process. Large data centers are typically supplied directly from distribution lines at medium voltage (1000V to 38,000V). The three main voltage classes in this category are 5KV, 15KV, 27KV and 38KV. Small facilities are typically supplied at 480V from utility pad mounted transformers. Multiple utility services are provided along with local generators. At some point within the power distribution, these alternate sources must be switched. Designers of large data centers must choose the voltage for switching operations. Figure 2 provides the medium voltage switching alternate to Figure 1. In this scheme, 15KV generators are switched with the two utility sources. Switching operation at medium and low voltage are very different. Vacuum interrupters are commonly used for medium voltage switching operations. SF6 is an alternate technology. In comparison, low

voltage power circuit breakers continue to utilize air break technology. Each device is optimized for long life and ease of maintenance. Power system parameters are also significantly different between medium voltage and low voltage systems. Compared to low voltage systems, medium voltages allow us to transfer power over long distances. Long cable lengths introduce additional capacitance, which affects the transient response of the power system. Frequent switching is the common characteristic of automatic transfer systems. As most transient phenomena are statistical, frequent operation naturally increases the exposure of the data center. These transients have been known and documented under various headings - since medium and high voltages were introduced into utility systems. However, such troublesome occurrences were rare within commercial power distribution. This is because, unlike data centers, typical commercial power is not frequently switched. Transient Recovery and Multiple Re-ignitions Transient Recovery Voltage (TRV) is the voltage that appears across circuit breaker contacts during the interruption process (Figure 3). Of special interest is short circuit TRV, which is the transient recovery voltage associated with short circuit current interruption. TRV is a high frequency oscillation and a function of power system capacitance and inductance. Calculation of TRV is not as straight forward as calculations pertaining to fault currents and load flow. Special transient simulations may be required to estimate TRV values. ANSI standard C37 series stipulate the TRV requirements for medium voltage circuit breaker application. However, for commercial power design, these requirements are often ignored and circuit breakers are specified solely on the basis of their short circuit and continuous current ratings. This can bring about a non-optimum application of circuit breakers. Multiple re-ignition is a phenomenon associated with the arc extinction process inside the medium voltage circuit breaker interruption chamber. If multiple re-ignitions occur, we encounter the possibility of other high frequency resonance problems within the power system. The exact physics of multiple reignitions is not thoroughly documented. Various researchers have provided explanations of this phenomenon and offered mitigation methodologies. The point of the above discussion is not to discourage medium voltage transfer. The point to be taken is that a more in-depth analysis of the power system is required where medium voltage transfer is employed. This illustrates the value of the engineering specialty for mission critical power. III. SWITCHGEAR CONTROL SYSTEM DESIGN Design of the automatic transfer system is an art of its own. Even though this art has been around for many years, it has not been adequately documented or standardized. However, there are some fundamental principles, or axioms, which can be applied to this field. These principles can also be incorporated into the project specifications. From each circuit breaker we derive three sets of signals in the form of open/closed contacts: 1. The auxiliary contacts give us the status of the circuit breaker 2. The bell alarm contact or overcurrent trip switch closes only when the low voltage circuit breaker trips due to a fault. For medium voltage circuit breakers the overcurrent relay output is used directly

3. The cell switch tells us whether the drawout type circuit breaker is in the connected or test position The first intuitive distinction to be made is by looking over the control schematics. Are the above three types of contacts used in the control system? Referring to figure 1, lets assume that the bell alarm contact from circuit breaker MA is not used in the control system. Now assume that a fault exists on bus A. Circuit breaker MA will trip in response to this fault. But if the bell alarm contact is not used, the automatic control system will start generator 1 and close circuit breaker GA into a dead bus fault a very non-optimum action for a mission critical system. So, in response to a fault, the system must lock itself out. The bell alarm contact must be used in the control system. If the cell switch is not used, circuit breaker maintenance will be hampered. Referring to figure 1, lets assume that circuit breaker MA is put into the test position. By putting the circuit breaker in test, maintenance personnel will be able to open and close the circuit breaker several times to assure its proper operation. But each time the circuit breaker is opened, the automatic control system is liable to attempt to perform other switching operations in order to correct what it perceives to be loss of power to the bus. The cell switch can tell the control system to stand down when circuit breaker testing is being performed. Cell switch must be used in the control system. It is not too difficult to distinguish the amateur from the pro in this field. The pro knows how to utilize all the tools available to him. You may be thinking that the above points are so commonplace that they will never be overlooked that is until you come across a system in the field that violates these basic rules. Microprocessor Control Nirvana What about the microprocessor revolution? The old analog devices have a tendency to crowd the doors and covers on switchgear equipment and they have no recording capability. With modern microprocessor control systems, we can create a database of system events, we can automate alarming and monitoring and we can have access to information anywhere such as on the world wide web. There is enormous advantage is utilizing microprocessor and communication technologies especially in a data center. But we should not ignore the fundamentals. Mission critical systems, by definition, must be fault tolerant, including a condition where the microprocessor based system is not operational. This can happen due to control power loss or other factors. In order to create a high level of reliability, we must design the system with maximum redundant automation. But in order to give the system full fault tolerance, we must have a manual back up, which is completely dependent on human interaction. So we must have two independent control systems. The automatic system and the manual system must be completely independent of one another. A problem arises when these two systems are combined. For example, it is common to run all input and output signals of circuit breakers into a PLC (Programmable Logic Controller). If the manual controls are in any way connected to the PLC, the purpose of manual control is defeated. Manual controls must have a completely redundant path to the circuit breakers they control. The way this paradox of automatic / manual control is resolved is by observing a set of basic guidelines: 1. A Manual / Auto switch must be provided to give the operator the choice between the modes of operation

2. The manual mode must have all anti-paralleling interlocking in place independent of the automatic system. The anti-paralleling interlocks will not allow two sources to be closed together on the same bus. 3. An analog voltmeter is necessary on each source. The old analog indicating voltmeters are indispensable for this application. Digital devices depend on control power availability. So here is the basic science behind this art: 1. Design your manual controls independent of any electronic systems. 2. Place your automatic controls on top of this manual layer. You can get as creative as you want with the automatic controls, using the latest technology with redundant microprocessors and communication systems. Control Power can be the Show Stopper Design of the control power is the most fundamental and the most important consideration for mission critical power transfer systems. If this step is over-looked, the most redundant automatic system will be dysfunctional when it is most needed. A reliable source of control power is the conditioned power from UPS (Uninterruptible Power Supply) equipment inside the data center. Use the conditioned data center power as the primary control power source. Use the various switched sources as alternate sources. Avoid conditioning and storage devices that may not receive sufficient maintenance over time. Where UPS equipment or storage batteries are used, these devices must be readily accessible and conspicuous. IV. CONCLUSIONS Design of mission critical power systems is an art and a science. The need for high reliability forces us to re-think our traditional design practices. Switchgear equipment is at the heart of data center power distribution. Switchgear reliability directly impacts data center reliability. In the process of re-thinking traditional design concepts we must not abandon the basic principles. Microprocessor revolution and communication technologies have changed the way we look at the world around us. But they do not change the fundamentals of this art.

Switchgear Application Issues for Mission Critical Power Systems, Part Two

ABSTRACT Within part one we explored selection of system voltage for automatic transfer operation. We also discussed several issues in the design of switchgear automatic control systems. In part two, we explore the issue of fault tolerance and the frailties of ground fault protection. I. INTRODUCTION One of the most basic considerations in the design of data center power is the choice of system grounding. Commercial power systems are typically solidly grounded. In comparison, modern data centers are often high resistance grounded. In a high resistance grounded system, the first ground fault will initiate an alarm instead of tripping circuit breakers. However, many data center designers continue to use the solidly grounded power system for various reasons. This article discusses fault tolerance and two basic pitfalls that data center designers face when utilizing solidly grounded power systems. II. DESIGN FOR FAULT TOLERANCE Examining Figure 1, if we have a fault on Bus A, the utility source will trip and the generator will not start. Therefore, bus A will not be able to supply the loads. Uninterruptible Power Supply (UPS) equipment on side A can only carry the load for a limited period of time until their battery runs down. However, static switches inside the PDU (Power Distribution Unit) equipment will continue to supply the load by transferring to bus B power. Therefore, the computer loads will not be dropped. Similarly, examining figure 2, we observe that a fault on the generator bus A will remove this bus. But side B can continue supplying power to the data center due to the existence of the medium voltage tie circuit breaker. If the tie circuit breaker were not included, the entire load would be dropped. Further improvement can be envisioned by supplying the UPS static bypass devices from an alternate source - third utility source backed up by a third generator. While this introduces other factors, such as synchronism of various sources, overall power availability is improved by increasing redundancy. Similar analyses can and must be applied at every point within the power system. III. THE FIRST PITFALL IN GROUND FAULT PROTECTION When we speak of a fault in any power system, generally we are speaking of ground fault. Ground faults are the most common type of fault. In cable distribution systems, they are usually the result of insulation deterioration due to aging. However, they can be caused by human error during equipment maintenance. Solid grounding provides a return path for ground fault currents. Therefore, tripped overcurrent protective devices easily identify the location of the fault. However, when multiple sources of power are tied together, ground fault protection gets complicated. One major source of complication is the way system neutral conductors are used. In order to supply 277 V fluorescent lighting a small portion of the total load in a data center - system designers often extend neutral conductors throughout the power distribution system. Figure 4 illustrates the neutral conductor connections for a portion of the system in figure 1.

With this arrangement of neutral conductors, a fault on bus A will have multiple return paths to the source. Normally we expect the ground current depicted in figure 4 to return through the neutral to ground bond of the utility transformer. However, as illustrated in figure 4, a portion of this current may return through the neutral to ground bond of the generator, travel over the neutral bus and return to the utility source as neutral current. This can complicate ground fault protection. Circuit breaker MA may not sense sufficient ground current to trip, while circuit breaker GA senses a ground current even if the generator is off. Therefore, some designers argue that the circuit depicted in figure 4 is a violation of the National Electrical Code (NEC) Article 250-6 due to the multiple ground connections employed. Local electrical inspectors may in fact interpret article 250-6 to disallow the connection shown in figure 4. But such interpretation is not universal. If we had utilized a single neutral to ground connection, (figure 5) we would still have a problem with ground fault protection. If the generator is on and the utility source is off, the fault current will return to the generator as neutral current defeating the ground fault system. Additionally, often circuit breakers MA and GA are in different switchboards. Considering figure 5, if the neutral conductor is somehow disconnected between bus A and the generator, the generator source will be ungrounded an undesirable condition and a violation of NEC. Therefore, single point grounding is not an optimum solution. The problem shown in figure 4 gets much more complicated when we consider the neutral conductors on bus B and the neutrals derived from the UPS sources. The real problem is in tying together the neutral conductors of multiple systems. Even though there is no standardization in this area, switchgear manufacturers can solve these problems by proper design of ground fault protection. This may necessitate employing extra current sensors and dual ground fault sensing units on circuit breakers. But this added complexity must be weighed against the need for extending the neutrals. If 277 V power is needed for fluorescent lights, a stable neutral can be easily derived downstream using an isolation transformer. Eliminating the neutral conductors from the remainder of the system will save significant amount of copper, which will help defer the cost of an isolation transformer for lighting loads. IV. THE SECOND PITFALL IN GROUND FAULT PROTECTION

This second issue is rather insidious, as it is not very evident. Figure 6 shows the system of figures 4 and 5, but with neutral conductors removed and each source properly grounded. Even though the system in figure 6 will respond correctly to fault currents, it can cause spurious tripping of circuit breakers due to circulating currents. Circulating currents sometimes exist when multiple sources of power are tied together. They appear due to differences in the magnitudes or phase angles of source voltages and they may exist due to harmonics. Third harmonic currents have been known to circulate between generator and utility sources under certain conditions. Figure 6 illustrates the circulating current. Ground fault protection of both circuit breakers MA and GA will interpret this current as ground fault, causing incorrect circuit breaker tripping. Therefore, the simple radial ground fault protection should not be applied to multiple source mission critical systems. If there were a direct connection between bus A and bus B such as by a tie circuit breaker the circulating current problem would get more complicated. The complication grows in proportion to the

number of inter-tied sources. But all such problems can be solved by proper design of the ground fault system. For example, the circulating current problem depicted in Figure 6 can be corrected by tying the ground fault sensors together as shown in figure 7. Working many scenarios of circulating currents and ground faults will prove that the circuit shown in figure 7 is workable. V. CONCLUSIONS

Fault tolerance must be engineered into the power system using an intuitive process. The system must have a method of continuing power supply through a redundant path to computer loads when there is a fault on any bus. Ground fault protection design is significantly simplified if the neutral conductors from multiple systems are not tied together. Stable grounded neutral can be derived close to the 277 V lighting loads where it is needed. Isolation transformers have been successfully employed in this application for many years. Presence of multiple sources necessitates evaluating circulating currents for mission critical power systems. Ground fault protection may malfunction due to circulating currents if the system is not properly designed.

UTILITY A GEN A

UTILITY B GEN B

BUS A

BUS B

PDU FROM ALTERNATE PDU

COMPUTER

Figure 1: Basic Data Center Power Distribution

UTILITY A

GEN A

GEN B

UTILITY B

BUS A

BUS B

PDU FROM ALTERNATE PDU

COMPUTER

Figure 2: Medium Voltage Version of Data Center Power System

TRV

FAULT

TRV1>C1
TRV1>C1 -C2 (Type 4) 20000 TRV1>XX0001-XX0011(Type 8)
CURRENT

-C2

(Type 4)
TRV1>XX0001(Type 4)

10000

Voltage (V)

-10000

-20000
TRV

-30000 0
SOURCE VOLTAGE

10 Time (ms)

15

20

Figure 3: Transient Recovery Voltage Due to Short Circuit Current Interruption. Circuit Breaker Contacts Part at Time Zero. Arcing Continues Until Current Goes to Zero. TRV Appears Across the Circuit Breaker Contacts As Soon As Current Goes to Zero.

10

UTILITY SOURCE A

GEN A

MA

GA

GROUND FAULT SENSOR

BUS A

FAULT

Figure 4: Multiple Return Paths Cause Ground Fault Mis-operation

UTILITY SOURCE A

GEN A

MA

GA

BUS A

FAULT

Figure 5: Single Point Grounding Does Not Solve Ground Fault Mis-operation Problem

11

UTILITY SOURCE A

GEN A

MA

GA

BUS A

Figure 6: Circulating currents can cause ground fault nuisance tripping

UTILITY SOURCE A

GEN A GROUND FAULT TRIP

MA

GA

MA

GA

BUS A

Figure 7: Properly Designed Ground Fault Protection Requires Special Connection of Sensors

12