Академический Документы
Профессиональный Документы
Культура Документы
Incident Category:
1 a) Actual or Potential Injury/Occupational Illness:
2 b) Process Safety incidents:
3
c) Miscellaneous incidents:
As GT-604 tripped, Flying Take-over of HRSG-611 occurred successfully from TEG to AA mode. However, HRSG-611
later tripped due to malfunction of fuel gas valve positioner (PIC-618AV)/Low air-pressure security (PIC-618AV
positioner was found faulty during subsequent start-up). Following GT-604 main circuit breaker trip, ILMS shed load as
per its logic. However, since PRC-2A and 2B were closing, GT-601 and 602 failed to normalize the system frequency.
GT-601 and 602 tripped on ‘Turbine Under-speed’ resulting in total site black-out.
T-0 = 11:25 AM
T= 0 s
1. Power supply failure in Bailey PCU-3
2. The generation of PFI (explained later) in Bailey causing the closure of PRC-2A and PRC-2B; fuel valves of
GT601 and Gt602 respectively.
T=15 s
3. Low fuel gas supply pressure alarms on GT601 and GT602.
4. Ramping down of GT601 and GT602.
5. GT604 tried to pick up the load lost by GT601/602
6. GT604 load reaches to a maximum of 25.2 MWs
GT601 and Gt602 respectively.
T=15 s
3. Low fuel gas supply pressure alarms on GT601 and GT602.
4. Ramping down of GT601 and GT602.
5. GT604 tried to pick up the load lost by GT601/602
6. GT604 load reaches to a maximum of 25.2 MWs
T=23 s
7. Breaker of GT604 trips
8. The entire load now shifted on GT601/02
9. Since GT601 and GT602 are already ramping down, their breaker immediately trips
10. GT601/02 lose their minimum operating speed
T= 34 s
11. Turbines of GT601 and GT602 also trip
12. Turbine of GT604 trips on high exhaust spread.
Ne
w
Se
qu
en
ce
of
Ev
en
ts:
In The investigation was carried out based on Event logs, DCS trends, DCS board-man feedback and visual inspection of
ve
the hardware. The process is explained below:
sti
ga
tio A. UTY-1 DCS:
n/
Int Uty-1 DCS is a hybrid of Bailey INFI-90 DCS as the back-end and Honeywell Experion as the front end.
er Observations:
vie Following observations were recorded from hardware visual inspection, DCS trends, event logs and DCS board-man
ws feedback:
:
(If
ap DCS Trends:
pli 1. On 12th March, ’14 around 11:25 am, the fuel valves (PRC-2A, PRC-2B) closed causing low fuel flow to GTs and
abl eventually resulted in their tripping. These valves are configured on Bailey PCU-3.
e)
DCS board-man feedback:
1. The DCS operator confirmed that at the time of tripping, various DCS PID controller faceplates shifted from AUTO to
MAN mode with outputs held at -6.2%. This observation was recorded on both Bailey OIS and Experion HMIs.
2. This problem occurred for majority loops on PCU-3. It is noteworthy that none of these observations were recorded
on PCU-1 (Urea-1), PCU-7 (HRSG-611) and PCUs at Urea-2.
B. GT-601 TRIPPING:
All the time stamps mentioned in this section are with reference to the time of GT604 Mark-6 Control room HMI.
Before tripping of GT601, it was loaded with 7.8 MWs of load; its Fuel Stroke Reference (FSR) was at 50.92 % while
Gas Control Valve hydraulic oil pressure feedback was 121.2 psi conforming to the FSR value. As can be seen from the
trip log GT601, the first alarm to appear was of GAS FUEL PRESSURE LOW triggered by the signal L63FGL and
indicates the dropping of fuel gas pressure below the alarm set point.
This alarm is followed by FUEL GAS SUPPLY PRESSURE LOW which is triggered by the signal L63FG2L, which is the
trip signal to the gas turbine.
NOTE: Upon looking this signal name in the Mark-V IO assignment file, it is confirmed that this is a trip signal:
L63FG2L QD1-DTBA-033 Gas Fuel Supply Pressure Low (trip) [63FG-2]
Finally, GENERATOR BREAKER TRIPPED alarm appeared, clearly indicating opening of breaker.
C. GT-602 TRIPPING:
All the time stamps mentioned in this section are with reference to the time of GT604 Mark-6 Control room HMI.
Before tripping of GT602, it was loaded with 11.8 MWs of load; its Fuel Stroke Reference (FSR) was at 65.8 %. As
opposed to GT601, the value of Gas Control Valve hydraulic oil pressure feedback is not included in the trip history file
for GT602.
Similar to GT-601, it can be seen from the trip log GT-602 that the first alarm to appear was of GAS FUEL PRESSURE
LOW followed by FUEL GAS SUPPLY PRESSURE LOW LOW, thus tripping the GT602 at the same time as GT601.
Finally, GENERATOR BREAKER TRIPPED occurred (exactly at the same time the breaker of GT601 tripped).
D. GT604 Tripping
Before site blackout, GT604 was running at a load of 18.8 MW while its FSR value was at 34.6 %.
The Control System Toolbox of GT604 logs every event and marks it as a triggering point in its trip historian. However,
by observing its normal logging operation it is also observed that it skips logging of repetitive data after a certain
amount of time.
The same observation is made while reading the sequence of events of GT604, that there is no relevant data before
11:26:10. This shows that there occurred no significant or new event during that time period unless recorded.
The first three alarms in event log of GT604, namely FUEL GAS NOZZLE PRESS.SENS.HIGH DEV./TROUBLE ALM
triggered at 11:26:10, Gearbox proximity probes trip multiply funtion cmd triggered at 11:26:11 and LUBE OIL FILTER
DIFF.PRESS.HIGH ALARM triggered at 11:26:13 are recurrent in nature.
D. GT604 Tripping
Before site blackout, GT604 was running at a load of 18.8 MW while its FSR value was at 34.6 %.
The Control System Toolbox of GT604 logs every event and marks it as a triggering point in its trip historian. However,
by observing its normal logging operation it is also observed that it skips logging of repetitive data after a certain amount
of time.
The same observation is made while reading the sequence of events of GT604, that there is no relevant data before
11:26:10. This shows that there occurred no significant or new event during that time period unless recorded.
The first three alarms in event log of GT604, namely FUEL GAS NOZZLE PRESS.SENS.HIGH DEV./TROUBLE ALM
triggered at 11:26:10, Gearbox proximity probes trip multiply funtion cmd triggered at 11:26:11 and LUBE OIL FILTER
DIFF.PRESS.HIGH ALARM triggered at 11:26:13 are recurrent in nature.
The next alarm in the sequence; GENERATOR BREAKER TRIPPED ALARM triggered at 11:26:16 shows the tripping of
the breaker. This alarm is signified by the signal L52GX changing its state from 1 to 0.
The turbine finally trips at 11:29:56; Master Protective Signal (L4_EVT=0).
By looking at the trends of GT604 it is seen that the turbine was running normally at a steady load of 18.8 MW until
11:26:07, when it starts gaining load. By 11:26:15 it reaches to a max of 25.23 MW when the circuit breaker trips. In
approximately 8 seconds GT604 picks up a load of 6.5 MW. At the time of tripping, its FSR had a value of 39.9 % while
the turbine speed (TNH) had ramped down to 96.7 %.
K
ey
i. Uty-1 DCS:
fi The observations mentioned earlier ascertain that the problem originated in PCU-3. Since upstream fuel
n valves of GT-601, 602 and 603 and associated HRSGs, SGs, steam system and other utilities are present on
di this PCU, it has become the life line for both plant-1 and 2.
n
g
s: What happened in PCU-3?
(L Based on the discussion with ABB Pakistan (OEM), Free-lance expert opinion by an ex-ABB Project
is Manager, system configurations and system manuals, it was ascertained that a Power Fail Interrupt (PFI)
t
in
was generated in PCU-3 which triggered the entire event. The entire event in PCU-3 is elaborated as follows:
lo
gi a) PFI alarm was generated
ca
l
b) A low voltage surge (below threshold) in system power supplies (+5/ +15/ -15 VDC) or I/O power supply
or
d (25 VDC) from MPS-II to system bus in PCU1, caused the PFI alarm.
er
s, c) PFI alarm caused to stop all the master modules (NPMs & MFPs) in PCU-3.
p
er
ti d) Low voltage serge in MPS-II was momentary and PFI alarm was not latched in IPMON01 module, so all
n master modules went in RESET (STOP >> RUN) mode.
e
nt e) When master modules (NPMs & MFPs) went in STOP mode, all associated Digital & Analog outputs went
fa
ct
in "Last State/ Value HOLD" state (including analog output to fuel PCVs for GTs), as configured in related
s function codes.
u
n f) But, when master modules (NPMs & MFPs) went in RUN mode (due to non-latching of PFI alarm in
c
MPS-II), it caused the MFPs to go in normal initialize then Execute mode, which resulted in initial zero value
o
ve to analog outputs (including analog output to PCVs for GTs). MFPs returned in Execute mode after 50
re seconds. After this, entire PCU-3 system restored and became fully operational.
d
in
th What is PFI?
e As per Bailey INFI-90 system manuals, PFI stands for Power Fail Interrupt. PFI is a signal that causes active
in
controller or processor modules (MFCs or MFPs) to reset and the communication system to be bypassed,
ve
st when generated in the INFI 90 OPEN system by an out of tolerance bus voltage. These bus tolerance
ig voltages are mentioned below:
at
io
n)
The existing Modular Power Supply (MPSII) in Bailey PCU-1 & 3 (installed at URUT-1) and PCU-4 & 5
(installed at Urea-2) has a feature that if any of the system bus voltages (5V, +15V) or I/O system voltage
(25.5V) falls below pre-defined threshold voltage, it generates a Power Fail Interrupt (PFI) signal which puts
the MFP (Bailey controller) into HALT mode and cause Analog Outputs to go into default state (Configured as
Last Value Hold in our system).
If PFI is configured in Non-latching state, then whenever the PFI is generated, it causes MFP to Reset
causing Analog Output to go to ‘Zero’ value. The system then normalizes in around 1 minute. This is what
happened in PCU-3 in site black-out.
If PFI is configured in Latching state, then whenever the PFI is generated, it latches and causes MFP to
HALT and Analog Output values to go into Last Value Hold. When PFI bit is manually reset from IPMON01
monitor on MPSII, it will cause MFP to Reset causing Analog Output to go to ‘Zero’ value. The system then
normalizes in around 1 minute.
1. PFI generation on 25.5 VDC bus fault (Field I/O system voltages)
2. PFI generation on 5 VDC and 15 VDC (Module System voltages):
Keeping in view the PFI configurations in Engro’s INFI-90 system, the entire can be narrated as follows:
In PCU-1, 3, 4 and 5 Modular Power System – II (MPS-II), IPSYS01 modules are used for system voltage (5
VDC, +15 VDC) and I/O system voltage (25.5 VDC) generation. 04 IPSYS01 modules are used in each
MPS-II power system
Failure in power supplies can be due to:
· Ageing:
As per Bailey INFI-90 system manual, recommended replacement cycle of IPSYS01 power modules is 05
years.
From maintenance records, it was found that 02 IPSYS01 modules in PCU-3 MPS-II were replaced in 2011.
The other 02 IPSYS01 modules in PCU-3 were of old vintage. No other maintenance record could be found.
Hence, it appears that these 02 modules were in-use since system commissioning.
· Non-friendly Environment:
The standard environmental conditions for MPS-II power system are as follows:
Environm
Temperatu
Operating e
Storage an
Relative hu
Air quality
The actual condition of the Power supply modules and PCU-3 is depicted in following pictures:
Overall, environment of PCU-3 cabinet was found deplorable. Although, Uty-1 DCS Room is a temperature
controlled environment, yet dust ingression is inevitable due to cable trenches and improper sealing of doors.
As evident from pictures above, a lot of dirt was observed on the IPSYS01 modules and IPMON01 of PCU-3
Power Supply. Main reason for this dirt is non-sealing of cabinet bottom.
It is highly probable that dirt caused reduced life of the capacitors and other electronics on IPSYS01
module which resulted in a voltage dip on power bus and subsequent generation of PFI.
A 3A fuse was found blown on one of the Digital Input Termination Units. This can be due to following
reasons:
Since, plant was soon started-up after the incident; the Digital Input card could not be inspected. Visual health
of field cables was verified and found to be ok.
From verbal references, it was reported that cabinet de-dusting activity of PCU-3 was last conducted in 2011.
Maintenance history shows that last Online PM was done on 18-Nov-11. Last PM3 for Uty-1 DCS in SD was
done in 4-Aug-11. The last shutdown work-order was initiated in Oct ’13.
Since, it was a momentary dip in voltage, it couldn’t be detected in a PM. However, there is a need to
improve the on-line PM procedure for noting voltage levels on IPMON01 monitor and proper cleaning of
cabinet filters.
1. Under-frequency protection at the Generator, which trips the Generator Breaker only
2. Under-speed protection at the Turbine, which trips both, the Generator and the Turbine as well
For GT601, GT602 and GT603, there is no frequency protection and the system is completely dependent on
the turbine under/over-speed protection. Generator & Turbine OEMs, Brush Electric & GE respectively, have
recommended over and under frequency protections for GT604. At the time of the subject incident, the
under-frequency settings of GT604 were as follows:
Frequency Delay
49.0Hz 5.0s
48.5Hz 0.5s
The over speed scenario is far less critical for the Generator, however, the same is not true for the Turbine.
To avoid this scenario, there are two protections available in Gas Turbine Generators.
1. Over-frequency protection at the Generator, which trips the Generator Breaker only
2. Over-speed protection at the Turbine, which trips both, the Generator and the Turbine as well
At the time of the subject incident, the Over-frequency settings of GT604 were as follows:
Frequency Delay
51.0Hz 5.0s
51.5Hz 0.5s
After discussion with our Network Consultant, Dr. Voelzke (Siemens Germany), we have evaluated that the
present over/under-frequency settings are quite stringent and need to be revised. The new settings are
determined as following:
Although these new relaxed settings will give GT604 more time to react to anomalies in the Electrical
Network, this would not have prevented a Black Out following this incident due to the extremely
detrimental effects of closing GT601 and GT602 fuel valves. Further relaxation in under-over frequency
protection settings is not possible since these settings are now close to the turbine over/under speed
protection settings.
GT604 Tripping
As is evident from G60 (Protection Relay for GT604) logs, GT604 Generator Main Breaker tripped on
under-frequency. The system frequency is recorded to have dropped as low as 48.4Hz, which is below the
trip settings. The reason for this drop is explained through the ILMS load trends. As can be seen in the MW
graphs, the load on GT601 and GT602 is gradually decreased due to closing of fuel valves of both turbines.
As a result, this load appears on GT604. The maximum load recorded on GT604 is approx. 25MW which is
beyond its load limit. The frequency of GT604 subsequently dropped and the Generator Breaker Tripped on
under-frequency. Since the fuel valves of GT601 and GT602 were already closing, both these turbines fail to
sustain the electrical load leading to Blackout.
As soon as GT604 is isolated from the Electrical Network, the turbine speed increases and the normal
Generator frequency of 50Hz is restored. However, after 3m40s the turbine trips as well. The cause of turbine
tripping is not visible in G60 or ILMS data.
Fig2. ILMS Trends - GT601, GT602 and GT604 MW Loads and GT604 Frequency
Note: Sampling rate for ILMS trends is 5 seconds. Although the graphs don’t explicitly show the chronological order of
events, they do depict the trends quite accurately.
ILMS Operation
ILMS utilizes an Event based Load Shedding Scheme since it is not possible to implement a frequency based
scheme in present Electrical Network due to its complexity causing power transfer limitation through
Reactor-1. Due to this restriction, each GT tripping scenario has to be individually studied and tweaked to
ensure a permissible power transfer through Reactor-1. As a result load shedding can’t be independent of
Events as long as this restriction is there.
During closing of GT601 and GT602 fuel valves, the Generator breakers for both GTs did not trip. Hence, as
per scheme, no action was taken by ILMS. The first event logged by ILMS is the tripping of GT604.
Appropriate loads were shed by ILMS accordingly, as is evident from the Event Logs. Soon after this load
shedding, the two other turbines, GT601 and GT602, also shutdown due to closing of fuel valves, resulting in
a Blackout. ILMS accordingly tripped/opened all MV breakers as per logic.
ILMS Modifications
Although a frequency based load shedding scheme would eliminate the need to rely on the occurrence of an
Event for load shedding, it is not possible to implement such a scheme in present Electrical Network due to
its complexity.
A project to replace existing Reactor-1 with new increased capacity reactor is in progress. Major scope of the
project has already been completed. The remaining scope requires EE620 Bus-3 outage. This is only
possible during complete Urea-3 outage and partial Urea-2 outage. Window for this modification has been
allocated during planed outage of Plant-2 from 4th-20th May.
Once the new reactor is taken in service, the next phase would be to implement a new ILMS scheme which
will couple frequency based load shedding with Event based load shedding. As per this new scheme, if
frequency of the Electrical Network degrades for any reason (e.g. GT tripping, fuel valve closing, load
rejection by any GT), ILMS will shed appropriate load to restore the frequency of the Network even if no Gas
Turbine trips. Although, if an event occurs, ILMS will not wait for the frequency to degrade and shed load
proactively Designing such a scheme involves quite a few complexities since it is tricky to determine which
scheme is to act in different scenarios. It is however, expected that current issues caused by Turbines will be
better handled with the new scheme.
Upon looking in Mark-V logics of GT-601, 602 and 603, it was found that signals are not coming in the trip
logic. It is much possible that if the GT-601 and 602 had tripped on Low Low Fuel Gas Pressure, GT_604
tripping could have been saved.
2. The black-out occurred due to ‘Event based’ reaction of ILMS instead of a hybrid of “Event” and ‘Frequency’
based design. ILMS only responded when GT-604 breaker tripped on under-frequency. Even in this case,
black-out was inevitable due to tripping of GT-601 and 602 on under-speed owing to closure of PRC-2A and
PRC-2B.
Evaluate & change PFI settings from Non-Latching to Latching in URUT-1 and Urea-2 TA - Turnaround Jehangir Alam Khan,Saad 12/31/2014
DCS Afzaal Shamsi
Remove PFI generation from 25.5 VDC. PFI to be kept for 5 V, +15 V due to requirement TA - Turnaround Jehangir Alam Khan,Saad 12/31/2014
in controller and communication cards Afzaal Shamsi
Ensure foaming is provided on bottom plinth for proper sealing of URUT-1 DCS Room SJ - Simple Job Jehangir Alam Khan,Saad 10/31/2014
cabinets Afzaal Shamsi
As a short term solution, addition of frequency based tripping of individual breakers TA - Turnaround Muhammad Imran 09/30/2015
on ILMS to be evaluated and implemented Khaliq,Muhammad Zaghum
Riaz
Ask GE about the slow response of FSR of GT604 upon addition of load EJ - Engineering Muhammad Hamza Khan 01/31/2015
Job
Process engineering to Evaluate providing Low Low Fuel Gas Pressure as a TRIP for EJ - Engineering Masab Javed 12/31/2014
GT-601/2/3/4 wherever applicable Job
On UREA-2, upgrade MPS-2 Power Supply to MPS-3 Power-supply. MPS-2 has phased TA - Turnaround Hira Shaukat 03/31/2016
out 6 to 8 years ago and has a history of failure in industry
On URUT-1 evaluate shifting from MPS-2 to MPS-3 power supply till the time Yokogawa TA - Turnaround Jehangir Alam Khan 06/30/2015
DCS comes on-line. Else, install re-furbished MPS-2 power-supply modules in URUT-1
DCS
Accountability
Measures
Taken:
Date of 12-03-2014 Rep 15-09-2014
occurrence of ort
incident due
on
# Extended Create Date Update By Old Date Extended
Date
Reasons for
sending the
report late (If
applicable)
3/13/2014 7:55:48 AM: Detailed Incident Report Routed to CN=Muhammad Imran Khaliq/OU=ECPLDHK/O=ENGRO and by
Mohammad Ismail
10/11/2014 7:32:21 PM: Detailed Incident Report Routed to CN=Mohammad Ismail/OU=ECPLDHK/O=ENGRO and CN=Arif
Jalil/OU=ECPLDHK/O=ENGRO by Saad Afzaal Shamsi
10/17/2014 3:17:23 PM: Incident Report Closed by Azmat Hayyat Bhatti
10/17/2014 3:18:09 PM: Incident Report Closed by Azmat Hayyat Bhatti