Summary Report of Workshop on Reliability of Discharge Equipment CEA Dam Safety Interest Group

Montreal, April 30 & May 1, 2001.

1.0

Introduction and Overview

This 2-day workshop was organized by CEA Technologies' Dam Safety Interest Group (DSIG) in response to an OPG proposal made at the 2000 Fall DSIG meeting. Mona Bechai and Klaus Faisst of Ontario Power Generation and Tai Mai Phat of Hydro Quebec were the workshop organizers. The objectives of the workshop were to: Document current Dam Safety Design Review Standards and Criteria for discharge equipment. Document current methodologies and procedures for testing of discharge equipment and remote control systems in actual or simulated adverse conditions. Identify reliability requirements for remotely operated equipment. Identify problem areas in operation and reliability of discharge facilities.

The workshop was attended by 42 delegates from 14 companies/organizations, including two other CEA Technology Interest Groups, the Hydraulic Integrated Resource Management Interest Group (HIRMIG) and the Hydraulic Plant Life Interest Group (HPLIG). The workshop featured 19 presentations on reliability issues with flow control equipment, including the present state of practice, lessons learned and experience by the member countries. The last half-day was structured as a panel discussion with the focus on common needs for future research and development opportunities. The panelists were:

Gary Salmon, Technical Coordinator DSIG (Chair) Don Coulson, Vice President, RSW Bill McStraw, Senior Mechanical Engineer, USBR

The entire content of the workshop has been recorded on Compact Disk (CD-ROM in pdf Format) and copies of the CD have been distributed to all participants. The content has been set up in the following sequence:

Summary Report Agenda 19 Presentations (in sequential order) List of Participants Presentation Outlines and Biographies Invitation to Workshop

2.0

Highlights

The highlights recorded in this summary report have been sorted into five topics: Risk Analysis as a Tool for Reliability Evaluation Inspections, Tests, Condition Assessments and Maintenance Design Reviews Human Factors and Operator Training Lessons Learned

2.1

Risk Analysis as a Tool for Reliability Evaluation

The spectrum of opinion expressed by workshop participants on this topic varied widely. Marty McCann from the U.S. National Performance of Dams Program (NPDP) explained that risk analysis is successfully practiced in other industries (e.g. Aircraft, Nuclear, Chemical). Although a component reliability estimate is unlikely at this time, risk analysis can still be a useful tool for reliability evaluation, because it can show which items are more critical than others , some times by an order of magnitude. The U.S National Performance of Dams Program has a wealth of information on dam incidents, even though, only a fraction of all incidents is being reported (perhaps 20%). The available information can be used to support engineering assessments; e.g. failure events of gates and valves (number of events vs. type of reason). In Sweden, risk analysis is not yet considered a well-established tool for dam safety, mainly because of the complex behaviour of the civil parts of the dam. Electrical and mechanical components are easier to deal with. Scottish & Southern Energy (S & SE) have introduced risk analysis in 1996, using a simplified FMECA approach. Equipment components are numbered for systematic analysis and numerically evaluated for their probability of failure. This value is then multiplied by an estimated "severity of consequence factor" "and an assumed "likelihood of detection" factor, to arrive at a numerical criticality value for that particular element or component. A composite criticality score for the entire discharge system is then computed from the criticality values of the individual components. As a result of the assessment process S & SE have been able to produce a planned form of action to rectify highlighted problems and investigate areas where knowledge is lacking as part of their asset management program. The USBR includes loss of human life in their risk analyses. They have some 3,500 gates and valves on a database and use the database for prioritization of refurbishment work. However, meaningful numbers and a methodology has yet to be developed in order to apply risk analysis as a tool for reliability evaluation. FERC doesn't have much comfort in numerical analysis of risk and wants to "divorce" risk analysis from reliability evaluation. In Australia, recent studies of spillway gate failures show that reliability assessment should consider not only equipment condition and performance but also human factors, poor design and policies. One of the hindrances in quantifying gate reliability is the lack of information on equipment performance and failures.

EDF (France) has lots of data for risk analysis in the nuclear field, but no comparable database in hydroelectric. The principal in making risk analysis a useful tool for reliability evaluation is good but more work at the foundation and training is required.

Ontario Power Generation observations are that failure rates exist for all kind of components; e.g. motors, breakers, relays. However, there is no specific data bank on discharge equipment failures because components and discharge systems are very site-specific with respect to their evaluation parameters; e.g. type, size, age, manufacturer, environment, life cycle use, maintenance history. A standard procedure is needed by dam owners to enter component failures into a shared data bank. This will require a free exchange of experience and information by a pool of contributors.

Other Comments and Observations by Workshop participants: Not all gate failures lead to catastrophic consequences. Large, high profile gate failures such as Folsom reduce public confidence in the industry. Risk factors should determine inspection rigor. Many mechanical systems fail without prior warning. Gates can fail in both open and closed position. Hence, the upstream and downstream consequences of each failure scenario must be considered. Most electrical component failures are random failures (not predictable). Therefore, all critical components should be backed-up by redundancy. Companies are down-sizing, hence corporate knowledge is getting lost which impacts on reliability. Contracting out of equipment maintenance is on the increase, hence there is potential for conflict between plant staff and outsourcing, with potential for more human errors. De-regulation will bring a reduction of people and more telemetry. As we add more and more programmable logic controls (PLC's) we are loosing the "feel" for the equipment. Evaluation of the reliability of discharge facilities is a very important part of an overall risk assessment for a dam.

2.2

Inspections, Tests, Condition Assessments and Maintenance

The workshop presentations and discussions confirmed that periodic inspections, tests, condition assessments and proper maintenance are the most conventional methods to demonstrate system and equipment reliability. Scottish & Southern Energy is using the results from inspections, tests and condition assessments on two fronts: (a) risk assessments for dam safety and (b) prioritization of refurbishment work. Inspection results for key components are recorded on standardized data collection sheets, specifically developed for gates. This asset management approach has identified the need for regular inspections of gates and valves both for safe operation and to determine long term maintenance and refurbishment plans. A program of inspections has been established based on the following format: weekly visit and visual check by operative; 3 monthly test operation by operative; yearly test operation by engineer; and 5 yearly engineering inspection. Hydro Quebec has three inspection levels: Routine inspections: 1 per year, mostly for electrical and mechanical components of heated gates. Regular or limited inspections: 1 per year to 1 per 5 years, to check gates, stop logs, gantry cranes, cable hoists and other components. This includes a full or partial (at least 10%) opening testing. Full Inspection: 1 per 10 years, synchronized with a facility reviews. The USBR is using qualified engineers for inspections, condition assessments and ranking of priorities. Test results for 3,500 gates and valves are stored on a database. The USBR's state of practice is: Test all system initiation modes for gate operation (local, remote, back-up power). Test headgates under full unit flow. Measure and record motor currents on gate hoists as an indicator of lifting forces. Verify all calibrations.

Hydro Quebec, the USACE and the USBR are collaborating in the development of a condition indexing technique to gauge the condition of discharge equipment and to prioritize maintenance requirements. In condition indexing, the weak elements (links) are identified. The failure potential of each element is evaluated. The concept is based on a standard framework of tests and measurements, systematic inspections and available site information from operating and maintenance personnel, and is then tailored as required to fit a specific site. The Australian Practice is to open gates fully where practical once a year, partly open some gates every month. Unfortunately, test results are not yet consistently recorded. FERC's Practice: Visually inspect each gate (Tainter Gate Initiative). Test often enough, analyze equipment for design deficiencies. Measure electric current drawn by motors during gate lifts. Five year safety inspections by independent consultants. Yearly site inspections by FERC staff (operations inspections). Five year full-height gate tests. Yearly certification of gate operation. Inspection efforts are based on consequences of failures. Yearly participation in conferences/workshops on discharge equipment. FERC has asked the Licensee's to document their lubrication procedures, in particular for trunnions of radial gates (Tainter Gate Initiative). Ontario Power Generation has two levels of tests and inspections: Functional Tests, carried out once per year, consisting of limited inspections and partial opening of all flow control devices. Design Tests, carried out during periodic reviews (every 5 to 7 years, depending on the structure's Incremental Consequence Category), consisting of a thorough inspection and full opening of the flow control device under design conditions. In the future, the embedded steel components and the normally inaccessible parts of the gates will also be included in these inspections. In addition, design checks will be carried out on system components affected by wear and deterioration. Ontario Power Generation has initiated a maintenance program known as Streamlined Reliability Centered Maintenance (SRCM), which is really a qualitative reliability-centered approach. It's main objective is to reduce the incapability factor of all production and discharge facilities. Maintenance is done according to assumed criticality measures. System components (e.g. gates, hoist motors, power supplies) are assigned criticality ratings based on regulatory requirements and experience (subjective judgement by joined staff). Component failures, as well as corrective and preventive maintenance for all hydroelectric plants will be entered into a data bank. Once completed, the data bank should provide valuable information on type and frequency of failures. Operation and Maintenance Manuals should be updated every year. Maintenance (and testing) of discharge equipment at N.B. Power is done prior to spring run-off. Proper maintenance/inspections inside of some of their larger spillway gates is not possible because of confined space access restrictions. To enhance overall reliability of retrofitted control systems, BC Hydro is subjecting new system components (new on the market) to extensively bench tests prior to their installation in the field. Some are subjected to seven different tests to verify their reliability prior to site installation.

2.3

Design Reviews

In FERC's definition a design review should identify the consequences of failure, the failure mechanisms and the required analyses for a given failure scenario. The USBR is performing selective stress analysis with earthquake loads, mainly on older gate designs. Their experience and recommendations are: Tainter gates are most at risk. More stress analysis with realistic seismic load assumptions needs to be done, particularly on high dams. Vertical lift gates are not nearly as much effected by seismic loads. Re-evaluate margin of reserve on factors of safety. Re-evaluate welded joints from inspection results. Prof. Wilson (California) is doing finite element stress analysis for the USBR. More research is needed in this field. Design reviews on headgates and intake gates should also include an air vent analysis, particularly on older designs, to re-evaluate negative pressure in tunnels and penstocks on emergency gate closure. Hydro Quebec's Recommendation: A design review should identify the "weak links" in the system, which in their experience has been: #1: Loss of power supply - must have redundancy #2: Problems with gate lifting mechanisms #3: Gate jamming (poor tracking) In Sweden, where discharge facilities are generally remotely controlled, the weak links were reported to be in the control system. As part of a design review, Ontario Power Generation is studying potential benefits and consequences of reduced gate heating in winter. If a plant has, say, 10 sluice gates but requires only a fraction of that number to handle load rejects plus winter flow conditions, the remaining gates could be allowed to ice up if the gates can withstand the ice loading. Comment by Don Coulson on "Old Gates - New Standards": Older structures are usually not designed to meet to-days standards (e.g. ice loads, seismic). However, their fitness for service can still be evaluated against the background of past performance, operation & maintenance history and their present state of repair based on tests and inspections. BC Hydro's reliability improvement projects at Seven Mile Dam and John Hart Dam were identified as good example available for a reliability case studies. They include a review of the design criteria and reliability improvements at existing dam sites to "best industrial practice". The improvements included multiple sensors with a "majority vote" to trigger an action; e.g. to open sluice gates before the reservoir reaches the critical maximum operating level.

2.4

Human Factors

BC Hydro and NSW (Australia) have highlighted the importance of the "Human Factor" in risk assessments. NSW have consulted a human psychology expert on the subject who concluded that the probability for human error is 3 times higher than for others failures. The USBR and FERC are taking steps toward formal certification of discharge equipment operators as a measure to enhance overall system reliability. Some failure examples, when analyzed, sound like a conspiracy (accumulation of circumstances + random bad luck + mistakes). For instance, at one dam site the power supply from a transformer to a gate hoist motor had the phases wired up in reverse. Hence, pushing the raise button made the gate go down and vice

versa. The reverse phase condition remained unnoticed for seven years. In another example relevant to risk analysis, power to a gate hoist motor was unavailable when needed due to maintenance of a component of the power supply system. This occurred in spite of the fact that the system had nine alternative power sources for this motor.

2.5

Lessons Learned

Saguenay Flood (Quebec) Following the Saguenay Flood in 1996, and conclusion by the Nicolet Commission, Hydro Quebec is putting more emphasis on: Re-evaluation of hydrologic risks under different floods, up to PMF. Systematic inspection, testing, maintenance and rehabilitation of spillways and outlet works. Wider sluice flow passages (at least 4 m) to pass large floating debris and sunken tree stems (Sweden considers 7 m a minimum width, based on physical model studies). Replacing wooden stop logs with metal gates (to permit gates to be opened under high reservoir levels). Alternative power sources for gate hoists. Access to gate structures at all times. Strong cooperation and involvement by all disciplines (civil, mechanical, electrical). Folsom Dam Spillway Gate #3 Failure (1995) This Tainter gate failed due to strut arm end moments, which were not considered in the original design ( design deficiency). The end moments were produced by trunnion friction, however, lubrication was not clearly deficient. The design deficiency could not be detected by visual inspection. Strut arms were slender and not well braced. Carlos Zumaran, (Hydro Quebec) reported on a drum gate refurbishment project. The gates were 8 m wide and featured a self-actuating operating mechanism with a counter-weighted wire rope & pulley arrangement. This simple system has been working problem-free without operators and communication links for 71 years, confirming the point that reliability can also be found in simplicity.

3.0

Conclusions

Reliability Guidelines There was modest support from workshop participants to the idea of developing reliability assessment tools geared to discharge facilities with emphasis on condition assessment and indexing techniques, as opposed to an abstract numerical analysis. The HIRMIG and the HPLIG will investigate their member's support and report back on their Group's interest in participating towards this development. Hydro Quebec, the USACE and the USBR have an ongoing project on condition indexing. Hydro Quebec will contact S & SE to discuss potential co-operation. Hydro Quebec may be in a better position to report on their efforts at the next DSIG meeting.

Inspection and Testing Pre-inspection work should include review of maintenance and repair history, operating and maintenance manuals, flow charts, EPP manuals and information from field staff. It should also include interviews with operators to check their knowledge. Inspection and testing should be scheduled to obtain maximum information. Testing should be done of all systems (including redundant ones) and should include a calibration exercise of remote as well as local/actual read-outs. Tests must be performed and documented by qualified engineers and technicians.

Design Reviews Inspections and testing alone will not reveal all deficiencies. The original designs must be reviewed and compared to current standards (for seismic loading, ice loading, etc.) Feedback on shortcomings and design deficiencies must be given to designers of new gates to avoid future problems.

Human Factors / Operator Training Dam operator training should cover: Theoretical subject knowledge. Knowledge of the specific site and systems characteristics. Refresher training and simulation of emergencies. Personnel should be certified before they are permitted to operate emergency discharge equipment. Sweden is developing a simulation river model that could be very useful in operator training. There is considerable interest from some workshop attendees in this model.

Abbreviations: ANCOLD ASDSO BC CEA DSIG EDF EPP EPRI FERC FMECA HIRMIG HPLIG NPDP NSW OPG PMF S & SE USACE USBR Australian National Committee on Large Dams Association of State Dam Safety Officials British Columbia (Canada) Canadian Electrical Association Dam Safety Interest Group Electricite de France Emergency Preparedness Procedure Electric Power Research Institute Federal Energy Regulatory Commission Failure Mode Effects and Criticality Analysis Hydraulic Integrated Resource Management Interest Group Hydraulic Plant Life Interest Group National Performance of Dams Program (Stanford University, California) New South Wales (Australia) Ontario Power Generation Probable Maximum Flood Scottish & Southern Energy U.S. Army Corps of Engineers U.S. Bureau of Reclamation