Will Oroville Spillway Gates Fail in Heavy Flows? Design Flaws & Fixes Risk Gate Binding?

Has DWR missed for over 40 years the accumulation of mud, debris, and silt that ended up jamming a Radial Gate?  For many years Inspectors believed that extensive leaking from Gates were normal.  What changed this belief in the failure of a binding Gate during a test?  Are the Gates closer to Failure with these new findings?

LOS ANGELES, CA (California Network) -  In 2007, during an official Federal Energy Regulatory Commission (FERC) Inspection, Radial Gate 4 jammed after only lifting 6 feet of its 33 foot travel [1].  From the perspective that all components of the Spillway Radial Gates are considered a High-Reliability System, what ensued in the subsequent Engineering Failure Analysis Report findings could only be read as an engineering nightmare.  Quoting the Report: "During Federal Energy Regulatory Commission required operational testing Spillway Gate No. 4 would only open to approximately 7 feet at which time the motor would trip offline from overload." "Initial inspection found heavy galling marks on the right side wall plates as well as protruding bolts on the seal assembly directly adjacent to the wall plates.  A bronze plate was also found between the wall and seal.  This plate was later determined to be from a retrofit done in 1974."  "The seal assemblies were removed and disassembled.  A large amount of mud and debris was found behind the seal.  The seal inflation piping was completely filled with mud and debris also.  One bronze guide shoe was damaged beyond repair."   "Two main items were attributed to the gate binding:  1. Lack of maintenance caused the system to degrade and become clogged with mud and debris.  2. Due to irregularities in the seal assemblies - it appears they were not properly adjusted for the proper clearance over the entire length of the seal."

This event uncovered an astonishing amount of damage, broken parts, and design issues so serious that it led to the complete refurbishment to all eight of the Spillway Radial Gates.  These issues are considered FERC Category 1 Failure modes that are of the highest rated level.  Failures of Radial Gates have been known to lead to potential catastrophic outcomes including dam failures [10][11].  DWR's own report to FERC noted that a major cause was from "Lack of maintenance" - which caused the system to degrade and become clogged with mud and debris.  How could a High-Reliability System be neglected to the degree to where the accumulation of mud and debris became a contributing failure mode?  How many years of this type of maintenance neglect of this mud and debris have been going on?  Forty years?

DWR Engineers also discovered that silt had entered into the piping that pressurizes the rubber seals, rendering the inflation/deflation system inoperable.  The hydraulically operated piping & rubber seal systems did not have any silt filtering nor any method of flushing the tubing for maintenance cleaning.  This design flaw wasn't discovered for forty years until gate 4 failed.  Apparently, there was no maintenance inspection program nor were there a DSOD inspection program of the hydraulic tubing systems that could have alerted engineers to this internal pipe & seal clogging issue.

The most severe findings were loose bolts and heavy scraping wear on the stainless steel side plates and the bronze guide shoe bars (see Fig 5).  The stainless steel plates are imbedded into the pier concrete column walls to form a smooth surface for the inflatable rubber seals and the bronze guide shoes to slide upon.  The scraping and galling marks were so significant that the stainless steel plate surfaces had to be repaired.   How could DSOD or Maintenance miss these dramatic scraping marks for so many years?  It finally took a broken bronze shoe plate becoming jammed into the sidewall seam for the combined failure of Gate 4 to awaken DWR to these serious and numerous issues.  DSOD did a "special spillway" inspection report soon after the Gate 4 failure and found that most of the gates had visible sign of "galling" (rubbing wear) on the side seal stainless steel plates [2].  DSOD and Maintenance further discovered that the side seal assemblies were highly irregular, bent, unaligned, and were conducive to be "imprecise to where seal leakage can be expected" [2].

Evidence suggests that DSOD inspectors had become accustomed to excessive gate seal leakage for many years [4][5][6][7][8][9].  Yet this "extensive leakage" was considered normal.  In a High-Reliability System, this should have triggered alarms as the seals were designed to be water tight with minimal seepage.  But, this serious multi-gate failure mode was considered "normal" by the consistent "satisfactory" approvals in the decades of Inspection reports.  This is very difficult to fathom in a high quality engineering background.

Another alarming finding was the discovery of "bending" of threaded bolts to overcome an offset alignment problem in the manufacture of the original Radial Gates (see Fig 4).  Bending of threaded bolts are known to induce micro-fractures at the thread inflections that could develop into crack extensions to the point of failure [18].  This should never occur in a High-Reliability System.  In seeking to fix this issue, DWR Engineering replaced the "bent bolts" with J-bolts but found that the offset alignment would not make a secure mating surface for the "J" part of the bolt.  Instead, DWR decided to use "eyebolts".  The image infers that DWR inserted a slight "spacer" to account for the "offset" differential.  This image is from the 2008 failure analysis report DWR provided to FERC (Fig 15 in the report).

The original design of the Radial Gate Side Seal Assembly and the intended mounting scheme is revealed in Fig 1.  Welded flange plates on the heavy approx 20 to 30 ton main Radial Gate serve as J-bolt mounting points where the end threads and nuts are used to "adjust" the spacing of the side seal assembly.  This adjusting process would allow the setting of the ultimate gap spacing between the rubber seals, the bronze guide shoes (3 per radial gate side), and the stainless steel seal plate in the Pier column concrete wall.   This pressure held design of the J-bolt may have been somewhat acceptable in the day, however this design is not robust as it has two degrees of misalignment freedom - rotation and sliding (see Fig 3 left).

Today's modern Radial Gate designs have revealed a new and powerful force dynamic that was not well known years ago.  Up to 60 ton Radial Gates have experienced a powerful "vibration" oscillation condition from critical flow levels in the gate openings.  A standing wave of high and low pressure oscillation develops on the lower skin plates of the radial gate that transfers a complex set of distortions and movements to the massive steel gate structure.  Indeed, experts are now attributing this powerful resonant oscillation vibration condition as a key component to the catastrophic failure of the Folsom Radial Tainter Gate in 1995 [11].   The power of this multi-mode form of oscillation in the massive trunnion gate structure would be dominant in forces applied to the Oroville Spillway Radial Gate design to these weakly coupled side seal assemblies.  The wave action of the resonance would easily focus high forces on the side seal assembly individual aligning J-Bolts (now upgraded to eyebolts).  This could cause either a shift in the alignment from "slip" or from "rotation" in the bolt compression area of the eyebolts.  The result could be a swift and dangerous form of a strong binding condition to the gate in a virtual "freeze" state.

The Oroville spillway Radial Gates Side seal assemblies & mounting bolt architecture were not designed for these powerful "dynamic" forces from a 20-30 ton structure**.  The side seal assemblies could be distorted and thus become overly pressurized in locations that could lead to an excessive frictional coefficient (i.e Binding failure).  Other seal areas could excessively gap such that debris could flow into and lodge causing another degree of an increase in friction coefficient [13].  The modern upgrade solution to this failure risk is a ruggedized & adjustable mating assembly that could endure the higher dynamic forces over and above the vulnerable existing "slip", "rotate" and bend design.

A new design flaw has been discovered in a DSOD 2015 Inspection photograph. At least one of the radial gates reveals mis-alignment offsets of the welded flanges that are so large that the DWR to FERC identified solution of an eyebolt will not align.  Thus, the photograph reveals that sections of metal, either a bar or formed metal, had to be bent upwards many inches to connect between the flange and the side seal assembly (see Fig 2, 3).  This greatly weakens the alignment holding strength while adding a new failure shift mode (alignment rotation bending moment).  Searching FERC/DWR communications & documents, there were no discussions where DWR notified FERC of this design modification.  As far as FERC has been informed, the 2008 report (see Fig 4, Fig 3) has only the "straight" eyebolt design.  This infers that FERC has not had the ability to provide feedback on the integrity of this new imposed design fix by DWR.   DSOD Inspectors have commented on the flange to side seal assembly misalignment issue with the gates [2].  It is unknown as to how many of these significantly misaligned connections are in all or some of the eight Radial Gates.  Mechanical deviations of this magnitude are significant regarding proper analysis in High-Reliability Systems.

The discovery of this significant flaw is after DWR has notified FERC that the full rehabilitation of all of the eight Radial Gates have been completed [3].

Another great concern to the Failure Mode Potential of one or more Radial Gates at Oroville is from the years of developing concerns over the "unknown" anchorage of the Trunnion Blocks.  These Trunnion Anchor blocks have a combined force of nearly 7+ million pounds of tensile force, by design, provided by the post-tensioned group of anchor tendons for each Radial Gate.  Another threat is the near 4 inch offset shear of the 5 foot thick Pier 10 for Radial Gate 8 that has a diagonal crack that is now near 16+ feet.  All that it takes is a very small shift in the Trunnion Axis anchorage to where the Radial Gate could "lock up" from excessive frictional coefficient forces.  Since the known number of significant cracks in the anchor tendons are distributed across all eight of the gates, the level of unknowns to the imbalance of anchorage strength could induce a small rotational shift in the Trunnion pin anchorage.   As the side seal plates of the Radial Gates are at or near 40 feet from the Trunnion pin pivot point, and DWR has identified that the "proper clearance" is 0.040 inch gaps to the gate sides to side seal plates, a shift of 0.017 inches from 40 feet away would shift the gate flush with the side plate.  How much frictional coefficient rise increase occurs from every incremental 0.001 shift to where the gate "freezes" would be determined by an actual failure.

However, the great uncertainty of not knowing the effective tension on 382 (2 have failed) anchor tendons does not give confidence to the now known susceptibilities to the design of the Radial Gate Side Seal Assemblies.

What could happen in a cascading Failure?  A frozen Radial Gate, if open, would be difficult to stop the flow through the stuck open gate.  A stack of five 8+ foot high plates with support beams, called "stoplogs", normally are used to block water from the entrance to a Radial Gate.  However, during a water flow condition, these large plates would be very difficult to successfully emplace due to the turbulence of the water [12].  These stoplog plates are designed to seal by hydrostatic pressure compressing rubber seals on the back side.  In trying to lower these plates, the rubber seals would engage from the hydrostatic flow pressure.  These plates are not designed to be "forced" in place in a high water flow state to address an emergency flow stoppage condition.  

Thus, a frozen Radial Gate Failure condition could lead to scouring erosion of the cobble rock inlet apron as it drops below the 830 foot cutoff elevation.  Erosion turbulence of the Spillway inlet lip could begin a penetrating erosion at a deeper level to the foundation to the headworks.  If any scouring erosion reaches any sections of the now known "highly weathered rock" or "highly erodible rock", a piping channel may develop to the other side of the headworks.  Then the risk of a "v" breach would develop.  If the turbulent erosive flow continues, the "v" breach could enlarge to the point of compromising the headworks and possibly beneath or adjacent to the upper main spillway.  At this point, it becomes a condition of where it is described as "loss of crest control".   Even if the flow is temporarily stopped, further spillway flows may intensify the prior erosion damage, leading back to the series of events that could end up again at "loss of crest control".

In this scenario, the Emergency Spillway would become a major component as it would have to act as a redundant backup to the damaged main spillway apron where the main spillway would likely have to be shut down.   The Emergency Spillway is not designed to handle the higher flows of the main spillway until the planned 2018 construction season in the buttressing of the RCC layers and partial downhill apron.  Thus, it is possible that the Emergency Spillway would face a flow it was not designed for.   This could lead to a repeat of Feb 12, 2017 where undercutting or back-cutting of the armoring would only give more time to face the potential damages that nature would decide in the outcome.

It may be that DWR is facing the perfect storm of "unknowns", flaws, cracks, and a race against nature as the headworks clearly seem to be going down the same path as what has happened leading up to the main spillway blowout failure [16][17].  Maintenance, Inspections, flawed engineering design issues, deferred known 50 year old aging and failing anchor tendons and the lacking of expertise to the level of Civil Law and Civil Liability in the "Standard of Care" in the profession of a known standard of a High-Reliability System will be the perfect storm.  Even if nature and conditions fare favorably on Oroville, the sum of the findings in engineering, maintenance, inspections, and the seeming lack of expertise to properly operate a High-Reliability System, which cannot fail (for the hundreds of thousands of lives in immediate risk), may become America's Fukushima - fully preventable in hindsight.  Only time will reveal if nature and favorable conditions will bless us.  However, the American public should be made fully aware of the full extent of what has been allowed to develop.  Americans should be made aware of how much risk was allowed to be shouldered (and still is at a strong degree of unacceptable risk) by the immediate downstream residents below this dam.

A special note to FERC:  Recently, four documents were specifically removed from your software search parameters on the FERC advanced search database.  These four documents were key references to recent articles on the anchor tendons, and on "void" filling.  The public documents are still on the database if a user knows how to do a generic search by a date window, but the specifics of removing these four documents from the text search box list is very specific and can only be deemed deliberate.  For those that are interested, the documents that were "masked" from a keyword search on the document numbers are:  20000509-0242, 20000419-0130, 19990322-0490,  20090519-0022.

** Estimation of the combined four steel Radial Gate "lift" cables, at 1 inch diameter steel, and a factor of safety of 5, yields a 33.4 ton lifting capacity [15].

Note: text markups, underlines, arrows, on images and documents for illustrative purposes and are not part of the originals.

References:

[1] DWR ENGINEERING REPORT 5/1/2008 - Gate No. 4 Assessment and Main Seal Replacement    https://elibrary-backup.ferc.gov/IDMWS/search/advResults.asp    20080701-0078(19428792).pdf
"During Federal Energy Regulatory Commission required operational testing Spillway Gate No. 4 would only open to approximately 7 feet at which time the motor would trip offline from overload." "Initial inspection found heavy galling marks on the right side wall plates as well as protruding bolts on the seal assembly directly adjacent to the wall plates.  A bronze plate was also found between the wall and seal.  This plate was later determined to be from a retrofit done in 1974."  "The seal assemblies were removed and disassembled.  A large amount of mud and debris was found behind the seal.  The seal inflation piping was completely filled with mud and debris also.  One bronze guide shoe was damaged beyond repair."   "Two main items were attributed to the gate binding:  1. Lack of maintenance caused the system to degrade and become clogged with mud and debris.  2. Due to irregularities in the seal assemblies - it appears they were not properly adjusted for the proper clearance over the entire length of the seal."
[2] DSOD 2008 "Special Spillway" Inspection report 1/14/2008 - 1/16/2008 Report date - "Reason for inspection: Special spillway inspection" "The side seals have been reinstalled at gate 4.  The j-bolts and seal inflation system will be competed in the near future." "Galling of the embedded seal plates has been noted at other gates".
During a routine operational test conducted in 2007, the owner found that gate 4 would trip the circuit breakers when raised about 6 feet.  The gate was taken out of service and the seal system was disassembled.  Galling of the embedded seal plates and brass rub plates was observed.  A more in-depth description of the findings will be provided in an upcoming memorandum by the Design Engineering Branch."  "Photograph 1 illustrates a problem encountered by the repair crew.  Note the space between the wall and the seal in the upper part of the picture, and the tight fit at the bottom of the picture.  This condition was typical at top and bottom on both sides of the gate." "According to the repair crew, the ends of the seal attachment plates were found to be bent outwards.  This condition was relieved to some extent by installing restraints at the top and the bottom of the gate as shown in Photograph 1.  The j-bolts will be used to press the center portion of the seals into proper alignment.  The fit is likely to be imprecise, and seal leakage can be expected." "The seal attachment plates were apparently fabricated with slight lateral offsets at some of the welded splices.  This apparently adds to the irregularity of fit." "When the head-operated seal inflation system was dissembled (disassembled?), the crew found that silt had plugged the supply pipes and portions of the seals.  The system will be rehabilitated using stainless steel pipe and flexible couplings."  "A method for flushing silt will be devised"
"During a superficial inspection of the other gates the repair crew found seal plate galling as noted below."
"Gate 1:  Galling along right seal plate, but none along the left plate."
"Gate 2: Little to no galling on either side."
"Gate 3: The right side was OK.  Some galling observed along the left plate."
"Gate 5:  Little to no galling on either side."
"Gate 6: Significant galling on the right side, but the left side looked OK."
"Gate 7: These seals were not inspected at the time because the gate was raised to provide access to gate 4 via the chute floor."
"Gate 8: Some galling noted along right plate, but left plate was OK."
[3] DSOD 2015 Feb 3 Inspection - 3/6/15 Report date - "The gate seal inflation systems at gates 6, 7, and 8 were completed recently and were successfully tested through full gate travel.  This completes the seal system modifications at all gates."
"Sheet 13 - 11. A Typical FCO radial gate is shown. No signs of overstress or damaged gate members were observed.  The paint coatings remain in satisfactory condition.  Some of the piping for the gate seal inflation system is visible at the arrows.  Modification of the gate seal systems at all gates is complete." "A row of tendon anchor rods is visible at right."
[4] DSOD 2006 Inspection report - 860.6ft reservoir level - 4/11/2006 Report date - "The gate seals were leaking extensively, which is typical during high head conditions"
"No indications of vibration were noted on the gates or lift cables."  "The flow at that time was about 31,500 cfs.  All eight gates were open equally, and the flow was symmetrical across the chute floor."
"Gate seal leakage was extensive but normal for these gates.  No recommendations are being made as a result of the inspection.".  (note: no mention of galling on stainless steel plates).
[5] DSOD 2005 May 17 Inspection - 6/7/2005 Report date - 872.03ft reservoir level - "The radial gate seals were leaking in amounts historically consistent with the reservoir elevation."  "Photograph 2 shows a tendon anchor connection that broke several years ago." (Gate 4 left upslope)
[6] DSOD 2005 Oct 12 Inspection 10/14/2005 Report date - 851.09ft reservoir level - "The radial gate seal leakage was normal."
[7] DSOD 2003 Jun 19 Inspection 7/9/2003 Report date - 898.96ft reservoir level - "The stoplogs were being installed at gate 3 during the inspections.  Field staff will evaluate the leaky seals."
[8] DSOD 2002 Nov 18 Inspection 12/4/2002 Report date - 699.48ft reservoir level - "The radial gate repair contract is complete.  Numerous missing and poor welds were found and addressed."
[9] DSOD 2002 Jun 13 Inspection 6/27/2002 Report date - 826.33ft reservoir level - "Modification of gates 5 through 8 is on schedule." "Water control was necessary to divert leakage from gate seals and stoplogs.  The right side seal on gate 3 was producing most of the leakage."
[10] Spillway Gate Failure or Misoperation - UNITED STATES DEPARTMENT OF THE INTERIOR - Bureau of Reclamation - Water Operation and Maintenance Bulletin No 202 Dec 2002 -
https://www.usbr.gov/assetmanagement/WaterBulletins/202dec2002.pdf
"The failure, misoperation, or use of spillway gates may cause downstream flooding that can range from minor to catastrophic. Downstream flooding is possible from any of the following:" "..displacement of concrete structural components; lack of maintenance; or other design or operational defects. If gates cannot be opened during a major inflow at a reservoir, dam overtopping and possible dam failure may result."  "Debris blockage of spillway gates impedes outflow, possibly leading to damage to spillway gates and/or overtopping and dam failure."
[11] VII-1. Failure of Radial (Tainter) Gates under Normal Operational Conditions - US Bureau of Reclamation - May 27, 2015 - https://www.usbr.gov/ssle/damsafety/risk/BestPractices/Chapters/VII-1-20150527.pdf
"Some other mechanisms that may lead to failed gate operation include failure of hoist wire ropes or hoist chains and gate binding."  "The ice and debris loads are unusual loads."
[12] Stop Logs for Emergency Spillway Gate Dewatering, Derek R. Freckleton; Michael C. Johnson, P.E., M.ASCE; M. Leslie Boyd, P.E., M.ASCE; and Dustin G. Mortensen, P.E.. - http://ascelibrary.org/doi/10.1061/%28ASCE%29HY.1943-7900.0000348
"Emergencies resulting in uncontrolled flow through spillway gates often lead to millions of dollars lost to repairs, lack of dam productivity, property damages, and risks to public health and safety. A considerable need exists for reliable emergency dewatering systems to prevent these consequences. Stop logs are the most commonly used dewatering system for spillway gates but are typically designed for maintenance purposes only. They have been found in most cases to be unreliable for emergency closure under flowing conditions."
[13] Dams & Civil Structures: A Challenging Gate Repair at Cowlitz Falls - Hydroworld - 07/01/2010 -
http://www.hydroworld.com/articles/hr/print/volume-29/issue-5/articles/dams--civil-structures.html
"A lifting force of nearly 140,000 pounds was required to break the bulkhead loose from the guides. The second unmodified bulkhead was successfully installed to continue the inspection and allow the removal of the failed sluice gate."
"The team was kept together to analyze the cause of the failure of sluice gate No. 1. Increased wheel friction was again found to be a contributing factor, as sediment had migrated into the bearing surfaces."
[14] Precision operation and machining - Trunnion Pin Tolerances - Trunnion Bearings and Thrust Washers - https://tss-static.com/remotemedia/media/globalformastercontent/industries/orkot_hydro_2/pdfs_1/Dam_Trunnion_articlepdf_5.pdf
[15] Appendix D Wire Rope - Bureau of Reclamation - Safety factor of 5 - Steel Cable -  https://www.usbr.gov/ssle/safety/RSHS/appD.pdf
16700 lbs (safe 20% of breaking strength)  Safety factor of 5  - 33.4 Tons for Radial Gate plus load friction
[16] Large 14 Foot Crack in Headworks? Cracking Radial Gate Steel tendons? Threat to Spillway yet no Repairs in 2017? - Catholic Online - https://www.catholic.org/news/green/story.php?id=74915
[17] Headworks Cracking Risking 3 Highest Level FERC Category 1 Potential Failure Modes?  New Design Flaws Discovered?  - Catholic Online - https://www.catholic.org/news/green/story.php?id=74965
[18]  Specification of threaded bar in structural applications - Journal of the Australian Steel Institute - Volume 47 Number 1 - July 2014 -  http://aefac.org.au/documents/AEFAC-Pub01-threaded-bar.pdf
"Threaded bar in its own right is not an ideal structural member. The threads act as notches and when subject to bending, material failure could occur at a lower load than that for a smooth bar due to stress concentrations associated with thread roots. The same reason makes threaded bars not suitable for dynamic load applications where they could be subject to premature fatigue failure. Furthermore, when the strength of the bar increases, its susceptibility to brittle fracture and fatigue failure also increases." "If the bending is done on an annealed threaded bar which is later heat-treated to achieve the necessary tensile properties this problem may be avoided."