We’ve all been watching the stories fly in over the last 24-36 hours about the power outage at the Superdome during the Super Bowl, so that won’t get rehashed here.
We’ve also hearing a lot of theories from disparate media outlets about why the outage happened – did Beyonce’s halftime show suck up too much juice? Was the Superdome unprepared for such a huge power draw? Was there a conspiracy on behalf of the 49ers, who were getting blown out of the water at the time? At least the first two have been debunked:
The exact cause of Sunday night’s blackout — and who’s to blame — remained unclear late Monday, though a couple of potential culprits had been ruled out.
It wasn’t Beyonce’s electrifying halftime performance, according to Doug Thornton, manager of the state-owned Superdome, since the singer had her own generator. And it apparently wasn’t a case of too much demand for power. Meters showed the 76,000-seat stadium was drawing no more electricity than it does during a typical New Orleans Saints game, Thornton said.
What we’re also now hearing is that there was at least documentation of concerns that power outages could occur due to a decaying power grid:
The cause of a 34-minute blackout at the Super Bowl remains under investigation, but public records released Monday show that Superdome officials were worried about a power outage several months before the big game.
An Oct. 15 memo released by the Louisiana Stadium & Exposition District, which oversees the Superdome, says tests on the dome’s electrical feeders showed they had “some decay and a chance of failure.”
Entergy New Orleans, the company that supplies the stadium with power, and the structure’s engineering staff “had concerns regarding the reliability of the Dome service from Entergy’s connection point to the Dome,” the memo says. Those concerns were due in part to “circumstances that have previously occurred with the electrical service regarding transient spikes and loads.”
And at least management tried to do something about the problem:
Authorities then authorized spending nearly $1 million on Superdome improvements, including more than $600,000 for upgrading the dome’s electrical feeder cable system, work that was done in December.
“As discussed in previous board meetings, this enhancement is necessary to maintain both the Superdome and the New Orleans Arena as top tier facilities, and to ensure that we do not experience any electrical issues during the Super Bowl,” said an LSED (Louisiana State and Exposition District) document dated Dec. 19.
Some preliminary investigations (same ABC News article) bring up some theories to the cause of the outage:
A lawyer for the LSED, Larry Roedel, said Monday a preliminary investigation found the replacement work done in December did not appear to have caused Sunday’s outage.
Entergy and the company that manages the Superdome, SMG, said Sunday that an “abnormality” occurred where stadium equipment intersects with an Entergy electric feed, causing a breaker to create the outage. It remained unclear Monday exactly what the abnormality was or why it occurred.
Apparently we have a series of uncertainty around the actual cause of the outage. I sense a little finger pointing without having actually applied full root cause analysis yet.
But really, it certainly seems like the outage could have been prevented because documentation exists notating the concerns about spikes in power consumption and the decay of pieces of the power grid, but it isn’t clear that the work done to prevent the potential power failures prevented the actual power failure.
It’s a method used to identify, analyze, and prevent such failures. And despite the somewhat intimidating name, it’s a relatively simple tool to use. Basically, you try to figure out all the ways something can fail and then come up with ways to prevent or mitigate the failures. The something can be anything… a product, service, system, or process. For each “failure mode”, you identify the potential causes, along with the impact severity on the customer, hence the term “effects analysis”. This helps ensure that attention is spent on the areas most likely to cause the serious problems.
And here’s his quick analysis for using FMEA with the Super Bowl:
A formal FMEA may have revealed potential problems in the Superdome’s power supply, lighting units, or maintenance procedures, which all could have been remedied. In the big scheme of things, nobody was hurt and no lives were lost. But it did delay a live TV broadcast for over 30 minutes, which seemed to change the momentum and course of the game. It’s all about reducing risk.
He’s right – very rarely can you guarantee that nothing will go wrong. However, something very big and very critical went wrong at the Superdome and it appears that proper application of FMEA could have minimized the risk.
What are the chances the Superdome used FMEA before signing off on the very expensive repairs that may or may not have caused/prevented the power outage?