In-Depth

Storm Warning

How Jefferson Parish planned for and responded to the worst disaster in U.S. history, and the lessons IT professionals can learn from it.

• By Michael Desmond
• 06/01/2006

The Jefferson Parish Emergency Operating Center (EOC) squats over the low, grassy plain south of the Mississippi River. It's a brutish, four-story block of a building that sits in the shadow of a massive radio tower. If the EOC building looks like a bomb shelter, it's because it once was.

On Aug. 29, 2005, that structure survived the greatest bomb ever dropped on the United States: Hurricane Katrina.

Nearly 100 souls had crammed into the fortified structure on the morning of Katrina's landfall. Most came from across Jefferson Parish. With a population of more than 450,000, the parish cradles the city of New Orleans in a great crescent, from the shores of Lake Pontchartrain in the northwest to the Mississippi River-bounding communities of Marrero and Gretna to the south and southeast. Inside the EOC building, emergency personnel worked to evacuate, assist and rescue citizens across this expansive swath of earth.

The Jefferson Parish EOC was well-equipped for the task. There were hardened landlines, 800MHz radios, and cell and satellite phones tapping multiple networks. There was a fully redundant computer network, a fleet of remote sensors and cameras and a detailed disaster plan promising a multi-layered government response. And, of course, there was a system of canals, levees and pumping stations designed to channel flood waters out of the area.

Within hours of Katrina coming ashore, every one of those systems would fail catastrophically.

Nearly 100 people crammed into the bunker-like EOC building during Hurricane Katrina.

Cascading Failure
Dr. Walter S. Maestri has been director of the Jefferson Parish Emergency Management Department since 1996. The Sunday before landfall, he watched as Katrina boiled to a frightening crescendo. Now a Category 5 storm, among the most powerful ever recorded, it steamed directly toward New Orleans.

But when the storm weakened and turned unexpectedly to the east just hours before landfall, Maestri was confident. More than 90 percent of parish residents had evacuated. Those unable to leave were housed at hospitals and shelters with supplies and backup power. His team was in place at the EOC. Key personnel stood by in neighboring parishes to provide Day 1 communications and recovery.

"Up to the breaching of the levees, this was a perfect model," Maestri says from his office on the fourth floor. "We knew what we thought could happen. We knew what we had to deal with. We had prepared the plans, spread the message. Everything was according to plan. No surprises, no surprises," Maestri's voice trails off. He sounds almost reluctant to go on. "Then of course the storm comes in -- and things happen."

The power failed first, victim of the thrashing winds that toppled poles, tangled lines, and threw down transformers and other gear. The EOC, like the Parish's two data centers, are supported by diesel generators. In the EOC and at local hospitals and other critical facilities, the lights stayed on and the computers kept humming.

Jason Phillips monitors water-level and pump-station status from his workstations at the Jefferson Parish EOC in Marrero.

Out in the field, the parish's network of nearly 200 radio-based remote sensor stations switched to battery backup power. Jason Phillips, supervisory control and data acquisition manager for the parish, watched as the real-time feeds displayed on his twin monitors began to drop off one by one. Most batteries lasted from 45 to 120 minutes before going dark. Wind meters, shorn of their directional vanes, stopped transmitting telemetry on wind speeds, while many lakeside sensors kept working until the waters swallowed them. The last sensor quit with a reading of 29.6 feet.

"The ones we lost went underwater. Where we had them mounted we never would think the water on the lake would get that high," Phillips says. "But it did."

Voice and data communications failed next. When the power quit, general Internet access went with it. Phone service throughout the area faltered or failed entirely within hours of landfall. At the EOC, staff members could receive calls throughout the storm, but were unable to dial out. It would take five days for BellSouth to restore service to the EOC, one of the highest-priority service points in the parish.

Communications Collapse
Of course, the parish disaster plan anticipated these outages. Staffers had been issued cell phones from a diverse set of service providers to help ensure voice communications. In fact, most service failed as cellular towers sustained damage, switching stations lost power and call volume swamped the networks. Even the dozen or so EOC-operated Motorola satellite phones faltered during the storm, their signals blocked by thick cloud cover and heavy rain -- though they would prove invaluable in the days that followed.

The one system that worked -- the parish's nearly 2,000 radios -- suffered a crippling blow when the repeater antenna on the 21-story Galleria building blew away. That severed links to police, fire, hospital and key emergency personnel situated on the north side of the Huey P. Long Bridge. When crews rushed to fix the damage after the storm, they discovered that T1 service between the Galleria and the EOC was dead as well.

Like an army slicing through the weakened flank of an enemy, Katrina had cut off the EOC from the East Bank communities pinned between Lake Pontchartrain and the Mississippi River.

The gutted interior of Casual Home Furniture, which took on five feet of water when New Orleans flood waters flowed down Airline Highway. The store today does business from a large tent in the parking lot.

These were some of the hardest hit areas in Jefferson Parish. Flooding devastated portions of Metairie, particularly in the south, where floodwaters from New Orleans followed Airline Highway into the parish. Powerful winds battered lakeside communities. Vital resources like the New Orleans airport and the I-10 highway linking the area to Baton Rouge were a top priority.

Technicians rushed to replace the antenna and set up a point-to-point microwave link between the EOC in Marrero and the Galleria building. That enabled 800MHz radio communication parish-wide for first responders, but less than two days later the link failed again.

As emergency coordinator for Jefferson Parish, Tom Rodrigue was ready to battle wind, water and line damage. But he was stunned when he learned that FEMA personnel had disconnected the freshly mounted antenna.

"We had an instance where we got our antenna back up and someone at FEMA came in and put their communications up, because of the operation that was going on [near the Galleria site]. At which time the sheriff went back up there and put [on] armed guards, took their antenna down, put ours back up and says, 'Anybody tries to take it down ...'" Rodrigue pauses as a co-worker finishes the thought, "… they're going to shoot you."

Even with its radio service restored to the East Bank, the EOC struggled to communicate with state, federal and other parish agencies. Jammed frequencies and incompatible radio networks stymied cooperation -- Rodrigue describes the system at the time as "inoperable." Those difficulties enhanced the isolation created by rising floodwaters and broken roads.

"We had to resort to runners," says Carolyn Capdeville, Jefferson Parish coordinator for emergency management. "It was the pony express -- people going from place to place carrying messages."

Tulane's Education Six months after a glancing blow from Hurricane Katrina flooded New Orleans, the city's downtown is a work in progress. Street lights remain dark and power in office buildings fails regularly. A half block from the Superdome stands the dark monolith of the Hyatt Hotel. The building's façade is an ugly patchwork of blown out windows and white plastic sheeting, a caustic visual reminder of just how much work remains to be done. Paul Barron, interim CIO of Tulane University, has a birds' eye view of it all from his 14th floor office on Poydras Street, right across from the Superdome. Like New Orleans itself, Hurricane Katrina nearly finished off the 172-year-old university, Barron says. "During the first 60 days, it was unclear if Tulane was going to survive or not." Calamity struck quickly. "That Friday night I was at a preseason Saints game, and people were talking a little bit about the hurricane. But nobody was talking about evacuating," Barron says. Certainly not Tulane's IT department. Tim Deeves, director of network services, recalls kicking off full system backups Friday evening. Because the storm struck on a Monday, the IT staff had all weekend to work. Deeves himself shut down the school e-mail system on Sunday, at 2 p.m. "I was a little bit worried," Deeves says. "At one point I was the only person in the building, I was worried about the elevator losing power. I said, ‘Chris, if you don't hear about me by 1 o'clock, call public safety and tell them to come get me out of the elevator.'" The plan called for a courier from the Tulane security center to accept the tapes from Deeves and transfer them to a nearby site. But the courier failed to show up. Deeves was told to leave the tapes out in the 14th floor hallway for pickup. When the courier finally arrived, he found the doors locked -- the building staff had fled. Those tapes would spend more than a week stewing in the late-summer Louisiana heat. As it turned out, it was a good thing the process broke down. Rick McGinity led a team into downtown New Orleans to recover gear from the disabled Poydras Building. "The interesting thing is the security center is four and one-half blocks from here. That was not very smart," Barron says. "We lucked out in terms of getting our tapes, because the security center is much lower than we are, and there was water. We got lucky." Planes, Trains & Automobiles With electronic transactions unavailable after the storm, Tulane President Scott Cowen decided that paying staff was the top priority. To do so, the university needed to recover the tapes Deeves had left on the 14th floor. It fell to Rick McGinity, director of operating system and database services, to get them. What follows reads like something from the script of the movie "Trains, Planes & Automobiles." McGinity and his team couldn't fly into New Orleans Airport -- that was under military control -- so they landed in Baton Rouge, picked up two staff members and drove to one of their homes. There they picked up a second car and followed what McGinity describes as "a whole stream of ambulances going about 100 miles per hour." They talked their way past several roadblocks. At the Jefferson Parish operations center set up by the West Bank Expressway, the team climbed into a deuce-and-a-half troop truck with a contingent of heavily armed police. They rumbled past the nearly-empty Superdome and circled the Tulane building on Poydras Street, seeking shallow water. National guardsmen, patrolling the streets on Sea-Doos, looked on as McGinity waded to the entrance, reached into the murky waters, and keyed the lock at the bottom of the door. The walk up to the 14th floor was uneventful, and the team was soon tearing open disk arrays and servers in the sweltering heat. "We pulled disk drives and the tapes, up until the point where we just couldn't pull anymore stuff because time had run out," McGinity says. By nine that night, the team was on a plane out of Baton Rouge on its way to Houston. McGinity, his team, and those tapes would eventually land in Vorhees, N.J., where Paul Barron had set up Tulane with a logical partition on an IBM mainframe at Sungard. Tulane was back in business. Its employees -- at least, the ones on direct deposit -- would get paid on time. -- M.D.

'We Couldn't Find Them'
Jefferson Parish was fortunate. Unlike surrounding communities, it could radio state agencies.

"St. Bernard Parish, for whatever reason, couldn't reach the state," Maestri says of the neighboring parish to the east, which was utterly devastated by flooding. "So what happened was St. Bernard Parish could reach us and we could relay messages to the state. We were relaying messages for multiple parishes.

"Communication interoperability is a tremendous problem, and nobody learned the lesson from 9/11. That problem, through Katrina, still exists," Maestri warns. "Our challenge is not interoperability, it's operability. The commitment has been made that everybody will share the same system."

Canal breaches in the Lower 9th Ward and at the 17th Street Canal (shown) devastated large sections of New Orleans. Many areas remain uninhabitable.

The city of New Orleans, however, simply dropped off the map. "They abandoned their EOC and set up their temporary operating facilities in different buildings downtown, and we couldn't find them. Their towers went down, their radio systems went down, or whatever, and we couldn't find them."

One pleasant surprise: cellular text messaging service stayed up. For several days after the storm, text messaging was the only reliable way to communicate with people beyond the parish radio network.

Staffers also found after the storm that cell phones with out-of-state area codes often worked when local cellular numbers would not. Even today, staffers like Capdeville carry a second cell phone with them. Still, not everyone was quick to adapt.

"The state office of Emergency Preparedness, they have a program called E Team that works on the Internet, and they wanted us to go over that," Capdeville says of requests her team tried to place with the state. "And we kept hollering at them on an 800MHz radio that we don't have Internet. You've got to take it this way or you are going to lose us. There was no way to get to E Team."

The EOC team finally convinced the state to accept requests by phone, but even that was dicey. "Eventually the only communications system to stay up during the entire ordeal was satellite telephones," says Rodrigue. "Had it not been for that we would not have been able to make any requests to the state."

"It's not the technology that saves anybody, it's the people who use the technology," Capdeville says. "That is missed a lot of times, the human part of this."

Data Center Two-Step
Jefferson Parish maintains two discreet data centers, located more than 10 miles apart. Identically configured, each includes a single IBM AS/400 iSeries midframe server connecting to a 2TB SAN and linked to each other and the EOC via frame relay. While the two centers house the same equipment, they serve different application sets.

A mile north of the Huey P. Long Bridge, the East Bank data center in the ninth floor of the Joseph S. Yenni Building houses utility billing operations, as well as a Linux server running an Oracle GIS database for mapping services. Critical financial and payroll applications are housed in the West Bank data center, on a parish campus in Gretna. There are also 35 Windows servers hosting Exchange, Internet Information Server, DNS and application for several departments. The parish operates about 1,400 total Windows clients.

"Ever since the early '90s the plan was to have two data centers with redundant capabilities," explains Ridley Boudreaux, director of Electronic Information Systems for the parish. "Now we didn't set them up redundantly, it's not like a hot situation where I can instantly say now we're doing our payroll on the East Bank. But we always bought identical iSeries machines -- or AS/400s -- with enough capacity to run all of the applications on one of them."

Katrina tore open the Yenni Building roof, disabling the data center and damaging the upper floors.

That decision, which helped the parish shave costs, got a severe test when Katrina washed ashore. High winds ripped the large air conditioning units off the top of the Yenni Building, tearing open the roof. Water from the storm and ruptured pipes poured through the upper floors of the building. "Even if you have generator power -- with no air conditioning, forget it," Boudreaux says. "When you have something as catastrophic as the roof coming off and water cascading through the building, a whole lot isn't going to stay operational."

Even with the East Bank site disabled, Boudreaux felt lucky. The storm had largely spared the West Bank facility and its critical financial applications. "We were extremely fortunate that the building on the West Bank came through with flying colors. The generator worked. Power remained to that building so that system stayed up," he says.

Daily procedure calls for the backup tapes produced at each data center to be transported to the other site -- ensuring that data will survive a fire or catastrophic event. With Katrina's approach, the disaster plan called for up-to-date backup tapes to be transported from the two data centers to the hardened EOC site in Marrero. But the tech responsible for the transfer was running late, trying to get his family out of the area. The tapes ended up in Houston.

"It probably worked out better that way," says Boudreaux, who had the tapes shipped to a temporary location in Baton Rouge. When it became clear the Gretna site had ridden out the storm, Boudreaux's team got to work.

"We got one of my people and somebody from payroll operations in that building by Thursday after the storm and immediately started working on payroll, which was due the following Friday."

Boudreaux says that the AS/400-based iSeries platform eased migration of East Bank data center applications to the sibling midframe in Gretna. In just a few hours his team had dumped the bits onto the working AS/400 and was up and running. "You take a backup of Exchange and try to put it on a box that's not identical to the one you took it off of," he says with a laugh.

Boudreaux himself was on the north shore of Lake Pontchartrain when the storm hit. Rather than return to the stricken parish, he headed west "through downed trees and power lines" to Baton Rouge, where the parish was establishing an emergency seat of government. There Boudreaux could find what he needed to recover his IT operation: Power, communications and access to systems, partners and vendors.

"When I left I took all of my contact lists in hard copy -- I printed it all out," he says. "I had phone numbers for all of our vendors, I had passwords with me."

Even as Boudreaux got busy in Baton Rouge, one of his contractors, Walt Barowka, struck out from the north shore, heading south toward the stricken Yenni Building. His mission: To recover as much equipment and data from the disabled data center as he could.

"On his way down, I think it was the Thursday after, coming down from the north shore, he saw some young bucks cutting trees along the side of the road. He said, 'Come on, I'm gonna pay you,' threw them into the back of his truck, and brought four guys down, plus a couple of his own people and a couple of my people. They just hauled equipment down all day long, hauled it over to the Gretna building."

By the end of the day, the impromptu IT rescue team had delivered to the surviving data center everything but the AS/400 midframe system -- which was too heavy to carry down the nine flights of stairs.

"It was heroic," Boudreaux says. "That's the word I used over and over again -- absolutely heroic. Everything worked."

'You Learn Lessons'
Well, not quite everything. Boudreaux says the parish got caught flat-footed when its Web site went dark during the storm. The local hosting firm suffered damage in the storm -- it took more than a week for it to restore service. In retrospect, Boudreaux says locally hosting the Web site was a mistake.

"In the parish we try our best to support local businesses," he says. "You learn lessons. Fortunately, we were registered with Network Solutions -- it took a matter of minutes to switch it over."

The switch over was made possible by CMA Technology Solutions, the value-added reseller in Baton Rouge that had helped set up the parish's two iSeries systems.

"I showed up at their door 1 o'clock on Tuesday and said 'I need some help.' They gave me an office, they gave me space on their iSeries, they gave me a brilliant programmer, who helped me set up a one-page Web site that afternoon and get our address changed with Network Solutions."

The site hosted a registration page to confirm the whereabouts of parish personnel. Boudreaux also pointed staffers to a private page containing his latest contact information and location, helping him stay in constant touch.

He also funneled all e-mail -- which was completely unavailable in Jefferson Parish -- to his site in Baton Rouge. Boudreaux couldn't e-mail staffers in the afflicted area, but he could review correspondence and pass on urgent items by cell phone or text message.

As for the AS/400 iSeries midframe left in the Yenni Building, Boudreaux's team reached it four weeks later, once electricity and air conditioning had been restored. For nearly a month, the system sat in broiling temperatures and high humidity -- still, it booted up normally, even with one hard drive failed and another showing failure indications.

"So, the system worked. It came up and worked," Boudreaux says.

Lessons in the Wind Interoperate: In a macabre repeat of the snarled communications that hampered response during 9/11, parish officials struggled to reach local, state and federal agencies. Area parishes are now working with the state to implement a unified standard for radio communications. Local backup is not enough: Organizations dutifully backed up data ahead of the storm, then housed the media less than 10 miles away. Likewise, Jefferson Parish locally hosted both its e-mail and Web servers. It took heroic measures to recover these systems. Seek alternatives: When all communication systems failed, one stayed up -- text messaging. Also, cell phones with out-of-state area codes proved more reliable than local accounts. Both are now cooked into the emergency plan for the parish. Forge-trusted partnerships: Ridley Boudreaux restored Web and e-mail service within two days of the hurricane by turning to the VAR that set up his IBM iSeries systems. The company provided server resources, office space in Baton Rouge and a skilled programmer. Don't overvalue technology: Katrina pulled down the entire working infrastructure of the New Orleans area. And yet, the state urged Jefferson Parish to submit requests via the E Team Web-based logistics application. It took creative and dedicated people to work around problems. Non-digital backup procedures should be considered and developed. Stay or Go: The best place for many managers was outside of the stricken area, where they had access to communications, Internet, power and transportation. Still, it takes boots on the ground -- like Walt Barowka carrying hardware out of the Yenni Building -- to get things done. Challenge assumptions: Jason Phillips learned that wooden sensor station poles ably weathered the high winds, while galvanized steel poles often bent at 90 degrees or snapped entirely. The parish is switching to wood poles. Produce hard copy: Critical data such as contact lists, emergency documentation and authentication data should be printed and distributed, for access during extensive power or network outages. Also consider distributing documentation on compact USB keys, so remote staffers have easy and portable access. Reject wishful thinking: Many IT plans worked within the framework of a three-day evacuation and failed to account for the effects of a direct hit by a hurricane. -- M.D.

The Gathering Storm
By the time you receive this issue, the official start of the 2006 hurricane season will be days away. Yet, the entire region remains fragile. Maestri says about a third of Jefferson Parish lives in temporary FEMA trailers, secured with nylon tie-down straps. Many will almost certainly topple in Category 1 hurricane winds, meaning that an approaching tropical storm may require a full evacuation. Yet the storm-related financial struggles of families in the area makes evacuation -- with its steep costs in fuel, hotel bills and lost work -- prohibitively expensive.

And then there's the infrastructure. Maestri describes local electrical service as being held together "with spit, bailing wire and scotch tape." He warns that many electric utilities in the area are now operating under bankruptcy. The health-care system is in even more dire shape.

Jefferson Parish EOC Director Walter Maestri warns that damage from Katrina could make the next major storm much harder to cope with.

"We have gone from 11 major hospitals functioning to three. A million two hundred thousand people are now dependent on three hospitals, in essence," Maestri warns. "We had a level 1 trauma center -- it's gone. We had two major medical schools functioning -- Tulane and LSU. Those doctors, those students are now no longer here. They've moved out."

The grim calculus of evacuation demands that Maestri, Capdeville and other managers at the Jefferson Parish EOC shuffle the deck and re-tune their models. Capdeville has been tweaking the computer-based evacuation models, lowering her assumptions about road capacities and taking into account issues like a higher percentage of vehicles becoming disabled on roadways.

At the end of the day, there's only so much technology can do. "I think one of the problems is that we, 21st century Americans, are totally dependent, in ways they don't even think about, on technology," Maestri says.

"There's a picture that we have that somebody snapped, one of our own photographers, of the CEO and chairman of the board of a Fortune 50 corporation who lives here, sitting on the curb eating an MRE [meal ready to eat]," Maestri says. "I mean, he has planes at his disposal and God knows what. And he's sitting at a curb eating an MRE, because all of his money, he couldn't get to it."