Launching a Disaster Recovery Plan Twice in the Same Month

Add bookmark

Having a strong Disaster Recovery Plan in place is vital for any shared services operation. In fact, many practitioners are quoting disaster preparedness as one of their key concerns for the future – witness the world-wide instabilities of recent years. Rinker’s shared services group, based in South Florida, had the opportunity to test its disaster recovery strategy not just once, but twice, when 2004's record Hurricane Season threatened South Florida’s tranquility – and with it, the support systems which Rinker’s worldwide networks relied on.

Frances and Jeanne. If those two names don’t mean much to you, then you probably weren’t in South Florida during the waning days of summer 2004. And if you weren’t in South Florida during that time, be very glad. Because during a few weeks, late last summer, the pristine, idyllic beaches of South Florida, which normally offer hours upon hours of sun drenched rest and relaxation, became a place of fury, of 120+mph winds, massive waves, torrents of rain and a profound exhibition of the power of nature.

This is the story of how two major weather events that took place within two weeks, and only a few miles of each other, not only made meteorological history books but devastated lives and changed the world for at least one shared services organization located in "hurricane alley."

This is also a story about people, planning, processes, systems and communications. It’s about how a group of people got together to implement emergency plans and keep their families together and safe while keeping the shared services organization of a worldwide, multi-billion dollar business running. Welcome to the story of Rinker Group Shared Services, and how they got through Hurricane Season 2004.

Background

Rinker Group is one of the largest heavy building materials companies in the world with over 600 locations in three countries, including the US, Australia and China. Rinker’s corporate office, its shared services organization and IT data center supporting the entire worldwide business is located in West Palm Beach, Florida. That means, if the shared services center lost systems, all of Rinker’s 600+ sites globally would lose systems. Additionally, all the US sites depend on the center to for various activities, not the least of which is payroll.

Rinker Shared Services consists of Business Services (which includes customer billing, accounts receivable, accounts payable, payroll, general accounting, master data, imaging, etc.), IT, HR and People Development. Rinker has SAP as its ERP, Microsoft for email, file and printer sharing, and a host of other applications. This describes the general structure of Rinker’s shared services organization, which does its part to help keep Rinker at, or near the very top, in most categories, when ranked against competitors. Rinker shared services is located under one – very strong – roof in beautiful West Palm Beach, Florida, not very far from those beaches where nature decided to vent its fury last summer.

2004 began like any other year. Rinker shared services organized meetings to assess where they were at as an entity within the company, and as individuals. Performance was evaluated; improvement plans created; goals designed to take Rinker shared services "to the next level" were put in writing. Then, around the time most people in the US are looking forward to a warm summer and planning vacations, Rinker shared services teams met to "look forward" to something quite different: Hurricane Season.

Hurricane Planning

Hurricane planning is part of life in South Florida. At home or at work, you have to have some idea of what you are going to do if a big one hits. West Palm Beach has been fortunate, in that no major hurricane has made a direct hit on the area since the 1930s. That does not mean the people who live and work there let their guard down. They plan for the worst, hoping those plans never have to be implemented.

2004 seemed to be going along pretty well for West Palm Beach, but not for Florida. In mid-August, Hurricane Charley slammed the west coast of Florida. Rinker has plants there, and so people at the corporate office got first hand accounts of the devastation...when we were able to communicate again. And while the Rinker shared services team breathed a sigh of relief at being spared a direct hit from Charley, all were uneasy because we knew that two and half months of hurricane season remained.

Rinker as a company did its part to help people affected by Charley get some relief, and at the same time get its impacted businesses back on line. Some of our shared services team drove to the west coast, bringing food and water to people hit by the hurricane. They also brought wireless internet modems to plants where downed phone lines had knocked these off the Rinker network. Back in West Palm Beach, NFuse delivered applications to those plants via the web. The Internet modems and NFuse got Rinker plants connected to mission critical systems and provided a way to get connectivity to the plants, while the phone company struggled to get service restored.

While dealing with the after effects of Hurricane Charley, everyone in South Florida kept their eyes and their web browsers on the weather. In the South Atlantic, a tropical storm named Frances was forming. Hurricanes are rated in categories, with 1 being the lowest and 5 the highest, or most powerful. We watched as Frances went from a tropical storm to a hurricane.

When it became a 1, it was still far enough away not to be a worry. But tracking maps gave our shared services team some concern. Projections showed Frances making landfall directly at, or very near, West Palm Beach, Florida. Rinker shared services began implementing their Disaster Recovery (DR) plan (see Figure 1).

Figure 1

(Click on the image to enlarge)

Figure 1: Tracking all 4 hurricanes - Rinker sites denoted by triangles

Preparing for Disaster Recovery

The Rinker shared services Business Continuity Plan (see Figure 1) is coordinated through five core teams:

the Information Technology Disaster Recovery Team and Recovery Site
the Information Technology Help Desk
the Business Process Support and Communications Team
the Corporate Office Support Staff
Travel Planning and Coordination

The first thing to nail down was who was going where. Rinker shared services planned to have two sites outside of Florida. One site was for IT systems recovery and one for Rinker shared services business continuity. A subset of the technical team would go to the IT systems recovery site while a subset of the business process team would go to the business continuity site. A third team was formed to manage communications to the business. Rinker used subset teams in order to have local staff on hand in the event the hurricane missed West Palm Beach.

Setting Up Remote Sites

While it seems straightforward to get people to sites outside of a hurricane area it is not a simple undertaking. This is because people have families and loved ones and when Frances was about 72 hours away it was rated a potential Category 3 hurricane. Naturally, no one wanted to leave their family behind to endure the storm and neither did Rinker. Travel plans included not only getting the Rinker shared services Disaster Recovery team to their recovery location, but their families as well.

With travel plans set, the IT Technical DR team went into action putting the recovery vendor on notice and gathering the required equipment, backup tapes, etc needed to recover systems at Rinker’s site in New Jersey. The technical team is responsible for maintaining the IT systems infrastructure including servers, network and supporting hardware and software. The team ran through their disaster recovery checklist to ensure everything was ready to go. At this point projections had hurricane Frances 48 hours before landfall headed straight for West Palm Beach. Rinker Shared Services operations continued normal operations. The IT Technical DR team was then split in two groups. The first half headed out at once to configure the systems at the recovery site. The second half of the team remained in West Palm Beach in order to get the latest systems backup possible. Projections reached a point where hurricane Frances was 24 hours away from landfall still on course for West Palm Beach. With backups in hand, the second half of the technical team headed to New Jersey.

At the same the time the technical DR team was set in motion, the Business Process DR team was also working. The IT Help Desk is critical to supporting business processes and is part of this DR team. The IT Help Desk DR team began determining the staffing required to support the business locations not directly impacted by the storm. At the 48-hour-before-landfall mark, a subset of the help desk team left West Palm Beach to position itself in a regional office in Las Vegas. At the 24-hour-before-landfall mark, the normal IT Help Desk 800 number was redirected to the DR office in Las Vegas, and to a bank of phones specifically isolated for the Help Desk staff.

The Business Process Support team staffed a support team in Las Vegas, as well. This team had people in place to ensure all business processes continued, regardless of the events in West Palm Beach. They were available to support the business as usual and answer questions related to the hurricane.

The Communications team also made their way to Las Vegas. One key learning for Rinker as a result of 9/11 is how important it is to keep communications flowing. As a result, Rinker adjusted its DR plan where email was moved to the top of the list of systems to be recovered. Also, a web site was hosted externally to provide information regarding the business and systems. The Communications Team’s job was to communicate to the business via email and also to update the external web site hourly.

The team that remained in West Palm Beach all the while continued implementing their plans. First, was making sure Rinker people were safe. Several Rinker employees needed assistance prepping their houses for the hurricane. A small team coordinated and installed hurricane shutters and plywood on the houses of those employees requiring assistance.

The remainder of the DR team worked in the corporate office. They ensured the building was secure, all corporate staff had safe facilities to stay in and purchasing the required supplies, such as fuel for the power generator.

On September 5th, 2004 Francis made landfall as a category three hurricane. About four hours into the storm, power was lost and the emergency power generator kicked in. The West Palm Beach data center continued to operate with systems and data communications feeding all 600+ locations. The DRP team in New Jersey was positioned to switch over the redundant systems if failure did occur in West Palm Beach. The help desk and communications teams in Las Vegas provided round the clock communications to all 600 locations.

The storm passed the next morning but left a path of destruction including downed power lines, trees and flooding. The IT DR and Business Support DR teams remained in place for five days while the West Palm Beach office and data center remained on generator power. On day five after Frances made landfall, power was restored. The West Palm Beach staff began to return to work and the Disaster Recovery teams were called back home.

And Again

Two weeks later, as we began to get back into our day-to-day routines and projects we realized that we were going to have to do it all over again. Hurricane Jeanne literally made a complete circle and turned towards South Florida poised to hit the coastline in almost the exact coordinates as Frances.

The DR teams went into action with their Disaster Recovery action items. Jeanne did hit the coast very close to where Francis made landfall. This time the power was lost for 3 days and the remote teams once again provided the communications, support and business continuity around the clock.

Conclusion

Frances and Jeanne changed the world for Rinker shared services. Rinker learned that they could withstand two major hurricanes and continue business operations as usual. And while Rinker is confident in its ability to do it all again, there are several areas of improvement for 2005’s hurricane planning. In the end it’s about people, planning, processes, systems and communications.

Lessons Learned

People first: distraction impairs performance. Find what your team members are worried about (i.e. loved ones, house, etc) and work to eliminate that distraction so that they can focus on the goal.
Never, ever, ever underestimate the value of communicating with your teams, your customers, your vendors, everyone.
Arrange for multiple war rooms: Technical, Functional, Management, etc.
Equip war rooms with multiple analog phone lines that do not rely on a switch.