By Dave DeWitte
dave@corridorbusiness.com
Business continuity planning can go a long way during a disaster, as UFG discovered during the September 2016 flood evacuation of its downtown Cedar Rapids headquarters.
One of the most common responses from employees asked about improving the company’s disaster response was to use a different type of tape to label computer monitors for relocation, as it had left some marks when removed.
“If we’re down to things like that, we’re in pretty good shape,” UFG Chairman and CEO Randy Ramlo remarked during the CBJ’s recent Economic Forecast Luncheon.
The secret was having a strong business continuity plan (BCP), which made it possible for the insurer to operate continuously throughout the five days employees were evacuated from their offices in the 100 block of Second Avenue SE.
UFG was actually finalizing requirements on a mutual assistance agreement for IT resources with Cedar Rapids Bank & Trust (CRBT) just before flood crest predictions were issued for the Cedar River the week of Sept. 19.
“We had planned to conduct a connectivity test once the written plan was completed,” UFG Corporate Communications Manager Colleen Scholer said, quoting a post-disaster review. “The test ended up being a live disaster recovery event based on executing our UFG disaster recovery plan and our requirements developed for the CRBT agreement.”
As a publically traded insurance company, UFG requires its IT department to process large amounts of data every day regarding billings, claims and other vital processes.
“Nightly production ran on time each night with online backup each morning by 5:30 a.m.,” Ms. Scholer said. “CRBT provided incredible cooperation and hospitality.”
Keeping the power on in most of UFG’s offices was a big part of the continuity plan. After the flood of 2008, the company began a multi-stage process of moving its electrical switch gear out of basement and first-floor spaces that were vulnerable to flooding. By simply flipping a switch, power could be cut off only to the building’s vulnerable lower levels.
Personal computers from UFG’s offices in the SCI Building and the 1st and 1st Building were moved to upper floors of the company’s building at 118 Second Ave. SE so they could be accessed remotely using Microsoft’s Remote Desktop Protocol (RDP).
“The Information Systems tech team did a great job moving PCs to our 118 building for RDP, failing over production servers to the Involta data center in Marion, setting up essential services at CRBT, and being at CRBT the entire week of the outage to get staff members up and running early Monday and troubleshooting any issues as the week progressed,” Ms. Scholer said.
The original plan had been for UFG to use CRBT’s digital subscriber line (DSL) service for internet access, but CRBT provided added bandwidth by sharing resources on its internal production network, Ms. Scholer said, making work go a lot smoother for UFG’s staff.
“Our employees were productive starting Monday morning, the first day out of the office, and no service issues were reported in daily senior management team conference calls the entire week,” Ms. Scholer added. “In fact, a check of completed tasks by our underwriting support staff in the Midwest region on day four had a higher count than the prior week after four days.”
The number of users on UFG’s internal Citrix network increased from 250 to more than 600 during the week of the flood. UFG moved two extra Citrix servers to Involta, its data hosting provider, to spread the load of additional users. IT resources were replaced in their former locations over the weekend of Oct. 1, and employees went back to their offices on Oct. 3.
Lessons learned
The value of a robust BCP was also illustrated by the uninterrupted operations of Informatics, the web services agency at 215 Second Ave. SE, Suite 300, in downtown Cedar Rapids.
Informatics was likely the only business to operate continuously from its downtown location during the flood of 2008, thanks to generator power, but it has made a number of updates to its BCP since then.
“We live and die by our ability to serve our clients,” Vice President and COO John Osako said. “Our local clients are very cognizant of the potential for flooding. They need their applications to operate no matter what they’re going through.”
Mr. Osako said the biggest change since the 2008 flood evacuation has been the relocation of its IT resources to offsite data hosting facilities, where they can be accessed via high-speed internet connections. The firm company also added new software to its catalog of IT resources, some of which is deemed critical for maintaining access during a disaster.
Preparing Informatics’ critical IT resources for evacuation last September took only about two hours, Mr. Osako said, and setting them back up in a different location took about the same amount of time. The work was completed on Sept. 25 after the evacuation was ordered by city officials, allowing the company’s 20 employees to work from home on Sept. 26 as city officials closed the downtown area and kept watch over temporary flood barriers that held back the river.
Members of the Informatics team stayed in touch while working from home using Slack, a desktop app for instant messaging and audio conferencing, and Zoom, a web conferencing service, Mr. Osako said, helping to maintain productivity.
The amount of effort and knowledge that goes into preparing a BCP varies widely depending on the company, he noted. While many firms have the IT expertise to prepare and implement a BCP in-house, he said others would be better off retaining a specialist in IT continuity and security.
Planning in advance is the important thing.
“The worst time to figure this out is in the middle of a disaster, because there’s so much going on,” Mr. Osako said.
