Strategic Tech Planning Is the Heart of Healthcare Disaster Recovery

When Alexander Popowycz became CIO of Rockledge, Fla.-based Health First in 2015, many of the organization’s IT systems lived in only one data center.

The situation was untenable for the integrated system, which operates four hospitals in an area notorious for hurricanes. If an especially large storm took down a single data center, Health First risked losing access to a variety of critical applications and information.

“There were a number of systems that weren’t dual-sited,” Popowycz says. “They didn’t have a backup environment at any other location.”

By the following year, however, as Hurricane Matthew moved toward Florida, Popowycz and his fellow hospital administrators stood ready to proactively shut down any noncritical systems and transfer crucial workloads to another site after introducing more redundancy into the IT environment. While the state ultimately avoided a direct hit from the storm, Popowycz was thankful for the upgrades.

“Most organizations don’t have the resources to make everything highly redundant and available,” Popowycz says. “You have to make choices.”

Disaster recovery remains a complex but vital aspect of every healthcare organization’s IT operation. A robust DR setup considers data center and network architecture, as well as processes such as planning, training and testing. Here, failure truly is not an option, and without the right balance of infrastructure and strategy, patient care can suffer.

SIGN UP: Get more news from the HealthTech newsletter in your inbox every two weeks

Protecting the Growing Reliance on Technology

Bruce Darrow, vice president of IT and chief medical information officer for Mount Sinai Health System in New York, stresses the importance of disaster recovery consideration early — as IT systems are designed — instead of adding on DR capabilities after systems are up and running.

“We think about our backup profile when we look at adding new applications to our portfolio,” Darrow says. “Where’s the primary? Where’s the secondary? For many of our systems, our primary data centers are local, and our secondary data centers tend to be hosted offsite.”

That’s because everything ­completely depends on IT, says Linda Landesman, a visiting lecturer at the University of Massachusetts Amherst School of Public Health and Health Sciences and author of "Emergency Preparedness for Health Professionals: Introduction to Disaster Response."w For instance, while doctors at one time relied on beepers, most now carry smartphones, she says. Today, the bulk of critical patient care systems increasingly relies on IT.

“Everything is tied into technology — the medical records, all of the systems that are conducting patient monitoring,” Landesman says. “If you were to go into a hospital room, particularly for the sickest patients, everything is plugged into the wall.”

Health First now commits all patient data to a minimum of two locations in real time, and also creates backups of those records in a third environment.

“The minute a physician enters data, it automatically gets recorded into two systems,” Popowycz says. The organization relies on VMware VMotion to enable the live migration of virtual machines; Health First proactively shifts virtual machines away from failing or underperforming servers.

Meanwhile, Baystate Health, an integrated health system in Springfield, Mass., recently replaced its aging data center infrastructure, moving toward a hyperconverged model that enables a high level of resiliency. The organization deployed VMware vSAN and VMware NSX on Cisco UCS C-series servers in three physical locations.

“We really said that disaster recovery, along with security, had to be a foundational piece of our design,” says Michael Feld, the provider's former acting chief technology officer. “This technology is a relatively inexpensive way of achieving a high level of disaster recovery, but also better performance.”

The new architecture’s durability has already been tested in real life. During construction work, one of Baystate’s physical data center sites was cut off unexpectedly from the other two. Thanks to the system’s design, nothing failed.

Placing Agency Protection in the Cloud

Officials at Baystate also expect to come to an operating agreement in the coming months with a cloud backup provider to act as a safety net.

To ensure business continuity, healthcare facilities increasingly turn to cloud-based backup and Disaster Recovery as a Service options. DRaaS protects organizations from losing data during service disruptions by automating live server functionality in a cloud-based environment. That means a much shorter recovery time — minutes or hours, rather than days or weeks — in the event of an actual disaster.

In 2016, nearly 47 percent of providers responding to the HIMSS Analytics 2016 Cloud Survey said they planned to use the cloud for business continuity and DR, up from 31 percent in 2014; 13 percent of respondents said their disaster recovery is cloud-based.

Diversified Options Seek to Fit Specific Disaster Recovery Needs

Health First now designs new IT systems with specific disaster recovery needs in mind. For instance, the health system recently consolidated five voice platforms into one.

“One of the key considerations was the ability to have a resilient environment for voice communications as a backstop,” Popowycz says. “We went through all sorts of disaster scenarios to determine what kind of service we could expect.”

Initially, the provider considered moving to an entirely cloud-hosted voice solution, but switched gears after realizing that a voice system with no on-premises component would shut down completely in the event of lost internet connectivity during a natural disaster.

Ultimately, Health First implemented a hybrid solution that utilizes cloud resources while still keeping part of the voice system in-house.

“We deployed a solution that, even if you weren’t able to make a long-distance call, you could call from floor to floor, to be able to do a phone consult,” Popowycz says.

The People People Behind Disaster Recovery Processes

Technology represents only one part of a DR strategy. Success also largely depends on creating detailed plans for what an organization will do during the course of a disaster, regularly testing those plans and ongoing collaboration with non-IT stakeholders.

Mount Sinai conducts drills to simulate how the IT department coordinates with the rest of the organization in the event of a disaster. Additionally, IT runs its own DR drills, simulating scenarios in which primary data sources become unavailable, and systems must be re-established from backups.

To ensure uptime, Landesman says, hospitals must develop plans for worst-case scenarios in which record-keeping systems and other IT-dependent applications may be lost.

“Now that very few people have paper records, that could be a real problem,” she says.

When hospitals receive warning before a storm, they might consider printing out hard copies of patient records that contain important information such as individual medications, Landesman says. It’s also essential for healthcare provider organizations to have backup communications systems in place, such as two-way radios, that can operate completely independent of technology resources.

Meanwhile, many younger doctors have little or no experience working with paper-based systems in a hospital setting. Darrow admits: “If you ask me to go to paper processes because IT systems are down, I at least have some experience of doing that. The further we get into the future, the fewer people who will have that experience.”

The key to mitigating such challenges is to establish clear downtime processes, ensuring that even digital natives will know what to do when the time comes, Darrow says. At Mount Sinai, a so-called “downtime box” is placed on every hospital floor, stocked with standard forms for progress notes, as well as specialty-specific forms tailored to the organization's various medical departments.

“As much as this is a discussion about technology, disaster recovery involves processes and planning just as much as anything else,” he says.