Around our Raleigh, NC office today everyone is talking about how they are preparing their homes for the unwelcome arrival of Florence. But you might be wondering how we’ve prepared our operations and support for such an event.
The answer is a combination of both procedural and physical preparedness including:
- A gas-powered generator at our office
- Redundant internet providers, firewalls, and network routers
- Daily backups of data to our on-site servers
- Weekly data backups stored offsite
- Encrypted database backups on AWS S3
- VOIP Telephone system to allow staff to work remotely
- Keystone Status Page to communicate database availability, even if we’re unreachable
- Contingency plans and equipment needed for remote database and customer support
We have prepared for events like we are now facing—whether the event was a hurricane, ice storm, or some other disaster. Part of our annual SSAE audit is to further review and refine those disaster preparedness plans.
For example, our office is equipped with a 60kVA natural gas generator to power the building in the event of an outage. If electrical service is disrupted, the generator will maintain power to our servers and communications systems as long as the natural gas line provides fuel. Keystone also has redundant fiber-optic connections to the Internet with separate vendors, along with dual network routers and firewalls. This increases the chances that our communications will remain operational throughout a natural disaster—if one network goes down, the other can take over.
Here’s a summary of some of the anticipated “events”, the measures to manage them, and the anticipated impact on customers:
- Loss of power at Keystone’s office
- Natural gas generator automatically kicks in to provide power to Keystone’s servers, and 2 workspaces for Keystone staff to be able to work.
- The generator was tested less than a week ago (Saturday 9/8).
- No discernable impact on customers.
- Roads are impassable / Unsafe for Keystone staff to come to work
- VOIP telephone system allows staff to respond to phone calls remotely
- VPN access allows staff to connect to the office network and work as if they were at their desks.
- No discernable impact on customers.
- Roads are impassable and power is out at Keystone staff homes
- Unfortunately, we don’t want staff trying to travel to the office if it’s not safe, and if they lose power/internet access from home, the staff won’t be able to support customers.
- Impact: KLAS Hosting will continue uninterrupted, but our response to support calls will likely be greatly reduced until conditions improve.
- Loss of communications at Keystone’s office
- Keystone has redundant fiber-optic Internet connections, from separate companies.
- KLAS Hosting would not be impacted unless *both* connections were lost. The systems will automatically accommodate the loss of one.
- Keystone’s telephone service would be impacted if our 2nd connection is lost. In this case, we will be reduced to only email communications.
- In the case of loss of both communications links, Keystone will transfer KLAS hosting operations to our cloud-based Disaster Recovery site. (more information below)
- Customer impact: depending on the number of communications links that are lost, the impact will range from being rather minor to quite substantial. Keystone will use the “Announcements” section of the new “Keystone Status Page” to communicate the current impact, and how to best contact us. It is accessible from the Keystone Status Page menu item on klasusers.com or this URL: https://uptime.statuscake.com/?TestID=emlREBtN3e
In the event of loss of both network connections, or power from the generator, we will begin the process of migrating customers to our new cloud-based disaster recovery site. This process takes several hours to complete, therefore customers will be notified when their database is available along with instructions on how to connect to the cloud database, if necessary.
Procedurally, we take steps on a daily, weekly, and monthly basis to plan for both minor and major disruptions and disasters. We have checklists of tasks to complete just prior to anticipated weather events such as Florence to make sure we securely lessen the impact and the time and effort needed to recover. Such regular and one-time planning includes making sure we have secure backups of your data as well as the ability to continue maintaining and providing access to it. Additionally, we’ve taken steps to allow our support service personnel to work remotely with continued access to your databases, and to our phone and email tracking systems.
Just recently we implemented StatusCake, a monitoring system with a Keystone Status Notification Page, which will tell you whether or not KLAS is up, even if you cannot reach us. In KLAS 7.7 (the next major release) each library / organization will have their own notification page to check their own database’s status. KLAS 7.7 will also allow you to access KLAS remotely via a secure https: connection rather than requiring a VPN.
Our physical office is not located in a floodplain, is near major electrical and telephone distribution centers, and all cables into the building are buried underground. During Hurricane Fran, the last major hurricane to affect Raleigh, services were restored to the area around Keystone current offices within 2 hours. Since then, the City of Raleigh Police Department has leased the lower floor of our building which means that restoring power and communications to the building is of an even higher priority to the utility companies.
We hope that all of you who are also in Florence’s path are able to stay safe. And for all of our customers, no matter what Mother Nature sends our way.
Rest assured that we're well-equipped to keep your data safe and any interruptions to your service at a minimum.
