Here in North Carolina we usually think of late summer / early fall as the start of hurricane season. Well, this year is different (as is everything else in 2020) and we've already had a couple of named hurricanes develop, with one hitting Outer Banks in late July. Therefore, we wanted to go ahead and review what we do ensure we can continue to serve your library, your staff, and your patrons after a natural disaster, and share the additional improvements we've made this year to our Disaster Preparedness and Recovery plans. We've spent a significant amount of time building infrastructure, reviewing procedures, and planning for an untimely incident.
In fact, much of what we put into place as part of our disaster preparedness plan is what allowed our staff to begin to work from home in March and continue to do so even today and for the foreseeable future.
Our September 11, 2018 Key Notes Blog Post was written as we faced the threat of Hurricane Florence and shared details about we'd implemented at that point including a combination of both procedural and physical preparedness such as:
- A gas-powered generator at our office
- Redundant internet providers, firewalls, and network routers
- Daily backups of data to our on-site servers
- Weekly data backups stored offsite
- Encrypted database backups on AWS S3
- VOIP Telephone system to allow staff to work remotely
- Keystone Status Page to communicate database availability, even if we’re unreachable
- Contingency plans and equipment needed for remote database and customer support
On August 15, 2019, Katy posted to share the news of some additional steps we'd taken to further enhance our disaster preparedness and recovery efforts including:
- Back-end changes to KLAS 7.7 to it quicker and easier to create and store database back-ups
- A move to storing back-ups in the cloud, so they are safe and retrievable no matter where disaster strikes
A new monitoring app so staff can be notified right away if something goes wrong with the servers and any emergencies can be dealt with as quickly as possible
Today I'd like to share this year's improvements to our disaster recovery and preparedness process and infrastructure, which include:
- Cloud-based databases running in multiple regions, to better place the system geographically near the library
- Incremental transaction data backups happen every 10 minutes
- Database backups are saved in the local region, as well as to a separate region. If a database hosted on the east coast has a disaster, there is a copy of the database backup in another region.
- Database backups are saved to the local server, as well as copied to S3 storage
- Database backups are also copied from Amazon's data centers to Google Storage
- A new automated system restoration process which had been manual prior to the beginning of 2020. As an automated process, it takes about an hour while the previous manual process took 10-12 hours at a minimum.
