Since 2018, we've made an annual post detailing our infrastructure and planning efforts to serve your library, your staff, and your patrons after a natural disaster, and share the additional improvements we've made in the last year to our Disaster Preparedness and Recovery plans. We've spent a significant amount of time building infrastructure, reviewing procedures, and planning for an untimely incident. For this year's post, we starting with our most recent upgrades and working backwards to our original September 2018 update.
2021 Updates & Improvements
To prepare for this year's post, we asked Lee what he felt were important updates or changes made in 2021, or for any other info he wants the KLAS Users' Community to know about our efforts. The two items he mentioned are:
- The recovery process is the the same as before, but note that restoration of databases is sequential (per server), not parallel. So, while a one-hour recovery for a database at top of the queue is possible others at the bottom of the list will have a longer wait as the recovery process works through the list.
- We have added weekly server snapshots to our AWS backup servers. For an end user, this makes no difference to restoration, but it makes Keystone staff's jobs infinitely easier.
2020 Updates & Improvements
Much of what we put into place as part of our disaster preparedness plan in 2020 is what allowed our staff to begin to work from home in March of that year and continue to do so even today and for the foreseeable future.
On September 1, 2020 posted a list disaster recovery and preparedness process and infrastructure improvements we'd make over the past year such as:
- Cloud-based databases running in multiple regions, to better place the system geographically near the library
- Incremental transaction data backups happen every 10 minutes
- Database backups are saved in the local region, as well as to a separate region. If a database hosted on the east coast has a disaster, there is a copy of the database backup in another region.
- Database backups are saved to the local server, as well as copied to S3 storage
- Database backups are also copied from Amazon's data centers to Google Storage
- A new automated system restoration process which had been manual prior to the beginning of 2020. As an automated process, it takes about an hour while the previous manual process took 10-12 hours at a minimum.
2019 Updates & Improvements
On August 15, 2019, we shared the news of some additional steps we'd taken to further enhance our disaster preparedness and recovery efforts including:
- Back-end changes to KLAS 7.7 to it quicker and easier to create and store database back-ups
- A move to storing back-ups in the cloud, so they are safe and retrievable no matter where disaster strikes
- A new monitoring app so staff can be notified right away if something goes wrong with the servers and any emergencies can be dealt with as quickly as possible
2018 Updates & Improvements
Our September 11, 2018 our first Emergency Recovery & Disaster Preparedness Key Notes Blog post was written as we faced the threat of Hurricane Florence and shared details about we'd implemented at that point including a combination of both procedural and physical preparedness such as:
- A gas-powered generator at our office
- Redundant internet providers, firewalls, and network routers
- Daily backups of data to our on-site servers
- Weekly data backups stored offsite
- Encrypted database backups on AWS S3
- VOIP Telephone system to allow staff to work remotely
- Keystone Status Page to communicate database availability, even if we’re unreachable
- Contingency plans and equipment needed for remote database and customer support
