See What we are Talking About!
Good Backups Gone Bad: The Downside of Assuming you Backups are Good
Steve Woody, CommitDBA Director
June 29, 2012
Of course all IT Managers and DBAs know that solid database backups are absolutely necessary. Well, maybe not all, since there have been cases where recovery was impossible due to a variety of reasons including apathy, sloppiness, lack of planning, lack of execution and other reasons. However, most professional IT staff in the industry understands that database backups are essential. This is true no matter what type of enterprise database - be it Oracle, SQL Server, DB2, MySQL, Informix and others.
Well structured organizations that house an “on-premise” IT infrastructure typically embed ITIL based Service Level Management (SLM) operation practices and implement risk management policies. With e-commerce, mobile apps, and enterprise application software the norm, all of IT infrastructure must be trusted with risk mitigation standards. This includes foolproof database backups. Solid backup execution is a key component of IT Continuity Strategy and one of the critical success factors of every organization that values its corporate data. Since mission critical data is an invaluable asset for any organization, it’s never good for a DBA to assume that your backups are good. So what are the “checks and balances” that DBAs can rely on to provide confidence in backup reliability instead of assuming their database backups are in order?
It is imperative to take the time to test various types of database recoveries (e.g. Hot, Cold, Logical, Full/Partial, Incremental/Differential, etc.) so that nothing is left to chance when a critical recovery is needed. The testing of backup and recovery scenarios helps to meet risk management initiatives and provides the DBA with less anxiety should a need arise to perform a point-in-time recovery. Many database recovery attempts have failed due to lack of adequate testing.
Database monitoring is typically thought of as an automated tool that monitors and checks for database internal incidents and performance degradation issues. While this is true, most vendor monitoring tools provide the ability to monitor internal and external backup red flags that alert the DBA of problematic issues with notifications relating to backup success factors. If specific tools do not provide this capability, then scripts can be developed to augment the monitoring tool to include backup monitoring. If an organization does not have a monitoring tool, then it is recommended to create custom scripts tailored to your environment.
Some organizations provide change management, change control processes and documentation while others have nothing. The purpose of change management is to ensure that standardized methods and procedures are used for the efficient and prompt handling of all changes. The documentation of any requested changes or just-in-time necessary changes for backup processes should be logged into a backup change file for the purposes of keeping track of backup modifications.
Scalability and Capacity
With the trend toward massive data overload, storage and backup capacity become a concern. Virtualization also presents additional backup concerns in large storage environments. Be sure to work with your IT management and team to discuss appropriate large volume backup strategies along with hardware, software and compression solutions for huge data volumes.
If you are running your own backup scripts but are concerned with massive data growth and scalability, it would be beneficial to investigate robust and feature rich backup utility tools. Database vendors provide their own backup utilities such as Oracle’s Recovery Manager or RMAN and SQL Server’s Backup Maintenance Plan. There are plenty of third-party vendors who also offer backup software utilities such as LiteSpeed for SQL Server and Oracle from Quest Software.
Backup Process Audit
It is good practice to audit your backup plans and processes once every six months at minimum. An audit protects the organization from any holes or gaps that might have occurred due to any database, system, network or other environment modifications.
While this article does not cover disaster recovery (DR), it is within the scope to mention that backups should be shipped off-site or replicated off-site in case of problematic issues. Vendors such as Iron Mountain, Barracuda Networks and other vendors provide these types of services.
New technology solutions include cloud based backup services where you can outsource your backups to cloud based service centers. Your databases are auto replicated to the cloud centers which offloads many of a DBA's on-premise backup responsibilities.
Solid and foolproof database backups must be the number one priority for the DBA in terms of priority rankings. It is necessary to put in the rigorous controls and solutions in order feel confident when a database restore is required in an emergency recovery situation. New technologies and service models should also be analyzed in order in improve backup requirements with increasing data growth. Finally, don't assume your backups are good! Take the appropriate steps to solidify your database backup and recovery.