Backup and Disaster Recovery
Commercial backup utility horror stories:
Novell:
At my first place of employment, the backup system was never tested. The environment used diskless workstations with all data housed on the server. I had to do a full disaster recovery after a VP decided to enter the server room and reboot the main server when they could not access their data. The untested backup software didn't work correctly with the servers fancy raid hardware. It took 24 hours to find a work around to restore the systems data.
At another former job, when Novell first introduced their new NDS directory service, no commercial backup vendor had a solution that could restore the Novell directory tree. The NDS tree houses all user and authentication data as well as info about resources of the NOS.
At another employer, the central corporate office dictated backup of Novell servers to tape drives housed on Windows NT systems using Arcserve. The system was never tested.
UNIX:
At a University, the main faculty and staff Solaris server was backed up via Arcserve to a Windows NT servers tape drive. A full restore had never been tested. Crackers broke in and fubared the system on Christmas eve. On attempting a restore, it was found that all subdirectories contained zero length files. One had to enter each subdirectory with the fancy Windows GUI backup tool and select all files in a subdirectory to restore them. On restoration, the file permissions and ownership were wrong. Thankfully I didn't start work there till mid January the following year.