Add true backup functionality to the existing cluster. Discussion started Oct. 2013. |
Challenges to doing backups: The problem to solve
The quantity and number of files can put demand on the headnode and may take too long
- Will the backup put too much demand on the computer, competing with the computer's primary research purpose?
- Will the backups take too long?
The number of files
- 100's of millions of small files, along with "normal" files.
The amount of data
Cost is an issue since note originally budgeted.
Goals
Capability to restore system and data hardware to all-new hardware
Deal with various failures, such as:
- file system corruption#
- hard disk failures
- RAID controller card failures
- motherboard failure
- other server hardware of sub-system failure
- fire damage of room*
- water damage of room*
- theft*
- malicious incursion into the system*#
* Requires an off-site copy to be effective.
# May require versioning to get prior copies of since-corrupted data.
Ability to have some versioning, if not too expensive an add-on.
Options
Option1: In-room, off-box, copy
HDs in a dedicated computing box.
Option1: Out of room
HDs in a dedicated computing box.
Hosted file service.
EZ-Backup.
- $850-1,650/yr, depending on level of file compression.
Ideas to consider for above options
Software to sync data, ideally with versioning
EZ-Backup
CrashPlan
Compare
Real-time Backup Version Retention
Backup Frequency and Version Retention
Backing Up Very Large File Selections
Use software that backs up changes to the partition, not looking at files per se.
