Challenges to doing backups: The problem to solve

The quantity and number of files can put demand on the head-node and may take too long

• Will the backup put too much demand on the computer, competing with the computer's primary research purpose?
• Will the backups take too long?

System uses ~800 cores, with 30 programs (per Yi He).

The number of files

• 100's of millions of small files, along with "normal" files.

The amount of data

• 3.1TB of data (research and users)
  • On 6TB partition
• 77GB OS data
  • On 160GB partition

Cost is an issue since note originally budgeted.

Goals

Capability to restore system and data hardware to all-new hard drives, if necessary

Deal with various failures, such as:

• file system corruption#
• hard disk failures
• RAID controller card failures
• motherboard failure
• other server hardware of sub-system failure
• fire damage of room*
• water damage of room*
• theft*
• malicious incursion into the system*#

* Requires an off-site copy to be effective.

# May require versioning to get prior copies of since-corrupted data.

NOTE: If other hardware than hard disks are required to be replaced, OS may need to be modified

Ability to have some versioning, if not too expensive an add-on.

Options

Option1: In-room, off-box, copy

HDs in a dedicated computing box.

Option1: Out of room

HDs in a dedicated computing box.

Hosted file service.

EZ-Backup.

• $850-1,650/yr, depending on level of file compression.

Ideas to consider for above options

Software to sync data, ideally with versioning

EZ-Backup

CrashPlan

Compare

• http://www.crashplan.com/consumer/compare.html

How CrashPlan backup works (technical)

• http://support.crashplan.com/doku.php/articles/how_backup_works

Real-time Backup Version Retention

• http://support.crashplan.com/doku.php/feature/version_retention_and_backup_frequency

Backup Frequency and Version Retention

• http://support.crashplan.com/doku.php/reference/version_retention_and_backup_frequency

Backing Up Very Large File Selections

• http://support.crashplan.com/doku.php/how_to/seed_archive#backing_up_very_large_file_selections

Software that backs up changes to the partition, not looking at files per-se.

One possibility Oliver found:

CA ARCserve D2D I2 Backups

• http://www.arcserve.com/us/products/ca-arcserve-d2d/ca-arcserve-d2d-i2-backups.aspx

12 minute technical review of D2D

• http://arcserve.com/~/media/Files/Videos/ARCserve/ca-arcserve-d2d-for-Linux-16-5.mp4

PDF Brief

• http://www.arcserve.com/us/~/media/files/productbriefs/ca-arcserve-d2d-r16-product-brief.pdf

Pricing

We'd like to know the educational price (if any), but here are the upper-bound prices:

http://shop.arcserve.com/Products/D2D/CA-ARCSERVE-D2D-FOR-LINUX-Product-plus-1-Year-Maintenance

• (List) Price: $732.00

http://shop.arcserve.com/Products/D2D/CA-ARCSERVE-D2D-FOR-LINUX-Product-plus-3-Year-Maintenance

• (List) Price: $976.00

Oliver can't tell whether one pays more for additional clients. The other OS options specify number of seats, so the above implies to Oliver a single price gets you unlimited clients(?) per server.