...

2) A list of things to consider doing when the a clusters is being turned off / rebooted for other reasons.A proactive maintenance should be scheduled for approximately quarterly, and no longer than 6 months

Examples of activities taken in past maintenance windows:

During this down-time, we intend to:

• Test new GPU software capabilities
• Update the OS of the storage system.
• Update the BIOS of the 4 GPU clusters.
• Update the UPS software to address current software's limitations.
• Confirm backups and review other system software configurations.

Maintenance Checklist

· BIOS / firmware updates (and why it's done...)

· Network / switches

· Synology / storage system Updates

...

· Add / remove nodes

· Kernel updates    o : Only security updates. Account / password sync

· OS Updates: Only security updates (via YUM)

· UPS / check / updates / settings

· Chkdsk 

Sequence taken during maintenance

· Schedule / notify representative / users

Evaluate hard drives

Verify backups

· Disable outside access

· Delete jobs -

· Chkdsk

· Shutdown headnode & nodes

...