On monitoring
Frequency: Every 1 day
Send via e-list: DNS-DIAPER-REPORT-L@cornell.edu
Content: Daily data entry count of the last 30 days
Frequency: Every 1 hour
alert_count = 0
| Target | Data source | Value | Expected (if) & Otherwise | Action (then) | Personnel |
|---|---|---|---|---|---|
| Database | BioHPC: test.sample | Count of entries filtered by:(date_time in last 48 hours) | >= 1 | // Do nothing | |
| == 0 (or timeout) | alert_count + 1 | All | |||
| Backend mobile APIs | mobile test server/api/monitoring | {
"api_env": "test",
"connected_db": "test",
"pipeline": "mobile"
} | Match all keys/values | // Do nothing | |
| else (or timeout) | alert_count + 1 | Chengyong, Vivian | |||
mobile prod server/api/monitoring | {
"api_env": "production",
"connected_db": "production",
"pipeline": "mobile"
} | Match all keys/values | // Do nothing | ||
| else (or timeout) | alert_count + 1 | Chengyong, Vivian | |||
| Backend dashboard APIs | dashboard test server/env | {
"api_env": "TEST",
"connected_db": "TEST",
"pipeline": "dashboard"
} | Match all keys/values | // Do nothing | |
| else (or timeout) | alert_count + 1 | Weihang | |||
dashboard prod server/env | {
"api_env": "PROD",
"connected_db": "PROD",
"pipeline": "dashboard"
} | Match all keys/values | // Do nothing | ||
| else (or timeout) | alert_count + 1 | Weihang |
if (alert_count > 0) {
put all error info (expected v actual) in an email
send the email to DNS-DIAPER-ALERT-L@cornell.edu
}
else if (current time between 08:50am and 09:10am) {
send an email "All status OK" to DNS-DIAPER-REPORT-L@cornell.edu
}
// else { do nothing; will run again at next hour; } |