• Created by user-0432d, last updated by Unknown User (ss2876) on May 15, 2022 2 minute read

On monitoring

  • REPORT (e.g. every 1 day)
  • ALERT (e.g every 1 hour)


Pipeline

Monitoring pipeline

Monitoring Job 1: Daily

Lambda function name: report_atlas 

Frequency: Every 1 day

Send via e-list: DNS-DIAPER-REPORT-L@cornell.edu

Content:

  • Daily entry count of last 30 days (sorted by date in descending order)
  • Users' cumulative count (sorted by cumulative count in descending order)
  • Users added in the last 30 days


Monitoring Job 2: Hourly (Daily)

Lambda function name: job2 

Frequency: Every 1 hour (Now also changed to Every 1 day)

Content:

  • Status: OK or any alert (details of errors if any)

Step 1: Init

alert_count = 0

Step 2: Run all rules

TargetData sourceExpected ValueExpected (if)
& Otherwise		Action (then)Personnel
DatabaseBioHPC: test.sampleCount of entries filtered by:
(date_time in last 48 hours)		>= 1// Do nothing



== 0 (or timeout)alert_count + 1All
Backend mobile APIs


mobile test server
/api/monitoring
{
  "api_env": "test",
  "connected_db": "test",
  "pipeline": "mobile"
}
Match all keys/values// Do nothing


else (or timeout)alert_count + 1Chengyong, Vivian
mobile prod server
/api/monitoring
{
  "api_env": "production",
  "connected_db": "production",
  "pipeline": "mobile"
}
Match all keys/values// Do nothing


else (or timeout)alert_count + 1Chengyong, Vivian






Backend dashboard APIs


dashboard test server
/env
{
  "api_env": "TEST",
  "connected_db": "TEST",
  "pipeline": "dashboard"
}
Match all keys/values// Do nothing


else (or timeout)alert_count + 1Weihang
dashboard prod server
/env
{
  "api_env": "PROD",
  "connected_db": "PROD",
  "pipeline": "dashboard"
}
Match all keys/values// Do nothing


else (or timeout)alert_count + 1Weihang






Frontend dashboardfrontend prod server
(Note: There is not test
 server for frontend dashboard.
Developers' local laptop is
used as test server.)

Source code of any page
e.g. /status 
In source code, inside
<div id="status" hidden="">...</div>
{
  "env":"prod",
  "connected_api":"prod",
  "pipeline":"dashboard",
  "api_base_url":"https://dashboard-prod.diaper.cf:5000"
}
Match all keys/values// Do nothing



else (or timeout)alert_count + 1Emanuele, Audrey

Step 3: Determine status & Send emails

if (alert_count > 0) {
	put all error info (expected v actual) in an email
	send the email to DNS-DIAPER-ALERT-L@cornell.edu
}
else if (current time between 08:50am and 09:10am) {
	send an email "All status OK" to DNS-DIAPER-REPORT-L@cornell.edu
}
// else { do nothing; will run again at next hour; }

Public function to send email

Lambda function name: sendEmail 

  • No labels