Resolved -
At 12:49:15UTC the process responsible for periodically checking Snitch histories was shutdown during a deploy but was not restarted as part of the deploy process. Around 20:10UTC we were notified by a customer that a Snitch wasn't failing when it was expected to be and discovered the process was offline as part of our investigation. At 20:22UTC we restarted the process which then kicked off checks for all potentially missing Snitches.
Unfortunately we did not receive an alert that the process was offline because, as we should have known, the alert would only have occurred if this process had been running. We're discussing fixes to ensure this kind of error doesn't happen again.
Apr 16, 08:30 EDT
Resolved -
Between 9:15UTC and 11:10UTC a worked process became wedged which caused roughly 12% of alerts during that time to be delayed and the dashboard to be outdated. We've resolved the immediate issue, worked through pending alerts, and we are looking into the cause of the lock up.
Apr 15, 05:30 EDT