System has been stable for several hours. We have a plan in place to automate future occurances that we will be deploying soon.
Posted May 04, 2018 - 10:11 EDT
Monitoring
As of 10:44 UTC we've caught up on the back log of check-ins and the system is back to being stable. We're still investigating the root cause and will be preparing a post mortem once we know more.
At this point we're keeping a close eye on the system to make sure this doesn't impact anyone should it start happening again.
Posted May 04, 2018 - 07:30 EDT
Investigating
We have sent out several false alerts due to delays in check-in processing. We're still investigating the root cause but have mitigated the issue for now.