Paused Snitches are alerting
Incident Report for Dead Man's Snitch
Resolved
We're really sorry for the trouble and confusion this caused. The issue itself was related to migrating to a new alerting backend that's been in the works for quite a while. We had hoped for it to be seamless (and it almost was) but we failed to double check the state of Snitches that were waiting for their first check-in. We plan to take another pass at our verification process to make sure we haven't missed anything else.
Posted Dec 15, 2021 - 20:15 EST
Monitoring
Pending Snitches should all be fixed and we're seeing alert metrics return to normal. We're waiting until the top of the hour after our basic hourly Snitches alert before we consider the incident closed.
Posted Dec 15, 2021 - 19:39 EST
Identified
We've identified that some Snitches that were pending (i.e. have never checked-in to Dead Man's Snitch) were alerting. We've reverted those Snitches which were pending and hadn't been paused to their original state. This should stop future emails for those Snitches.
Posted Dec 15, 2021 - 19:20 EST
Investigating
We're investigating an issue related to a backend migration that caused some previously paused Snitches to become unpaused and start alerting again. We're working to repause those Snitches.
Posted Dec 15, 2021 - 18:53 EST