Resolved -
Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.
Jan 28, 10:39 EST
Monitoring -
We've deployed a fix for the issue and our worker system has worked through all of the queued jobs. We're monitoring to make sure the system is stable and will be reviewing steps to avoid this in the future.
Jan 28, 09:57 EST
Update -
We believe we've identified the issue which has been causing one job (sending iOS push notifications) to block our queue. We're working to deploy a fix now.
Jan 28, 09:48 EST
Identified -
The system we use to send alerts and emails is currently backlogged due an upstream service issue. We're
Jan 28, 09:39 EST