Alerts and emails are once again being sent in a timely manor. We've identified the root cause as a job which had a missing timeout which was causing issues due to an upstream connection issue. This highlighted that the monitoring we have in place for our worker system needs to be reviewed as we didn't receive an alert that there was an issue.
Posted Jan 28, 2025 - 10:39 EST
Monitoring
We've deployed a fix for the issue and our worker system has worked through all of the queued jobs. We're monitoring to make sure the system is stable and will be reviewing steps to avoid this in the future.
Posted Jan 28, 2025 - 09:57 EST
Update
We believe we've identified the issue which has been causing one job (sending iOS push notifications) to block our queue. We're working to deploy a fix now.
Posted Jan 28, 2025 - 09:48 EST
Identified
The system we use to send alerts and emails is currently backlogged due an upstream service issue. We're
Posted Jan 28, 2025 - 09:39 EST
This incident affected: Push Notification Alerts (iOS Push Notification Alerts) and Email Alerts.