Everything is looking good and has been stable the last hour 🎉
Posted Nov 25, 2016 - 12:39 EST
Update
We've finished processing all check-ins that were left on our failover queue. Everything is looking to be back to normal. We'll continue to monitor the situation in case of further issues.
Posted Nov 25, 2016 - 11:26 EST
Monitoring
The issues with our primary queue server appear to be fixed and we've reenabled it for check-in processing. We are handling all new check-ins as they arrive but are still working off the backlog of check-ins from our backup queue service.
Posted Nov 25, 2016 - 11:14 EST
Identified
Our primary queue server is currently having CPU issues causing check-ins to take longer to both process and respond to clients. We're working with our service provider to address the issue. We've failed over check-in processing to a secondary service but are having issues keeping up with message rates. We expect check-ins processing to be delayed until our main queue server is back at full capacity.