Introduction:
Customers experienced issues accessing the PrinterLogic platform (US region).
Issue Summary:
On February 9th, between 7:48 AM MST and 8:49 AM MST, a segment of our US instances were not accessible. From 8:17 AM MST until 8:57 AM MST, badge release was not available in the US region.
Impact: Major outage - Customer instances were not accessible.
Affected Regions: printercloud.com - US region
Root Cause:
On February 9th, a surge in traffic exposed a DNS misconfiguration, overwhelming select endpoints. Internal Service communication faltered until a rollback had been completed.
Solution and Mitigation:
Engineering teams received automated alerts of the situation and began investigating.
Once identified, DNS configurations were promptly rolled back and propagated across the infrastructure. Instance availability improved between 8:17 AM MST and 8:49 AM MST once propagation completed. Following the resolution of instance availability, a service adjustment was necessary to restore badge release functionality.
The configuration rollback has resolved the Service disruption, and impacted Services are now operating as expected. Teams have reviewed and identified the DNS misconfigurations, with additional measures being implemented to ensure proper operation of the configurations within internal testing environments before rolling them out to customer environments.
Conclusion:
This issue has been resolved.