PrinterCloud US - some instances are loading slowly / experiencing delays / receiving 400 errors.

Incident Report for PrinterCloud

Postmortem

Introduction:

On January 3, 2023, A subset of customers with instances hosted in the US were intermittently unable to utilize PrinterLogic SaaS.

Issue Summary:

Impact: A subset of customers were unable to utilize PrinterLogic SaaS between 01:12 PM MT and 6:14 PM MT.

Affected Regions: US

Root Cause:

Our investigation revealed a high number of concurrent connections on the database servers.

Solution and Mitigation:

The Engineering and DevOps team were able to mitigate the connection issues and stabilize the environment, engineering teams are working to optimze recovery efforts to reduce the time to recovery for the affected services.

Conclusion:

We have restored all impacted services. We continue to mitigate any future occurrence.

Posted Jan 05, 2024 - 20:03 MST

Resolved

This incident has been resolved, a postmortem will be posted in the next 48 hours.
Posted Jan 03, 2024 - 20:28 MST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jan 03, 2024 - 18:37 MST

Update

We are continuing to investigate this issue.
Posted Jan 03, 2024 - 16:27 MST

Update

We are continuing to investigate this issue.
Posted Jan 03, 2024 - 14:14 MST

Investigating

We are currently investigating this issue.
Posted Jan 03, 2024 - 13:20 MST
This incident affected: PrinterLogic | SaaS US (PrinterLogic | SaaS, MSP Portal - US).