CA instances intermittently experiencing "400 Bad Request" errors when accessing Admin Console

‌

Introduction:

On June 18, 2024 between 10:48 MST - 11:48 MST, a subset of our Canadian customers experienced “400 Bad Request” errors when performing any action requiring a database connection. 

‌

Issue Summary:

Customers affected during the outage were intermittently unable to process user login requests, or other core functionality.

‌

Resolution: 

Engineering teams stabilized the environment, determined the cause and deployed a fix to address an unexpected host state.

‌

Root Cause: 

A host entered an unexpected state after handling an unusually large request.

‌

Solution and Mitigation:

The system was stabilized by replacing the bad host. Monitors were put in place to detect this issue, and the appropriate teams are working on improving request handling.

‌

Conclusion:

We recognize this had an impact for our Canadian customers during normal working hours. We are committed to enhancing our architecture to prevent smaller services being able to impact overall service availability. We thank you for your patience as we worked through this service disruption.