RESOLVED: Some users may have been encountering 500-page errors in Canvas
Click here to show/hide earlier details
UPDATE: 27 Nov 2024 - 10:43am
Canvas have confirmed that this issue occurred from 17:18 to 17:38 on Friday 22 November 2024. They posted the following details on this issue:
Details
On November 22nd, 2024, at 10:15am MST (17:15 GMT), the Canvas Operations team made a change to create new Domain Name System(DNS) records for domains that were assumed to be no longer in use. However, the domains were still active, and the new DNS records were quickly picked up by services that required them, including critical infrastructure needed by Canvas application servers to interact with databases. This caused a surge in HTTP 500 errors across the platform.
At 10:16am MST (17:16 GMT), automated alerts notified the team of the increased errors. Upon investigation, it was confirmed that the DNS changes were responsible for the issues. At 10:18 (17:18) users were completely unable to access Canvas. By 10:33am MST (17:33 GMT), the team reverted the DNS changes, and Canvas functionality was fully restored shortly thereafter.
Mitigation
Going forward we are updating our internal teams processes for when these types of DNS changes are made. We will first verify with our tooling the relevance of the domain being changed and any impact the changes will have within Canvas.
Conclusion
We understand the importance of Canvas functioning properly as needed and we are working diligently to ensure that this behavior is not an issue moving forward.
to receive email alerts of updates.