↑ Top

UNC Canvas issue Dec. 6: What happened and what we’ve changed to ensure it does not reoccur

On December 6, 2023, the data that Canvas uses from ConnectCarolina was improperly generated. Canvas finished processing this faulty data shortly before 2:00 am, resulting in ill-timed course access issues for faculty and students intending to use Canvas during the next several hours. Specifically, all Fall 2023 courses were temporarily concluded and all Spring 2024 courses were temporarily deleted from our UNC Canvas instance. FA23 course sites were visible but not editable, whereas SP24 course sites were not visible at all. While these issues were temporary and did not result in irrecoverable data loss, they were acutely felt and demanded rapid correction and thorough post-incident analysis.

Issues were prioritized and corrected in the following order: ConnectCarolina FA23 enrollments by 9:30 am, missing SP24 courses by 2:20 pm, and missing manual enrollments for missing SP24 courses by 2:30 pm the next day. The ITS Ed Tech team worked as quickly and carefully as was possible while root causes were being investigated in collaboration with Instructure Canvas technical support. This analysis continued until the afternoon of December 12.

Data from ConnectCarolina can be loaded into Canvas using a few different methods. The original method used at the time of the incident quickly processes changes in data by only updating records that have changed since the last update. This method allows institutions to set a threshold that prevents significantly changed files from being processed. ITS Ed Tech’s reading of Canvas’ documentation led us to believe that the faulty file exceeded this threshold and should not have been processed. After several days of analysis, Instructure Canvas technical support found that the threshold had not been exceeded, but that Instructure’s calculation of the threshold was more complex than documented. The new method adopted after the incident no longer uses thresholds but instead processes enrollments in separate streams hourly to prevent corruption.

This issue will not recur because we understand how it happened and instituted preventative measures. Data corruption is not an unexpected problem, but the safeguards available to prevent further issues were not well-documented and therefore not sufficiently understood.