Dear RamCT Community,
CSU has been working diligently with Blackboard to isolate and resolve rare instances of assessment failures in RamCT. A minor patch was released November 1st to enhance diagnostic messages written at assessment time. This patch was thoroughly tested on our development system, and then tested dozens of times once placed on the production system.
Later in the week, a new assessment failure was reported but the logging was insufficient to pinpoint the problem. Blackboard subsequently requested that CSU deploy a version of the same patch with additional logging enabled. This patch was applied during a maintenance window that was completed approximately 9:30 AM Saturday, 11/08/2008. Rigorous testing was not performed this time, as Blackboard assured our staff that it was the same patch configured with new logging parameters. Unfortunately, Blackboard had introduced a coding error which prevented the system from saving assessments.
A RamCT system administrator was called at home Sunday afternoon regarding assessment failures since the patch was installed. Within an hour, the patch was removed and normal service was restored.
As a result, several changes are being implemented:
- The RamCT College Coordinators and Application Administrators will develop a test regiment to be performed before any patch is applied. Testing will be conducted first in the development environment, and then again once applied to the production environment.
- An escalation procedure is being developed and made available to the College RamCT Coordinators, and to key faculty they appoint within their college.
- Enhanced monitoring tools are being developed to detect sudden increases in the number of items emailed to RamCT support. Once a threshold is reached, on-call staff will be alerted of a potential problem.
We are confident that the above procedural changes will further enhance the stability of the RamCT system. After the service outage this past weekend, RamCT has been available 99.389% of the time during the Fall ’08 semester. Please accept our sincere apologies for the recent outage.
Scott Baily
Interim Director, ACNS
