The Fire Department’s decades-old EMS computer aided dispatch system (CAD), which began experiencing problems on Monday including several unplanned intermittent outages that prompted switching to a back-up system, has been fully stabilized as of this morning.
Technicians determined the problem was caused by corrupted “disc arrays” – hardware storage components in the CAD system – and the defective equipment has been replaced. Beginning last night, technicians gradually restarted and re-integrated the two primary servers that run CAD with the back-up server on which the system has been operating since Monday. With three servers integrated and in full operation, the EMS CAD system once again has appropriate levels of redundancy in place over multiple sites.
On Monday, and briefly yesterday when diagnostic work was being done, the system was down for a total of 90 minutes. During that time, all calls to the 911 system went through and were handled by operators and dispatchers. Of nearly 12,000 calls handled by EMS staff this week, 96 percent were successfully handled through the CAD system. The remainder were handled manually by operators using cards to assign and track incidents—a back-up process that has been in place and used when necessary for more than 2 decades.
The EMS computerized dispatch system will be continuously monitored over the next several days to ensure that it continues to perform satisfactorily.
The 30-year-old CAD system is scheduled for a complete replacement in 2015 as part of the city’s overhaul of the 911 system.