A Comparison of Three Metrics to Identify Healthcare Associated Infections

Background: Identifying healthcare associated infections (HAIs) is an important but time-consuming task.  Although surveillance by infection control personnel using National Healthcare Safety Network (NHSN) criteria remains the gold standard, automated surrogate measures are gaining attention for their potential to streamline intra-facility data collection and standardize inter-facility comparisons.

Objective: To compare 3 methods of HAI identification: traditional NHSN surveillance, a commercial data-mining approach, and International Classification of Diseases 9^th Revision (ICD-9) coding.  

Methods: We monitored catheter-associated bloodstream infections (CA-BSIs) for 3-month periods in 2009 in each of 6 intensive care units (ICUs) of a large university hospital.  NHSN surveillance was completed by experienced infection control practitioners.  A Nosocomial Infection Marker report was generated by data-mining using the MedMined™ Data Mining Surveillance service (CareFusion Corporation).  ICD-9 coding was carried out by veteran billing personnel.  The reports were derived independently and then compared.   Cases classified differently by the 3 approaches were reviewed by 2 infectious diseases physicians to determine their appropriate designation as either true or false positive CA-BSIs.  We calculated the sensitivity and positive predictive value of each of the 3 methodologies.

Results: A total of 56 CA-BSI were identified in the 6 ICUs (bone marrow transplant 20, cardiac 3, medical 13, neurosurgical 6, surgical 11, high-risk neonatal 3) during 7,154 catheter-days.  The overall CA-BSI rate was 7.9 per 1000 catheter-days.  NHSN surveillance detected 46 cases, and there were no false-positives.  Data-mining identified 48 cases but also 44 false-positives.  ICD-9 coding documented 7 cases and 5 false-positives.  Overall sensitivities of the 3 metrics were: NHSN surveillance 0.82, data-mining 0.86, and ICD-9 coding 0.13.  Positive predictive values of the 3 measures were: NHSN surveillance 1.0, data-mining 0.52, and ICD-9 coding 0.58.

Conclusions: While comparable in regards to sensitivity, there was an important difference in the positive predictive value of traditional NHSN surveillance versus data-mining.  Any consideration of implementing an automated surveillance system must take into account the expertise and time necessary to review cases and eliminate false-positives.  Otherwise, data-mining was too inaccurate for use in providing feedback to ICUs or for public reporting.  The ICD-9 coding scheme lacked both sensitivity and predictive power as a metric for identifying CA-BSIs.  The NHSN approach has imperfect discriminatory power and inter-rater reliability, but the adoption of either surrogate metric does not enhance HAI reporting performance from an accuracy or efficiency standpoint.