Use of Electronic Reporting and Annotations by Infection Preventionists (IPs) to Describe Differences in the Application of Surveillance Criteria

Background: Although the National Healthcare Safety Network (NHSN) provides standardized surveillance definitions, subjective judgment is required in their application.  We previously reported an inter-rater kappa of 0.42 amongst IPs classifying bloodstream infections, but information on how IPs apply surveillance criteria in practice is limited.

Objective: Describe how IPs with different levels of training classify episodes of central line associated bloodstream infection (CLABSI).

Methods: Electronic health records (EHRs) for IP review were abstracted from actual EHRs at a 121 bed Veterans Affairs (VA) facility.  114 patients hospitalized between 2000 and 2005 with positive blood cultures were randomly sampled.  Data including microbiology, bed movement, caregiver notes, antimicrobials, and chest radiograph reports were provided in familiar, web-based EHR format. 18 IPs were recruited from 138 VA facilities and randomly placed into one of 4 groups to review identical records.  IPs reported whether they were an NHSN participant, certified in infection control (CIC), or without formal training.  IPs submitted surveillance decisions electronically: use of data and time was tracked. An objective measure to identify CLABSI based on microbiology, time since admission, and presence of central venous catheter was applied to all 114 records.

Results: IPs were from facilities representative of the national VA system and included 4 NHSN participants, 9 CIC certified, and 5 with no formal training.  Each of 114 records was reviewed by 4 or 5 IPs, for a total of 512 reviews. While the objective measure identified CLASBI in 39% of 114 records, IPs reviewing the same records all agreed to CLABSI absence in 55 (48%) and presence in only 9 (8%), with disagreement in 50 records (44%). Aggregated reporting decisions stratified by IP level of training were similar in most measures as shown in the table below, but variation within these groups is notable.  Disagreement often occurred with violations of NHSN criteria and when infections were classified as secondary.  Although at least 1 IP noted secondary bacteremias in 103 of 114 records, in only 14 of these 103 did a majority of IPs identify a secondary source.           

Conclusions: Disagreement in CLABSI reporting occurred in nearly half of records reviewed by IPs.  Overall, there was a 2 fold difference reporting CLABSI amongst IPs regardless of training.  Errors due to violations with explicit NHSN criteria were found even in NHSN participants, and led to CLABSI under reporting.  Differences in classifying secondary infections were notable.  Use of more explicit automated objective criteria would over report CLABSI rates, but would improve comparability across facilities.