Interrater Reliability in Applying National Healthcare Safety Network Definitions

Background:

In 2010 The National Healthcare Safety Network (NHSN) celebrated its 5^th anniversary.  Standard application of its definitions assures consistency in surveillance for healthcare-associated infections (HAIs), increasingly used by states for public reporting.  Clinical definitions utilized for treatment of patients’ infections differ from NHSN’s in the determination of infections.

Objective:

The objective was to ascertain interrater reliability of the application of NHSN surveillance definitions of HAIs.

Methods:

This five-hospital system’s 11 infection preventionists (IPs) and medical director took quizzes from The American Journal of Infection Control (AJIC) written by experts from NHSN.  The results of the online quizzes were reported anonymously to the test-taker as well as confidentially to the director of infection prevention and control for purposes of determining interrater reliability. 

Results:

The same question in the first quiz (#2) was answered incorrectly by all who missed getting all answers correct.  It involved a patient scenario with cultures of unrelated organisms, a central line-associated bloodstream infection (CLABSI) with Enterococcus faecalis and an intra-abdominal infection (IAB) with Escherichia coli. The persons who answered incorrectly did not feel that both HAIs should be reported, which prior to this quiz would have resulted in underreporting.  In the second quiz, all but one person got all questions correct, so interrater reliability increased from quiz #1 to quiz #2.  The third quiz was recently sent to participants.

Code of IP or  medical director	Quiz#1                                                (4 questions about CLABSI and IAB Infection)	Quiz #2                                          (3 questions about CLABSI and Catheter-associated Urinary Tract Infection)
A	4/4	3/3
B	4/4	3/3
C	4/4	3/3
D	4/4	3/3
E	3/4	3/3
F	4/4	3/3
G	3/4	3/3
H	3/4	2/3
I	4/4	3/3
J	3/4	3/3
K	3/4	3/3
L	4/4	3/3
# Correct/Total       (% Correct)	43/48 (90%)	35/36 (97%)

Conclusions: Utilizing NHSN quizzes allowed the person to get immediate feedback on NHSN definitions anonymously.  Sharing results with the director enabled determination of interrater reliability between the system’s 11 IPs responsible for mandatory reporting of HAIs in Pennsylvania to NHSN.  Including the medical director assisted in understanding the difference between surveillance and clinical definitions.  Since public reporting of all HAIs is by hospital name in PA, interrater reliability within the system and between hospitals in the state is vital.  State reviewers will be doing audits of hospitals with unexpectedly high or low HAI Standardized Infection Ratios (SIRs), so this exercise will improve our accuracy.  This system’s IPs and medical director will continue to take these quizzes as they are released in AJIC.  As shown in the first two quiz results, improvement occurred from quiz one to quiz.  A high level of interrater reliability is vital to allow meaningful hospital comparisons of SIRs of HAIs for public reporting.