Performance of Predictive Models for Identifying MRSA Carriage on Admission

Background: Predictive models to identify unknown MRSA carriage on admission may optimize targeted MRSA screening. However, common approaches to model selection can result in overconfident estimates and poor predictive performance.

Objective: To compare the performance of various models to predict unknown MRSA carriage on admission to surgical wards in 4 European hospitals.

Methods: A prospective cohort study was conducted in 13 surgical wards with universal MRSA screening. Multivariate logistic regression models were used to predict probabilities of unknown MRSA carriage on admission. Approaches used were stepwise backward elimination (“conventional” model), Bayesian Model Averaging (BMA) which accounts for uncertainty in model choice, and a model including only variables selected at least half the time when applying these approaches to repeated random sub-samples of 50% of the cohort-data (“simple” model). To assess model performance, cross-validation against data not used for model fitting was conducted.

Results: Of 2901 patients enrolled, 111 (3.8%) were newly identified MRSA carriers. Risk factors associated with MRSA carriage are shown in the Table. After repeated random sub-sampling cross-validation, the mean c-statistic was 0.64 for the “conventional” model, 0.65 for BMA, and 0.69 for the “simple” model. All estimates were in the range of limited prediction ability. When each hospital was used in turn as the validation set, the discrimination of these models to predict MRSA carriage was low with all c-statistic values for the “conventional” model being less than 0.6 except for 1 hospital in Barcelona (c-statistic = 0.76). The BMA approach performed better (c-statistics 0.58 to 0.80) and the “simple” model was most discriminatory (c-statistics 0.64 to 0.80).

Conclusions: The predictive performance of models used to identify unknown MRSA carriage on admission was limited, reflecting heterogeneity of risk factors between European hospitals. When comparing model selection approaches to develop MRSA risk indices, simpler models may perform better than more complex models.

Variable	"Conventional" logistic regression model			Bayesian Model Averaging				"Simple" logistic regression model
	Odds Ratio	95% CI	p value	Posterior Probability	Odds Ratio	95% CI	p value	Odds Ratio	95% CI	p value
Female sex				2.3	1.0	0.9-1.1	0.89
Age	1.02	1.004-1.03	0.01	44.8	1.0	0.99-1.03	0.42	1.02	1.004-1.03	0.01
Diabetes				7.8	1.0	0.8-1.4	0.80
Chronic skin disease	3.0	1.5-5.8	0.002	49.3	1.7	0.5-5.3	0.37	2.9	1.5-5.6	0.002
Hospitalization (<1 year)	2.2	1.5-3.3	<0.0001	100	2.2	1.5-3.4	<0.0001	2.2	1.5-3.3	<0.0001
Nursing home resident	3.4	1.6–6.8	0.001	82.4	3.1	0.9-10.3	0.07	3.4	1.7-6.9	0.001
Skin wound / sore	2.7	1.7-4.4	<0.0001	100	2.9	1.7-4.7	<0.0001	2.8	1.7-4.6	<0.0001
Antibiotics (<6 months)				2.6	1.0	0.9-1.2	0.89
Urinary catheter	4.5	2.0-10.3	0.018	72.4	2.9	0.7-12.8	0.15
Tracheostomy				3.1	1.1	0.6-2.0	0.90