Background - Traditional audit methods are limited in their ability to provide short
feedback loop to identify underperforming surgical units in time for them to respond
appropriately. Moreover, case mix and other confounding factors limit the usefulness
of crude mortality figures. More advanced industrial methods such as cumulative
sum method (CUSUM) have therefore become of interest to surgeons.
Hypothesis – Continuous monitoring of outcome in aortic aneurysm surgery using
CUSUM technique (with optimisation using fractional polynomial mathematical
models) can be applied, and do provide significant higher and more accurate
detection rate of outliers when compared to traditional audit methods.
Methods – Using anonymised records from National Vascular Database (NVD),
three monitoring systems were applied in real time: Cumulative mortality (reflecting
traditional audit process), funnel plot, and CUSUM (SPRT). VBOHM risk score was
used to adjust for case-mix. Outliers were detected using different detection levels
(h) and odds ratios (OR) with variable mortality rates (p). Performance of the three
monitoring models was compared using direct alarm signals, sensitivity and
specificity analysis, receiver operating curve (ROC), and average run length (ARL).
Choosing control limits to maximise efficiency was approximated using direct
simulation, Markov chain, and fractional polynomial techniques.
Results –In-hospital mortality following elective Abdominal Aortic Aneurysm (AAA)
repair between 1995 and 2011 in 140 centers were monitored. Compared to
traditional audit methods, CUSUM has significant sensitivity to the outlier status of
each vascular unit, with average number of CUSUM alerts of 0.89 when there is no
outlier status, rising up to 23 alerts when there is an outlier status. Maximising the
sensitivity and specificity of detecting outliers by CUSUM technique (also called incontrol
ARL) while minimising false alarms (also called out-of-control ARL) was
achieved using different range of values for control limits (h) and odds ratios (OR).
For best CUSUM performance, values of OR=3, p=3, and h=1.25 has been shown to
detect outliers correctly in 53% of case, and reject correctly in 59% of cases. This
corresponds with CUSUM sensitivity of 80% and specificity of 80%. CUSUM has a
positive predictive value of 78% and negative predictive values of 82%, with
accuracy reaching 80%. Fractional polynomial technique and CUSUM simulation
behavior were shown to correlate well (R > 0.88,
|Date of Award||Sep 2016|
Developing better methods for monitoring performance in vascular surgery
Jibawi, A. (Author). Sep 2016
Student thesis: Doctoral Thesis