If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
We previously showed that an ultrasound-guided i.v. catheter insertion (USGIV) simulation-based mastery learning (SBML) curriculum improves the simulated USGIV skills of paediatric anaesthesiologists. It remains unclear if improvements in simulated USGIV skills translate to improved patient care.
Methods
A cohort study was conducted from August 2018 to August 2020 to evaluate paediatric anaesthesiologists' USGIV performance in the operating theatre before and after they participated in the USGIV SBML curriculum. Paediatric anaesthesiologists' use of ultrasound for successful i.v. insertion and first-attempt i.v. insertion success rate with ultrasound were compared before and after training.
Results
Twenty-nine paediatric anaesthesiologists completed training. Unadjusted analysis showed a significant increase in the percentage of i.v. catheters inserted with ultrasound for successful i.v. catheter insertion (9.5–14.5%; P<0.001) and first i.v. catheter insertion attempt success with ultrasound (5.5–8.9%; P<0.001) from before to after training. Multivariable regression analysis showed higher odds of ultrasound use for a successful i.v. catheter attempt (1.79; 95% confidence interval [CI]: 1.11–2.90; P=0.018) and first-attempt success with ultrasound (4.11; 95% CI: 2.02–8.37; P<0.001) after training.
Conclusions
After completing the USGIV SBML curriculum, paediatric anaesthesiologists increased their ultrasound use for successful i.v. catheter insertion and first-attempt success rate with ultrasound for patients in the operating theatre.
Ultrasound-guided i.v. catheter (USGIV) insertion increases first-attempt success rates and decreases time to cannulation in paediatric patients with known difficult i.v. access.
Randomized controlled trial of ultrasound-guided peripheral intravenous catheter placement versus traditional techniques in difficult-access pediatric patients.
Use of ultrasound guidance for central venous catheter placement: survey from the American board of emergency medicine longitudinal study of emergency physicians.
Use of ultrasound guidance for central venous catheter placement: survey from the American board of emergency medicine longitudinal study of emergency physicians.
SBML is an intense form of competency-based learning, in which learners use deliberate practice on a simulator and are expected to reach a high skills standard (minimum passing standard [MPS])
Those who are unable to meet the MPS participate in further deliberate practice until they can be reassessed and reach the standard. Our SBML curriculum significantly improved paediatric anaesthesiologists' USGIV skills in a simulated environment.
However, it remains unknown whether these improved skills are also associated with improvements in patient care.
The aim of the current study was to determine if paediatric anaesthesiologists' participation in the USGIV SBML curriculum improved i.v. catheter insertion outcomes in the operating theatre setting. We hypothesised that after SBML, paediatric anaesthesiologists would have a higher use of ultrasound for successful i.v. catheter insertion and a higher rate of first-attempt success with ultrasound compared with before the training intervention.
We performed a quasi-experimental study to test the association between anaesthesiologist participation in a USGIV SBML curriculum and i.v. catheter insertion outcomes at a tertiary care paediatric hospital in Chicago between August 2018 and August 2020.
We evaluated the change in anaesthesiologists' ultrasound use for successful i.v. catheter insertion and first-attempt success rate with ultrasound in the operating theatre before and after completion of SBML. The Ann & Robert H. Lurie Children's Hospital of Chicago's Institutional Review Board approved this study, and the requirement for written informed consent was waived for both the USGIV SBML training of paediatric anaesthesiologists and electronic medical record (EMR) review of patient records.
Intervention: SBML
The USGIV SBML curriculum was delivered between August 2019 and January 2020 inclusive. The curriculum has been described previously.
Briefly, paediatric anaesthesiologists first underwent a USGIV skills pre-intervention test on a paediatric venipuncture pad (Simulab, Seattle, WA, USA) using 22G 45 mm B. Braun Introcan Safety® IV catheters (Bethlehem, PA, USA) and high-frequency ultrasound probes (L25xp and HSL25xp; SonoSite X-PORTE, Bothell, WA, USA). The 25-item skills checklist was developed by an expert panel of eight paediatric anaesthesiologists, two interventional radiologists, and two interventional radiology nurse practitioners using the modified Delphi technique. The MPS was set by the same expert panel using the Mastery Angoff technique and was determined to be 23 out of 25 items correct.
Subsequently, they participated in a 2 h session consisting of a procedural skills video, a live demonstration of USGIV insertion, and deliberate practice on live models for ultrasound scanning and a simulator for ultrasound-guided dynamic needle tip positioning (Fig. 1). Finally, participants were required to meet or exceed an MPS using the same skills in a post-intervention test on the simulator. Participants who did not meet the MPS engaged in more deliberate practice until they could be retested and meet this standard. We recorded participants' characteristics and clinical experience, including age, sex, years in practice, and number of USGIV inserted in their career and in the past 6 months.
Fig 1Demonstration of paediatric venipuncture pad for ultrasound-guided i.v. catheter insertion shown in (a) transverse orientation and (b) longitudinal orientation.
All paediatric anaesthesiologists were eligible for inclusion in the study. As reported previously, no anaesthesiologist had received any formal training in USGIV before the intervention, but some concepts and skills may have been acquired informally through vicarious learning.
By January 2020, all anaesthesiologists had completed the SBML USGIV training.
Data source
To evaluate the impact of the training on patient care, we constructed a data set of successful i.v. catheter insertions that occurred in the operating theatre during the study period using data queried from the EMR (Epic, Verona, WI, USA). The unit of analysis was i.v. catheter insertion. As such, patients could appear multiple times (i.e. multiple rows) in the data set if they had multiple i.v. catheter insertion sites on their body, because they had multiple procedures during the observation period, or both.
We also extracted data from the EMR on patient-level information, including age; sex; weight; BMI; ASA status; race/ethnicity; history of cardiac, renal disease, and prematurity; and inpatient or outpatient status. A history of one or more of renal disease, cardiac disease, and prematurity was abstracted from the International Classification of Diseases (ICD)-9 and ICD-10 Clinical Modification (CM) billing codes associated with each patient's medical record. ICD-9-CM and ICD-10-CM codes associated with renal disease were 580–589, 753, N00–08, P96, Q61–64, N10–16, and N17–19; codes associated with congenital heart disease were 745–747 and Q20–28; and codes associated with prematurity were 765, P05, and P07.
Outcome
Only information on the successful i.v. catheter insertion attempt was included in the EMR. For each i.v. catheter insertion, the following data were available: who inserted the catheter, number of attempts, and ultrasound use for the successful insertion. We defined two USGIV binary outcomes: (i) successful i.v. catheter insertion with ultrasound use and (ii) successful i.v. catheter insertion at first attempt with ultrasound. For example, if landmark techniques failed twice and ultrasound was used successfully on the third attempt, this would be categorised as a successful i.v. catheter insertion with ultrasound but not as a first-attempt success with ultrasound. Because of the limitations of the data, we were unable to assess who performed prior (failed) attempts and whether ultrasound was used for these attempts. Unsuccessful i.v. catheter insertion attempts could have been performed by hospital staff, trainees, or untrained or SBML-trained anaesthesiologists.
Exposure
The exposure of interest was the USGIV SBML training status of the anaesthesiologist at the time of successful i.v. catheter insertion (SBML-trained vs traditionally trained). SBML-trained anaesthesiologists were those that successfully completed the USGIV SBML training. A successful insertion was classified as performed by an SBML-trained anaesthesiologist if the successful insertion occurred after the date when the anaesthesiologist completed training. If the same anaesthesiologist completed the successful i.v. catheter insertion before they participated in SBML training, the insertion was classified as performed by a traditionally trained anaesthesiologist. At the study institution, there are no set protocols that determine which patients need ultrasound guidance for i.v. catheter insertion or which clinician performs i.v. access. Ultrasound use is at the discretion of the anaesthesiologist.
Statistical analysis
We tested the association between training completion and USGIV outcomes by comparing the use of ultrasound for successful i.v. catheter insertion and first-attempt success with ultrasound
for i.v. catheter insertions performed by paediatric anaesthesiologists before (‘traditionally trained’) and after USGIV SBML training.
To assess the association of SBML training on i.v. catheter insertion outcomes, we performed an unadjusted analysis comparing the number of attempts (categorised based on EMR documentation: 1, 2–3, 4–5, and 6+), ultrasound use for the successful attempt, first-attempt success, and patients requiring more than three attempts between anaesthesiologists who were traditionally and SBML trained using χ2 tests. In the adjusted analysis, we modelled the probability (log-odds) of the outcome using segmented regression with generalised estimating equations (GEEs). The GEE model uses a ‘logit’ link with an ‘exchangeable’ correlation structure for data clustered at the anaesthesiologist level. The underlying model using a generalised linear model expression is as follows:
where g(.) is the link function, Yij is the outcome for an anaesthesiologist i at time j, tj is the time j for the i.v. insertion, and Aij is the indicator for training status of the anaesthesiologist i. The betas are the population-level parameters with β0 as the average at time 0, β1 is the slope or change of the outcome over time had no training occurred, β2 is the change in the outcome attributable to training, and β3 is for the interaction between time and training status referring to any changes in the association of the training with the outcome over time. βX are the parameters and values for all other covariates adjusted for in the model.
The model assessed the impact of the training using a binary term for exposure (insertion before or after completion of training), a linear term of time in months (time zero is August 1, 2018), and an interaction between the exposure and time term. The interaction term allowed testing for any time-changing associations between the training and the outcome. The model also adjusted for selected patient characteristics believed to affect i.v. catheter insertion success based upon our previous work and literature search: age (categorical), sex (binary), weight (categorical), race and ethnicity (categorical), inpatient status (binary), and having comorbidities (binary for kidney disease, cardiac disease, and prematurity).
Randomized controlled trial of ultrasound-guided peripheral intravenous catheter placement versus traditional techniques in difficult-access pediatric patients.
Given the time and time–training interaction terms in the model, we calculated and plotted the odds of successful insertion with ultrasound use and first attempt with ultrasound at different time periods, had all i.v. catheter insertions been performed by SBML-trained anaesthesiologists to the odds that all i.v. catheter insertions been performed by traditionally trained anaesthesiologists. Analyses were conducted on complete data for all confounding variables. All analyses were implemented in R 4.1.0 (R Foundation for Statistical Computing, Vienna, Austria). GEE was done using geepack (https://cran.r-project.org/web/packages/geepack/geepack.pdf) and marginal probabilities using emmeans (https://cran.r-project.org/web/packages/emmeans/emmeans.pdf).
Sample size justification
Previous exploratory analyses of our EMRs have shown that there are roughly 1000 i.v. catheters inserted in the operating theatre every month, with anaesthesiologists inserting at least 35% of them. With 350 i.v. catheter insertions every month, an estimated 6300 i.v. catheters would be inserted in 18 months. Based on historic data before initiating the curriculum in 2018, we estimated that at least 5% of successful i.v. catheter insertions were performed with ultrasound. A χ2 power analysis showed that a total sample size of 5900 was needed to achieve a power of 0.90 (alpha 0.05) to discern an increase in ultrasound use for successful i.v. catheter insertion of 2% (from 5% to 7%), reflecting a 30% increase in ultrasound use.
Results
Thirty-six paediatric anaesthesiologists were available for inclusion in the study; seven were excluded because they served as proctors/instructors for the SBML curriculum. The remaining 29 anaesthesiologists completed the training. Three participants completed the SBML training as fellows and then were subsequently included as anaesthesiologists during the study observation period. Baseline characteristics of the anaesthesiologists at the time of USGIV SBML training are described in Table 1. The median anaesthesiologist age was 36 yr (inter-quartile range [IQR]: 33–42), 58.6% were female, and the median years of clinical experience was 9 (IQR: 5–13). The majority (79.3%) of paediatric anaesthesiologists had inserted more than 10 USGIVs in their career, whilst most (72.4%) had inserted fewer than 10 USGIVs in the past 6 months.
Table 1Paediatric anaesthesiologists' characteristics and experience with paediatric USGIV. IQR, inter-quartile range; USGIV, ultrasound-guided i.v. catheter insertion.
There were 8881 i.v. catheter insertions during the study period. Unadjusted patient care outcome associations with anaesthesiologist training status are shown in Table 2. In unadjusted analyses, anaesthesiologists showed significantly greater use of ultrasound for successful i.v. catheter insertion (9.8–14.1%; P<0.001) and a higher first-attempt success rate with ultrasound (5.1–8.5%; P<0.001) from before to after training.
Table 2Unadjusted clinical care outcomes of i.v. catheter insertions performed by traditionally trained (pre-intervention) and SBML-trained (post-intervention) paediatric anaesthesiologists. SBML, simulation-based mastery learning.
Multivariable analysis using GEE showed that the odds of using ultrasound for successful i.v. catheter insertion was significantly higher after the anaesthesiologist completed training (odds ratio [OR]: 1.79; 95% confidence interval [CI]: 1.11–2.90; P=0.018; Table 3). The estimated OR of the outcome between SBML-trained and traditionally trained anaesthesiologists for selected time points is shown in Figure 1a. If all insertions had been conducted by SBML-trained anaesthesiologists since August 2018, the odds of ultrasound used in successful attempt are significantly higher compared with the scenario where insertions were done by traditionally trained anaesthesiologists (Fig. 2a). Over time, however, the OR gradually approaches one (null), suggesting improving skill or increasing use of ultrasound amongst traditionally trained anaesthesiologists (in the no SBML training scenario).
Table 3Generalised estimating equation analysis comparing the likelihood of ultrasound use on successful i.v. catheter insertion attempts between SBML-trained and traditionally trained anaesthesiologists adjusted for patient characteristics and comorbidities (N=8881 i.v. catheter insertions). CI, confidence interval; SBML, simulation-based mastery learning. ∗Other includes all racial and ethnic categories less than 5% of total (Asian/Pacific Islander, multiple races, and prefer not to report).
Fig 2Change in odds ratios (dots) with 95% confidence intervals (dashed vertical lines) of SBML-trained vs traditionally trained anaesthesiologist i.v. catheter insertions over time for (a) ultrasound used in successful insertion attempt over time and (b) ultrasound used in first successful attempt over time.
Results were also similar for the odds of using ultrasound in first-attempt successful i.v. catheter insertions (Table 4). The odds of this outcome were significantly higher after completion of the training (OR: 4.11; 95% CI: 2.02–8.37; P<0.001). When accounting for training status, time, and training status–time interaction collectively, the OR of first-attempt ultrasound use over time between an SBML-trained vs traditionally trained scenario showed significantly higher odds of the outcome. However, the OR also approached one (null) over time, suggesting that odds of ultrasound used in the first successful attempt would have increased without SBML training (Fig. 2b).
Table 4Generalised estimating equation analysis comparing the likelihood of first-attempt success with ultrasound between SBML-trained and traditionally trained anaesthesiologists adjusted for patient characteristics and comorbidities (N=8881 i.v. catheter insertions). CI, confidence interval; SBML, simulation-based mastery learning; USGIV, ultrasound-guided i.v. catheter insertion. ∗Other includes all racial and ethnic categories less than 5% of total (Asian/Pacific Islander, multiple races, and prefer not to report).
We showed that training paediatric anaesthesiologists in USGIV using an SBML curriculum was associated with significant improvements in ultrasound usage for successful i.v. catheter insertion and the rate of first-attempt success with ultrasound. Our study adds to the literature, demonstrating that SBML results in improved patient care. To our knowledge, this is the first study to evaluate SBML with anaesthesiologists as participants. SBML-based training of anaesthesiologists has potential positive implications for maintenance of certification in anaesthesia and other continuing medical education activities.
Despite a long history of using simulation to improve patient safety, there are few studies in the anaesthesia literature that have examined the effect of simulation curricula on patient care outcomes. Bisgaard and colleagues
The effects of graduate competency-based education and mastery learning on patient care and return on investment: a narrative review of basic anesthetic procedures.
reviewed simulation studies on procedures also performed in anaesthesia, finding that the majority of simulation studies that reported patient care outcomes focused on central venous catheter insertion. These studies showed a decrease in complications and an increase in return on investment after simulation-based training.
Two studies evaluated the effects of simulation training with only deliberate practice but no MPS requirement. In the first study, Smith and colleagues
evaluated the learning curves of 12 anaesthesia trainees and found that 95% of fibreoptic intubations in patients were completed within a benchmark of 2.5 min after simulation training and deliberate practice. In the second study, Udani and colleagues
Comparative-effectiveness of simulation-based deliberate practice versus self-guided practice on resident anesthesiologists’ acquisition of ultrasound-guided regional anesthesia skills.
used deliberate practice to train anaesthesiology residents in subarachnoid block and found that the residents scored higher on simulated checklist items after training, but they found no difference in block success rate or completion time on real patients.
Our study aligns with previous evidence demonstrating that SBML is associated with improvements in patient care. Prior studies have shown improvements in patient outcomes after SBML for resident physicians in skills, such as thoracentesis, paracentesis, and central line insertion; for nurse training in USGIV; and for patient training on ventricular assist device self-care.
Performance of peripheral catheters inserted with ultrasound guidance versus landmark technique after a simulation-based mastery learning intervention.
However, to our knowledge, this is the first study showing that SBML for anaesthesiologists was associated with improved patient care.
The positive effects of our USGIV SBML curriculum may be even higher if used at other institutions for two main reasons. First, every anaesthesiologist participating in the SBML intervention had prior experience performing USGIV, despite not receiving a formalised training curriculum. In fact, approximately 25% of the anaesthesiologists who participated in our study met or exceeded the MPS on the simulation pre-intervention test, indicating a high baseline level of skill before training began.
Second, all anaesthesiologists who participated in our study supervised paediatric anaesthesia fellows and certified registered nurse anaesthetists who had previously participated in the USGIV SBML curriculum.
Because the curriculum was developed over several years, the anaesthesiologists in our study may have learned vicariously from these other SBML-trained participants as part of supervising patient care. This may account for the findings seen in the OR of the outcome plots for selected time points (Fig. 1). Vicarious learning from others who have undergone SBML has previously been demonstrated for central venous catheter insertion.
Our study had several limitations. First, we used a quasi-experimental design without a ‘true’ control group, which makes it difficult to attribute improvements in clinical outcomes to our SBML intervention. However, we used both a time and time/training interaction variable in our regression modelling to account for temporal trends, and we demonstrated that SBML training status was still associated with significant increased odds in the use of ultrasound for successful i.v. catheter insertion and first-attempt success rate with ultrasound. The GEE modelling indicated that the highest odds of successful i.v. catheter insertion with ultrasound and first-attempt success with ultrasound were seen at the time of training and decreased thereafter, indicating a decrease in association of training with these outcomes. As there were no changes to institutional i.v. catheter insertion protocol, availability of ultrasound machines, or USGIV training, we believe that both outcomes increased over time because of an increase in skills from vicarious learning. Second, the study was conducted at a tertiary care children's hospital, which may limit the generalisability of our findings to other settings. Third, we did not measure skills decay; it is possible that skills decayed over time, as there was no further training for the anaesthesiologists after completion of the USGIV SBML curriculum. However, other studies have shown that SBML-based training can slow skills decay.
Finally, only the method used for the successful i.v. catheter insertion attempt was documented in our EMR. We were unable to determine whether ultrasound or landmark techniques were used for previous insertion attempts or if other personnel made the previous unsuccessful insertion attempts. Therefore, we had to use the overall number of attempts (whether US was used or not) as the denominator to calculate first-attempt success with ultrasound rates. We believe that this limitation explains why our first-attempt success rates for USGIV insertions were much lower than the rates of 41–86% reported in the literature.
However, our overall first-attempt success rate of 70% (Table 3) is consistent with previously published estimates. We believe our data reflect real-life practice because many anaesthesiologists will attempt landmark techniques first (because it is faster and children are under anaesthesia) and then move to ultrasound when unsuccessful, or they will start with ultrasound only when patients are suspected to have difficult i.v. access. Because the same denominator was used for i.v. catheter insertion attempts before and after SBML-based training, we believe that these increases in first-attempt success are valid and reflect the actual care patients received in this setting.
In conclusion, anaesthesiologist participation in the USGIV SBML curriculum was associated with improvements in the use of ultrasound for successful i.v. catheter insertion and first-attempt success rate with ultrasound. Further studies are needed to evaluate the impact of USGIV SBML using concurrent controls. Our next steps are to continue to implement our SBML curriculum in anaesthesiology and study whether improvements in patient care are sustained over time. We also plan to expand our curriculum to paediatric nursing on the inpatient wards.
Authors' contributions
Study conception: HAB, MT, JHB.
Study design: HAB, MT, JHB.
Data acquisition: JH.
Data analysis: HAB, AR, JH, JF, JHB.
Data interpretation: HAB, AR, MT, JF, JHB, MP.
Drafting of article: HAB.
Revising of article: AR, MT, MP, JH, JF, JHB.
All authors approved the final version of the article and are accountable for all aspects of this work.
Acknowledgements
The authors would like to acknowledge the anaesthesiologists at the Ann & Robert H. Lurie Children's Hospital of Chicago for participating in the simulation-based mastery learning curriculum.
Declarations of interest
The authors declare that they have no conflicts of interest.
Funding
Society for Education in Anesthesia research grant in 2019.
References
Reigart J.R.
Chamberlain K.H.
Eldridge D.
et al.
Peripheral intravenous access in pediatric inpatients.
Randomized controlled trial of ultrasound-guided peripheral intravenous catheter placement versus traditional techniques in difficult-access pediatric patients.
Use of ultrasound guidance for central venous catheter placement: survey from the American board of emergency medicine longitudinal study of emergency physicians.
The effects of graduate competency-based education and mastery learning on patient care and return on investment: a narrative review of basic anesthetic procedures.
Comparative-effectiveness of simulation-based deliberate practice versus self-guided practice on resident anesthesiologists’ acquisition of ultrasound-guided regional anesthesia skills.
Performance of peripheral catheters inserted with ultrasound guidance versus landmark technique after a simulation-based mastery learning intervention.
00100-9&id=gr1.jpg){kind=link}
00100-9&id=gr2.jpg){kind=link}