Abstract
Background
This study aimed to develop a test with validity evidence for abdominal diagnostic ultrasound with a pass/fail-standard to facilitate mastery learning.
Method
The simulator had 150 real-life patient abdominal scans of which 15 cases with 44 findings were selected, representing level 1 from The European Federation of Societies for Ultrasound in Medicine and Biology. Four groups of experience levels were constructed: Novices (medical students), trainees (first-year radiology residents), intermediates (third- to fourth-year radiology residents) and advanced (physicians with ultrasound fellowship). Participants were tested in a standardized setup and scored by two blinded reviewers prior to an item analysis.
Results
The item analysis excluded 14 diagnoses. Both internal consistency (Cronbach’s alpha 0.96) and inter-rater reliability (0.99) were good and there were statistically significant differences (p < 0.001) between all four groups, except the intermediate and advanced groups (p = 1.0). There was a statistically significant correlation between experience and test scores (Pearson’s r = 0.82, p < 0.001). The pass/fail-standard failed all novices (no false positives) and passed all advanced (no false negatives). All intermediate participants and six out of 14 trainees passed.
Conclusion
We developed a test for diagnostic abdominal ultrasound with solid validity evidence and a pass/fail-standard without any false-positive or false-negative scores.
Key Points
• Ultrasound training can benefit from competency-based education based on reliable tests.
• This simulation-based test can differentiate between competency levels of ultrasound examiners.
• This test is suitable for competency-based education, e.g. mastery learning.
• We provide a pass/fail standard without false-negative or false-positive scores.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Martínez-Ares D, Martín-Granizo Barrenechea I, Souto-Ruzo J, Yáñez López J, Pallarés Peral A, Vázquez-Iglesias JL (2005) The value of abdominal ultrasound in the diagnosis of colon cancer. Dig Organo Of Soc Esp Patol Dig 97:877–886
Torloni MR, Vedmedovska N, Merialdi M, Betrán AP, Allen T, González R et al (2009) Safety of ultrasonography in pregnancy: WHO systematic review of the literature and meta-analysis. Ultrasound Obstet Gynecol 33:599–608
Anders Ericsson K (2008) Deliberate Practice and Acquisition of Expert Performance: A General Overview. Acad Emerg Med 15:988–994
McGaghie WC (2015) Mastery learning: it is time for medical education to join the 21st century. Acad Med J 90:1438–1441
European Society of Radiology (ESR) (2013) Organisation and practice of radiological ultrasound in Europe: a survey by the ESR Working Group on Ultrasound. Insights Imaging 4:401–407
Garg M, Drolet BC, Tammaro D, Fischer SA (2014) Resident Duty Hours: A Survey of Internal Medicine Program Directors. J Gen Intern Med 29:1349–1354
McGaghie WC, Miller GE, Sajid AW, Telder TV (1978) Competency-based curriculum development on medical education: an introduction. Public Health Pap 68:11–91
Downing SM, Yudkowsky R (2009) Assessment in health professions education, 1st edn. Routledge, New York, p 108 and p 143
Ghaderi I, Manji F, Park YS, Juul D, Ott M, Harris I et al (2015) Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 261:251–262
Østergaard M, Ewertsen C, Konge L, Albrecht-Beste E, Bachmann Nielsen M (2016) Simulation-Based Abdominal Ultrasound Training – A Systematic Review. Ultraschall Med 37:253–261
Schallware (2016) Specifications of simulator. Schallware, Germany. Available via http://www.schallware.com/
European Society of Radiology (2015) Guidelines & recommendations - Appendix 5. EFSUMB. Available via http://www.efsumb.org/guidelines/guidelines01.asp
Thinggaard E, Bjerrum F, Strandbygaard J, Gögenur I, Konge L (2015) Validity of a cross-specialty test in basic laparoscopic techniques (TABLT). Br J Surg 102:1106–1113
Thomsen ASS, Kiilgaard JF, Kjaerbo H, la Cour M, Konge L (2015) Simulation-based certification for cataract surgery. Acta Ophthalmol 93:416–421
Dyre L, Nørgaard LN, Tabor A, Madsen ME, Sørensen JL, Ringsted C et al (2016) Collecting Validity Evidence for the Assessment of Mastery Learning in Simulation-Based Ultrasound Training. Ultraschall Med 37:386–392
Jacobsen ME, Andersen MJ, Hansen CO, Konge L (2015) Testing basic competency in knee arthroscopy using a virtual reality simulator: exploring validity and reliability. J Bone Joint Surg Am 97:775–781
Konge L, Clementsen PF, Ringsted C, Minddal V, Larsen KR, Annema JT (2015) Simulator training for endobronchial ultrasound: a randomised controlled trial. Eur Respir J 46:1140–1149
Konge L, Albrecht-Beste E, Nielsen MB (2014) Virtual-reality simulation-based training in ultrasound. Ultraschall Med 35:95–97
Ericsson KA (2008) Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med Off J Soc Acad Emerg Med 15:988–994
Bransford JD, Schwartz DL (1999) Rethinking Transfer: A Simple Proposal with Multiple Implications. Rev Res Educ 24:61
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Guarantor
The scientific guarantor of this publication is Michael Bachmann Nielsen.
Conflict of interest
The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.
Funding
The authors state that this work has not received any funding.
Statistics and biometry
One of the authors has significant statistical expertise.
Informed consent
Written informed consent was obtained from all subjects in this study.
Ethical approval
Institutional Review Board approval was obtained.
Methodology
-
Prospective
-
Observational
-
Performed at one institution
Rights and permissions
About this article
Cite this article
Østergaard, M.L., Nielsen, K.R., Albrecht-Beste, E. et al. Development of a reliable simulation-based test for diagnostic abdominal ultrasound with a pass/fail standard usable for mastery learning. Eur Radiol 28, 51–57 (2018). https://doi.org/10.1007/s00330-017-4913-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00330-017-4913-x