Reproducibility in Simulation Surgical Training

As the staff from Global Surgical Training Challenge and Intuitive Foundation prepare for their site visits to meet with evaluation partners and finalist teams, one of the critical questions they will seek to understand is whether the modules are consistently reproducible across different environments and with different learners. 

In the context of research, the U.S.-based National Academies defines reproducibility as obtaining consistent results using the same data and code as the original study. Any number of factors can lead one researcher to come up with different results than a researcher using the same study protocol. The evaluation process in the GSTC will seek to identify the factors that might lead to inconsistent results. 

Each of the four finalist modules is intended for learners with specific levels of previous training and experience. Tibial Fracture Fixation, for example, aims to teach non-orthopaedic surgeons. Each module outlines specific standards of what is considered competence of skills, measured through peer and self-assessments. Each module features objective quantifiable measures to assess knowledge, judgment, and manual skills. 

Over the last year, the teams have refined their modules to promote consistent results. The instructions for building surgical simulators, as well as the didactic learning elements, are posted on the open-source platform Appropedia. They act as a sort of recipe for users to undertake the training. The teams have conducted usability studies of web or mobile apps. They have also ascertained clarity of the instructions to build the simulators, including how easy they were to build. The critical result is whether the modules consistently meet the standards designed within them. 

Some of the changes that teams have undertaken include changing verbiage in the instructions to make key points more clear, or implementing accommodations for reduced Internet connectivity in certain environments. In some cases, teams have adjusted measurements of the simulators to create more realistic conditions.

Over the next few weeks, the GSTC and Intuitive Foundation teams will be able to observe first-hand how learners use the modules. They will meet with the finalist teams and evaluation partners for real-time support and trouble-shooting. These activities mark the beginning of the last phase of the Global Surgical Training Challenge as the teams finalize their modules for the judging panel in December.