Sustainability, maintenance & quality assurance

Guiding questions:

How to sustain the use of VPs?
How to update VPs in the curriculum?
How to evaluate the quality of VPs?

The development of VPs collections and their implementation are often supported by grants. However, even the most successful projects have to face the moment when the source of funding drains out and new ways of sustaining the initiative are needed. To keep an initiative alive for a long time was never an easy task and the methods depend on the setting. However, there are some that turned out to be successful.

One of such approaches is relying on a collaboration across institutions [Berman 2011]. It was often observed that universities suffer from the non-invented-here syndrome and insist on developing and maintaining their VPs by themselves [Casanova 2019]. In the long run, the costs are difficult to meet, but when divided across many partners become affordable. Sometimes, payment of a subscription fee by users of the VP collection might be required to cover the basic needs [Berman 2011].

It is important that the group of users develop a sense of ownership and feel responsible for its content. This can be implemented by establishing an editorial board that consists of stakeholders (e.g. directors of clerkships) [Berman 2011] interested in the use of VPs at the university. It also helps to align the collection with current national or university learning objectives catalogs [Altmiller 2021] as well as with changes in the professional practice [Morrissey 2014] and modify the content accordingly. For instance, the COVID-19 pandemic introduced several changes in the clinical reasoning process and hospital practice and consequently required an update of an existing collection of VPs as was described by Hege et al. [Hege20]. The VP system should enable evolution and end-user customization of the cases [Botezatu 2010] and empower the teachers to do the update by themselves if they wish to [Zary 2009].

For a sustained long-term use of VPs, the collection needs to reach a critical mass of users [Kolb 2009]. It has to be tightly integrated into the mainstream curriculum because sidetrack add-ons usually do not survive [Casanova 2019]. Leaders at universities must emphasize the importance of VPs and their lasting support [Djukic 2012]. A VP integration also requires time to be established and mature. For instance Kolb et al. in retrospect judged that a 3-year period of the NetWoRM project was perhaps insufficient for successful implementation of the VPs in all centers [Kolb 2009].

Finally, a sustainable use of VPs requires accountability [Botezatu 2010]. There should be a mechanism to report feedback. Even the unsuccessful uses and less favorable events should be analyzed and conclusions drawn. Feedback can be collected using established or self-developed questionnaires. On the one hand, established tools usually are better validated and permit the results to be compared across different implementations. On the other hand, homegrown surveys can be tailored to the local needs. In the category of ready questionnaires, the eViP project has proposed two VP evaluation tools which were later on further developed and validated by Huwendiek and colleagues. One of them is a tool to measure the quality of VP for the purpose of fostering clinical reasoning [Huwendiek 2015]. The other is a set of tools to evaluate the quality of VP curricular integration: a student questionnaire and reviewer checklist [Huwendiek 2009]. These tools have been adopted by iCoViP and are now available on our website together with the results of the pilot evauations. Sobocan and Klemenc-Ketis developed a psychometric tool to measure acceptability and attitudes of medical students towards the use of the VPs in the classroom [Sobocan 2016]. Last but not least, Kleinheksel and Ritzhauptand developed an instrument to measure the adoption and integration of VPs in nursing education [Kleinheksel 2017].

Besides questionnaires, it is also possible to use observational studies (ethnographic approach) to evaluate how students in small groups interact with VPs [Kulasegaram 2018, Edelbring 2018]. When collecting verbal feedback on the VP use, Hirumi et al. suggest to first ask the students what went well in the VP before addressing problems. This helps in directing students’ emotions in a positive manner [Hirumi 2016]. Formal usability studies may involve professional human-machine-interactions tools as eye movement trackers [Boeker 2006, Stevens 2006]. It is also possible to analyze user activities based on VP system logs. This was described in more detail in the section on learning analytics. Lastly, it is of course possible to organize formal studies to explore the effectiveness of VPs in reaching desired learning outcomes in comparison to alternative methods (e.g. simulated patients, human patient simulators) or different VP design or implementation variants. Such evaluation may follow even a randomized controlled trial study and be published as research studies. Kononowicz et al. recently published a systematic review with meta-analysis of such studies [Kononowicz 2019]. When you intend to advance your evaluation to the level of a research study do not forget to seek permission of an ethical review board.

Recommendations:
– Encourage your colleagues to support you in sustaining the use of VPs by giving them a sense of ownership of the collection. Invite them to voice their opinion on the quality and let them co-decide on the development of the integration in a VP editorial board or special task force.
– Save time to evaluate the quality of VPs by using established questionnaires available (e.g. from iCoViP)
– Use also the VP systems logs to detect quality problems.

Phase of the curriculum	4th year of medical school
Goal in the curriculum	The iCoViP collection was implemented to practice selected VPs in the subject of Occupational Medicine (a voluntary subject in the University of Zaragoza in the 4th year of medical school).
Effective use of resources	The collection was used asynchronously so it was not necessary to have a computer infrastructure or a specific room and all students in Spain have laptops or access to the medical school computers. However, issue of updating content not yet solved.
VP alignment	The collection is used while students are doing their clinical placements having a teacher revising the responses and explaining to students most frequent mistakes.
Prioritization/relevance	For Occupational Medicine, the use of the iCoViP collection was a mandatory assignment needed to pass, but did not add points to the final grade.
Relation to other learning activities	None
Time allocation	One week was given to complete the assignment but more time was available for those who wanted to do the activity remote from home.
Group allocation	Students worked alone at home, when the educator discussed the solution of the VP, it was allocated in class (face-to-face or online).
Presence mode	Asynchronous
Technical Integration	It was integrated into Moodle.
VP use orientation/training	VPs were related to a specialty of medicine, but were randomly assigned to students regardless of where in the specialty they were rotating at the time.
Technical infrastructure	We used the technical infrastructure of the iCoViP project without any additions at the University of Zaragoza.
Learning activities around VPs	The teacher explained each VP and how to resolve them after getting the responses of all the students.
Assessment	Students had to complete the VPs, but there was no grade at the end. Only credit (pass/fail).
Quality assurance, maintenance, and sustainability	Frequent revision of the VPs by medical doctors.

Phase of the curriculum	Year 3 and 4 of medical school
Goal in the curriculum	Students learn basic steps of clinical reasoning including identifying relevant findings, developing differential diagnoses, deciding about a final diagnosis, ordering tests to rule out / confirm differentials, and suggesting treatment options.
Effective use of resources	Course tutors needed time to familiarize themselves with the VPs and time had to be planned during the synchronous meetings to discuss the VPs
VP alignment	VPs were part of the modules (e.g. Abdomen, cardiovascular system, pulmonary system, etc.) in year 3 and 4 of medical school and aligned with the objectives of these modules based on key symptoms. In addition a pool of 41 VPs was available for deliberate practice across key symptoms and diagnoses.
Time allocation	5 VPs / module
Group allocation	Students could choose whether to work in groups or individually.
Presence mode	Students could decide when and where to work on the VPs during the period of the module.
Technical Integration	VPs were integrated into the school's learning management system Moodle via a SingleSignOn interface.
VP use orientation/training	No specific familiarization, but general introduction at the beginning of year 3.
Learning activities around VPs	Depending on the modules other learning activities were embedded.
Assessment	The topics of the VPs were part of modules assessment.
Quality assurance, maintenance, and sustainability	We used the built-in feedback functionality to receive qualitative feedback from students and VPs were part of the regular evaluation activities of the medical school.

Phase of the curriculum	Year 1 and 2 of medical school
Goal in the curriculum	Students learn basic steps of clinical reasoning such as identifying & prioritizing findings and composing a summary statement. They also can follow the reasoning process of the VP author concerning differential diagnoses, ordered tests, and treatments.
Effective use of resources	Course tutors needed time to familiarize themselves with the VPs and time had to be planned during the synchronous meetings to discuss the VPs.
VP alignment	VPs were part of the longitudinal clinical course and aligned with the other modules in year 1 and 2 (In case of Augsburg this was Contact, Movement, and equilibrium). We aligned the key symptoms of the VPs with these modules, so that students worked in parallel on these VPs and the corresponding module.
Time allocation	15 VPs over two years / 5 VPs per module.
Group allocation	Students could choose whether to work in groups or individually.
Presence mode	Students could decide when and where to work on the VPs during the period of the module.
Technical Integration	VPs were integrated into the school's learning management system Moodle via a SingleSignOn interface.
VP use orientation/training	At the beginning of year 1 students were introduced into clinical reasoning and how they can train this ability with VPs.
Learning activities around VPs	Depending on the key symptoms during the longitudinal clinical course other learning activities were embedded.
Assessment	The topics of the VPs were part of the clinical longitudinal course assessment.
Quality assurance, maintenance, and sustainability	We used the built-in feedback functionality to receive qualitative feedback from students and VPs were part of the regular evaluation activities of the medical school.

Phase of the curriculum	Basic sciences/pre-clinical years: - At Jagiellonian University Medical College in Kraków we have integrated it in the “Introduction to Clinical Sciences” course in 2nd year of medicine. - At University of Porto in the “Propedeutics/Semiology” course in the 3rd year of medicine.
Goal in the curriculum	- To provide the students with an opportunity to challenge their knowledge at home in between classes with an interactive, clinical-oriented task. - To support learning by linking basic science knowledge with clinical reasoning. Students had the opportunity to practice on undiagnosed cases the skill of differentiating common symptoms, such as dyspnea, abdominal pain, headache.
Effective use of resources	Use of iCoViP VPs in native language of the students. No extra cost needed.
VP alignment	- In Kraków we have selected a few VPs (seven) with common diseases (e.g. pneumonia, pancreatitis, pulmonary embolism) with common symptoms. - In Porto, VPs are chosen according to the common symptoms to promote clinical reasoning.
Prioritization/relevance	- In Kraków the completion of all VPs is mandatory. - In Porto, VPs are introduced on a voluntary basis and are available on demand.
Relation to other learning activities	We used the time students had at home between seminars (Kraków: between on-campus based PBL sessions). VPs were spaced-activated (a new VP appeared biweekly synchronized with changing topics of the PBL seminars that focus on different leading symptoms).
Time allocation	Around 30 minutes biweekly, repeated 7 times in a semester.
Group allocation	Students worked alone at home to reflect but could consult the VPs with their peers or instructors in small groups of the face-to-face PBL seminars.
Presence mode	Students worked on the VPs asynchronously and self-directed at home to have time to reflect and consult textbooks.
Technical Integration	VPs were integrated with a course to which all students were enrolled on the official university learning management system Moodle using Learning Tools Interoperability (LTI) interface.
VP use orientation/training	Students were provided with an introductory email with instructions and had additionally the opportunity to technically practice using CASUS in a parallel “Telemedicine” (Medical informatics) course. Instructors received test-codes to practice the VPs at home.
Technical infrastructure	Students used their own computers from home. An email address was provided to a person responsible for technical support.
Learning activities around VPs	Students were asked to complete concept maps for all VPs they solve. Moreover, in Kraków students were provided with links to additional online articles to help them with topics difficult at this stage of education.
Assessment	Students were asked to complete the cases prior to the end of the term. They were not given grades for the activity - just credit. Their answers were randomly inspected to see common mistakes and provide general feedback to all students.
Quality assurance, maintenance, and sustainability	We checked the diagnostic accuracy of individual cases (and detected one case with imprecise diagnosis). Students evaluated the course using the iCoViP case collection evaluation questionnaire.