AugMedicine: AR for medical diagnosis, Lung Cases
A collaboration between the Centre for Innovation and the Leiden University Medical Centre resulted in a HoloLens application called ‘Augmedicine: Lung cases’. This application integrates sound, visualizations, 3D projection and touch to create a learning environment that realistically simulates the process of listening to the lungs. This application allows medical students and doctors to train themselves on a wide range of scenarios in diagnosing patients that present with shortness of breath.
This collaboration began with the following inquiry: How can doctors make a correct diagnosis from a simple complaint like shortness of breath? As a doctor, it is vital to be able to accurately diagnose patients that have breathing complaints. This process is challenging to learn, since clinical reasoning and auscultation (the process of listening to internal sounds of the body using a stethoscope) require a high level of both cognitive, psychomotor skills. To overcome this learning challenge, the Centre for Innovation helped create an innovative learning tool in Augmented Reality. Our goal was to to explore if this technology would provide a missing link between theory and practice in the diagnostic process.
Leiden University Medical Centre
LUMC: Prof Marlies E.J. Reinders, Drs. Arianne D. Pieterse, Dr. Franka Luk, Dr. Beerend P. Hierck, Manon Zuurmond, and students: Willemijne Meurs, Angelos Youssef, Nicky Bruin, Sybren van Hal.
Centre for Innovation: Jelger Kroese MSc, Leontine van Melle MA.
Departments of LUMC involved:
Department of Internal Medicine, Department of Anatomy and Embryology, Department of Pulmonary Disease
How it works
The HoloLens is used to place a 3D model of an anatomical virtual torso on top of the body of a real person. By using a real stethoscope that is positionally tracked by the HoloLens, students can interact with the virtual torso. As they touch the hologram with the stethoscope, an audio sample of a lung sound that was recorded from a real patient is played back. The type of audio recording that is played back is determined by the position at which the stethoscope touches the hologram. Through this, students learn to distinguish sounds of various lung diseases and learn how the sounds they hear relate to the position at which they place the stethoscope on the body. Since the application is networked, it can be used in a social learning setting where groups of students collaborate under the supervision of a teacher.
In addition to developing the app, the CFI created a gamified scenario for integrating the app into the medical curriculum. In this scenario, students meet an actor in the role of lung patient. The virtual torso is projected on the actor’s body. By interviewing the actor and requesting additional information sources, such as radiology images and a patient file, students have to come up with the correct diagnosis and formulate a plan for treatment of the patient. By this, students can practice the full process of diagnosing lung diseases in a simulated environment that integrates many of the social, embodied and multimodal components of a real-world setting.
Augmented Medicine: the value of AR for teaching clinical reasoning skills
Besides studying lung cases in AR, there is immense opportunity for this technology to teach medical interns clinical reasoning skills with 3D models and holograms. Within the Internal Medicine programme, we developed an application that helps translating 2D viewing of CT imagery to 3D perspective. This application will be released later in 2019. As a next step in our collaboration with LUMC, we have the ambition to offer students the possibility to study clinical cases of the abdomen through a new HoloLens app. The CFI is working to guide the medical faculty in their further explorations to use and implement the HoloLens in educational programmes. Besides using these applications in the Internal Medicine Masters programme, LUMC is simultaneously researching the effect of the Hololens apps on teaching and learning. The research is supposed by professor Marlies E.J. Reinders of internal medicine.