Virtual Reality Training Environment for Surgical Operations

The long and complex training of surgeons is intended to prepare them as best as possible for the successful performance of complex procedures such as bone operations. With the help of virtual reality (VR), it is possible to realistically depict surgical situations and test training with virtual patients free of risk. However, current methods still have their limits in practice, as there is no haptic feedback, i.e. noticeable feel via the finger or hand. The aim of the VIRTOSHA project, which is carried out by the consortium consisting of the University Hospital Bonn (UKB), the TH Köln, MindPort GmbH as consortium leader and Haption GmbH, is to develop a VR training environment for realistic perception during surgical operations.

“During their training, future surgeons need to experience what operations feel like and train their fine motor skills. Therefore, we are developing a new VR training tool that enables lifelike interaction with the tools. We are designing a drilling and screwing simulation with haptic feedback robotic arms and will implement the characteristics of different types of tissue. In this way, the resistance when drilling and screwing into the bones can be replicated and interventions can be trained realistically,” says PD Dr. Kristian Welle, senior physician at the University Hospital Bonn (UKB), who is responsible for the project at the UKB together with Prof. Björn Krüger.

(v. l.): Anna Jansen, David Lähner, Markus Wellmann, Kristoffer Waldow, Prof. Dr. Arnulph Fuhrmann, Dr. Turhan Civelek, Prof. Dr. Björn Krüger, Dr. Jérôme Perret, PD Dr. Kristian Welle, Julian Marques. (Bild: Universitätsklinikum Bonn/PD Dr. Welle) (Image: Universitätsklinikum Bonn/PD Dr. Welle)

Simulating Different Patients

By analyzing real operations, the steps and options of a procedure, the behavior of tissue, implants and surgical tools, as well as the haptic feedback, i.e. the noticeable feedback via the finger or hand, are recorded. Various processes and avatars, i.e. virtual representations of people, can be combined to create new training scenarios and train the collaboration of interdisciplinary surgical participants.

“We use software to design different scenarios where tissue and bone properties can be adjusted to simulate different patients. A realistic representation of the tissues and bones as well as the precise integration of hand movements are crucial for the real feeling in the VR environment,” says Prof. Björn Krüger, head of the “Personalized Digital Health and Telemedicine” research group at the UKB’s Department of Epileptology.

The team at TH Köln is developing the AI-based simulation system with which users can realistically simulate interventions in human hard and soft tissue such as bones, periosteum and muscle tissue during training. The researchers also calculate when the surgical tools hit the simulated tissue in order to optimize the force feedback of the haptic arms. “We are bringing our many years of expertise and know-how in the application of haptic technologies and the real-time simulation of deformable materials to the project,” reports Prof. Dr. Arnulph Fuhrmann from the Institute of Media and Imaging Technology at the TH Köln.

Authoring System for Wide Applications

“In the past, attempts have been made to train surgical procedures using virtual reality,” says David Lähner, founder and CEO of MindPort. “We have developed a new approach to this goal: Using an integrated authoring tool, surgeons will be able to create new content and update existing content themselves. This allows many more different cases to be covered. Our long-term vision is that a 'digital twin' of the patient can be used to practice in virtuality before the real operation begins.”

“This approach is not only exciting for the training of our new colleagues, but will also greatly simplify the introduction of new implants,” adds PD Dr. Kristian Welle

About the Project

The research project VIRTOSHA (author's tool for virtual reality training simulations of osteosynthesis with haptic feedback and tissue simulation) involves MindPort GmbH as consortium leader, the University Hospital Bonn (UKB), the TH Köln and Haption GmbH. The project is funded with 1.5 million euros as part of the Gesünder.IN.NRW innovation competition. The project runs for three years.