Development of Augmented and Virtual Reality Applications for Surgical Planning and Training

Great advances have been made in the field of medical imaging over the past decade. The instrumentation nowadays allow for greater resolution and faster imaging thereby greatly increasing their value in diagnosis as well as surgical planning. However, the advances made in instrumentation have not been mirrored in advances in the visualization of medical imaging. While techniques have been developed for 3D rendering of medical images, such efforts are still limited to 3D renders done on a 2D screens. Current advances in augmented and virtual reality provide great opportunities to finally break through this barrier of visualizing 3D medical images on 2D screens. Development of such AR/VR technology for true 3D visualization of medical images for surgical planning, training and implementation are ultimate goals of this team.

Currently, there are three types of true 3D virtual representations Virtual reality. The first is the type used by Oculus Rift and similar devices which project stereoscopic images to separate screens or a split single screen in front of each eye. At the other end of the spectrum are devices such as Microsoft HoloLens, which enable an augmented reality experience in that virtual objects are stereoscopically projected onto the edge of a see through screen. The third entry to this world are Mixed Reality devices, where virtual objects are superimposed on a streaming video of the outside world in real-time to obtain a more seamless integration of both worlds. All three techniques are applicable to the visualization of medical imaging. However, there has been very little concerted effort to fully utilize this capability in surgical planning, training as well as a reference during surgery.

The team is looking for Computer Science, Biomedical Engineering, Biology, Business (Innovation and product development), and Arts (human centered design) students who want to participate in leading efforts at VCU to innovate AR and VR based approaches for application in the planning of complex surgeries and surgical training.

Projects will include:

• Identifing the challenges faced by surgeons and interventional radiologists with respect surgical and non-surgical planning, implementation and training that can be solved using AR and VR based approaches
• Developing and implement a pipeline for the conversion of CT and MRI scan data into an augmented reality platform for surgical planning
• Developing virtual reality flythrough approaches to navigate the anatomy of the human body
• Developing augmented reality based 3D medical imaging for use as references during surgical procedures
• Developing augmented/virtual reality based approaches for the training of medical students and residents

Interested? Contact Prof. Dayanjan S Wijesinghe (wijesingheds@vcu.edu)