NA-MIC Project Weeks

Back to Projects List

AR in Slicer

Key Investigators

• Alicia Pose Díez de la Lastra (Universidad Carlos III de Madrid, Madrid, Spain)
• Simon Drouin (École de Technologie Supérieure , Montreal , Canada)
• Jose Carlos Mateo Pérez (Universidad de Las Palmas de Gran Canaria, Gran Canaria , Spain)
• Pablo Sergio Castellano Rodríguez (Universidad de Las Palmas de Gran Canaria, Gran Canaria , Spain)

Presenter location: In-person

Project Description

Microsoft HoloLens 2 has demonstrated to be an excellent device in many clinical applications. They are mainly used to display 3D patient-related virtual information overlaid to the real world. However, its processing capacity is quite limited, so developing complex applications that require medical image processing is quite convoluted.

A good solution could be to perform the difficult computations on a specialized software on a computer (i.e. 3D Slicer) and send them in real time to HoloLens 2 so that it can focus solely on visualization. Up to date, there has been a lack of software infrastructure to connect 3D Slicer to any augmented reality (AR) device.

During the last year, Universidad Carlos III de Madrid (Madrid, Spain) and Perk Lab in Queen’s University have worked together to develop a novel connection approach between Microsoft HoloLens 2 and 3D Slicer using OpenIGTLink.

The results of that work are publicly available at this GitHub repository.

The current solution is implemented in a 3 elements system. It is composed by A Microsoft HoloLens 2 headset, the Unity software, and the 3D Slicer platform. The HoloLens 2 application is not directly built on the device, but remotely transferred from Unity in real time using Holographic Remoting.

Objective

Evaluate the transferability of the aforementioned project to other AR devices. Specifically, we’ll focus on the VARJO XR-3 headset.

Approach and Plan

1. Connect Varjo headset to Unity.
2. Find a way to remotely render information from Unity to the headset.
3. 3D Slicer creates an OpenIGTLink server.
4. Unity, containing the AR application, creates an OpenIGTLink client that connects to the server.
5. Currently, when the application is executed in the Unity editor, it starts sending and receiving messages from 3D Slicer. Simultaneously, it wirelessly streams the app to Microsoft HoloLens 2 using Holographic Remoting. Try to replicate the same with Varjo.

Progress and Next Steps

So far, everything works for HoloLens 2. Our current application transfers geometrical transform and image messages between the platforms. It displays CT reslices of a patient in the AR device. The user wearing the glasses can manipulate the CT plane to see different perspectives. The application was build for pedicle screw placement planning.

Our main goal for this week is to replicate the exact same application in the new device.

Illustrations

No response

Background and References

Check out our app in this GitHub repository. This repository contains all the resources and code needed to replicate our work in your computer. You can also have a look at this demo of the functioning of the app on HoloLens 2.

Transfer of geometrical transforms from HoloLens 2 to 3D Slicer:

Transfer of images from 3D Slicer to HoloLens 2:

Results

It worked!

Conclusions

The system seems to be easily transferable to any AR or VR headset as long as it can be connected to Unity, either via wifi or with a cable