NA-MIC Project Weeks

Back to Projects List

SlicerVR - Restore Interactions

Key Investigators

• Csaba Pintér (EBATINCA, Spain)
• Simon Drouin (ÉTS Montréal, Canada)
• Andrey Titov (ÉTS Montréal, Canada)
• Lucas Gandel (Kitware)
• Jean-Christophe Fillion-Robin (Kitware)

Project Description

The main controller interactions in SlicerVR have been broken for about a year, some interaction types even longer. It would be crucial for keeping SlicerVR usable to make the interactions work again.

Kitware and Robarts (Jean-Christophe Fillion Robin, Lucas Gandel, Sankhesh Jhaveri, Adam Rankin) have been investing resources and effort in rehauling the AR/VR backend in VTK for a while, thus now we have a new OpenXR backend and restructured libraries SlicerVR is built on. The goal is to give a small push to their efforts in terms of SlicerVR interactions during the project week, towards restoring at least the previous feature set.

Objective

1. Fix the main controller interactions
   • Flying (joystick forward-backward)
   • Grab and move objects (trigger down and move)
   • Two-controller world move/zoom (i.e. 3D pinch)
2. Customization of controller buttons. Either via the
   • Method in-place (functions integrated here, see also here)
   • Json manifest files (see here)

Approach and Plan

1. Continuous coordination with Kitware / Robarts about our potential to help during the week
   • Make sure the absolutely necessary core VTK changes are in place by the start of the week (RecognizeComplexGesture virtual etc.)
2. Set up VR workstations at the hotel to be able to test and develop
3. Make sure event pipeline reaches the SlicerVR functions handling the events
4. Fix interactions: fly, grab, pinch3D

Progress and Next Steps

1. Restored support for complex gesture (grip) through SlicerVirtualReality#109 and VTK MR-9892
2. Restored partial support for interactions like Select3DEvent and ViewerMovement3DEvent through SlicerVirtualReality#108
3. Cherry-picked said commits, built, and confirmed that the fly and grab&move features worked
4. Confirmed that the actions manifest works on the HP Reverb 2 headset
5. Started work towards restoring the “Pinch 3D” complex gesture (saving initial physical to world transform and manipulate world as the controllers move)
6. Created coarse UML model of the relationship between SlicerVR and VTK VR to help troubleshoot the module.

Illustrations

https://user-images.githubusercontent.com/1325980/216611394-4e46ff9a-21b1-4e96-8f0b-48f81da5866a.mp4

Background and References

Past project week pages

• Project week #37 page
• Project week #35 page
• Project week #34 page

Pinter, C., Lasso, A., Choueib, S., Asselin, M., Fillion-Robin, J. C., Vimort, J. B., Martin, K., Jolley, M. A. & Fichtinger, G. (2020). SlicerVR for Medical Intervention Training and Planning in Immersive Virtual Reality. IEEE Transactions on Medical Robotics and Bionics, vol. 2, no. 2, pp. 108-117, May 2020, doi: 10.1109/TMRB.2020.2983199.