Author: AV

Today we aim to capture the basic information about virtual and augmented reality (VR & AR) and their applicability in healthcare applications. 

Both VR and AR are well known topics in the scientific community and they had quite a significant improvement in the last 5-10 years. Nowadays, developing a solution based on VR or AR is at the best of times as the number of available hardware and software solutions continues to grow.

The developers, and with them the scientific community, have many tools that are helping them build better solutions compared with the ones proposed 10 years ago as many system engines provide consistent VR and AR support (e.g Unity, Unreal, etc.).

Healthcare is one of the beneficiaries of these technological advancements. The new level of interaction available through these technologies is a good fit for medicine and healthcare in general, although there are a few challenges and issues for successfully applying virtual  and augmented reality in a feasible manner.

There are at least two areas where these technologies can bring a plus for this area:

a. For rehabilitation, where the possibility of executing a large number of exercises is speeding the recovering process for example the stroke survivors(compared with limited hours of kinesiotherapy) .

b. For medical education, where the beneficiaries will be especially youth individuals that are learning medical notions in an interactive environment

Background

Virtual and augmented reality are both technological systems and their goal is to immerse the users into each specific environment. VR displays a fully artificial environment where the user should believe that is the real world. Also, “VR is a realistic, real-time, 3-dimensional (stereoscopic) computer simulation of physical objects and space”.

According to this article from Journal of Virtual worlds Research(JVWR14), over the years there was no clear consensus about the meaning of VR in medicine. There we find that some shared the same vision of VR in their reviews as they saw the VR as a “collection of technologies that allow people to interact efficiently with 3D computerized databases in real time using their natural senses and skills”. Others ( Riva (2003); Steuer (1992)), described VR correlated with the human experience: “a real or simulated environment in which a perceiver experiences telepresence”. Schultheis (2001) mentions that VR is “an advanced form of human-computer interface that allows the user to interact with and become immersed in a computer-generated environment in a naturalistic fashion”, while Bellani and Fornasari (2011) view VR “as only a simulation of the real world based on computer graphics”. 

The definitions mentioned above are underlining two approaches of VR in healthcare: VR as a simulation tool and VR as an interaction tool. 

On the other hand, Augmented Reality blends real environment and virtual elements while virtual reality is targeting to display only the virtual environment. AR is adding virtual elements over the real-world display and according to R. T. Azuma et al, “A survey of Augmented Reality” in Presence, AR is based on techniques developed in VR and interacts not only with a virtual world but has a degree of interdependence with the real world. 

Augmented Reality systems have three major components: 

1. Tracking and registration; 

2. Display technology;

3. Real-time rendering.

The main motivation of the usage of medical augmented reality lies in the “need of visualizing medical data and the patient within the same physical space”. 

The advantage of AR is the fact that the users are feeling comfortable because the presence is highly achieved. This is a consequence of the fact that the users are still present in the real environment while the virtual elements are superimposed on top of it. On the other hand, in VR the medium (environment) can be fully controlled as opposed to AR and this factor is important when it comes to minimising the external disturbing factors.

For example, a user with an HMD and some sound proof headsets can be fully isolated by outside factors that can disrupt the attention from the operations executed in the virtual environment, while this scenario is unachievable in AR.

Both VR and AR are considered reliable methods to simulate realistic experiences and this makes both technological systems a good fit to reproduce real situations for training or educational purposes, into a safe environment.

There are two basic AR software implementation types: marker-based and markerless.

Results showed that a higher sense of presence was shown for AR invisible marker systems compared with the visible marker ones [IACG15]. A marker-based application solves the problem using visual markers detectable with computer vision methods (e.g. 2D barcodes). For example Corporis application uses marker-less tracking using group plane detection. For this reason, no additional markers are needed to position the 3D Models in the 3D space.

Moving forward, depth perception issues were noticed with medical AR systems while using semitransparent structure overlay onto visible surfaces. In AR the virtual elements are drawn on top of the real environment and besides that, factors like the lighting can affect the output image and as a solution “seven depth cues are evaluated with rendering using depth-dependent color and the use of aerial perspective shown to give the best cues.”  (Ziv Yaniv and Cristian Linte, “Applications of Augmented Reality in Operating Room”, in book: Fundamentals of Wearable Computers and Augmented Reality, second Edition, pp.485-518, DOI: 10.1201/b18703-23, 2015)

Besides VR and AR there is a third notion named Mixed Reality (MR) and often this one is confused with AR. While AR refers to a system in which an enhanced version of the real world is available for the user, Mixed Reality refers to a system that combines real and virtual objects and information. The enhancement elements are virtual and can include objects and information. Basically, in Mixed Reality the physical and digital objects coexist and interact in real-time. 

The figure below  displays the real and virtual environments and the blend between them.

Another important aspect regarding the AR is the fact that it shouldn’t be limited only to the graphics while it targets the real environment augmentation as at a larger scale the augmented reality can be extended to a multimodal approach (Ha. Scharffenberger and E. van der Heide, “Multimodal Augmented Reality – The Norm Rather Than the Exception” in Proceedings of the ACM Workshop on Multimodal Virtual and Augmented Reality, DOI:10.1145/3001959.3001960, 2016). 

After many psychological studies that investigated the adverse consequences of the incorrectly rendered focus cues in stereoscopic displays it was found that the these might contribute to the commonly recognised issues, such as: distorted depth perception, diplopic vision, visual discomfort and fatigue and degradation in oculomotor response (H. Hua and B. Javidi, “A 3D Integral Imaging Optical See-Through Head-Mounted Display” in Optics Express, 22(11), 13484-13491. DOI:10.1364/OE.22.013484, 2014).

Cybersickness is an important topic while developing for VR and it was noticed that some design changes can reduce cybersickness as this is not only related with the hardware specifications. In the following lines a few examples are provided:

 1. In a virtual environment, the developer should use ramps instead of stairs (based on a survey from Oculus Rift Tuscany Demo).

 2. A reduction of cybersickness was noticed if the HUD (Heads-up Display) elements are blended in the 3D scene, instead of using the classic 2D elements. This was strongly observed while developing our applications and the positive impact of this design. 

3. FOV and Focus are key elements and they should be properly calibrated. For example, the human eye is automatically focusing for near or distance. The best approach will be using an eye tracking device to see where the user is looking into the scene.

Today we went over some basic details about the definition of VR and AR and some very basic usages they have in healthcare. As you could have identified already, these technologies offer many possibilities however there are certain gaps that make them still in development. With continuous development all of these can be minimised but only with a dedicated effort in the following years.

Please let us know if you enjoyed this article and if there are other items you would like to know more about. 

All the best! A.V.

(not a ChatGPT product)