Virtual presence: Today and in the Future.

This is a quote from Ready Player One – the book by Ernest Cline about a future where almost everyone on Earth has a vizor (VR glasses) and is connected in virtual space. Recently I had a quite interesting discussion where we are heading with all this (XR), and if it’s possible to replace our physical contact (person to person) within a virtual space. The more we thought on it the more questions we’ve raised. But let’s start from the beginning. Let’s take a look at the current state of virtual presence and how far are we from dystopian reality which is introduced in the book? Will humans in the future use mostly virtual presence, or maybe we still have a long way to come ahead of us? In this article I will elaborate on every aspect of our communication and will seek the answers.

In one of my posts: The future of communication, I wrote why I do believe that our future communication will change from video calls into the virtual presence. It won’t happen overnight, but obviously – video calls are not the end of the road here and with time virtual presence will be as usual for us as video calls are today. We also need to look at communication from a bigger perspective and break it down into its components. We will explore what is possible already and what will come in the following years.
As we all know –verbal language is only a small part of communicating with each other, the most important part of reading our interlocutor is subconsciously analyzing the posture, mimic, eye movement and other body language elements. So let’s take a closer look at the list:

1. The face
2. The eyes
3. Distance/touch
4. Hands and fingers
5. Feet
6. Mirroring gestures
7. Head movement
8. Body posture
9. Smell and pheromones

1. The face

We all know that the face is the most important element of body language. I could write an essay on how crucial for us is to see the face of another person to judge the emotions. Instead of that, I recommend to you to watch this short video from The Wired where former FBI agent, Joe Navarro summaries what we can read from the face only. And after watching let’s compare it to where we are with VR.

As you see our faces is a complex element which can communicate different things in a second. Reading other people faces is in our genes, we evolved with it, and even if we don’t have comprehensive knowledge we can do it on the subconscious level. How does it relate to virtual presence?

Today: Not possible yet. But…

Nowadays it’s not possible to see our interlocutor mimic in VR. In most of the solutions, you can only see people’s avatars (in most cases the avatars are animated). But the animations won’t have anything to do with the actual mimic of your face. Although when you check the work from Facebook research lab on animated avatars you will see this aspect will be covered soon.
Of course – this raises further questions like: What will happen when people will use it to pretend to be another person, or what if they use software to fake the emotions. You can read further about in my article: Deepfakes – welcome to the world of deception. Putting ethical aspects aside do believe this element will be covered soon in the virtual presence.

2. The Eyes

Of course, the eyes are a part of our face, but I decided to give them separate attention since studying the eyes is one of the most important aspects when communicating with others. This aspect on its own is very complex and multilayered. What are the eyes behaviours we are reading:

• Directions: We are judging if the person we are having a conversation with is looking in our eyes (is being honest) or rather trying to avoid our eye-contact (potentially is hiding something or being shy)
• Pupil dilation expresses excitement or boredom. When the pupil expands, we can assume there are emotions involved.
• Blinking rate indicates the level of stress. When it is connected with touching the face, it could mean that the person is lying or insecurity.
• When looking up, it could mean that person is thinking about the answer. Looking up and to the left, mean that the person is trying to recall something from memory. It could also indicate insecurity or a lie.

Today: Not possible yet. But…

Companies are working on tracking our retina (e.g. Vario, which is tracking the position of the retina and renders only part of the VR experience in high resolution to save processing power). Facebook (former Oculus) is also experimenting with eyes movements and applying them to animated avatars (as shown in the video above). Actually, eye tracking is already implemented in current headsets like Vario or VIVE Pro, and companies use it e.g. to track the attention of the consumer to the layout of the product in shops. Although in terms of virtual presence, there are no solutions yet which will allow us to read the eyes of other people. It will come together with realistic face avatars.

3. Distance/proximity

We can tell if the person feels comfortable with us. When he/she moves towards us and keep close distance most probably feels comfortable. We can easily tell in the room filled with people who are friends and have a better relationship since they are hanging/sitting together. People tend to group with others they feel comfortable with, or they find similar in some way. If you are a new employee in the company – you can fairly easy determine the structure in the firm since people usually hang out with the people they are spending the most time with (in this case – departments). Of course, there are sometimes highly social people who are exceptions from this rule, but in general, it works that way. Touching another person is of course, the ultimate sign that the people feel very comfortable with each other. So the interpersonal distance tells us a lot. Right now when we live in times of COVID19 I must admit that this aspect is especially tough for me. I like to hang around and hug other people and it just feels unnatural to wave to them from distance. So this gives us a taste of how it feels to be in virtual presence.

Today: Possible

It is possible to determine the position in VR. But in reality – it’s not very useful yet. The reason for that is that in VR space you feel neither comfortable nor uncomfortable based on the distance. My opinion is: the more realistic avatars will be – the more will protect our own private zone. So we can say – this aspect is covered (more or less) and will develop with time.

Touch in VR
In terms of touch, there are experiments (e.g. Tesla suit, the most advanced solution on the market yet) with full-body suits which can simulate the touch (or even hits) but it’s still a long way ahead of us in this area.

4. Hands and fingers

It’s important also to pay attention to the hands and fingers. When the person is stressed he/she starts to fiddle with their fingers, and changes the hands’ position often. When a person is holding an object between us it might seem that it’s creating a barrier. When it’s playing with the object – it might mean that it’s either nervous or bored. When we emphasize, we tend to show our fingers spread wider than usual in the open position. So there are a lot of nonverbal cues connected to our hands and fingers.

Today: Possible

The research Facebook is doing on its hand gestures is truly astonishing. It’s already possible on Oculus Quest to see our hands and gestures and accuracy is really impressive. Look at the video below what is already possible.

And this is when it gets really interesting. It’s the beginning of the road since the studies on human gestures are much more developed right now. If you take a look at what Facebook research lab did on Dense Hand Surface Tracking with Elasticity – it’s really close to reality.

Of course all this in lab environment, but if they will be able to do it with the cameras embedded in VR glasses – we can say that gestures, hand and fingers tracking are fully implemented in VR and virtual presence.

5. Mirroring gestures

Mirroring gestures tell us if the person we are talking to is connected to us. If it’s subconsciously mirroring our gesture mean that he/she agrees to what we are saying. (check the video with former FBI agent above if you didn’t yet).

Today: Possible. But…

It’s possible in theory but because all aspects in virtual presence are rather clunky, it’s not very useful yet. The same as with proximity and body posture. But it will come with time when aspects like full body avatars and advanced finger and hands tracking will be addressed.

6. Head movement

How the head reacts tells us if a person is interested in you (looking straight into your direction), or could be not interested in what you have to say (looking nervously around). Nodding means it’s agreeing with us and shaking to left and right means the opposite. So it’s important to see the other person head. Is it possible in VR?

Today: Possible

We can say that with the current state of technology we can read the head gestures since the VR googles with 6 DoF (6 Degrees of Freedom) can track the head’s position and its movement in space. It will be even more informative when we will have a full face and eyes tracking.

7. Feet

During a conversation, we can also observe the feet layout. If feet are directed towards the door it might tell us that subconsciously the person want’s to leave the room. If they are directed into our direction and they are steady – we can assume that person is feeling relaxed. If they in constant move – it might tell us that the person is nervous or impatient.

Today: Not possible.

In most of VR experiences, you won’t see your feet since they are not the most important part of the experience. I don’t anticipate it will change any time soon since there are more meaningful parts of the virtual presence which needs to be addressed first.

8. Body posture

At the end let’s talk about our body posture. It’s a mixture of all elements above. The body posture actually reveals a lot of information. Open body posture with hands spread wide means that the person feels comfortable and it’s not afraid of you. Closed body posture with arms crossed on the chest – obviously, the person is in more protective itself pose. Although sometimes crossed arms could mean that individual feels confident.

Today: Partially possible.

Because of body posture is a mixture of so many elements which I listed above, in theory, it’s possible, but in reality, we can’t say what the person is trying to say since we can’t really judge the body posture in virtual presence yet. Facebook is experimenting with the full-body animated avatars and the results are really promising. So maybe in a couple of years, we could address all aspects above using full-body avatars. Look at the video from experiments recordings.

9. Smell and pheromones

This is the most controversial one. Human being also uses their sense of smell when talking to another person. Nice smell of perfumes mixed with skin can be appealing and change the way we are treating our interlocutor. Also, animals use pheromones to mate, and there is a theory (although not scientifically proven) that humans are doing the same. Nevertheless, we all know that smell is a part of our personality and influence other people. So where are we with this aspect in VR?

Today: Not possible.

Although there are some experiments including smell like Feel Real startup – we still have a long way ahead of us in this topic. And it’s one of the most complex ones since there are millions of different nuances in this category. Therefore I think this element will be implemented as one of the last.

Summary of virtual presence:

Conclusions

Even if it’s not perfect yet, virtual presence will be without doubt the future of our communication on distance. It’s getting there slowly but surely step by step. The next questions we should ask ourselves is – what influence on our lives it will have and what the psychological and ethical consequences are of using such technology (deepfakes, isolation, influence on our wellbeing to name a few). As always, time will tell. 

Hope you enjoyed this article. If you think that I missed something in this article or something should be corrected please drop me an email at bartek@xrhumanity.net.