[00:00:00] Speaker A: Welcome to the Heart Rate Variability Podcast. Each week we talk about heart rate variability and how it can be used to improve your overall health and wellness.
Please consider the information in this podcast for your informational use and not medical advice. Please see your medical provider to apply any of the strategies outlined in this episode. Heart Rate Variability Podcast is a production of Optimal LLC and Optimal HRV. Check us out at optimalhrv.com Please enjoy the show.
Welcome friends to Heart Rate Variability Podcast. I mean, I'm here today with a very special guest. I read his article on VR and social cohesion and like some of his other research as well to prepare for this article and how he brought in heart rate variability to one of his recent studies. And I'm so glad to have Stefan Strober onto the show today to talk about your expertise. So my friend, just to start us out, I'd love to just get an introduction of yourself. I know I tried to like get the right pronunciation of the name there. So thanks for your patience with me. How you say it is so beautiful. So I just love to get like a little bit of introduction of yourself before we jump into this most recent article.
[00:01:22] Speaker B: Sure. Thanks so much for inviting me to your show. So I listened to a few podcasts, find it really interesting.
So a little bit about myself.
So yeah, I'm actually a computer scientist, so it's maybe a little bit different from your normal audience that you have that are more psychologists and physiologists.
But nevertheless, I'm also very interested in the topic. And what I am doing is actually I'm interested in virtual reality, mixed reality and extended reality, and especially about how we interact in those environments. And I don't know if you have some experience with virtual reality includes the concept of embodiment. That means you are embodied as an avatar. You can look down, you can see your hands, you can see the other people having their own avatars that they can freely choose.
And I find it a very interesting field to do research in this domain from a technological side of you, but also from a psychological side. Because if you want to interact in this environment, it's very important to look how humans interact in the real world and understand. And then we can apply it also to creating new technologies where people can interact in a more sophisticated way, let's say, and more enjoyable, they produce more creative results and so on. So I have been working in this like since 10 years from a very interdisciplinary perspective.
And yeah, I just find it extremely interesting how we interact in this virtual world, but also now how we interact with embodied AI agents I find also really interesting that can mimic many of the human behavior. And also now you can talk to them.
And it's also maybe difficult to distinguish between am I talking to an agent? Am I talking to a real person?
So this is definitely now a really interesting line of research as well.
And yeah, so I apply some theories of, from psychology and neuroscience to technical developments.
[00:03:35] Speaker A: I love that. I would love to get your opinion because, you know, I'm a big fan of virtual reality. I had the Oculus one, I have the Oculus three. I couldn't afford the Apple version of it. And that may have been a good choice long term as well. I mean, the one.
It's an interesting field because at one point almost you could think about Pre and post ChatGPT release.
The initial version is with Meta, Mark Zuckerberg and a lot of others.
There was like a year or two in there where we were all going to live in the metaverse. And I was, you know, interested. My background being mental health. Like, could you do therapy that way? Is that, is that a better way for virtual sessions that, you know, as you use the word embodied, is it a better experience for the individual? So I had a lot of questions and you know, as a lot of things. So I bought the headsets. I couldn't find a lot of productivity uses for it. And none of my friends wanted to spend $300 to be in VR meta world with me at the time. You know, so I just love to, as you've been in this science and thought a lot deeper than I have, just kind of, where do you see this? Is it more like, hey, once we get the hardware correct, we'll all be more in virtual reality or mixed reality, which is a fascinating experience as well. I just love to get your opinion on where the field, where you see the field sort of evolving to nowadays.
[00:05:18] Speaker B: Yeah. So basically where to start?
So we are using it like in research a lot like in psychology and neuroscience. It is an extremely valuable tool because it allows you to study human behavior or also animal behavior actually in interactive settings, like for example, human usually in psychology, neuroscience, you have psychophysical paradigms where you see some stimulus on the screen and then you have to press a button. And this is extremely good because it's extremely controlled and you can make it comparable.
But VR has the potential to study interaction in real world, interactive setting. So my PhD, I was doing a lot of studies with social perception and table tennis. So I had people playing table tennis in virtual reality. And then I can Manipulate exactly what you see two during the game. And if you know table tennis, it's really challenging, highly dynamic, fast.
And then we looked like, what information do you pick up about your opponent in this task? And we found really very interesting results that are quite different from the typical psychophysical studies, which are very controlled but not really representative of the real world. And so there's a lot of examples of the social psychology where of course it's something else. If you see a picture of somebody on a screen or if you're standing in front of a lifelike avatar that represents a specific person, you can make it much more ecologically valid as it called, like how your results apply to real world context.
So that's. So in this field and also in the medical field, we use it a lot for.
Yeah, so we do some studies on body perception in eating disorder patients, you know, that have a, have a misrepresentation of their body dimensions. And there you can also give them an avatar that looks like themselves, the personalized avatar. And then you can manipulate or they can manipulate exactly to recreate the avatar. Like they see themselves like in a mirror. And they have to recreate the avatar that they have in the head basically. And then we can study the thresholds, how they represent their own body and it can be also used for treatment. So actually I'm working a lot in the clinics in different domains with virtual reality. So they are the context. It's really extremely promising, I would say. But yeah, in the consumer sector I think there's still a lot of problems with the hardware and usability that makes it just.
Yeah, we have this hype cycle. It's always with VR and there was one again like two, three years ago, but. But it's slowly, I mean it's slowly taking off, I would say, but going a little bit more in the direction of smart classes and xrp.
So definitely I see the future there, but yet will still take time, I think.
[00:08:24] Speaker A: Yeah. And then there's the interesting work you've been doing with the social connection piece as well. And again, as somebody who's admittedly played games, I really avoid the social connections that you might get in a gaming environment.
Just because I do, I want to escape reality. I don't want to be reminded of human beings while I'm in whatever virtual scenario I'm in.
[00:08:52] Speaker B: But.
[00:08:52] Speaker A: But I'm curious because your work is obviously beyond just the gaming environment. And so it's sort of just kind of as we start to bring in your most recent literature, kind of what informed your Your idea of bringing the social connection, social coherence into and then using heart rate variability to measure that. So a little bit of the history leading up to this great article that is more recent.
[00:09:24] Speaker B: Sure. So basically I was interested. So there are different ways how we can communicate using long distance. Like for example, now we meet each other in video conferencing call.
And as you know, in the corona pandemic, everybody had to go online and everybody was in the zoom calls all day. And then there was this effect called zoom fatigue. I don't know if you heard about it. Like, yes, in the Zoom, then you get tired and really like can burn out symptoms and stuff.
And there's not really a good explanation why this is happening.
And then I got interested to come. So I met with some of my students, I met in virtual reality and I had somehow the subjective experience, it was much smoother somehow.
[00:10:15] Speaker A: Yeah.
[00:10:16] Speaker B: Even though like you have the glasses, which is like a little bit tiring, but on the other hand you have like, yeah, you're sitting on the table, virtual table, you see the others, you can look at them, you can talk with them. And I found it much more relaxed for myself. And then I was wondering, why is this the case? Right. And so I was thinking about it and then with my student Zara, we from psychology. And so we were designing an experiment where we just compared different types of communication. So like we had one scenario where people were meeting in Zoom, one scenario where people were meeting in virtual reality, and then we still had a control condition obviously when they met in real life. So our three different conditions.
And then we were thinking, okay, how can we measure how good the social interaction and the performance works in this different media? And so we use the classic things that you use in research in computer science and virtual reality, which is one, one concept is called social presence. It's how much you feel the other person is there close to you, you know, like that you can talk to the other person. That's usually measured with questionnaires, but you can also use it measured behaviorally. But that's really a way often used measurement tool.
And then we were obviously also interested in performance and we were especially interested in creativity because in the zoom conferences, like when you meet with your colleagues and work, the creativity in the workspace is extremely important.
And there was one important study that showed that in virtual reality.
I know, sorry, I mix it up in Zoom meetings, the creativity is decreased compared to real life.
And this study was suggesting that we should be cautious using technology as a default.
And then I was interested if virtual reality might be maybe better than zoom situation.
And so we set up this experiment and then I met with Jens Prisoner, who's a psychologist in the University of Constance, also in Germany. And he is a specialist in heart rate variability. And he had this idea. He's also co author in the study or last also in the study.
He had the idea to also measure synchrony, heart availability, synchrony. And that was the first time I also heard about, heard about this concept and I found it really interesting. So I read the literature and I mean, you're talking about this all the time. I don't have to explain hardware, but we know it's linked to the stress system.
And now maybe I talk a little bit about the concept of synchrony.
[00:13:24] Speaker A: Yeah, that'd be great.
[00:13:26] Speaker B: Which is basically how heart rate variability aligns over time between two or more individuals.
And so synchrony is also measured a lot in psychology or social psychology. And it happens on different levels.
It happens for example on a behavioral level.
So there's so called motor synchronia. That means like you are matching posture, gait.
Yeah, like head nodding and like your facial expressions and everything subconsciously when you talk to another person or when you're in a social context.
And that's one level on the behavioral level. But then interestingly it also happens on the physiological. Yeah, on a physiological level that means we also synchronize our heart rates and also heart rate variability or respiration, skin conductance. So there are a lot of studies measuring synchrony on this physiological levels as well.
And then you also have neural synchrony. There are also a lot of recent studies using Fnius and Eg to measure neural synchrony.
For example, when you're watching a movie together, then you also synchronize on a neural, neural level.
And in general you can so say all these concepts are somehow connected. Right. So for example, I mean, of course if I synchronize on a behavioral level, then I will also, yeah, synchronize on physiological level also on a newer level. So it's not really clear what is the causal direction here. So it depends a lot on the task. For example, if I watch a movie together, then I don't have behavior, I'm just sitting there. But still the brain activity synchronizes. And there are a lot of theories why is this happening. But one theory, what I like is that degrees. So like in social interactions the degrees of freedom are very huge. So you don't know what's happening. You Know, like could go in any direction. Right. And what we try always is to reduce complexity somehow. But if we synchronize and we move similar to the other person, then it reduces complexity in ourselves and makes it easier to predict what the other person is going to do next. Yeah, it's for prediction and reduction of variability.
And so. Yeah. And the hard revealability synchrony is especially interesting I think because it's connected to the stress system.
[00:16:11] Speaker A: Yeah.
[00:16:12] Speaker B: And that is also very important for social interaction.
So for example, what is the most stressful thing people can experience? And one of the most stressful thing is to give a presentation in front of very critical people. That's the so called Trier social stress test which has been also invented in Trier in Germany here, but it's widely used.
And that's this aspect. I think it's stress system is very important for social interaction and that's why I think it's a really good marker. So. And it's also extremely easy to measure. So you just have to attach a sensor on the, on the chest and then you can easily measure it. I mean if you measure EEG or something in a social interaction task, it's very complicated because you have movement artifacts.
[00:17:09] Speaker A: Yeah.
[00:17:09] Speaker B: And to measure multiple people, that is, that is really challenging. I think so. But to come back to our experiment, so we had in the, we had like the three different conditions, so real world setup, face to face meeting, virtual reality and zoom.
And we had the creativity task and we had groups of three people. So we wanted to not only look at two persons completing the task, but really three people because it increases the complexity also of the task. For example, sometimes in a, if you have only two people, then always one is speaking, the other is listening. But if you have three people, like anybody can speak and the other people looking at this person and so on.
So yeah, so and then we ask people in the different conditions to be creative. So there's also standardized tests for this. Of course there's a so called alternative uses task. So you just give them some random objects like let's say a paper clip and then you have to say, okay, now you have five minutes time as a group to find as many possible use cases for this thing. For example, a paperclip you could use, I don't know, could build like an antenna for radio or you could poke somebody or you can use it as like, I don't know, like to open a lock, you know.
[00:18:42] Speaker A: Yeah.
[00:18:43] Speaker B: And then you have 5 minutes time to come up with as many solutions as possible.
And Then we just record the solutions and then there is. Yeah, you have to analyze different things. Like one is called flexibility, which is so like one is just how many answers do you find? Yeah, but that's a little bit biased because you might have extremely similar use cases.
[00:19:11] Speaker A: Right.
[00:19:12] Speaker B: Then you have also so called flexibility. That means like how many different categories of solutions do you find? So this is usually associated with divergent thinking and creativity.
[00:19:25] Speaker A: I would love if I could just interrupt you really quickly and then get back. I think that, I mean, I'm struggling a little bit. I know this is kind of difficult to communicate through a journal article, but like I'm trying to picture. I can picture myself in the zoom on the paperclip question in person, no problem. I could put myself easily in that situation.
For those that might not be as familiar with VR or for folks like me who are playing kind of first person oriented games where I might see my hand shoot a bow or climb a mountain or something like that, I'm trying to picture, even as a VR user, am I seeing people with paper clips?
What is the interface for the virtual reality? Just to give our listeners something to visualize in their mind of what your study subjects were seeing.
[00:20:29] Speaker B: Actually, that's a very important point. You're right. So I didn't fully explain this, but it's really important because in the real world, face to face conditions, they sit on a table like three people sitting on a table facing each other. And in virtual reality, they are actually in different rooms and each of them wears a head mounted display like Oculus Quest 3 we used, I think.
And then they are virtually transported in a virtual office on a.
They have also like a table and they're sitting exactly like in the real world, but like in a virtual world. So they can see each other's avatars. We also have, we also used personalized avatars. That means we took a picture and then we recreated the avatar of this person of every participant. And then they can see each other and can.
Yeah, can use their hands, for example, for gesturing. They can turn their heads. But what is really important here, what we didn't do is we didn't do eye tracking and we also didn't animate facial expression. Actually.
[00:21:39] Speaker A: That was good. Yeah, that was. I think that's such a key point. Yeah.
[00:21:43] Speaker B: Yes. The review of us were extremely critical about this point that we didn't use facial expressions because of course they are also really important. Right. And also eye tracking, eye movements. And it's possible to do this now relatively easily, but at this time it was still not the main scope of the study.
So. Yeah, and also in the zoom, they were also sitting in different rooms. And what was also very interesting with regards to this, all other studies previously that investigated synchrony. People are usually sitting in the same room.
[00:22:21] Speaker A: Yeah, exactly.
[00:22:23] Speaker B: So, and the question was also, if you are remotely connected, can you also think, do you also synchronize your heart rate variability?
[00:22:32] Speaker A: Right.
[00:22:32] Speaker B: That's also really interesting to us because some people would think, no, if I'm disconnected, I will not synchronize at all. So.
[00:22:41] Speaker A: Well, especially without the emotional. And I like we, we, we, we assume. And this is why I can't wait to talk about the findings of this. Like, I mean I, you know, and to just to nerd out, you know, like Stephen Porges work around like the vagal connections to kind of like where my beard is.
And just knowing that like during COVID when a mask covered this area, I'd have. My connection with people would be. So I felt so limited because I have no idea were they, did they have gas, were they angry, were they laughing? Like, you know, it's so hard to get the non verbal part of that emotional piece, which is where again, I found your research incredibly compelling. As we move to the results of.
Yeah. How's any of this get communicated over VR? Because even our conversation, both of our hands are moving, facial expressions. I, you know, even though I'm kind of looking at a camera and looking at a visual representation of you, you know, through a camera, you know that there is some of those, you know, things that I would rank high on, things that would be needed for social connection, social coherence, those sort of things, you know, in hrv sort of. That would have been my guess. And anything else you want to tell us about the study, but this is where your findings just made me have to reach out to you and have this conversation.
[00:24:22] Speaker B: I think one. So it's. I think you're completely right. So the non verbal cues are of course extremely important. Facial expression and also body posture. One advantage of VR is that you see the whole body posture in zoom. You see only the face.
[00:24:37] Speaker A: Yeah.
[00:24:38] Speaker B: I think another big difference is why zoom is maybe not the most optimal medium is because you don't have the shared spatial reference. That means like there are three people. Then everybody's looking at me all the time. Right. Like as.
But even if I'm not talking, everybody's looking at you.
[00:25:00] Speaker A: Yeah.
[00:25:01] Speaker B: And that's like a 2D plane. But like that's not, I mean from an evolutionary point of view, it makes no sense that people are sitting in front of you looking at you. And that's of course stressful even if you're not talking. And in VR you really have the spatial reference. It means like one person is talking the other people looking at this avatar and you have a natural turn taking and yeah, and attention processes. So but okay, let's continue.
So now I explained to you the setup, so maybe now I can talk about results.
[00:25:38] Speaker A: Awesome. Yes please Dale.
[00:25:40] Speaker B: So okay, so the first marker, what I mentioned is the so called social presence which is assessed with a questionnaire. So that's a standard measure in this type of research.
It just means like how close you feel the other person is close by.
And there we found not too surprisingly that the lowest or the highest social presence you have in the face to face condition sitting next to each other, the lowest we find actually in the video condition.
And we are in the middle. It was significantly higher as video but also significantly lower as face to face. So that kind of makes completely sense that we know already from previous, previous research.
Then the creativity.
So we measure different things. But this flexibility and divergent thinking I think is the most important marker there we found actually also similar pattern. Best performance was in the face to face condition, video condition was worse and VR was some in the middle. So you produce more creative results in the VR but face to face is still the best.
And then now let's look, I still have the data here.
Let's look. In the heart rate variability synchrony.
And there we were really surprised because the heart rate variability in the virtual reality condition was almost identical to face to face condition. But there was a huge gap to the view conditions. That means in VR you synchronize your heart rate variability as good as in the face to face condition. But there's a huge gap towards in the video condition. And this was extremely surprising because okay, we thought maybe a little bit better in the VR compared to the video, but that we didn't find a difference between video between virtual reality and face to face was extremely surprising.
And I think it's also first time that one could show that hearted visibility synchrony happens also when you are remotely connected.
Yeah, virtual reality. So we find this really very fantastic result. Even though the VR condition was not perfect, as I said, there were no facial expression and no eye tracking. But still alone being present in the virtual metaverse like scenario caused our hearted visibility to synchronize.
[00:28:24] Speaker A: So I'm curious and I always give my guests the opportunity to Speculate, which means no one, and I always throw this out, can hold you accountable for some speculations. Right. That we know. Like, you know, that we need research to back this up. But I'm just curious, like, I mean, I think part of it, your, the, the first research findings can help explain the, the heart rate variability. Like if it's, if it's better than video calls. And I'd be a little worried honestly if like VR was better than face to face on some of these metrics, I'd be like, whoa, whoa, whoa. This is, you know, are you sure you're resetting like so, you know, I think it's a little surprising, but yet I think like, so, so we got the, the VR beating, the video calls, you know, on a, on, on those earlier findings to get to the heart rate variability.
Do you have any like speculations of, of what might be going on to sort of, sort of get us to that? Yeah, face to face still might be the best. But if you for whatever reason can't do that, I think about like international companies with budgets to get everybody on VR sort of what, why, what might be explaining something that would seem to be justifying companies and others therapists thinking seriously about this going to virtual reality when face to face isn't tangible.
[00:29:59] Speaker B: So, okay, so first of all, I can only speculate.
[00:30:04] Speaker A: Yeah man, you're allowed to speculate.
[00:30:08] Speaker B: I don't know, I don't know the answer since like our studied study had some limitations that I said. But so I personally think it has to do with the spatial reference. That means like in VR you are like, you have your like similar to the real world. You have like a fixed location and there's a turn taking you look at people like, so that's, that's, that, that is my guess.
But so we would, we would have to do future studies to investigate how for example like facial expression or eye tracking and so on enhances or maybe the minimalist effect so that I can only speculate. But I think what is really also cool about this work is that maybe we can use this really as a marker to assess the quality of the social interaction or have an additional marker. For example, if we use virtual reality in like, let's say in a clinical setting with patients and maybe we can measure the connection between the doctor and the patient using this heart durability synchrony as an easy measure and can also use it like in human computer interaction, for example, if you want to improve remote collaboration to see what helps and what doesn't help for Improving the interaction. So I'm just glad that we kind of maybe in the future can use this as an additional marker of the other markers of the quality of the social interaction.
So as a measurement tool. Right. And you can also use it, the nice thing about this heart rate variability synchrony, you. You can measure it more or less in real time with a little bit of a delay. And it tells you maybe directly if something happens on a social level, if an avatar comes too close to you or like if, you know, like if something.
Yeah. So there's a lot of possibilities for future research in this field.
And yeah, we are also working now already on new studies.
And yeah, I think it's just the first step, I think, to use this also in the human computer interaction community to, as a, as an experimental result.
[00:32:31] Speaker A: I love that. I mean, I'm curious what. And again, in the realm of speculation purely here, but one of the things that your article got me thinking of because I was okay, what are the things that we know would allow for increase in heart rate variability and increase in that synchronization?
And the first thing that popped into my head that doesn't always exist to face to face is like safety as well. And I wonder if I'm in a room full of strangers.
[00:33:06] Speaker B: Absolutely.
[00:33:07] Speaker A: Where we're, we're not necessarily. I got an avatar. Yeah, you can judge my avatar if you want. But it's, you know, there might be even more psychological safety in a VR space where, I don't know, some of those background things that we have going on is, do I have food in my teeth? You know, does this person find me attractive? Or, or, you know, do I have gas or whatever, like range of kind of those basic human things that go on in any interaction. I'm assuming these people were. Did not know each other going into it.
You know, if there was some additional safety benefits that might have been even greater than face to face, but, you know, we're even worse off in the virtual reality.
You know, that was one of my questions I kind of wanted to kind of throw out for speculation is if we had that social psychological safety in VR that might beat the other ones, even though, again, face to face would win maybe overall with that, that's a
[00:34:23] Speaker B: really interesting question, I think.
And yeah, like, we know from studies in social psychology that the synchrony is always thought as a kind of social glue.
What clues, groups of people together. And there's a lot of studies showing that, I don't know, if you synchronize your behavior, you are more liked by a stranger. For example, that.
Yeah, already. It starts already. Mother infant synchrony predicts, I don't know, immune function and stress resilience. So it happens as a synchronization process happen already. They are innate. They already happen. Like when the baby is born, it synchronizes already the vocal, the vocals. And. Yeah, so that's such a fundamental mechanism, I think. And yeah, what people. People most often don't realize that this fully immersive metaverses, let's say they also often research finds that the same social mechanisms also are in play when you enter this virtual immersive realities.
[00:35:33] Speaker A: Yeah.
[00:35:34] Speaker B: And another example, what just comes in my head is like interpersonal distance. So it's also studied a lot in.
In social psychology, there's a certain distance that you stand close to each other when you have a communication. It's also different in different cultures and between.
[00:35:51] Speaker A: Yeah.
[00:35:52] Speaker B: And so on. And you can measure this. And research found already, like in the 90s, that in virtual reality you can very well replicate this mechanism. So people also standing the same distances to the avatars. And so.
Yeah, so that.
And. But the problem is, of course, virtual reality is kind of anonymous, so we don't know who's driving the avatar, for example. Yeah.
But I have found already in some of this immersive words, people already using also.
I don't know, they display, for example, their heart rate in order to signal their emotional states. So in VR you can do many more things. Of course. You can embody a different avatar.
You can be any person that you want to be, but you could also, yeah. Show your emotional states. You can show your, I don't know, like your. Your heart rate, for example, and can make it visible to other people.
[00:36:55] Speaker A: Yeah.
[00:36:56] Speaker B: So there I think it's also going the direction of biofeedback, maybe. So I think there's a lot of opportunities one can play around in VR that you cannot do in the real world, I think.
[00:37:08] Speaker A: I love that.
[00:37:09] Speaker B: And that makes it really an experimental.
Experimental field. Very interesting experimental field, I think so.
[00:37:18] Speaker A: Cool. I. I would love to. As we. As we kind of start to wrap up here, there's two questions. One is just like the nerd inside of me has to ask an expert on this, and then I want to see where the future might go. But I think one of the. The big informers of future work around VR is. Is the technology.
I think that it's been.
I'm okay wearing an Oculus 3 for a while.
I think it gets uncomfortable. But you're so immersed in the world that sometimes you forget you're wearing it. So I find my tolerance for that is if I'm engaged with things, I just kind of forget and I'm in this immersed world. But also that technology, if I were three people kind of maybe looking at facial. I mean, there's no way to kind of record my facial expressions right now while I've got that on.
I checked out, I got a new pair of glasses. I thought about looking at some VR glasses and then it seemed like mostly social media, which I honestly tried to spend little time on anyway. So, like, I didn't really need that. So I'm curious where, as somebody who is way more connected to this field than I am, do you see any promising developments with technology?
And augmented reality I think is going to become more.
More and more what we talk about here with the glasses that wear like normal glasses. Just that we could sort of utilize some of the findings in your research where there might be more emotional expressions, more so than a happy face. I click kind of thing. Like, where do you see the technology going that might be able to support some of the really powerful things you've seen with your findings and maybe overcome some of the technological barriers that you have faced as well?
[00:39:22] Speaker B: I think there's a lot of technical developments, a lot of research going on. And also what you can do now with AI and new display technologies and new sensor technologies is already, I think it's extremely promising, but unfortunately it always takes a time until you can make also a product out of it because much of this technology is still quite, let's say, expensive and not mainstream yet. And I think the problem with VR headsets is always if you want to sell them, they cannot be more like than, I don't know, $500 or less. And so it's very much coupled to the development of mobile phones because they use the display technology and you could build much better devices already, but then they would be extremely expensive. I think there's also a lot of research in. Yeah. Including the physiological states into the VR experiences for research purposes. Lots of experimental VR headsets that measure, I don't know, apneas and yeah.
And like physiology. And then you can use also machine learning to predict internal states like anxiety, stress and so on. This, I think this works already quite well also on the level of rendering or simulating realistic bodies, which is also important for social VR experiences. Obviously right now, if you go in this world, the bodies look like really bulky and eerie and not, I mean, yeah, it looks more like Comics, right? Yeah. But there's also a lot of research on creating realistic bodies using machine learning and 3D scanners where you can scan like millions of people and then train some statistical body models and so on. So this you can make already much better.
So I think if I look at all the developments, I'm quite excited about the Next, let's say five to 10 years because a lot of new things will come and maybe I think it will go more in the direction of spatial computing where reality and virtual reality kind of blends in.
And I imagine also maybe the future of zoom or video conferencing will be more like some sort of augmented reality where you have your normal glasses. Right. And then you're sitting on the table. You're sitting. I'm sitting now in your kitchen. I don't know. And can talk to you.
[00:42:01] Speaker A: Yeah.
[00:42:01] Speaker B: But it will be probably, probably not be fully full VR I guess, but more just that you have limited information that you display and that you can still at the same time interact with the real world or with real people.
And so I think there's a lot of problems to solve. More like also in human computer interactions how you can make this really work well so that people are not disturbed in their daily activities, but they can still communicate and interact with the digital world.
[00:42:35] Speaker A: Absolutely. And I just see like articles like yours just gives a. I really hope provides the emphasis. Even though I still know there's billions of dollars going into virtual and augmented reality and spatial computing, whether Meta is throwing, you know, 300 billion a year at or not anymore, there's still a lot, a lot of people doing a lot of really great work. But like that's what. And I do a lot of virtual trainings and there's nothing, I mean I appreciate the opportunity to do it, but I'm, I'm doing a training into a camera and everybody else's camera may be off and honestly if they're on, I, that's even more distracting in some ways because when I'm attending a virtual training, I'm usually checking email and multitasking as well. So I appreciate, I appreciate, you know, what the normal human being is doing during these trainings. And it's just like you lose so much of that in that environment and I think a conversation with you is a lot better.
But then you put a third or a fourth person in and people are falsely raising there. There's no real flow in the same way to that to an in person experience. And I'm, that's where I got really Fascinated with VR is, you know, one working with a lot of clinicians and coaches.
It seems like there's a lot of potential there, you know, and in your research, you know, I think really positions the question is, can we do this better in VR than we could over zoom, knowing that, hey, if you can get somebody to an office, that's obviously ideal, but now we get a lot of biometric information and that could be really cool as well. So I just think that there's so much untapped pieces here and, you know, it's a hardware problem in some extent, but boy, is our software ability to progress with AI getting quicker and quicker too, which maybe it's three and not five years. Because I'm ready for a pair of glasses I want to wear all the time with that.
I'd love to wrap us up here. I know you mentioned your research has continued, and I've learned not to ask researchers to talk about their current research unless they really, really want to. But I'd love for you to also look into the future of research.
As somebody that threw out some really amazing findings that I would not have guessed as I was reading the setup of your article. And then the findings, some really powerful and as any good research, powerful findings that lead to other powerful questions yet to be answered. I'd love to just hear, as you speculate on the research front, what do you. What are those questions you're like, you know, kind of just dying to answer yourself and you know, that you hope others will look at as well.
[00:45:49] Speaker B: Yeah. So, like, so, for example, we are now very interested in. To understand the. The underlay mechanism somehow of the synchrony of this physiological synchrony. Awesome. And I think with virtual reality, it's very good because you can easily manipulate the visual cues that you see about the other person. So all this difference is what we just talked about, like between VR and zoom, for example. Now we can do look more precisely. Okay. We can, I don't know, maintain the spatial reference, but people cannot look at each other. Or we can include facial expressions and can see how this changes the synchrony. And by this understanding this become closer understanding. Why do humans synchronize and what are the. Yeah. What are the visual information necessary or like the multisensorial information necessary to. For this synchronization to happen?
[00:46:47] Speaker A: I think I love that. Do. Do you. And this is total speculation, but with the numbers of the VR trending more to the side of the face to face than the zoom, and with you recognizing and us talking about the limitations of VR is there. I mean I left your article thinking, oh, if we could get emotional expressions and hardware, do we reach a point where virtual reality, we could create a virtual reality environment for collaboration that even if you're in the same office building and you could meet in person, that VR could be better.
That's kind of what I saw in your research, to be honest. Which was it wouldn't be like before reading your article, I would not have thought I would be questioning that. But it seems like we could manipulate an environment to a task with better hardware that could allow us to maybe go beyond the face to face environment. Am I way just taking this too far or do you see a future where we may be able to create environments around a specific task in VR that would outperform the face to face environment?
[00:48:17] Speaker B: I think in principle it's possible, I would say because if we know for example what cues are important for pro social behavior, we can maybe highlight this cues somehow.
And if this doesn't work for sure you can do some training. Because if you have the number, let's say our heart rate synchronization is, I don't know, 25 or something, you know, like. And then we can test different things how we could improve it and can see in real time the number changing. So we could I think potentially improve. But this is a really interesting question. I think with the flexibility of choosing an avatar and choosing what cues you present and in which way and VR, I think there is a potential to make it even better for specific tasks. If you know what the task is, I think it could help. So for example, now there's a lot
[00:49:17] Speaker A: of,
[00:49:19] Speaker B: let's say companies that produce mechanical parts where they train their employees with virtual reality because it's really first of all a good task for virtual reality because you can really practice dismantling or assembling something and then you can do really good training, standardized training within a relatively short time. And this works supposedly very well.
[00:49:51] Speaker A: So.
[00:49:51] Speaker B: So I think for very specific tasks, I think it's similar for social interaction for very specific tasks, I can imagine that it will work better in VR than in real life. But I don't think in our experiments that we will ever find more synchronization in VR. I don't know. But even if you have facial tracking and eye tracking. But let's see, that's the most exciting thing about research that you have some hypothesis and you do an experiment and then you find out you completely wrong.
[00:50:23] Speaker A: Well, here's, I'll kind of leave you with this as a funny example of what I was thinking with your article. So I love snowboarding, and I've been a snowboarder for 25 years. I'm kind of getting better at it in my old age, and I'm really working on doing a 180 jump. And I was supposed to do that this season, but never got up the courage to do that. However, in my VR snowboarding game, which is weirdly realistic, I can do a 720. Now, what I need you to do is to help me create an environment as a trainer, as a therapist, as a coach, where I can be like 10 times better in VR than I am without VR. That's, that's where that, I'll give you that analogy as a challenge is like, hey, I'm such a better snowboarder in VR and I understand exactly, you know, because I don't. I'm not going to go to the hospital unless I run into the wall because I can't see where the wall is. And there's some differences there. But I think it's an interesting analogy to think about where, and I think folks in my field are guilty of this in the mental health space is we own this. This is a biological, psychological, BioPsychosocial system where VR technology doesn't have any place. And then Covid made us realize that virtual training is not horrible. And I actually prefer it because I don't have to drive 45 minutes in traffic to see my therapist, so I can just jump right on, do the session, jump off, and continue my day.
And I know we're not anywhere close to this in reality, but I would just like, why could we get there? Because when I'm face to face with you, I don't know your heart rate variability, I don't know your emotional regulation, I don't know your heart rate. So I just find it a really interesting way to think about it is, hey, I'm a 10x better snowboarder in VR than I am in real time. Now, I enjoy being on the mountain a lot more than I enjoy putting on the VR headset. But at the same time, I think we all need to drop our preconceptions of what this technology might do for us. And I'm going to own my own speculation here with that. But it's a fun, I think, way to put it, is what is possible with this amazing technology that is evolving.
I think more rapid than affordable hardware will allow us to, you know, kind of experience right now.
[00:53:12] Speaker B: I think it's always a fine Line the difference between what happens in virtual reality and in the real world. Because of course, I mean, it depends on your goal, right? If you want to become a better snowboarder, then it might help your motivation. Yeah, like if you are good in a virtual reality, but if you want to learn real skills and you have to study also how this transfers to the real world.
And yeah, but there are specific use cases where it's really good. For example, also the Germany soccer is very important. And for example, the soccer teams, they also use this technology to training, right? Like, because if you do strategic training, then it's different if you see it on a screen or if you see it from an egocentric perspective where they put the player inside a specific situation and tell, okay, you have to run now here and not there, right? Like the trainer can manipulate everything. And this seems to be really effective because otherwise not all the big sports teams would use this type of technology also.
[00:54:17] Speaker A: Right? And it goes back to my snowboarding analogy. I learned, and this is really wonky in the snowboard, I learned switch, which if you're not familiar with that term, you usually have a dominant foot in snowboarding and switches when you ride your weak foot. And you would see, you would think, hey, I'm just switching feet. But what I found out is it takes you. I'm an expert advanced snowboarder. It takes you back to your first day on the mountain learning a whole new skill, weirdly that you can just turn yourself around and be really good at something. You're really terrible at it. And so with VR in my game, I just practiced really boring runs. Switch. And while it didn't make me an expert switch rider right away, I got the feel for it in a different way that, you know, I wonder, like with communication skills or hey, I'm training in front of a thousand person VR audience of avatars of maybe people that, you know, should say, I generated avatars.
You know, it still gives that skill development and you know, it just opens so many fun opportunities that I think a lot of people don't aren't open to because they have this, they pigeonhole VR as this thing that maybe their kid puts on and you know, breaks their ceiling fan or something like that. You know where it is, I think that your articles and your research are opening up. No, just the spatial dynamics of a relationship might contribute to synchronization in ways we couldn't really look at before. And how do we set up an in person meeting? Do we have a table between us? Is there, you know, it just allows so much AB testing as well that I'm really excited to follow your work moving forward.
[00:56:16] Speaker B: Thank you so much.
[00:56:17] Speaker A: Awesome. Well Stefan thank you as always. I'll put the link to the article because it really is a must read for the HRV nerds out there because like I said I love that you included this variable. I hope it sticks in future research that you're doing and when and if it does I would love to have you back on to talk about it because yeah like I said I just you blew my mind and yeah yeah I had so many questions and excitement about this conversation. So thank you for your work in this field and I look forward to following it in the future.
[00:56:58] Speaker B: Thank you so much. I really enjoyed this conversation. Thank you.
[00:57:01] Speaker A: Awesome as always. You can find show notes links to the
[email protected] and as always we'll send you. See you soon.
[00:57:11] Speaker B: See you soon. Thank you so much.
