In the course of gathering intel for my new book I needed to get an informed view on the rather confusing terminology of the various digitally mediated realities that we’re seeing into these days; VR, AR, MR and XR.

So I arranged a chat with John Rose-Adams who works at XR Stories and sees many of these projects before him in a days work …

Hey John! I know what VR (Virtual Reality) and AR (Augmented Reality) are. Then MR (Mixed Reality) and XR (Extended Reality) are a little bit harder to define, so I wonder what your take is on those?

In mixed reality (MR), there are devices here already, HoloLens and Magic Leap are two, where you’re layering realities on the real world. So if virtual reality (VR) occludes real reality and replaces it in your full field of vision, with a virtually created world or a 360 video, so everywhere you look is a new reality, then mixed reality enables the real world to be one of the layers that you can see and then you can variably occlude and interchange with virtual world layers.

The ultimate end point of this in sci-fi, is that in people’s eyes is the ability to have mixed reality. And so adverts come out of walls and the world you see, appears to be augmented.

Currently it’s wearable glasses, so the off position is that it’s just a piece of glass and then projected onto that glass are other layers of spatialised reality that you can mess around with. Magic Leap are really fun. They’re really fun in the sense that you predominantly are seeing the world around you, it’s like being inside an iPad, but the iPad has completely disappeared. So your head is inside the technology, but the predominance is the real world that’s around you still.

Partly the way they work is like all augmented reality (AR) devices, they’re scanning the real world and working out where the planes and the edges are so they can place things. You can have a character appearing to be dangling their legs off a real table. And if you walk around the table, they’re in the same place on the table as you move around.

And then extended reality (XR), I always think of it being the ends of the full parentheses of what these technologies really are. So, things like pure audio, immersive audio, for example, that works off-screen, there is no screen mediating that kind of stuff, but that’s got to be included in what we’re talking about by extended realities. Then the internet of things, you can have your Alexa doing extra stuff with you, chatting to you, you can be part of stories mediated by those kinds of devices. I think that’s all in the mix. So XR is a non-exhaustive category of everything that doesn’t fit in the central VR, AR and MR categories.

A couple of years ago I googled the term XR and there were two definitions. I actually liked the one which turns out isn’t correct. So XR, is mostly understood as being extended reality, but I prefer the other definition, in which the is X is an algebraic X, so it’s interchangeable for a value. So the X could be V, A or M, or any other interchangeable thing which describes as something reality. And I thought that was preferable.

It’s all converging. I wonder if the “X” will become the most useful descriptor, it is gaining cognitive currency as a term of itself. Because like I said, virtual reality is heading towards mixed reality, augmented reality is only like it is because of the screen and the technology that has been available. Once the technology becomes ephemeral and the screen is so close to your eyes it is a set of glasses, then essentially that is what mixed reality is now classified as.

They’re all converging to sort of similar things. And aspects will become redundant over time. So VR won’t be the same version as we know it now. And neither will AR.

Have you got any good examples of each one or what you think is interesting in those formats?

Virtual Reality

It’s interesting to know where the best examples flock to in terms of technologies. I think video games have a natural home with virtual reality. You can get games as amazing as the “Half-Life Alyx” game, which is based off a wildly popular cult first person shooter and puzzle solving game, but in VR it’s properly brilliant and creepy and immersive in a full game sense. Whereas VR also sometimes attempts to be a virtual work environment in certain products but they are not necessarily enjoyable as a way of doing that task. Video games in VR. Cool. Other scenarios, not necessarily…

It’s a natural kind of progression. Because if you’re a gamer, you might have an amazing monitor in front of you, which is ultra-high resolution, as wide as you can to get as much of your field of view on that screen, possibly even a curved screen to really bring it to you. Well, a VR headset, it’s that, but it’s all around as well. So there’s a natural progression towards VR in video games.

It’s changing all the time though. VR is becoming mixed reality because that was what its end point always was. The fact is that the first VR headsets had no visual pass through, now all headsets have some visual pass through and the real top end ones they’ve got full pass through, so you can put it on and it’s mixed reality because the camera’s on the front are full resolution colour cameras. So you’re looking at what’s in front of you in real detail as well as the full resolution virtual environments.

Mixed Reality

In terms of mixed reality, where Magic Leap pivoted towards as a company was away from media and entertainment and games applications, (although they are still part of that world), and towards enterprise applications. Things like training in high risk environments, how to fix an oil rig from the outside without actually dangling off the side of an oil rig, army military type applications, and also, things like architecture – on site you can overlay a 3d CAD version of what you’re building onto the space that you’re working on, you could be doing complex wiring systems in factories where you have to really get everything in the right place and there’s multiple people, multiple trades working.

Those enterprise applications seem to be where MR and by example, Magic Leap as a company, are heading. That technology has explored what home it might have and is heading that way.

The sci-fi side of my brain says that the personal version of mixed reality in the future will be being able to personalise your entire world, like choosing to make every object you see in a day pink, choosing to replace the head of everybody you see for Boris Johnson’s head for a day, because you can, and with audio, you can also make it sound like Boris Johnson.

It’ll be fun. A form of entertainment in itself. And if the personalisation tools are really cool, then you can genuinely do it how you want, you can create worlds that other people like to enjoy

Or you plaster adverts on every conceivable surface and sell real estate around the world so that people with headsets will see it, all that kind of jazz. But at the moment it seems to be that enterprise is really going with it, you can kind of see why, if you’re training a mechanic and they can open a bonnet and the headset can help explain where the things are, get to the source of problem quickly, you can call up the schematics, whatever technical details are needed, it tells you which wrench to use immediately, all that kind of stuff.

Augmented Reality

And then augmented reality, seems to be trying to find its home or at least mobile device bound AR is. I’ve not seen any really obvious, compelling story applications for it. It seems to be an interesting way of augmenting location-based experiences but then immediately you come across the problem that you really need 5g in a location to give you enough data and bandwidth, to be able to kind of make something super interesting happen in that moment for lots of people.

I feel like AR’s the most ubiquitous, it’s on everybody’s mobile device more or less, but it’s the least useful, or hasn’t found its most obvious home, but maybe I’m doing it a disservice.

With “Pokemon Go” as an example that is used a lot as a really successful piece of IP that uses an AR and geo location really well. But Niantic themselves then really struggled with another successful preexisting piece of IP, the Harry Potter AR game “Wizards Unite“, and the reviews of that were really mixed. In theory they were hitting the same sweet spot, a massive fan base, successful (and already proven) technology, but it didn’t take off.

I had to butt in here… Krish: I’d love to do stuff via a mocap suit transposed into a cartoon character that sits on people’s floor and just talks to them, it doesn’t look like me or sound like me, but it’s me doing it. (Or, if I’m terrible at it, get an actor to do it), but just being able to put things into people’s own places will have a big impact.

And so all of a sudden, you’ve got this personal relationship with something virtual, a media thing that’s happening and it’s in your space. And these are new types of kind of emotive experiences. Like in the recent Wallace and Gromit AR game “The Big Fix-up”, having a little widget running around in my conservatory or a rocket taking off, I was completely bowled over and it’s not super complicated to do that, but when it works, all of the potential is realised in an emotional type of response, which is fantastic.

… back to John:

Then make that AR on a mixed reality headset, a really light set of glasses that are for all intents and purposes, no more cumbersome than a set of optical glasses, and Alexa being a virtual person who is in your house with you, walking around and when you’re in the kitchen, you can beckon them over and ask them a question or get them to help you with cooking. All the things that at the moment are just audio, it doesn’t need to be, so there’ll be a future version of Alexa, which will be a metahuman type AR thing. And because you’re wearing mixed reality glasses Alexa is completely just there with you.

At the moment you could do that with a mobile phone, a handheld screen, the reason you would never think to do that is because it’d be an absolute faff. if you’ve got the headset on you just look. And so, that would equally work with cartoon characters or your favourite movie star –  without going too Black Mirror, the technology is very able to do this kind of stuff because it’s what it’s designed to do. Where will we go in the next few years?

What are your thoughts on interactive storytelling in these areas?

Like at the moment, I think the most fun, interesting interactive stuff, is not necessarily narrative and is more experience-based stuff like that you can go and see Mix Master Mike playing a gig in a virtual space, stick on a VR headset and go to your gigs like that now.

And that’s partly COVID, but also partly because you can play a gig to 2 million people like that, whereas you were limited by tour dates and venue capacity before that. And the stuff that happens in Fortnite, Marshmello, they did a gig for millions of people. So there’s all that sort of interactive, communal social experience type stuff.

But on the interactive storytelling side I think it’s really interesting that VR experiences, even though they’re created by game engines, are working out what interactivity is when it’s not a game. So if you watch some of these award-winning VR shorts that come through Venice and other film festivals, they are mostly lean back and they are great, they’re very immersive, but they’re not necessarily interactive, and when they are interactive, they’re quite economic with what that interactivity is.

I played a preview of “Goliath“, which is a documentary piece of VR, that’s been developed by Anagram who are based in London. It won the VR strand at Venice film festival. It’s a documentary about the experience of an individual who lives with schizophrenia and has found a home in the video games community. So it’s a straightforward documentary subject, but the treatment that you can give it in VR means that it can be so many things all at once. And so the experience is this constantly evolving and changing visual interactive world where the story is linear, but how you mess about with the story and do things whilst it’s happening, enhances what you’re hearing.

So you don’t change the outcome. There’s no outcome to change. It’s somebody’s story that they’re telling you, but what the technology enables you to do is change the experience. And it’s really cool. So there’s bits where you play in an old style arcade game, which is beautiful. It goes through various video game genres and there’s a Streets of Rage bit where you have to avoid these people. You have to avoid them because you don’t want to engage with them because you’re feeling insular at that point in that story, and it just keeps ranging. And it changes to a mad psychedelic DJ thing where you’re on the decks and there’s all this stuff going on, it’s just visually incredible.

And all of that stuff is where we’re at right now, interactivity can be many different things because media is so diverse, it’s not just a video game, it’s not just TV and it’s not just a book and it’s not just an information delivery system. It’s everything all muddled up at once. All of what interaction means and can mean is all jumbled up as well. And we’re finding different threads and figuring out – how do we connect them together? And what is fun and engaging in the way we traverse all of this stuff?

What’s winning this year at the “John” Awards then?

I really enjoyed Goliath and in my John awards ceremony, the VR category goes to Goliath, incredible. Then, what Gazelle Twin and Kit Monkman Associates and the University of York were able to do with the Zoom medium when they did the “Absent Sitter” prototype at the York Mediale Festival in 2020 was really incredible. And I thought that was a great example of creative constraint. So with what they had to play with, which was a Zoom, and what they managed to get out of, it was quite staggering to me. And it was everything I’d want in terms of immersive and unsettling and thought provoking and just great. They’re bringing an iteration of that project to ARS Electronica, so you might be able to catch it there. And it it’s evolving, because that’s the nature of these things, they’re improving on it and they’re exploring a sort of a hybrid present audience and remote audience aspect to it. And one of the reasons why immersive experiences like this one resonate so well is because they also speak to the medium that they are working in. They were exploring the nature of presence and absence and performance and the nature of contracts with the audience member and then used the metaphor of a seance, which of itself is about absence and presence and theatre and suggestion. And then that becomes what you then receive when you’re participating in that thing. So it’s great. It’s talking to its own kind of mechanisms and messing about with them, and I love that kind of stuff. That’s not AR, MR or VR, that’s just screen-based wizardry, sonic wizardry and storytelling ability. Stereo sound was probably the most complicated thing. So that gets the John immersive experience award, it was very screen mediated, but pretty cool interactive storytelling.

I’m also interested in the trends, so what Swamp Motel are doing is cool. And also Darkfield Radio. The emergence of a ticketed scheduled event, so it’s got a theatre like quality to it, I buy a ticket and enter into a transaction. I’ve got a time I start, it may or may not be mediated by live performers or whatever. And then what you get, is something that could never be done in the theatre or could never be done effectively in a video game really. And certainly isn’t what’s on the TV. I think is really exciting because it’s another strand. These types of things have been accelerated and made more acceptable by COVID and us being remote from each other, but things where you can bring one mate or six mates from wherever they are in the world together for a story. I think they’re brilliant. And I think that as a trend is really, really amazing. We should see more of it because it’s very doable and it’s very creatively fulfilling. So the trend award goes to Swamp Motel and Darkfield Radio!