Term paper, Fall 2016, for Foundations of Human-Computer Interaction course taught by Hamid Ekbia at Indiana University, Bloomington. 

 

Why am I here?

Being an Entity in Virtual Reality

 “Your job is partly to anticipate, partly to control the thought processes of the audience. To give them what they want and/or what they need just before they have to “ask” for it — to be surprising yet self-evident at the same time. If you are too far behind or ahead of them, you create problems, but if you are right with them, leading them ever so slightly, the flow of events feels natural and exciting at the same time.” [1]  ~Walter Murch on editing in traditional cinema

 

 

Story happens in the mind, the canvas is between the ears.

You strap on a virtual reality headset and headphones. The video starts. Silence. And then a soft sound brushes along in your left ear. A scene fades in around you. Where am I? What’s going on here? What am I supposed to do? What can I do? You scan the wild west around you; a new frontier; a land of opportunity. You are an adventurer here, or maybe a bandit? No, you were born to be a gunslinger. As you stumble through the thickets of this exotic trail, your journey unveils.

Audiences today are so sophisticated that they can absorb several flows of information gushing in more rapidly than ever before. They desire for a more intimate connection to their entertainment. Virtual reality is the natural evolution of this need; the natural evolution of story. It exists as an enticing release to capture their unbroken attention, a flight from materiality like never before experienced.

 

Every time the audience puts on a headset to experience a virtual reality creation, they’re letting the medium put them into an altered state, like a digital drug. With the right sounds, the right visuals, and more, an illusion can be held making the experience a memory that can become part of you. This is the clear divide between experiences in virtual reality versus television, film, and all other artistic mediums for that matter. Film is where someone can share the story of an experience, whereas virtual reality is where you can give someone an experience that they can generate their own stories from. The core of virtual reality is an experience and presence. Nothing else. It’s fundamentally about being somewhere. Other mediums are a window one could glance away from, a door you could shut.

 

i. An experience can be defined as being able to be articulated or named; has a beginning and end; inspires behavioral and emotional change and can also include co-experience where there is creation of meaning and emotion together through a product’s use [2].

ii. Presence is the feeling of the sensation of being somewhere [3]. To be truly immersed the valid actions supported by the system and the actions that a user can perform should behave in the way you expect them to.

Telling a story

Stories are told through tons of different mediums and the craft has been in a state of evolution as long as we have. Cave paintings, graphic novels, theater, opera, books, movies and games among many others. Any good story transports audiences through its storytelling. Good pieces of entertainment transport audiences through their visual storytelling. Many creators would say this is their language with an audience, and they’ve developed this vocabulary and become accustomed to it. In a sense, they’ve always been creating virtual realities. Visiting a virtual world can overwhelm you with possibilities. The deep, creative implications of actually feeling like you’re somewhere else are transparent, evocative, and thrilling. How do you create a beautiful experience that brings meaning to peoples lives?

The possibilities start when the audience enters the world rendered before them and begins to explore and manipulate a malleable space. So the challenge is for storytellers to create an innovative piece of entertainment that unconditionally immerses and instills wonderment in audiences for the duration of the story’s performance. But what is the right storytelling that will create absolutely suasive entertainment in virtual reality’s current embryonic state? This presents untapped storytelling models that are epitomized by an existential question: do you exist in this world and can you make a difference?

 

What is my purpose?

To exist in a virtual reality world means you have what is known as agency, or control. The degree of agency you have determines how much of a difference you can make in the virtual world. Agency comes in two forms in storytelling: local agency and global agency. Local agency can be thought of as the moment-by-moment changes you engage in: talking to people in the world, spilling a cup of coffee, exploring. It’s basically plausibility, meaning that all suggested affordances give all possibilities so if it looks touchable you should be able to touch it!

But what if that spilled cup of coffee belonged to a certain character and they noticed? What if not only did they notice, but it affected them so much that they stopped what they were doing and engaged with you in an upsetting manner? What if this engagement caused the story to unfold differently, just one spilled cup of coffee? This is what effect global agency has. Each moment-by-moment change can add up to significant changes in the story.  

 

In Coming Out Simulator 2014 each part of the story can be either local agency events or could potentially lead up to you changing the story completely all through text dialogue choices!

Another interesting story where you engage with high agency is Façade, an interactive drama with artificially intelligent non-playable characters (NPCs). It’s a short experience that promotes being played over and over again trying different strategies with characters who feel real and is one of the only interactive drama games that has natural language input offering both local and global agency to the player.

So going back to the question, “do you exist in this world and can you make a difference,” in regards to virtual reality means for very different models of storytelling depending on the purpose of the narrative, whether it’s for thrills, empathy, joy, wonder, drama, comedy, etc.

You do not exist and you cannot make a difference!

You enter a virtual world, but find you are like a ghost; you simply do not exist to any of the characters in the universe around you. This has been the standard model for the majority of filmed entertainment, but its use in virtual reality raises newfound complexity with the audiences’ relationship to entertainment. The audience is not permitted trespass into the virtual space, unable to exert any influence on anything. The storyteller is an omnipresent magician weaving the story under their control, all the action and exposition only but granted to the audience upon their bated breath. Entertainment content also transfers easily over into 360-video and virtual reality, but this does not imply the audiences’ physical presence digitally or permit them any influence. You have neither local nor global agency. You can look around anywhere, but where you look makes no difference. None of your actions change anything about the experience.

 

newwavenewwave

 

Some simple innovations can start to enhance this style of storytelling though. They can give the sense of making a difference like using spatial audio to vastly change the experience. In the live action virtual reality experience New Wave, found in the Within app, the story unfolds around a couple at the beach and depending on whether you’re looking at the conversation to your right or to the left will determine which conversation you’ll be able to hear. So even though the video will be the same, you can get an entirely different experience of it based upon what audio you hear. This alone could essentially transform an experience from you not existing and making no difference to, seemingly, making a difference.

 

You do not exist but you can make a difference!

Now, let’s say the piece of entertainment gives the audience some power to affect the outcome of the story. This means of giving options to the audience converts the experience from a passive to interactive one. It’s almost like bestowing god-like decision making powers upon the audience. However, this is an illusion of influence since the audience is limited by the predetermined set of choices laid out by the storyteller who has also in turn laid out a series of outcomes. This mode of storytelling exists in the “choose your own adventure” style of stories.

 

In Duet, a 360 animation by Glen Keane, this is done more subtly as the audiences’ gaze dictates the progression of the narrative. These gaze-based triggers do have limited interactivity with exploration, but this interactivity has no real consequence to the story that’s unfolding. So in this sense you are exploring an environment and impacting how you personally experience the story within the local agency constraints, but if this doesn’t change the story at all then you have no real global agency as to how the overall story unfolds.

 

Another interesting example of this style is an immersive Broadway theater piece in New York called Sleep No More where the audience can explore 100 different rooms and choose which of the 21 parallel stories to follow, but their choices aren’t making a difference to the overall story arc of the actor’s actions. The audience does not exist within this realm of the retelling of Macbeth through interpretive dance.

 

You do exist but you cannot make a difference!

The state of virtual reality has taken great liberties with this form of storytelling. It is relatively new where the audience exists in the world as a sort of plot device, but nothing is asked of them, and once again, nor are they permitted to act or tip the fate of events. The audience isn’t given any choices to change the outcome of the story, but yet they are addressed as if they were a character within the story. These experiences are told usually from a first person perspective where others are directly addressing the audience and provide a voyeuristic experience that can transport you into the shoes of another person, but yet you’re essentially a mute with no way to respond or react within the story. This style of storytelling provides ample room for experimentation. For example, the audiences’ gaze can shift between points of view so they can potentially be in different spots of a crowd at say, a concert or sports game.

 

In Invasion, you exist by having your presence be known to the character, the white bunny. While there is still no local or global agency, the style does promote immersion into the story through a sense of wonder! 

There is also a huge market for horror film makers to utilize as they can give a thrilling experience more intimately than ever before. There are numerous examples of horror in virtual reality that utilize this storytelling style excellently.

In Guy Shelmerdine’s virtual reality experience Catatonic, the audience is placed in the shoes of a wheelchair bound patient as they are wheeled around a psychiatric ward of a 1940s era hospital. In some cases where the developers are showcasing the film, the audience can actually sit in a wheelchair with added surprises while wearing the headset for further immersion.

 

In the Insidious 3 trailer for virtual reality, the same principles from Invasion are seen. This first-person style of storytelling is a logical step from the standard model of film. From here, some interaction can be given and the audience can feel like they’re in the world.

 

In Terror Film’s virtual reality experience The Chosen, the audience awakens in a dark room with the lights flickering as supernatural forces are made wary of your presence. It’s quite a thrilling vignette of potential for this type of storytelling in virtual reality.

 

This has also been used in creative ways outside of virtual reality, too. In House of Cards, Frank Underwood sometimes casually looks into the camera as if talking to (note, not with) the audience. He addresses the obscure audience, not you as a character or a plot device in the shows universe. It’s similar to when a narrator speaks to the audience like in Shakespeare’s plays, for example. This style is also used commercially very successfully by amusement parks where you are literally on a ride as part of the characters’ journey.

 

You do exist and you can make a difference!

This is the most transformative combination of existing in a world and being able to make a difference. The universe acknowledges your existence as you interact and you are bound by its laws and underlying story. You have the capacity to veer from the set story and it will reset, adapt, or even evolve based on if it as a free form or rigid story according to the storyteller’s intentions. Every action the audience takes, no matter how small, causes something within the story to react to you, but also your influence can be crucial to how the overall story unfolds. This might just be the holy grail of virtual reality experiences where you’re interacting with authentic artificially intelligent characters, and the nature of these interactions can be contributing and weaving towards a vastly different range of outcomes. 

This area of storytelling has flourished in the arena of video games where both local and global agency has manifested. Façade, mentioned earlier was an example of this end of the spectrum.

 

Chrono Trigger, by Square Enix, is a fantastic role playing game in which small actions can build to a butterfly effect (quite literally in the story of this game) and diverge into over 20 endings for the game.

 

deusexdeusex

more: https://en.wikipedia.org/wiki/Deus_Ex

Deus Ex, by Eidos, was one of the first of its kind to incorporate an extreme level of detail to local agency in 1999. Not only was local agency detailed, but your actions could change how entire factions of characters react to you. Making a mistake like siding with one character or another can significantly alter the course of the game and its ending.

 

How do you create a story?

Achieving presence

What sets virtual reality apart is this idea of presence, the feeling of having the sensation of really being there. The above storytelling styles present an experience in different ways to achieve presence. This creates an entirely new set of opportunities and especially creative constraints. Those constraints are that stories must support the medium. Every piece of content created for virtual reality has to ask the question, “Why VR?” as part of the development process. Why is this story better told in this medium? What can be done to leverage the strengths of virtual reality? So creating content for virtual reality needs to take into account how and why these environments, characters, and situations are better suited for virtual reality then they would be anywhere else. To work out the challenges of telling a story in virtual reality, content creators can use everything from using traditional film techniques to scene staging and the many ways to grab the viewer’s attention. The ultimate goal is to build a truly immersive cinematic experience that was never before possible.

Storytelling has a new vehicle and the destinations and journeys have exciting possibilities, but how do you start actually making a story in virtual reality? You would need to have the interactive mechanic you’re employing harmonize with the narrative. It’s hard to do this with a script and storyboard since they don’t take into account presence, which includes interactivity, variable points of view, branching dialogue options, etc.

 

perspectivesperspectives

 

A great example of a story utilizing interactivity well with the intention being to promote empathy was Perspective by Rose Troche. She utilized virtual reality to tell a story about perspectives giving the audience a direct experience of four different points of view revealing how our perceptions and biases can shape our eyewitness testimony of a police shooting.  Each individualistic mindset of the characters gives a variation of the overall story where a police shooting occurs between two officers and two black teens. To get the objective perspective, then you must watch all four stories, and to further complicate the nature of the narrative, the order in which you watch these stories can drastically change an outsiders view of what happened. The goal of this was to leave audiences’ questioning their own biases and really empathizing with each of the faulty characters.

 

Embedding yourself in another character is a very powerful use of virtual reality, which can then give you a new perspective on your own life. This is a step beyond other mediums capabilities, and the ultimate potential has yet to be explored and is unknown [3]. Watching a movie and merely witnessing a scene unfold evokes a different quality of experience than when being directly addressed as a character in the story.

Eric Darnell, of Baobab Studios and creator of Invasion mentioned earlier, gives his take on the process for creating their short animated virtual reality experiences. Moving away from empathy, he feels there is a sweet spot between interactivity and storytelling to generate a childlike sense of wonder, awe, and inspiration. [4]

Before you start working on anything, he says you should be able to pitch a concise 1-2 sentence description of what you want to make to anybody and if they react well then it’s worth working on. He gave an example of the DreamWorks movie he worked on, Antz,

“there’s this colony of Antz and they all think the same, except one ant who wants to think different...and he’s played…by Woody Allen [4].”

To map a story utilizing virtual reality you need to make the leap from abstract 2D to 3D. Eric says to just start by sitting down and writing. Draw the story, sketch and collaborate heavily. Get feedback and then circle back and try again. Think about the architecture of the story; is it long or a short story? The longer it is the more the pieces have to connect and production could take years. Iterate until you see something forming. Are there themes emerging? Fine tune things to the themes. Don’t start with a message right away going into your story, but rather ideate on something exciting and compelling for virtual reality. This process should be set in stone before you even create the first frame. The connection needs to be made between what you’re trying to do with your story and how the mechanics of telling that support it. Ideas don’t necessarily have to fleshed out. They can be vignettes that you test and get critique on. One example they do at Baobab for presence in their stories is the bat test: if someone swings a bat at you and you duck, then you felt presence! 

 

greyboxinggreyboxing

 

Oculus’ Story Studio goes about actually testing interactivity as part of their creative process [5]. They prototype using software, or as they refer to it “grey-boxing,” which speaks to the fact that mechanics are tested using simple grey shapes as proxies for sets and characters. The goal is to isolate and improve the most fundamental aspects of the audiences’ agency. These prototypes are vignettes of presence, but they inform big concepts like whether a space is fun, challenging to navigate, or uncomfortable in some way. They then go further and test using higher fidelity prototypes. This solves for emotional presence in testing as, 

“every layer of immersive texture we add seals another leak from reality [5].”

Everything in a virtual space relates to how characters in it act, what they say, and how you interpret those actions. In order to derive meaningful conclusions about the direction of a narrative, the creators have to be emotionally present to observe results, which can be difficult in a world made of grey boxes. These tools and techniques are more about helping storytellers figure out the strengths and weaknesses of telling a story within the medium of virtual reality.

 

Rob Morgan of GameStory offers further insight into achieving this level of presence. His ideology is that,

“less is more, and in virtual reality less is even more [6]”

He says that the experience should talk to the audience less by supporting their experiences rather than narrating them. Characters in the world have to act like real people and engage you on a human level both emotionally and socially. There are whole levels of social and behavioral interaction that we expect to have with other humans and so you can’t treat characters in a virtual reality experience the same way that you might in a 2D experience. For example, there are social cues that you’d expect a human to react to based upon where you’re looking, whether you seem like you’re paying attention or if you’re threatening other people in some way. Voice acting and graphics go beyond just their look and feel in virtual reality as humans are really great at detecting incoherent things, or fakeness. It is a vital element of immersion that the acting and world be believable rather than unauthentic. There’s a whole range of interaction that we demand and expect to have, and with these additions we could add another dimension of immersion.

The costs of interaction & presence

While interaction seems intuitively tied to an amazing virtual reality experience, there is a trade-off that needs to be treaded carefully. Both Oculus Story Studio and Eric Darnell believe that if you’re paying attention to the narrative in front of you then you’re far less inclined to be absorbed by your surroundings and allow yourself to be transported somewhere else. On the flip side, if you’re too swept up in the environment and the spectacle of virtual reality, you find yourself paying a lot less attention to the narrative action. What is the sweet spot where you are present in the moment and also have a level of investment in the characters and the story?

 

lostlost

 

Oculus Story Studio’s Lost is a good example of their journey to find the sweet spot [7]. When audiences’ entered their experience they often missed the first couple of minutes of the story because they were inspecting the surroundings, or gazing up at the detailed moon in Lost’s case. In response, they extended the opening sequence by a minute to give audiences time to absorb into the experience and acclimate to the virtual reality. They then had their character directly acknowledge the audience at one point, which in turn had audiences’ feel more connected with the character and their reactions reflected curiosity instead of hesitance. They concluded a storytelling model where the audience exists, but cannot make a difference was the course they wanted to go on. In their initial iterations, they employed the storytelling model where the audience doesn’t exist nor can they make a difference. As they put it,

“this absence of viewer agency creates an invisible wall between you and the virtual environment. The actors ignore you, the world remains indifferent to your presence, and yet you are so undeniably there…it doesn’t feel right [7].”

The viewer has to be an entity in a scene, but should they have the ability to directly affect the world if it would benefit the sense of presence? If there are interactive elements, how are they conveyed to the audience and also that it doesn’t feel gimmicky? And how are the limits demonstrated? What if there’s a story going on and audiences’ are given the ability to pick the leaves off bushes or touch an insect all while a story is being told in front of them? The problem to solve for presence is establishing relationships between the narrative, the environment, and the audience to come together.    

 

halflife2halflife2

 

A game that had this sort of narrative before virtual reality was Valve’s Half-Life 2, a story-driven game where there were no cutscenes and the story was told entirely from a first person perspective throughout. In order to guide the player, the game utilized the player looking at particular things based on lighting and sounds. Every level was designed in such a way that the story and environment complemented each other. The game was redeveloped to run in virtual reality.     

 

The Magician behind the scenes

A storyteller can never be 100% sure where their audience will be looking, but they can “partly anticipate, partly control the thought processes of the audience [1]" by evaluating the entirety of the experience, stepping back as an objective observer of their design. They let the flow of their creation take them on a journey through the overall experience. One trick of doing this is working backwards. Storytellers can decide on the outlined journey beforehand and then, just like a storyboard, break it apart only to find threads that can help them craft these worlds together from the start. Cues are the hooks to the threads as they are used to focus the audiences’ attention like sprinkling pieces of music, deliberate sound effects, sensations of touch, color play, animation and more. The storyteller that constructs the experience must face the harsh reality of their work and ask what will a naïve audience do naturally in this world?

“Put [yourself] in place of the audience. What is the audience going to be thinking at any particular moment? Where are they going to be looking? What do you want them to think about? What do they need to think about? And of course, what do you want them to feel?” [1] ~ Walter Murch

 

Some of the best examples of this in modern entertainment have to be Uncharted 2 and Journey. They are exemplary in terms of cinematic storytelling.

You awake on a train hanging vertically off a cliff in Uncharted 2. You scurry to get out and reach solid ground as the game plunges you into interactions with the events unfolding for pure progression of the narrative. Navigating through the game’s interfaceless screens requires cues from specially triggered dialogue from characters, a subtle bell ringing upon glancing at an interesting path, the terrain almost pointing towards a destination; all converge to seamlessly guide the player throughout the story. The interactions from the player are limited in the grand scope, but the pure awe and adventure the game provides visually pulls you in the best it can into a 2D screen. It is a peak into the potential virtual reality can have if the medium was transferred to the 3D.

In Journey, there is no dialogue, no text, and no interface of any kind. Just you, alone, in an endless sprawling desert. You are guided simply by your will to keep exploring and moving forward in the landscape. Certain unique interactions with silent characters cue you to look away at another part of the scene. That part of the scene, that you previously disregarded, now becomes a discoverable element that you are drawn towards. The audience has a shared experience with the world and feels more part of what’s happening and are compelled to explore more and see what else happens instead of resisting and merely leaning back to watch. The creator could have just made a map or compass pointing to the desired locations the player should go, but then none of the flow would have happened. The experience may still have continued fine, but it wouldn’t be as dynamic or memorable.

 

 

The Canvas

Virtual reality from the audience perspective is an indulgence, and from the storyteller a creative opportunity unparalleled; the ability to hack consciousness in the waking world. The boxes that we put art forms into, painting, filmmaking, writing, can all be removed. Audiences put on the headset and give themselves an experience. It’s never the same experience twice; they come back over and over again. I believe they come out with a better appreciation for the beauty of life.

Virtual reality fuses the audience into a synergic relationship with the story, providing glimpses of new existences and perspectives that leap into the abyss away from the familiar consumption of our living room couches. The intrigue towards the world, its characters and story, the control you have over it, will require the audience of the future to engage alongside their entertainment with remarkable intimacy, with unforeseen emotional bonds and commitments. This profound reciprocity between storyteller and audience is an ongoing evolution of constant exploration and understanding, fueling creative rumination of stories.

Virtual reality’s innate transcendency is redesigning storytelling not on a stage, or words on a page, or behind a glass screen, but totally immersed within screen.   

 

References

 [1] Brillhart, J. (2016, January 12). In the Blink of a Mind — Prologue. Retrieved December 2, 2016, from https://medium.com/the-language-of-vr/in-the-blink-of-a-mind-prologue-7864c0474a29#.pp9fj3fkl

[2] Jodi Forlizzi , Katja Battarbee, Understanding experience in interactive systems, Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques, August 01-04, 2004, Cambridge, MA, USA [doi>10.1145/1013115.1013152]

[3] Slater, M. Place Illusion and Plausibility Can Lead to Realistic Behaviour in Immersive Virtual Environments. Retrieved December 2, 2016, from http://www0.cs.ucl.ac.uk/staff/m.slater/Papers/rss-prepublication.pdf

[4] Darnell, E., & Bye, K. (2016, February 2). Storytelling in VR & the Tradeoffs of Empathy and Interactivity. Retrieved December 2, 2016, from http://voicesofvr.com/290-storytelling-in-vr-the-tradeoffs-of-empathy-and-interactivity/

[5] Billington, P. (2016, April 20). Protovis. Retrieved December 2, 2016, from https://storystudio.oculus.com/en-us/blog/protovis/

[6] Morgan, R., & Bye, K. (2015, April 16). Rob Morgan on Narrative Design in VR & escaping the uncanny valley via interactive social behaviors in NPCs. Retrieved December 2, 2016, from http://voicesofvr.com/125-rob-morgan-on-narrative-design-in-vr-escaping-the-uncanny-valley-by-implementing-interactive-social-behaviors-in-npcs/

[7] Burdette, M. (2015, November 18). The Swayze Effect. Retrieved December 2, 2016, from https://storystudio.oculus.com/en-us/blog/the-swayze-effect

[8] Dolan, D., & Parets, M. (2016, January 14). Redefining The Axiom Of Story: The VR And 360 Video Complex. Retrieved December 2, 2016, from https://techcrunch.com/2016/01/14/redefining-the-axiom-of-story-the-vr-and-360-video-complex/

Image sources (Unless otherwise stated, images are taken by the author as screenshots from videos or gameplay or open sources):

House of Cards: http://www.bolumrehberi.com/images/tv-show/House-of-Cards/house_of_cards_wallpapers_1920x1080_03.jpg

Chrono Trigger: http://static2.gamespot.com/uploads/original/43/434805/3072122-0612653297-chron.jpg

Uncharted 2: http://www.videogamesblogger.com/wp-content/uploads/2009/09/uncharted-2-hanging-train-car-wallpaper-concept-art.jpg