Review and Retrospective

Over the past six months I have been working with Flutter and Firebase to develop a framework for a cloud connected interactive smartphone  expérience.

The app delivers virtual objects into the  user’s real-world space with AR. The user searches and collects virtual objects over time. Each find unveils new interactivity, for example an object may be examined, pressed, dialled or looked through.

This functionality is inspired by the choose your own adventure books and text based games for home computers I played incessantly as a child growing up in the eighties. An era that witnessed the blossoming of interactive text through which the reader becomes entwined within the narrative thus experiencing the medium through their own agency within it.

While traditional books are interpretative with a static linear route from start to finish, interactive books allow for a story to unfold in different ways based on the reader’s input.

Furthermore, with computer driven narrative, the text production becomes dynamic. Espen Aarseth, a pioneering figure in computer narrative studies, popularized the term ‘cybertext’ to differentiate machine generated dynamic text from other forms. Describing the phenomenological feel of cybertext as “constantly reminded of inaccessible strategies and paths not taken, voices not heard” (Aarseth, 1997 p.3)

Marie Laure Ryan, an authority in narrative studies, recognises a need to think of computer games as a distinct form of narrative, “we need to expand the catalog of narrative modalities beyond the diegetic and the dramatic, by adding a phenomenological category tailor-made for games.” Because classic literary narratology does not account for the experience of computer games.  (Ryan, 2001)

Douglas Brown, an expert in game theory and design, notes, in his 2015 thesis, that computer games requires wilful engagement: “Since generally refusing to use agency is tantamount to not reading the text at all, this fundamental interac tivity of gaming makes the gamer something other than an audience.” (Brown, 2012, p.12)”

My research in this area has led me to examine the nature of game narrative in the mobile computer setting whereby real-world spaces fuse with the phenomenological character of cybertextual narrative.

This technical experimentation and theoretical research has driven the development of Noirscape, a film-noir themed mobile app game in which the participant engages with time bleeds (a metaphor in line with the app’s Noir crime genre) where fictional artefacts bleed through the porous boundary of a fictional past and real present, across virtual and real space.

For the app, I developed a black & white vignette AR filter to further blur the boundary between what is fiction and what is real. The virtual artefacts cast convincing shadow and reflect light inside the real-world space. The user composes a scene and then saves a snapshot which serves as a unique piece of Noirscape narrative.

A 2015 study sought to explore the potential of AR as a “general people’s creative medium”. It found AR to be a good storytelling environment because of the way specific locations or real-world objects become subject matter and stimulate imagination.  The results from the study conclude that even simple AR narrative provides users with a positive playful experience of in-situ narrative creation. The  paper identifies a future challenge of expanding the AR narrative designs to allow users “intuitive creation and weaving of a story encompassing scattered multiple scenes and objects”. (Yanghee, 2015)

A 2018 piece in the International Journal of Human Computer Interaction notes how the technological development around augmented reality and its dissemination into consumer culture are still in its infancy but nevertheless found that positive user intentions are driven by the novel affordances granted by the technology. The paper suggests further research should anticipate more developer display devices to keep up with the rapid development between humans and perceived reality” (Hamari, Malik et al 2018).

It for this reason I opted to instead use 360 degree photography and video for Noirscape’s outdoor interactive flashback scenes which, following experimentation, I concluded were not satisfactorily achievable YET with consumer level AR.

Flashback is a common narrative device in classic film noir through which a first-person narrative sequence from the past is embedded in the present, the narrator is usually long since deceased yet speaks from the perspective of the present.

“Narrative is a… doubly temporal sequence (says Genette, 1980)… : There is the time of the thing told and the time of the narrative (the time of the signified and the time of the signifier).”  (Genette 1980 p33)

I had originally intended to work with a local actor friend, to film a sequence of flashback footage; however, we had to cancel our plans due to pandemic restrictions. So, I had to pivot and adapt.

I sought stock footage but despite searching many resources I was unable to find anything suitable. And this would only provide a finite amount of material from the same actors; licensing is very costly too.

I then turned to 3D animation software and digital character design, this was very new to me. Although it involved quite a learning curve, I have become comfortable with the software and can now build and direct my characters in as many bespoke scenes needed. Furthermore, I can deploy my digital characters and animations to other formats like marketing videos and even VR environments.

While 3D character feedback was highly positive; testers were negative about the digital voice over. Even though I had sourced a gaming AI voice service, users disliked the robotic quality. So, I sourced a professional voice actor online and forwarded the scripts and project details. I was very happy with the results and look forward to working with them on new content. The combination of real human voice with virtual character works well and my testers agreed.

The flashbacks are triggered by the geographical position of the mobile device. Clues provided through user interaction with the previously discovered artefacts send the player to hidden locations in their town where the flashback sequences are uncovered.

While my use of flashback is directly inspired by cinema; I came across interesting research discussing  flashback use in a highly acclaimed 2017 desktop / console game; the paper, titled “Interactive Flashbacks: A Case of Agency in What Remains of Edith Finch” presents the flashback concept as “creating a unique process of retelling of past events” . “A specific kind of narrative, in the ‘past’ of a gameworld ; pieced together through the player’s (spatial) exploration.”; and a “herenow which is the time of the spatial exploration”.  It is interesting to me because what I am doing is similar; although Noirscape takes the spatial exploration beyond the purely virtual.

The development of mobile technology, says Ryan, Foot & Azaryahu, GPS and augmented reality counters the tendency of computers to lure sedentary users into virtual worlds by replacing simulated environments with real-world settings and by sending users on a treasure hunt in the physical space. (Ryan, Foot & Azaryahu, 2016, pp102)

Through the activity of searching the physical space of a local town for fictional content, there is a dual narrativity taking place: the story being discovered and the story of its real-world discovery.

To give Noirscape some structured user progress feedback the app contains a puzzle feature called a stumper. The stumper involves the collection of enigmatic visual codes, which are attributed during the app experience. The aim is to complete the set, the stumper screen therefore serves as a gamified progress bar.

I am currently sourcing funding and supply for a physical boxed product to accompany the app. I am in contact with a local independent hobby store in the town where I am to launch the pilot. To access the app you must enter the shop and peruse the shelves until you find Noirscape, purchase it and take it home. A process, which I like to think of as part of the game, itself.

The box contains a set of physical stumper pieces, when the player has collected all the pieces within the game, the physical pieces can be assembled to trigger a marker-based AR experience; an animated scene to draw a close to the experience. As there can be a great number of combinations and positions for the stumper pieces, this allows the development of new seasons, with fresh narrative content available through in-app purchase.

The box set contains, among other items, a private detective’s ID card. This is used to sign the player into the app using near field communication technology –  a batteryless radio transmitter which contains a unique ID code to identify the player.

Current Status

The pilot app is built and has been user tested for some time through Google Firebase Release & Monitoring to identify and fix crashes and bugs as well as the Google Play Store step by step test and release workflow.

The app is currently published in the Google app store and open for pre-registration while  I continue to create more narrative content for the AR artefacts and flashbacks using 3D animation software working with the voice actor. This content is added through database and cloud based asset delivery so it can be incrementally added to the experience without requiring additional releases.

The process of publication to the store was quite involved as I am publishing in both French and English language. Likewise, the app itself is internationalised so that a user can select their interface language. The cloud based delivery functionality is also locality aware and versions of narrative content is stored for both French and English. Further languages can be added without touching the main code.

I am currently working with a homemade version of the physical box set, using my own printed stumper pieces and working NFC card. I have an approved Kickstarter campaign which I intend to launch in the coming weeks to help raise funds to finance an initial batch of the physical products. I have also published a website with links to the Play store and the Kickstarter campaign, as well as a social media presence through Twitter and Facebook.

References

GENETTE, G. (1980). Narrative discourse: an essay in method. Ithaca, N.Y., Cornell University Press.

Apartoglou, Harry. “Interactive Flashbacks: A Case of Agency in What Remains of Edith Finch.” Paper (2018): n. pag. Print.

Ryan, Marie-Laure, et al. Narrating Space/spatializing Narrative : Where Narrative Theory and Geography Meet . The Ohio State University Press, 2016.

Nam, Yanghee. (2014). Designing interactive narratives for mobile augmented reality. Cluster Computing. 18. 309-320. 10.1007/s10586-014-0354-3.

RYAN, Marie-Laure, Kenneth E. FOOTE and Maoz AZARYAHU. 2016. Narrating Space/spatializing Narrative : Where Narrative Theory and Geography Meet Columbus: The Ohio State University Press.

Brown, Douglas. (2012). The suspension of Disbelief in Videogames.

Ryan, M. L. (2001). “Beyond myth and metaphor: the case of narrative in digital media.” Game Studies Vol. 1, Issue 1. http://gamestudies.org/0101/ryan/ [accessed April 2021]

AARSETH, E. J. (1997). Cybertext: perspectives on ergodic literature. Baltimore, Md, Johns Hopkins University Press