Creative Development Journal
Welcome to my journal! My name is Mat Wright and I’m a creative app developer. I use this journal to document my research and development.
I previously experimented with combining the Unity game engine and Flutter (Google’s UI toolkit for natively compiled cross-platform apps) back in late 2019, early 2020; with mixed outcomes. At the time, I was working on an AR prototype using Vuforia, a framework for which a Flutter plugin wasn’t (and still isn’t) available. In any case Flutter is essentially a 2D platform while Unity is one of the most popular realtime 3D authoring tools. On the flip side, Unity has poor support for native-feel mobile interface. So, the combination of these two technologies should be a great match.
I recently developed functionality to allow an app’s users to sign in using a business card. The business card has an embedded batteryless, short-range communication device known as a Near Field Communication (NFC) tag. The user simply positions the business card close to their mobile device (usually top rear) and data is transferred from the static , wireless tag to the mobile device. Power is transferred from the mobile device to the NFC tag when placed in close proximity thus the tag containing the data does not require its own power supply. I have received a number of messages asking me to document the process.
Over the past six months I have been working with Flutter and Firebase to develop a framework for a cloud connected interactive smartphone expérience. The app delivers virtual objects into the user’s real-world space with AR. The user searches and collects virtual objects over time. Each find unveils new interactivity, for example an object may […]
My research interests have led me to want to understand more about the processes and determining factors behind the concept of ludological narrative and examine whether it could be worthwhile carrying out primary research to examine how mobile computing combined with augmented reality has already, or will through future innovation, enhance the existing narrative mode of computer games or whether this emerging medium may constitute a distinct narrative mode of its own.
Researching the history and significance of the femme fatale in classic and revivalist film noir has provided rich insight to the complex nature of a characterisation that might otherwise be easily taken for granted. The common themes of beauty, sexuality and danger which emanate from the popular notion of the femme fatale only betray the true sensibilities of the subject matter.
Noirscape uses NFC (Near Field Communications) technology to add some fun to the signing in process. A detective ID card is among the items that will be included in the boxed version of the game but they’ll also be available via the future website and onlin merchandise store.
NFC is a tiny, low-cost, batteryless radio transmitter which allows app developers to embed the featherweight, waferthing device into clothing, toys, posters and just about anything else. In my case, I am using a business card format. The actual NFP cpmponent is inside the plastic and is only about an inch in diameter. For the purpose of my testing I have created my own version using card and a stick’on NFP emitter.
The classic film noir era played out under the Hollywood Production Code, a strict set of guidelines that governed movie content throughout the nineteen thirties and forties. The Production Code meant specific restrictions on behaviour and language: no sex, no nudity, no blasphemy, no swearing… all of this meant that film makers had to get even more creative in order to express film narrative using metaphor, lighting, shadows, composition, focus and language.
Finding hidden objects in an augmented reality [AR] experience is fun; especially when the items have interactive qualities which trigger new content, clues and narrrative. Furthermore, given the quality of 3D assets, AR lighting, shadow and reflection prediction and generation; coupled with a the Noirscape black & white filter, it’s a great little feature to be able to capture an augmented view within the home and share with others on social media or messaging.
Augmented Reality (AR) provides a way to place realistic looking virtual objects into a realworld scene. While the object may merely exist upon the screen of a phone; there are features to AR which combine the worlds; fiction and reality; beyond the two dimensional surface of a smartphone. For example, when a 3D object is placed within the AR scene; it may not be physically present upon the targetted surface; but certain qualities of the said surface are projected into the machine generated final composition (horizontality, width, height and distance of the plan) and ultimately this data is processed within the mind of the person who momentarily accepts the presence of a fictional item within their immediate realworld environment.
This is a demo of an interactive scene using 360 photography, 3D character design and animation pulled together in Adobe After Effects to produce a WEBP image file which then has interaction through sound and buttons using the Flutter mobileapplication development framework.
Noirscape experiments with a cross reality approach to narrative. Participants search for and find fictional objects in their own physical home through the AR (augmented reality) feature. One of these objects is a door and it’s keyhole a doorway between an augmented view of one world and the entirely fictional world of another. Furthermore, Noirscape binds the narrative with physical spaces in the nearby town. In my case and for the purpose of the pilot version of Noirscape, this is the French town of Bourg-en-Bresse.
While the allure of Noir is an assumption I hold about the target audience; it is just as important that the audience is interested in role-playing, interacting with other participants and within the context of their local town. I made assumptions about attitudes towards a product which is tailored to a person’s home town as well as the concept of leaving one’s mark withing the fictional world of the experience. In order to test and respond further to these assumptions I carried out some primary research which I have detailed below.
A demo of my vintage rotary phone dialler build with Flutter with sound effects and number dialling recognition. This an interactive element linked to AR artefacts found within the game.
I created a telephone dial using Flutter’s CustomPainter API. This permitted me to draw the various elements onto the screen relative to the size of the screen. The problem now is : how to detect which number has been dialled.
The Noirscape participant collects fictional objects in the Augmented and Immersive reality parts of the experience. Beyond the fun of finding these items and seeing them appear in-situ on your living room table; they also provide important props and gameplay. The objects are associated with actions. Sometimes an object will have multiple actions, sometimes they’ll depend on other objects to work together. Here are a couple of examples of interactive fictional objects I’m currently developing.
While, in itself, the narrative of Noirscape is to be interpreted at will by it’s participants, the structure of the environment in which the experience will take place will of course provide the underlying props and backstory along with the opportunity for interactivity with these props, people and different types of spaces – fictional, semi-fictional and realworld.
I was able to scale my 3D object correctly using a combination of GLTF settings and ARCore config. With some Shader work within Flutter I’ve created a Noiresque look in which the vintage 1940’s 3D telephone I got from Sketchfab (see link in video description) is positions consistently in the AR of ‘Mixed Reality’ world based on the detected horizontal Plane, the Pose of the object and of course the World Coordinate Space.
In order to get to the bottom of the issues I have been experiencing with ArCore for Flutter I decided to test and document different combinations of factors. Given that there is a multitude of possible test cases with different types of 3d object file, different versions of Sceneform and heaps of other stuff; I wanted to test each case one by one and record my findings, as I’m convinved this is doable with Flutter and ARCore.
I encountered a number of issues with getting complex 3d objects into a Flutter AR app. The Flutter AR components takes either a remote glft file, a format designed for loading 3 objects in a web browser; or sfb a format specific to Sceneform, a 3D rendering component for smartphones.
After much deliberation about how to handle the narrative element of my app I believe I am close to pivoting in a slightly new direction. Although I’ve identified suitable source material for a narrative, it isn’t really what I want to do at a personal-creative level. Although, I understand that in business apps, the user is of course the center of all design considerations; and in terms of the app’s features that will be the case; however, I am very attracted to the idea of approaching this app-game with some artistic license. One of my early desires, as a ‘creative’ prior to committing to the general direction of the app’s premise, was to be able interact with the users, who I tend to describe as ‘participants’ in my ongoing thought processes.
Over the last 12 weeks I have taken a problem – the withering public interest in smaller city and town centers as a point of inspiration and ideated, through the help of academic research and experimentation, a concept for an ‘app-game’. I have a name for the app, a theme, and even an overview of the user journey through the app from purchase to completion. I now need to build the app.
Over the last nine months I’ve taken a number of intenstive Adobe workshops, each of which culminating in an exam and if passed, recognition as an Adobe Certified Associate in the related software and field of practice (video, graphics, etc).
The Noirscape app narrative design features several conceptual layers.
The boxed product is sold on the high street in selected specialised stores. The product’s cover design will be highly stylised to the ‘Noir’ look and feel. It is clear from the cover illustration that this is a hybrid board-app game.
To design an immersive and interactive smartphone application using 360 video and GPS to send users on a film noir themed investigation into the physical space of their local town.
I am currently working on a conceptual design document inspired by BJ Fogg, founder, and director of the Stanford Behavior Design Lab.
Following on from my reflection and study-notes about the porous boundary between the fictional [or synthetic] world of electronic narrative platforms and the realworld, I would like to now switch my focus to the role of realworld within the scope of interactive networked narrative; I shall focus particularly upon the idea of spatialised narrative i.e. the intersection of space and narrative; with the aim of exploring the synergistic potential of combining space, place, computing and interactive narrative.
In my previous post I compiled some of my thinking and study notes around the notion of implementing interactive, branching narrative using the smartphone medium. This got me thinking more about the uniqueness of the smartphone app as a creative interactive medium as opposed to book or video. I considered how the Choose Your Own Adventure books I played as a kid were in fact book-games just as games I played on a tv screen or monitor are video-games. More recently I played the 2018 Netflix movie-game called Black Mirror: Bandersnatch.
Like so many other kids growing up in the 1980’s I spent a heck of a lot of time playing Choose Your Own Adventure (CYOA) books as well as playing/making text based adventure games for my Commodore and Sinclair home computers. I recall spending hours upon end staring at a tv screen with a flashing cursor, writing simple commands within a text game : go west, sit down, pull chord, enter hole. Often I’d be stuck in a seemingly neverending sequence of typing the same commands and just not being able to unblock new content.
During my recent experiments I have been working with Adobe After Effects to apply special effects and animations to 360 degree video footage. My first experiments were quite successful. I was able to do what I set out to do, with a couple of areas that would need some further attention. One such area was the exportation format from After Effects which was producing an unusual effect whereby the 360 image was not wrapping correctly.
360 video files can be quite big, the short ones I have been experimenting with are around 10Mb and even on a fast wifi connection there was a significant delay while the video is downloaded in its entirety before the 360 video player starts playing. This would provide a poor user experience in terms of the wait. I could potentially add some gameplay functionality into this waiting time; but it’s still a big ask – to download large files; especially if the user is outdoors and using mobile data, which may be limited or even expensive to the user.
Following on from my 360 video experimentation in Adobe After Effects I wanted to test out options for embedding my 360 videos within a a mobile app using Flutter.
I would like to explore the possibility of shooting studio footage – for example of character acted scenes, using a green screen which would be integrated into the town footage. This would mean I could reuse the same studio footage in multiple towns and therefore offer the app to a greater number of users. As my theme for the app is Film Noir, I could also use other props and footage such as period vehicles and so on.
The aim would be to create a ‘Pilot’ version of the interactive experience at a local town, to which I have easy access and which I know well.
I would be looking to pitch the pilot to the local population as well as the tourist board and local newspaper.
Although I consider this app to be foremostly an artistic endeavour, this doesn’t mean I can simply ignore user research. A creative app is quite pointless if no-one ever interacts or engages with it. My hypothesis is that that people react with greater curiousity and desire to engage with a creative piece when they feel […]
Over the last 6 weeks I’ve spent much time ideating for a new app. I know that I want to make good use of the technical skills I’ve been developing over the last couple of years in augmented reality, digital illustration, applied machine learning and somehow bring it all together with my gamification research as well and my philosophical interests.
It’s been a couple of years since I became keenly interested in cross reality (XR) app design and development. To begin with, I was experimenting with Google Daydream VR (virtual reality) and had also recently acquired an Oculus Go headset; which had become widely available and popular. At this time I began thinking about ideas […]
Although I’ve built several mobile apps previously using web technologies Cordova and Ionic, I was never fully satisfied with the quality of the end products. In most cases, this was because of the un-native ‘look’ of the web based ‘hybrid’ apps. I recall experiencing much frustration using native plugins too.
Introduction AISpy is an experimental app concept built using Flutter, a cross-platform toolkit made by Google. The app uses TensorFlow Lite, a deep learning framework for on-device inference. The app combines artificial intelligence and gamification to acquire user submitted source images to enhance the underlying machine learning object recognition dataset. The app protoype uses a […]
The second-annual international Flutter hackathon, organised by the global Flutter Community, took place over the weekend 27/28 June 2020.
The 48 hour app jam invites teams of up to five members to build an app using Flutter – Google’s UI toolkit for building natively compiled applications for mobile, web and desktop from a single codebase.
The purpose of this article is to explore how elements of UI (user interface) design evolved from the analogue world of physical, three dimensional switches, dials and buttons. It also examines potential current and future roles for skeuomorphs (digital components which look like real-world objects) within the field of software and product design.
I am reviewing the first chapter of Douglas Brown’s thesis on the suspension of disbelief in video games. I chose this chapter for analysis because I wanted to understand the foundations upon which the thesis is based – the case for games as a distinct medium. One of my own research interests is the nature of VR (virtual reality) and I am interested in the implications of Brown’s conclusion within the field of VR creativity.
Within this post I will provide a step by step guide to create a Google Cloud Firestore driven contact form for a Flutter web app; including details about how to build the form, validate the input and set up security rules before finally saving the data to the Google Firestore database.
I recently decided to test drive Google’s ARCore Extension for the Unity game engine to find out how Cloud Anchors work. Cloud anchors allow AR (Augmented Reality) apps to create virtual objects within a physical space and persist them across multiple user sessions. This means that different users should be able see and interact with the same virtual objects at the same time from an AR app.
John is a french baker. He has been baking bread in his village bakery alongside his wife for over 30 years. John gets up every morning at 4am to make sure there is plenty of fresh bread on sale when the bakery opens it’s doors at 7am. At which time John’s wife, Matilda, joins him in the shop to serve the customers. The bakery is very busy every day. John continues to bake bread until early afternoon and then he returns home for some rest and goes to bed at 8pm. Matilda remains at the shop until it closes at 7pm. There must never be any left over bread at the end of the day but at the same time they must never run out either, or else they’ll lose customers.
Last summer, I was working on a prototype for an AR app that would allow users to create virtual objects in real world spaces. I carried out the work using the Unity3D game engine. My last commit to the repository was five months ago.
Since those last commits I’ve been busy working on a few Flutter projects (see previous posts). I now have an new opportunity to go back to the AR project, so I have been reflecting upon how I might combine these two technologies,
The Challenge I recently completed my first experience building a mobile app with Flutter while working as part of the Newspeak Creative. Off the back of that experience I decided to take part in the Google Flutter Clock challenge. I only found out about the competition a couple of weeks before the deadline but decided […]
Over the last 10 weeks I’ve been working within a small group of developers collectively known as “Newspeak Creative”. We have designed and developed a revision app for GCSE school children studying Orwell’s Animal Farm for their final exams. I was charged with writing the code and general technical development of the app while comrades […]
Our proposal to build a GCSE revision app, as described in my previous post, was accepted by the client. The team settled on an art style for the app’s avatars and began compiling content. Work has also begun on building the app itself. Following technical research into the feasability of using a game engine for this project, the team decided to proceed with a different platform and I will discuss this decision in this post.
I am currentlly working within a small team of three creative app developers to deliver an interactive mobile app for a UK secondary school. The main purpose of the app is to assist year 11 (15/16 y.o.) students with their GCSE English Literature revision. The theme of the app is George Orwell’s classic novel, Animal Farm. Since receiving the brief a few weeks ago we have been putting together a proposal and I would like to reflect on the process so far.
Introduction During my recent agile sprints, I completed the minimum requirements for two key user stories. Firstly, my player can sign into the app using Facebook. This makes it quick and easy for the user, providing they have a Facebook account. As I have integrated authentication through Firebase, I am able to add further social […]
“As a player I want to position my lovelock in the realworld so that it stays in the same place. “ Week Ten User Story Sprint Introduction Originally, I had aimed to allow players to place a virtual lovelock in a realworld physical location. For example, a player would be able to attach their virtual […]
“As a player I want to sign in using my facebook account so that I can save time.” Week Nine User Story Sprint Introduction From a player’s perspective, signing up for an app on a mobile device can be an offputting chore. It usually involves entering an email address and then having to verify it, […]
Introduction I have spent the previous six weeks exploring software kits, programming frameworks and 3D graphics techniques. I have been focused on developing skills which are relevant to building apps for virtual and mixed/augmented reality; otherwise collectively known as XR – eXtended (or in some cases ‘cross [x]’) reality. To start putting these skills to […]
Introducion A feature of Agile methodology, the persona is a kind of fictional character, invented to represent a typcial user who will be engaging with the app your team is going to develop. Storyboard Agile development consists of a storyboard. This is a collection of short stories, which act as a placeholder for a real […]
During the last week I participated in a 48 hour app jam. The theme for the jam was decided using VNA (verb, noun, adjective) ideation cards. In addition to the VNA derived theme I decided to constrain my app to being a VR game to be built using Unity. This was following on from my […]
Introduction To explore a new technological platform with a view to getting under the hood and coding something up is one of the most exciting prospects, as a creative app developer, I can think of! In today’s connected world, apps have become a ubiquitous feature of our daily lives. At night while I sleep, my […]