Learnings in 5 1/2 chapters.
1- Where it all starts.
Journalists and media practitioners of my generation bring the full experience of the rapidly developing timeline of media evolution to the newsroom and to the classroom. With adding up now twenty six years of intensely working in the field of innovative methods and immersive technologies for quality storytelling in journalism and communication, my priority became to help all interested to better understand how to easily adopt, evaluate and improve the use of Now Age tools and skills.
Digital Now Age methods can be leveraged for all newsrooms, professionals, organisations, communities and citizen users aiming to proactively take part in the digital transformation with one’s resources and goals at a time. My Now Age Storytelling partner and journalism innovator Devadas Rajaram and I are passionate about spreading mobile immersive storytelling skills as the gateway to the evolving future of journalism and storytelling. At present and for the next few years mobile tools and skills prove being the best starting point to dive into the digital communication ecosystem.
Digital mobile methods offer the middle ground, resonating with your audience and blending with their communication behaviour in the most compelling, far reaching, affordable, accessible, high quality, cost-cutting ways. Furthermore, they will enable you to deep dive into the digital space and master the next phases of digital transformation and evolving technologies. However, in the news industry the full potential of mobile, 360-degree/VR and augmented reality technologies is underrated and immersive storytelling still treated as an exclusive area or gimmick, as Devadas Rajaram points out (August 2018).
In the next paragraphs readers can learn more about two key trends in Now Age content production: First, understand better why and how we ought to demystify the use of emerging technologies for journalism and authentic storytelling. Followed by, secondly, a short travel along the potential of mobile immersive storytelling for journalism and communication. Obviously it represents a paradigm shift for journalism by implementing user-transformative, seamless and easy-to-adapt methods, causing doubts and reluctance by professionals, used to think in expensive high-end equipment and exclusive business lines.
However, recent success stories of newsrooms or media houses going digital only and mobile first as the likes of “De Correspondent”, “The Independent”, “Ebner Verlagsgruppe” show that the transit to digital mobile solutions, including diversified product portfolios and revenue streams, will reward newsrooms and media houses with an amplified voice, make their stories stick with their users, increase reach and retention rates and build a strong brand, providing them with the advantage of gaining trust, the most significant currency during an era of trust-crisis in news and media.
And, yes, don’t worry, after all it’s not rocket science.
2- Why mobile immersive storytelling became the new normal despite being ignored in significant parts of the news industry and educational institutions.
In her bestselling book “The signals are talking” American futurist Amy Webb argues that the “promise of exciting new technologies sometimes obscure real change that‘s actually underfoot” (2016, p.44). She illustrates her point by quoting Peter Thiel’s notorious slogan “We wanted flying cars; instead we got 140 characters“ from his manifesto “What happened to the future” (2011).
Webb insists that the over centuries evolving dream of flying cars illustrates what happens to an innovative approach, if its potential and consequences are not measured within a net of nodes, connecting categories of science, society and technology. She uses “flying cars” as a metaphor for a hard-to-realize (for mass adoption) but easy to imagine mobility concept, reaching a far wider public attention than real trends. Part of the reason is that real trends often look on their surface simple and less spectacular. Yet they fulfill a basic human need, making them lasting and sustainable, and turn out being under their simplistic surface complex, powerful, realistic and having far reaching consequences for individuals, organisations and societies.
Laser technology in its early days comes to mind. Although engineer Theodore Maiman, widely credited as the laser inventor, defined it as “a solution seeking technology for problems” as Ulrich Hutschek highlighted in his “Innovation telling” presentation (2016), laser technology remained at first a decades-long underestimated method, which later evolved into a wide variety of applications. Today most people will be knowingly or unknowingly in touch with laser technology in their every day lives. Possibly at a supermarket check-out, in a health check situation, looking at an atomic watch or using laser-based audio-visual players to only mention a few. The same happened to smart phone technology which still remains underrated in its complete potential as a fully equipped media house on the go for professional news and communication industries.
So, while flying cars remain unrealistic for mass adoption, although every other day you’ll read about them, mobile devices including smart phones, voice-based assistants and drones have revolutionised the entire communication and business system of societies worldwide in less than twenty years. While the internet became the second home for people, smart phones became their first-of-its-sort second brain, accessed now by two-thirds of the world’s population, building the gateway to upcoming third brains and fourth brains. These will be realised by emerging mobile devices like Magic Leap’s lightwear glasses, lenses and implements, serving users with real-time hands-free layers of 3D augmented reality and virtual reality information connected to the internet of things.
However, tragically the mobile communication ecosystem remains to date in large parts of the news and media industry as well as in education under-appreciated in its full quantum leap and paradigm shift. But not by everyone. Futurists like Amy Webb or Kevin Kelly anticipated the impact of mobile long ago. Pioneers like Devadas Rajaram worked with mobile technology for news since 1998, look at “Voice2Cell”. Early as 2007 German sports outfitter Adidas started to integrate 3D-modelling in its internal development and design processes for its over forty thousand shoe models, as innovator Detlef Mueller said at his non-public FMX talk 2018. A step, lifting the company up on the level of one of the most innovative global players.
Looking at the first movers who acted within the time-span of the last two decades, the question needs to be addressed why so many journalists and newsrooms keep a perseverant reluctance towards digital change. This seems at times of such a grand self-destructive dimension best comparable to the ignorance of climate change, threatening life on planet earth in plain sight. In this article I seek to find some answers and look into solutions to change this to the better.
2 1/2- Let’s take a global snap shot to explain why digital adoption ought to be faster and fully adopted by journalists, storytellers and educators.
The traditional economy and communications in the news and media industry have been digitally disrupted for about two decades. Basically transforming markets, information and communications into a fluid network, connecting virtually everyone and everything, including all data, in no time and from anywhere by using the hyper-local, interactive, real-time digital space.
Studies and research by the Reuters Institute, Future Today Institute and others, display year after year the rising numbers of trending mobile devices on global scale and used for all forms of communications. With the mobile devices digital literacy became a super-power and must-have resource for open societies in the Now Age.
The tragic point here is: The failure to explore, apply digital mobile skills and spread digital literacy in the knowledge-based industry, affects, eventually harms, all of us. Societies worldwide face a void of digital literacy. As much as the radical new and constantly changing digital methods are challenging, as much is it problematic to resist the change and ignore digital methods used by the majority of citizens worldwide. The dimension of this void can now be observed in the broadened trust-crisis shaking traditional institutions at the heart of the knowledge-engine of open societies, in the news, media and education industry, following the widely unprecedented impact of cyber-wars, misinformation and commercial abuse of small and big data. Additionally, this void is spurred by fascists and populists who early and successfully anticipated how to turn the digital communication system into a digital mal-information system.
It can be rightfully argued that the current disinformation front could take advantage of the high level of digital media illiteracy inside and outside of newsrooms, going hand in hand with leaders in management and politics who lack the understanding how to measurable anticipate future directions.
Journalism innovator Nonny de la Peña mentioned in her SXSW 2018 “convergence keynote”, the level of knowledge in newsrooms and organisations about the use of immersive technologies for purposeful storytelling is depressingly low. We all can do better in building a sustainable future for independent journalism and media, if we want to foster them as efficient check-and-balance filters and information engines for digital network societies. If we won’t get it right now and adapt to the transforming digital communication system, we all will lose much more than embossed routines.
Said this, the imbalance in comparison of billions of unique users, penetrating the internet with the use of their mobiles only, and the slow adaption to mobile as well as to immersive technologies in the news and media industry is striking. Maybe part of the problem is triggered by both the misleading names of “phones”, “watches” and “glasses” and the increasingly tiny sizes, making mobile devices prone to be underestimated from the look on their simplistic surface. Funny enough, too, it’s totally possible to completely ignore the impressive high-tech features of mobile devices and prioritise cosmetic and fashion judgments. But, eventually, at the end of the day it’s inexcusable to underrate the full potential of mobile related methods and tech platforms, softwares and apps.
Take for example Twitter real-time blogging, Snapchat augmented reality stories and Twitch live-conversations. Twitter, with its in the meantime extended 280 characters, demonstrated its powerful potential first and foremost in breaking news and disaster situations like the plane emergency landing on the Hudson River 2009, the 2010 Haiti earthquake, or the 2015 Paris attacks, to name most significant ones for social good and responsible information distribution in real-time. In reverse, currently by having been turned into a radicalising policy tool by authoritarian leaders worldwide following the examples of ISIS or president Trump, a criminal suspect and white supremist, whose 2016 campaign ironically was strongly supported by Twitter-sceptic Peter Thiel.
However, the fact that technology can be abused for evil is as old as the history of human tools and can’t be used a priori against it, as you can’t blame the knife for the killing. We rather ought to make sure that we understand the technology and use it better and more efficiently than populists and cyber-warriors and understand how to take Now Age precautions against unlawful and irresponsible use of digital communication technology.
Looking back at the cyber-war, establishing an advanced fake system targeting the Brexit vote and the US election 2016, and then, more recently, in contrast the 2018 amateurish Zuckerberg senate hearing as well as the old-fashioned mechanisms of the DSGVO data regulation in the EU — introduced with laudable intentions but putting the burden manually on individual users, small-sized and medium-sized organisations and businesses — we may conclude that none Now Age precautions have been put in place, instead law and policy makers in open societies seem being caught in defensive and reactive mode. We see them applying outdated methods of data protection to Now Age cyber-attacks and new digital phenomena instead of leading people and economies into the Now Age era of smart methods, artificial intelligent technologies and post-heroic strategies.
Let’s look once more at Twitter and its current real-time storytelling formats. With its 280 characters tweets including multimedia content and longer threads and moments, in-built live-streaming tools, and messaging and chat functions the platform offers a perfect example for how journalists and storytellers can repurpose emerging digital technology for journalism and storytelling in a way unforeseen by its inventors and initial use-cases.
Another example I’ve used before are Snapchat’s smart, artificial intelligent, augmented reality lenses and face-detection filters, which can be seamlessly used for source protection, if necessary, to enable interviewees to share their stories while their identity is being protected.
I suggest to describe the method of exploring popular new technologies in order to use it for unintended journalistic use-cases and reach out to formerly hard-to-reach audiences as applying ‘open innovation’ to journalism and storytelling. I’ve defined it as “Innovationtelling” applicable with two main directions: On the one hand, you may take a given technology and repurpose it for journalism, in order to reach more users where they are and with the same technology they use — like mobile phones for multimedia storytelling in real-time, or Snapchat for investigative reporting. On the other hand, journalists and storytellers may take their responsibility on a new level by informing and educating users about inevitable, real technological trends in media and societies and empower them with the knowledge how to master these technologies. For example, how to use 360-degree/VR video, responsive virtual reality, augmented reality, algorithms, drones, voice-based robots and many more technologies, already on our doorsteps or “Soonish” arriving.
3- Learnings from the early days of virtual reality storytelling in journalism.
Only six years ago, journalist and educator Nonny de la Peña pioneered with the first interactive immersive journalism story “Hunger in L.A.” (2012), using responsive virtual reality 3D engineering. The story is best experienced with VR goggles, bringing users to the scene of a vast number of poor people queuing up for free food on a sidewalk in downtown L.A., outside of a food bank. Suddenly one man in the queue goes in diabetic shock, faints, falls down and lays helpless on the pavement, surrounded by desperate whispering bystanders. “Hunger in L.A.” is based on a true story and eyewitness reports. It was produced on a low-budget and published for an audience and an industry completely unaware of the new immersive 3D technologies and the potential for purposeful experiential storytelling in journalism deriving from it.
It would even take two more years before Google’s VR compatible Cardboard became available as the first untethered mobile goggle for consumers to be able to easily experience this sort of stories. Back then in 2015 and 2016 The New York Times sent Google Cardboards to ten thousands of its subscribers because the cardboard is affordable, seamless, intuitive, user-transformative and easy to use. One year on the first high-end, tethered and costly VR goggles for mass adoption were released, first by Oculus (Rift) and later by HTC (Vive). Subsequently VR compatible web-browsers and platforms became accessible, too.
Looking back at 2012 as the year, when immersive 3D technologies entered the mass-market under the labels Augmented Reality, 360/Virtual Reality, responsive Virtual Reality and Mixed Reality, we have to understand that all of those are based on mixed technologies superimposing different layers of 3D information on our physical reality. Obviously that makes them attractive for powerful new ways of storytelling. But at the same time the different labels as well as the high-end costs scare away non-tech affine users to freely experience and apply these new communication technologies. All in all these technologies are far away from being branded and educated with ease, instead seemingly demanding creators and users with advanced tech knowledge and able to distinguish and choose between different types of 3D technologies.
Creators and users of tech-affine user personas were for long only subjects in the gaming industry, military complex or fundamental research. However, this exclusivity proves only true in the Now Age, if one will look only on the surface of the immersive technologies, leading to a grave misconception which in parts causes the slow market adoption. Instead, by now, it has become critical for the news industry, media and education institutions to make the effort to understand fully the potential of seamless mobile methods and applications for immersive storytelling. And to spread this knowledge which ought to be taught actually as the alphabet for digital literacy, basically accessible by all creators and users worldwide, tech-affine or not.
Therefore we can learn from successful technology startups — like Apple, Youtube, Warby Parker, Uber — that users quickly and seamlessly adapt to new evolving technologies, if those will make their everyday life easier and offer an intuitive, mobile and more and more also a hands-free experience. This success pattern makes it likely for 360-degree/VR and Augmented Reality technologies as well as for Augmented Reality cloud-based search to unfold the biggest potential for present and future mass-adoption especially for the news and media industry. It’s likely that responsive Virtual Reality and Mixed Reality which cuts users off from their physical reality will be left behind. But the basic presupposition is, that journalists and storytellers stop counting on the concept of exclusive, costly and high-end technologies and hard-to-reach stories.
As a side note, to help spread digital literacy, it may make sense to use one umbrella notion for all possible 3D layered reality. Like the umbrella notion “XR realities”, following, as to my knowledge, the initiative of the Berlin based 360-degree/VR and responsive VR creators platform VRagments by Stephan Gensch and Markus Boesch.
Nonny de la Peña and her team followed from the start the ‘innovationtelling’ strategy, bringing it later to new mastery and continue doing so. From the start it meant to them finding practical solutions for their goal to marry compelling journalistic content and innovative 3D technologies in order to empower users with a full body story experience and move her or him in the middle of the story. The centre of this approach is defined by both the benefit for users, providing them with new means and knowledge for active citizenship, and the impact for journalism to turn underrepresented, complex and powerful stories into compelling experiences in 3D, sticking with their audience.
De la Peña’s immersive journalistic 3D stories established the new term “story living”. She and her team engineered self-made solutions by repurposing 3D gaming technology for journalism. Therefore they recreated an authentic spatial environment in 3D, mixed it with original voices and spatial ambience sound as well as with motion captured 3D protagonists. They engineered and rendered the video story product in a game engine, like “Unity”, and published it in responsive VR compatible browsers and headsets. Either the simple, untethered Google Cardboard for experiencing 360-degree/VR. Or the tethered, high-end HTC Vive with controllers for experiencing responsive virtual reality including motion control. In 2012 the team even invented the first responsive VR goggles — more bulky at that time than now — the foundation of the Oculus Rift, subsequently developed by Lucky Palmer, back then a member of Nonny’s group.
While success stories look from their end perfect, it’s important to also speak out about the troubles innovators have to overcome, in a male business world holding an extra level of spice for female innovators. In the aftermath of “Hunger in L.A.” and her far-reaching experiment de la Peña became at first marginalised, with superiors and colleagues in education and media organisations labelling her work as crazy and unethical. The latter — accusations of “unethical inventions” — seems being a common deathblow argument in the academic world around the globe, to silence innovators who either bring discomfort to a long mapped field of expertise or achieving mind-blowing stuff, going beyond comprehension of many. In both cases innovators, especially women innovators, are seen as threats and made pariahs. Unfortunately the public hardly hears of those who don’t overcome the hurdles placed in their way.
Looking at the timeline of events unfolding in 3D immersive journalism since 2012, we understand that de la Peña overcame the obstacles in her ingenious drive for journalism innovation, leading to her own company “Emblematic Media”. Her achievements are outstanding as a woman journalism innovator and mediapreneur, even more so, if taking into account that the vast share of investments continues being directed to men-led media companies, in the US as well as in Germany and industries around the world.
Nonny’s research and work pushed the boundaries of digital immersive journalism to the paradigm of “duality of presence”. She proved that well researched and produced responsive 3D stories about climate change (“Greenland melting”, 2017) or domestic murder suicide (“Kiya”, 2015) reach millions of users. In the Now Age the “duality of presence” represented by isolated Virtual Reality story experiences behind bulky goggles, cutting us off from our natural physical environment for the time we’re in the story, is evolving to what I call the “plurality of presence” represented by the increasingly simultaneous interaction with a variety of responsive immersive 3D layers superimposed on our physical environment.
The most seamless and user-transformative “plurality of presence” is for the time being realised with the use of smart mobile devices, superimposing spatial audiovisual 3D layers in real-time on our immediate physical environment, without cutting our senses off from any reality, unlike VR goggles.
4- Let’s focus on mobile immersive storytelling — and why it’s the gateway to the present and near future journalism and storytelling.
The internet has entirely changed the way we communicate. It created a revolutionary new matrix: the digital communication ecosystem. It’s worth our time to briefly reflect what are the key changes for storytellers so far.
The core of digital communication — prior to the advent of quantum computing — is defined by digital acceleration of how we’re processing an increasing amount of small and big data, by connecting users in real-time or live through accessing the internet from potentially anywhere at any time, instigating hyper-local interactivity on global scale, and, last, not least, by nurturing a conversational communication style in favour of compelling, multimedia, interactive, casual, game-affine information formats.
The universal devices and methods to enable access to the digital communication space are mobile phones, including feature phones and smart phones, which in 2018 altogether more than half of the world population got at their hands, often more than one device per user. For 2020 it’s anticipated that 62.6% percent of the world population will penetrate the internet with a mobile phone, making roughly 4,78 billion users.
Because the matrix of the current digital communication is user-centred and moving away from top-down and hierarchical structures, digital storytellers and content producers ought to put a process in place which helps them to constantly identify, understand and connect to the changing behaviour of users, starting with listening and knowing how to begin a conversation.
This concept proves “design thinking” methods as extremely useful for journalists and storytellers, due to their natural focus on constructive solution seeking on a longer timeline, consisting on repeatedly cycles of conversation, development, implementation and feedback rounds. By the way, this may give us another powerful example for applying “open innovation” as the method behind repurposing design prototyping for storytelling.
The user-centred system inherent in the digital mobile communication ecosystem leads to a totally different storytelling. Users are neither longer forced into an one-view-point story perspective nor into the so-called receivers’ end. Instead users are now an active partner in an ongoing interactive conversation. Listening skills, far from being a trendy buzzword, became a new soft competence, representing the starting point of successful storytelling in all communication areas.
Consequently the digital communication system in the Now Age is by definition based on a “mobile mind-set”. A concept in reverse to values and structures which dominated newsrooms and media over a many centuries long time span. Back then communication was framed in a system widely based on top-down communication, static sender-receiver workflows, little to no diversity representation, male-dominated hierarchal chains of command, and value-set prioritising status and seniority in newsrooms and media companies, to name the most obvious framing criteria.
In the Now Age the transit to digital mobile communication demands change management actions taking short-term, middle-term and long-term success into account, while individual users, teams and entire organisations migrate to new tools, skills, workflows, distribution and revenue models. The change management process for newsrooms and communication units is challenging in many ways. The transition in many cases has to follow up on top of traditional structures still in place. Another challenge is the fact, that media owners have been laying off staff by merging newsrooms, using decreasing revenue from traditional streams as a short-sighted argument. By way of contrast, independent newsrooms and media outlets of all sizes — like The Guardian, The Independent or taz — have been rewarded for undertaking the long-term change process into digital only newsrooms and media outlets. It might be one of the rare situations without any alternative, and first movers worldwide have actually been proven successful.
At Now Age Storytelling we advise newsroom teams and communication units to master two competence areas in order to successfully transit into the digital ecosystem: One area is defined by mapping and adapting to the constantly changing digital ecosystem, developing a mobile mind-set for embracing life-long learning, opening up to experimenting (trial and error), rethinking the structure of knowledge transfer and storytelling, and prototyping genuine agile workflows, build agile teams and strategies for diversified product portfolios, micro-communities and diversified revenue streams. The other area is defined by adapting to the practical skills to shape, produce and evaluate high quality digital information, content and story formats, in particular with mobile devices in combination with all suitable and emerging technologies and softwares.
Both areas will develop in parallel and over a longer time span. All who master these competences will be well prepared for the emerging technological forces shaping our digital network societies and economy, for example XR technologies, the Internet of Things and Quantum Computer processing.
Last, not least, 3D modelling, experiential storytelling and spatial storyliving are the features which are now key in digital mobile storytelling. When we earlier looked at the groundbreaking example of de la Peña’s works, it helped us to understand how immersive storytelling brings users in the middle of the story. Another colleague of de la Peña and an early adopter of VR journalism is Dan Pacheco, currently teaching at the Newhouse School, Syracuse University. In 2014 he co-produced “Harvest of Change”, a multimedia immersive report about how climate change impacts farming. His team used responsive virtual reality 3D scenes and interactive 360-degree videos as parts of a long-form immersive report. Pacheco explains in his talk at “MobileMe&You 2017” how immersive 3D experiences blend with the natural perceiption of humans, bringing us information in the way we already perceive the physical world around us. Experiencing 3D content takes away the distance and stimulates users’ empathy, not only by bringing them into the middle of the story, but also by enabling them to haptically interact through doing, touching, moving and structuring the story in their own control and pace.
Consequently immersive storytelling contradicts the concept of impartial and distant journalistic storytelling, wherein newsrooms, reporters and users have been trained for centuries. Snipping out the first person storyteller and traces of story construction, I’d call the “deus ex machina” approach. The outdated concept translated in the early days of journalistic 360-degree/VR storytelling into the key effort to make storytellers vanish from the spatial scene, and reporters went into hiding on a 360-degree photo or video scene or had to repair the 360-degree content in the stitching-based post-production.
Nowadays this concept is slowly changing into the first-person experience of news, represented by confident, authentic first-person reporters who act as guides-on scene and connect with first-person users. An example is “Four Walls. Inside Syrian Lives” by Rashida Jones, one of the first journalistic 360-degree/VR productions. Other examples can be found in the reporting style at early towards mobile and 360-degree reporting moving newsrooms like Léman Bleu and EuroNews360.
In the light of the ongoing trust crisis in open societies, it could be an extraordinary grand error to stick to the old concept of “impartial journalism”. The digital mobile ecosystem gives newsrooms and media a vivid alternative at hand, working on basis of first-person storytellers who take up arguments but also take a stand, who report various perspectives but also tell a solution. The new trust-model is represented by individual storytellers turning into influential brands like David Fahrenthold, and by corporate brands building on a strong user-, subscriber- and donator-base on the basis of trusted individuals, trust campaigns and interactive information crowdsourcing bonding with user knowledge; for example, The Guardian, The New York Times, Ryot, Boom or FUNK. Furthermore these brands all use an extensive mobile social strategy with referral avenues to web-based hubs.
In the news industry several big players — for example The New York Times, Washington Post, USA Today, or The Guardian — have made it their mission to explore immersive storytelling, educate the public about new information technologies, and integrate mobile and emerging technologies into the ongoing change process of their digitised newsroom’s workflows and revenue models. Their explorations contributed profoundly to the understanding of the present and future of journalism embracing innovative experiential technologies and story concepts.
However, their often resourceful efforts might have discouraged smaller players with less resources to take up this innovative avenue and also build digital products, include mobile workflows and implement new revenue streams. Partly, because the misconception of immersive technology as cost-intensive rocket science with an exclusive storytelling perspective still prevails. Partly, because many professionals trained in traditional methods and formats simply underestimate the quality and power of mobile. Partly, because the mobile ecosystem fosters agility, interactivity, consent and solution finding, while at the same time the mind-set, prevailing in many newsrooms and organisations, sticks with top-down chains-of-command.
This idle state contrasts tragically the obvious advantages provided by mobile devices: Easy access to user-transformative, seamless Now Age technology. Operational mode, easily switched between live-streaming, real-time storytelling (with a short period of post-production) or long-form storytelling (with an extended post-production). Creators’ choice from a wide variety of free or one-time purchased apps and softwares, formats, workflows, platforms, which can be genuinely designed and adapted for the most suitable native stories and content.
5- Why the digital mobile ecosystem is ideal for immersive storytelling.
Don’t let yourself scare away by high-end gears, coding threats, costly resources and imperatives of exclusive or hard-to-reach storytelling. Instead start to look at the reasons and mechanisms, making immersive storytelling an inherent method of mobile storytelling. For individual users it’s the new normal, it’s overdue that newsrooms, communication departments, organisations and businesses learn to appreciate user-transformative and mobile technology and how it plays out for the creation and reach of immersive stories. Let’s wrap-up with zooming deeper into the mobile immersive ecosystem.
Smart phones are likely the simplest and most powerful brain-computer-interfaces and already highly in use across the world. They are first best technology in fulfilling human needs. Among those, the needs for speed of connectivity, ease of communication and efficient automation. Therefore the tiny media houses in our pockets are equipped with the complete power of a media house in its own right, opening up a completely new horizon for immersive storytelling — with a finger tap.
The technology evolved so fast during the last few years that we currently find easy-to-access and easy-to-use cameras and creators platforms in smart phones, wearables, glasses and lenses. 180-degree and 360-degree spatial camera functions are in-built in almost all smart mobile devices. And, honestly, who wants to look at the limited information in a landscape rectangular picture, if you can have with the same effort the full information of a 180-degree or 360-degree vertical or round visual / video. Available to affordable costs are now also mobile compatible equipments like the Insta360 One camera which can be directly plugged into smart phones. All these tools are part of the digital mobile ecosystem, offering a super easy real-time creation process for spatial visuals — from shooting and editing to publishing, all without the need of coding or stitching skills.
To a greater degree it’s important to reflect that immersive experiences in journalism and communication are not alone about technology, not first and foremost empathy machines, and certainly not about the most perfect illusional experience. Instead it’s about the most authentic, seamless and relatable story experience users can engage with on the go, or, if necessary, also stationary.
Scientific definitions describe immersion as the measurable degree to which responsive VR stimulates the user’s receptors leading to a surrounding spatial experience (a.o. Jason Jared, The VR book, 2016, p.47). Yet the common use of the notion “immersive experience” goes long back and starts with the hands-on illusion of a spatial experience bringing the user’s imagination into other worlds. Like painted stained glass in gothic architecture, translating the human ability of physical 3D perception into the fine arts and craftmanship.
Hands-on immersive experiences lead the path to understand the power of mobile immersive storytelling based on a complex combination of seamless high-tech and liquid, immersive storytelling methods, creating a multi-layered “plurality of presence” in our immediate physical environment, to be experienced by interacting with the mobile screen in diverse sensual ways. Most impressively demonstrated by nearly daily new developments at “6D.AI”, one of the leading tech companies working on augmented reality applications and the AR cloud.
Animated gifs combined with text, floating text boxes over visuals, audio transformed into video, 360-degree video combined with interactive hot-spots offering 3D information and animated 3D objects — all represent different degrees of immersive experiences, having in common that they all can be created and experienced by the use of a smart phone, intuitively, with finger taps, untethered, seamless, interactive.
Mobile phones also introduce the vertical ratio aspect, a compelling simulation of a beginning spatial experience related to both the natural human 3D perception and vertical body stature. Start scanning your environment with a mobile held in vertical aspect, you’ll get a much more spatial illusion than in landscape. Landscape is the ratio aspect users have been primed into by the dominant TV and movie screens for the last century. Now with the evolving immersive technologies, vertical, round and spatial experiences are taking over — like it or not — telling intuitively and naturally immersive stories.
Finally, sound is back in its own right, impacting about 70% of a spatial story experience by evoking the acoustic surround illusion and guiding users to hot-spots of action by audio clues placed in the acoustic panorama or 3D space. Nonny de la Peña uses original voices from 911-emergency calls, from interviews and ambience sounds in “Kiya” (2015), recreating a scene of lethal domestic violence, and in “After solitary” (2017), recreating the experience of solitary confinement, using for the first time responsive volumetric VR technology. People generally like to listen to voices and oral stories, which pull listeners directly in the middle of the story, making one understand the immersive power of voices and accountable stories. Sound and voices are fundamental human experiences which are powerful for mobile immersive storytelling preceding high-end responsive virtual reality 3D storytelling. Another recent, truly compelling example is the podcast “Inside an WWII American war crime”, pointing to the intense immersive experience in its title.
If we start looking at storytelling in the Now Age as an era of mixed technologies, mixed story formats and layered plurality of presence, we’ll soon appreciate the wide variety of tools and formats we have in common with our audience. On the one hand, this gives us the option to create more seamless experiences resonating with our users who concentrate on the story instead on the technology. On the other hand, we gain more creative freedom since there are neither exclusive formats nor “must” have tools.
An example gives the interactive web-documentary “The Industry” 2017, investigating the Dutch illegal drug industry. The journalists makers decided to not create a spatial 360-degree/VR video experience. Instead they used single interactive 3D models with hot-spots offering more in-depth information in additional texts and multimedia formats. There chosen methods frees users from the restricted one view point of a 360-degree video or the being cut-off experience coming with a VR headset. The final product gives first-person users more options to freely choose from and explore all story elements presented in a variety of media formats, in their own time and pace.
The same approach can be seen in “Harvest of Change” (2014) or “The Wall” (2016). Both embed 360-degree/VR and responsive VR scenes as elements of a broader, complex multimedia interactive story.
It’s safe to say, there is no one format or technology that fits all themes and products — and luckily there doesn’t have to be this magic one and only.
Sound and interactive multimedia and 3D information layers are key ingredients for immersive storytelling easily produced with any mobile phone. Mobile devices can be used to create compelling immersive experiences preceding any advanced stories resulting from the use of high-end technology. Furthermore, smart phones can be used for creating and experiencing seamless and interactive augmented reality and 360-degree/VR information and stories.
In itself the digital mobile ecosystem is a native immersive storytelling space. Long before new technological forces will emerge and take over, mobile storytelling ought to be used as the starting point to deep dive into creating seamless immersive experiences. The next technological forces will build on these skills sets.
So, when do newsrooms and communication departments jump onto the mobile immersive storytelling train? It’s right on track with our users and audience.