writing with a camera

Immersive Storytelling with 360 Video

Title slide

Thrown together by David Tamés on February 25, 2020, revised on November 15, 2024, see the last slide for acknowledgments and copyright information.

This presentation begins with a brief history of apparatus-based image-making, from early uses of linear perspective around 1415 to contemporary immersive works. Fundamental terminology and concepts are introduced, including immersion, presence, place illusion, plausibility illusion, 360 video, 3DoF, virtual reality, and 6DoF. The presentation then covers eight evolving best practices for 360 video storytelling and closes with a discussion of assembling a production kit for production 360 video works.

See also: Ambisonics Recording Kit Demo (workshop presentation) and Cinematic VR Immersive Storytelling (article)

Photo Credit: Giu Vicente, @giuvicente

ca. 1415: Linear Perspective System (Filippo Brunelleschi)

One of the important antecedents for the development of virtual reality was developed in Florence, in the early 15th century. According to art historian Giorgio Vasari, the architect and inventor Filippo Brunelleschi sometime around 1415 developed the linear perspective system while experimenting with the depiction of some landmarks in Florence. The linear perspective was a point of no return for Western art, the technique enabled artists to represent a tridimensional space on flat surfaces and this approach influences to this day the ways we experience and design our environments, artificial or natural. The linear perspective was the first important step in the process of immobilizing the viewer in front of pictures created with the purpose of deceiving our senses to make us believe we are looking at a real scene.

The linear perspective was the first important step in the process of immobilizing the viewer in front of pictures created with the purpose of deceiving our senses to make us believe we are looking at tridimensional scenes. Every element of the scenery, whether it is a tree, a person, or a building, is positioned to comport with the viewer’s line of sight. The space produced by the linear perspective is mathematic, artificial, and exists only through the beholder. By indicating a “best” point of view, that is, where the illusion is most effective, this perspective erects a hierarchy; whoever stands at the center is potentially more important than the viewers who gaze from one side.

Source: The Links Between Linear Perspective and Virtual Reality Technology by Filippo Lorenzin (Hyperallergic)

1435: De Pictura (Leon Battista Alberti)

Published in Latin as De Pictura in 1435 and in Italian the following year (Della pittura, dedicated to Brunelleschi), Alberti’s treatise codified the monocular perspective Brunelleschi had demonstrated around 1413, transforming painting from a mechanical craft into an art grounded in geometry and optics. Alberti imagined the scene as a visual pyramid in which rays extending from the viewer’s eye to every point on the object, with the picture plane understood as a cross-section or “open window” (finestra aperta) cutting through that pyramid. His construction method (later called costruzione legittima) used a centric point on the horizon and a distance point to build a geometrically correct tiled floor, giving painters a repeatable procedure rather than intuition. The window metaphor, the fixed monocular viewpoint, and the mathematics of the pyramid became the scaffolding of Western image-making for the next five centuries, through the camera obscura, photography, cinema, and eventually the virtual camera of CGI (Computer Graphics Imagery), game engines, and synthetic cinema.

For more on this history, see The Virtual Window: From Alberti to Microsoft by Anne Friedberg (The MIT Press, 2009)

ca. 1470: Ideal City

The Ideal City panels (Urbino, Baltimore, Berlin, c. 1480–90) traditionally attributed to Piero della Francesca but now more commonly given to Francesco di Giorgio Martini, Luciano Laurana, or their circle) are often read as the first pure perspectival environments in Western art: paintings whose subject is not an event or a figure but constructed space itself. Nearly empty of people, organized around a single centered vanishing point, they function less as depictions than as computed rooms, spaces one is invited to inhabit with the eye. The panels make explicit what Alberti had theorized: that a mathematically constructed space can be the artwork, not merely its setting. They are also closely tied to the origins of Renaissance scenografia (perspectival stage design), arguably the first practical technology for placing a body inside a constructed perspectival environment. This lineage that runs through the panorama, the stereoscope, cinema, and the VR headsets. The Ideal City is, in this sense, an early virtual environment in the strict sense: it was a coherent, uninhabited world waiting for a viewer to complete it.

The hierarchy is political as well as geometric: the privileged viewpoint was often literally the throne, the pew, or the patron’s chair, and the vanishing point acted as its mirror. Vilém Flusser would later argue that such apparatuses program their users while appearing only to serve them. Virtual reality inherits this logic almost unchanged: a single centered viewer, a computed pyramid, a world that exists only through the one who beholds it, even as it promises to free the body it still requires be held still.

For more on this history, see The Virtual Window: From Alberti to Microsoft by Anne Friedberg (The MIT Press, 2009) and The Links Between Linear Perspective and Virtual Reality Technology by Filippo Lorenzin, Hyperallergic, March 1, 2021.

Late 17th and early 18th century: Baroque theatre

Baroque theatre (roughly 1650–1750) is the missing hinge between Alberti’s painted window and cinema’s framed screen, the point at which perspective construction became architectural and the viewer was placed nside a room built around its geometry instead of in front of a picture. The proscenium arch functioned as a three-dimensional Albertian frame; behind it, receding wings (telari) and painted backdrops were laid out along orthogonals converging on a single vanishing point. That point, the sightline from the royal or ducal seat at the center of the hall functions so that the sovereign saw the illusion at its geometrical perfection and everyone else saw a slightly skewed version of the same scene, a literal optical hierarchy of the audience. Stage machinery (e.g. grooved wing carriages, cloud borders, trap lifts, thunder runs, etc. ) was deliberately concealed above, below, and beside the visible scene, producing the first systematic apparatus of concealment: a deceiving surface supported by hidden mechanism, the direct ancestor of cinematic continuity editing and the models and game engine code building the world while hiding the mechanism behind the scenes doing the work. The Castle Theatre at Český Krumlov (1766, rebuilt on a 1680s structure) survives as the most complete working example in the world with original wings, backdrops, machinery, lighting fixtures, and manuscripts intact and it is still used for a small number of performances each year, functioning as a time machine for the Albertian apparatus in its architectural phase. Photograph source: Wikimedia Commons.

For more on this history, see The Virtual Window: From Alberti to Microsoft by Anne Friedberg (The MIT Press, 2009) and The Links Between Linear Perspective and Virtual Reality Technology by Filippo Lorenzin, Hyperallergic, March 1, 2021.

1792: Panoramas, Robert Barker

Barker’s patent (1787) and the Leicester Square Rotunda (1793) mark the moment the Albertian window became a cylinder. The frame dissolved, the viewer stepped onto a central platform, and painted illusion wrapped continuously around them for 360°. What held the apparatus together was architectural concealment: a raised viewing platform hid the painting’s bottom edge, a velarium (a suspended canopy) hid its top edge and diffused an overhead skylight so no lamp or window betrayed the scale, and a darkened entrance corridor dilated the pupil before the visitor emerged into the lit scene. A careful choreography of arrival that contemporary immersive media reproduces. The result was the first mass-market immersive medium, drawing hundreds of thousands of paying viewers in London, Paris, Berlin, and New York across the 19th century.  Image Source: TBD.

1792: Panoramas, Robert Barker

Viewers flocked to Barker’s Panorama, paying up to 3 shillings to stand on a central platform beneath a skylight that bathed the painting in even, sourceless light, an experience contemporaries did not yet call “panoramic” (the adjective first appeared in print only in 1813). At the semi-circular Panorama of London, painted as if viewed from the roof of Albion Mills on the South Bank, visitors could purchase a souvenir set of six engravings that modestly recalled what they had seen; laid end to end, the prints stretched 3.25 meters. Image source: TBD.

1822: Diorama, Louis Daguerre and Charles Marie Bouton

Daguerre and Bouton’s Diorama (Paris, 1822) then added time to space: translucent painted scrims lit alternately from front and back, shifting across a choreographed cycle, so that a cathedral interior could pass from dawn to midnight or a mountain valley from sunshine to avalanche, the temporal dimension that 360° video restores today, and the ancestor of every VR experience that changes its world around a still-centered viewer.

In 1822 Louis Daguerre, a painter and stage decorator, along with painter Charles Marie Bouton, invented “realistic illusion shows” or Dioramas. Together they opened The Diorama in Paris a venue to show their large scale panorama paintings, with special light effects…

1822: Diorama, Louis Daguerre and Charles Marie Bouton

…at the time it was one of the most popular spectacles in Paris, through the clever play of light, Daguerre could make one scene dissolve into another. Parisians were treated to the sight of an Alpine village before and after an avalanche, or Midnight Mass from inside and outside the cathedral, accompanied by candles and the smell of incense.” See: The Invention of the Diorama-Daguerre and Bouton Listen to the “Diorama Drama” episode of Between the Ears from BBC Radio 3, Lance Dann brings to life various techniques for delighting and mystifying our imaginations, asking what it takes for us to be given pleasure by immersion, https://www.bbc.co.uk/programmes/m000drdq

1822: Diorama, Louis Daguerre and Charles Marie Bouton

…at the time it was one of the most popular spectacles in Paris, through the clever play of light, Daguerre could make one scene dissolve into another. Parisians were treated to the sight of an Alpine village before and after an avalanche, or Midnight Mass from inside and outside the cathedral, accompanied by candles and the smell of incense.” See: http://dioramadaily.blogspot.com/2013/02/the-invention-of-diorama-daguerre-and.html (Diorama Daily) and listen to the Diorama Drama episode of Between the Ears from BBC Radio 3, Lance Dann brings to life various techniques for delighting and mystifying our imaginations, asking what it takes for us to be given pleasure by immersion.

1850s: Stereoscope: 19th century virtual reality device

A stereoscope is a device for viewing a stereoscopic pair of separate images, depicting left-eye and right-eye views of the same scene, as a single three-dimensional image. A typical stereoscope provides each eye with a lens that makes the image seen through it appear larger and more distant and usually also shifts its apparent horizontal position, so that for a person with normal binocular depth perception the edges of the two images seemingly fuse into one “stereo window”. See Stereoscopes: Nineteenth-Century Virtual Reality Devices (Arts Quarterly, New Orleans Museum of Art, October 19, 2017).

1850s: Stereoscope: 19th century virtual reality device

The stereoscope, demonstrated by Charles Wheatstone in 1838 and dramatically improved by David Brewster’s lenticular design in 1849 brought binocular depth perception to consumers. The stereoscope is the first widely distributed apparatus to add binocular depth to the Albertian single-point construction, producing what Jonathan Crary describes as a radically new kind of observer: isolated, privately positioned, and optically absorbed in an image whose spatial illusion depends on the careful occlusion of everything outside the viewing hood. It is, in short, the 19th century head-mounted display, and the structural ancestor of every stereoscopic VR headset that followed.

Two photographs of the same scene, shot from positions roughly matching the inter ocular distance of the eyes, were placed side by side in a viewer that presented one image to each eye; the visual cortex fused them into a single scene with convincing depth, a phenomenon Wheatstone had first isolated using mirrored drawings a decade before photography existed to supply the images. Oliver Wendell Holmes’s simpler, cheaper handheld model (1861), which he refused to patent, became a mass market item. By the 1860s tens of thousands of stereo cards circulated through parlors in London, Paris, New York, and Boston, carrying viewers to the Alps, the pyramids, Niagara, the ruins of the American Civil War, and a vigorous trade in tourist exoticism and pornography. See Techniques of the Observer: On Vision and Modernity in the Nineteenth Century by Jonathan Crary (MIT Press, 1990) and The Stereoscope and the Stereograph by Oliver Wendell Holmes (Atlantic Monthly, June, 1859, the essay that popularized the device.

Images (1) Theodore Lilienthal, Bird’s-Eye View from Moreau’s Restaurant, Mouth of Bayou St. John, 1867, stereoscopic albumen prints, The Historic New Orleans Collection (2) Late 19th-century advertisement for a stereoscope.

2016: Notes on Blindness (Ex Nihilo, ARTE France, AudioGaming)

The VR work that stood out for me at Sheffield Doc/Fest in 2016 was Notes on Blindness: Into Darkness, based on the audio diaries of John Hull, a writer who gradually lost his sight and documented his experience using an audio cassette recorder. The work was originally produced for Samsung Gear VR using Unity by creative directors Arnaud Colinart and Amaury LaBurth as a companion to the documentary film Notes on Blindness (Peter Middleton and James Spinney, 2016), however, it uses a completely different visual language than the film. The work immerses you in articulations of sounds visualized as outlines fading in and out of your visual field as Hull narrates his experience, enabling you to feel what the loss of sight might be like as sounds take on a new role in revealing the texture and topography of your environment. Notes on Blindness. Into Darkness was the first VR experience that felt to me like an eloquent use of the emerging audiovisual grammar of VR. Image credit: ARTE France. Caption: VR project was produced by Ex Nihilo, ARTE France, and AudioGaming, based the feature documentary Notes on Blindness (Peter Middleton, James Spinney, 2016)

2016: Notes on Blindness (Ex Nihilo, ARTE France, AudioGaming)

Art direction was shared among three designers, Arnaud Desjardins, Fabien Togman, and Béatrice Lartigue but Lartigue’s year-long role as art director decisively shaped the finished visual language. Her guiding principle, as she later put it, was that “the world is drawn with particles“ rather than render Hull’s environments as solid objects, she built them as sparse clouds of glowing points that become visible only when activated by sound, wind, rain, or vibration passing through the space. The point of view is realistic, but the visual translation of Hull’s inner world is deliberately abstract, staging his description of an intermittent, fragmented world in which things exist while they are heard and quietly dissolve when they are not. The ground at John Hull’s feet is rendered more densely than the surrounding space, anchoring the body and giving the otherwise abstract environment a physical horizon. Paired with binaural audio that organizes the scene aurally before it is ever organized visually, the effect inverts the Albertian contract: space is produced, moment by moment, by the listener’s embodied presence, a working demonstration of a presence-image logic, a unique affordance of immersive media. See: Designing for VR: Notes on Blindness: Into Darkness by Béatrice Lartigue, Medium, June 28, 2016.

2016: First Impressions (The Guardian)

First Impressions (The Guardian, 2017) based on cutting-edge research in neural development and color vision in infants, provides viewers with an immersive experience of the world from an infants point of view during the first six months of life. When I first saw this five-minute VR micro-documentary experience at Sheffield Doc/Fest, the setting included bean bag chairs and large, colorful blocks to set the context. Assistants helped you put on the viewer and headphones. At first, you can only see blurry objects in a monochrome field as a narrator explains whats happening regarding neural development. Color enters your visual field slowly, at first reds and greens, and eventually blues and yellows. The sharpness of the objects around you increases gradually as you look around the environment of your bedroom. You see and hear the coming and goings of your parents, another child, and the family dog, as the narrator explains how crucial social interaction is during this phase of neural development. First Impressions was created by the Guardians in-house VR team, including Nicole Jackson and Francesca Panetta, and is part of a series of editorially independent VR films.

2016: 6x9 (The Guardian)

6 x 9 (The Guardian, 2016) 6 x 9 was the first 360 documentary produced by The Guardian. When I saw it at Sheffield Doc/Fest, I was led into a 6 x 9 cell to watch it…

2016: 6x9 (The Guardian)

…enhancing the impact of this video that places you inside a cell measuring 6 x 9 feet and telling the story of the psychological damage that can result from the isolation of solitary confinement.

2017: Pearl (Patrick Osborne)

Pearl (Patrick Osborne, 2017) offers an engaging fixed POV animated narrative that unfolds around you; your only agency is where you look, but its a heartwarming, well-told story that provides a metaphorical ride through time and place.

2018: The Joy of Frogs (The Guardian)

Every spring, ponds around the UK start stirring, and frogs come out of their winter slumber to mate. The Joy of Frogs (The Guardian, 2018) offers a unique perspective on this story that is as old as time. It’s a lot of fun to watch, and it puts you in a specific place at a particular time and place that makes a lot of sense, providing a value-added experience that the flat presentation lacks, this work makes good use of the affordances of 360 video while retaining compatibility with the flat presentation version.

Immersion

Janet Murray, in Hamlet on the Holodeck: The Future of Narrative in Cyberspace (The MIT Press, 1997) wrote, “Immersion is a metaphorical term derived from the physical experience of being submerged in water. We seek the same feeling from a psychologically immersive experience that we do from a plunge in the ocean or swimming pool: the sensation of being surrounded by a completely other reality, as different as water is from air that takes over all our attention, our whole perceptual apparatus.”

The general consensus seems to be that immersion involves being or feeling surrounded by something, however, a number of alternate propositions address what it is that surrounds the individual. In my Narrative Basics and Designing Extended Realities courses I make a clear distinction between “Immersion” and “Presence,” drawing the distinction primarily from Sanchez-Vives and Slater (2005).

Immersion is a property of the technology. It is the objectively describable and measurable set of stimuli the system delivers, and related to the number of sensory channels engaged, and fidelity within each channel’s submodalities (e.g., shape/color/value for vision; volume/pitch/tempo for audio). Because it is technological, immersion is under the direct control of the experience designer.

Presence, on the other hand, is a property of the participant’s subjective experience. It is the emergent internal physiological and psychological state, in other words, the subjective sense of “being there” that arises in a person immersed in an immersive environment. Because it is subjective, designers cannot produce presence; they can only set up conditions that elicit it, and the same immersive system may yield very different levels of presence across participants.

Mel Slater uses a color-science analogy to make this concrete: immersion is to presence as a wavelength distribution is to the perceived color of a sunset. The first is an objective description, the second is how the perceiver experiences it. Presence rests also rests on two illusions described by Mel Slater: The place illusion (I am somewhere else) and the plausibility illusion (what is happening is really happening), and it fails through breaks-in-presence, typically caused by glitches in the technological infrastructure (lost tracking, collisions with real objects, latency). This is where the objective side (immersion) determines whether the subjective side (presence) can hold.

References: “From presence to consciousness through virtual reality” by Maria V. Sanchez-Vives and Mel Slater in Nature Reviews Neuroscience, Volume 6, 2005, pp. 332–339, doi.org/10.1038/nrn1651; The VR Book: Human-Centered Design for Virtual Reality by Jason Jerald (ACM/Morgan & Claypool, 2016).

360 video storytelling: evolving best practices

We’re still learning how to make the best of 360 video, especially in terms of interaction, however, these best practices will give you a running start and help you avoid the most common rookie mistakes. Photo: Giu Vicente, @giuvicente

Image: https://www.digitalartsonline.co.uk/features/hacking-maker/360-video-projection-how-its-created-why-its-worth-your-time/

360 video is different than virtual reality, though it shares some family resemblances

360 video and virtual reality both deliver a surrounding experience using a head-mounted display (though 360 video may also be experienced in a dome theatre) and both rely on spatial sound, but there are some significant differences that should be appreciated. 360 video is recorded with a spherical camera rig from a fixed point in space, and the viewer is placed exactly where the camera was, able to turn their head and look around, however, they can’t walk around and they do not experience parallax. This is may be described as offering three degrees of freedom, or 3DoF, to the participant. Virtual reality, on the other hand, is rendered, that is, a game engine computes the scene in real time from wherever the tracked head is located in 3D space and often the participant may move through space. This is described as six degrees of freedom, or 6DoF. We can think of 360 video as a panorama with a timeline, centered on the Albertian viewer who can now swivel. Virtual reality is what I call the presence-image, synthesized based on the position of the tracked head and body beholding it. The family resemblance is real, however, the ontology of the two forms are dramatically different.

Best practices are a working consensus about what tends to produce good outcomes for practioners doing work under similar constraints. Consider best practice as scaffolding rather than rules or laws. The purpose of learning best practices is to develop fluency that makes informed departure from established guidelines productive.1

Evolving best practices: 1. Record spatial audio (Ambisonics A or B) on location and mix in Ambisonics in post

Stereo or binaural recording is not going to cut it for 360 video sound unless you don’t let your viewer move their head when they watch your video. If viewers can move their head, then you’ll need to record spatial sound. Some 360 video cameras record in the Ambisonics format. For double system ambient sound recording for 360 video, consider using an Ambisonics microphone (e.g. Rode NT-SF1 or Sennheiser AMBEO VR). You’ll have to deal with the microphone appearing in the scene either straight above or directly below the camera, however, if you’re using a multiple lens camera like the Insta360 Pro, you can hide the microphone in one of the overlaps between lenses or place it in a rig just under the camera and mask it out. Professional sound mixing software (e.g. Reaper, Fairlight, ProTools, etc.) supports sound mixing Ambisonics audio and you can monitor your 360° mix on stereo headphones through binauralization. See also: Ambisonics Recording Kit Workshop.

Evolving best practices: 2. Record dialogue with wireless lavalieres

Record dialogue with wireless lavalieres. Given the wide frame of 360 videos, the on-camera microphone is going to record really good ambient sound but it’s not going to record good dialogue. It’s crucial to use wireless microphones to capture dialogue in a double-system sound configuration. In postproduction you can pan the dialogue with your subject or actor’s movement in the scene. Professional sound mixing software (e.g. Reaper, Fairlight, ProTools, etc.) supports panning mono sources in an Ambisonics mix. See also: Ambisonics Recording Kit Workshop.

Evolving best practices: 3. Avoid excessive camera movement

Avoid excessive camera movement. With 360 video you place your viewer in the center of the scene, allowing them to experience your stories from a first-person in the scene perspective, as if they are in the scene. Because of this, you want to avoid any discord between camera movements and the movement of the viewer’s body. They are embodied in the scene, therefore, their own bodily movements can be disorienting and in some situations cause nausea (what many people call “VR sickness.”). It is important to keep any camera movements very smooth, or not at all. If movement is required, offer the viewer a frame of reference or blur the peripheral vision during the movement (e.g. Google Earth). Good examples of fixed position 360 video documentaries include First Impressions (The Guardian, 2017), The Joy of Frogs (The Guardian, 2018). Image: thefulldomeblog.com, collection-of-360-video-rigs (accessed February 25, 2020, no longer available online).

Evolving best practices: 4. Avoid tilting the camera, keep the horizon level, and in most cases, shoot at eye-level

Your viewers will feel uncomfortable if the horizon is not level. If a shot is not level, fix it in postproduction. Avoid tilting the camera because this takes control away from the viewer and diminishes their sense of presence. For more details, see VR Video Formats Explained by Matt Rowell (360 Labs, January 25, 2019).

Evolving best practices: 5. Avoid panning (let the participant control panning on their own)

Allow the viewer to look around on their own terms, don’t pan the camera for them! VR sickness is when the movement does not match our own movements, disconnect between our head movement what what we expect to see. When people watch 360 video using a headset, you want them to feel as if they are in the midsts of the scene unfolding before them. Allow them to explore the scene from any perspective they choose by moving their head. Therefore, avoid panning the camera, as this takes control away from your viewer and diminishes their subjective experience of presence. 

Evolving best practices: 6. Spatial storytelling privileges space and mis-en-scène

Editing 360 video is an evolving craft, so give yourself time to test your work with an audience early and often. Unlike rectangular screen-based video which benefits from frequent cuts to keep the reptilian brain engaged and the mammalian brain constructing meaning from the juxtaposition of shots, 360 videos is a completely different beast. Your viewer needs time to construct meaning deductively as they explore the scene before being given another scene to process. Mis-en-scène takes on more importance, spatial storytelling is different from cinematic storytelling.

I was fortunate to see one of the best examples of this: Four Feet: Blind Date (Maria Belen Poncio, 2018, 19 min.) at Sheffield Doc/Fest in 2019. Anxious to explore her sexuality, Juana, an 18-year-old woman in a wheelchair, overcomes her fears, doubts, and an inaccessible city to meet ‘Felipe’ for a blind date. Together they discover how their bodies feel. This cinematic VR experience stands out because of the combination of excellent writing and storytelling-relevant use of point of view and how your gaze is guided throughout the experience as you are invited into Juana’s world from a perspective closer to her own. See: Short film ‘4 Feet: Blind Date’ portrays intimacy while wheelchair-bound by Isabella Balandran (Los Angeles Times, High School Insider, March 28, 2019)

Evolving best practices: 7. Direction is more about offering cues instead of framing and cutting as in traditional cinema

Since 360 video places your viewer in the center of the scene and have the ability to look around, you are now co-directing the experience with you. Keep in mind viewers will look around the scene based on what catches their attention. If there’s an important object or gesture you don’t want the viewer to miss, you will need to attract their attention in order to guide their gaze. There are a range of approaches for doing this, some are less subtle (e.g. the narrator saying “look down” in The Joy of Frogs) but you can also use sounds, object movement, graphics, lighting, text overlays, etc. to draw the viewers attention. You also need to write and edit with the understanding that viewers may not see everything that’s in the scene.

When people watch 360 video using a headset, you want them to feel as if they are in the midsts of the scene unfolding before them. Allow them to explore the scene from any perspective they choose by moving their head. Therefore, avoid panning the camera, as this takes control away from your viewer and diminishes their sense of presence. In Traveling While Black (Roger Ross Williams, 2019) is a cinematic VR experience about the Green Book, a critical guide for African-Americans struggling to travel safely in the United States during the Jim Crow era. This work makes excellent use of your ability to look around as the story unfolds. In one particularly effective scene, you can change your gaze to look at past and present.

Evolving best practices: 8. Contrary to early advice, jump cuts are acceptable, however, make sure they are motivated

The issue of jump cuts in 360 video: While narratively working as a flashback, this 360 video experience covers events that occurred before the time frame of television show’s first season, it is, in fact, a contemporary memory. We’re experiencing the protagonist Elliot’s current recollections of a much earlier event, and unlike some early advice to avoid jump-cuts in 360 video, The Mr. Robot VR Experience uses jump cuts without disorienting the viewer. Rules were meant to be broken, but it helps to understand the parameters around the rule you are breaking. See: The Mr. Robot VR Experience, Storytelling, and the Future of Immersive Media by Kendall Whitehouse (On Technology and Media, July 31, 2016) and 360 Jump Cuts: Avoiding Whiplash by Sarah Hill (Medium, November 28, 2016).

Assembling a 360 video + spatial audio production kit

Technology is constantly changing, however, at the time of the most recent revision of this presentation, this kit, which I assembled for a fiction project, provides an example of an effective solution to shooting 360 fictional, journalistic, and documentary work.

  • Camera: Insta360 ONE RS 1-Inch 360 Edition Camera with extra batteries
  • Camera Accessories: Insta360 Vertical Microphone Adapter for ONE X2 and RS Series, Insta360 ONE RS Invisible Mic Bracket (and velcro to hold the Tentacle Sync E mkII)
  • Camera Support: Benro Slim Travel Tripod (Carbon Fiber) and small grip gear (for mounting the camera with the microphone directly under the camera on the tripod)
  • Ambisonics Sound Recording: Sennheiser Amber VR Mic (spatial audio capture from camera position) with Sound Devices MixPre-6 II Mixer/Recorder with Ambisonics Plug-in and Sony MDR-7506 headphones; worth considering: for fiction work the spatial soundscape can be created during postproduction, so this component may be skipped.
  • Spot Microphone Recording: Tentacle Sync TRACK E Audio Recorder with Lavaliere microphone, one for each actor (position in 3D surround can be panned in postproduction)
  • Timecode: Tentacle Sync E mkII (for feeding timecode to Insta360 and MixPre); The MixPre can jam sync, however, the Tentacle simplifies sync since all devices can be synced with the Tentacle App on your smartphone; the Tentacle Sync E mkII feeding timecode to the camera also records reference audio; Use of a Timecode slate is optional.

Comments on this kit

A challenge to resolve when shooting fiction 360 video is how to record actor’s dialogue in a 360 environment. This is resolved by using Tentacle Sync TRACK E Audio Recorders with a lavaliere microphone as spot microphones on each actors for clear dialogue. This device records both audio and timecode, and can be synchronized with other Tentacle Sync devices with their smarphone app.

Recording spatial sound from the camera position is performed with a Sennheiser AMBEO VR Microphone (positioned with small grip gear right under the camera which can be composited out of the frame (especially if you use “look straight down” for navigation and menu access in the experience). Another option is to mount it above the camera in the seam between the two cameras and it can be composited out of the scene. The Sennheiser Ambeo VR Microphone is connected to a Sound Devices MixPre-6 II with the Ambisonics plug-in which supports recording Ambisonics audio and monitoring binaurally. While the MixPre-6 II is capable of jam-synching timecode, I feed it timecode with a Tentacle Sync E mkII since that allows me to synchronize the timecode of the MixPre-6 II, the Tentacle Sync E mkII connected to the camera and the Tentacle Sync TRACK E all in one fell swoop using the Tentacle Sync smartphone App. I can also use this with the Tentacle Sync Timebar Timecode Slate if I want to slate my shots. Every device in this configuration is either fed timecode or generates it’s own. This is the best architecture when working with a camera that does not support timecode.

The AMBEO VR Mic captures first-order ambisonics as A-format from its four tetrahedral capsules. With the Sound Devices Ambisonics Plug-in, A-format can be converted to B-format in the recorder, or the raw A-format can be preserved for conversion in post with Sennheiser’s free AMBEO A-B plugin. Shooting A-format for post conversion allows orientation corrections after the fact. The TRACK E handles spot-miking, which is essential, because an Ambisonic microphone six feet from a speaker captures the room more than the voice.

Mounting the mic directly under the camera on the tripod, with grip gear, is recommended. The AMBEO’s body and shock mount obscure the tripod column in the nadir, which means one nadir patch in post covers the mic, the tripod, and the grip hardware all at once. Much cleaner than trying to patch them separately.

Always carry a second battery. The ONE RS 1 runs roughly 50 minutes per battery at 6K, less in cold. Plan on two or three spares depending on the length of your shoot day. Storage-side, the 1-Inch 360 Edition writes at bitrates that demand UHS-I V30 at absolute minimum, however, V60 is the safer spec for long takes at 6K 30p. I usually purchase SanDisk Extreme SD cards. Cheap cards will drop frames silently and corrupt files.

Wind protection for the AMBEO VR microphone is essential. My kit includes a Rycote InVision Microphone Suspension that came with the microphone, along with a Rycote BBG Windshield and a Rycote BBG Windjammer. Better wind protection is offered by a Rycote Stereo Cyclone Ambisonic 1 Windshield System designed specifically for the Sennheiser AMBEO along with a Windjammer for high wind conditons.

Monitoring audio is a challenge with 360 video. While the MixPre-6 II handles headphone monitoring cleanly and in binaural with the Ambisonics plug-in, if you place it under the camera tripod, you will also have to rig a wireless connection (or hide a long headphone cable) to monitor recording, or run a long micrphone cable from the camera position to the location of the recorder (Sennheiser offers long extensions their special cable for the AMBEO VR microphone). That said, given the Tentacle Sync TRACK E Audio Recorders record 32-bit audio you don’t need to worry about setting levels with it. In addition, the MixPre-6 II supports a Wingman app that connects to the recorder with Bluetooth for remote operation.

To contact me use this contact form.

You are free to use these materials for educational and research purposes. Suggested attribution: Immersive Storytelling with 360 Video by David Tamés.

This presentation was revised on November 15, 2024, it is a condensed version derived from three guest lectures: (1) “Immersive Media Art and Storytelling,” March 1, 2022 for IHFILM 342-01 Digital Cinema, Professor Martin Roberts, Keene State College; (2) “Immersive Media Art and Storytelling,” April 14, 2021 for CDGD 567 VR for Designers and Storytellers, Professor Sofie Hodara, Dynamic Media Institute, MassArt); (3) “From Interaction to Immersion: How XR transforms media design,” February 25, 2020 for JRNL 3610 Digital Storytelling and Social Media, Professor John P. Wihbey, Northeastern University.

© 2020, 2024 by David Tamés, some rights reserved, released under the term of Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0) https://creativecommons.org/licenses/by-sa/3.0/ license, however, some materials contained herein are copyright by third parties and are used under fair use guidelines and cited whenever possible.

Photo: Giu Vicente, @giuvicente

Footnotes

  1. The term “best practice” is sometimes used cynically by industry professionals to mean “what lawyers or insurers require,” which is a narrower sense than the craft meaning of the term I use in this context; I do not want to evoke the “compliance connotation” of the term. For more on the concept and social process of best practices, see The Reflective Practitioner: How Professionals Think In Action by Donald A. Schon (Basic Books, 1984). ↩

Backlinks