How to Produce a Virtual Ensemble Project
Donovan Seidle of the Calgary Philharmonic Orchestra takes you through the process of creating a physically distant or virtual ensemble. Watch the full Facebook Live video here, or read this post in blog format below.
I use a 2010 MacPro for editing and producing. I’ve found it a great operating system for my creativity, and love its workflow. However, this type of work can be done on either a Mac or PC, and many of the tools I use work on either operating system. My list of tools is not exhaustive, and you’re free to explore the many options out there – there are a ton of great tools, and free ones too! We’ll go through this workflow on both Mac and PC. I’m not a video editor: I’m approaching this more as an audio guy, so if there are people on your team, or yourself that have tips and tricks that are better than these, please use them!
Our goal is to create synchronized music videos from many participants using current online technology. For ensemble work, there isn’t a solid solution to practice or perform together online that works. This is because of timing latency (the time it takes for something to transmit from one network location to another over the internet). There’s also trouble with audio dropouts, or unintended noise reduction. Some of these can be dealt with in the settings of Zoom or WebEx, but the quality of the audio experience for an audience is lacking – and we want to produce something better than that.
An ensemble has to be synchronized – in a recording session this happens with a conductor and a click track: we won’t have the luxury of being able to tune to one another, and adjust our tempo and rhythmic timing to one another – so a click track is a sure way forward. In order to do this, we need tools. Whatever platform you’re doing this on, here’s what we need:
Audio Converter Application: A way to convert audio to whatever format we want. Switch, $25 – Mac & PC
Video Converter Application: To downsample and standardize the many videos we will be receiving – we have little control over the devices being used to record. Uniconverter, $70 – Mac & PC
Multi-track Audio Editor (Digital Audio Workstation): To change timing, mix together, and tweak our final mix using all of the audio from our participants. Logic Pro X, free trial – Mac. Reaper, free – Mac. Cuckos Reaper or Cakewalk, free – PC.
Audio Editor: If needed Audacity, free – Mac & PC.
Video Editor: To compile all our videos into a mosaic, and make them all start at the same time. Final Cut Pro, free trial – Mac. Davinci Resolve, free – Mac. Filmora, subscription- Mac & PC. Sony Vegas Pro, $70 – PC. Adobe Premiere Pro, subscription – PC.
Define your project goals
Clearly define your project goals. Start small, and once you get the hang of it, you’re ready to expand. Try a test project first, which you can do on your own! Some guiding questions:
- Is this an audio-only project, or with video?
- How many performers do you need? Think about the final layout and “look” of your mosaic. Will it be an unbroken shot from start to finish, or will you have featured performers?
- How long is the piece?
- Are there tempo changes?
- Will the participants record sections of music, or the whole thing all at once?
You can approach your project differently depending on your answers. Make sure you answer all these questions – they will inform you of what you need.
Synchronizing players
In a performance context, we are all synchronized on stage with a conductor, our ears, and our brains; which navigate the sometimes tricky dissonance between what we see, and what we hear. For a “recording session” like this, a well-defined beat that goes WITH the click, is really helpful. We don’t have the benefit of being able to push and pull our timing against our colleagues as we play – but we have a precise click that will keep us all in line as we separately record our music.
Ideally, we want some natural “push and pull” of the tempo; not just a steady metronome (which is also an option, and great for pop material).
The simplest solution
The simplest solution that allows us to collaborate with other musicians is an app called Acapella. You can record a video/audio track, and then record over the top of this up to 8 more times as we listen. By listening to ourselves carefully, we can stay on track, and harmonize and arrange.
Pros and cons: This cool tool provides us with a simple click, and our own recordings as a guide track. But after we record something, we can’t adjust timing or the mix; and the small number of screens isn’t a workable solution for the orchestras we love to work with, nor is listening to a single line of music suitable for synchronizing a whole symphony. We’re going to have to create something more flexible and precise.
You’re probably thinking “why can’t I just have a conductor on the screen, waving the all-powerful baton?” The success of that is dependent on consistency between your players: a baton largely gives subjective information: how precise is your beat? What happens when the music gets faster or slower, more rhythmic? A conductor is not meant to be a metronome: and there’s a lot more happening in an ensemble than paying attention to the beat. Your players are responding to each other, and the information dissonance between what they see, and what they hear, and navigating that difference in the moment. In this medium, they’re absolutely unable to respond to another player, unless there’s a guide track, and objective tempo information like a CLICK. A conductor is certainly helpful for conveying emotional information and direction in this context
Simple Click
Our next sync method begins with something steady: a metronome. This is the simplest click track we can have. For a steady click, you can even just record your metronome into your recording device.
But if we want to make sure our music has tempo or meter changes, we have to design that in.
Designed Click Track Grid
Our next sync method is a designed click track grid. This is the approach that most audio productions and movie scoring composers use.
This method is precise, it mirrors the scores that we have (in our heads or in our hands, as closely as we design into the grid, at least). I often use this method for my writing. It is as precise as a metronome and follows the score as closely as I’ve designed my grid.
Adaptive Click
We can also have the music “follow” an existing guide track. Let’s create that music guide, and do a technique called Beat Mapping, or Reclocking in order to match the grid of our project to the ever moving fractional tempo changes of all those individual beats.
This is the most musical solution. This music can come from anywhere – you could play it in, you could conduct as you hummed the part (which could be muted after you place your click in), it could be a pre-existing recording that you like…. Anything.
We have to make sure video and audio will be synchronized the whole way through the project. I find a visual and aural snap or clap to be very useful when lining up video and audio. We’ll do this in this next video. I’ll be using this final video as a little project, and apply the next steps to it, moving forward.
Your Guide Video
Include a greeting, a sync point (clap together on a count), and a few beats before the music starts. You could also include in your email to participants: a guide video link, sheet music, participant video submission link, recording instructions.
Receiving Submissions
Sending out the video: Guides like this can be created and sent out to your participants; you can host the guide on YouTube, your website, or send it to your players as a video file via email. Receiving large video files can be tricky. I like to use Dropbox, but Google Drive and WeTransfer work well for this too.
Processing Submissions
Once we receive submissions from our performers, the EASIEST way to proceed is just to bring them into our video editor, align them, and set up our mosaic. To solve issues of balance, it’s usually best to do some audio editing at this stage, and to make sure that all of our videos are in the same format.
Our first step is to make all the videos the same format. I like to work with Quicktime videos smaller than 720p, especially if we’re making a mosaic.
Math Moment: If we want our videos to have a 16:9 aspect ratio, this means something like 1920 by 1080 (1080p), or 1280 by 720 (720p). 720p or 1080p refers to the RESOLUTION of the video: how many pixels are in every frame. Less pixels means a smaller file size and less resolution. If you enlarge a tiny video too much, it looks pixelated and strange. Think about your final product, and decide whether you need full resolution video (for close-ups), or if small is fine for everyone.
Audio Processing
I like to edit the audio – sometimes the mix or balance isn’t right, because of recording volume. Sometimes we need some studio magic to make it sound great. First the simple steps:
There are likely more things to tweak. We can start to edit timing, and if we really want, get into pitch correction, and advanced stereo imaging.
We have our video mosaic already, so let’s bring in our completed audio, line it up, with the click point, and then create a fade in and fade out.
At this point, you have your video. You can host it on your website, send it out to your friends, family and social media platforms (as a link to a YouTube video, or uploading the file itself).
Review
- Set your goals – know what you want to creatre
- Create your guide (click, guide video/audio track, sync point clap)
- Email participants (include how you want to receive submissions)
- Receive & process submissions (convert, trim, separate audio, create video, process audio, assemble and render)
- Deliver / host final product