Keeping It All In Sync: Part Two

Stefanie Diesi, Staff Engineer

Earlier this week, I wrote about the importance of synching audio and video, something that we use in many applications. We’ve already discussed why it’s necessary, and today I want to talk about the different ways to do it, some easier than others.

Adjust the Video
The most common and easiest way to correct for these offsets is to adjust the presentation time of the video frames to match the timestamps on the audio samples. This is a relatively simple adjustment to implement, and there’s a low impact on the user. If a video frame stays on the screen for 32 or 34 ms instead of the standard 33 ms, the viewer is unlikely to be able to tell the difference. As this image shows, small adjustments to the display time of video frames over time can easily correct for differences in the capture clocks. Here, the solid lines indicate the playback without modification, and the dashed lines indicate the playback after adjusting the video to match the audio.

stefanie 2

Adjust the Audio
Some applications don’t allow us to adjust the playback rate of the video, or perhaps the audio and video must be synchronized to an external clock. In these cases, we have to adjust the audio, which is more complicated. The added difficulty is due to the fact that an audio sample has a duration. This is easiest to understand in the context of pausing a movie. The last video frame remains on the screen for as long as the movie is paused. On the other hand, the last audio sample plays and then there’s silence. This property is why we can’t just adjust the start time of an audio sample. Moving a sample earlier would clobber the previous sample, and moving it later would result in a gap in the audio. Instead, we have to look into different options.

Adjust Audio the Easy Way
The first—and far easier—method for adjusting audio to match a particular clock is to simply drop or repeat a sample. We track the difference between the desired time and the current timestamp of the audio sample. Whenever that offset is greater than the duration of a sample, we add or drop a sample to make up the difference. In this example, the audio clock is running slow, so several samples are dropped to help the audio catch up to the video.

While this method is by far the easiest audio adjustment to implement, it’s also the most coarse correction. The listener may or may not be able to detect the audio discontinuities depending on the sample rate and the type of audio being recorded (a college lecture versus a symphony for example).

stefanie 4

Adjust Audio the Hard(er) Way
If simply adding or dropping samples periodically results in unacceptable audio artifacts, we need to try a more sophisticated technique: approximate the audio samples. Instead of dropping a sample, we could take the average of the sample to be dropped and the one after it. The average sample would replace the two input samples. This method is a linear approximation.

Higher-order approximations can be used as well. These involve fitting a curve or a spline to a series of N audio samples and then creating either N+m or N-m audio samples based on that approximation. Finally, we could perform a Fourier Transform on batches of samples to obtain a very high fidelity model. Then, use this model to extract a different number of samples that closely match the original input. Below is a simple representation of how the approximated audio samples (dashed black lines) might represent the original audio samples using these higher order approximations.

stefanie 5

Each of these audio adjustment options has less noticeable artifacts than simply dropping or repeating a sample, but they also have increased cost both in terms of processing power and computation time. So much so that the Fourier Transform option may only be practical for post-processing a recording and not for real-time applications.

What do I choose?
As with most design decisions, there is no right answer. All of the methods for correcting A/V sync problems have their own advantages and disadvantages. Each choice will need to be evaluated against the application, the requirements, the hardware, and the user experience to choose the option that best suits the needs of the project. We’d love to hear your comments about which methods have worked best for you!

 

Cardinal Peak
Learn more about our Audio & Video capabilities.

Dive deeper into our IoT portfolio

Take a look at the clients we have helped.

We’re always looking for top talent, check out our current openings. 

Contact Us

Please fill out the contact form below and our engineering services team will be in touch soon.

We rely on Cardinal Peak for their ability to bolster our patent licensing efforts with in-depth technical guidance. They have deep expertise and they’re easy to work with.
Diego deGarrido Sr. Manager, LSI
Cardinal Peak has a strong technology portfolio that has complemented our own expertise well. They are communicative, drive toward results quickly, and understand the appropriate level of documentation it takes to effectively convey their work. In…
Jason Damori Director of Engineering, Biamp Systems
We asked Cardinal Peak to take ownership for an important subsystem, and they completed a very high quality deliverable on time.
Matt Cowan Chief Scientific Officer, RealD
Cardinal Peak’s personnel worked side-by-side with our own engineers and engineers from other companies on several of our key projects. The Cardinal Peak staff has consistently provided a level of professionalism and technical expertise that we…
Sherisse Hawkins VP Software Development, Time Warner Cable
Cardinal Peak was a natural choice for us. They were able to develop a high-quality product, based in part on open source, and in part on intellectual property they had already developed, all for a very effective price.
Bruce Webber VP Engineering, VBrick
We completely trust Cardinal Peak to advise us on technology strategy, as well as to implement it. They are a dependable partner that ultimately makes us more competitive in the marketplace.
Brian Brown President and CEO, Decatur Electronics
The Cardinal Peak team started quickly and delivered high-quality results, and they worked really well with our own engineering team.
Charles Corbalis VP Engineering, RGB Networks
We found Cardinal Peak’s team to be very knowledgeable about embedded video delivery systems. Their ability to deliver working solutions on time—combined with excellent project management skills—helped bring success not only to the product…
Ralph Schmitt VP, Product Marketing and Engineering, Kustom Signals
Cardinal Peak has provided deep technical insights, and they’ve allowed us to complete some really hard projects quickly. We are big fans of their team.
Scott Garlington VP Engineering, xG Technology
We’ve used Cardinal Peak on several projects. They have a very capable engineering team. They’re a great resource.
Greg Read Senior Program Manager, Symmetricom
Cardinal Peak has proven to be a trusted and flexible partner who has helped Harmonic to deliver reliably on our commitments to our own customers. The team at Cardinal Peak was responsive to our needs and delivered high quality results.
Alex Derecho VP Professional Services, Harmonic
Yonder Music was an excellent collaboration with Cardinal Peak. Combining our experience with the music industry and target music market, with Cardinal Peak’s technical expertise, the product has made the mobile experience of Yonder as powerful as…
Adam Kidron founder and CEO, Yonder Music
The Cardinal Peak team played an invaluable role in helping us get our first Internet of Things product to market quickly. They were up to speed in no time and provided all of the technical expertise we lacked. They interfaced seamlessly with our i…
Kevin Leadford Vice President of Innovation, Acuity Brands Lighting
We asked Cardinal Peak to help us address a number of open items related to programming our systems in production. Their engineers have a wealth of experience in IoT and embedded fields, and they helped us quickly and diligently. I’d definitely…
Ryan Margoles Founder and CTO, notion