Keeping It All In Sync: Part One

Stefanie Diesi, Staff Engineer

These days, we employ audio and video applications in everything from IoT to embedded products, and one challenge that comes up time and time again is synchronization. Synchronization between audio and video streams is the most common case, but some applications require synchronization between multiple A/V streams, the latter being much more complicated. For the purposes of this two-part blog post, we’ll consider the basic case of synchronizing a single audio and video stream, today talking about why we need to synchronize, and later this week talking about how we do it.

Why do we need to synchronize?

Digital audio and video both consist of discrete-time approximations of a continuous system. A typical recording will have thirty or sixty video frames per second and about forty-four to forty-eight thousand audio samples per second, but there is wide variability of these values based on the capture device and the application. If there are separate capture devices for audio and video, it’s reasonable to envision a case where each device will perform its own encoding and even its own timestamping. We can visualize the audio/video synchronization with an illustration like the following where vertical lines mark video frames and horizontal lines denote audio samples. In real life, there can easily be hundreds of audio samples for every video frame,  but for simplicity’s sake, only four are illustrated here. This image represents the video frames (vertical lines) and audio samples (horizontal lines) for a single recording where the same color indicates that the data was collected at the same time. The clock used to timestamp a particular sample is commonly referred to as a media clock, and here’s what the A/V stream would look like if the audio and video capture devices share a media clock:

stefanie 1

However, when a scenario involves capture devices with separate media clocks, it is unlikely that those separate clocks will march in lockstep. (That kind of synchronization can be obtained using genlock.)  Separate media clocks mean that timestamps on a video and audio frame, captured at the same time, will drift apart over time like in this image:

avsyncprob

 

Due to the different capture rates of the camera and microphone, the playback is not synchronized. So now we know why synchronization is so important. On Friday, I’ll talk about how to adjust both audio and video accordingly in order to eliminate sync issues.

 

Cardinal Peak
Learn more about our Audio & Video capabilities.

Dive deeper into our IoT portfolio

Take a look at the clients we have helped.

We’re always looking for top talent, check out our current openings. 

Contact Us

Please fill out the contact form below and our engineering services team will be in touch soon.

We rely on Cardinal Peak for their ability to bolster our patent licensing efforts with in-depth technical guidance. They have deep expertise and they’re easy to work with.
Diego deGarrido Sr. Manager, LSI
Cardinal Peak has a strong technology portfolio that has complemented our own expertise well. They are communicative, drive toward results quickly, and understand the appropriate level of documentation it takes to effectively convey their work. In…
Jason Damori Director of Engineering, Biamp Systems
We asked Cardinal Peak to take ownership for an important subsystem, and they completed a very high quality deliverable on time.
Matt Cowan Chief Scientific Officer, RealD
Cardinal Peak’s personnel worked side-by-side with our own engineers and engineers from other companies on several of our key projects. The Cardinal Peak staff has consistently provided a level of professionalism and technical expertise that we…
Sherisse Hawkins VP Software Development, Time Warner Cable
Cardinal Peak was a natural choice for us. They were able to develop a high-quality product, based in part on open source, and in part on intellectual property they had already developed, all for a very effective price.
Bruce Webber VP Engineering, VBrick
We completely trust Cardinal Peak to advise us on technology strategy, as well as to implement it. They are a dependable partner that ultimately makes us more competitive in the marketplace.
Brian Brown President and CEO, Decatur Electronics
The Cardinal Peak team started quickly and delivered high-quality results, and they worked really well with our own engineering team.
Charles Corbalis VP Engineering, RGB Networks
We found Cardinal Peak’s team to be very knowledgeable about embedded video delivery systems. Their ability to deliver working solutions on time—combined with excellent project management skills—helped bring success not only to the product…
Ralph Schmitt VP, Product Marketing and Engineering, Kustom Signals
Cardinal Peak has provided deep technical insights, and they’ve allowed us to complete some really hard projects quickly. We are big fans of their team.
Scott Garlington VP Engineering, xG Technology
We’ve used Cardinal Peak on several projects. They have a very capable engineering team. They’re a great resource.
Greg Read Senior Program Manager, Symmetricom
Cardinal Peak has proven to be a trusted and flexible partner who has helped Harmonic to deliver reliably on our commitments to our own customers. The team at Cardinal Peak was responsive to our needs and delivered high quality results.
Alex Derecho VP Professional Services, Harmonic
Yonder Music was an excellent collaboration with Cardinal Peak. Combining our experience with the music industry and target music market, with Cardinal Peak’s technical expertise, the product has made the mobile experience of Yonder as powerful as…
Adam Kidron founder and CEO, Yonder Music
The Cardinal Peak team played an invaluable role in helping us get our first Internet of Things product to market quickly. They were up to speed in no time and provided all of the technical expertise we lacked. They interfaced seamlessly with our i…
Kevin Leadford Vice President of Innovation, Acuity Brands Lighting
We asked Cardinal Peak to help us address a number of open items related to programming our systems in production. Their engineers have a wealth of experience in IoT and embedded fields, and they helped us quickly and diligently. I’d definitely…
Ryan Margoles Founder and CTO, notion