Equalized video latency is often more important than absolute latency

Tho Autruong

One tradeoff that is common to achieve low video latency—in applications where the available data rate in a channel is constrained—is to refrain from sending I-frames. By their nature, I-frames are large and would clog the transmission in a constrained channel. Instead, smaller sub-regions can be encoded and sent, periodically refreshing the entire image. It turns out that HEVC, which has more flexibility in partitioning the video frame, is well suited for low-latency applications for this reason.

Low latency video is important for any channel where the video is being used as a part of the control loop: For instance, when the viewer of the video is controlling a joystick to change the camera position. Low latency is also important in videoconferencing, where people are having an interactive conversation.

But when is low latency not that important? Answer: When the application is not real time, which is the case for the vast majority of video that we consume. We may think that news or sports on broadcast TV is in real time, but any TV program is far from being real time. Even security video from an IP camera is not particularly real-time. That’s because when you allow for latency in your video delivery system, you can achieve a much lower overall bandwidth for the same quality level.

“Confusing second-screen app”

Last year Cardinal Peak had the opportunity to visit with a leading consumer electronics company for an engineering face-to-face meeting. We were shown a demo in which video from a set-top box was being displayed on both a TV and a tablet. For this project, new ways were being devised for controlling and viewing the video from the tablet. Typically this is known as a “second-screen” application designed to enhance the user’s TV-viewing experience.

Vintage TV

To the visitors, however, the user interactions on that second screen became quickly confusing, because the video on the tablet was a few hundred milliseconds behind what the TV was showing. This short of a delay might seem inconsequential, but the discrepancy was really annoying. Didn’t I just see that? What is now?

The explanation given was that the video was first decoded by the STB (for subscriber authentication) then re-encoded by the smart TV and streamed wirelessly to the tablet to be decoded. The delay resulted from that extra re-encoding/decoding step.

“User bothered more by relative delay”

In the ensuing engineering discussion, suggestions were offered. What delay values would be acceptable to the user? What can be done to reduce the latency? For every question asked, there was an equally valid answer as to why it was not technically feasible to reduce the latency any further. It seemed like the discussion was going nowhere until someone suggested that maybe the user does not care about absolute latency and is simply bothered by the relative delay between the two streams.

So if the TV would also decode the same re-encoded video streamed to the tablet, the two streams would be better matched. The user would not know that the video on the TV is further delayed. The TV would need to perform a seemingly extraneous decoding step but to the user, the product offering would be near perfect, at least as far as video latency is concerned.

Categories: Tho, Video

Cardinal Peak
Learn more about our Audio & Video capabilities.

Dive deeper into our IoT portfolio

Take a look at the clients we have helped.

We’re always looking for top talent, check out our current openings. 

Contact Us

Please fill out the contact form below and our engineering services team will be in touch soon.

We rely on Cardinal Peak for their ability to bolster our patent licensing efforts with in-depth technical guidance. They have deep expertise and they’re easy to work with.
Diego deGarrido Sr. Manager, LSI
Cardinal Peak has a strong technology portfolio that has complemented our own expertise well. They are communicative, drive toward results quickly, and understand the appropriate level of documentation it takes to effectively convey their work. In…
Jason Damori Director of Engineering, Biamp Systems
We asked Cardinal Peak to take ownership for an important subsystem, and they completed a very high quality deliverable on time.
Matt Cowan Chief Scientific Officer, RealD
Cardinal Peak’s personnel worked side-by-side with our own engineers and engineers from other companies on several of our key projects. The Cardinal Peak staff has consistently provided a level of professionalism and technical expertise that we…
Sherisse Hawkins VP Software Development, Time Warner Cable
Cardinal Peak was a natural choice for us. They were able to develop a high-quality product, based in part on open source, and in part on intellectual property they had already developed, all for a very effective price.
Bruce Webber VP Engineering, VBrick
We completely trust Cardinal Peak to advise us on technology strategy, as well as to implement it. They are a dependable partner that ultimately makes us more competitive in the marketplace.
Brian Brown President and CEO, Decatur Electronics
The Cardinal Peak team started quickly and delivered high-quality results, and they worked really well with our own engineering team.
Charles Corbalis VP Engineering, RGB Networks
We found Cardinal Peak’s team to be very knowledgeable about embedded video delivery systems. Their ability to deliver working solutions on time—combined with excellent project management skills—helped bring success not only to the product…
Ralph Schmitt VP, Product Marketing and Engineering, Kustom Signals
Cardinal Peak has provided deep technical insights, and they’ve allowed us to complete some really hard projects quickly. We are big fans of their team.
Scott Garlington VP Engineering, xG Technology
We’ve used Cardinal Peak on several projects. They have a very capable engineering team. They’re a great resource.
Greg Read Senior Program Manager, Symmetricom
Cardinal Peak has proven to be a trusted and flexible partner who has helped Harmonic to deliver reliably on our commitments to our own customers. The team at Cardinal Peak was responsive to our needs and delivered high quality results.
Alex Derecho VP Professional Services, Harmonic
Yonder Music was an excellent collaboration with Cardinal Peak. Combining our experience with the music industry and target music market, with Cardinal Peak’s technical expertise, the product has made the mobile experience of Yonder as powerful as…
Adam Kidron founder and CEO, Yonder Music
The Cardinal Peak team played an invaluable role in helping us get our first Internet of Things product to market quickly. They were up to speed in no time and provided all of the technical expertise we lacked. They interfaced seamlessly with our i…
Kevin Leadford Vice President of Innovation, Acuity Brands Lighting
We asked Cardinal Peak to help us address a number of open items related to programming our systems in production. Their engineers have a wealth of experience in IoT and embedded fields, and they helped us quickly and diligently. I’d definitely…
Ryan Margoles Founder and CTO, notion