One tradeoff that is common to achieve low video latency — in applications where the available data rate in a channel is constrained — is to refrain from sending I-frames. By their nature, I-frames are large and would clog the transmission in a constrained channel. Instead, smaller sub-regions can be encoded and sent, periodically refreshing the entire image. It turns out that HEVC, which has more flexibility in partitioning the video frame, is well suited for low-latency applications for this reason.
Low latency video is important for any channel where the video is being used as a part of the control loop: For instance, when the viewer of the video is controlling a joystick to change the camera position. Low latency is also important in videoconferencing, where people are having an interactive conversation.
But when is low latency not that important? Answer: When the application is not real time, which is the case for the vast majority of video that we consume. We may think that news or sports on broadcast TV is in real time, but any TV program is far from being real time. Even security video from an IP camera is not particularly real-time. That’s because when you allow for latency in your video delivery system, you can achieve a much lower overall bandwidth for the same quality level.
Contents
“Confusing second-screen app”
Last year Cardinal Peak had the opportunity to visit with a leading consumer electronics company for an engineering face-to-face meeting. We were shown a demo in which video from a set-top box was being displayed on both a TV and a tablet. For this project, new ways were being devised for controlling and viewing the video from the tablet. Typically this is known as a “second-screen” application designed to enhance the user’s TV-viewing experience.
To the visitors, however, the user interactions on that second screen became quickly confusing, because the video on the tablet was a few hundred milliseconds behind what the TV was showing. This short of a delay might seem inconsequential, but the discrepancy was really annoying. Didn’t I just see that? What is now?
The explanation given was that the video was first decoded by the STB (for subscriber authentication) then re-encoded by the smart TV and streamed wirelessly to the tablet to be decoded. The delay resulted from that extra re-encoding/decoding step.
“User bothered more by relative delay”
In the ensuing engineering discussion, suggestions were offered. What delay values would be acceptable to the user? What can be done to reduce the latency? For every question asked, there was an equally valid answer as to why it was not technically feasible to reduce the latency any further. It seemed like the discussion was going nowhere until someone suggested that maybe the user does not care about absolute latency and is simply bothered by the relative delay between the two streams.
So if the TV would also decode the same re-encoded video streamed to the tablet, the two streams would be better matched. The user would not know that the video on the TV is further delayed. The TV would need to perform a seemingly extraneous decoding step but to the user, the product offering would be near perfect, at least as far as video latency is concerned.