Live video and audio content is no longer a novel phenomenon in today’s world of online live streaming. As technology advances, it’s transforming how we consume content — and the demand for digital audio and video products is booming. In spite of streaming media’s popularity, one essential element of the puzzle remains overlooked: encoding formats.
At its simplest, streaming media involves the delivery of media content over a computer network in real time. But without encoding, streaming media wouldn’t be possible. That’s right, no bumping your favorite tunes on Spotify, no Netflix and chill, no binging your favorite shows on HBO Max, Hulu or another streaming service, and no connecting with colleagues over Zoom.
But what is streaming media encoding? Are there different kinds of encoders? Which one is best for your project? This blog post will highlight the most popular encoding formats and how they are used to package media for streaming.
Contents
What is Streaming Media Encoding?
Encoding is converting raw media (whether audio, voice or video encoding) into a digital format that’s compatible across many different devices.
To send streaming media content across the internet, live or recorded data must be compressed to be efficiently and cost-effectively sent to another location. With encoding, videos and audio files are typically shrunk down from gigabytes to megabytes for more efficient storage and transportation.
Whether encoding occurs on an app, as part of a digital product, or on a camera or recording device, encoding is vital to streaming since bandwidth limitations and bulky file sizes would otherwise hinder how data is transmitted and received. Uncompressed files would make streaming content untenable, and content has to be in a format that most smartphones, laptops, PCs and other gadgets can interpret.
A live or recorded stream is sent to a server, where the content is chunked into smaller segments, with each a few seconds in length. The video/audio content is then encoded using one of many encoding formats before the device at the receiving end of the stream decodes the encoded data back to its raw form and plays the content. In practice, we use media codecs to facilitate encoding and decoding of media for this process.
Encoding Compression & Decompression Process
Part of the encoding process includes containerization. Audio and video are usually put into a container for storage and broadcast. Containerization is done for a number of reasons but one of the most important is to present the audio and video together with timing information so the player can keep the content in sync. Inserting subtitles or multiple audio and video tracks into some containers to carry multiple streams and alternate tracks for streams with alternate camera angles or languages are also possibilities. MPEG-2 transport streams are the historical container-favorite format due to adoption by the cable industry, yet the modern growing favorite is MPEG-4 (mp4) — not to be confused with fragmented MPEG-4, which is used to chunk up media for streaming and is not technically considered a container. On the way in, we “multiplex” the video and audio together into a container, and on the way out of the video file, we “demultiplex” back into the elementary audio and video streams.
Now that we have a basic understanding of encoding, let’s dive into different kinds of codecs and their tradeoffs.
Different Codecs and Trade-Offs
Codecs are devices or programs with algorithms that perform compression and decompression of the data stream. They are compression technologies content distributors use to shrink content into a streamable size and format. Made up of an encoder and decoder, codecs compress and decompress files and represent the compressed data in a standard format.
Lossless and Lossy Codecs
There are two kinds of codecs: lossless and lossy. Lossless codecs reproduce the exact same file after decompression, while lossy codecs yield a similar version of the file but not an exact duplicate after decompression. Unfortunately, some lossless codecs can’t adequately compress data rates for streaming, but as long as you have the bandwidth you can stream it. The trade-off for lossy codecs, however, is compression vs. quality — the more compressed the file, the lower the quality.
Other compression trade-offs include the application’s visual quality requirements, the transmission channel’s environmental characteristics (speed, latency and error) and the source content format. Compression also assumes trade-offs between the visual quality requirements and the application’s other needs, including storage, latency tolerance, etc. Finally, real-time encoding implementation might demand increased processing, which can drive costs up.
Video Codecs
Video codecs compress video data, while audio codecs do the same for recorded sound, and the rest of this section will highlight the most common video and audio codecs.
H.264/AVC (Advanced Video Coding)
The H.264 video codec is the most common video codec. Also known as AVC (Advanced Video Coding), H.264 is the industry-standard video compression technology, helping creators record, compress and distribute their online content. H.264 is widely supported since it delivers good video quality at lower bitrates.
Despite not being the most cutting-edge compression technology, H.264 makes up the vast majority of encoding output. It plays on virtually any device, is frequently produced in the MPEG-4 container format, is often incorporated with the AAC audio codec and can be packaged into .mp4, .mov, .F4v, .3GP and .ts containers.
If you wish to use the HLS streaming protocol defined by Apple with Apple devices, you must take an additional step and package H.264 media as fragmented MPEG-4 segments or use an MPEG-2 transport container. Apple devices only support H.264 and H.265 video with HLS. Apple’s HLS website has full details.
Learn more about H.264 in our blog posts on the world’s smallest H.264 encoder and the H.264 sequence parameter set and picture parameter set.
H.265/HEVC (High Efficiency Video Coding)
Also called High Efficiency Video Coding or MPEG-H Part 2, H.265 is a video compression standard designed as a successor to H.264. The H.265 codec compresses information more efficiently than H.264, resulting in video files of comparable quality that don’t take up as much storage space and require less bandwidth to stream.
Unlike the H.264 codec, H.265 supports 8K resolution but can take significantly longer — up to 4x — to encode than its predecessor. By creating smaller files, H.265 decreases the bandwidth required to view these streams, making it a great option for high-resolution streaming, but the patent and royalty drama surrounding H.265 has resulted in the development of the AV1 codec (see more below).
VP9
Developed by Google, VP9 is an open, royalty-free video coding format. The primary competitor to H.265, VP9 is supported by YouTube, Chrome, every Android phone, all new iOS devices, Mozilla’s Firefox and Apple’s Safari — making it second behind just H.264/AVC regarding compatibility across browsers and devices.
A more advanced and higher-quality compression technology than H.264/AVC, VP9 is a solid option because it performs about as well as H.265/HEVC and is virtually as compatible as H.264. If the longevity of the H.264 codec indicates what’s next for VP9, the codec will continue providing value for some time.
AV1
The first codec released by the Alliance for Open Media, the AV1 codec is also open and royalty-free. Built as a successor to VP9, AV1 intends to eliminate the royalties associated with H.265 by democratizing high-quality video delivery and playback.
With tests exhibiting the achievement of 34%, 46.2% and 50.3% higher data compression than other solutions, the AV1 codec is significantly more efficient. The challenge for AV1 today is that devices capable of decoding AV1 streams are lacking despite recent Amazon-led efforts to increase adoption, so the codec is not yet supported on a large scale.
VVC/H.266 (Versatile Video Coding)
Finalized in 2020, H.266 — aka Versatile Video Coding (VVC) — is a block-based hybrid codec, making it very complex and very capable. Intended to take the place of H.265 and H.264, the H.266 codec was created to work with everything from 4K streaming to 16K streaming. Unfortunately, it faces the same royalty issues as its predecessors.
Despite those concerns, VVC is a new standard for compressing and sharing video files in high-quality formats that companies should keep their eyes on. In recent years, developers have made significant progress in increasing bandwidth savings and overall performance, but entirely too many unknowns surrounding the codec persist.
Video Encoding Formats — Advantages and Trade-Offs
| Encoding Format | Advantages | Trade-Offs |
|---|---|---|
| H.264 / Advanced Video Coding (AVC) |
|
|
| H.265 / High Efficiency Video Coding (HEVC) |
|
|
| VP9 (Google) |
|
|
| AV1 (Alliance for Open Media) |
|
|
| H.266 / Versatile Video Coding (VVC) |
|
|
Audio Codecs
Since most video is also captured with audio, the audio component must also be addressed with audio codecs to effectively package media for streaming.
Advanced Audio Coding (AAC)
The most common audio codec, AAC — or Advanced Audio Coding — is a coding standard for lossy digital audio compression. Designed to succeed the MP3 format (see more below), AAC generally achieves higher sound quality at the same bit rate. AAC is used by Android, iOS, iTunes and YouTube, but higher-quality alternatives exist.
Note that AAC supports multiple profiles, each with increasingly better quality per bit. AAC-LC is the lowest quality per bit profile. Despite AAC-LC’s relatively poor performance, it is also royalty-free and supported by all devices that claim AAC support. The more advanced and better quality HE-AAC profile requires royalty payments for encoders (decoders are always royalty-free) but can achieve significantly better bit rates without losing quality. Most modern players support HE-AAC, but some older players may not.
MP3
Originally defined as the third audio format of the MPEG-1 standard, MP3 is a coding format for digital audio. The file format that arguably made portable digital media possible and popularized digital media players like Apple’s iPod, allowing users to exchange audio at scale online and carry hundreds of songs in their pockets, MP3 is supported by nearly every audio device but offers inferior compression for perceived quality when compared to AAC. Despite the better quality, AAC is less popular than MP3 which enjoys a massive market share.
Other Audio Codecs
Other audio codecs include AC-3, G.711, Opus, Speex and Vorbis.
AC3 is a lossy format designed for the efficient encoding of surround sound. Today, it is also used for applications such as TV broadcasts, satellite radio broadcasts, digital video streaming, game consoles and more. G.711 is a narrowband audio codec that provides toll-quality audio. Originally designed for use in telephony, recent G.711 enhancements utilize lossless data compression to reduce bandwidth usage and increase audio quality by increasing bandwidth.
Opus is the highest-quality lossy audio format. While it is open-source and royalty-free, it has yet to be widely adopted. Opus will, however, make Speex and Vorbis obsolete. Specifically tuned for the reproduction of human speech, Speex is an audio compression codec used on VoIP applications and podcasts. Finally, Vorbis was designed as a nonproprietary alternative to AAC but is less advanced than Opus.
Audio Encoding Formats — Advantages and Trade-Offs
| Encoding Format | Advantages | Trade-Offs |
|---|---|---|
| Advanced Audio Coding (AAC) |
|
|
| MP3 |
|
|
| AC3 |
|
|
| G.711 |
|
|
| Opus |
|
|
| Speex |
|
|
| Vorbis |
|
|
Together, audio and video codecs are the lifeblood of streaming media. Involved at every step from shooting to editing to encoding streaming media files for delivery, codecs enable A myriad of encoding formats, which have their own trade-offs.
What Codec to Choose When Packaging Media for Streaming
When it comes to streaming media, choosing the right codec can make a huge difference in the quality of the user experience. The most common mistake we see is organizations opting for the “best” quality at the lowest bandwidth and using a codec that doesn’t work with 90% of the devices on the planet.
Usually, people are sending video content to mobile or web devices, so considering which codecs are supported by iOS, Android and the web is important. Whether users will be utilizing Sonos, Alexa, Amazon Fire, set-top boxes or other popular media clients is another important factor since each has its own list of supported technologies — it’s a veritable Rubik’s cube!
The table below is a partial list of video codecs and whether they are supported by iOS, Android and some common web browsers. In addition, a given format may not allow encoding due to legal issues such as royalties, and the encoding may not be compatible when mixed with certain container formats. Streaming support may be more limited still and of course, things are always changing.
Video Codecs Supported by iOS, Android and Web Browsers (by Version)
| Encoding Format | iOS | Android | Chrome | Edge | Safari | Firefox | Opera |
|---|---|---|---|---|---|---|---|
| H.264 / Advanced Video Coding (AVC) | 8+ | 3.0+ | 4+ | 12+ | 3.2+ | 35+ | 25+ |
| H.265 / High Efficiency Video Coding (HEVC) | 11+ | 5.0+ | 107+ | 12+ | 11+ | No | 94+ |
| VP9 (Google) | No | 4.4+ | 29+ | 14+ | 14 (macOS) & 15 (iOS) | 28+ | 10.6+ |
| AV1 (Alliance for Open Media) | No | 10+ | 70+ | No | No | 67+ | 57+ |
Note that some browsers may not have the same support for desktop and mobile versions. They tend to be closely but not exactly matched.
Also, H.266/Versatile Video Coding (VVC) hasn’t been widely adopted enough for general use. The standard was finished in mid-2020 but may be experiencing delays due to 8k and 16k video adoption slowing and complicated royalties.
For more information, check out this reference guide on video codecs and audio codecs, or “can I use” browser support tables.
For a deeper understanding of streaming media, check out our blogs on understanding and selecting streaming protocols, best practices for using WebRTC for interactive video streaming, and considerations for scaling streaming apps.
Reach Our Experts in Selecting Streaming Media Codecs
At the end of the day, the success of your streaming media development project goes well beyond selecting the right codec for compression. Many other considerations impact how streaming media must be packaged for delivery.
The Cardinal Peak team recognizes that consumer demands are evolving and has the expertise across a range of streaming architectures, video product development and audio product design to deliver the best possible streaming experience. Connect with our streaming media experts to learn how we can create a custom streaming solution that balances quality, cost and client support.