Content helps keep our world connected. From music and podcasts to sports and news, digital content must be available to users — everywhere, on every device.
To meet increasing consumer demands for a streaming media solution that brings people together through the content they love, a major media broadcast company contacted Cardinal Peak to discuss the development of a next-generation media streaming platform.
The Challenge
Previously, the customer’s media streaming platform relied on third-party services. While the offering had evolved to include several advanced features, like time-shifted playback to replay past content, it couldn’t achieve the company’s overall strategic goals. To scale the solution and meet the needs central to its evolving business plan, the customer needed to implement proprietary streaming media services.
Beyond reducing cost, owning its own streaming media platform allows our client to accelerate the development of new features and unify disparate services so the solution is easier to maintain at scale. Cardinal Peak was engaged to assist with the system architecture, implementation and delivery. In all phases of the project, we worked closely with our customer’s engineering and DevOps teams to develop a world-class media streaming platform that runs on state-of-the-art cloud services and is designed to scale.
The Solution
While many cost-effective off-the-shelf solutions are available, our customer’s needs, in this case, require a customized solution to achieve cost-efficiency and business goals. With consumers expecting solutions that stream content to everything from their smartphones to massive 4K TVs, collaborating with a partner that possesses in-depth experience with a wide range of streaming architectures is essential.
A customized media ingest and HLS streaming platform? By replacing costly third-party services, Cardinal Peak enabled proprietary, scalable streaming — delivering seamless, time-shifted and targeted content with robust failover and proactive monitoring.
How We Developed a Streaming Media Subsystem to Address Complex Integration Challenges
At the onset of the engagement, we pursued a streaming media platform design that included two subsystems.
Subsystem 1
The first subsystem focused on content ingest centered on an encoding service to acquire live and prerecorded content and metadata from the customer’s remote studios and other sources and package it for HLS streaming in the cloud. This stage included evaluating several third-party media encoding services and designing and estimating what it would take to build an encoding service. We, unfortunately, encountered numerous challenges with third-party options, from cost to lacking features required for ad insertion and metadata handling. Working with our customer to evaluate trade-offs, we ultimately decided to build a media ingest service.
Cardinal Peak designed and developed a containerized service to handle all aspects of media ingest, transcoding, metadata processing and recording to cloud storage. The solution included working with DevOps to devise and implement a Kubernetes solution to retain a number of instances equal to the number of active stream connections, plus a set of hot standbys for rapid failover. This proved to be especially challenging since our encoders needed to run 24/7, making our service a long-running process. Long-running processes like this are a cloud anti-pattern and took some time to solve. Our final solution scaled up and in to optimize cost while retaining robust failover capability.
In addition to the encoders, we also developed a centralized management service to manage and distribute encoder configuration and publish encoder status for metrics and monitoring through a pub/sub model using MQTT.
Decoupling the design allowed us to resolve issues in determining what encoder instance is handling what source for real-time instance management and monitoring. This approach also enables the option of deploying our encoder to edge hardware with AWS IoT and managing it with KubeEdge without impacting our encoder management services.
A move to edge hardware was interesting to our customer because it:
- Allows their teams to save cloud computing costs.
- Provides the option to eliminate the off-the-shelf hardware encoder they are currently using.
- Delivers an encoding option to third parties that may want to host streams on their platform.
- Allows them to manage and monitor their encoders the same way, irrespective of where they are deployed, lowering the burden on operations to maintain the system.
- Provides a mechanism that can be used to deploy other services at the edge in the future if needed.
The same encoder service was also designed to support a deployment model that accepts encoding jobs off a job queue to transcode “static” media files, such as ads and shows, into HLS-packaged versions for streaming, thereby maximizing our reuse of the code base for multiple use cases.
Subsystem 2
The second major subsystem is a highly scalable HLS server capable of handling targeted ad insertion and time-shifted playback of content. Again, third-party options were considered, but our customer’s strategic vision relied upon increasingly complex and customer-specific rules for media distribution that formed the basis of their business strategy. Cardinal Peak worked with the customer to design a highly scalable, rule-driven HLS stream builder capable of selecting content from multiple sources and synthesizing an outbound stream specific to the user.
Our HLS service effectively builds a completely custom stream of content for the end user or can simply rebroadcast the content it receives. The service also allows the seamless mixing of static content like ads, creatives, songs and whatever other content is desired.
Targeted ads may be inserted on ad breaks, or the service may offer alternate content if the user is a subscriber. The HLS service is designed to be stateless in order to facilitate traditional horizontal scaling, but streaming is unfortunately a stateful process. To achieve scale, we leveraged Redis and local in-memory cache to manage state and retain horizontal scaling for livestreaming efficiently and cost-effectively.
Time-shifted playback had the unfortunate side effect of bypassing some of our cache strategy in the original design. The original design included a SQL database to track stream segments, but we quickly realized the sheer volume of data streams generated, coupled with time-based access requirements, made the database a performance bottleneck — and a high-cost center to scale.
Working with our customer’s data architect, we were able to replace the database with Amazon S3, AWS’ Simple Storage Service, and effectively implement a time series database right on top of S3 without requiring a database at all. This significantly increased our scale and decreased costs.
Streaming Media Platform Development, Reporting & Metrics
In addition to the streaming media platform development, our client needs reports on what content and ads are played when and how frequently. These reports are crucial to claim ad revenue and pay applicable royalties on the content it broadcasts.
Another key feature of our streaming media platform design is sourcing detailed user consumption data of streams and inserted content. Our engineers evaluated several possible solutions, but given the high volume of data and the need to reliably retain said data and encrypt it inflight to protect personal information, we chose to use Kafka.
Kafka met our base data requirements and made our streaming data easily available to other business units. Other teams could consume our data to report on ad impressions, track song plays for royalty payments and perform business analytics to understand user engagement and improve programming.
Kafka Connect also made it easy for us to save our data to S3 for long-term storage, satisfy data retention requirements, and push data to other systems, such as Big Query or Snowflake, as needed.
Our solution also sourced application metrics and data that allowed for proactive application monitoring of operations. For example, when we stopped receiving data from a source or ads were no longer being received, an alert was raised. We integrated with Datadog for display and alert purposes.
Streaming Development to Deliver Content Everywhere
On the surface, the product we built is a classic media streaming platform. The solution essentially takes digital content from one or more sources, packages that content into a normalized format and streams it in a scaled fashion. The development of streaming solutions are typically more of an integration project, requiring careful setup and tuning, but we can also help with custom solutions to meet more demanding business needs and deliver the best streaming experience possible.
While many cost-effective off-the-shelf solutions are available, our customer’s needs, in this case, required a customized solution to achieve cost-efficiency and business goals. With consumers expecting solutions that stream content to everything from their smartphones to massive 4K TVs, collaborating with a partner that possesses in-depth experience with a wide range of streaming architectures is essential. At Cardinal Peak, we have worked on every aspect of audio/video streaming systems, including cloud and mobile, and our streaming service development expertise empowers brands to revolutionize how we entertain ourselves.
Contact us today to learn how we can help you with your next streaming development project!
