An essential element of modern cloud architecture, the best message queues provide — among myriad additional benefits — communication and coordination for distributed applications. They significantly simplify coding and improve the performance, reliability and scalability of applications decoupled into smaller, independent building blocks.
As discussed in previous posts, a myriad of important factors, including how much your organization is willing to spend and how much effort you’re willing to exert, need to be considered when determining the best message queue to suit your application. In part one of this blog series, choosing a queuing service, we established a computer science 101 background into queues and developed an understanding of how to choose the best queue data structure before gaining awareness of queue pricing cost drivers in the series’ penultimate post. Armed with this essential knowledge, we can now dive into some of the generally accepted techniques to adhere to when selecting the best messaging queue for you.
Contents
Message Queue Best Practices
When it comes to choosing the best message queue for your application, the advantages are use case-dependent. However, whether you’re using your queue for cross-service communication, consuming a significant amount of data from your clients to do analytics, integrating applications, scheduling or leveraging the power of a queue data structure to perform some other task, following the message queue best practices outlined in this post will help you pick the right queue.
Look at Key Metrics
There are literally dozens of queue solutions on the market, each with varying use cases, types and subfeatures, which can make selecting the optimal messaging queue a mind-boggling endeavor. Since the advantages of using different queues for different applications depend on the use case, it is critical that you decide what your key metrics are and narrow those down. Example metrics include latency, max throughput, max message size, and fanout capacity.
As is the case with any application, if you aren’t measuring the success of your efforts, you could be missing out on a more cost-effective and easier-to-use queuing service. From performance and reliability to scalability and decoupling, understanding the key metrics for your application will help you most effectively balance the queue’s cost with the amount of effort your developers need to put in in order to develop, deploy and maintain applications.
Plan Ahead to Choose the Best Message Queue
Once you’ve determined your key metrics, it’s time to start planning ahead by modeling what you need to ensure that whatever queue service you pick will meet your predicted load plus some overhead. Figure out what your data load is going to look like — including transaction counts, message sizes, etc. — and model it to understand your throughput and other metrics. From there, it will be important to make sure you carefully evaluate all of the queuing services to make sure that they not only meet your basic criteria for transactions, message sizes and throughput but also that the cost model is going to work well for your organization.
By modeling your queue application and conducting a cost analysis upfront before you commit, you’re better able to do the following:
- effectively assess the risks and opportunities for each queue service
- proactively address any issues that might arise
- revise and update your model
- ultimately improve the performance of your application by choosing the optimal queue
Evaluate Client Libraries
Another best practice is to evaluate what your client libraries look like. If you’re writing in a common language like Python or JavaScript, there is more than likely a client library you can add to your development projects, therefore it’s no big deal. However, if you are coding in a little less common or newer language, it is important to ensure there is a client available that will integrate with your chosen queue service. If not, you should be prepared to write one.
You might find a stylish and sophisticated queuing solution, but if it doesn’t have the right client for what your application requires, it will not interact with the API.
Push vs. Pull Model
Going hand in hand with that client is the push-pull model. Some queues are set up in such a manner that developers actually just request a message (the pull model), which works well in some cases and is convenient if you have specific security needs and want to allow traffic out but not in, for example.
The other option (the push model) is to subscribe, and the message queue — when it’s ready — will push elements and make calls to you. This choice is useful because developers do not have to poll the queue. Making more efficient use of network and system resources. However, this option requires opening the firewall to create the connection, which of course opens the door to other hazards and challenges.
Things to Consider When Choosing the Best Message Queue
Message Queue Security
Speaking of network security, a critical procedure in selecting the best message queue is to think about your security model and how you want to access the queue and have data moved around. You might also give thought to whether you need encryption in your queue. If you’re building a medical device that requires HIPAA compliance, for example, you are required to encrypt data both in transit and at rest, which means your message queue ideally supports encryption or you need to be prepared to implement it.
One-Subscriber or Pub/Sub Fan-Out Model
Most message queues are designed to have one publisher and one (or more) subscriber(s), so if you are developing on a microservices architecture, you only need a one-subscriber model because the first microservice instance available is going to pick up the message and process it. You don’t care who picks it up, just that it gets done. In the case of a broadcast event or a configuration change that multiple services need to know about, you might want to choose a queue that’s more of a Pub/Sub fan-out model, where you can publish once and multiple services receive a copy.
Thus, considering your use cases around fan-out is another good thing to do upfront to ensure you make the right choice.
Message Queue Integrations
With a significant amount of direct queue integrations with its services, Amazon might be a good option if you are considering a direct managed integration versus rolling your own. Lambda is a great example here. A lot of Amazon services — SNS, SQS, even some CloudWatch events — are natively integrated with Lambda, so developers don’t have to worry about the headaches of setting up, managing or configuring the integration because Amazon takes care of it.
The Best Message Queue is Future-Proof
Perhaps most importantly, you need to contemplate what lies ahead. While not too many developers moonlight as fortune tellers, it is important that they have enough capacity planned for the long haul.
If you think you’re going to grow rapidly or incur high costs, for example, then you might opt for a managed cluster, or possibly even operate your own cluster. Managed services can save a lot of pain and agony in setting up and managing a cluster, however, this approach comes with an ongoing cost because you’re basically paying for a web service to be constantly running behind the scenes. That being said, if you’re going to be running a lot of data 24/7 or you anticipate doing that soon, you might want to consider taking on a little greater cost load upfront versus just integrating with something that meets your needs today.
At the end of the day, there are truly a dizzying array of message queue services, and there is no one-size-fits-all solution on the market.
To Select the Best Message Queue for Your Application, Adhere to these Best Practices:
- Look at your key metrics
- Model everything upfront
- Use the most important features you need as a filter to narrow down your choices
- Do some more cost modeling to genuinely identify the right choice for both now as well as the future
If you understand message queues and what they do and recognize the cost drivers associated with your queuing service, following the best practices discussed here to select the optimal queue for your application will be that much easier. Hopefully, this blog series demystifies some of the most significant complexities surrounding selecting a queue and simplifies the process for you. If not, feel free to connect with us — we’re happy to help you select the best message queue for your application!