When it comes to software testing, “ready to ship” is actually a very high bar.
While it’s easy enough to build proof of concepts and prototypes, releasing the perfect product in the real world remains challenging. The criteria I always tell our quality assurance team to meet is “are you ready to give this product to your mother?” What I mean is are you ready to give this product to someone who:
- doesn’t have the specialized tools to get at its inner workings?
- has no idea how the system is put together?
- doesn’t have the developer right next to them to help them debug and determine why it failed?
Contents
Software Testing Basics
Before we bring a product to market, we need to determine whether it will work in any environment, on any network, on anyone’s phone, in anybody’s house — and we must have a high degree of confidence that it will indeed meet those requirements. How do we answer those questions to determine whether a product is ready to give to mom? By running a bunch of tests!
In order to check whether the actual product matches expected requirements and is defect-free, we need to run the same tests many times. So, we have to write tests, we have to be consistent in the execution of those tests so that we can run them again and again, we have to be flexible enough to deal with any changes in the requirements — wherever they come from — and we have to document the tests that have been done.
Clearly, the process of evaluating and verifying that a software product or application does what it is supposed to do would be made much simpler with a software testing tool that allows QA engineers to write the test cases, track the execution of the test cases and different configurations for the test environment, and be able to track trends regarding what has been successful or failed over time. Plus, the best software QA testing techniques involve reporting issues back to developers or tracking them back to system requirements so that testers can say, “this test verifies that this specific requirement is done.”
Source: https://www.gurock.com/testrail/
TestRail — Our Software Testing Tool of Choice
Given the importance of tracking versions, tests, trends over time and the ability to know that test steps are well defined, a tool that manages all test planning activities would be incredibly helpful. Even though a myriad of test case management software tools exist, the one I like best — and the one we use at Cardinal Peak — is TestRail.
Why? Because TestRail does one thing — and does it well. While a lot of test case management tools try to do everything from version control and bug tracking to project planning and test case management, TestRail only focuses on test case management. The tool empowers Cardinal Peak’s QA testing team to efficiently manage test cases, plans, and runs in alignment with development sprints, execute tests and track the results that matter most — all in the same place.
The tool also has a rich API for being able to log the results programmatically. Of course, some tests are run manually by our quality assurance engineers, but we write automated tests for others. When those automated tests run, TestRail lets us record all of that information through its API. Plus, because the interface for automation is just a web API, we can tie our automated QA testing tools into TestRail, whether we’re working on an embedded or cloud project or a mobile application. Since TestRail doesn’t assume anything about the execution environment, it is versatile as well.
In addition to having a UI that is laid out in such a way that fits my mental model of how we want to test, TestRail is customizable. We can add custom fields to track specific features, and we can even create our own custom templates for both test runs and test plans.
Another important piece of the puzzle is reporting and communicating back to our customers. At the end of the day, clients want to know if their project is ready to ship. In order for our team to answer that question, we need insight into how many tests are passing/failing, the known failures (if there are any), and we must be able to demonstrate, with a high degree of confidence, that we have coverage of the project’s unique requirements and the product has passed all tests.
Finally, TestRail integrates well with our bug-tracking system so that we can report issues back to developers or have traceability back to the system’s requirements, linking tests back to found bugs. This integration also allows us to track the version of the software that was executed and to assign sets of tests to all the various testers on our team so that they can execute tests in parallel.
Check out the TestRail case study about Cardinal Peak for more information!
Test Planning in the Real World
If you’re curious how expert QA testers actually approach planning their testing in real environments, check out the video below of the May 12th panel discussion with other seasoned QA testing professionals, sharing advice and help you improve your test planning processes.
Of course, I was excited to discuss how we manage and develop test plans, test planning best practices and my real-world test planning experience. I also talked about testing at Cardinal Peak and how we support the product development and engineering our talented team does.
Hosted by TestRail, Test Planning in 2021, Part 3: Test Planning in the Real World highlights what “test planning” really means in practice, common test planning mistakes and what to learn from them, and how you can start improving your test planning process today.