Jira as a perfect testing tool

Testing in Jira is getting popular among the most successful companies. There are many reasons for choosing this test management solution. The first is that the Atlassian’s suite is usually well-known by all team members, even by the newcomers and beginners. The grand majority of people working in IT have a chance to at least see how Jira works at some point, even if it comes only to project management. Secondly, Jira was dedicated to tracking bugs, so it gives multiple possibilities for keeping on track with the progress, relations, and creating detailed documentation. On the Marketplace, there many Jira extensions that allow not only requirements test management but also the whole range of different actions. This is why it seems to be a perfect solution for the whole team – everyone can get what’s necessary to execute complete software development and release the fully functional product in the end. Additionally, Jira provides its users with real-time reporting, which supports communication, saves much time, and helps with decision making. Testbytes, Testing and QA Consulting Company presents the reasons on their infographic below.

Requirements and Test Management for Jira vs Xray

RTM for Jira is almost 2-years old, so it’s a relatively young but fast-growing application. Xray, on the other hand, is one of the oldest leading testing plugins available on the Marketplace. Both claim to be a complete solution for test management for Jira, available for Cloud, Server, and Data Center. They guarantee consistent testing, where the whole team can work together on the project on requirements, tests, reports, and bugs. Thanks to this, the scattered process will no longer be an issue, even when it comes to complex projects and large teams. The extensions support each stage of the process in the manual as well as automated testing. The apps make it possible to work with analysts, managers, and developers from the very beginning. Only this level of organization results in releasing the greatest software products in the end.

Sounds promising, but is it really so? Do all the functionalities work equally well on both extensions? We would like to analyze this in the present article. We chose four of the most often asked-for features and observed how they really look like in action. Read on to get to know the results.

User-friendly interface and intuitiveness of the app

When we log into an application for the first time, we expect intuitiveness that allows going through each step without the need of checking the documentation every other minute. Of course, the best apps provide their users with instruction but nevertheless, the majority of the team members would rather start working right away than read for a few hours before even turning the extension on.

RTM for Jira was designed to be as familiar for Jira users as possible. It has a simple, yet eye-friendly interface, where the consecutive steps are listed on the left side of the screen. The user can go from Requirements to Defects and through all the reports without the necessity to integrate any other tools. The look and feel of the extension make it possible to start working without any additional training. The user can see the relations between the objects by clicking on the dedicated Relations tab visible from the detailed view of each testing element. Moreover, testing objects in RTM, such as for example Requirements or Test Cases are standard Jira issues that allow to connect them with tasks, Epics, and Stories.

Xray also provides its users with seamless Jira integration and testing objects saved as native Jira issues. The interface of the app is transparent, which enables the development and testing teams to work together using just one tool. Just as RTM, Xray matches Jira look, so it is all the user can look for: clear and intuitive. The app grows fast, but still, everything is stored in one place, so there is no need to look for the data in a repository. The significant difference between the two apps is that in XRay requirements aren’t included in the process. It’s possible to integrate them using a separate tool, but the default settings of the extension don’t provide the requirements management features. What’s more, after clicking on elements from the issue list, Xray redirects users to Jira issue view and closes the internal list at the same time. In RTM selecting a testing object just opens the new view inside the app.

Traceability of the testing objects

When it comes to traceability, we all know that it’s crucial in successful test management. This is why the creators of both testing extensions came up with the features that optimize tracking relations between the testing objects. Xray provides its users with a dedicated Traceability Report. It promises its users the possibility of tracking all connections from Requirements to Defects, in both forwards and backward directions. It has a form of a colorful, table, easy to understand in just a second. If the requirements tool is integrated with the app, it allows seeing how Defects are related to Requirements, analyze the statuses of Test and Test Runs, and how they affect Test Execution. The report is customizable with Basic and Advanced filters, which lets the users get the exact data without the unnecessary information that could be confusing.

As it was mentioned before, RTM for Jira has a dedicated Relations tab for all elements, which presents their connections. Thanks to this, just by clicking on a particular Requirement, a Test Case, or another testing object, the user can see in which Test Plans or Execution are they included, what Defects they have caused, and much more. Requirements and Test Management for Jira app has also the Traceability Matrix and Requirement Coverage reports. Both are also flexible as XRay’s report, as the user can choose proper data using filters. The first report makes it possible to see the relations between two base-lined issue types at a glance, and the second shows all the connections of the chosen Requirements on a transparent table. With these features, it’s really hard to omit something, so the team can be sure that all initially planned functionalities are safely covered by the necessary further steps of testing.

Support for manual and automated tests

As both apps make it possible to store requirements, tests, test plans, test executions, and defects in a single place, they surely support consistent manual testing. Requirements and Test Management for Jira additionally offers the tree-structured view for all objects, which makes the work much easier to organize. Folders and subfolders enable grouping or categorizing issues as the team sees fit. Sharing results and dividing tasks become logical and simple. When it comes to automated tests, RTM for Jira gives its users REST API endpoints made for dealing with tests and importing execution results from automated processes. For now, it’s available for Test Cases, Test Plans, Test Executions, and Test Case Executions. The team is now working on the enhancement of REST API for Requirements, Defects, and Tree structure. You can track the progress on the RTM’s roadmap. It’s also worth remembering that as Requirements and Test Management for Jira app is based on working with native Jira issues, it’s possible to use Jira’s native REST API to update some fields or add attachments/comments.

Xray app provides great support for automated tests. In addition to what RTM includes, so the built-in and Jira’s native REST API for the user to benefit from, the extension also enables multiple integrations with the other tools. The strong point of the extension is that manual and automated tests are managed the same way, which makes it possible to report automated results from the CI environment in case of requirement coverage evaluation. There are several CI tools that the user can connect with Xray, like e.g. the most popular Bamboo and Jenkins. XRay offers two types of tests that may be used for automated testing: Cucumber for tests in the specified language and Generic, for all other automated tests.

Transparent reporting

Requirements and Test Management for Jira gives its users two reports which enable them to track the progress of testing: Test Execution and Test Case Execution reports. The reports are filtrable and customizable. They show the status of each Test Execution, as well as the included Test Cases’ percentage of completion. TE and TCE reports display which objects are Done, To do, In progress, or Fail, in order to spot the defects and fix them right away. Moreover, they make it possible to estimate the time left to complete each activity and the overall process. The strong point of both is their graphic form, which is easy to read by all team members and stakeholders, no matter their experience or knowledge level. As the reports are automatically generated, creating them is quick and intuitive, which is why it can be done anytime the manager asks for the current work status.

Xray provides on the other hand provides its users with Coverage Analysis. The report has a similar objective as RTM’s Test Case Execution described above. The functionality enables to analyze testing issues from different perspectives or dimensions, such as the Latest, Version, Test Plan, Test Environment, or Final Statuses. The statuses are shown in a horizontal bar chart, divided by colors. It’s possible to visually group the results (for example by component), filter them, choose the model of analysis, and the way of data presentation (flat or hierarchical). The report allows analyzing the current status of release or past releases, as well as requirements coverage in a given version. The chart is also supported by a drill-down. When the proper section is selected, the table appears and it shows the total number of tests that are linked, Passed, Failed, or marked with the status Not Run/Unknown (Other).

Be the jury and pick your winner

There is no doubt that both RTM for Jira and Xray are two great apps for testing inside Jira. Implementing one of them to the process will be beneficial for the whole team and help achieve the goal of each software company: releasing many bug-free, fully functional products, which sweep the competition every time. It seems that Requirements and Test Management for Jira has an advantage with built-in requirements and more friendly reports, but Xray beats it with more integrations with external tools, which supports test automation. The final choice as always belongs to the users. And in this case, it’s a difficult one, as the differences are really hard to spot. In order to make it a bit easier, we’ve prepared a comparison table for you, and also, we recommend digging into the documentation.

• Requirements and Test Management for Jira
• XRay