Tests and test management apps
What TreeCraft shows of your tests today, with tests kept as Jira work items, and what to expect from Xray, Zephyr Squad and Zephyr Scale.
On this page
What TreeCraft can show
TreeCraft shows Jira work items. A test appears as a row when the app that manages it keeps the test as a Jira work item. A result, such as passed or failed, appears only when it is also in Jira, for example as the work item's status.
Results that a test app keeps in its own storage are out of reach. TreeCraft is built to Atlassian's Runs on Atlassian requirements and sends nothing outside Atlassian, so it doesn't call another app's own service to fetch them.
The test results columns count how many tests cover each story, and how many passed, failed or have not run, from Jira's own data. They read a result from a test's Jira status. Reading a result that an app writes into one of its own fields or properties is not built yet.
Tests as plain Jira work items
This works today, with no test app. Tests are work items of a type you choose, such as Test, and their status says how they went, for example To Do, Passed and Failed.
- Link each test to the story or requirement it covers, with a Jira link type, or make tests sub-tasks of the story.
- Build a hierarchy. Add the stories with an Add work items from JQL rule, and bring the tests in under them with an Add linked work items rule (pick the link type and the direction) or an Add child work items rule for sub-tasks. See Rule types.
- Show the Status column to see each test's result. Colour it with conditional formatting so that failed tests stand out.
- Add the test results columns: Tests, Tests passed, Tests failed, Tests not run and Stories with passing tests. Someone with Control tells the hierarchy which work types are tests and stories, which links tie them, and which statuses mean passed or failed. See Test results settings.
The columns count only the tests each person can see in Jira. For other counts, formula columns work on the same rows. For example, COUNT#children{1} counts the rows directly below a story.
Xray, Zephyr Squad and Zephyr Scale
We haven't installed these apps on a site to check, so everything in this section is not yet verified on a live site. It describes how each app is known to store its data in Jira Cloud. We will update this page once we have checked.
| App (Cloud) | Tests as rows | Coverage: tests under the story they cover | Pass or fail results | Status |
|---|---|---|---|---|
| Xray | Expected: tests, test plans, test executions, test sets and preconditions are Jira work items. | Expected, through Jira links: the coverage columns should count them. | From the test's Jira status only. A result that Xray keeps in its own storage, or writes into a field or property, can't be read yet. | Not yet verified on a live site |
| Zephyr Squad | Expected: the Test work type is a Jira work item. | Expected, if your team links tests to stories in Jira. | From the test's Jira status only. Executions and test cycles live in Zephyr's own storage, so they can't be read. | Not yet verified on a live site |
| Zephyr Scale | Expected: no, test cases aren't Jira work items. | Expected: no, Zephyr keeps the traceability outside Jira. | Expected: no. | Not yet verified on a live site |
Where a test is a Jira work item, everything in Tests as plain Jira work items applies to it, including the coverage columns. The result is the exception: the status of a test in these apps may not say how it went, so the passed and failed columns may not match the app's own results.
Xray, Zephyr Squad and Zephyr Scale are trademarks of their owners. Plugincraft is not affiliated with or endorsed by them.
Other test apps
Any app that creates tests as Jira work items should behave the same way: its tests appear as rows, linked to stories through Jira links. Its coverage should count the same way. Its results can be read only from the test's Jira status for now. Reading a result that the app writes into a Jira field or a work item property is not built, and we haven't checked which apps write one.
Questions about the app? Write to info@plugincraft.ca.