Jenkins integration
Last updated
Last updated
For the interaction between aqua and Jenkins, the famous open source continuous integration tool, we have developed a plugin that can be installed in Jenkins and that allows you to trigger Jenkins jobs from aqua and vice versa, meaning that you can trigger execution of the test scenarios in aqua from a Jenkins job.
Unit test results can also be written back to aqua to take advantage of aqua's powerful reporting feature. In addition, aqua links can be parsed from a repository's comments (e.g. Subversion or Git) and displayed in Jenkins. This chapter details the steps required to set up and configure integration with Jenkins.
Before starting the installation of the Jenkins plugin, please ensure that you have the right Jenkins plugin corresponding to your aqua installation version number and your Jenkins installation version number. If you are not sure, please contact our support team.
Be sure that you have created the dedicated automation agent and the user with the 'Jenkins' licese type to ensure that the integration will be successul for the given project.
The Jenkins plugin is not available from the Jenkins plugin store. The Jenkins plugin file is delivered as "aquaJenkinsPlugin.hpi" and can be downloaded from the 'Installations files' page.
To install this plugin, open the Jenkins dashboard and select Manage Jenkins on the left-hand side. Then select Manage plugins in the middle.
Now you can upload the plugin file from your computer, or paste the link to download the Jenkins plugin into the URL field.
Please note that to upload it via the URL, you need to be logged in to aqua.
To configure the plugin, please go to the Manage Jenkins and select the Configure System section.
Scroll down until the section called Aqua Plugin.
As you can see in the screenshot, the plugin requires the following information:
Agent code
An agent code that identifies the plugin in your aqua installation. Retrieve this code by opening aqua. Right-Click the corresponding project and choose “Project Configuration” => “Automation”. On the appearing window add a new agent, call it Jenkins (or similar) and save your settings.
Agent Webservice URL
The Webservice URL for aqua Agents. Retrieve this information from your administrator. You can check this URL by opening the URL in your browser. You should see a website entitled “aquaAgentWebservice”.
Agent active
This activates your Jenkins Plugin.
Main Webservice URL
The common Webservice URL for retrieving test cases. Retrieve this information from your administrator. You can check this URL by opening the URL in your browser. You should see a website entitled “MainService”. This is only used for the direction Jenkins -> aqua
aqua user name
A valid aqua user name that is used for authentication. This is only used for the direction Jenkins -> aqua
Password
The corresponding password that is used for authentication. This is only used for the direction Jenkins -> aqua
Name of your aqua instance
When you have completed your settings, please save them by clicking Save in Jenkins. You can check if your settings have been successful by opening your aqua Client.
To trigger the Jenkins Jobs from aqua, open your test in Jenkins and select 'Configure' tab
Then the source code repository URL needs to be provided.
If you would like to run the maven test, please be sure the proper configuration was adjusted
To publish the JUnit test results, first make sure that the test results are displayed in Jenkins and fulfill the test report XMLs field.
Unit test results are shown in Jenkins. However, to get access to this information in aqua, you have the option to write back the results into a template test case in aqua. The first step is to create this test case in aqua. Open a new test case, give it a name, e.g. “Unit test results”, then save and close the test case. Then go to your Jenkins Job, add a new Postbuild Action called aqua Notifier” and enter the Id of your aqua test case.
When executing this Jenkins job now, your Unit test results are written to your aqua test case and stored as a new execution in this test case. Please be aware that in addition to the creation of a new test execution, the test description and all test steps are replaced with the information from Jenkins, every time the Jenkins job is executed. Therefore, any information stored in the test case description or test steps will be replaced.
Do not save the Unit test results into the same test case that triggers a Jenkins job!
You can also use a variable for the Test Case Id (e.g. $TestCaseVariable) if you have a parameterized Jenkins job and want to store the results in different test cases in aqua. For this purpose, all test cases from the variable values must exist in aqua.
Then open the corresponding test case in aqua. Check whether, under the test step -> 'Automation' tab, the appropriate job ID from Jenkins was inputted.
Now you can execute the given Test Case in aqua with the usage of the Jenkins agent.
A Jenkins job can be parameterised. If you want to override a specific parameter, e.g. Environment, you can parameterise your aqua test case (see the Data tab in the left-hand menu). When the given Jenkins job is triggered via aqua, the given value will be set correctly in Jenkins.
Once the execution is done you can check in aqua the information collected during it including the logs.
aqua test scenarios can be triggered from Jenkins. This is a convenient way to run your automated tests in aqua after a successful build in Jenkins. To add this trigger, add a build step called aqua test scenario Trigger. This will add the following form to your Jenkins job.
If no Agent is specified in the aqua Test Scenario, then it will be taken from the Agent Name value in Jenkins. The output path creates the report in Jenkins, and then the results can be published in Jenkins - useful if you want to integrate aqua with Jenkins and then trigger some Ranorex jobs.
When you run tests using Jenkins, aqua receives general information and separate execution logs for each test, offering a detailed breakdown of what happened during the execution.
The specific timestamp when an event takes place, such as the start or completion of a build, the triggering of a job, or the execution of a particular step within a pipeline.
Types have been added to improve the log classification.
JENKINS_GENERAL_RESULT: This type logs the start of the execution and provides an overview of all tests, including their names and status (e.g., FAILED or PASSED).
JENKINS_TEST_RESULT: This type contains individual execution logs for each test, allowing for a more granular review of test results.
Type
The "Type" column helps users quickly identify whether a build step succeeded, failed, or simply provided informational output.
Message
The test results and the name of the test case are displayed.
Detailed message
In Jenkins, the following information is displayed in the XML reports under specific tags:
Standard Output (<system-out>)
This includes regular logs and outputs generated during the test run, such as informational messages and status updates.
Standard Error (<system-err>)
This section shows error logs and diagnostic messages, helping you quickly identify any issues that occurred during execution.
Error Message (<failure> or <error>)
These tags capture failure messages, typically from assertions or exceptions. They provide details about why the test failed.
Stack Trace (<failure> or <error>)
This tag includes a detailed trace of the code failure, showing exactly where the error occurred, making it easier to diagnose and fix issues.
In aqua you will the see the following information
For passed tests
If there are no details: Message: "Standard Output: No output." is displayed.
If additional details are available, the following may be shown:
Standard Output (<system-out>): Displays the regular output. If empty, the message will read, "Standard Output: No output."
Standard Error (<system-err>): Displays error-related information, if any. If there’s no error output, it won’t be shown.
For failed tests
The following information may be displayed:
Standard Output (<system-out>): Shows the output generated during the execution. If empty, the message will read, "Standard Output: No output."
Standard Error (<system-err>): Provides diagnostic messages, if any. It won’t be shown if there’s no error output.
Error Message (<failure> or <error>): Displays specific failure messages from assertions or exceptions. If there’s no error message, it will not be displayed.
Stack Trace (<failure> or <error>): Contains the trace of where the error occurred in the code. If no stack trace is available, it won’t be displayed.
Large messages in Standard Output are truncated. If the full Standard Output is needed, it can be viewed in the reports under the Files tab.
Minimum Jenkins version required to run with the new aqua plugin upgraded to 2.462.3 (latest stable version, as referenced here).
The plugin can now only be built with JDK 17 or higher (It is recommended to use JDK 17).
Configuration changes to the plugin might require Jenkins restart
Item IDs can have at most 6 digits
aqua build steps for Jenkins do not support the Jenkins pipeline (please use the freestyle instead)
The name of your aqua instance can be found by opening your aqua Client. The footer shows the appropriate information, in our case “aquatest-oracle”.
