In follow up to Joachim Hofer’s series and Matt’s post on Jenkins and Jubula, I thought I’d write a short guide with some tips on using Jubula to test multiple applications, or multiple instances of the same application, during one test run.

The use case for doing this is more often than not some kind of shared data. Application A enters the data and application B should receive it. If someone is working on the data in application A, then a user in B should only have read privileges etc.

Working with multiple applications for tests in Jubula requires the use of two constructs: Test Jobs and the autrun command.

 

Test Jobs – an overview

A Test Job is actually nothing more than a collection of Test Suites to be executed after each other. You can create a Test Job via the context-menu in the Test Suite Browser.

Once the Test Job has been created, you can open its editor via double-click and add Test Suites to it via Drag&Drop. In the Properties View for each referenced Test Suite, you’ll have to select an AUT ID. This is the unique identifier used to be able to find the correct application at runtime. Every AUT you start has (or receives) an ID – more on IDs later.

What types of applications?

The first point to look at is whether the application(s) to be tested use the same toolkit (GUI library). If so (i.e. both are RCP applications), then there are no limitations on the interactions you’ll be able to perform in either application.

If the applications use different toolkits (i.e. one application is Swing and the other is RCP), then you will have to set the toolkit for the Project to concrete.

This will allow you to start and test both applications but will mean that the RCP-specific actions usually offered in the modules for RCP and SWT are not available. You’ll not be able to perform actions such as working with a GEF canvas or tree tables, for example. The “unbound_modules_concrete” actions (which are valid for Swing and RCP, and contain most of the necessary actions to work with applications) are available for writing tests.

Getting the applications started

Before we get to the various ways of starting applications for testing, a tip in advance: If you will be starting the same application twice, then use two AUT Configurations (they will share an Object Map so you don’t have to repeat the mapping). If the applications you are testing are different, then create a new AUT Definition for the second.

Once that’s cleared up, you can choose one of the ways below to make your applications available for testing:

Testing interactively

If you plan on starting the test manually (i.e. by pressing play in the Integrated Test Environment), then you can write AUT configurations for each application to be started, and start each application individually via the Start AUT button on the toolbar. The AUT IDs as defined in each AUT configuration can be selected for the individual Test Suites in the Test Job. As all the applications are started manually before the test, you don’t need to worry about starting any other applications via autrun. Having said that, this approach is usually only interesting when you are trying out your tests. Normally, it’s preferable for tests to be started via a CI tool (in our case, one that starts our Test Executor with the necessary parameters).

Testing in unattended builds via autrun

If you want to use the Test Executor to run your Test Job, then you’ll need to have some way of making sure that your applications get started. Unlike for Test Suites, the Test Executor cannot (currently) start an AUT from a configuration. The way to start an AUT without requiring an AUT Configuration is to use the autrun command. This is available from Java 5 for applications that can be started by an executable file.

autrun is a small program that can be found in the autrun folder in the installed (standalone) version of Jubula. Navigate to this folder in a command line client and enter the following:

autrun.exe
–a <the hostname of the agent you want to connect to>
-p <the port number for the agent>
-w <the working directory for the AUT>
-i <the ID that that AUT should receive when it is started>
-<TOOLKIT>
-e <the executable file>


The options available for the toolkit parameter are –rcp, -swing and –swt (HTML applications can’t be started using the autrun option). For RCP applications (i.e. Eclipse), you will also need the parameter –k .

Bear in mind that AUT IDs are unique throughout the Project. Start your AUT via autrun with a different AUT ID than those already used in the Project.

An example autrun command for an Eclipse application could look like this:

autrun.exe –a localhost -p 60000 -w C:\Programs\Eclipse -i Eclipse_autrun -rcp –k en_US -e Eclipse.exe

If you are currently connected to the Agent on the port number specified, then you should see an entry in the Running AUTs View in Jubula when the AUT starts. If the AUT is not already declared in the project, then it will be displayed as unknown:

For any AUT started by autrun, you need to enter an AUT Definition (not a Configuration) in the Project Properties. There is a specific section for AUTs started by autrun and you can open the dialog by selecting the (as yet unknown) AUT and selecting “Create AUT Definition” from the context-menu.

Fill in the necessary details and click OK. Now this AUT ID is known and can be used for this Project. You can select the AUT ID in the Test Job Editor for the necessary Test Suites.

In terms of your test, you can now do the following:

1. For the first AUT, make sure that it is started from your CI tool as a part of the test setup. This is a prerequisite for the Test Job.
2. For any other AUTs in the test, you have a choice:
   • You can call autrun from your CI tool for each AUT required
   • Or you can start a small script from within the test itself (via Execute External Command).

The former option has the benefit that the AUT gets started even if specific parts of the test don’t run, but does mean an overhead in the CI scripts. (You will, however, have to start the first AUT from the CI script, as the Test Job cannot run otherwise). Starting an AUT from within a test is usually lower overhead, but it does introduce a dependency between separate Test Suites in a Test Job. However, the very idea of a Test Job is to link dependent Test Suites, so the dependencies exist anyway.

Applications that start other applications

Some use cases have a different set up entirely. The AUT itself causes another AUT to be started. Jubula can deal with this situation based on the following assumptions:

1. The AUT Agent has been started in Lenient Mode. This allows AUTs that are started by other AUTs to connect to the Agent. You can switch to lenient mode for the AUT Agent via the context-menu on the system tray icon, or you can start the Agent with the parameter –l
2. The AUT that is started has the same toolkit as the AUT it was started from (e.g. RCP)
3. If multiple AUTs are started in this way, their order for the test is known.

If these three assumptions hold true, then you can define and use AUT IDs for applications started in this way by declaring the newly started AUTs as ID+1 ID+2 etc.

Switching between applications

Once you’ve:

• made sure your applications will all be started
• declared the AUT IDs they will use in the Project
• created a Test Job containing the necessary Test Suites
• selected the correct AUT ID for each Test Suite (this is even necessary when you are testing different instances of the same AUT)

… then you can start your test.

What you will notice fairly quickly, however, is that you will most likely need some way of making sure the correct application is in the foreground. Jubula cannot force this, as there are no clear-cut platform- and window-manager-independent commands to do so. It is up to your genius as a tester to make a module that will always bring the correct application to the foreground.

As luck would have it, we’ve worked with this construct a few times and have found the following module to be useful:

The Test Case performs a click in the active window – or tries to. If the window that receives the click is not the expected application (based on the AUT ID), then an “Action Error” is thrown. The Event Handler reacts to an Action Error and performs “Alt+Shift+Tab” using “External Key Combination”. This brings the next application to the front. The Event Handler is specified with the Reentry Type “Retry”, meaning that the failed Test Step (the click) is retried once the Event Handler has been executed. If the application that is now in front is the correct one (i.e. has the right ID), then the Test Step “click in active window” is marked as successful. The “Number of Retries” lets you define how often to do this loop. Once the AUT you require is in front, then the test is marked as successful and your actual tests can begin.

If you’re working on Mac systems, then it is advisable to alter your settings to make Alt+Shift+Tab switch between your applications in order (Alt+Tab alone switches between two applications, and Mac systems use another key combination by default to tab through a list of applications).

This little module lets you have as many windows and AUTs running as you like – you’ll always get to the right one eventually (just make sure the retries are set high enough)!

Test results and variables

Between each Test Suite, the test results for that Suite are written to the database (no accumulation of results for the whole Job as yet). However, if you want to work with variables between Test Suites, that’s no problem. A value stored as a variable in one Test Suite is available for the rest of the Test Job.

Testing multiple applications is one of the more complex use cases to automate. Hopefully these steps will make it easier to get started. At some point I’d like to see support for starting AUTs from configurations for a Test Job and a cumulative test result for the whole Test Job, but the current system already let you start your AUTs and then run and analyze your multi-AUT tests to get those tricky use cases automated.