KWWidgets/GUI Testing/Squish

Squish

• Squish Home
• Squish for Tk, only for Unix
• about €2500
• Win32/Qt/Java/Web/Tk, scripting in Tcl/Python/JavaScript/Perl, recorder, window inspector, breakpoints/verification

Prerequisites

• Qt3. Squish for Tk will not work with Qt4. Make sure you install all Qt3 development packages. On a Ubuntu OS, this (most likely) involve: libqt3-headers, libqt3-mt, libqt3-mt-dev, qt3-apps-dev, qt3-assistant, qt3-designer, qt3-dev-tools, qt3-linguist, qt3-qtconfig, etc.
• Tcl/Tk 8.4 or 8.5
• KWWidgets

Download

• Download a Squish for Tk evaluation license

-rwxr--r-- 1 barre barre 6587889 2008-09-04 13:08 squish-3.4.1-eval-tk-src.tar.gz*
-rwxr--r-- 1 barre barre      19 2008-09-04 13:08 squish-3-license*

Installation

• Reference: Squish Installation
• Unpack the archive

barre@tetsuo:~/build$ tar -xzf /home/barre/tmp/squish/squish-3.4.1-eval-tk-src.tar.gz 

• Configure Squish

barre@tetsuo:~/build$ cd squish-3.4.1-eval-tk-src
barre@tetsuo:~/build/squish-3.4.1-eval-tk-src$ ./configure "--with-qtdir=/usr/share/qt3"
 "--with-tclconfig=/opt/tcltk8.5.2/lib/tclConfig.sh" "--with-tkconfig=/opt/tcltk8.5.2/lib/tkConfig.sh" 
"--with-pydir=/usr/share/python"

Note that the configure script may not find your Qt3 distribution correctly: use --with-qtdir to point to the appropriate directory. An IMPORTANT note for building/running Squish is, you have to build Squish with the the same version of Tcl/Tk used by the AUT (Application Under Test) , because it doesn't work if Squish and its AUT(KWWidgets or Slicer) are built with different versions of Tcl/Tk. Use --with-tclconfig and --with-tkconfig to point to your own Tcl/Tk distribution, if any (here /opt/tcltk8.5.2). Also, you will need to install libxss-dev package if it is not installed.

• Enter your license key (found in squish-3-license)

Configuring Squish. For license information please read the LICENSE file.
By configuring and building Squish you are accepting the terms of this license.
Enter license key: ***-*****-*****-***
Checking for C++ compiler ......... g++
Checking for g++'s version ......... 4
[...]
Done with configuring. Full log in config.log. Now type './build'.
* Build
<pre>
barre@tetsuo:~/build/squish-3.4.1-eval-tk-src$ ./build

Testing KWCallbacksExample

In this section, we will exercise KWWidgets' KWCallbacksExample example.

• Build KWWidgets examples; set KWWidgets_BUILD_EXAMPLES to ON, make sure the ./bin/KWCallbacksExample example has been built and is working properly. It should display a simple window with a single slider inside of it. Fancy. When the application exits, make sure you select the "Do not show this dialog anymore" checkbutton in the confirmation dialog.

KWCallbacksExample

Create a new Test Suite

• If you are new to Squish you may want to read: Squish Concepts and Creating the First Test. The Squish for Tk documentation is located offline in your build directory at squish-3.4.1-eval-tk-src/doc/book/index.html, or online at http://www.froglogic.com/download/book/.
• Start the Squish IDE

barre@tetsuo:~/build/squish-3.4.1-eval-tk-src$ ./bin/squish

• Create a new test
  • Select "Create New Test Suite" at the Welcome page, or "File|New Test Suite..." in the main menu bar.
  • Enter a Suite Name (say kwcallbacksexample). Select where the test suite and its files shall be saved (say ~/tmp/kww_test/). Hit "Next".
  • Specify which toolkit the application is using. KWWidgets uses Tcl/Tk under the hood, therefore select the Tk radiobutton. Hit "Next". NOTE: this page will *only* show up if Squish was built to support *several* toolkits; if only one toolkit is available (hopefully, Tk), this page will not be displayed and the appropriate toolkit will be picked automatically.
  • Select Tcl as the scripting language. Hit "Next".
  • Select the Application Under Test (AUT): pick the the ./bin/KWCallbacksExample application found in your KWWidgets build tree. Note that even though we are using Squish for Tk, we are actually exercising a KWWidgets *binary application*, not a Tcl/Tk *script*. Hit "Finish".

Create New Test Suite

Pick Test Suite Name

Pick Scripting Language

Select AUT

Create a new Test Case

• Select "Create New Test Case", or "File|New Test Case..." in the main menu bar or right-click on suite_kwcallbacksexample in the left tree and select "New Test Case...". Name it, say, Test1. This should create a tst_Test1 case in the left tree and a test.tcl file in your test suite directory. Note that a test suite can store multiple test cases at a time (which would be executed one after the other).

New Test Case

Record the Test Case

• Right-click on tst_Test1 and select "Record...", or select "Test Suite|Record..." in the main menu bar. Accept the defaults and click "Record". This should launch KWCallbacksExample. Note the "Squish Control Bar" window displaying "Recording test case 'tst_Test1'".
• Move the slider in the KWCallbacksExample window from 10.0 to a different value, say, 50.0. Note a new (and lengthy) entry in the "Squish Control Bar" window: "scrollTo KWCallbacksExample.vtkKWWindow0.vtkKWFrame3.[...].vtkKWMyWidget0.vtkKWScale0 50000".
• Exit "Recording" mode by closing or exiting the KWCallbacksExample window, or hitting the "End Recording" button in the "Squish Control Bar" window.
• Back to the Squish IDE, the script test,tcl has been updated to reflect your interaction with the AUT:
  1. waitForObject makes sure a widget is ready and has been mapped on-screen (here, our scale widget). The widget name itself is quite long as it reflects the hierarchy of parent-child widgets in a typical KWWidgets application. The name of each child in this chain is created at run-time by concatenating the name of the C++ class for that widget to a unique ID among its siblings. The last child in this specific example should be vtkKWScale0, which is the de-facto KWWidgets C++ class used to wrap the Tcl/Tk scale widget we just interacted with.
  2. invoke is a Squish command that will act on a widget. Its first parameter, scrollTo , will scroll the scale slider to its new value, 50.0 (displayed as 50000).
  3. snooze 12.5; snooze statements force the script to wait for N seconds. These calls are generated to ensure that the script always runs with the same time-delays as when it was recorded. They are added because in some cases an action may take some time to complete and the application may not be ready for the following action in time.
  4. invoke closeWindow is another Squish command that will close a specific window. It is not clear at this point what Squish is closing; one would have expected :KWCallbacksExample.vtkKWWindow0, but this might be the first widget that is being deleted in the whole application instead. In any cases, this line can be ignored and deleted; the application will still exit properly. Let's shorten the snooze command to 1 second, since it is now the last statement (or remove it altogether).

Record Test Case

Test Case Recorded

Execute the Test Case

• Right-click on tst_Test1 and select "Execute", or hit F5, or select "Test Suite|Execute" in the main menu bar. This should run the whole test, move the slider to 50.0 automatically, and bring you back to the Squish IDE. Note the contents of the "Test Log" tab in the "Output" panel at the bottom:

START  Fri Nov 7 17:11:24 2008 Start 'tst_Test1' (Passes: 0, Fails: 0, Fatals: 0, Errors: 0, Warnings: 0)
   END    Fri Nov 7 17:11:25 2008 End 'tst_Test1'
    End of test 'tst_Test1'	

Execute Test Case

Insert a Verification Point

Verification points (VP) can (and should) be added to make sure the application behaved correctly, i.e. that once executed by Squish, we end up in the state we expected. There are multiple ways to add a VP; in this case, we will insert a breakpoint where the verification should take place, execute the testing script again until it "breaks", and insert the VP itself at this point; we will then inspect (spy on) the AUT's widgets to choose the widget we want to check for validity (in our case, the scale widget).

• In the test.tcl window, insert a breakpoint on the line right *after* the invoke scrollTo command line, say on the snooze command line, for example (you can manually type such statement if you removed it previously, just like if you were editing Tcl code). Click in the gray column immediately to the left of the line, where its number is displayed, to insert a breakpoint.
• Execute the test case. Squish should execute the test, then halt it at the breakpoint; the line in which the debugger breaks is highlighted in yellow.
• Start the Squish Spy on the currently running but halted AUT. This is achieved by choosing "Spy|Spy AUT" in the menu bar.
  • Below the editor two new views appear. To the left, it shows the object hierarchy and to the right the property tab. When the property tab is active, it shows the properties of the current object. Each object and property has a check box. By checking a property, a verification point for this property is created.
  • Rename the Verification Point (the entry above the object hierarchy), or leave it to VP1.
  • Select the vtkKWScale0 widget in the object hierarchy:
    • by opening the hierarchy tree until you reach vtkKWScale0 (check the test.tcl script to find the proper parent-child path), or
    • by selecting the scale widget *directly* in the AUT: invoke "Spy|Pick Object" in the menu bar, wait for Squish to switch to the AUT GUI then click directly on the scale widget.
  • Select the getvalue property checkbutton in the property tab: it should reflect the *current* value of the scale widget (here, 50.0). This is the property we want to check for the test to pass.
  • Insert the Verification Point in the test.tcl script by clicking on the checkmark icon in the toolbar next to the VP name. Note that the tree in the "Test Suite" panel on the left should reflect the new VP as well.
• Stop the script by choosing "Test Suite|Stop" in the menu bar. You should now see a test vp command in the test.tcl window. This is the specific command that will make sure the conditions created for this verification point are met and that the test is passed.
• Remove the breakpoint (by clicking on it). You may also remove the snooze statement, or move it before test vp.

Insert Breakpoint

Stopped At Breakpoint

Choose Verification Point Property

Verification Point Inserted

Validate the Test Case

• Right-click on tst_Test1 and select "Execute", or hit F5, or select "Test Suite|Execute" in the main menu bar. This should run the whole test, move the slider to 50.0 automatically, check the Verification Points and bring you back to the Squish IDE.
• After the test run completed, you see the result of the verification point in the test log view at the bottom of the Squish IDE. This is a detailed report of the test run and would also contain occurring failures, errors, etc. If you click on a test log item, the Squish IDE highlights the script line which generated the test result. The Verification Point passed, yes!

START Fri Nov 7 17:33:16 2008 Start 'tst_Test1' (Passes: 1, Fails: 0, Fatals: 0, Errors: 0, Warnings: 0)
  PASS Fri Nov 7 17:33:17 2008 /home/barre/tmp/kww_test/suite_kwcallbacksexample/tst_Test1/test.tcl:4: VP1:
    Object property comparison of :KWCallbacksExample.vtkKWWindow0.[...]vtkKWScale0.getvalue' passed
    '50.0' and '50.0' are equal (VP1)	
  END Fri Nov 7 17:33:18 2008 End 'tst_Test1'               
  End of test 'tst_Test1'	

• By implementing custom report generators it is possible to process test results in many different ways, for example to store them in a data base, or to output them as HTML files. The default report generator simply prints the results to stdout, or when being used in the Squish IDE they are displayed in the test log view. You can also save the test results in the Squish IDE as XML by right clicking on the test results and choosing Save Results As.

Verification Point Passed

Run the Test Case on the command-line

A whole test suite or individual test cases can be run from the command-line, thus allowing them to be part of nightly regression dashboards.

• Make sure the Squish Server is running:

barre@tetsuo:~/build/squish-3.4.1-eval-tk-src$ bin/squishserver &

• Pass the test suite to the Squish Runner executable:

barre@tetsuo:~/build/squish-3.4.1-eval-tk-src$ bin/squishrunner --testsuite ~/tmp/kww_test/suite_kwcallbacksexample/
2008-11-07T17:42:40 START Start 'suite_kwcallbacksexample' Test 'suite_kwcallbacksexample' started
Debug: Client thread created, sock: 8
Debug: Connection established from 127.0.0.1
RUNNERID: 247170001
AUTID: 247170002
AUTHOST: 127.0.0.1
AUTPORT: 4322
RUNNERID: 247170001
Debug: Client thread created, sock: 9
Debug: Connection established from 127.0.0.1
application registered  247170002
Initializing AUT context ...
2008-11-07T17:42:41	START_TEST_CASE	Start 'tst_Test1' Test Case 'tst_Test1' started
... done.
2008-11-07T17:42:42	PASS	/home/barre/tmp/kww_test/suite_kwcallbacksexample/tst_Test1/test.tcl:4:
VP1: Object property comparison of ':KWCallbacksExample.vtkKWWindow0.[...].vtkKWScale0.getvalue' passed
   '50.0' and '50.0' are equal (VP1)
2008-11-07T17:42:43	END_TEST_CASE	End 'tst_Test1'               	End of test case 'tst_Test1'
2008-11-07T17:42:44	END	End 'suite_kwcallbacksexample'	End of test 'suite_kwcallbacksexample'
*******************************************************
Summary:
Number of Test Cases:	   1
Number of Tests:	   1
Number of Errors:	   0
Number of Fatals:	   0
Number of Fails:	   0
Number of Passes:	   1
Number of Expected Fails:	   0
Number of Unexpected Passes:	   0
Number of Warnings:	   0
*******************************************************

• If VTK_DEBUG_LEAKS was turned ON, you may notice some debug leaks. This seems to happen when the AUT is terminated by closing the window directly (wrong signal maybe?). The test will still pass though. One workaround is to use select "File|Exit" in the AUT's menu bar, which will exercise the menu bar at the same time!