Testingmachine is used to test eGovernment services. The eGovernment services can vary from country to country but generally it grants citizens access to important documents and information. In most EU countries there are additional services like paying property tax that can be done online. Most governments in Europe are leaning in this direction in order to decrease administrative overhead.
Another challenge to overcome will be testing eGov services on mobile platforms, more specifically automating tests under Android. Good news is this is possible using Selenium. The one major obstacle that we face is getting around the smart card login. So far we are unaware of getting this to work under Android unless you patch the kernel and this is of course not an option. More documentation and research is needed concerning this.
Now that we have explained what testingmachine is and what we are using it for we can begin explaining how you can become a developer.
Getting the code for the project is pretty straightforward.
You can either clone the repository using git with:
git clone https://github.com/tis-innovation-park/tm-vmm.git
In addition you can download the zip file from the online repository here:
If this is your first time using Git, here is a resource to help you on your way:
Building and installing
./configure --prefix <installationdir>
sudo make install
In this file you can configure settings you want to use as default in your clients. It is perfectly possible to override these settings in your individual client configurations. For a list of variables, see section Configuration syntax in the full manual of Testing Machine.
Using Virtual Machine Manager
We thought it might make sense to explain very briefly how to use the virtual machine manager, just to get your feet wet and understand a little what is going on before we get to the more interesting stuff:
Starting with guest operating system visible
tm-vmm --start-client <CLIENTNAME>
tm-vmm --start-client Ubuntu-14
Starting with guest operating system headless
tm-vmm --start-client-headless <CLIENTNAME>
tm-vmm --start-client-headless Ubuntu-14
tm-vmm --check-client-status <CLIENTNAME>
tm-vmm --check-client-status Ubuntu-14
Becoming a Virtual Machine Manager developer
The Testing Machine is currently made up by Virtual Machine Manager (tm-vmm) and documentation on how to write, execute and automate tests of eGov sites in particular but also other software.
Virtual Machine Manager (vmm)
tm-vmm is made up by bash scripts that let the user manage various virtual machine software in a general way. See the tm-vmm manual for more information.
Directory containing all programs visible to the user.
tm-vmm - main program for the management of virtual machines. This script is basically a parser for the functions as stored in the scripts in the scripts directory.
tm-vmm-auto - outdated script previously used to automate stuff
A directory containing scripts and other files that can be used while developing Testing Machine. Nothing of this stuff needs be included in the distribution.
This is the directory containing the manuals and other documentation. The manuals are written in Markdown format.
*.md - Source version of the manuals
generated-files - all files generated from the .md files are placed here
pics - pictures used in the manuals
faq-pics - pictures related to the FAQ
Various examples of how to use Testing Machine. We will remove this directory since we switched to using a separate repository (tm-examples) for this purpose.
This is where the functionality of Testing Machine is stored.
loggers - directory containing various scripts that implements different logging functionalities.
Adding bash code or fixing bugs
Add a new command line option
If you want to a new command line option to tm-vmm you should update the bin/tm-vmm.tmpl file. Make sure to add code to parse the command line and also as an entry in the help printout.
Update the documentation accordingly.
Add bash functions
If you want to fix bugs or add new features add this in the scripts directory. If you add a bash file, make sure to update the Makefile.tmpl accordingly.
All documentation is written in Markdown format. We use pandoc to generate html and pdf.
Each chapter/section is written in separate files to make them more easily re-usable. Since there's no *include* functionality in Markdown we had used the Makefile to cp/cat together the various manuals. Perhaps not the most optimal way of doing things - but it works. If you have improvement suggestion you're more than welcome.
If you add a markdown file, make sure to update the doc/Makefile.tmpl accordingly.
Building Testing Machine
The scripts and Makefiles in Testing Machine contains variables used to store the installation dir. This is needed to source all bash files.
The scripts and Makefiles are created, with the variables described above updated according to the installation dir, from templates.
Starting with eGov testing
Since you now know how to add functionality to testing machine we thought it would be beneficial to help those who wish to get started with testing web services. Our examples are based on the services that the Province of Bolzano use but it should be easy to understand and modify so you can apply the same thing to some web service you would like to test.
Getting the code for testing eGov services can also be done using Git. There are some example tests that can be run however we first need to install some packages on the host operating system. Use your package manager, here we use apt:
apt-get install openjdk-7-jre-headless ant
Now the environment is setup and we can clone the directory containing the examples
git clone https://github.com/tis-innovation-park/tm-examples.git
This is an extension of the simple Testing Machine example that can be found in the parent of this directory. It contains some Bolzano-specific tests, and is using Selenium.
The init method (in the TMText.java file), which is annotated with @BeforeClass, will set up the driver object for us, and it does so before the actual testing start. The actual tests are contained in the SeleniumTests.java file.
Finally, the driver is closed with a method annotated with @AfterClass.
Build and run the project simply by placing yourself in this directory (the directory containing build.xml) and type:
should be replaced with either AndroidDriver, FirefoxDriver, or ChromeDriver. When ChromeDriver is used, you must set the path to the driver file:
ant -Dtm.seleniumDriver=ChromeDriver -Dwebdriver.chrome.driver=<Path>
When AndroidDriver is used, you can set the optional tm.seleniumUrl parameter in this way:
ant -Dtm.seleniumDriver=AndroidDriver -Dtm.seleniumUrl=<URL>
The Ant command will compile the classes and run TestNG, which in turn will generate an HTML summary of the tests in a "testng_output" subdirectory. (If you want to change this directory, all you have to do is to set the -Dtm.testngOutput parameter. The results can now be copied to any machine using SSH.
The tests in the tm-examples/eGov/bolzano performs Selenium based tests that check various things such as links on a page, if a certain text is present, if the header of the page matches the website and so forth. The Selenium documentation has a lot more detail, automating Selenium is covered here but not how to write Selenium tests, that is out of scope
Starting with eGov testing on a virtual machine
Once the host environment has been setup and the guest virtual machine is up and running we can now start testing on the virtual machines that we setup in the previous tests. Lets fetch the necessary Selenium jar that will be needed.
Now we copy this over where it is needed, replace <CLIENTNAME> with the appropriate VBox name
tm-vmm --client-copy-file selenium-server-standalone-2.37.0.jar <CLIENTNAME>:~/selenium-server-standalone-2.37.0.jar
To proceed with the next step we need to make sure that port forwarding is enabled from the host port 4444 to the guest port 4444, this is done in the settings of VirtualBox:
Settings > Network > Advanced > Port Forwarding
VirtualBox will route connections to localhost on the host port to the port on the guest.
Everything is now in place and we can begin to run our tests, once again from the host issue the following command:
ant -Dtm.seleniumDriver=FirefoxDriver -Dtm.seleniumURL=http://localhost:4444/wd/hun -Dtm.testNGOutput=test_results/<CLIENTNAME>-`date +%Y-%m-%d`
The results that have been generated can now be viewed locally, or copied to some remote destination if needed.
scp -r testng_output someuser@someserver:/somedir
To make things even simpler there is a bash script that does all the heavy lifting for us. Not not only that but we can also run the same suite of tests on multiple guest VirtualBox machines that we have setup
In the same directory, tm-examples/eGov/bolzano you can simply run:
tm-vmm --clients-exec script.sh “test0;test1;test2”
Now tm-vmm will start the same action for clients test1, test2 etc etc. If you want the script to perform the tests on all clients, simply leave out the client names, or specify the ones you want the tests to run on.
The main part of setting up the environment is now complete. The script can be automated using cron. The actual work here comes in writing the eGov tests which can be rather time consuming but it is well worth it. Every eGov website will be slightly different, we hope however that with the examples we have provided that anyone using this will quickly understand how to become an eGov developer, by expanding on the existing Bash code base and being able to modify the Selenium tests that have been written in Java. To conclude, once the setup with the virtual machines is complete, it is simply a matter of writing tests for Selenium, that can then be run against different browsers on virtual machines without having to change any code in the tests.
Try to be as precise as possible when reporting bugs. The more information we get the bigger chance we have of fixing the problem.
Use the mailing list below to report bugs
How to join: clone the repo, try it out -- join the mailing list.
For more information read the developer guidelines.
We have one mailing list for the project: firstname.lastname@example.org
Join this list here: https://lists.testingmachine.eu/cgi-bin/mailman/listinfo/community
If you send emails to this list as a non subscriber chances are it will get lost.
If you want to report a bug:
Social Media https://twitter.com/FSCRS