Posted by: craigtech | April 28, 2009

Automated functional testing with Selenium, CruiseControl.net & MsTest

I’ve had a real problem with perceived “quality” on a project I’ve been working on recently – this is no slant on the developers, but in my opinionstems from the fact that the customer organisation has a testing team larger than our development and test teams put together, and this has put an unneccessary strain on the working relationship.

The only way I could think to get on top of this was to take as much of the repetitive regression testing as possible (a manual regression cycle was lasting around 4 hours), automate it, and run it frequently from our CI server.

My requirements are:

  • The tests themselves need to be maintainable by our testers (who have minimal development knowledge), with a simple point and click interface.
  • The tests need to be run in firefox, IE6 & 7.
  • I need to be able to collate and track the testing metrics; test counts/passes/fails, etc
  • I need to track screenshots during the test process so that testers can spot-validate the results of the automate tests.

Having researched the options out there Selenium seemed a good fit; so I began by writing a simple one-test suite.  I’ll describe the process I went through, the problems I came across, and the solutions I found here.

Selenium IDE is a great little firefox plugin that gives exactly what we need as far as the testers are concerned.   My only slight concern is that some of our applications are IE only, so we’ll need to look at a recording application for IE further down the line.

The first problem I came across was when attempting to run this html suite against the selenium server jar.   No matter how hard I tried I couldn’t get them to run.  I googled about and found it was a recurring theme, but there was a suggested patch for selenium, which I attempted to apply, but got no joy.  As a quick alternative a tried using the C# generated NUnit code instead, and it worked a treat.  Rather than waste too much time trying to track down the issue I decided that using C#coded tests would be a workable option – I made the simple change to move from NUnit to MSTest though.

Having decided this I then had the problem of how to allow the testers to maintain the tests in the selenium IDE HTML format, but have C# tests for run-time.  I wrote a simple parser that searches a given directory for test suite files, follows the links to the tests themselves, and generates the tests.  The generated code is then compiled down to a .net assembly to be run by MSTest.

So, I now had my automated tests in place, the next problem was to get it running under cruise control.net.  This brought another set of challenges:

  • Selenium RC server currently runs from the command line, but I’m intending to initially have two projects running these selenium tests; rather than fire up a selenium server for every project, or attempt to use the same server across multiple projects my preferred approach was to run selenium as a windows service.
  • Our cruisecontrol server is intended to be a lightweight system, and doesn’t have visual studio installed, I could have ducked out of this problem & used NUnit but I’d read about how to get it running standalone and decided to get that up & running at the same time (it turned out someone else had already tried to do this  on the server, but not got it to work).
  • Our selenium tests write out images to disk, and these images need to be included as part of the build report.  It turns out cruisecontrol.net doesn’t deal with non XML output very well.

So getting selenium running as a windows service was relativelly easy in the end, using the Java Service Wrapper.   The two problems I came across here were:

  • You need to put the selenium jar inside the bin directory of the JSW – this was counter intuitive to me, but it worked.  So I ended up copying the contents of the JSW zip file into a directory D:\selenium\selenium-server-1.0-beta-2, and then copied the selenium-server.jar, from the Selenium RC download inside the bin directory of that, i.e. D:\selenium\selenium-server-1.0-beta-2\bin\selenium-server.jar.
  • The conf file D:\selenium\selenium-server-1.0-beta-2\conf\wrapper.conf required changes:
    –  Add another classpath “wrapper.java.classpath.3=selenium-server.jar” below “wrapper.java.classpath.2=../lib/wrapper.jar” 
    –  Modify “wrapper.app.parameter.1″= to be “wrapper.app.parameter.1=org.openqa.selenium.server.SeleniumServer”
    –  Modify the ntservice.name, display name & description to be something relevant, i.e “wrapper.ntservice.name=seleniumserverwrapper”, “wrapper.ntservice.displayname=Selenium Server Beta 2”, “wrapper.ntservice.description=Selenium Server Beta 2”

With that setup you can run the install testwrapper.bat file, and start the service.  Selenium is then running and can be connected to using the .Net client (ThoughtWorks.Selenium.Core.dll another part of the Selenium RC download).

This article described how to get MsTest up and running standalone.  My only concern with this was the need to use the /noisolation flag when invoking MsTest, but for now this will be fine.

[2009-05-06 Update]
I’ve found the reason for needing the /noisolation flag – I was getting an error:

Failed to queue test run ‘cedmunds@ZEUS 2009-05-06 14:41:24′: Failed to get host
 process location. Host process is not available.

It was because the key below did not update correctly when I ran the reg file, and  after re-entering the key manually all worked fine

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools]
“HostProcessLocation”=”D:\ContinuousIntegration\System\VisualStudioTools\\Common7\\IDE\\vstesthost.exe”

[/update]

This article described how to include images in the crusie control build report.  I haven’t actually gotten around to including this yet, but I’ll report back when I do.

It’s also worth pointing out that the system uses projects in SVN to store the tests; the testers need to keep the SVN repository up to date (we use tortoisesvn for this purpose) and I think it would be a great enhancement to Selenium IDE to integrate with SVN/team system.

In short, with a relatively small amount of custom code I’ve managed to get these technologies working together in around 2-3 days.  If that can free up just 2 testers from half a day of mindless regression testing each, twice a week, the effort will have repaid itslef in a couple of weeks!  As long as you’re able to write your tests in Firefox (or hand craft them) then this is a perfect approach.

P.S. The selenium HTML -> C# transformation code isn’t completed yet, in fact I’m going to see if I can harness the code within Selenium IDE to do the transform. When I’ve finalised the approach I’ll update this, and share the code via CodePlex if applicable.

Advertisements

Responses

  1. Thanks for the informative post. I’m wondering if you’re running cruise control on a server with an actively logged in user? The one problem we’ve encountered with Selenium and cc.net is that Selenium just won’t work on our build server because it needs a logged in user in order to open browser windows. Have you ran into this?

    Thanks

    • Hi Grant. Yes, we run both Java service wrapper around selenium and cruise control under domain accounts, but I don’t see why you wouldn’t be able to run the JSW/Selenium service as a local user though. How are you running your selenium scripts?

  2. We’re running the scripts as you describe, via CruiseControl.Net kicking off NUnit tests, which the Selenium C# drivers. With the service we created, we attempted both local and domain accounts. We weren’t using the Java Service Wrapper, however, and I think I’ll give that a go in the next week or two. Thanks for your time.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: