Having wrote about the top ten things I discovered in my first year at Codeweavers, I figured it would be time for a follow up after the past two years. In no particular order, a collection of the biggest lessons I have experienced.

• Design by Contract
• Test Driven Development (TDD) is a tool
• Design is Important
• Don’t tie yourself to a Framework
• The Importance of Tools
• Acceptance Testing need not use the Full Stack
• Program for Change (Open/Closed Principle)
• Reinvent the Wheel, Often
• Do it right – violate YAGNI
• Practice, Practice, Practice

I’ll expand on these topics over time in future posts.

At the time of writing the latest version of Firefox (version 13) has just been released. Bare in mind that a week ago I updated our Selenium bindings so that we could use Firefox 9+ for running our browser tests.

The latest release is another great release for the Firefox team, except there is software out there will be broken. The software in question I’m talking about is any code that uses Selenium 2.22.0 that was released 2012-05-29. It turns out the bindings only work for Firefox 12 or less.

For whatever reason any tests that used Selenium this morning just stopped working for us – and others. The tests in question caused the runner to hang as no window could be opened. I’m not sure what causes this, as the browser is essentially the same to the end user, bar some new features. Not being a Selenium developer I cannot comment how or why this has happened, nor can I suggest the Selenium team should be version agnositc.

Our solution in the end was simple. Turn off the auto updating and downgrade the browser. I’ve blogged about this in the past, but since Firefox 10 – the team are adopting a “silent” update process. This is great for end users. Imagine the countless man hours saved if IE6 had shipped with an auto update feature? The problem now seems to be in the hands of developers.

Another attempt to make this problem more obvious has been to add a check prior to our tests running to ensure that it can open a window. If this fails or hangs, we display a useful error message indicating that the browser in question is not compatible. This is due to the fact that it is not immediatley obvious what the problem is. More confusion occurs when some machines will execute the tests with no problems at all.

Tools -> Options -> Advanced -> Update Tab

So if you use Selenium and Firefox – ditch the auto updating. Manually update your bindings and check compatability for now…

Back during the later part of 2011 there was a common theme occurring in our retrospectives each week. How can we replicate our live environment as close as possible?

We took steps to achieve this goal by creating a single machine image to ensure all our machines were configured correctly. Another quick win was to ensure certain aspects of our live data was restored to our local development databases during the night. This enabled us to take stack traces from our logs, and quite literally paste them into our IDE and replicate the users problem instantly. Without the same data set we could have seen different results. Despite these positive steps, there was a missing link in our replication process. How do we simulate the traffic of our live environment? As an example, we average anywhere from four to five thousand calculations per minute with our current web services, with our local and demo environment no where near this figure.

During 2011 I found myself involved in many deployments in which despite heavy testing I was uneasy. On our demo environments we could throw the same amount of load against our services, yet sometime after deploying our service would fall over. We would quickly have to revert and go back to the drawing board. The problem we had despite our traffic being mimicked in terms of volume was the load was not real. Our customers however have many more variations of requests that we were simply not predicting. The other obvious issue was during local development, the service may well handle the same volume of traffic, yet once live and the process has been running for a few hours – things might go bump. Factors such as memory or timeouts being the culprits here.

Collectively we had a few ideas on how to solve this. We looked into low level solutions such as directing traffic from IIS/apache towards other servers. We examined other load testing tools, and we even contemplated creating our own load creator. This internal tool would go over our database and fire off a number of requests at our demo environment. I felt uneasy with all these solutions. They were not “real” enough. I wanted the real time traffic to be submitted to our demo services, only then could we have full confidence in our work.

My idea was rather radical in the sense it was so easy, yet dangerous enough that it might just work. I proposed we integrated our own service, into itself. In other words, just before our service returns the results of the calculation, it takes the users request and submits it again, against our demo environment. The same service would be recursively submitting into itself. In order to ensure we did not affect the speed of the service, the submission is performed via an async call, meaning if this second call was to die the live service would be unaffected. The obvious downside here was that in order to test this, we needed to deploy the changes to our live service. This was achieved via a feature toggle, meaning at any time we could turn the feature on or off without affecting any customers.

The end result of this was that when the feature is enabled, the traffic on our live service is sent to our demo service. This allows us to deploy experimental or new features and changes to the demo environment and check them under real load, with real time data. If all goes well after a period of time we can deploy to our live service, if not we roll back and no one is the wiser.

Having finally achieved more stable end to end tests via Selenium, we figured it would be worth while sharing how we achieved this. The following are six steps we’ve found that you can do to make Selenium tests more stable.

1. Turn off automatic updates for your browser/plugins
2. Set your IIS (or equivalent) app timeouts to zero
3. Create a base Selenium Fixture for use in your tests
4. Update to the latest version of Selenium
5. Warm up your apps prior to testing
6. Ditch Selenium – test at the API level

Turning off automatic updates seems like a no brainer, but after a fresh install we forgot to do this once and spent some time figuring out why Firefox would not load on the CI server. It turns out that the “You’ve just updated” window was blocking the test from continuing as it had stole focus.

The second point is with regards caching and the general responsiveness of your application. We have a few applications that take about thirty seconds to fully warm up due to the huge data set they rely on. If we can build this cache once, then store it for as long as possible, subsequent hits to the app should be instant. In other words, we try to mirror our live environment as much as possible.

Our custom test fixture attribute enables the ability to modify all Selenium tests in one go. We found that from 3am to 5am our databases undergo maintenance, therefore we do not run our regression tests during this time. All this took was one change within the attribute to apply to all tests. For example:

public class RegressionTestFixture : TestFixtureAttribute
{
    public RegressionTestFixture()
    {
        Category = "Regression";        
        OnlyRunTestsWhenDatabaseIsActive();
        // Etc...
    }
}

We simply inherit from NUnit’s TestFixtureAttribute and use this custom attribute rather than the standard TestFixture attribute. The inheritance is required to ensure that third party tools such as test runners still work as expected.

Previously we were using Selenium 1.x with Sauce RC. Having ditched this and upgraded to Selenium 2.x we’ve been able to update our browsers to the latest versions, in turn this allows improved speed and stability when runing the tests.

On our local development machines the application you are working on is often in memory, meaning subsequent hits should be much faster after all dependencies are loaded and cached. The issue we discovered on our CI server was that after a fresh build of the whole codebase, the initial hits to the applications would be very slow. To combat this we added a warm up step to our build. Just before the tests are run we would perform a HTTP GET to invoke our applications start up processes. This added somewhere in the region of thirty seconds to the build, but the increase in stability is staggering. No longer will Selenium report timeouts.

Finally the fastest end to end tests come from not using Selenium. Ditching the browser completely and testing as high up in your API is the quickest, and most stable solution. Combining this thinking, with a handful of dumb Selenium tests that just check for the likes of 404s seems to be the most optimal solution currently.

Having done these at some point over the past few months we’re starting to get to a more stable point with our Selenium tests. We’ll be looking to take this forward with future tests and hope to enjoy continued stability.

Recently myself and another colleague wrote an acceptance test for a feature that had yet to be implemented. For this end to end test we used Selenium, after all we wanted to test the whole feature so this made sense. Our test performed some simple user input, performed a calculation and checked the response. The problem with the test was it was very brittle. If the application had not recently been used, the massive data set the application relied on would not be cached.

To get around this we added a few Thread.Sleep() statements into the test. This worked rather well for the majority of test runs, but sometimes these pauses were not long enough. On the other hand sometimes the data was cached, meaning these sleeps would be unnecessary. One resource which has recently done the rounds was regarding useful advice about using WaitForPageLoad() and WaitForCondition(). WaitForCondition will only execute once a condition has been met, such as a element becoming visible. This meant that for the times when the dataset was in memory the test would be executed immediately, while the times when the data was being loaded, the test would simply wait until the test was ready to move on. This was a very simple, yet highly effective tweak to our tests. The execution time went from roughly thirty seconds, to just less than ten seconds in one case.

This was not the end of the battle to achieve more stable Selenium tests. Some of our tests were still rather flaky. Some mornings we would enter work, notice the red build and discover that the several failed tests were down to Selenium timeouts. During the daytime however, we rarely had these issues. In order to fix these problems I increased the frequency of builds. The idea being the more we run our builds the more chance we have of spotting the errors. After all, if something was to fail at 2am, I am unlikely to care. 2pm however, and the team will be all over it. By making the problem more visible, we would be forced to fix the outstanding issues.

The aim was to make the tests as fast as possible, while maintaining stability. One thing the excellent Growing Object-Oriented Software (Goos) touches on is the aspect of not needing to perform end to end testing at the GUI all the time. The benefit of not touching the UI is huge. Your tests are faster, they’re more stable and a heck of lot easier to write. The other nice benefit of testing from an API point of view, rather than the browser is it forces you to decouple your app from the views. If you’re not writing fat models and skinny controllers, you’ll have adapt in order to test as much of your application as possible without hitting the UI.

What about the remaining part of your feature that is not covered by the application? I like to imagine this part as the tip of an iceberg. As this area is small enough the actual UI testing you need should be minimal. So here we can let Selenium do what it is good at. Click things. Selenium is great at this. All you need to do at this level is check for 404s, incorrect page titles and a few other mundane aspects of the UI. There should be no need to check if your actual application is correct at this level. For correctness, you should have a large suite of fast, isolated, unit tests.

Another point to consider is how often your view actually changes, in comparison to your actual underlying API. A designer should be free to move things, rename content, add images and so forth without breaking tests. As long as there is a calculate button somewhere on the page, and said button takes you to a result page, who cares about everything else? Likewise the code underneath can be consistently changing behind the scenes, as long as the API remains constant, our tests should always be valid.

For the technical low down on some of the ways we are achieving more stable end to end tests, check out six tips to speed up Selenium tests.

Image courtesty of Natalie Lucier