Recently we had @kevinrutherford in to talk about object discovery. While TDD is a great tool, it is no excuse for some sort of design process. If you don’t, you’ll most likely end up with a mess. This session was based around this principle.

One point that was discussed what that the first few seconds of a test for a new class is the most important part of the TDD process. It is here where you will decide whether to take a state or interaction based approach to testing. This first test will dictate the structure of the new class. Once you start with tests, it becomes difficult to evolve or change the design of a class without friction.

There is generally two types of code at Codeweavers.

• Procedural, e.g. how a computer thinks, e.g. after step x, execute step y.

  car.engine.starter_motor.start()

• Object Oriented, e.g how we think in real life. We tell people to do things.

  car.start()

The trick comes from the fact that just because you use C#/Java/etc.. most of the time you aren’t actually writing OO code. You often end up writing procedural code in a rather obtuse manner. A takeaway from this afternoon was to try and perform more CRC (class-responsibility-collaborator) sessions.

CRC (Object Cube)

During this session we performed a modified version of a CRC. One thing I always find with CRC sessions is how useful they are. The problem I and others find is when it comes to day to day development, I’m too eager to start coding without performing some sort of up front design.

Events

In order to write flexible OO code, you need to hide as much state as possible. While this is great in practice it turns out to be very difficult to achieve in the real world. One method of getting around this is to make use of events aka the observer pattern.

Kevin used a wiki as an example. Consider a page that is updated. If this page was to fire a changed event, then anything that is listening for these events would be notified. They observers could then react once the event was received. The nice thing about this approach is that the code obeys the open/closed principle. New features can be added without the need for the change to have a large, rippling effect.

Nouns and Namespaces

When I was first exposed to object oriented programming, apart from believing that inheritance was the coolest aspect of OO, I was also led to believe that using nouns when designing classes can be useful. It turned out that this was a fallacy. If you follow this advice you’ll end up with a small collection of classes that do everything. The biggest revelation I had during the session came from the importance of namespaces. Using nouns for up front design can be extremely useful if you remember that the noun can potentially be a namespace. This means you’ll end up with a handful of namespaces that are relevant to the domain in question. Inside these namespaces you will have one or more classes that do one thing, and one thing well.

Closing

At the end of the session I was left with many questions and new ideas to test and try out. Using events to hide state, yet allow other objects to interact when required was a real eye opener. However it turns out that when actually trying to implement these ideas in code, it is actually quite difficult due to my current thinking. The plan for the next month will be to explore these ideas in more detail and see what affect it has on my day to day development.

A recent blog post by Steve Klabnik concluded with a statement about tossing unit tests if you have end to end tests covering the code in question.

Don’t be afraid to change the tests! As soon as you’ve verified that you’ve transcribed the code correctly, don’t be afraid to just nuke things and start again. Especially if you have integration level tests that confirm that your features actually work, your unit tests are expendable. If they’re not useful, kill them!

A few people on Twitter found this odd, and I’d have included myself in this statement a while back.

https://twitter.com/F1nglas/statuses/160776765353111552

Kent Beck’s TDD screencasts changed my view on deleting unit tests however. During the later videos, he actually deleted some tests. Pretty much all TDD resources don’t really mention this. One of the key points beginners learn is that if you break any tests, you’ve introduced a regression. This is not always the case. If you follow the rule of never deleting ANY tests you encounter you are going to be stuck with someone else’s implementation forever. Likewise unit tests are there to drive design, not enforce how something works. I remember discussing deleting unit tests with my work colleagues and finding Kent’s videos pretty shocking at the time. I mean deleting unit tests!?

The more I do TDD, the less this statement becomes so jarring. For example.

[sourcecode language="csharp"]
public Result DoSomething()
{
// Implementation here.
// Pretty straightforward.
return result;
}
[/sourcecode]

Consider a test for the above behavior, such as we get the result back in a particular state. Pretend the logic is rather simple, and it does not warrant a separate object. Any other developer should be free to come along and change the internals of this method. As long as we get a result back in the correct state, the test should be valid. The test should not care that we are using strings, lists or whatever internally.

Occasionally I find tests like this hard to pass. In other words, I feel like the logic is correct yet the test fails. Maybe I’m using a new language feature, or a language feature that seems to be not working as I expected. If so I’ll break out a new unit test that tests the implementation. Such tests are often refereed to as learning tests. Here with a smaller focus I often become aware of what I’m doing wrong. Following Kent Becks example, I ditch the test after and move on.

I feel this sums up my feelings nicely.

https://twitter.com/avdi/statuses/160780747827974145

I and others are not saying bin every unit test you have that is covered by end to end tests. Unit tests are great, you can run hundreds in a matter of seconds. They have their place as part of the development process, but do not find yourself working against them. However I am saying you should delete any test which relies on implementation details. I am saying bin any test which does not make sense. I am also saying bin tests as part of a refactoring session as long as you have test coverage higher up. If you don’t have test coverage such as acceptance tests, you cannot be sure you have not broke anything after the refactor.

Having finally achieved more stable end to end tests via Selenium, we figured it would be worth while sharing how we achieved this. The following are six steps we’ve found that you can do to make Selenium tests more stable.

1. Turn off automatic updates for your browser/plugins
2. Set your IIS (or equivalent) app timeouts to zero
3. Create a base Selenium Fixture for use in your tests
4. Update to the latest version of Selenium
5. Warm up your apps prior to testing
6. Ditch Selenium – test at the API level

Turning off automatic updates seems like a no brainer, but after a fresh install we forgot to do this once and spent some time figuring out why Firefox would not load on the CI server. It turns out that the “You’ve just updated” window was blocking the test from continuing as it had stole focus.

The second point is with regards caching and the general responsiveness of your application. We have a few applications that take about thirty seconds to fully warm up due to the huge data set they rely on. If we can build this cache once, then store it for as long as possible, subsequent hits to the app should be instant. In other words, we try to mirror our live environment as much as possible.

Our custom test fixture attribute enables the ability to modify all Selenium tests in one go. We found that from 3am to 5am our databases undergo maintenance, therefore we do not run our regression tests during this time. All this took was one change within the attribute to apply to all tests. For example:

public class RegressionTestFixture : TestFixtureAttribute
{
    public RegressionTestFixture()
    {
        Category = "Regression";        
        OnlyRunTestsWhenDatabaseIsActive();
        // Etc...
    }
}

We simply inherit from NUnit’s TestFixtureAttribute and use this custom attribute rather than the standard TestFixture attribute. The inheritance is required to ensure that third party tools such as test runners still work as expected.

Previously we were using Selenium 1.x with Sauce RC. Having ditched this and upgraded to Selenium 2.x we’ve been able to update our browsers to the latest versions, in turn this allows improved speed and stability when runing the tests.

On our local development machines the application you are working on is often in memory, meaning subsequent hits should be much faster after all dependencies are loaded and cached. The issue we discovered on our CI server was that after a fresh build of the whole codebase, the initial hits to the applications would be very slow. To combat this we added a warm up step to our build. Just before the tests are run we would perform a HTTP GET to invoke our applications start up processes. This added somewhere in the region of thirty seconds to the build, but the increase in stability is staggering. No longer will Selenium report timeouts.

Finally the fastest end to end tests come from not using Selenium. Ditching the browser completely and testing as high up in your API is the quickest, and most stable solution. Combining this thinking, with a handful of dumb Selenium tests that just check for the likes of 404s seems to be the most optimal solution currently.

Having done these at some point over the past few months we’re starting to get to a more stable point with our Selenium tests. We’ll be looking to take this forward with future tests and hope to enjoy continued stability.

Recently myself and another colleague wrote an acceptance test for a feature that had yet to be implemented. For this end to end test we used Selenium, after all we wanted to test the whole feature so this made sense. Our test performed some simple user input, performed a calculation and checked the response. The problem with the test was it was very brittle. If the application had not recently been used, the massive data set the application relied on would not be cached.

To get around this we added a few Thread.Sleep() statements into the test. This worked rather well for the majority of test runs, but sometimes these pauses were not long enough. On the other hand sometimes the data was cached, meaning these sleeps would be unnecessary. One resource which has recently done the rounds was regarding useful advice about using WaitForPageLoad() and WaitForCondition(). WaitForCondition will only execute once a condition has been met, such as a element becoming visible. This meant that for the times when the dataset was in memory the test would be executed immediately, while the times when the data was being loaded, the test would simply wait until the test was ready to move on. This was a very simple, yet highly effective tweak to our tests. The execution time went from roughly thirty seconds, to just less than ten seconds in one case.

This was not the end of the battle to achieve more stable Selenium tests. Some of our tests were still rather flaky. Some mornings we would enter work, notice the red build and discover that the several failed tests were down to Selenium timeouts. During the daytime however, we rarely had these issues. In order to fix these problems I increased the frequency of builds. The idea being the more we run our builds the more chance we have of spotting the errors. After all, if something was to fail at 2am, I am unlikely to care. 2pm however, and the team will be all over it. By making the problem more visible, we would be forced to fix the outstanding issues.

The aim was to make the tests as fast as possible, while maintaining stability. One thing the excellent Growing Object-Oriented Software (Goos) touches on is the aspect of not needing to perform end to end testing at the GUI all the time. The benefit of not touching the UI is huge. Your tests are faster, they’re more stable and a heck of lot easier to write. The other nice benefit of testing from an API point of view, rather than the browser is it forces you to decouple your app from the views. If you’re not writing fat models and skinny controllers, you’ll have adapt in order to test as much of your application as possible without hitting the UI.

What about the remaining part of your feature that is not covered by the application? I like to imagine this part as the tip of an iceberg. As this area is small enough the actual UI testing you need should be minimal. So here we can let Selenium do what it is good at. Click things. Selenium is great at this. All you need to do at this level is check for 404s, incorrect page titles and a few other mundane aspects of the UI. There should be no need to check if your actual application is correct at this level. For correctness, you should have a large suite of fast, isolated, unit tests.

Another point to consider is how often your view actually changes, in comparison to your actual underlying API. A designer should be free to move things, rename content, add images and so forth without breaking tests. As long as there is a calculate button somewhere on the page, and said button takes you to a result page, who cares about everything else? Likewise the code underneath can be consistently changing behind the scenes, as long as the API remains constant, our tests should always be valid.

For the technical low down on some of the ways we are achieving more stable end to end tests, check out six tips to speed up Selenium tests.

Image courtesty of Natalie Lucier

We try our best to ensure our test code here at Codeweavers is free of duplication. Not only this but I would argue that we give it the same care as the production code, which is a good thing. Despite these best intentions, over time test code can rot. That nicely factored test fixture you wrote a few months ago? Now it’s a stinking mess of duplication. This can happen for numerous reasons. One of the factors for this degradation occurs when new test fixtures are added.

We often use a test fixture per behavior, so our tests are often grouped by behavior, for example “GivenASuccessfulQuote” or “GivenAUnsuccessfulQuote” would be test fixtures for testing the quotes behavior. These fixtures would test the happy path of a quote and erroneous behavior respectively. The issue comes when we wish to add a new feature. Imagine we wanted the unsuccessful quotes to return a result with a message explaining why the quote was unsuccessful. We’d often start by writing a new fixture, adding the required set up and relevant asserts/verifies. You might argue that we should look for similar fixtures to add to at this point, but I prefer duplicating code (intentionally or unintentionally) first to get the feature working.

Recently having carried out a similar process mentioned above myself and another developer noticed that there was duplication, albeit slightly different between our new fixture and the unsuccessful quote fixture. We never wrote the other fixture, so we never planned for our new fixture to become so similar. I tried rather unsuccessfully removing the common setup code between the two classes. The revelation came when we took the assert from our new fixture and pushed it up into the unsuccessful quote scenario. This assert failed as we were missing the expected message that was not being set up correctly. After moving this expected message from the new fixture up into the unsuccessful fixture the assert passed. Once this went green, we pushed the assert out of the test we had just migrated to and all was well. The new message fixture could be deleted, after all it was behavior of an unsuccessful quote. Having done this we removed in the region of about twenty lines of code. I was stunned at how well this worked. We had essentially tested our test code worked.

The process was as follows – Remember, it’s only worth doing this if the behavior you are testing makes sense in the fixture you wish to migrate to. Putting all your tests into one fixture defeats the purpose of grouping behavior via test fixtures.

Steps:

1. Write new fixture – make its test(s) pass
2. Identify duplication between fixtures
3. Migrate if it makes sense for the behavior you are testing to belong on a new fixture – otherwise try and remove duplication another way
4. Push assert into an existing test method
5. Run tests
6. If the tests pass you’re done, or if they fail move the required setup from the new fixture into the current fixture
7. Run tests
8. If the tests pass – move assert(s) into separate test method(s) otherwise go to step 6
9. Delete new fixture
10. Run tests

I have to thank Paul Shannon for this gem as he was the one who explained how to do this. He plays the process down but the fact that this is a safe way to remove duplication between tests is great.

Pseudo code worked example

Part 1:
[sourcecode language="csharp"]
class GivenAUnSuccessfulQuote
Setup()
ThenItShouldBeInValid()
ThenItShouldDoOtherStuff()
end
[/sourcecode]

[sourcecode language="csharp"]
class GivenAUnSuccessfulQuotesMessage
Setup() // This is near duplication above
ThenItShouldReturnAInfoMessage()
ThenTheMessageShouldDoOtherStuff()
end
[/sourcecode]

Part 2:
[sourcecode language="csharp"]
class GivenAUnSuccessfulQuote
Setup()
ThenItShouldBeInValid()
ThenItShouldDoOtherStuff()
ThenItShouldReturnAInfoMessage() // Moved this from the fixture below
ThenTheMessageShouldDoOtherStuff() // Moved this from the fixture below
end
[/sourcecode]

[sourcecode language="csharp"]
class GivenAUnSuccessfulQuotesMessage
Setup() // This setup is near duplication of the fixture above
end
[/sourcecode]

Part 3:

Run tests

Part 4:

Move required setup from the GivenAUnSuccessfulQuotesMessage until the GivenAUnSuccessfulQuote fixture is passing.

Part 5

Delete the GivenAUnSuccessfulQuotesMessage fixture.