Test framework characteristics

Some of the elements of a test framework we should not compromise on such as:

  • Speed
    • When I started off, I got the argument of – if we can run the pack at night and even if it takes hours then it should be fine. It doesn’t work like that. Your test pack is your constant companion, your best friend in the developing world. You need it to run at any time. The theory of running at night may not strike you as a bad idea at first, but here is how it isn’t great either:
      • What about failures?
        • Any failures reported will be picked up the next day. The priority is somewhat reduced on these as the reaction is a bit delayed. The site is fine, a few tests are failing no biggie.
        • Who is going to monitor them?
        • You’ve fixed the tests in theory, now when they run the next night all should be fine. Turns out it isn’t. What do you do now? Make another fix and test it again the next day? That is 3 nights already of failing tests, 3 days that it should have been useful to you but it hasn’t been all because of speed. And this cycle rarely ends.
    • Moral of the story, just don’t compromise on it. I’ve got a hard limit that the pack must finish within 5 minutes no matter what kind of test it contains. Here are some ideas:
  • Reliability
    • A test pack is only as good as its reporting. If it reports false positives or false negatives, you can hardly trust it. If there was a QA reading this, they’d tell ya. Might as well chuck it away. If it has false negatives which is often the case, the strategy is to just run the tests manually to verify and all is fine. But why go through the pain when you can avoid it? Over the years i’ve applied numerous strategies to overcome this issue while apps have become a lot more complex. Here are the ideas:
      • Cut down on UI tests if you can. If you aren’t running a single page app with JS (SPA) then use a headless intelligent crawler such as Goutte. Not only is it reliable but its 2 to 10 times faster than the UI alternatives. UI tests produce flakiness due to transitions into pages etc that happen. Whils’t you can’t avoid having UI tests completely, you can assert the major part of the site such as form submission, page redirection and other content based on submission without UI tests just as service layer tests. Tools such as Behat allow you to run JS enabled and Goutte in conjunction based on the tags you mark your tests with. Best of both worlds.
      • Don’t use wait – spin it. Using wait means slower tests and your test only tries to action something once before it falls over. Using the spin method will try to perform the action every second and when it successfully does, it will move on – making your pack faster and more reliable.
  • Maintenance
    • An often neglected part – you need to think ahead. If you want to maintain a test suite, you need to be able to run it pretty quick and get feedback pretty quick as well. Its fine when all passes, but who takes care of the failing tests? Its always a chore. To help out with this, have a look at this extension https://packagist.org/packages/genesis/behat-fail-aid, this gives you a lot of debugging information for free, even for JS apps! You’ll fix the problem a lot sooner when you’ve got all the facts at hand. It’ll produce screenshots, read the debugging bar, read js errors, print the current url and much more and present all of that in a very nice error message.

Spin method:

* @param callable $lambda Must return true or throw an exception.
* @param integer $wait
* @return void
protected function spin(callable $lambda, $wait = 3)
    $lastErrorMessage = '';

    for ($i = 0; $i < $wait; $i++) {
        try {
            $result = $lambda($this);
            if ($result) {
                return $result;
        } catch (\Exception $e) {
            // do nothing
            $lastErrorMessage = $e->getMessage();


    throw new ElementNotVisible($lastErrorMessage);

You may also like...

Leave a Reply