Tuesday, March 11, 2014

Functional Testing (without Acceptance Testing) in Agile Development

Functional testing tools can be used in various ways. Today I outline the ways that I like and dislike, to use functional testing tools in an agile development environment.

Topics covered

Agile Testing Pyramid

  • Misuses of functional testing tools
  • The best uses of functional testing tools
  • Functional testing and TDD
  • Well written functional tests
  • Functional Testing Smells
  • Differences between Acceptance Testing and Functional Testing
  • Summary
  • Further reading

Let’s get dislike out of the way first.

Misuses of functional testing tools

Acceptance Testing

Unfortunately, the word acceptance testing is broadly misunderstood and in fact in most instances that I hear it used, it would be better replaced with the term “functional testing” when I hear what was meant. I’m not going to go too deeply into what acceptance testing is and isn’t but have a short section toward the end of this blog to discuss this topic further. I will just say here that I don’t like acceptance testing and I think there are better uses for functional testing tools.

Replacing Manual Test Cases

Another common mistake I see with functional testing is the misunderstanding that there is a one to one relationship between manual test cases and automated functional tests. Yes you can use functional testing tools in this way but I would discourage it. (You can in fact remove manual testing completely by following the agile testing pyramid.) It is certainly not a one to one replacement and trying to do this results in an unwieldy and unsustainable suite of tests.

The best uses of functional testing tools

Happy Path / End to end / Sanity check / Smoke test

Running smoke/sanity tests at the end of every build that checks that your application does all the basic functionality that it is designed to do makes a lot of sense. Here is an e-commerce example: - can I select a product from the first page of the web app and go through each page to purchase the product affirming that the mock or fake services have been called and the product order placed? If your app does not do the simplest variants of the main functions that it is designed for, then you have a serious problem. That is why it is a good idea to run sanity checks / smoke tests on each build. Functional testing tools are a great tool to run these “happy path” end to end scenarios automatically.

Integration testing

Similar process as end to end testing but limited to testing partially through your application. For example, I need to ensure that the log file is being written to with the correct information and the file is in the correct location. I might run one (or a handful) of functional tests to verify this functionality (and for the rest use mocks in unit tests). Generally, you are in the integration testing realm when your test is touching databases, the network or the file system.

Environment testing

If your build environment is finicky, you can start your build by asserting that the environment is correct. Some examples are: - correct version of Java, version of Maven, version of the OS, you can connect the dev database, the database is in a known state, etc. These types of tests are useful to convey information about the system and can stop a whole lot of pain in debugging environmental issues and keeping a team informed of when and if they need to change/update their environment. Also a great example of using tests to replace documentation.

Build testing

Your build scripts might be doing some pretty complicated things. I like to ensure that no one can accidentally break the build by using post build artifact tests. Are all the pieces in the artifact and in the right places? Has the end artifact been minified / encrypted / obfuscated as expected? Is there even a build artifact? Does the size look correct/odd? Is the version numbering correct? Has it been auto deployed into the dev / test environment? Does it start? Is the artifact named correctly? Etc.

Build scripts are code. Treat them as a first class citizen and you will save yourself a bunch of time in debugging faulty builds. That includes testing them (and refactoring)!

Things you think will break (high risk features)

Consider: - There is an edge case that you had to write some pretty strange code to cater for. The likelihood or frequency of this issue happening is low but the cost of it not getting handled correctly could be very high. Try as you might, you worry that the unit tests and coding clarity might still not be understood by an unsuspecting developer that may stumble across it in the future and refactor or change it incorrectly. An answer is to add additional test coverage to the unit tests and cover the edge case with a well-constructed functional test. Good test code documents your system (as does well written source code itself). Make your tests (and code) as readable as possible and be sure to express clear intent in your tests. That way, when someone else is refactoring or changing the system and your test fails – they will know why that weird code was there and put it back to the state required to pass the test(s). Functional test coverage can be used this way to ensure that the edge case remains accounted for and documented.

Things that keep on breaking (fragile features)

Follow the same strategy as high risk features.

Things that are impossible (or at best, extremely difficult) to unit test

A good agile development team will avoid technologies and frameworks that make testing difficult. However, there are times that this is unavoidable and functional style testing is all that is available to you.

Bug fixing

One of the agile rules I was taught was – before you fix a bug, write a failing test. Sometimes that test makes more sense to be a functional or integration test than a unit test. It might be such a strange edge case that it is best captured in a functional test (as well unit tests).

Legacy systems

When it comes to working on a legacy system (one that has very low to no test coverage) the more tests the better. Functional testing makes a lot of sense along with as many unit tests as you can manage.

Functional Testing is Not TDD

Test Driven Development/Design (TDD) is a practice associated with unit testing. You could write a functional tests before coding and in some ways follow the same paradigm but this is not TDD! It is commonly referred to as ATDD. I'd rather not go down that rabbit hole here but just wanted to make it clear that the use of functional testing tools IS NOT Test Driven Design / Test Driven Development (TDD)!!!!! Learn TDD, then functional testing and you will have a better idea of when and how to write functional tests.

Attributes of Well Written Functional Tests

  • Express intent
  • First class citizen – keep your tests refactored, in source control and run often
  • Grouped into sensible suites

Functional Testing Smells

Intermittent failures / false positives

Not uncommon in functional testing unfortunately. I overheard from a dev team recently - “Oh yeah. If the frank tests fail just try run them again. Sometimes they fail the first time.” Doh! Functional testing smell. (Frank is a functional testing tool for iOS.)

Tests taking too long

Functional tests typically do run slower than unit tests and the cumulative time it can take to run these can get considerable if you are not careful or take some action to ensure you can keep a ten minute build. E.g. if you need to test a timeout then make the timeout period injectable and inject  short period.

Another practice I have seen is to pull slow running tests out into their own test suite. If the dev machine build time is exceeding ten minutes (including running the functional tests) then you could remove this suite from your local build and have it run on the CI server instead. (Requires a healthy CI process.)

Test suites too large

Unless you keep an eye on your functional tests and refactor, refresh and change them as needed, your suite will begin to grow out of hand and you will likely have duplication. You are creating technical debt if you do not maintain this suite the same way you should be maintaining your code and unit tests i.e. as a first class citizen and refactoring mercilessly. Refactoring functional tests is not as straightforward as unit tests but you need to be disciplined about this or the suite will become so large that it will end up getting thrown out.

How does Acceptance Testing differ to Functional Testing?

My definition of acceptance testing is writing a suite of automated tests that aim to verify that a story (or suite of stories) is implemented correctly i.e. functions in the way the writer intended it. More often than not these tests are functional/integration/end to end in nature rather than unit tests because that is the way that we describe the acceptance – in terms of functionality. The tests do not have to be written prior to any code but if you did, then you are in the realm of ATDD (Acceptance Test Driven Development).

Having practiced acceptance testing for a period of time in earnest, I found I didn't like the practice and feel it unnecessary if you are following the agile testing pyramid. I dislike it for many reasons including that acceptance testing encourages teams to invert the testing pyramid and create waste in the form of tests that provide little to no value in the broader life of development of the product.


Functional testing done right is another tool for ensuring quality in your code and product. Use it wisely, follow the agile testing pyramid paradigm and you have a great tool in your agile development tool box.

Further reading

This article describes a great approach to use functional tools for working on legacy systems. He calls his process ATDD and I guess it would be if the tests match acceptance criteria for a chosen story. Regardless of semantics, I like his approach for legacy system testing. http://www.infoq.com/articles/atdd-from-the-trenches

The Ten Minute Build - one of Extreme Programming primary practices. Running functional tests needs to fit inside this ten minute window - James Shore's description

Is Acceptance Testing a dead horse - article written by myself on how the practice of Acceptance Testing and the inversion of the testing pyramid

Merciless Refactoring - another Extreme Programming practice. Do it!

The great divide of Acceptance Testing in the agile community - http://www.infoq.com/news/2010/04/dont-automate-acceptance-tests

Agile Test Pyramid - description (and minor rant on it's inversion) by Martin Fowler