Acceptance test driven development, or ATDD, is a collaborative practice wherein application developers, software users, and business analysts define automated acceptance criteria very early in the application development process. They then use the acceptance criteria to guide subsequent development work. As John Ferguson Smart explains in this JavaWorld feature, ATDD is a simple process change that can have far-reaching implications for your development projects.
From acceptance tests to ATDD
The idea of acceptance tests — a set of tests that must pass before an application can be considered finished — is certainly not new. Indeed, the value of testing an application before delivering it is relatively well established.
Traditionally, testers will prepare test plans and execute tests manually at the end of the software development phase. Acceptance testing is done relatively independent of development activities. In some organizations, QA departments also use automated testing tools such as HP’s Quick Test Pro; but, again, this activity is generally siloed away from the rest of the development activity.
Testing an application after it has been developed has a number of significant drawbacks. Most importantly, having feedback about problems raised at this late stage of development makes it very difficult to correct bugs of any size. This results in costly rework, wasted developer time, and delayed deliveries.
ATDD takes a different approach. Essentially, ATDD involves collaboratively defining and automating the acceptance tests for upcoming work before it even begins — a simple inversion that turns out to be a real game changer. Rather than validating what has been developed at the end of the development process, ATDD actively pilots the project from the start. Rather than being an activity reserved to the QA team, ATDD is a collaborative exercise that involves product owners, business analysts, testers, and developers. And rather than just testing the finished product, ATDD helps to ensure that all project members understand precisely what needs to be done, even before the programming starts.
In addition, acceptance tests are no longer cantoned to the end of the project and performed as an isolated activity. Instead, ATDD tests are automated and fully integrated throughout the development process. As a result, issues are raised faster and can be fixed more quickly and less expensively, the workload on QA at the end of the project is greatly reduced, and the team is able to respond to change faster and more effectively.
ATDD in practice
Let’s consider how ATDD typically works in the context of an agile project. As a rule, a software project aims at delivering end-users with a number of high-level “features” (sometimes called functionalities or capabilities). A feature is a general value-proposition relating to something the application can do for the end-user, expressed in terms you might put on a product flyer or press release: for example, a feature of an online real-estate lease-management application might be “Manage property repairs.”
Features are generally too big to implement all at once, so they are broken into smaller, more manageable chunks. In agile circles, these chunks are often expressed in the form of user stories — a short sentence capturing what the user wants from a particular piece of functionality. For example, user stories for the “Manage property repairs” feature might include “Issue work order” and “Approve invoice.”
A user story cannot stand alone, however; it is merely the promise of a conversation between developers and users about a particular requirement. The details about what needs to be implemented will arise from this conversation. It will then be formalized as a set of objective, demonstrable acceptance criteria. For example, you would need to specify acceptance criteria for “user can approve an invoice for an amount less than the agreed maximum” and “user cannot approve an invoice if the price exceeds the agreed maximum.”
Acceptance criteria determine when a particular user story is ready to be deployed into production. But they do much more than record what should be tested at the end of an iteration. Acceptance criteria are drawn up as a collaborative exercise, at the start of the iteration, with developers, testers, and product owners involved. As a result, they help ensure that everyone on the team has a clear vision of what is required. They also help provide clear guidelines for developers as to what needs to be implemented. (These guidelines are even more effective if the developers doing the programming are practicing Test Driven Development, or TDD.)
ATDD and TDD
TDD, or Test-Driven Development, is a highly effective development strategy that helps developers write code more accurately and precisely. The low-level requirements used to drive TDD are directly derived from the high-level acceptance tests, so the two techniques complement each other: automated acceptance tests describe the high level business objectives, while TDD helps developers implement them as requirements.
Note that acceptance criteria are not designed to be exhaustive — there will be more technical tests for that. Instead, they are used as much for communication as they are for verification. They take the form of working examples, which is why ATDD is sometimes referred to as “specification by example.”
Acceptance-test driven development is not just limited to agile projects. Even teams using more formal and detailed use cases, or more traditional approaches such as the Software Requirements Specification (or SRS) documents, can benefit from having verifiable, automated acceptance criteria as early as possible.
Automating your acceptance tests
A key part of acceptance criteria is that they are automated. They are not simply stored in a Word document or Excel spreadsheet, but are living, executable tests. This is important — for ATDD to be effective, the automated acceptance tests need to be run automatically whenever a change is made to the source code. So it is vitally important to have a tool that will integrate smoothly into your build process, and that can be run on your build server with no human intervention.
Automated acceptance tests not only serve to test the application: they also provide an objective measurement of progress (in agile projects, working software is considered to be the only true measure of progress). The tests can also give an idea of the relative complexity of each feature and story, because a functionality that is long and complicated to test is likely to also be long and complicated to develop. This in turn can give a useful heads-up to product owners needing to set priorities.
Although you certainly can write automated acceptance tests using conventional unit testing tools such as TestNG, there are a number of dedicated ATDD tools. These tools are focused as much on communication and feedback as they are on testing.
ATDD tools
ATDD is more an approach than a toolset, but there are a number of tools that can make things easier.
FitNesse is one of the earliest ATDD tools. Using FitNesse, users enter their requirements in tabular form in a Wiki, and developers write code behind the scenes to run the test data stored in the Wiki against the actual application. When the tests are executed, the table will be colored according to whether the tests succeeded or failed. FitNesse is very useful when your acceptance test criteria can be expressed in terms of tables of data and expected results, although it is also used to express acceptance tests as a series of steps.
More recently, other tools have emerged that support Behaviour-Driven Development, or BDD. This technique encourages developers to think in terms of the behaviour of an application, and to express their low-level technical requirements using a narrative approach. Cucumber is a popular tool from the Ruby community, that allows you to express your acceptance criteria using the “given-when-then” structure commonly used in agile projects. It is also easy to use Cucumber with Java. JBehave uses a similar approach, with stories expressed in text files and tests written using annotated Java classes. Easyb is a similar tool based on the Groovy language.
Concordion is another more recent ATDD tool. In Concordion, acceptance tests are expressed in the form of HTML pages containing free-form text and tables. Java classes are then used to analyze special tags placed in these pages, in order to execute and display the results in HTML form.
All of these tools place a high emphasis on readability and communication. Listing 1 illustrates how one of the earlier acceptance criteria might be expressed using Easyb:
Listing 1. A user scenario in Easyb
scenario User can approve an invoice for an amount less than the agreed maximum"
{
given "the User has selected an open invoice",
and "the User has chosen to approve the invoice",
and "the invoice amount is less than the agreed maximum amount",
when "the User completes the action",
then "the invoice should be successfully approved",
}
Once the acceptance criteria are defined in this way, the corresponding test code can then be written in more conventional programming languages such as Java, Groovy, and Ruby.
In addition to showcasing Easyb, this code snip shows the communication focus of ATDD tools. Automated acceptance criteria are expressed in high-level terms that makes sense to business managers as much as to software engineers and programmers. Most ATDD tools also generate reports that express the test results in familiar business terms. Tests that have been written in this way, but with no backing test code, will be marked as “pending.” At the start of an iteration, all of the acceptance criteria will be in this state. As development progresses, the next step will be to implement them, which is where the actual code that tests the application is written. So these reports not only tell you what tests pass and fail, they also provide a way to track the progress of your project, by indicating what work remains to be done.
Taking a slightly broader perspective, automated acceptance tests are like any other automated tests — they should be stored in your version control system and executed periodically on your Continuous Integration server (at least on a nightly basis, but preferably whenever a change is made to the application source code). Getting fast feedback when acceptance tests fail is essential. You can also configure your CI server to publish the results of the acceptance tests where they can be easily consulted by non-developers. Fortunately, modern CI tools such as Jenkins integrate well with virtually all of the common BDD tools.
Craft Coder 