The need for acceptance testing has become too important to our product and is now included in our automated build environment (more info on automated build environment here). While Test Driven Development (TDD) does produce good quality on the specification of the requirement from the component side, it doesn’t adequately target “how the system” should act from the end user perspective. There’s a lot of UI specific tests that need to be performed on the views to ensure that the correct functionality is doing what it was intended to do. In an agile development environment, like Extreme Programming, these black box system tests are known as functional tests which tests the validity of a user story; as opposed to unit test which tests the verification of a specification (written TDD test-first style in our office, which helps but are still from the point of view of the developer). On the other hand, functional tests can be written by testers, business analyst or the developers, but since they are from the perspective of the end user, or subject matter expert, they are best written (or at least specified) by the business analyst. During the teams research we wanted to find a tool that can be integrated into our environment as well as easy enough for a business analyst to write.
Here’s a resource to the 4 testing quadrants as identified by Janet Gregory. Image taken from http://agilevancouver.ca/sites/agilevancouver/files/speakerslides/Vancouver-Quadrants.pdf
Here’s a list of functional testing tools identified by Gerard Meszaros and added to by others in the industry . These functional testing tools focus mainly on the 2nd Quadrant above.
The one I am currently trying out iTest2, basically uses Watir under the hood but includes a nice IDE (actively being developed) and an abstraction on top of Watir called RWebSpec. It’s easy enough for the business users to add functional tests (especially if they’re using the recorder), but complex enough for when the developers/testers need to access to the COM interfaces which gets at the browser’s document object model, DOM, without need for directly interfacing with the UI. So far this tool is starting to prove itself but we’ll need more time to see if it is the most effective one out there.
Things still left to do:
- Include functional tests as part of our story breakdown planning and estimating process (Example here and they can act as requirements artifacts/documentation).
- Try out StoryTeller as a solution to automated acceptance testing in .NET.
- Assess how we’re doing in the 4 testing quadrants (above), then alleviate any shortfalls.
- Figure out how the database gets loaded with information from our functional tests and how tests can be independent from each other (test servers, scripts).
- Move to a better functional testing as compared to plain old record/playback because it can become cumbersome to maintain.
- Stay on top of software testing as the advancements in this area are changing daily http://www.testingreflections.com/