Why Most Software Test Plans Fail

That most software test plans fail is apparent in the abominable state of most software products' quality. The blue screens of death, security breaches and unanticipated, quirky behavior, etc characteristic of all too many software products means that each product's test plan failed.

There are many reasons why a test plan can fail. Even an absolutely perfect test plan can fail if management does not allow sufficient schedule for the test effort to verify that the product meets its quality requirements. Sadly, most test plans are far from perfect and would fail given even an infinite amount of time. These test plans fail because they only consider whether or not the product under test meets its functional requirements.

This statement may seem like heresy to many. The immediate question is, "What else should a test plan consider beyond the functional requirements?" The answer is that the test plan needs to be created in a manner that considers two huge factors that are only minimally addressed by the functional requirements. These factors are the product's end use environment and the software development environment used to develop the product.

To some extent the product's use environment is considered in creating most test plans. This consideration is usually in the form of use-case, user stories or user task descriptions that define how the user interacts with the software. Unfortunately, at best these user interaction descriptions only fully define the positive interactions of the user with the system. It is generally assumed that any user interaction not so described is forbidden or impossible which is a reasonable hope but typically false.

The roots of this problem go back to the very concept of testing to a functional specification and predictive software development methodologies. When most software was only used in a controlled environment with trained users this approach made sense. With mass market software and, especially, software implementing web functionality, there is absolutely no control of the environment and users range from simply untrained to downright hostile crackers seeking to exploit any weakness.

At a minimum, the software test plan needs to consider where the software product will be used and by whom. Further, the product should address ensuring that the execution environment (e.g., shared libraries, OS version, etc.) are compatible. As an example, the test plan for a web based software product needs to address abuse and misuse of the site as well as whether it meets its functional requirements. Anything less than this will probably result in the site being at least defaced and possibly exploited in a way that allows the exploiters to retrieve valuable user data.

This brings us to the other shortfall in most software test plans. How well the development organization can meet the stringent requirements of hack-proofing a public web site or developing a highly reliable application is highly dependent on the how well the software development organizations functions as a coherent team. Rigid, predictive software development methodologies address this issue by including management structures that provide visibility into how well the various pieces of the development effort fit together. This insight comes at the expense of having a document driven, predictive development methodology that is anything but agile,.

Agile development methodologies supposedly address this issue by continuously testing the evolving product. This works to some extent but only if the test plan takes into account the lack of formal coordination between people developing the various pieces of the product. This is a nice way of saying that the test plan still needs to exercise and test internal interfaces whether the interface be between pieces of software or between two or developers working on related pieces of the product. Unfortunately, this requires the test team having insight into the developmental product that is difficult to achieve and maintain under schedule pressure. The alternative, though, is a product with inherent vulnerabilities that, if not found by the test team, will be found by the hackers.

Testing to functional requirements only addresses one third of effort required to verify that a software product meets its requirements, is suitable for use in its intended environment and does not suffer from internal construction flaws. Only by addressing the product's intended use environment and the organizational weaknesses inherent in the development methodology as part of the test effort can the test plan really be said to verify the product. One need only look at the number of complaints users have with most software products to see that very few software development organizations address these addition facets of testing a software product.

Cheers,
Dave