You are hereA New Chore for Sisyphus / Chapter 5 - Pretending the problem doesn't exist is a really bad idea

Chapter 5 - Pretending the problem doesn't exist is a really bad idea


By DaveAtFraud - Posted on 06 July 2010

RFC 1925-8: It is more complicated than you think.
RFC 1925-9: For all resources, whatever it is, you need more.
(corollary) Every networking problem always takes longer to solve than it seems like it should.

Chapter 2 presented a laundry list of problems that occurred during the development effort described in Chapter 1. Use of these ill-considered practices were not the result of an incompetent or inexperienced development team but were driven strictly by a management demand to meet an impossible schedule. The resulting product exhibited the predicted near-term and long-term consequences. The near-term consequences of these practices resulted in the project not being even close to completion nor stable within the requested schedule. The long-term consequences meant that the product was not stable and predictable even a year later. The predicted issues with building a new version on top of this shifting and unstable foundation continued to reverberate within the development organization. Apparently simple requests for enhancements and functional changes were often met with schedules that entailed several months of effort since a re-write of the affected code would be required. Sometimes the response from the development organization to such requests was that the request simply could not be met with the existing product.

Senior management, of course, would like to blame the developers and the test team for these flaws. Surely, this fiasco couldn't be attributable to management's edict to complete the project within an impossible schedule. That these problems arose directly from attempting to solve too “big” of a problem in too little time can easily be shown. Each practice described in Chapter 2 cannot succeed and inevitably leads to the consequences described given any significant level of inherent difficulty in the project being attempted. Further, when such a project isn’t constrained by inherent difficulty, application of an appropriate agile methodology to the problem yields high quality results in as short a schedule as is possible. Given a complicated problem or a simple one, applying the non-methodology of the death march takes longer, costs more and yields a product with abysmal quality.

When management attempts to force a project to completion on an impossible schedule, the practices described in Chapter 2 are likely to occur as the developers respond to management edicts by foregoing accepted software engineering practices. To be sure, these “alternative” practices are never a good idea. Such practices will typically cause at least superficial problems for any project even if the level of inherent difficulty for the effort is relatively low. When these practices become widespread within a project and that project has a significant level of inherent difficulty, the near-term and long-term consequences are an absolutely predictable result.

With most large development efforts, there are both relatively easy and more difficult portions of the effort. Applying the practices detailed in Chapter 2 to the “easy” parts of a project gains very little (the task could have been done quickly using accepted software engineering practices). The real damage is done when such practices are applied to the difficult tasks. Since the difficult portions of the project are exactly those tasks that end up taking longer than originally hoped regardless of the project phase, this tends to be exactly what happens. These tasks remain intractable as the schedule slips and management demands that they be completed “somehow.” The very nature of the tasks that are least forgiving when attempted using the practices described in Chapter 2 causes these tasks to attract such practices as a quick fix to a slipping schedule.

Revisiting the list of sins

We now look again at each practice described in Chapter 2 and show how the consequences described are not just the result of bad luck, an inexperienced development team or lack of testing. These practices arise when too hard of a problem and too short of a schedule put the development team firmly between a rock and a hard place. There are neither resources nor, more critically, enough schedule available to “do it right.” Unfortunately for the development organization and, eventually, the users of the product, adopting these practices, whether consciously or unconsciously, does not solve the underlying inherently difficult problem.

Previously we considered the near-term and long term consequences of these practices. The near-term consequences of these practices impact the quality of the current effort while only somewhat shortening the time when the development organization is in the coding phase. Even after coding is officially done, a significant amount of code gets written in the form of patches, kludges, lash-ups and bug fixes. These get hurriedly created in order to get the product to the point that it can be inflicted on customers and the development team rallied for yet another suicidal charge to create the next release.

The long-term consequences are those that will arise after the program is released. Customers will attempt to use the ill-begotten product of this effort and the development team will attempt to enhance the functionality of the current release and create the next major release. Like Sisyphus rolling his rock up the hill only to see it roll back down again, the hill is still there as is the rock. The effort required to get the rock to the top of the hill is not reduced by putting off the task nor by hiding from it. Until the underlying inherently difficult problem is solved, the software development problem before the team will not go away. Like Sisyphus and his rock, the development team will always find itself at the bottom of the hill. If anything, the rock becomes heavier and the hill higher and steeper. With each new release, the development team now faces not only the task of implementing the new functionality of the next release but also the effort required to overcome the lasting ill effects of the previous efforts.

The following analysis will show how these consequences are the inevitable result of the practices when applied to a problem with high inherent difficulty. By way of comparison, the effects of each practice are also considered if the practice is instead applied to a project with low inherent difficulty. That most of these practices have little or no consequence when applied to projects with low inherent difficulty goes a long way toward explaining how a development team can become complacent when working on such efforts. These practices, however, ambush the same team when an inherently difficult project is on their plate. Regardless of the level of inherent difficulty of the problem at hand, these practices should always be considered completely unacceptable. The little gain achieved by these practices on projects with low inherent difficulty hardly compensates for the risks they introduce. Further, a project with low inherent difficulty can be completed more quickly and with higher quality by using an agile development methodology that doesn't flout accepted software engineering principles.

Management demands to immediately start coding

Management demands to start coding before functional specifications or at least a preliminary design have been developed are an indication that management doesn’t understand the effects of high inherent difficulty on developing a software solution. More likely, those calling for such work don't understand software engineering in general. This fallacy can best be stated as, “Code solves the problem therefore one should start coding immediately to solve the problem sooner.” At the time this code is being written, neither a full understanding of the problem nor the scheme for solving that problem exists. This lack of understanding means that code written before the functionality of the system to be created is understood will probably result in both integration problems and product consistency issues.

Besides lack of consistency in the final implementation, this lack of understanding will frequently lead to the developer making assumptions regarding input and output validation, error handling, and a host of other environmental factors. These assumptions may or may not be met when the remainder of the system gets implemented. The result is integration issues and bugs “discovered” by customers when they attempt to use the system in ways not expected by the developer but allowed once the full system is implemented. Frequently, such bugs lie latent until some future version of the system assumes that this code works or behaves in a particular fashion. It is only when the development team attempts to provide additional functionality that they to find out “the hard way” that the code does not behave as expected.

Low inherent difficulty in the project being attempted means that there will be a low level of functional coupling between program elements. Early coding on such a project may not raise consistency issues and the lack of strong data coupling means that boundary checking and data validation may not be particularly significant issues. If such issues do arise, they can usually be easily fixed since minimal functional coupling means that the problem fixes will only have a local impact. On the other hand, when applied to a problem with a high level of inherent difficulty, the potential problems will not become visible until late in the development cycle. While it is possible that the developer may omnisciently guess correctly for every design decision required to develop the early code, this is highly unlikely. The more likely result is the code will not be consistent with other portions of the program that are implemented later, after the project is better understood. At a deeper level, the developer will not have insight into how this particular portion of the program needs to interact with other program elements. He or she will be left guessing at how the other program elements expect this particular piece of the puzzle is supposed to behave. Again, these guesses may be correct in some cases but probably will not be in all cases.

If the early coding is fairly isolated or only serves to better understand the problem (e.g., it represents a prototype), chances are that no issues will arise. As management pressure increases and more code is generated prior to the overall system design being understood, the likelihood grows that the code developed will be inconsistent and not mesh well with other code and especially other early code that is also being written in the same knowledge vacuum. Apply still more management pressure due to slipping schedules as the inherent difficulty of the project remains intractable to easy solutions and it becomes less likely that this code will be treated as a “throw-away” prototype and, instead, it will be incorporated, warts and all, into the developmental baseline.

Given a project with a high level of inherent difficulty, both the near term and long term consequences spring from incorporating this code into the production baseline. The pieces of software developed in isolation will invariably not have the same look and feel as the remainder of the project while interfaces to the remaining project functionality will not be well defined or understood before a functional analysis is complete; assuming one is ever done. Likewise, lack of a functional context will mean that boundary values and the execution environment will not be well defined when the code is written. Some of these issues will be discovered during the integration phase of the product or by the documentation team. Many will not be discovered until either a user stumbles upon them or some new version of the product attempts to exercise the program in ways not anticipated by the developers.

Lack of functional requirements

The act of creating functional requirements or of collecting and documenting user stories provides invaluable insight into the expected functionality of the overall system being developed. Frequently, interrelationships are discovered that even the customer or users did not explicitly know. This is particularly true when dealing with projects that involve solving problems with a high degree of inherent difficulty. If the problem at hand is not inherently difficult, there will be only minor interrelationships and dependencies among the program elements. If the problem to be solved is inherently difficult, performing a functional analysis is how to discover these relationships and dependencies.

If the effort is not inherently difficult, agile development techniques to capture the functional requirements as “user stories” allow a development to step off at nearly the same pace as working without formal functional requirements documents. User stories provide a sufficient understanding of what needs to be built while user approved test plans provide a validation of both the product and the developer's approach. Even if the project turns out to be inherently difficult, the user stories provide a basis for understanding the functionality to be developed from the user's perspective. Analysis of these user stories and the generation of test plans from them can be used to identify any significant level of interdependence between program elements before any code is written. If this analysis reveals significant dependencies among program elements or there was an a priori assumption that the project would have a high level of inherent difficulty, a functional analysis is the means to identify and understand the program dependencies. The functional specifications that are the result of this analysis allow for an appropriate effort to be applied where such dependencies exist.

If the project has a high level of inherent difficulty and no functional analysis is done, the interdependencies and data coupling that characterize inherently difficult projects will eventually become apparent later in the project. What should be functional components will generally be, at best, only partially apparent from the system level specification but the relationship and dependencies between these components will rarely be visible at the system level. Since there is no coherent view of the product's desired functionality, it is hardly surprising that the developers instead create a patchwork that implements the individual components with little or no coordination between the pieces (such coordination would take time away from writing code). This uncoordinated development effort drives the near term consequence of an uncoordinated, inconsistent product.

Performing a functional analysis and capturing the results of that analysis in functional requirements provides significant insight into what needs to be developed in order to meet the system level requirements. This insight includes both looking inward into the functionality required to meet specific system requirements and looking outward to the user and how the user will perceive and use the system to be developed. Not performing a functional analysis on an inherently difficult project will generally result in functionality being implemented wherever it seems to conveniently fit. Convenient here doesn't just mean where the functionality can easily be added to the program. The convenient solution could also be an available developer who sees “a way” to provide the needed functionality regardless of how well it fits. Adding functionality wherever it can be made to fit means that changes to the system will rarely be confined to a single functional component. Likewise, the user will perceive this near random acretion of functionality as an inscrutable user interface to an even less understandable product.

The data coupling and interdependencies between program components of an inherently difficult project have to eventually be taken into account. The desired system is not possible unless these dependencies are implemented correctly. The result is, changes must be made to the existing code base throughout the development effort as each “new” dependency or interrelationship is discovered. In order to minimize the schedule impact of retrofitting these dependencies, these changes often are implemented as kludged patches to the design; driven by expediency and not correctness. As this baggage of patches accumulates, the design becomes more and more brittle. The hurriedly designed patches lock interdependent components into a behavior that fits neither component's design. The kludged patches and lack of design coherency between components that are the result of not bothering with functional requirements drive the long-term consequences of latent bugs and an inflexible and inconsistent implementation. The resulting brittle, inflexible product rules out achieving consistency by anything short of a ground up re-write. Also, as noted in Chapter 2, each broken interface and kludged patch required to make the system work represents a potentially exploitable security hole even if the product works correctly under nominal conditions.

Lack of preliminary design

As with the development of formal, functional requirements, creating a preliminary design for the overall project can also be skipped if the project at hand has a low inherent difficulty. Applying an appropriate agile methodology will generally allow the project to proceed with the practices of the particular methodology ensuring that the resulting product is coherent and consistent. The functional design for such projects tends to be a collection of minimally connected capabilities. Trivially, the design reflects the independence of the functional components.

For projects with high inherent difficulty, lack of a preliminary design usually results in the development team degenerating into several warring camps. Each clan attempts to justify why the rest of the project design is “wrong” and needs to be “fixed” to match their view of how the final product should work. Each clan is right and wrong at the same time since each is attempting to cast the remainder of the project into a design that fits their narrow, function-centric view of the overall project. Since there is never time for a functionally coherent design to be created, patches, splices, lash-ups and other contortions are used to make the disparate, disjointed pieces apparently work as a whole in the current product revision. The fact that these kludges and patches only paper over fundamental design differences between the different “warring camps” in the development group still shows through to the user as inconsistencies in the final product.

The long-term impact to the project is a direct result of the kludges, patches, splices and lash-ups required to get the various pieces to somehow work together. The product that results from such a development approach is brittle and fragile. Both the brittleness and fragility result from the kludges that respect neither “side’s” design but simply map one design to the other in a way that works at the moment. Attempts to change the system in later releases then break at these boundaries. The resulting product is fragile since even minor inconsistencies in the data will frequently cause breakage at these same points. The increased risk of failure is the result of having three independent pieces of software (the two functional pieces and the kludged patch that forces them to work together) that only minimally work together as a whole.

Projects with low inherent difficulty that are worked using an agile methodology will not see this issue. The need for intricate interaction between separate program functional areas isn’t present in projects with low inherent difficulty. Likewise, even the non-methodology of a death march can stumble through such a development effort. The results of the death march development approach will be far less satisfactory than those that could be achieved using an appropriate agile methodology. There is even some small probability that project may still be completed in spite of the non-methodology used.

Lack of detailed design

Detailed design is the point in program development where functional requirements and preliminary design are translated into specific, implementable algorithms suitable for coding. This applies to the executable code that will define the logical operation of the system and to the interfaces implemented within this code between the various parts of the system. It is only during detailed design that many of the issues that constrain an inherently difficult development effort are fully resolved. Interfaces are defined, any complex algorithms are identified and designed and the interaction between specific program elements finalized.

For projects with low inherent difficulty using an agile methodology, individual developers or developer teams accomplish this step with minimal or only informal interaction required with other developers or developer teams. A low level of inherent difficulty allows the system level detailed design to be skipped although individual developers will still need to create a detailed program design for their assigned components. Even under agile development methodologies, stream-of-consciousness coding (coding without a detailed, program design) leads to a poor implementation. As with skipping the preliminary design, the non-methodology of a death march may not be severely impacted by not creating a system level detailed design if the problem at hand has a sufficiently low level of inherent difficulty. On the other hand, superior results will be achieved in less time by applying a suitable agile methodology.

If an informal level of interaction and coordination is not sufficient due to the inherent difficulty of the project, the project is probably not suitable for development using an agile methodology nor is it suitable for a death march. In either case, if the project attempts to continue without settling the detailed interfaces and interactions required, these issues will show up in the current effort as integration difficulties. Testing and debugging techniques will inefficiently and incompletely take the place of the skipped design effort. For the current release, kludged lash-ups and patches will again be required to make the otherwise incompatible pieces work together. If the schedule pressure is sufficient, such patches, kludges and lash-ups will be developed to only meet the narrowest interpretation of the requirements for the current effort. This saves time for the current effort but leaves the task of correctly mapping the mismatched abstractions onto each other for some later release. The resulting fragility and brittleness directly lead to the long-term consequences of an unstable, easily broken and virtually unmaintainable system. Any lack of completeness in the patch code will eventually show up in later versions where aspects of one or both abstractions will be left dangling.

Lack of common abstract objects

Abstract objects allow programmers and eventually end users to work with program entities as if they were the corresponding real-world objects. This means that the underlying implementation of the object is masked from everyone except those actually concerned with implementing the object. Other developers and possibly the end user only see an abstract interface that presents the set of legal actions that can be requested. This may sound simple but the technique is very powerful. It means that program correctness can be much more easily determined. With an abstract object, the details of an object's implementation need only be understood and verified by the programmers developing the object. The remainder of the program and programmers work only with the abstraction as presented by the developer defined interface. By not defining common abstract objects, many more parts of a program have to deal with details of the raw data or the underlying system. Rather than implementing such details once, the implementation is repeated by multiple programmers and, chances are, each will do it in their own way.

If the problem has a low inherent difficulty, there will be no significant, common, abstract objects and this problem will not arise. When the project is inherently difficult, the creation of abstract objects allows complex parts of the system to be hidden from all but those who must implement each such object. The price to be paid for achieving this simplification of the system implementation is the time it takes for developers to understand both their piece of the puzzle and the needs of the other developers. The developers of each object must not only take the time to understand how to implement their object but also understand the needs of any other components that will interface with their object (although such needs should have been documented in the functional requirements).

When a development team implementing an inherently difficult project plows ahead without agreed to, common, abstract objects, the near term consequence is the system takes longer to integrate. Many parts of the system will typically each have to deal with raw or unabstracted data without the benefit of having an abstract interface to encapsulate it. Each such instance becomes a potential source of errors making the current development effort more difficult. Abstract objects whether common or which implement a unique capability also provide easily testable components. Such components allow the integration team to rapidly ascertain that specific components are or are not behaving correctly.

Common, abstract objects provide an additional benefit in the form of well understood, self-contained reusable components on which future development can be based. Without common, abstract objects, future development is saddled with understanding the intricate interactions that were defined, as needed, to meet some previous iteration's requirements. Even if done correctly, the direct interactions that characterize an interface that does not employ abstractions will still be a rat's nest of inscrutable logic. This situation will be made much worse if many of the other ill practices described herein are also followed with the result that the required logic is replicated at multiple locations in the code. Typically, such logic will only be understandable by analyzing the source code and any comments the developers deigned to provide and hopefully maintained through the crush of the development effort. Such quagmires of dense code represent a major obstacle to any attempt to extend the system's functionality for a future release.

Continual or frequent “refactoring”

Refactoring, as noted previously, is the agile development equivalent of a re-write. Agile methodologies emphasize two guiding principles that make such refactorings inevitable: “keep it simple” and “you aren't gonna need it” (YAGNI). When properly implemented in an agile methodology, these underlying principles mean that each iteration contains only the functionality that it requires without overly complicating a given iteration by partially implementing functionality that may be required in a later iteration. The affected code is then refactored in a later iteration to implement any additional functionality only when it is actually needed.

Whether the development team is actually following an agile methodology or only using agile terminology to describe their efforts, continual or frequent refactoring of the same functionality without any apparent convergence on a final, workable implementation indicates that the functionality in question probably needs to at least be rethought. For projects with low inherent difficulty, at worst, the functionality to be implemented is volatile and the multiple refactorings are required to allow the user to experience different approaches to solving the problem. In this case, hopefully, the users will settle on a particular approach and the project can move on. The possibility also exists that this functionality may represent an aspect of the project that is inherently difficult. Unfortunately, by the time the project has reached the stage of multiple refactorings of a particular component, it is somewhat late in the game to suddenly decide that additional functional analysis of the affected functional component or the entire system is required. Had the development team taken the time to develop functional requirements and a preliminary design, the inherent difficulty of the particular component would have been evident and alternative approaches to meeting the system level requirements could have been explored.

The near term consequences are borne by the integration team and the schedule as the repeated, non-converging refactorings mean that the project remains unstable. If the functionality being refactored affects multiple pieces of the product (that is, it implements an inherently difficult aspect of the project), the entire project integration schedule is delayed. In this case, the inherent difficulty of the project means that there is little likelihood that a single developer can somehow code his or her way to a local solution that solves multiple interdependencies. At this point, the development team has little or no choice but to implement a sub-optimal solution which generally means that one of the multiple iterations that seems to cause the least damage and/or provides the most functionality will be inflicted on the users.

The long-term consequences only arise if some of these individual refactorings are released to the end users. As noted previously, if the need for multiple refactorings is driven by requirements volatility, this is the only way such a problem can be solved. It will result in changes to the user documentation but it is preferable to saddling the users with a function that doesn't do what they need. If the multiple refactorings are a result of the project attempting to code their way through an inherently difficult project, an ever-changing product confronts the users and forces them to re-learn how to deal with a new implementation at each product release. If the new implementation means that some level of significant data conversion will be required to maintain compatibility with previous releases, the impact on the development team is even worse since this represents additional functionality to be developed and tested. Finally, the problem will continue from product release to product release until the team takes the time to engineer the complex component. It is possible that some randomly released refactoring will provide the requisite solution. Unfortunately, since there is no functional analysis to support this conclusion, the development team is just as likely to continue substituting other refactorings.

Cut and paste sub-classing

Cut and paste sub-classing is really just a result of the development team surrendering to the inevitable as the consequences of no functional requirements, no preliminary design, no detailed design and no identification of common, abstract objects comes to a head during coding. Common functionality that should have been identified and designed prior to coding is now identified as non-reusable components that are copied and modified to meet each new use.

If the project has a low level of inherent difficulty, the need for such an activity would be minimal or non-existent. There would be little overlap or common functionality among the project components that would drive the need to replicate functionality. Given a project with high inherent difficulty, the team only stumbles over such common functionality as the implementation progresses. At best, they may be able to leverage a previously written component through real sub-classing to ease the development stress. More likely, they only come to the realization that several pieces of the project should have similar functionality during coding and end up with a “stretch-to-fit” implementation. Each new use results in another local copy of the base code or some previously copied branch. This new copy is then hacked to meet the new use. If the common functionality represents some aspect of the inherent difficulty of the project, the various copies of the code will eventually have to be made to behave the same while still meeting the unique, local requirements of each copy.

The real cost of cut-and-paste sub-classing first shows up during the integration and test phase of the current effort. The test team is forced to deal independently with each instance of the copied functionality. This effort will be even harder if there are no functional requirements to guide the test team as to what each instance should be doing and the extent to which each instance should either behave the same or reflect unique functional requirements. The unique capabilities and quirks of each copy are discovered and the test team is faced with resolving whether the uniqueness is specific to a particular instance or should be common to all instances. Likewise, finding and documenting a bug in any one instance provides no information as to whether the same bug exists in any of the other instances.

If the development team attempts to implement a follow-on release of the product, the long term consequences are even worse. Modifications to what should have been common functionality will need to be implemented in each instance of the copied functionality. Alternatively, the development team may attempt to “do the right thing” and actually collapse the various copies of the code into a single implementation. This may or may not work depending on what level of unique functionality the various copies have acquired prior to the unification effort. A valid functional design implemented through common, abstract objects would have prevented this. Functional analysis would have allowed the development team to put the unique characteristics of each usage into proper perspective.

Unrealistic unit testing

Without functional requirements all unit testing can accomplish is to show that a piece of code does what the developer intended. Since there is no authority in the form of functional requirements to define what this behavior needs to be, whatever the developer had in mind while coding must be correct1. Further, lack of abstract objects and a detailed design may mean that any data dependencies are defined to be whatever the individual developer decides they should be. Developers may informally coordinate their efforts but sufficient schedule pressure means that such coordination will be done only on a time available basis rather than as a prerequisite to development.

If the project is not inherently difficult, such dependencies tend to be internal to the developers' code and each developer will bear the consequences of any false assumptions. When a project is inherently difficult, both internal and external dependencies will exist. If the program was hastily designed, these dependencies will be large, intricate and deep as opposed to the small, clean and tight of a well-done design. For the current product development effort, the result is the integration team will be forced to attempt to explore these intricacies during testing but with little or no visibility into the underlying implementation or any functional requirements. The end result will be a system that, at best, is known to work for only a limited range of data values and relationships. Typically, the system will behave erratically for other data values or any change to the execution environment. This lack of predictable behavior across varying inputs gives rise to the long-term consequences. Eventually, another version of the system will be created and the developers of that version will assume that the behavior of the individual pieces will remain stable. This assumption is most likely wrong.

Hide the problem

Several of the practices listed above have significant negative near term as well as long term consequences. These consequences are far more intractable when the underlying problem to be solved has a high level of inherent difficulty. The worst situation occurs when the development team is unable to persuade management that a particular piece of the problem needs a greater effort than the current schedule allows but the test team is able to demonstrate that a real problem exists within the implementation. At this point, the only alternative is to come up with a means of masking the problem from the end user. This may take the form of restarting some part of or the entire product up to and including a reboot of the entire system.

Hiding the problem doesn't work regardless of the methodology. At some point, the underlying problem will have to be addressed. If the problem being worked is not inherently difficult, the likely cause of the problem can usually be isolated and an appropriate fix implemented. When the problem being hidden is just a manifestation of some aspect of the inherent difficulty of the overall problem, the solution will not be so easy. Any of the ill-practices described above can result in instabilities that can only be fixed by developing the system in a manner that recognizes the inherent difficulty of the task at hand. When such instabilities are discovered by the test team late in the development cycle (i.e., after coding is supposedly done), there is rarely an option of re-doing the development.

One apparent option is to mask the problem by hiding the most visible consequences from the user. This may take the form of denying the user access to badly behaved data that should be accessible or restarting some or all of the product to contain memory or resource leaks. The near term consequence is instability in the product since additional cases triggering the problem will eventually be discovered by testers, other developers and eventually users or customers. Long term, the underlying cause of the problem must eventually be found or the product will continue to be unstable in future releases. There is also the possibility that the specific data that causes the instability is of interest to the user and the user will not be pleased to have the system intentionally hide what they want to see.

Lack of user interface design/consistency

The underlying inherent complexity of a product is, to some extent, reflected in the user interface. Programs with low inherent complexity tend to either have a single, simple interface or there is at most only loose coupling between the various interface elements. Under such circumstances, developing a “use-case” or “user story” as to how the user would utilize the functionality and then applying a user interface style guide is sufficient to ensure the resulting product will have as much of a common “look and feel” as is possible. The user interface to such a system is, more or less, a collection of at most loosely connected functionality.

As with the program itself, the user interface for an inherently complex program tends to have multiple, related facets which need to reflect a coordinated idiom for the underlying program to appear as a coherent whole to the user. Forcing the development of such an interface onto too short of a schedule to allow such a coordinated idiom for the interface to be developed directly leads to a jumbled, disorganized interface. Typically, this interface design reflects each developer's unique view of the overall project. This view is obviously through eyes that have become adjusted to only seeing a single, myopic view of the problem.

For the current effort, such an interface means additional work for the documentation and test teams. In order to document the program, the documentation team must attempt to create a coherent view of the program by tying together the functionality of the disjointed elements. Since this view is based on the idiosyncrasies of the project as implemented, changes to the interface and the underlying functionality to fix any of the problems previously described then impact the documentation. This impact goes beyond just requiring a change to the description of say how to use a particular screen since the documentation attempts to provide a coherent view of the product including the idiosyncrasies2. The test team sees the issue the other way around. Idiosyncrasies in the interface result in either problem reports or the need to change test plans. Any test automation that reflects what a particular screen actually does or how it actually behaves also must be changed.

Longer-term issues arise as the customers attempt to use the product. The lack of a coherent interface means that the user will need highly accurate documentation, which is unlikely given the effect of the near term issues on the documentation team. If the product documentation is not sufficient, the user will need to continually contact the product support team to get answers to questions, work-arounds for the quirks that remain, etc. If the problem is severe enough, customers will look for an alternative product.

Extreme focus on only meeting the minimal interpretation of the requirements

The purpose of requirements is to specify what must be built in order to fulfill the user's needs. Since the product has not been built at the time the requirements are written, the requirements obviously must leave a large amount of detail to interpretation by only specifying what must be done without going into how it should be accomplished since this would constitute design. The point of performing requirements analysis and creating a preliminary design is to validate the completeness of the functional requirements. The customer and the developer can then agree that a program built as specified by the functional requirements and according to the preliminary design will meet the user's needs. If the absolute minimal interpretation of the functional requirements or, even worse, the system requirements is used, the user will be saddled with, at best, a very incomplete program and, at worst, a program that doesn't work.

If the project has a low level of inherent difficulty, this will result in a collection of partial solutions. Each such solution may be complete in the sense that it handles a particular subset of the possible inputs but there will be cases or conditions that are not handled. If an agile methodology is being used for the development effort, this just means that either additional iterations are needed, or, as will be discussed later, a methodology such as DSDM will only implement the highest priority functions. The solution is incomplete but the project still provides some level of functionality.

While it is possible to design and code to an incomplete vision of the specified product, eventually the test team will attempt to test that product and the documentation team will attempt to explain how to use the program to the user. If the project has a high level of inherent difficulty, the incomplete implementation will become manifest as the disjointed functionality will neither allow test plans to be written nor documentation to be created. The near term impact runs the gamut of the problems described above depending on how deeply the incomplete implementation affects the project.

At some point, the test team, the documentation team and the customer demand that the product actually work. Bug reports or engineering change requests are used to inefficiently connect up the disparate parts of the product. Recognizing these issues as the product nears completion means that the same patch and kludge approach to achieving a functional product as described for other ills will be the only approach to fixing such problems that the schedule allows. This means that the resulting final product will again be brittle and fragile with all of the implications for future development that such characteristics imply.

Attempts to test quality into the code

Programs with low inherent difficulty can be effectively tested by developers since the low data and functional coupling characteristic of these projects means that each piece of the program virtually acts as a standalone unit. The issues that occur are generally confined to a localized portion of the program and can be easily isolated and fixed. For programs with high inherent difficulty, finding design and requirements level issues during test means that there is very little choice as to how to fix the problem. Worse, if the problem reflects the inherent difficulty of the project being attempted, there is little likelihood that the problem can be easily or correctly fixed.

Given an inherently difficult project that is implemented using the above practices, there is pitifully little chance that any amount of testing will allow the issues and latent bugs that result to be found; let alone fixed. For the most part, testing is unsuitable for discovering design and requirements level bugs. This unsuitability of testing for finding these bugs is what gives rise to the adage that, “You can't test quality into code.” The deep rooted nature of the bugs, instabilities, and inconsistencies that result from the above practices will continue to plague users regardless of the test resources expended. The same schedule pressure that resulted in those practices will ensure that only the most obvious, trivial and superficial bugs will be found during testing.

Attempts to test quality into the code will generally show up as a need for more and more testers since the number of problems being found does not decrease as each attempt to fix a given problem only gives rise to new problems. Typically, management will insist on superficial testing of the user interface while attempting to ignore the implied and even explicitly visible problems with the underlying implementation. Testing can, at best, find implementation errors. If the project has a high level of inherent difficulty, design and requirements errors such as those described above go much deeper than simple programming errors. Attempts to fix them generally just moves the manifestation of the design or requirements inconsistency elsewhere in the program.

As noted previously, finding and attempting to quickly fix such problems late in the game means that only a patch and kludge approach has any hope of meeting the schedule. The design and requirement errors are still present in the code base even if kludged fix-ups and work-arounds temporarily hide them. These issues eventually show up as user reported problems or, worse, security holes. At some point, a new version of the product will be attempted and the underlying inconsistencies will have to be addressed or sufficiently worked around. If the work-around approach is taken, the next version's design must be significantly “adjusted” to not aggravate the design issues latent in the previous version.

Attempts to automate testing of functionality still being developed

As noted in Chapter 2, automated testing of a stable product can provide significant efficiencies and much more in-depth and repeatable testing than can be accomplished by using strictly manual methods. As a fall-out from the previous item, management will sometimes assume that more testing (as measured strictly by CPU cycles utilized) will somehow result in a higher quality product. The only way of achieving such additional testing without impacting the schedule is to automate test execution. The obvious fallacy in applying this approach to a project developed using the ill practices described above is that the continually changing product is unsuited to test automation. Automated testing is best suited to test activities such as repeatable regression testing and exercising all of the possible combinations and permutations of input values allowed through a complicated interface. Such test automation is only feasible once there is a reasonable probability that the interfaces under test are stable and function correctly under nominal usage. The value of such automated testing increases further once there is some reasonable possibility that the underlying logic that supports the interface may also function correctly.

If the project has a low level of inherent difficulty, there is a reasonable probability that relying heavily on automated testing will be valid due to the “stove-pipe” nature of the overall project. If the project has a high level of inherent difficulty, the most important bugs to be found lie deep within the product where off-the-shelf, automated test tools are unlikely to be of much use. If the project has a high level of inherent difficulty, the result of this approach is to divert test personnel from finding such “deep” bugs and instead diverts their attention to creating test automation scripts which only verify that the current incarnation of user interface behaves as expected. If the logic that underlies the interface content is still unstable as is likely if the project has a high level of inherent difficulty, this effort is at best trivial but more likely totally useless. The question is not whether the display works but whether the functionality that underlies the display is correct.

The diversion of test resources to develop test automation is generally wasted due to the rapidly changing nature of a product being developed on a compressed schedule (many changes are required to fix the bugs that result from the above practices if nothing else). The test automation that is laboriously developed is rarely of use for more than a small part of only the current effort. To the extent that a false sense of security is developed through the use of test automation, there will be a tendency for more of the deeper problems to reach the end user. This is especially true if the testers writing the test automation scripts discover that their scripts have a longer life time if they only exercise the system using “safe” values and settings.

Proliferation of patch releases

The ultimate result of all of the above practices can be summarized as the user ends up with a buggy, unstable, brittle and fragile product. If the project has a low level of inherent difficulty, it may be possible to attack the bugs individually or, at worst, in small groupings since it is unlikely that many of the bugs are related. Thus, the bugs can be addressed as they are found and there is a good likelihood that the project will stabilize and the irritants will be removed. For projects with high inherent difficulty, the opposite is true. The origin of these bugs often tracks to multiple pieces of apparently unrelated and supposedly working code. Fixing these bugs involves solving the design issues that were not addressed in the initial haste of the project. Since the user needs a working program, the only option is to release patches that attempt to put a band-aid on each problem.

For inherently difficult projects, the ultimate origin of these problems lies in the lack of system level design for the overall project. If this is the case, quick fixes tend to just move the problem elsewhere with the result that yet another patch is eventually required. If the origin of the problem goes to some unresolved design or requirements issue then the problem can only be fully addressed by actually redesigning the program correctly and re-implementing it. The end user pays for the early short term thinking by being asked to put up with an intermittently broken product and a continuing stream of patches and updates that purport to solve the problem but which, in the end, only move the problem to a different location within the program.

Omens

Unlike Sisyphus, a development team can at almost any time in the development process decide that the current effort cannot be accomplished within the requested schedule. As the project progresses, the signs that the project has a high level of inherent difficulty become more and more obvious if the development team only decides to not ignore these signs. At the project phase of each of these ill practices, an honest evaluation of the situation would reveal the dangling interfaces and required complex interactions that characterize a development effort for an inherently difficult project. The alternative is to follow the course of the project described in Chapter 1 with death march developers using agile development terminology to describe their efforts right down to the multiple, non-converging “refactorings” that never ended.

Appearances are not reality

If the problem to be solved is complex, imposing a flat organizational structure (e.g., the agile methodology “bull-pen”) and assuming a “stovepipe” program design only hides the complexity of the actual problem from management view. The complexity of the product does not suddenly go away simply because there is no organizational expression of the complexity and management decrees that each piece can/should/must be developed in near isolation. In this circumstance, developers will, naturally congregate into groups that correspond to a reasonable functional decomposition of the project. However, the lack of an organizational structure that corresponds to this decomposition hides critical elements such as design interfaces from management oversight. Given sufficient schedule pressure and an inherently difficult problem, design consistency issues will eventually show up as integration difficulties as the informal interface agreements beget numerous “square peg/round hole” type problems. Under these conditions, the pieces developed by different individuals simply don’t fit together. We can also expect all of the other systemic problems described in Chapter 2.


1Problems are usually met with the response that the program “works as designed.”

2This effect may be part of the momentum behind the “Information Mapping” approach to software documentation. With information mapping, there is an explanation of how to use each control of every interface element but there is no attempt to tie together the capabilities of the system. The real problem of how to use the system is left to the user. This may be appropriate for office automation tools such as word processors and spreadsheets but is woefully inadequate for special purpose applications.


This work is copyrighted by David G. Miller, and is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.