You are hereA New Chore for Sisyphus / Chapter 11 – Summary and Conclusions
Chapter 11 – Summary and Conclusions
Engineer's Lament: There's never time to do it right. There's always time to do it over.
As the inherent difficulty of the problem increases, there is a corresponding decrease in the ability of end-item testing to verify correct operation of software that purportedly solves the problem. This is the crux of the problem for all efforts to accelerate software development. By the time a program that attempts to solve an inherently difficult problem gets to the quality assurance team, the process used to create it has already established its fundamental level of quality. End-item testing can reveal some of the latent problems but will rarely shed any light on misunderstood requirements, bad design decisions or poor software engineering techniques. Worse, these problems only rarely show up as overt bugs. If they do, it is only as product inconsistencies, hard to reproduce instabilities, and quirky or inconsistent responses. On a schedule constrained development effort, a patch or quick-fix will be the only option that the schedule allows for addressing such problems.
Agile development methodologies achieve their fast paced development at the expense of being unsuitable for solving large, complex problems. Tools are limited to only helping write the pieces of code that become the solution to the problem. Tools also assist with ancillary tasks such as managing those pieces and superficially testing the resulting product. Likewise, off-the-shelf solutions may solve pieces of a problem but there is much art required in successfully putting together these pieces to solve even part of an inherently difficult problem. Finally, outsourcing all or part of the development effort does nothing to reduce the inherent difficulty of the problem to be solved. The time it takes to discover the solution to the problem is still the gating constraint on how fast the solution can be developed. Making the creation of this solution somebody else's problem, if anything, makes creating it more difficult and may result in irreparable harm to the in-house development organization as a result of the knowledge and skills that move to the outsourcer.
The minimum schedule required for a software project is determined by three factors. The first two factors reflect the nature of the problem itself. These are the problem's size and complexity. Of these two factors, project size has historically been the most important. The complexity of the problem now is also a constraining factor since the inherent difficulty of a project may also exclude the use of an agile methodology to rapidly develop the solution. Likewise, the inherent difficulty of the project restricts how much can be gained by the use of software tools or outsourcing some or all of the development effort. The final factor determining a project's schedule is the ability of the group assigned to solve the problem. Although the experience and intellect of the group are factors that somewhat determine this ability, the nature of human cognition and the ability of the group as a whole to analyze the problem and discover and validate a proposed solution are actually the critical constraints on how fast this process can occur.
Of these three factors, only the size and complexity of the problem is even partially within the control of the development group. Only by reducing the size and complexity of the project being attempted can the required effort and schedule be reduced. This may seem strange since it would seem that additional personnel could easily be hired to increase the team's problem solving ability. Unfortunately, the functional analysis that is required for dividing a large, inherently difficult problem into coherent pieces that can be worked in parallel is rarely amenable to simply adding more workers. Solving the interdependencies of such a problem becomes more difficult as more people attempt to concurrently work the problem. The constraint here is the ability of a group of people to work together to solve a problem none of them completely understands. The only way to significantly reduce the time that is required for this effort is to reduce the complexity of the problem to be solved.
It is not just the time required to perform a functional decomposition for large, complex problems that causes their schedule to be longer then desired. If complex interdependencies remain in the solution even after an optimal functional decomposition is achieved, the problem will require the rigor and schedule of a predictive methodology throughout the development effort. The only problem is that following such a methodology means that the project, as originally defined, will take far longer than is desirable. Again, the only way to significantly reduce the schedule required for the development effort is to reduce the size and complexity of the program to be developed.
Irresistible force meets immovable object
Software schedules are somewhat malleable by increasing or decreasing staffing levels. This seems to have been extrapolated to a belief that any schedule is possible if only enough manpower is applied. This belief was known to be false over thirty years ago when Fred Brooks compiled the original edition of The Mythical Man-Month. Additional data from 63 projects included in the 20th anniversary edition of The Mythical Man-Month support the original assertion with hard data1. As the project described in Chapter 1 illustrated, ignoring this fact will rarely yield acceptable results. The project will not be completed within the demanded schedule and attempting to force it to completion sooner than is possible will result in deep, systemic problems. Unfortunately for both software developers and software users, management continues to ignore this constraint.
Market pressure, higher management pressure, investor pressure, and bonuses tied to “performance” contribute to the continued lack of acceptance of this constraint. To some extent this is due to another fallacious assumption. This is the assumption that all software bugs can be fixed by finding some relatively small, erroneous section of code and rewriting that code so it no longer exhibits the problem. That is, there is a belief that all bugs are shallow2. Extreme schedule pressure and excessive over-staffing are known to result in significantly more bugs per line of code. This would not be significant if all bugs were shallow. Unfortunately, some bugs affect the usability, maintainability, stability, security and reliability of a product. Worse, none of these characteristics of a product are amenable to improving through superficial, end-item testing. While even a very small problem can keep a program from operating, the visible consequences of a bug are by no means an indication of how hard it is to fix the root problem behind the bug.
All software problems are not shallow. The practices described in Chapter 2 created problems that adversely affected the core functionality, usability, maintainability, stability, security and reliability of the product. A death march has no methodological safeguards to prevent these practices but they can also occur on any project following any software development methodology. All that is required is an inherently difficult problem and sufficient schedule pressure to bring about the practices described in Chapter 2. Unfortunately, all software development methodologies are vulnerable to the extreme schedule pressure that can give rise to these practices. That is, all methodologies break down under sufficient schedule pressure. Even the most rigorous predictive methodology can be followed superficially so that the requisite work products are apparently produced but the engineering behind these artifacts isn't performed. Worse, in many organizations, such schedule pressure creates a “foxes guarding the hen-house” situation since the same management that is demanding impossible completion times is also charged with enforcing the development methodology3.
An inherently difficult problem can create the “perfect storm” for a software development project. A problem that cannot be quickly solved confronted by a development team that is under the gun to provide such a quick solution. Either the schedule constraint must be relaxed to allow sufficient time to develop the functionality or the scope and complexity of the functionality must be reduced to the point that the development can be completed within the available schedule. If neither constraint is relaxed, the only alternative is to remove functionality by accident. This may be in the form of functionality that exists but can't be executed, functionality that only works for restricted, nominal cases, or system instabilities that mean the product's functionality is only available some of the time. In any case, the product has less than the full functionality requested. There is also the possibility that the software development practices adopted in order to meet the schedule will introduce functionality that wasn't intended. This “unintended functionality” can sometimes be exploited to subvert the system.
But there's more!
Given enough testing, bugs at the level of coding errors can be found and fixed. The fate of the Mars Climate Orbiter indicates that finding and fixing coding errors only after they cause a problem is not a viable approach for some projects. A commercial software vendor may even be able to get away with this approach given a software product with few competitive alternatives. Unfortunately, the extent of the problems created by forcing a project onto too short of a schedule go far beyond simple coding errors. The bad design decisions, lack of problem analysis, and near-term shortcuts that neglect long-term considerations result in the creation of an unstable foundation. Future versions will face increased and accumulating difficulties that will make future schedules either longer or even less attainable.
Systemic errors, instabilities and security holes are the immediate cost of an overly aggressive development schedule. The secondary effects are a degraded ability of the development organization to create quality products in the future. Some employees will see the futility in this approach and will leave if they can find a viable employment alternative. Those who remain will resign themselves to slapping together a product that they can't be proud of. Their approach to future development becomes, “What can we get away with shipping?” Hardly the attitude that will result in a world class product.
The following items enumerate the critical conclusions reached at various points throughout this book:
- The minimum schedule duration required for any software development projects is constrained by the inherent difficulty of the problem to be solved. Inherent difficulty is a measure of the complexity and level of interdependence of the problem to be solved. Large, complex, highly interdependent problems cannot be solved quickly. It may be possible to solve small, complex, highly interdependent problems relatively quickly. The ability to solve such problems is highly dependent on both the problem itself and the specific individuals tasked with solving the problem. This dependency on the individual team members and the problem they are faced with makes this ability very unpredictable. Also, this approach does not scale up for larger problems.
- The death march as a software development methodology is bankrupt. If the problem to be solved is not inherently difficult, it can be solved more quickly and a higher quality product produced if the development organization applies an appropriate agile methodology instead. If the problem is inherently difficult, the death march approach will take longer, cost more and, at best, produce a deeply flawed product as compared to applying an appropriate methodology that can effectively solve such a problem.
- A death march development effort does not just result in a product with numerous superficial errors. If the problem to be solved is inherently difficult, the software development practices that typically occur during such an effort will result in deeply rooted systemic problems that will lengthen the schedule for the current effort and continue to slow future development. In addition, such practices undermine the stability and security of the product. The ingredients that result in such practices are an inherently difficult problem to be solved and extreme schedule pressure. Thus, these practices can occur under any software development methodology when sufficient schedule pressure is applied to a team faced with an inherently difficult problem.
- Many of the flaws that are apparent in a range of current software products probably have occurred due to the development organization ignoring the previous item.
- Agile methodologies can produce a high quality product in a relatively short amount of time so long as they are only applied to problems with a low level of inherent difficulty. High inherent difficulty constrains agile development methodologies because it requires that the development team collaboratively and concurrently solve a complex, interdependent problem. Agile development methodologies do not include the engineering infrastructure or activities required for solving such problems.
- In order to solve inherently difficult problems, the development team must reach a shared understanding of how to solve a complex, interdependent problem. Tools are unlikely to significantly reduce the time required for this activity since tools can, at most, facilitate communication. They are no substitute for collaborative problem solving ability.
- Outsourcing part or all of an inherently difficult problem only transfers responsibility for solving that portion of the problem. The difficulty of the overall problem is unchanged. The time required to solve the problem is still constrained by how long it will take the development team to create a solution to a complex, interdependent problem. To the extent that creating this solution will require open and free communication with the outsourcer, outsourcing part of the effort may actually extend the schedule.
- The same property that causes outsourcing or other methods of “bringing in” the schedule not to work can be turned into a significant competitive advantage. Building and growing a development team that knows how to solve inherently difficult problems for a particular problem domain is a process that takes time. Once such a team exists, they can solve problems chosen from their domain of expertise as fast and as well as such problems can be solved. This capability becomes a competitive advantage since the developer of any competing product is faced with attaining this same level of expertise in order to be competitive.
- The difficulty of a problem need not be evenly distributed over the software that solves that problem. This means that the inherently difficult portions of a problem can be collected and architecturally isolated. This approach allows a careful, planned approach to be applied to developing the inherently difficult portions of a project while the remainder of the project has a large degree of flexibility in choosing a development methodology. Regardless of whether a different development methodology is used for developing the portions of the project that aren't inherently difficult, a shorter development cycle is feasible for these portions of the project.
- The engineering analysis required for this approach is not wasted regardless of whether the problem is or is not found to be inherently difficult. If the project is not inherently difficult, the analysis removes significant risk and, to the extent that the problem still requires some level of coordination between program elements, provides early visibility into these critical dependencies. If the project is inherently difficult, this analysis is the first step in collecting and architecturally isolating the complex, interdependent logic. Development of the program components that solve this portion of the problem will need the problem solving capabilities of a predictive development methodology for their entire existence within the software life cycle.
- It is possible to practice a predictive development methodology that does not include “high ceremony” design reviews. The level of ceremony and the formality of published artifacts from a predictive methodology can be tailored to fit development organization and customer requirements. What is needed for most development efforts is understanding and agreement; not piles of paper. The piles of paper and formal reviews associated with predictive development methodologies are artifacts required by specific customers and are not central to predictive methodologies.
If an event such as shipping a product later than desired cannot be avoided, it makes sense to manage the situation to minimize the damage. Most software development organizations instead stick their collective heads firmly in the sand and proceed as if the inevitable won't happen. Something that sort of resembles the desired product is eventually created but often only after taking longer than the estimate for doing it “right.” Further, this product has both superficial and deep-rooted problems that mean it has far less than the purported functionality. That is, the approach most software development organizations take when confronted by a development problem that will take longer than desired maximizes the damage done by creating a poor quality product later than desired. So much for managing the situation.
1Citation of research by Dr. Barry Boehm for his work Software Engineering Economics in The Mythical Man-Month (Page 273-4).
1The open source development assertion that, “Many eyes means that all bugs are shallow,” is patently false. A glaring example was the controversy over the virtual memory model for the 2.4 Linux kernel. This is an example of a design problem that could only be corrected by an extensive re-write. The existence of the problem may have been easily deduced. On the other hand, the effort required to fix the problem and verify the result was far from “shallow.” A more accurate statement would be, “Many eyes mean that coding errors are easily found, fixed and verified.”
1This is one of the reasons measures of a software organization's maturity such as the Software Capability Maturity Model (SCMM) insist that the quality assurance organization be independent from project management.
This work is copyrighted by David G. Miller, and is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.