Author Biography:

In addition to my formal education (BS and MS from Ohio State in Mathematics), my twelve years in various software development roles at TRW in the 1980s and early 1990s provided me with the equivalent of a post-graduate education in practical software engineering. Rather than continue in management after leaving TRW, I have continued to work in or near the software development “trenches” and have held up my experience in small and medium scale commercial software development efforts to what I learned while at TRW. On encountering a real-life death march at my previous employer, I realized that there was a need to show that the effects of such an effort go far beyond just a burnt-out development team with A New Chore for Sisyphus being the ultimate result.

The introduction to A New Chore for Sisyphus provides additional explanation as to how I came to write this book.

Topic Description:

A New Chore for Sisyphus is a pragmatic and somewhat sarcastic analysis of why attempting to force a software development project onto too short of a schedule not only does not produce the desired product within the specified time but also results in deep, systemic problems that will continue to haunt both the development team and the users of the product. This takes the form of dissecting just such an effort I was involved with and then draws generalized lessons that are applicable to all attempts to significantly accelerate a software development effort.

Market pressures drive software development organizations to attempt larger and more complex development efforts on ever shorter schedules. In the end, not only is the schedule not met but the product is riddled with systemic bugs that affect both its quality and the ability of the development team to build the next version. Patches are applied and an attempt is made to sort out the mess but the next version is already late and so the development team members put their collective shoulders to that task. Unfortunately, like Sisyphus and his rock with each new version they keep finding themselves at the bottom the hill. Worse than Sisyphus's punishment, each time the hill becomes higher and steeper and the rock heavier for the developers since they are faced with new requirements, a need to make up for the previous late release and the burden of the accumulating problems.

Attempts to point out that methodologies exist for engineering larger efforts are met with the derisive response, “There isn't time for that.” There seems to be a common set of fallacious assumptions behind this assertion. Foremost among these fallacies is that the converse of Parkinson's Law (work expands to fill the time available) is both true and can be applied to a software development effort. This results in abbreviated schedules and the belief that any bugs that result will be evenly distributed and easy to fix. That all bugs are easy to fix is the second critical fallacy as many of the practices unconsciously adopted by developers under extreme schedule pressure result in deep-rooted, system problems that adversely affect a software product's reliability, maintainability, stability and security.

Based on these two fallacies, software development organizations continue to attempt to develop too much functionality in too little time. What seems to be lacking is a chain of causality that directly links the practices brought about by extreme schedule pressure to the difficulties experienced later in the current development effort. Likewise, there is no acknowledged link between these practices and the problems that will be encountered when the development team attempts to build the next version of the product. Sisyphus fills this gap by providing this chain of causality. This direct linkage of (mal-)practices to results under these circumstances is unique to A New Chore for Sisyphus.

To a large extent Sisyphus “connects the dots” between The Mythical Man-Month and Peopleware. That is, the personnel practices that Peopleware warns against are shown to directly lead to the project failures predicted by The Mythical Man-Month. This connection seemed obvious to me as I managed various software development groups. Unfortunately, as the project described in chapter 1 of Sisyphus shows, accepting this connection is by no means universal and, if anything, is the exception in many companies and software development organizations.

Outstanding features:

Sisyphus deals with identifying which projects are amenable to schedule acceleration using agile methodologies or other approaches and which are not. Most significantly, the book provides an objective approach for making this determination. In addition, Sisyphus spells out in explicit detail the problems that will occur if a project that isn't suitable for accelerated development is forced onto too short of a schedule using any approach. Further, these problems are shown to result in deep design errors that will seriously impact the stability and security of not just the current release but also of any future release that attempts to build on this base.

Specific points addressed are:

1. Deep, systemic bugs, potential security flaws, and system instabilities are an unavoidable result of imposing an impossible schedule on a development team. Further, these problems are not amenable to attempts to “test quality into the code.” In fact, attempting to “test quality into the code” is a symptom of these problems.

2. The concept of inherent difficulty of software projects is identified and explained.

3. The role of high inherent difficulty in making the bugs, flaws, and instabilities of the first point inevitable when the schedule is highly compressed is shown.

4. Inherent difficulty is shown to be the limiting factor in all attempts (death marches, agile development, tools, or outsourcing and open sourcing) to significantly accelerate software development efforts.

5. Rather than being only a constraint on how fast a product can be developed, being able to solve inherently difficult development problems can be a competitive advantage if understood and appropriately exploited.

6. An approach is outlined that allows a software development team to determine if their current effort is constrained by inherent difficulty and to then apply appropriate development methodologies to these pieces of the project.

Competition:

Dreaming in Code (Crown) by Scott Rosenberg was released after I had pushed Sisyphus out for review to friends and colleagues. Dreaming in Code is, to some extent, a direct competitor of Sisyphus although my conclusions are quite different. I have not had a chance to read Scott's book but I believe that Sisyphus provides a more useful analysis in that Sisyphus explains how to recognize and overcome situations where “software is hard.” I would even go so far as to state that the analysis presented in Sisyphus predicts the failure of th project chronicled in Dreaming in Code.

In addition to Dreaming in Code, a few classics of software engineering such as The Mythical Man-Month and Peopleware touch on these topics only at a high level and without the insight into the problem that the concept of inherent difficulty gives. Specifically, these books do not show how forcing a development effort onto an abbreviated schedule has a massive impact on the product's near term and long term quality. To some extent, the authors of these works incorrectly assumed that readers would heed their warnings regarding failed projects but this doesn't seem to be the case. In addition, there are also a number of books that explain how to apply various development methodologies that don't really address the central theme of Sisyphus: what to do when the problem is hard.

Marketing Plan:

The core audience for Sisyphus will be those specifically concerned with software engineering in its classical usage as how to create better software. This book will also appeal to all software developers and their managers but especially those who are under competitive pressure to produce a high quality product on a short schedule. This probably means everyone who develops software commercially since I rarely hear of projects with copious schedules.

Sisyphus can probably best be promoted through reviews in professional publications such as ACM Queue and IEEE Computing. In addition, exposure on web sites that attract software developers and IS and IT professionals such as Slashdot would also be beneficial. Finally, the topic addressed by Sisyphus screams to have unstructured discussion such as would be provided by a blog. I would find it quite interesting and probably useful for creating a follow up to Sisyphus to serve as editor for such a web site.

Table of Contents for A New Chore for Sisyphus

Prologue – I like to use humor so the prologue has Zeus giving Sisyphus a new punishment of being a software development manager. The appropriateness of this analogy is pointed out at multiple points in Sisyphus.

Introduction – The insights in Fred Brooks' classic The Mythical Man-Month (TMMM) and how they are still ignored were a primary driver in the writing of Sisyphus. I tie what I will cover in Sisyphus to TMMM and explain what the book is about and its structure. I attempt to briefly show that what I will be showing for software development projects is rooted in simple, common sense.

Chapter 1: Failure is not an option (more likely, its a requirement) – Describes the unhappy tale of the death march I alluded to in the prospectus. I then introduce how this book analyzes software development efforts including some terminology and set the stage for the remainder of the book by noting that software project difficulty has a number of different drivers with the size of the project being only one.

Chapter 2: Porcine aerodynamics – Introduces the idea that the cost of a death march goes beyond staff burn-out. Once this idea is developed there is a detailed analysis of the various software development practices that occur on a project under extreme schedule pressure. This analysis includes discussion of both the near term (this release) and long term (future releases and customer experience) consequences of these practices.

Chapter 3: Just assume time travel is practical – This chapter takes a theoretical view of software development projects and the choices that exist for getting the project done within the requested schedule. This analysis leads to a statement of the inherent difficulty conjecture: there is a minimum feasible schedule for accomplishing a software project and attempts to force a project to completion in less than this amount of time won't work. This concept is them elaborated.

Chapter 4: It is always something – Inherent difficulty is shown to be the result of project complexity and the need for the project team to collaboratively and concurrently solve complex, interdependent problems. The human limitations of this process are then shown to be the gating constraint on how fast a particular software development problem can be solved.

Chapter 5: Pretending the problem doesn't exist is a really bad idea – This is the first chapter in the second section of Sisyphus. In this and the next three chapters, the various ploys to somehow accelerate a software development project are shown to not work when the problem at hand has a high level of inherent difficulty. Chapter 5 kicks off this analysis by looking at why a death march must fail when confronted with an inherently difficult development effort.

Chapter 6: It is more complicated than you think – Same analysis as Chapter 5 but for agile methodologies.

Chapter 7: Chasing the accident – Same analysis as Chapter 5 but for software development tools.

Chapter 8: Make it somebody else's problem – Same analysis as Chapter 5 but for outsourcing. I also point out that the understanding of how to develop software to solve inherently difficult problems constitutes a competitive advantage that can be built upon.

Chapter 9: The “wrong” answer – Summarizes the lesson thus far and proposes an engineering approach to the problem.

Chapter 10: How to manufacture software – Apply the approach outlined in Chapter 9 and show how the partial solution can be made complete. This is accomplished by architecting a software product that is constrained by inherent difficulty to isolate the inherently difficult components. This approach is analogous to manufactured goods which have a core functionality that changes only slowly while various peripheral features are added or changed much more frequently.

Chapter 11: Conclusion and summary – A few parting shots and a summary.

Epilogue – Sisyphus almost escapes his punishment.

Appendices – Summarize the material from Chapters 2 and 5 in tabular format. Also, since I use the observations from RFC 1925 to introduce each chapter, I include RFC 1925 as an additional appendix.