Tobias Mayer pointed out a classic paper in the scrumdevelopment group today. Parnas has created many papers that are now viewed as classics in the software development literature. This one points out many of the reasons why software development is an empirical process that requires an inspect and adapt approach as opposed to rigorous initial design.
Parnas, David L. and Clements, Paul (1986) A Rational Design Process: How and Why To Fake It. IEEE Transactions
I keep a copy of the Parnas collected papers around to remind me that many aspects of what we are doing with Scrum were pointed out by leading thinkers in computer science 20-30 years ago. Most people in the software business today have never read any of these papers which is why we continue to make foolish mistakes. Of course, Scrum was created by people who had read these papers to formalize what was known to work.
Hoffman, Daniel and Weiss, David (2001) Software Fundamentals: Collected Papers by David L. Parnas. Addison-Wesley Professional.
There are companies who feel that waterfall is working well for them and projects are being delivered on time and on budget (the budget is high, the time is long, and customer satisfaction is low). It is important to realize that they have created a illusion of a predictable process. When you peer under the covers and talk to the people really doing the work, you always find that the Parnas principles apply.
WHY WILL A SOFTWARE DESIGN “PROCESS” ALWAYS BE AN IDEALISATION?
We will never see a software project that proceeds in the “rational” way. Some of the reasons are listed below:
(1) In most cases the people who commission the building of a software system do not know exactly what they want and are unable to tell us all that they know.
(2) Even if we knew the requirements, there are many other facts that we need to know to design the software. Many of the details only become known to us as we progress in the implementation. Some of the things that we learn invalidate our design and we must backtrack. Because we try to minimize lost work, the resulting design may be one that would not result from a rational design process.
(3) Even if we knew all of the relevant facts before we started, experience shows that human beings are unable to comprehend fully the plethora of details that must be taken into account in order to design and build a correct system. The process of designing the software is one in which we attempt to separate concerns so that we are working with a manageable amount of information. However, until we have separated the concerns, we are bound to make errors.
(4) Even if we could master all of the detail needed, all but the most trivial projects are subject to change for external reasons. Some of those changes may invalidate previous design decisions. The resulting design is not one that would have been produced by a rational design process.
(5) Human errors can only be avoided if one can avoid the use of humans. Even after the concerns are separated, errors will be made.
(6) We are often burdened by preconceived design ideas, ideas that we invented, acquired on related projects, or heard about in a class. Sometimes we undertake a project in order to try out or use a favourite idea. Such ideas may not be derived from our requirements by a rational process.
(7) Often we are encouraged, for economic reasons, to use software that was developed for some other project. In other situations, we may be encouraged to share our software with another ongoing project. The resulting software may not be the ideal software for either project, i.e., not the software that we would develop based on its requirements alone, but it is good enough and will save effort.
For all of these reasons, the picture of the software designer deriving his design in a rational, errorfree, way from a statement of requirements is quite unrealistic. No system has ever been developed in that way, and probably none ever will.
English 
French 
Spanish