Many software development projects run afoul of the "Iron Triangle" of project management: the interrelated constraints of Scope, Schedule, and Resources. Simplistically, a change in any one of these constraints has a necessary effect on either one or both of the other two. If the three constraints are out of balance with each other at the beginning of the project, then one or more of them must change/break or the resulting quality of the project is destroyed by artificially meeting the constraints by cutting corners.
The real crux of the problem is that it is easy, even unavoidable, for these three constraints to be out of balance at the beginning of a project. To make matters worse, they can get more out of balance as the project proceeds. There are many reasons for this problem that all stem from the fact that we have only limited ability to control, or even truly understand, all three constraints and the relationships between them.
The main reason we specifically avoid waterfall-style software development is that it is based on a generally false premise that one can, prior to building any of it, accurately describe in exact detail how a complex software system should work when it is completed. Inevitably as a software project progresses, conditions that dictate the scope of the project will change in any of a number of ways: the team understanding of the problem domain evolves, the business has new or different ideas on the direction the project should go as it sees the system take shape, the underlying needs or priorities of the business shift because of external events or changes in strategy, and so on. This is why waterfall development projects have historically had such a high failure rate: the process used to initiate and manage the project was too inflexible to adapt to the changes that emerged as the project evolved. Waterfall projects attempt to lock in the scope of the project at the very beginning and then intentionally make it difficult to change that scope once the project has started. For any project to have a reasonably high chance at success, the way it is initiated and managed must accept, and even embrace, significant changes in scope as the project proceeds. This means it is not possible to understand the true scope of the project at its beginning and means we cannot fully control this constraint at the beginning of the project, unless we are willing to do so artificially by blindly ignoring factors along the way that tell us the scope ought to change in order to meet the goals of the project. All this provides much of the underlying basis for agile processes and the highly improved success rates agile projects enjoy.
Estimating the time it takes to develop software for a given scope is imprecise, to say the least. The challenge is that software developers generally never build the same thing twice. If they did need the same exact thing as something they had previously built, they would just copy it. So all software development is essentially custom development. Thinking of software development as analogous to "manufacturing" or "construction" is a mistake. There is no fully developed blueprint for a software project in advance of its development – that is what the source code itself becomes. At best, there is a general outline of what the fully detailed blueprint will become. As each piece of code is being developed there are new bits of analysis and design occurring constantly – it is as much art as science. Once the blueprint of the software has been fully expressed in the form of source code, the only "construction" that occurs is when the source code is compiled or interpreted, and that is fully automated and effectively takes no time. All this is not to say that software development should not be estimated, just to say that the any software development estimate is a general guideline at best: an educated guess, but still a guess. As long as these estimates are taken in the proper spirit, they can be useful for general planning purposes and establish a baseline for a project. But they should never be taken as absolutes or as guarantees. Further, taking into consideration all the real uncertainty of scope and the inevitability of change during software projects, working on detailed estimates of large amounts of scope at the beginning of a project is largely pointless. Even if software development estimates were highly accurate, they still aren't actually useful if the scope they are based on is destined to change before the development is actually performed. Agile processes accept and embrace this whole problem by only performing rough estimates at a high level at the inception of the project and defer detailed estimation until development of specific areas of scope are imminent.
Resources are perhaps the most predictable and most controllable aspect of software development. When we are talking about resources, we are of course mostly talking about people. Sure, we might also need to spend some money on hardware and development tools and so forth, but the costs of software development are primarily human costs. For any given project, we need people with various skill-sets that will work well together as a team. By choosing well qualified people that have good team chemistry, we establish the essential core for a productive and successful project. However, one pitfall in establishing the resources for a given project is the "Mythical Man-Month" problem as expressed by Fred Brooks in his essays on software development where he argues that "adding manpower to a late software project makes it later" (now known as Brooks' Law) and where he colloquially illustrates the problem by stating that "Nine women can't make a baby in one month". While these colorful statements are clearly hyperbolic, the point here is that larger teams do not lead to proportionally faster or broader results on a given project. The additional overhead of communication and coordination required across a larger team leads to diminishing returns that eventually actually slow the project down instead of speeding it up. This is especially true when adding people to an already established team that is perceived as "falling behind" on a schedule that was probably flawed from the beginning, since the new people need the existing people to help bring them up to speed and then require extra communication and coordination from the core team. With upfront understanding and planning in light of the "Mythical Man-Month" problem, there are techniques that can be employed to speed up a large overall effort. These will still have diminishing returns, but sometimes that additional cost is justifiable given the urgency around schedule. Many of the agile engineering practices exist explicitly to enable teams to collaborate with a smaller amount of time invested into communication. Many of the agile processes can be run in parallel by cleverly dividing up large sections of architecture for different teams, but even this still requires a significant amount of coordination between teams simultaneously working on modules that must interact with each other in a larger system.
Embracing the Elastic Triangle
Of course we would all love to be in a world where we could accurately define the scope, schedule, and resources of a development project at the outset and have that project proceed to completion exactly as defined. Hopefully the above has disabused us of any notion of that being realistic. So, with all this said, how do we avoid getting trapped in the Iron Triangle? The reality is that we must allow at least one of the points of the triangle to vary as the other two points change or become better defined, making this an Elastic Triangle. Convincing stakeholders of this can be difficult since it is asking them to proceed on a project with some amount of uncertainty. But surely informed flexibility is always a better approach than ignorant rigidity. So which point shall we allow to vary? This depends on the specific circumstances surrounding the project. Generally it makes sense to lock in on a reasonable number of people to work on the project and optimize the size of the team to what is most efficient and effective – this will best control the ultimate cost of the project. With that point locked in, then drive towards either a specific definition of scope or schedule and allow the other to vary freely. Better yet, allow the interplay between both scope and schedule to vary as the project evolves and manage them in tandem. Of course all the constraints must live within certain reasonable boundaries, so it may be that all of the points need to be adjusted as a project evolves. The main point is to be flexible, to recognize changes in knowledge and circumstances quickly, and to adapt immediately in order to keep the project on track for success.