Eighty percent
For as long as I have worked in the field, the reported failure rate of large IT projects — data warehouses, "transformations", big-data platforms — has hovered around eighty percent. The number gets argued about, but nobody argues it down to twenty.
Take it seriously for a moment and follow the arithmetic. These projects run for years. A mid-level engineer with five years of experience has been on two, maybe three of them. That gives roughly even odds that they have never worked on a project that succeeded. Not once. They have never seen what it takes, from the inside, for a data platform to reach production and stay there.
Worse, they may not know it. Corporate communication declares everything a success, and from an individual's seat the logic is simple: I got paid, so it was a win. "Paid but never deployed" is the most comfortable outcome a project can have — everyone can call it a success story, and nobody has to run it.
So the mid becomes a senior and passes on their "experience". The methods that were followed on the failed projects — the pseudo-agile ceremonies, the practices believed to work despite never being checked — become the things they know. Material like DORA, which actually measures what distinguishes delivery from theatre, gets dismissed as not how we do things here. The comfort zone accumulates years and goes on a CV.
From the project leader's chair it looks like this: statistically, one out of two non-junior people in the room has some practical idea of what works. It is hard to tell which one, because people are opinionated in both cases, and the ones who failed before are often the more agitated. When you hear "this is what we did last time and it worked", the questions to ask are boring and precise. How many production deployments did you do? How many incidents were reported? How was your restoration process improving? Is the thing still running? The usual answer, once you dig, is that the previous project was a success story that never really deployed.
This is how the failure rate sustains itself. Almost every new project is run by people who have never experienced how a successful one is led, designed, built and operated. It fails, the practices spread, everyone moves on to the next one. Like a virus.
I can confirm the statistic from my own career: most of the projects I worked on ended in some sort of failure. I have also been on the ones that shipped — a few of them — and I know what they looked like from inside. The differences were not exotic. Someone reduced the problem before building it. Errors were designed for first. Estimates came from history rather than hope. Engineers deployed to production early and often, and the people who understood the business were given tools they could operate.
That is the entire pitch, and I try to keep it modest. I cannot promise a project will succeed. What I can usually do, fairly early, is tell which of the two kinds it is turning into — and describe what the ones that shipped did differently, because I happened to be there for a few of them.