Eight Ways to Understand Why (Project Failure Analysis)

A Big Project: ‘Fire Control’

In 2004, a major project was commissioned in England to reduce the number of control rooms used to handle fire service emergency calls from 46 to 9 Regional Control Centres (RCCs). The benefits planned included greater resilience as well as efficiency gains. It was intended to implement the new RCCs between 2011 and 2012, but during 2010 it became clear there were serious flaws with the project, which was scrapped by the end of that year.

There were two inquiries into what happened, one for the UK Parliament Public Accounts Committee (PAC)14 and another by the UK National Audit Office (NAO).15 The reviews found similar issues. The table maps their key findings against the ‘eight ways to understand why’ in Table 9.2.

To deepen our understanding concerning what caused things to go off track, I spoke to Charlie Hendry (former Fire Chief of the Kent Fire Service), who was involved in the project:

‘I was involved in the later stages of the Fire Control project. It was a much more complicated project than many realised. First, it involved new technology; second, this sort of thing had never been done before in the Fire Service; third, and most important of all, the local Fire Services hadn't really bought into it.

The lack of buy‐in was partly because of factors such as changes in roles, potential job losses and who was going to be in control of the RCCs. But it was also because of genuine concerns and questions, such as the likely loss of local knowledge when using a regional model. Remember, in the context of a fire, if you misunderstand where a fire is taking place, you are putting property and lives at risk.

Table 9.2 Overview of Key Findings Concerning the Fire Control project

Report	National Audit Office (July 2011)	Public Accounts Committee (September 2011)
Ways to understand why
Appropriate goals	* 1. Underestimate of the complexity of the project * 2. Unrealistic costs and savings	* (A) Failure to understand the complexity of the IT system
Communication and information		* (B) Weak project oversight
R2A2	* 3. Governance arrangements were complex, which led to unclear lines of responsibility	* (C) Despite the scale of failure and waste, no one has been held accountable
Design	* 4. Poor contract design impeded the resolution of issues	* (D) Insufficient early milestones
External factors	* 5. Insufficient grip to sort out early problems with delivery by the contractor	* (E) Weak mechanisms to manage sub‐contractor performance
Change/maintain	* 6. Changing direction	* (F) Failure to apply checks and balances from the start
Human Factors	* 7. Levels of support incentives	* (G) Failure to secure the cooperation and support of others * (H) Including over‐reliance on consultants
Resources and priorities	* 8. High staff turnover and poorly managed consultants	* (I) The focus was on building the new RCCs

I attended project board meetings that were centred around following the Prince II project methodology. Sadly, the mention of Prince II tends to bring me out in a rash nowadays because I saw an excessive focus on what was going on at a micro level, which got in the way of bigger, less tangible factors.

I felt we were losing the wood for the trees. Numerous problems were encountered years before the final decision to stop the project. But it seemed to me that that if you reported problems, said you were working on them, and provided new deadlines, then that was fine. It seemed fine to defer things into the future, where there appeared to be time and capacity, without seeing the cumulative effect of doing this. Furthermore, I think what got missed were the cultural and behavioural challenges that hadn't been scheduled to be addressed until much later in the project.

My sense is that not enough was done early on to challenge what was being planned against the “on the ground” hopes and fears of people who used existing systems on a day‐to‐day basis. If there had been the right kind of engagement at the early stages, it would probably have been realised that there was the kernel of a good idea to progress but not through a top‐down, technology‐based, one‐size‐fits‐all solution.16

As a result of not getting the buy‐in right at the right level, in detail, during the early stages, you end up with a flawed design concept, assuming the wrong processes will work in the future and following a flawed implementation path.

I made myself unpopular on occasions because I could sense that practical problems were building up. And it was tricky because the project had been going on for years. So, much money had been spent already. I could sense how invested many players were in the project and how much they would lose if they admitted defeat. I feel I did my best to raise concerns, as did others, but the reality was that it probably needed someone independent to dig into the various problem areas, as well as the hidden cultural buy‐in problems at an operational level, to then call out “the emperor has no clothes”.’

Charlie points to a common problem: how easily we can underestimate the behavioural and cultural aspects of what seems to be primarily a project concerning systems and infrastructure. He offers a few additional reflections about target setting and accountabilities:

‘There is an interesting question about how we think of targets. On the one hand, you can have a target, as in Fire Control, where it's a specific new, big, national system, costing millions and taking years. Alternatively, you can set more modest aspirations that are more a “direction of travel”, such as “We need more resilience in our control centres, and it will be useful if we try to save some money as well”. From this sort of target, alongside pressure on budgets, you can get more of an organic, emergent approach that tries new ways of working in a “bottom‐up” manner. On the downside, no fanfare and no swanky new system, but on the upside, no hundreds of millions spent on consultants, no distractions from business as usual and no newspaper stories about wasted taxpayer money.

There is also a fundamental problem with the way responsibility and accountability work at a senior level, especially in relation to projects and programmes in the UK public sector. In theory, people at senior levels are given specific accountabilities to run organisations effectively and efficiently. This can work when the organisations’ boundaries are clear, and measures of success are realistic, specific enough and not subject to “finessing”. But when you turn to projects and programmes, with steering groups and governance boards, you automatically get a blurring of roles and responsibilities because decisions are often made on a collective basis. Worse, big projects are often multi‐year and people move on, so by the time a project gets into difficulty, a good proportion of the key personnel will have changed.’

Charlie highlights a problem with accountability across organisational boundaries. Steering groups and committees can make it difficult to trace who decided what, especially when staff change. An audit trail needs to preserve both the decisions and the information available when they were made.17 As the saying goes: ‘The secret of success is having someone to blame for your failures’.

The points Charlie raises can be found in other projects that have ended up disappointing stakeholders. Here is a ‘fact pattern’ that you might see:18

Concerning Prevention:

• (P1) The goals may overlook the project’s complexity and risks.
• (P2) People with practical knowledge may be asked to support a plan before their concerns have been heard.
• (P3) Board and steering group meetings provide checks, but limited time and data can make those checks ineffective.
• (P4) Consultants may not challenge the project’s ambition unless they are independent and specifically asked to test it. Judging that ambition is particularly difficult early on.
• (P5) Group pressure can make it hard to raise concerns in key meetings without appearing overly negative.
• (P6) Finding serious obstacles and estimating their effect on the budget and timeline takes time and investigation.

Concerning Detection:

• (D1) Project data and information was supposed to be able to highlight critical issues (often red/amber/green ratings alongside criticality);
• (D2) however, there are multiple reasons (psychological and structural) why metrics may be poorly designed or inaccurately reported (especially at the early stages of the project), and
• (D3) there are limits around what information and data can be pressure tested before it reaches the board or a steering group, and
• (D4) some key information may only come to light towards the end of the project at which point delivery to time, cost and quality may be challenging.

Concerning Recovery:

• My experience is that it often takes a visible setback or a concern from an influential senior executive to trigger a fundamental rethink of a project or programme, including detailed exploration of serious ‘Plan B’ options.

These problems can reinforce one another. The source maps their connections in a high-level causal loop diagram,19 using the eight ways to understand why, shown in Diagram 9.3.

Formal project controls can fail when they do not reveal these underlying problems. Worse, as Charlie explains, the very presence of project management dashboards, consultants, etc., can mask subtle, hard‐to‐measure issues, such as the absence of real buy‐in to operational matters, a clear understanding of how these would be addressed and whether there was sufficient time and budget left to address them.

Andrew Wright stresses the importance of having safety margins and clear priorities:

‘It's not just an urban myth that projects tend to be over‐optimistic on benefits and light on risks and extra costs. More and more research tells us this is an endemic problem in both public and private sectors.

The mindset you need to develop is one in which you expect to see scenarios for the project and critical dependencies and force yourself to imagine how you would feel if things were wrong by a factor of one and a half or even two times. When you think this way, you can start to be more thoughtful about balancing big spending on major projects versus more modest upgrades. You can also be stricter about project priorities, what is “must have” and what is “nice to have”.’

Gary Klein’s project pre‐mortem technique asks a team to imagine that the project has failed, then work backward to identify what might have gone wrong. They can then use these insights to either prevent problems from happening in the first place or to detect these problems early on before they go off track.20

Without mechanisms to pierce an understandable ‘can do’ mindset, it will often be very hard for those directly involved in a project or programme to say, ‘The emperor has, or may have, no clothes’. So, steering groups and project managers with concerns about a project or programme need a humble, questioning, frame of mind and may need to seek independent assurances around what is happening, being prepared to listen to any concerns without seeing these as an affront. Likewise, you need high‐calibre auditors to do independent assurance in a way that will be sensitive to its impact on a project but be rigorous. A project management consultant explained:

‘There is always pressure in a project environment and therefore lots of scope to get management information that isn't always 100% accurate. So, to get robust assurance, you need people who know what they are doing in terms of project management disciplines, who can dig into the right things and focus on the essentials of what the context demands.

Having the right skill sets matters greatly because even minuscule fractures in critical areas can lead to problems later. For this reason, the more significant and more demanding the project or programme, the more you need to budget for high‐quality independent assurance because most project teams are too close to the action to see the importance of certain fractures.’