Re-release: How I Stopped Worrying and Learned to Love Complexity
From Project Manager to Scrum Master
I published this article three years ago on Medium. I’m honoured that my journey of grappling with complexity plays an important role in ’s must-read book Driving Value with Sprint Goals: Humble Plans, Exceptional Results, which he released just a few months ago. Seems to me this is a good moment to offer you the opportunity to read the whole story. I hope you can appreciate the re-release.
From Project Manager to Scrum Master
”In complex environments, what will happen is unknown” — Pieter, quoting the Scrum Guide
I was at my company for five months when I agreed to manage our most urgent project. My colleagues responded tauntingly. “There’s no way that anyone can pull it off” was one of the nicest remarks. “You took the shortest route to the exit” was already a bit more foreboding. As was ”You are much braver than I am”. It scared me a little. These were indications that I had my work cut out for me.
“You are much braver than I am.”
People told me that I wasn’t the typical Project Manager. However, I always followed established Project Management practices. I created plans based on the input from people that were going to do the work. I wouldn’t change the planning under senior stakeholders’ pressure.
I would only do this when the developers confirmed that there were ways to do things faster, like decreasing the scope. I would bring all parties together to facilitate the discussion of alternatives, creating a safe environment for everyone.
I was transparent about issues and risks. I informed the steering committee as soon as disruptions impacted the plan. I had tough discussions to defend delays and higher costs.
I shielded the developers from politics. I let them find the best solutions. I managed the project, not the people working on the project.
All of these are established Project Management practices. Yet many colleagues worked differently. They obeyed the uninformed requests of the steering committee, instead of informing them properly.
They reported that everything was going well while knowing that this wasn’t the case. They forced a schedule upon people and punished those who didn’t meet the timeline.
Our Y2K project
I had one thing going for me. Failure was not an option because of the problem we had to solve. Twenty years before — when merely a start-up — the company had assumed that our number of clients would never exceed 9,999.
”If we grow to have more than 9,999 clients, we’ll be filthy rich!” That was then.
Since then the online payment industry has grown exponentially. The client number was used all over the place. We had our very own Y2K problem. There was no other possible outcome than success. I didn’t realise then how much this would help me.
The initial plan
I coordinated the efforts of 4 teams who said they were Scrum Teams. But they didn’t have Product Owners. Instead, they had me. These teams would do the coding and testing. We also had to change several external applications.
This made life increasingly complex.
I believed in the strength of long-term planning, as long as we did it thoroughly and updated the plan based on new insights. Changes were part of the deal, although the process to approve them was truly cumbersome.
You had to update the original plan and create a new version, check with the teams if they would be able to change their priorities, fill in a separate change approval form, submit an official request and wait for the next Change Control Board (CCB) meeting.
If you didn’t convince the CCB, you had to re-issue the updated change request and do the CCB meeting again. But there was more. Issuing a change proposal meant losing face. You didn’t plan well enough, you didn’t have your project under control.
Some colleagues avoided the burden by simply ignoring the change process, and painting a rosy picture at the steering committee. They had watermelon projects: green on the outside, and red on the inside. Hoping that the tide would turn magically.
I hated this cover-up practice. I hated the change control process less.
The first months
Our project started well. However, early in the project, we had identified a risk. We needed to build a new environment and — as soon as we were ready — switch to this environment to use the new client number format.
While many steps of this undertaking were unchartered territory I thought I could manage it as I managed the rest of the project. So it was part of the Project Plan and I managed the risk. Until the risk turned into a big problem.
All hell broke loose
As part of building up the new environment, all data needed to be replicated from the existing database to the new one. We had a lot of data and the database was old. It served its purpose for day-to-day work. It proved to be unfit for large-scale data replication. This was catastrophic.
We had unknown errors that only could be repaired with the expertise of the database vendor who also didn’t have all the answers readily available. We discovered patterns in the database that we didn’t expect, forcing us to repair the scenario and start again. We also found that the replication would take ten times as long as we expected. With that, all hell broke loose.
I felt I was lost in the woods. I didn’t know which road to follow now. I had no clue how this would impact timelines and costs. For any other project, this would be a reason to pull the plug. My project had to succeed. So I needed to find other ways.
I discussed my predicament with many people and I got all kinds of suggestions. The worst advice was: “Look for new opportunities elsewhere”. Then I had a chat with one of the developers, Pieter.
He started complaining about how we misapplied Scrum. I had heard that one before but didn’t understand what we did wrong. The teams had been delivering new iterations of the software consistently. They were very predictable and hardly came with unpleasant surprises. I had dismissed his remarks as ranting from a dissatisfied colleague.
Then he said something that struck me: “In complex environments, what will happen is unknown. We should use Scrum to deal with complexity, not to plan our increments of work.” Something in my head clicked.
“In complex environments, what will happen is unknown” — Scrum Guide 2017
I leaned back and started thinking. But couldn’t wrap my head around what Pieter just said. Then I leaned forward and asked, “How does Scrum help to deal with complexity? And what IS complexity?”
He laughed out loud, then responded, “Complexity is everywhere around us. Our environment is complex. Our products are complex. Your project is particularly complex. And you are completely missing the mark. With your brilliant planning which you have to update every month. With your risk management. With you trying to predict what will happen three months from now.”
I didn’t know how to respond. But it turned out that I didn’t need to because Pieter wasn’t finished. He leaned forward too and resumed his reprimand, “Scrum is a framework that helps you deal with complexity. I suggest you study it. The guide is only 19 pages, so if you start reading now, you should be finished in an hour. Then your journey will start.”
Inspired by Scrum
This is not a story where I magically switch from traditional project management to Scrum. It didn’t happen this way. But the realisation that I couldn’t know what would happen was a solace.
Now I had to bring this to the steering committee and I knew that I had to come up with strong arguments. I needed to prepare well. I sat together with the Database Administrators (DBAs) and discussed everything we knew that needed to happen, assessing the steps.
As experienced as they were, the vast majority of things to do were new for them. We created a plan that would cover a week, learn from what we did and then create a new plan. Yes, this is the part inspired by Scrum.
I would discuss with the steering committee weekly too, keeping them in the loop. I was ready to convince the steering committee that we didn’t know how long it would take, how many people we needed to do the job, and how much it would cost.
I started my plea with, “We need to make this work, we can’t fail.” I made sure that everyone realised that we had no option but to look out of the box. Then I asked the DBAs to explain the road ahead, with all the uncertainties and ‘first-time’ actions.
To my surprise, the steering committee embraced our approach. I guess it all made perfect sense. All it took was the realisation that we couldn’t predict the unknown. Sparked by one remark from Pieter, like a butterfly setting many things in motion.
Active participation, ownership and progress
Things didn’t go smoothly from that moment on. Far from it. We stumbled upon something new all the time. Some issues were small, others were pretty serious. After three weeks, our plan of approach had radically changed. And it had expanded. But at the same time, we were making progress.
Sometimes we took one step forward, two steps back. But most of the time we moved ahead and every time we learned something new, our path became clearer.
By forming clear goals every week, we could easily identify the individuals who needed to chip in, create the plan with them, and have them involved. With that, the sense of ownership grew, and dedication grew.
Suddenly, we were at the end of our checklist. We reached our destination. We had changed our complete platform and the functionality went “live” flawlessly.
Many people spent many hours coding, configuring, replicating, building, testing, testing, and even more testing. At the start, we completely underestimated the effort, the time, the complexity, and the costs. But we had to get the job done. And we did.
My learning journey
When I agreed to be the Project Manager of this endeavour, I didn’t expect it to be my last project in this role. As time progressed though, I realized more and more that the established way to manage projects didn’t work for us.
The product environment was too complex. There was no way that I could plan everything upfront, to create a plan that would be remotely close to what would happen in reality.
When I realised that my project was complex, I changed my approach accordingly and the pieces of the puzzle started to fit. I had found a way to manage expectations realistically and moved the ball downfield, step by step, inspecting and adapting constantly.
My view on delivering products had changed drastically. I started as a Project Manager, and at the end of the project, I became a Scrum Master. And I started to write about Scrum.
Thanks for a great article. I am interested in how you communicated the long term plan and how you forecasted whether you would actually be able to deliver on time. Working in short iterations taking small steps is of course the way to go, but how did you ensure that the progress you made was enough to finish until the go live date?