Kanban has been making in-roads at Mozilla, and this is no small feat considering the size and complexity of the organization. I’d like to think our team has influenced folks, either to adopt Kanban, or to at least think about alternative development methodologies. Below is the history of how Web Productions came to use Kanban and the lessons we learned along the way.
In The Beginning
In the beginning, development in our group was painful. Way back in February 2012, tasks would often come in via IRC and there was no way to handle “fires” or limit the amount of work going on. Developers and project managers were getting frustrated. Tasks and releases were managed either via lists in bugzilla or as bug lists in an etherpad. We could not tell the status of tasks nor how much work we had.
We knew there were problems, but we did not know what they were or how to solve them.
Lesson #1: Software Is Only Part Of The Puzzle
We initially thought the solution could be found in a third party tool. To this end we began evaluating a bunch of project management tools including Libreplan, Tinypm, Liquid Planner and Team Box. We even evaluated two tools that could be used for Kanban: Kanbanery (which we wound up using a year later) and Trello. Evaluating each of these took a considerable amount of time and we quickly realized we didn’t know enough about our problems to evaluate possible solutions. We didn’t know if our group had the type of problems that these products were designed to solve.
Lesson #2: Know Thyself
It was clear we needed to understand how work flowed through our team and what specific issues we had. During an on-site in April 2012 we mapped out our work flow. Below are screenshots of this exercise:
1. Too much work-in-progress (WIP).
2. Unable to prioritize.
3. No source of “truth” for projects–we could not see the “state” of a project.
In retrospect, we should have mapped out our work flow and identified our problems before searching for a solution. We needed to identify and understand how we worked and what our problems were prior to thinking about changes or improvements.
Lesson #3: Small Changes Are Better
Given a better understanding of our group’s problems we started to evaluate solutions. I have had great success with Agile/Scrum on past projects, as has one of our senior developers, Paul McLanahan. Together we decided that a Sprint-based approach could be a starting point. If we could get our tasks visible somewhere outside of Bugzilla, we could begin to address problems one and three. Using Scrum and Sprints could help address both problems one and two.
Around this time Paul had been developing Scrumbugs, a tool that would enable managing backlogs and sprints using data from Bugzilla. We knew we had to do something, and given our teams’ familiarity with Agile, using Scrumbugs looked to be a low-risk approach. However, it was going to be a big change involving adopting a new process (Agile/Sprints) and a new tool (Scrumbugs). At a minimum it would be an experience from which we could learn, and at a maximum it would solve our problems.
Scrumbugs was tightly coupled to bugs in our issue tracker Bugzilla. It used Bugzilla as a data source and allowed us to view tasks and create Sprints. It was, and continues to be, quite clever. For us, Bugzilla is not just a ticketing system, it’s where work gets organized and discussed. Any tool that we used had to somehow mesh with Bugzilla. We decided Bugzilla would still contain the ‘what’ we had to do, while Scrumbugs would be where we managed the ‘when’.
As part of implementing Scrum we wanted to build out Paul’s side-project to have the features we wanted. In April 2012 we started conversations and wireframing how this could work. Some sketches of this process are below:
The amount of work required to get Scrumbugs working for our projects was not insignificant. Additionally, I wanted to expand Scrumbugs to handle multiple projects and combined teams. In retrospect this approach had many flaws:
- Implementing workflow changes alongside building a system was too much.
- Before we realized any benefit we had to design and build the tool.
- We were changing too much at once. Folks were learning a new process and a new tool.
- We adopted an inflexible Scrum system.
- You needed to adopt the workflow we designed.
- Changes to workflow required changes to code.
- We designed a system around Sprints and releases.
- As we moved to continuous deployment our original model began to break down.
- Our designed model did not reflect reality.
- I outlined how I wanted us to work, not how we actually worked.
- The steps to get from one to the other were not clear and in some cases not practical.
Despite my team’s initial missteps, Scrumbugs has proven to be an awesome tool for other teams at Mozilla, especially those working on released based projects and those using traditional Scrum.
Lesson #5: Look For Immediate Benefits
While we were spec’ing and coding scrumbugs life went on unchanged. We couldn’t realize benefits until the system was close to completion. In May 2012 the following was still happening in IRC (names changed to protect the innocent):
10:03 DEV: PM: pong
10:03 PM: DEV: http://foo.mozilla.org needs somewhat urgent design refresh.
10:03 DEV: sounds great
10:03 PM: DEV: awesome should be pretty straight forward
10:04 PM: DEV: here's the bug
10:04 PM: https://bugzilla.mozilla.org/show_bug.cgi?id=foo
10:04 DEV: PM: are you suggesting I do this?
10:04 PM: if you have the availability
10:04 DEV: there has to be a better way of assigning work than "whoever is on IRC"
10:04 DEV: I'm in the middle of a foo sprint
10:04 PM: PERSON: said you were looking for good first bugs on foo.org
10:05 PM: DEV: no worries if you can't take this.
10:05 DEV: sure. design refresh just sounded large
10:05 DEV: PM: I'll take a look
10:06 PM: DEV: cool. these urgent requests are rare I promise
10:06 DEV: has seen several already
Lesson #6: Implement A Flexible System
We began using Scrumbugs in late August 2012 and immediately ran into problems. Our assumptions about workflow were wrong and we had technical issues with data being updated from Bugzilla in a timely manner. It wasn’t until October 2012 that most of the bugs were worked out and the system became usable (this was not an insignificant amount of work and for a side project, Paul did an amazing job).
In practice using Scrumbugs on our projects was painful:
- Team update meetings turned into one-hour+ bug triage sessions.
- While we had a list of bugs for a sprint, these became outdated almost immediately as new ‘on fire’ tasks came in.
- We had no way to limit work in progress, the bugs in the sprint were always in flux.
- We did not have an accurate representation of the state of tasks-you could not tell what was in QA or what was live.
- Folks did not get immediate advantage from Scrumbugs and did not want to use it.
- We did not have metrics to inform improvements.
- Improvements we did want to make involved code and we were limited by who could work on it.
- As more teams used scrumbugs, each team wanted to tailor the workflow to their needs. This was not possible given that we couldn’t tailor the tool for each individual team.
Lesson #7: Don’t Be Afraid to Abandon What Isn’t Working.
In September 2012, around the same time we were rolling out Scrumbugs, our team picked up the project management and development of Mozillians. I started this project using Scrumbugs and ran into many of the issues outlined above. I decided to use Mozillians as a Kanban pilot project using Kanbanery, a tool I had been playing with for personal task management.
Mozillians was a great candidate for a pilot project because:
- It had a small team, and most of us were new to Mozillians.
- It had struggled in the past to get promised features released on time.
- Tasks were all over the place: multiple bugs filed for the same thing.
- The project was generally in a rough state due to hacky code and lack of resources.
- The project had previously had high developer turnover.
My initial goals for Mozillians was to test Kanban as a concept, and if it worked, use it as an example for a larger roll-out. Around this time I was reading Kanban: Successful Evolutionary Change for Your Technology Business by David Anderson. I followed his guidelines to set up a board that mirrored our existing workflow, limited work-in-progress, and populated the board with tasks from Scrumbugs. Below is our initial board:
The process worked surprisingly well and later we began to identify blockers and make tweaks to our workflow. In the screenshot below you can see tasks accumulating in the “Testing” column–this was a sign of inefficiency in our process, that once identified, could be resolved.
They key attributes that made this successful:
- We modeled the board using our existing workflow–initially almost nothing with regards to workflow changed.
- We had a visualization of our process and the current state of work.
- The tool was flexible; we could make small or drastic changes easily.
- The tool was not tightly coupled to another system. Tasks that were not in Bugzilla, or not worked on by our team could still be represented and tracked.
Lesson #8: Build On What Works
We had great success with Mozillians and Kanban. We shipped a previously delayed feature (API) on-time and the project was turning into one that developers actually wanted to work on. This was not solely due to Kanban, but it definitely helped. (Kudos to the team, they did a great job.) Now that we’d proved it could work we had to roll it out to additional sites.
I began moving other projects into Kanbanery in late 2012. Firefox Flicks was moved in November 2012, and Mozilla Reps was moved in December of the same year. To further illustrate the visualization differences between Kanbanery and Scrumbugs, below are 2 screenshots of the Mozilla Reps project in Scrumbugs and Kanbery respectively:
The Big One: Mozilla.org
It wasn’t until February 2013 that we had enough positive momentum to move Mozilla.org to Kanban. While there was some initial pushback, there were also immediate benefits and I was surprised at some of the more positive feedback. Choice comments from IRC:
"I normally hate processes but so far I like kanban :p Maybe there is something zen about moving little boxes between columns."
"It's all about having data an[d] numbers when you say to someone 'no, we can't do that in that amount of time'."
"I'm usually resentful and afraid of all new things--especially when they have names I've never heard of, but the Kanbanery stuff seems promising."
Lesson #9: Document Policies & Encourage Refinement
Now that we had Kanban working for the majority of our sites, we documented our policies on how the board should be used:
Teams started learning about blockers and implementing specific tweaks to the process. In June 2013 we wrote a set of guidelines specific to the workflow on mozilla.org:
Lesson #10: Things are not perfect and that is OK.
Kanban and Kanbanery helped to address our main issues with work-in-progress and prioritization, along with giving us an accurate depiction of the state of our projects. However, as we solve each set of problems we realize new ones. Our current challenges are (and there are probably more):
- Each project has a board, because developers work on multiple projects–work-in-progress is not fully represented in one place.
- There is still some work that is not accounted for.
- We have not figured out how to extend Kanban to our contributors.
- Work upstream (creative and UX) is usually not accounted for on the board.
- Lack of swim lanes has made it difficult to manage some classes of work. For example: emergency fixes or specific subsets of tasks.
We came to Kanban by trying other things. It turned out that the most flexible process with the smallest barrier to entry allowed us to develop a process tailored to how we worked. Things are not perfect, but consistent improvement is the goal, not perfection. In the future I’ll write about the stats we are collecting and how we use those to measure and improve our work.