Structuring our Development Teams and the move towards Domain Driven Teams
As Codeweavers’ development team has grown over the years, one of the challenges we have often faced is how to structure our development teams.
A little history
We've practised Agile Development techniques for around 9 years, back when we only had a single digit number of developers in our team. This meant we had a single team all working towards a common goal of developing our new software platform (it's probably also worth mentioning here that our platform is built with a Service Oriented Architecture (SOA) approach as a number of different applications which come together to form the Codeweavers Services. This will be an important aspect in some of our later decisions).
As our client base grew, so did our development team. I won't take you right back to the beginning but to around 3 years ago when we decided to split our developers into two distinct teams, one team dedicated to our Showroom and Lender Decision applications and the other dedicated to everything else just like our core development team had always been. This made sense at the time, we wanted a dedicated stream of work on our new Showroom application whilst still continuing to deliver to our clients in the same way we had always done.
This didn't stop our core team from continuing to grow with new recruits and one day, someone saw a morning team standup of that core development team and noticed how ridiculous it looked. I'm not sure I've ever seen a morning stand-up (or any meeting in fact) with 15 people be particularly productive. Our teams were effectively structureless and some might say without purpose, particularly our core team, and whilst techniques such as pair programming were certainly helping keeping some order we clearly needed to do something. Since then we've gone through a number of iterations on our development team structure and last week we went through another following a retrospective.
Let me take you on our journey...
Client Based Team
sIn the Summer of 2015, we decided to break down our development team into multiple teams each based around our larger clients. At that time we had Work in Progress (WIP) queue size of 4 so this made sense to give each client based team a single WIP limit. After a couple of months, we came to realise that all of our bigger clients were eating up our WIP queue and this wasn't giving us time to focus on smaller pieces of work that we could perhaps just knock out in a couple of days or to continue developing the Codeweavers Platform in the direction we wanted it to continue going. Our initial fix for this was to spin out a small team of 3 developers who curiously named themselves the Ninja squad. They challenge was to grab those smaller pieces of work that were getting sidelined and deliver them quickly whilst maintaining our high standards. In reality, after a couple of months, this team got bogged down in support tickets and pieces of work the other teams didn't really fancy. Not an ideal situation.
The client-based team's concept definitely had its strong points, however. Our largest clients were effectively getting a dedicated development team for a set period of time until that team moved on to work with another client and our developers no longer had to contend with 15-20 man stand-ups every morning. We were definitely delivering but there was a rather large elephant starting to appear in the corner of the room. As our developers were working on tasks they were moving between many of the different applications that build up our Service Oriented Architecture. This meant that the domain and architectural knowledge each developer had to have for each and every application was vast. Far too much for most developers and particularly daunting to those new to the business. Many developers felt like they were splitting their time across too many different applications and never really becoming the expert in an area that many of them were striving to be.
App Based Teams
To counter the elephant in the room, we did a little research and then devised a plan. Taking inspiration from companies such as Spotify and Netflix we decided to break down our development team differently into a number of smaller teams based on our architecture. Each team is structured with a flat hierarchy and are empowered to make decisions and choices that will decide the current and future direction of the applications they are associated to.
This was an important change for us in many regards. It allowed developers to become more specialised in a certain area of our architecture whilst also giving them the motivation to really improve that area, whether it be code quality, performance or new features. We’ve seen this in action too over the last few months, with many applications seeing significant improvements. This was the real win for us with the App Based Teams approach, the focus on specific areas of our codebase has allowed developers to really take pride in their applications.
One issue with this approach was the new challenges it has given other departments in the business, mainly how they find the right team to talk to about a piece of work. There was certainly a big learning curve as it was never particularly important for teams like our Client Support team to know exactly what functionality a particular backend application had in our architecture.
We also initially struggled with how we would still deliver work for our clients with a team of developers no longer aligned to our client work. It turned out this wasn’t much of a problem after a few initial niggles. We had developers coming out of their teams and pairing up across the team to deliver client work that spanned multiple applications and it actually turned out that quite a bit of the client work often only hit a single team’s applications.
How did we implement the change?
We came up with 5 distinct teams, each had responsibility for around 2-4 applications and we roughly grouped these applications into similar domains. Our next step was to work out an ideal number of developers for each team, based on historical knowledge of how much work each application usually has going on in it and what we expected to see in the near future.We wanted teams to be 4-6 developers with a range of experiences and skill sets but we also didn’t want to dictate to the developers which team they will be going into. Also, some teams would likely suit being a little smaller and some a little larger. So, we got all the developers into one room and drew something that looked a little like this on the whiteboard.
We then gave every developer the opportunity to pick the team they wanted to join and also the option to pass on making a decision. Going around the room each developer made a choice and a few decided to pass. We then went around again with those who passed and the board slowly but surely filled itself in.
You might be thinking we would end up with unbalanced teams, or teams we had worries or doubts about but on the whole, this actually worked really well. We made one suggestion for someone to switch teams to help boost a team’s experience levels but it seemed everyone was very happy. We spent the rest of the day having a bit of a desk reorganisation to support the new teams and giving them their own working areas and off we went.
Domain Driven Teams
The next iteration of our app based teams is to move towards the teams being associated with our Problem Domains rather than a particular piece of code or application. We’re calling these Domain Driven Teams. One hope from this is to break down some of the walls that have appeared between teams not wanting to tread on each other toes by going into another team’s applications. In reality, we all share the codebase so we shouldn’t have to worry about this.
#domain*noun*noun **domain**\; plural noun **domains**- a specified sphere of activity or knowledge.- \the country's isolation in the domain of sport\"" - field， area， arena， sphere， discipline， sector， section， region， province， world - \""the report was very significant in the arts education domain\""
So what changes?
We stop thinking about pieces of work and teams being related to a particular application and start thinking about the domain a MMF (Minimum Marketable Feature, often referred to as a Minimum Viable Product) or Support Ticket mainly involves.
Here is what each team is broadly responsible for:
Dealer and User Organisation & Authentication- Vehicle Information- Incoming Service Integrations
Lead Management- Proposal Workflow- Customer Management
Finance Package Management- Finance Calculations- Quote & Statistics Management
Web Application UI and UX- Documents and Printing
A new trial team we are running with a slightly different approach- Delivering cross-functional vertical slices for clients - Will be interacting with other teams when moving into their domain areas
So it’s ok to write code in other team’s apps?
Yes! If a team’s current MMF (Minimum Marketable Feature, often referred to as a Minimum Viable Product) crosses application boundaries then they should own it all the way through, don't pass the buck at the edge of what you feel is your application. If you feel you need some expert knowledge then turn to another team or individual that may have more experience in this area for advice but don't just throw it over the wall. This goes for support tickets too.
What about application error logs?
We actively track application error logs on our live monitoring screens and everyone should take ownership of logs that occur and take a look at them. Be pragmatic about this and take ownership when an error log looks like it is in your domain. Communicate with others following your initial investigations should you need to. We should be keeping our monitoring screens green and investigating logs as soon as possible.
How will Support Tickets be distributed?
Pretty much as they are now but hopefully tickets will end up with the right team the first time. Teams own certain domains and tickets are usually related to a certain type of problem. Rather than our Client Support team or Product Owners trying to guess the application, the code is in they will now pass the ticket to the team that owns those types of problems.
Why the change to Domain Driven Teams?
There were a number of points raised in a recent retrospective that was related to ownership, throughput and bottlenecks. Following the discussion, we iterated towards a domain ownership model rather than an app ownership model to help remove some bottlenecks (work required in our Web Services tier) and to fix code that is logging or clearing support tickets faster (Taking responsibility for logs and support in areas that are a little fluffy e.g. Legacy Applications).
The lecturer in me cannot resist...
Some further reading!
Domain Based Teams may lead up towards a more Domain Driven Design (DDD) approach when building software. Here's a bit of a classic book on the subject:
Why so many changes to the team structure?
It's in our blood. At Codeweavers we don't believe in standing still and we are always looking for ways to try to improve. We're strong believers of many of The Toyota Way principles and particularly Kaizan, to continuously improve. This is just one of the many areas we focus on to try to be more effective as a team.