In metal working there is a process called annealing where cycles of heating and cooling are used to modify the chemical structure of the metal to achieve desired properties. This process was the inspiration for a type of software algorithm called simulated annealing.
Simulated annealing is a probabilistic technique for finding a global optimum to a function. The idea is if you are trying to find a maximum value, a natural way to proceed is to follow a gradient to higher values. In the chart below, imagine you are at the peak at x=3.3 (blue dot). If you look left values get lower, if you look right values get lower, so things are great so you can stop searching, hurray!!
However, if you were able to see far enough right you would see a much higher peak (at around x=7.2)!
The simulated annealing algorithm injects randomness by making the search jump random distances to the left or right to force it out of a local optimum. Initially, these are large jumps but over time they get smaller. The key insight is that early in the search it is unlikely you are anywhere near the global optimum so big jumps are needed to see if it can find a taller peak. But overtime smaller jumps are better since it likely has found one of the best peaks already and does not want to jump away from it.
Software teams and local optimums
What does this exploration of annealing have to do with running a software team? The connection is that a software team may be stuck in a local optimum.
A software team is a combination of people, process, and culture. Most teams eventually find a set of processes and ceremonies that work well enough. This includes the routine meetings, the project management methodology, and customs the team follows. Finding stability is great since it lets the team focus on the work. But it is important to periodically question these processes (akin to a probabilistic shift like in simulated annealing).
If it ain’t broke, don’t fix it could it be better?
Why change if your team is working well? Why not just let things continue to operate smoothly?
I’ve had this feeling many times. My team is operating well, we are getting stuff done and things are fine. But are we growing? It is difficult to grow doing the same thing you have always done. Even if we are working well, is there a way we could be even better?
Therefore, it is useful to routinely try to find a way to spur a shift or jump, try something new, experiment with a new process or a new way of working. These do not have to be monumental jumps. If your team is operating efficiently, look to find smallish shifts to how you work. If that goes well, repeat and do it again. You may end up far from where you started. However, if there is an area your team is struggling in, larger shifts may be useful. As things improve, decrease the size of changes (but don’t stop).
I have been through this process many times, both in the large jumps and the small ones. Years ago, my team was overall struggling. We had a ton of knowledge silos where multiple people on the team were the only ones who could work in certain areas. If they were out sick or busy and that area needed work, we ground to a halt. Our quality was low and team cohesiveness was poor. This was time for a big shift. We had a team discussion and proposed a 3 week “experiment” to switch our methodology which included adopting pair programming. After the 3 weeks we held a retrospective meeting to discuss the results. We reviewed what was working and what wasn’t and continued this process for the next few months, making increasingly smaller jumps until the team landed on a process that was productive and sustainable.
At other times when the team is operating well, we still routinely look for small jumps, to see if we can find incremental improvement: changing how we handle bugs and live site work, changing documentation strategies, or changing meeting cadence and ownership.
It may be in some areas you are efficient and productive, and those areas warrant smaller jumps but in other areas you may need larger jumps.
The size of the jumps are dependent on the teams’ efficiency and productivity in the area being considered for change.
Culture of experimentation
The easiest way to implement small or large shifts on your team is by building a culture of experimentation. No change should be thrown down as an edict. “Thou shalt work this way now” does not work. The manager of a team does not own its process and culture, the team collectively owns it, and the team must collectively decide to try a change.
Some larger changes may take more than 3 weeks to bear fruit, but after trying something for a couple of weeks, people will often have a greater appetite for iterating further on it. If people hate it, then scrap it and try a new experiment.
A successful experimentation culture hinges on the manager allowing changes from the entire team even if they may not personally love the idea. I have had the team decide to try changing how we work in ways that were not my preference. But the team owns its process and honoring that is key. These changes were small, and they ended up working out and leading to the team operating better. Even more than that, it builds trust in and instills the experimentation culture and reinforces that the team owns their process.
In search of a global optimum
Just like in simulated annealing, these large and small jumps are all in search of a global optimum value, or in this case the optimal team configuration. In reality, there is no absolute optimum for a team. But the process of change is how we can continually find incremental improvement. The team proposes a change, time-boxes it, discusses its results and hopefully things get a little bit better (or if they do not that is fine since it demonstrates willingness to try and move on). Then do it all over again.