The Agile Mindset

Agile is the not-so-new silver bullet for software development, but also for any kind of business process. We are working agile has become synonymous with we are doing it right. Still it feels to many people that agile doesn't work. This article goes back to the roots of agility and shows why it fails to work for many organizations.

Why Everyone Has Gone Agile

Traditionally, software development adopted practices from other engineering practices, which resulted in the Waterfall Model and followed a typical planning process: specify what you want (like for a bridge: how long it has to be, how much weight it needs to carry), make a detailed plan, execute that plan. Even when building bridges you will encounter unforeseen issues such as the quality of the ground, but the general idea of how to build that bridge will stay the same.

But software is much more versatile than bridges. Once a bridge is built, there is not a lot that you can or would want to change. But software evolves all the time, software is never finished. And this has become more severe with the internet. While 30 years ago, software would be developed in cycles of several months, pressing the whole thing on a bunch of CDs and shipping them by mail. Nowadays updates can be done every day just spreading the software over the internet. This is both a blessing and a curse: bug fixes and new features can be shipped quickly, but this also reduces time for thorough testing and integration with the entire system. Everything in software has become more complex: size, dependencies with other software systems, requirements of being connected and always available, rapid change of technologies and tools.

And this means that the traditional way of software development doesn't work any more (if it ever has).

Back to the Roots: What is Agile?

I like to consider the Manifesto for Agile Software Development as the holy scripture of agile development. Historically, it is not the beginning of it, but rather the culmination of different efforts and frameworks with similar intentions. What I like about the Agile Manifesto is that it does not describe specific procedures and practices, but a mindset. And this is exactly the point that is often missed.

It just consists of these four value statements:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

By its very nature, it shows the complexity of software development. It doesn't say do this or don't do that, it just prioritizes: one thing over another. It doesn't say nobody needs documentation, it just states if you are lacking time and need to make a hard decision as to where to invest your efforts, let it rather be working software than comprehensive documentation.

Agile Frameworks are not Agile

The Agile Manifesto sounds more like philosophy than like what managers (and some developers) want: a process model, or at least some kind of tangible rules, so that if it doesn't work, the rules are to blame, not the people.

So there are different initiatives to provide more specific decision rules. Most of these frameworks (for lack of a better word) preceed the Agile Manifesto and have been a basis to develop it. But I feel that by their very nature of providing specific rules, they violate the manifesto, at least if the rules are applied without the underlying mindset.

Scrum is probably the most favorite framework for managers as it is basically the Waterfall Model in shorter cycles. The method is based on putting up a plan at the start of a sprint, which clashes with the fourth value of the Agile Manifesto (Responding to change over following a plan). I don't want to denigrate the procedure as such, because I think the original proponents were very well aware that inside a sprint change may happen, and they do not require the team to meet the goals they set at the beginning of the sprint. But in practice the sprint goals are a tug of war between stakeholders inside and outside the development team. Everybody wants everything and quickly and once the team has added a feature to the sprint, they are basically caught in a kind of contract with all the negative consequences of nightshifts and buggy code.

Scrum has become so popular that there are job descriptions and certifications for Scrum roles, such as Scrum Master. But according to the original model, a certified Scrum Master is a contradiction in terms. The point of Scrum is that a team organizes itself and that some poor devil has to take on that role for a while, which is to keep the backs of the developers clear from outside disturbances. So the Scrum Master will deal with the outside world, while the other developers have fun coding. But in lived practice, the (certified) Scrum Master has become the Chief Contract Negotiator, which contradicts the third value of the Agile Manifesto (Customer collaboration over contract negotiation).

Extreme Programming is another framework of agile development, but with a completely different focus. It concentrates on best practices of programming. What Scrum is for managers, Extreme Programming is for developers. It comes with a set of values that are similar, or at least compatible, with the Agile Manifesto. It also provides a set of rules that includes a kind of process model like in Scrum, but also programming practices. For example, it emphasizes testing. Not just due to Extreme Programming, automated testing and test-driven development have become a holy cow. Certain test schemas are nowadays required by law (e.g. in banking or safety-critical systems).

There is nothing wrong with testing. Every software should be tested in some way or other and none of the practices propagated by Extreme Programming are wrong. But I have seen developers start writing tests before even understanding the requirements of the program they were about to write. My impression is that developers can hide behind the mantra with testing you have done nothing wrong. But there is a catch: tests cost time and money. Not only writing the tests, but maintaining them. Each time the program gets changed, the tests need to change as well. So writing tests too soon can be a major cost factor in software development. This is the reason why documentation is considered as a lesser value than working software in the manifesto. Nobody has a problem with documentation, but just like tests, it has to be adapted along with the software and that is costly and takes away time from coding.

And then there is Kanban, which was developed for industrial production. The whole point of agile development is that software is not an industrial product, it is always individual. By its very nature, software exists only once. You can run it on different computers, but there is no real production process other than maybe the deployment on different machines. The main contribution of Kanban to the typical software development process described below is the use of a Kanban Board, or Kanban Cards. The idea is that a product moves through different stages in the production process. In software development, typical stages are design, implementation, testing (the exact steps prescribed by the good old Waterfall Modell). So the processes and tools (first value of the manifesto) return through the back door.

As human beings we have the weird habit of interpreting rules literally, especially if we find them hard to follow. I don't actually know where the catholic tradition of fasting is put down in rules (I'm sure it's not in the Bible), but I always find it interesting to observe how the rules are lived. Many people in Germany don't follow the full fasting month, but at least they don't eat meat on Ash Wednesday and Good Friday. But those days are anything but abstinence, they are days of feasting. Since meat is forbidden, on those days just about any delicacy is served that does not contain meat: cheese, herbs, bread, eggs. It seems to me that this completely goes against the idea of fasting and doing with less. But people follow the rule, they don't eat meat.

The same happens with agile frameworks. The ideas are good, but any rule that is followed to a tee becomes useless. You have to understand the idea behind it. I would find it more reasonable to eat a little leftover meat on fasting days than to celebrate a feast of anything but meat. And it's the same with agile frameworks and their rules. Rather than writing tests just because Extreme Programming tells us to that testing is important, we should be careful to consider when tests are helpful and necessary and when they are just a burden.

The authors of the agile manifesto understood this perfectly. They were practitioners trying to establish a mindshift towards better practices. And in a way they were successful: everyone wants to be agile. But at the same time they failed: everyone wanted the benefits without going through the pain of shifting their mindset and habits.

Agile Mixup Practices

While none of the specific methodologies really meet the ideas of the agile manifesto, the practical application does it usually even less. What I often see is an ad hoc mixture of principles from different agile frameworks. A typical one is this:

• Teams work in sprints, i.e. in fixed temporal cycles, such as one month.
  Sprints are a concept of Scrum and the idea is to emphasize the iterative nature of the process. Scrum imposes certain events on a sprint: it starts with planning the goal of the sprint, requires tracking of progress and a daily scrum meeting during the sprint and ends with two meetings: the sprint review and the sprint retrospective. The review is for all stakeholders to evaluate the progress of the project, while the retrospective is for the team to improve the way of working together.
  Many teams use many or most of the processes and meetings required by Scrum. But that does not necessarily mean they are true to their character. For example, the retrospective is often facilitated by tools to evaluate the overall mood in the team, followed by a neverending discussion, possibly a set of improvements that nobody ever looks at after the meeting is over. So processes and tools are chosen over individuals and interactions, violating the first point of the agile manifesto.
• For the sprint planning, teams try to predict the size of task that should get done during the sprint. Since every developer knows that time estimates are usually wrong, the more typical forms are now story points or t-shirt sizing to emphasize the fact that it is just an estimate.
  The origin can be traced to different agile methods. Scrum does not explicitely state how a sprint is to be planned, but since its length is fixed and you want a set of tasks as a goal, it is clear that you need some kind of prediction. Extreme Programming has the concept of user stories to describe a unit of work, as a replacement of traditional specification documents. User stories are supposed to contain enough detail to be able to estimate the time it takes to implement, and that time also defines its size: if it takes more than three weeks of implementation, the story is too general and needs to be broken down, if it takes less than one week, it is too detailed and should be combined with other stories.
  Ron Jeffries has a very good article summarizing the problems with story points where he says: I like to say that I may have invented story points, and if I did, I’m sorry now.
  Predictions and goal parameters are my personal pet peeve and I will get back to them below.
• To keep track of the work in the sprint and for future sprints, teams often use Kanban Boards as a tool.
  There is nothing wrong with that tool as such. But it has become an industry of itself. Job descriptions ask for experience and proficiency with specific tools such as Jira and teams spend hours refining their descriptions of the work to be done (rather than using the time to actually do the work). Kanban Boards are also convenient for management to supervise (or rather oversee) teams. This is clearly a breach with the first principle of the Agile Manifesto. The tool becomes a replacement of interactions.

It is important to understand that software development is not (just) for nerds. Software development lives within a management culture. And in a way, all those difficulties of software development are also true for businesses, where markets, customer expectations and information change so quickly. But managers still stick to traditional procedures that work (more or less) for industrial production, but are by no means adapted to individual requirements and ongoing change.

So the resulting practices are what many call cargo cult. You take words and shallow procedures from agile frameworks (that by themselves are not really agile) and hope for the best. This doesn't work, but at least nobody is to be blamed, because the company has tried to be agile.

In his article Playing to Win [1], Gary Klein presents an interesting model of how to improve performance: you want to reduce errors and improve by gaining insights. But most organizations focus on avoiding mistakes. This can be done nicely with following process models and best practices. But in the process of avoiding mistakes, many organizations fail to leverage insight, creativity and opportunities. As Klein writes Managers don’t have the same tools for fostering the up arrow [insights] as the down arrow [avoiding errors]. They don’t know how to increase insights. And, besides, if workers fail to gain insights, no one will notice, whereas errors are public and invite blame.

Stop Optimizing

So the current practices are a foul compromise between the original principles of agile development and the cozy habits of both managers and developers of following well-defined process models and avoiding mistakes (and thereby blame).

The point of agile development (or management) is that there is no fixed process model to follow!

Repeat this sentence to yourself. Do it before and after each meal and include it in your bedtime prayer. Repeat it until you believe it.

As argued above, software development means to create complex systems that interact with other software systems and users, that change over time, that need to be adapted to new technologies and requirements. This is what Gigerenzer (and before him Savage) calls ``large worlds'' [2], i.e. setups where you do not know all the parameters and cannot make accurate preductions. In such a setup, optimization fails to produce the best outcome (as Gigerenzer has shown empirically in many instances).

What do I mean with optimization? The idea of optimization is to find the best possible result, in a mathematical sense. The mathematical procedures for optimization need a model of what needs to be optimized, and that means knowledge about how some action will change the world. Sounds abstract? Here is how this gets into the agile mixup procedure described above:

• Sprint planning means to define a goal, the state of the software at the end of the sprint.
  (See my article Goals Considered Harmful on my issue with goals.)
• To plan the sprint there will be some kind of prediction of how long it will take to implement a feature, so the team knows how many features (or other tasks) can be included into the sprint.
• This plan implies a kind of liability: you promised to do this, so why haven't you?

The main issue is prediction and the expectation of your predictions to be correct. As human beings we love to predict. And for some instances prediction is possible. If you have manufactured a thousand cars, you can probably predict pretty well how long it will take to produce the 1001st. But think about cooking a new recipe. You may have a rough estimate of what it will look like, maybe there is even a time estimate, and the recipe can sound very simple when reading it. But while doing it, you may not be fully sure of the size your are supposed to chop the carrots or whether the consistency of the sauce is creamy enough to go on to the next step. You find the problems the moment you are in the situation.

And this is the everyday situation in software development. You have to cook a new recipe every day, often not even being sure which recipe it is. And even if you have done it before, it was in a different kitchen with different tools. You never know what awaits you. And we can never know about the bugs we build in. Sure, we can put that on the programmers: just be more careful and don't put bugs into the code. But it is not so easy. Coding means to define what you want into the last little detail. And often the problem is not the coding, but forgetting about certain details (why would a user type in a letter instead of a number? this value can become negative?) or failing to see interactions with other components of the software. This is the price we pay for the complexity and the power of our software.

If you try to apply optimization in unpredictable situations, you will never get what you want. So what can you do instead?

The magic word is trust. Trust in people. The agile manifesto assumes that software is developed by talented individuals who are not only technically savvy, but also understand the requirements of the software from a business perspective. As human beings we are very capable of dealing with uncertainty. We are not optimization machines, for a good reason. Our inbuilt decision-making capabilities can deal with change and with unpredictable situations. We just have to relearn to trust ourselves and our fellow human beings in a world where optimization is considered the only way of achievement.

Process models, agile frameworks and rules can help, but only if they are well understood and adapted according to the situation. Trying to replace human judgement by tools or procedures is never a good idea.