Agile, Design and Buddhism

Why this blogpost?

Google returns more than 100 million hits for "agile development". So why add another blogpost to the collection?

Agile development is often misused as a buzzword to demonstrate that a company is modern and innovative. But you are not agile if you just get a certificate as a Scrum master. For being agile you have to change your mindset. Only if people understand, accept and appreciate this mindset, will they profit from the advantages of agile development. For me the agile mindset is very much related to the mindset connected to Design Thinking (another misused buzzword) and the basic ideas of Buddhism.

Why agile?

When you approach software development, the first step will probably be that you start some small project by yourself. You have an idea, you try to put it into code. You have more ideas and add them to your code. If you just keep adding code, you come to a point when each change or addition takes ever more time. Some decisions that you made at the start of your project may have made sense back then, but with growing functionality, they may have become outdated. The structure of your code does not automatically adapt to its size.

The pain of growing code becoming unmaintainable is even worse when you work in teams. You spend more time reading code and working around early decisions that are not ideal any more for your current project structure and size. This, of course, costs money. So companies are trying to avoid this situation.

The first attempt was the typical western-minded approach to plan everything ahead. If a growing structure is a problem, we try to plan ahead and find a structure that will satisfy us not just now, but also in two or even ten years' time. To do this, you have to develop a very clear and detailed concept of all project requirements and set up a structure that you will never have to change. This approach is known as the waterfall model: you spend a lot of time to analyse the requirements and design a software architecture (i.e. a specification of how different parts of the code interact). The wishful thinking of executives is that with the specification documents that fall out of this time-consuming and costly process, the implementation is a simple translation from detailed specification into code. It must be so simple that you can just send it to anyone in the world (preferably in low-wage countries) who knows how to write code. Then you test the software and release it. To add a tiny bit of realism, it is allowed to go one step back. So when you find an error when testing, you are allowed to go back to the implementation phase to fix the bug. But you are not allowed to question the basic architecture, also because this would be an organizational challenge. The steps of the waterfall model are supposed to be executed by different specialists. And the requirements engineer that has written the requirements specification has long moved on to other projects.

You may have noticed my depreciation of the waterfall model. It simply doesn't fit reality. I have long tried to use planning algorithms to build useful robots, until I realized that planning works only in very restricted, artificial situations. Humans hardly ever plan. Even if you think you make a plan, you basically just give yourself a rough guideline. So when you "plan" your weekend trip, you will fix on a destination and possibly book some transport and a hotel, but you will decide on the spot what to have for breakfast, what activities to do, and which photos to take. And the reason why humans hardly plan is that it is no use in a world that is constantly changing. The same is true for software. It is impossible to known what your software will look like in two or ten years. The technologies will have changed, fashion (in terms of user interfaces and technology) will have changed, legislation will have changed, and most importantly: requirements arise when users use the software. User experience designers know that asking users about what they want is hopeless. Observe people and let them use a product for the tasks they are supposed to fulfill, and you know the real requirements [1].

Agile development offers an alternative to approach the problem of growing code bases. It rather goes back to the simple model of starting a project and adding on to it. But in agile development you do not just add code, you also change and remove code. The basic idea is that your code develops over time, along with the requirements and changing conditions. This sounds simple, but is hard in practice, especially in western societies where waterfall-like thinking is the default. For some strange reason, people try to make fixed plans when problems get hard, even though they are not doing this for solving the complex problems of their everyday life. The unique human trait of being able to plan ahead seems to get sometimes out of control when we overuse it. Agile does not forbid planning, but it focuses much more on the present than on the future. While the waterfall model tries to solve complex software with a rigid formula, agile relies on the individuals to be aware of the situation and make good decisions.

Why Buddhism?

For agile to work, all stakeholders have to commit to an open, flexible and empathetic mindset. Instead of dreading change (as you do in the waterfall model), you appreciate new challenges. It is also a question of mutual trust, especially trust in the capacities of the development team. For agile to work, you need phases where you restructure your code. During this time, clients and users will not see any difference, but it is necessary to get back to a stable code base that will then allow for implementing new features quickly. Developers must earn the trust that those phases are necessary by making sure they are not just satisfying their own vanity to build the most beautiful code, but be aware of the trade-off between code quality and costs.

Awareness, openness, empathy, and a focus on the present moment happen to be qualities taught by Buddhism, mostly through the method of meditation. I think those teachings can help everyone to develop the mindset that is necessary for agile development to work.

The following parts of this blog series use the structure of Buddhist teachings [2]: the ground lays out basic principles, the path provides specific guidelines of how to obtain an agile mindset, the fruit points the way forward with a reminder that the path is difficult.

Agile software development is considered the latest and greatest. It is not that new, however. The Mainfesto of Agile Software Development [1] is now 20 years old and its ideas are much older. In their 1978 book "The Elements of Programming Style" Kernighan and Plauger provide a collection of simple principles that convey similar messages, especially of how to write and rewrite good code. The principles of the design thinking hype range back into the 1960's [2]. Buddha did not think about software development, but his more than two-and-a-half-millenia-old wisdom is aimed at dealing with life in general, which includes solving complex problems.

Past, Present and Future

Responding to change over following a plan [1]

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. [3]

Agile development—unlike the waterfall model—faces the fact that the world changes. Change can come from the outside, like changes in legislation, trends or fashions. But it also comes from the inside, from your and your team's experience, mood and understanding.

And both types of changes are natural and good! While changes in the societal or economic environment are often perceived as a threat, the changes in your own mind are considered downright failure. If you redesign your software (be it the user interface or the internal structure of the code), one can hardly avoid a feeling of disappointment. Why didn't I do it right in the first place? Even without explicitly formulated new requirements, we have to accept that finding a solution is always a process. And such a process includes changes of mind. There is nothing wrong about a growing understanding, it would be wrong to ignore your current state of mind and just stick to an old plan that doesn't work any more.

You shouldn't chase after the past
or place expectations on the future.
What is past is left behind.
The future is as yet unreached.
Whatever quality is present
you clearly see right there, right there. [4]

Keep the system uncluttered with extra stuff you guess will be used later. Only 10% of that extra stuff will ever get used, so you are wasting 90% of your time. [5]

If we accept that the world changes, our focus will move away from the future towards the present. We could claim that as intelligent humans we can foresee the future, but if we could do that, there would not really be any change to fear as the future would already be known and be somehow fixed. The future may not be fully unpredictable, but there are many factors inside and outside of ourselves that we cannot know in advance.

One of the really hard tasks in software development is to balance out past, present and future. Since many of us are used to overplan, the simple YAGNI pricinple ("You aren't gonna need it") is a good guideline to help us focus more on the present than on the future. But of course, there are aspects of your program that you cannot change on a daily basis. The larger a program becomes, the harder it is to change its general architecture. This is why the agile manifesto places change higher than plans, but it does not forbid them. Now and then you have to sit down, ponder about the learnings of the past, the requirements of the present, and some vague ideas of what lies ahead.

Also the past can take its toll, coming back to the issue of changes. Your present program structure or the libraries you have been using may have served you well in the past, but they may not be adequate any more. Changing such basic aspects in a project is a risky and costly issue, since you may have to rewrite a lot of code. The ideal of agile development claims that your code be simple and clean, but revising even the best code imaginable takes time. And such phases of code restructuring are especially hard to justify, because clients and users do not see any difference, but they do see your invoice.

Part of the agile mindset is to accept these quandaries as a normal part of software development. I think the biggest problem is that developers tend to question their own ability (why didn't I do it right in the first place?) and therefore have a hard time communicating and justifying needs for change. But I can assure you: you are not alone! If you find you need to rewrite code, it is a sign of strength that you are aware of what you are doing, not weakness.

Empathy

Individuals and interactions over processes and tools
Customer collaboration over contract negotiation [1]

Another factor that unites agile development, design and buddhism is their focus on people, including users, team members, clients, and ... yourself!

Buddhism teaches that the place to start is always you [6]. When you understand yourself and accept yourself with all your traits you can support others. All individuals are interconnected. The way you think about and treat others, will always come back to you in some way. Benevolence or loving kindness is practiced as a form of meditation. This does not only mean that you are kind to everybody, it also means that you put yourself into the shoes of others to better understand their motivations [7].

Whether you start meditation practice or not, the ideas of trying to understand others in terms of their goals, experiences and also moods, is very helpful for any kind of communication. If your client pressures you to get some feature done, she may be in competition to some other department. Or if a team member criticizes your sloppy coding style, he may have trouble at home and his harsh words may have nothing to do with you. Empathy for others will not solve all your conflicts directly, but it offers you new ways to think about a situation and find creative solutions.

When it comes to user needs, design (thinking) comes with a toolbox of techniques [8] . Agile development has picked up some of them, for example in form of user stories [9]. It is helpful to know and apply such techniques, but never use them as a standard recipe. Be always aware of what you want to find out and use your empathy to understand what your users want and need (wanting and needing are not always the same, but both have to be attended to!).

Non-Attachment

The insight of Buddha is summarized in the Four Noble Truths [10] [6] :

1. Suffering is an innate characteristic of life.
2. Suffering is caused by craving/ desire/ attachment.
3. One can break free of suffering.
4. The Noble Eightfold Path is the way to achieve this.

Whether you are seeking enlightenment or not, being aware of your attachments can help you make better decisions when developing software. We all have established opinions and habits, which are generally a good thing, but can make us blind to other opportunities.

Attachment is the base problem of the things we have discussed above: if you attach yourself to your decisions in the past, you may not be able to recognize their present inadequacy and live on with an inadequate solution or code that becomes hard to maintain. The disappointment you feel when you have to revise your former decisions, say on your architecture, comes from your attachment to your past experiences and to the attachment of trying to prove to anyone that you are perfect. In design you sometimes hear the advice to "kill your darlings". Decisions from the past, and especially those that you are particularly proud of or that you have been accustomed to for a long time, are not good just because you made them. Be prepared to let them go when you or the world have changed into something that needs a different solution.

This part has focused on the common principles of agile development, design and buddhist teachings. In the next part we discuss how to apply them in practice and how to grow your awareness with every line of code you write.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. [1]

Both agile development and buddhism emphasize the importance of the individual. One reason why projects using the waterfall model usually fail is the intrinsic assumption that people are "human resources". Just as you can interchange one coin for another you can interchange a programmer by another. But nothing could be further from the truth. I have advised more than a hundred students on theses and projects, and each was a unique mix of talents, skills and personality. As we have learned in the last part, the source of all suffering is attachment. So if you adhere to waterfall-like approaches, detach yourself from the idea of interchangable people.

The following pieces of advice are supposed to help you find your own path. There are no recipes that you can follow along. Never switch off your brain, but be aware of what you are doing and why. This is also important to keep in mind when you read about agile development procedures such as Scrum or Extreme Programming. All of them contain useful tools, so it is worth knowing them, but never follow them as a dogma.

Observe

Ultimately, happiness comes down to choosing between the discomfort of becoming aware of your mental afflictions and the discomfort of being ruled by them. [2]

The fourth Noble Truth of Buddhism tells us that we can end suffering by following the Noble Eightfold Path. We don't have to follow it all along to become good programmers, but at least the first step can definitely help us: Right view or right understanding [3][4].

This step advises us to always try to see the true state of affairs. Buddhism openly denies the existence of one universal truth, so it does not mean that we try to establish total objectivity. It means that we should become aware of our own predjudices and habits and try to see the world through different lenses.

Meditation is the most general technique for learning to be more observant. By quieting our mind and consciously observing our thoughts, physical state and feelings, we start to realize that they are just one way of looking at the world. By doing so, we will encounter negative thoughts and feelings, maybe even physical pain. The goal of meditation is to not judge these observations. I am feeling tired today, alright good to know. I feel ashamed because the structure of my code does not fit its present purposes, how could I be so stupid? Oops, don't judge! Just let us say these are my thoughts and feelings, there is nothing right or wrong about them, it is all just going on in my head.

Whatever passes through your mind, don’t focus on it and don’t try to suppress it. Just observe it as it comes and goes.[2]

One of the values of Extreme Programming is Courage [5]. It takes a lot of courage to face all your thoughts and the things that are going on around you. It is so much easier to hide away from unpleasant truths, but if you want to build good software, you better build it for reality. Your users annoy you as they are criticizing the colors of the user interface when you spent so much time making the input process as efficient as possible? That is ok, be annoyed, but do not shy away from the fact that other people have other priorities than you. Because only when you recognize a problem can you try to find a solution.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. [1]

In agile development, you explicitly observe your development process. One of the hardest decisions is how much time to spend on what. It happens that it takes you ages to get a library to run only to realize afterwards that the fancy animation or modern-looking widget does not solve any problem on your agenda. Or you find yourself implementing feature after feature and after a while you stop being productive because you are overwhelmed by errors that get ever harder to find and fix because you have not taken the time to clean up your code when you added to it. Whatever you do, it is impossible to find the perfect use of time, but this should not keep us from trying to improve. Again it is important not to judge. We usually try to avoid admitting that we spent too much time on this or too little time on that. But the fact that you have found it out is a success. If you realize these instances, you can use them to learn from them and in the future become more aware of how you invest your time. This takes time for everybody, so don't be hard on yourself, but celebrate your progress.

One of the earliest lessons I was taught by my father was that Buddhists don’t see the mind as a discrete entity, but rather as a perpetually unfolding experience.[2]

Experiment

With observation comes openness to other points of view, and thereby to different solutions. Design methods emphasize that you explore different paths before you pick one solution. In any design (thinking) book or website you will find a plethora of techniques for ideation, sketching and prototyping. Here is just a quick overview:

Ideation is the process of coming up with possible solutions. The most well-known one is brainstorming, which, however, is controversial as to its efficiency and effectiveness. If you want to use brainstorming, you should have a look at Ideo's 7 brainstorming rules [6]. Ideation is the practical instantiation of the buddhist idea of openness. Just let any idea happen without judging, you can always discard it afterwards. Allowing any idea to enter your mind will help you leave your normal way of thinking and funny ideas can sometimes be a step on the way to really good ideas. There are design techniques that you can practice even without a specific problem at hand or as a warm-up exercise for an ideation session. For example, when you are waiting for the bus, try to imagine as many possibilities as you can of why the bus is late (anything is allowed from traffic jams to the driver having been kidnapped by aliens). This way of thinking is called divergent thinking and is used by psychologists as a measure of creativity.

When you develop different ideas, you usually want to materialize them somehow, because your brain will not hold more than four of five ideas in short-term memory. Often we write them down (if you want to look professional, use colorful sticky notes), but you have other options like drawings (no pieces of art, just to get your idea across) or even physical entities, built from paper clips, sticky tape or anything else you have at hand. In the coding world you can start with some kind of pseudo code just to get a feeling of where it gets tricky.

Write first in an easy-to-understand pseudo-language; then translate into whatever language you have to use. [7]

Sketches are a simplified version of prototypes. You can get philosophical about what counts as a sketch or a prototype. My distinction is that sketches are for yourself or your team, while prototypes are a bit more refined and can be discussed with clients and users. A prototype can range from a drawing on paper (illustrating a user interface or a software architecture) to a running program. In agile development actually any program is a prototype, because one is never really finished. Instead of splitting hairs over the terminology, let us concentrate on the purpose of prototyping. Prototyping is about reducing uncertainty. You have an idea of what your solution will be like, but there are many factors that you cannot figure out at the drawing board: will users accept your user interface? Can state-of-the-art speech recognition libraries cope with a Swabian dialect? Can the data you measure from some machine really predict when the machine will break down? Especially for central questions of your task, try to answer these crucial questions as quickly as you can. So if you want to build a system with speech recognition, get samples of (real!) speech and run them through recognition libraries before you start to implement your dreams of what is happening after the utterance has been recognized.

Conversely, make sure you only prototyp what you do not know. Try to leave out as much as possible aspects that you already know. For example, you want to develop a program where people can upload a photo and you visualize different hats on their photo. The question here is not whether you will be able to come up with lots of funny hat variations, or whether you can find a smooth process for users to upload a photo and choose a hat. The question is whether your algorithm is good enough to really recognize human heads on any kind of photo (or you find out what the prerequisites for the recognition are so you can tell the users).

Build the spike to only addresses the problem under examination and ignore all other concerns.[8]
[Spike solution is a term for prototyping from Extreme Programming]

So when you develop a solution, be open to alternatives and evaluate alternatives with everything you have: use your experience, your rational thinking capacities, and try things out. And be honest: if you cannot get your speech recognition going in your experiments, do not stick with the idea and hope to get it right in the final product (because you won't), but get back to the drawing board and ask yourself why it had to be speech recognition in the first place. Maybe there are other solutions to fulfill the same purpose.

Also note that some aspects are hard or impossible to prototype. AI topics are a tyipcal instance. If you want to use machine learning to predict failures of some hardware component, you have a chicken-and-egg problem: for machine learning to work you need data, but gathering data is expensive and you would like to know which data you need. To a certain degree you can start with few data, visualize them and get a feeling of what is possible. But statistics is tricky. It is well-known from science that measured effects often disappear the more data you gather. So if you spot something in a small dataset, you can simply be lucky. In such situations use your experience in combination with prototyping and accept that reality is tricky.

Balance

Experimentation methods are mostly aimed at preparation: you think about possible solutions and you reduce the uncertainty in them. But do not forget that at the end of the day you have to deliver a working product, not a summary of the fifty ways that do not work. So you have to find a realistic balance between thinking and doing.

Ariely and Norton [9] show that people can think both too little and too much. Thinking too little might mean you sit down to code without even having understood the problem. Thinking too much can result in being stuck in a waterfall-like thinking phase that never ends. And as hard as it is to accept for executives: there is no silver bullet to solve this dilemma. All you can do is to gather experience and rely on your intelligence.

Working software over comprehensive documentation [10]

Thinking and doing is just one aspect that needs balancing. The four principles of the agile manifesto [10] demonstrate this by their formulation with "over". The balance between software and documentation goes back to the consideration of past, present and future. The code is there, it is the present state, while documentation most of the time shows either the past (if your program has evolved, but the documentation has not, as is often the case) or the future (in case of specification documents).

Also be aware that agile development promotes "working" software, not "beautiful" software. Agile methods are somethimes used as an excuse to tune code just for the sake of having beautiful code. Code quality is important, but do not try to measure it with simple metrics such as test coverage. The quality of code and the tools you use to ensure this quality depends entirely on the situation, they need not even be the same over your whole project. You may have a rather stable core of functionality of which you are rather sure that it is not going to change a lot and that is the basis for the rest of your code. In such situations, you should invest time to really make it stable and maintainable. Other parts may be evloving more quickly. Don't write tests for code that is about to change, you will have to code everything twice. If you are attentive to how your project evolves, where you have trouble with bad code ("Don't comment bad code — rewrite it." [7]) or are rewriting tests all the time, you will find the right balance of code quality and development efficiency.

The last part was about the path towards an agile mindset. So what awaits you at the end of it?

If you expect to obtain superhuman skills for writing software without errors in a jiffy, I have to disappoint you. Software development is and remains a slow and expensive process. Accept this. The difference that agile development makes compared to waterfall-like procedures is that the slow expensive process leads to useful stable software. Agile development also takes away some risk of software development. By its iterative nature, you always know what you have and where you are. In the waterfall model too much time is usually spent analysing and writing specifications. The time for implementation, testing and bug fixing is usually underestimated in the first place, and reduced by overrunning the timelines for analysis and specification. With an agile approach, you may not always be able to implement all the features that you wanted to, but you will not end up empty-handed.

Obstacles

Nobody will contradict when you suggest to use agile development methods for a software project. From the executive to the trainee everybody agrees that agile development is the only way to go. And everybody claims that they are being agile.

If you look closer, most projects define architectures, interaction concepts, APIs etc. before the first line of code is written. Projects start out with developing whole design systems for the user interface without knowing what the program will do and how users will interact with it. Do you hear the water splashing? Talking about agile is not the same as thinking and doing agile. But talking won't get the job done, so we better get started with the hard part.

The problem with agile is that it sounds reasonable, but it contradicts our thinking habits. I do not know whether this is due to our dualistic, rationalistic culture or whether all humans tend to fall into the waterfall-trap as soon as a task gets complex. It may have to do with our urge for security. When we make plans and draw architecture diagrams we get the feeling of having control over the problem. And plans cannot really go wrong (nobody ever looks at them afterwards). Reversely, when you code, things go wrong all the time. You thought some feature would be a matter of half a day and you end up trying one library after another for two weeks. Or you thought you had finished something, but it produces errors over and over, whatever you do to fix them.

Another obstacle is our estimation of time. Thinking out a good concept of a software is a hard task, but it is relatively fast compared to the implementation. The common expectation is that by drawing an architecture on paper you have done the main job (the "brainy" job if you want). With a good concept it should be a piece of cake to implement it. But it is the other way round. Programming basically means to laboriously translate everyday concepts into mathematical notation. The real difficulties are usually discovered while you are programming, so your abstract concept may become outdated. When people solve problems in everyday life, they constantly switch the level of abstraction. When you plan your vacation, you skip between the big picture (when will I be where) and the smaller picture (I want to visit this temple). Contrary to our intuition, people do not plan top-down, but it is an intertwined process [1][2] where your overall trip may be influenced by the opening times of the temple you want to have on your way. Hoping that your abstract architecture concept solves the problem in one go is an illusion. The abstract concepts have to evolve with the implementation.

While users and clients may be impatient to get as many features on their screen in as little time as possible, you may have to argue in exactly the opposite direction with team mates who are trying to produce "clean code". Stable, well-managed code is the basis of agile development, but often techniques are overinterpreted as fixed rules without considering the context. For some programmers (and executives) testing routines become an end in themselves. Or fixed processes such as two-week sprints. Or pair programming and code reviews. All these techniques are useful in the right context. Testing routines by themselves do not make better code (they can actually make it worse because most of your time goes into adjusting tests instead of programming the main thing). Pair programming can make sense especially to learn from one another, but it is not the technique for everybody in every situation. Here again a basic human motivation kicks in: the want for simple rules. Thinking is tiresome and we all wish for simple routines that take away the strain from thinking by ourselves. Fixed rules have the additional advantage that you are not to blame when something goes wrong, the rule is to blame, not you. Agile development only works when thinking individuals follow a common goal, rules can never replace mental effort.

All those thinking patterns are not just coming from others. You would not be human if you didn't carry them in yourself. In addition, most of us are plagued by self doubt about our abilities. Why did it take me so long to find this simple bug? Why didn't I get my code structure right from the start? Why am I the only one who has not heard about technique XY? You can be sure that every programmer has exactly the same doubts as you have. The more I talk to others the calmer I get, because we are all struggling with the same issues. You should always try to improve your skills, but see it more as a process than as a goal. No matter how much you know and how skilled you are as a programmer, there will always be more to learn.

Liberation

To cope with the obstacles on your path to an agile mindset, first accept that they exist. And then, slowly, take one step at a time.

Your own mindset is the place to start. If you know exactly what you are doing and why, it will be much easier to justify your approach towards others. Your to-do-list will always be too long to ever be worked off, so you have to prioritize. If people want you to do this or that, give them a chance to show empathy by sharing your to-do-list with them. When being asked "can you implement feature X", I ask back: "is it more or less important to you than feature Y that you asked for last week?". Make people aware of time and resource constraints and the implications of different options ("if I do not clean up this messy code now, every upcoming feature will take me twice as long"). It also helps to remind people of their decisions ("it was you who wanted this feature, I told you it would take up all my time for four weeks") to enhance their learning process.

If agile development is so hard, is it worthwhile? — Yes! Your reward will be running software that you can be proud of. Often the easy path is to just follow along in the waterfall-thinking process and fail along with the whole team. This makes your life easier on the surface, but in the end you waste your time. Agile development is a constant struggle with yourself and others. It will make you stronger and more self-confident as a person. And when you take it on, you can make tangible contributions.