It's common sense in the agile community that design is emerging during the development. Some extremist state that modelling before coding is not necessary at all. Well, that is not true. But is it necessary to create design documents to capture the models?
Every problem solving for even a little complex problem is done by an evolutionary process. Evolutionary means that variations are created and the best variant is selected. If design is a solution for a problem, then the process to get to a design will look like this:
- The developer creates variations of possible solutions in his brain. By playing virtually with it, he finds and selects the best solution.
- Then the developer will collaborate with other developers and compare the possible solution. They select again the best solution or may even find a better one. This step creates a common understanding.
- The final step is that the solution is put into the code. That really proves it and the common understanding from the second step will help if multiple developers implement the solution. During this the solution may even be amended iteratively by going back to step 1 and 2.
First of all it's important to note, that the inital step is to create a possible solution in the brain of the developer. That requires creativity and experience. Good developers will come up with good solutions. But nobody is perfect and other developer with different experience may come up with other solutions.
In the second step the solutions can be merged together. That works best with synchronous collaboration and a modelling technique. An asynchronous review process (send review comments via email) is inefficient. Also, developers can come up with solutions that wont be selected - they are wasted. It's a nature of the evolutionary process that variants are wasted. The question is, how much effort has be invested into the variant and is it possible to create a variant with less effort.
The third step will transfer the solution - at this point it is only an idea of a solution - into the code. Only with the code the solution becomes real.
In which part of this process do we need design documents? While a developer thinks about the solution, he can write it down. That can be done with a word processor, modelling tool or paper and pen. He is maybe so bright that he actually don't need to write it down. Given that in the second step the solution variant may not be selected, he should invest not much effort. But he needs something to show the solution to others. In the ideal scenario, he has it in his head and draws it on a whiteboard by talking to others. After the second step the developers may want to fix the common understanding, because it may take some days to implement it. You could take a white board photo and store it on the wiki. After the third step the solution is in the code. The common understanding may have changed a little.
The light weight documentation (paper/pen, whiteboard, photo) may not be useful when you have large systems and bigger changes. But that would require a more high level design, aka architecture. An architecture is more long term, may change but not quickly. Also a developer cannot implement the architecture in a few days. It is more a guidance for lower level designs. Architecture and high level design documents can be reasonable. And - documentation for later maintainance is reasonable as well. It depends on the effort to what level starting from the highest level you want to create the documentation.
I have found a post on the State of the .NET Culture. It reminds me on a project which I did recently in the role of the Scrum Master. There were some .NET developers in the team. We took over a large website that was running on Sharepoint and that was developed by another supplier for a fixed price. Our goal was to get it under control and to deliver regular releases to the live environment. The code was not very good and there were no automated tests. We started with the first Sprints and could see very soon, that we cannot deliver the expected velocity.
The developers had no expecience with the XP software engineering practices. No test driven development. No refactoring. Although visual studio has some build-in refactorings, they did not use it. We were trying to find the .NET developers on the market with this background, but were not successful. What I also found difficult to handle was Sharepoint. In order to have a developer to start developing, you need a Windows Server with a Sharepoint on it. It was difficult to set up. All that lead us to the conclusion, that we should replatform the website to Java or PHP.
Yes, of course the software was bad quality in the first place. Maybe with the right skills to build good .NET software one can be a well paid developer. Microsoft makes it very hard to achieve this. With a JDK, Eclipse and Maven I can be much quicker. But maybe that's possible on .NET too?
A few weeks ago I had a nice chat with a senior manager in a large IT organization. This company is transforming to agile methods, first in R&D and now in IT. He is part of the enterprise architecture and IT governments group. His fear is that with more and more agile teams, the group wont have an influence on the project's architecture decisions.
One can argue that in the pure agile world, the architecture does no come from architects. It will be done during the project from the team. Okay. What level of architecture are we talking about? System architecture or enterprise architecture? These are different levels of planning. One build houses and the others plan the city.
But what is the goal of enterprise architecture? Is it that just the projects have to follow a plan? No. The goal is that the architecture supports the projects, that future projects have less costs assigned. In agile words: the enterprise architecture must improve the velocity of a team. Hence enterprise architects are stakeholders. They must work with the product owner to on long term goals.
I honestly think that the enterprise architecture must get its input from the business strategy. If you have a business strategy that is somehow a longterm goal of the business, you can align the architecture to it. But which company has a well defined strategy? Sometimes the business does not have a strategy. All they say is that IT must be flexible. What if in an agile company the enterprise architectures takes a main source of input from the projects. Enterprise architects work in the project with the team and the product owner to form its high level plans for the architecture. Yes - enterprise architects must get involved.
A Scrum Master is a role that is supposed to support the Scrum Team. In general a Scrum Team is a self-organizing system. Systems can be stable or instable. I would see a Scrum Master as the one who is responsible to stabilize the system. A Scrum Team should also follow a goal, that is somehow a lower level goal of an organization, i.e. the company that the Scrum Teams works for. It's also a resposibility of the Scrum Master, but mainly of the Product Owner, that the goal of the team is in line with the company's goal.
Having said that, we cannot describe the Scrum Master as a servant of the team. He is a servant, but he is more than that. A Scrum Master must have a potetial to influence the team members. According to business sciences we have five sources of power:
- personal charisma
- expert knowledge
Which of the sources of power can be used? To create a self-organizing team, the Scrum Master cannot use punishment, because that will kill it. The Scrum Master has the legitimation to control the Scrum Rules. But the rules are made by the team and the team may choose rules that lead to the wrong direction. The Scrum Master must lead - with personality, knowledge and rewards.
I found the blog of Bredex, a small company in the north of Germany I have worked 10 years before. They started to blog last year and have quite some nice posts on it. One post diskusses whether feature driven development is a double edged sword. Alex wrote that they experienced in a project that focusing on feature delivery is causing problems in the software quality. Refactorings that must be done to keep the software maintainable will not be done, because the team wants to deliver features. Yes - that is the main goal of agile software development - deliver a high business value quickly.
Recent financial crises must teach us another aspect. We can live a high quality live on the cost of the future. Some like to overspend and increase their debt. The same you can find in software projects. If you try to deliver as much features as you can, you will increase your technical debt. Even without much pressure from as business department a team can be overcommitted and will deliver features on the cost of quality. The bad quality can be visible to the customer, due to too many defects in the delivery, or it can be unvisible due to an awkward design. The last one is technical debt.The first one - high defect rates are usually tackled with proper conventional quality assurance. The technical debt can be tackled with contiuous refactoring.
But how can you measure your technical debt? By automated tools that output software quality metrics. One of these tools is dependometer - Valtech's open source solution. Another very good tool is Sonar from codehaus.
If you are planning a sprint and you have delivered 15 story point and your technical debt is not increasing, then you are running on a reasonable velocity. But if the quality metrics show you an increase of the technical debt, then you are too fast and you should put less story points into your next sprint and plan tasks to refactor or clean up the code.
Some questions pop into my mind now: How do I explain this to the customer? If I report the velocity, they will see a drop of it. So better not to report the velocity? Or start with a low pace assuming that you can get faster in later sprints? Maybe we can report the quality metrics too - defect rates and technical debt. It really depends on our customer's nature.