Companies use nowadays more and more Commercial Off The Shelf (COTS) software products. I have seen many of these projects that launch a big software product within an larger IT organization. My feeling is that it not always helps the organization and that it is usually a very costly endeavour.
I'm looking for explanations, why these projects are causing so much effort. One explanation could be found in the number of dependencies that a software product has to other IT systems. The more interfaces a COTS product has, the more complex is the integration. But why is it more complex than integrating a software written from the scratch? There are two reasons:
- We know less about the internals of a COTS product. Since it's not written by the developers themselves, if a problem during integration occurs, it will be more difficult to find the causes of the problem.
- The COTS product implements more than required. A product must be adaptable in many organization. It come with features that one organization finds useful, others don't. But it contains all the features. It will also have some kind variability build in - to adapt it to all different situations.
Because of these two reasons, the variety of a COTS integration is higher. If a company that integrates a COTS software has to deal with a larger variety, we must have in the organization with a higher variety, to be able to control it. This is due to Ashby's Law, a very important law in cybernetics. If one system wants to control another system than it must have a higher variety.
So there are two ways of dealing with it:
- Decrease the variety of the system that you want to control. We do that for instance in coding software by type checking or information hiding.
- Increase the variety of the system that wants to control the other. We do that in software development by self-organized teams. Those teams are able to have a higher variety than a team lead by a command-and-control style manager.
Okay - so what we know is that COTS will have a higher variety - maybe. If a COTS product has less interfaces to other systems/components the integration will not amplify the uncertainty into the project, and thus may still be manageable.
Related to my last post on the need for documents during the design of software, I came across the topic of Design Thinking. Unlike analytical thinking, Design Thinking is aimed at building up ideas. It fits perfectly to agile methods, because it requires iterations and team work. I would say that agile methods include many parts of the theory of Design Thinking.
Here are some statements that I want to make related to Design Thinking:
- A team should know the theory of Design Thinking in order to support best the generation of ideas. Agile coaches should explain to team, how ideas and solutions are created and what hurdles will probably in its way.
- Our education has formed us to pick the first best idea and carry on with it. That may not be the very best idea to solve a problem. We have learnt that creating bad ideas is unacceptable. Children are very creative and can come up with lots of variations. We learn at school and from our parents what is right and get punished for what is wrong. Therefore we try to create right ideas and hesitate to communicate bad ideas. This blocks the idea generation within ourselves and within a team. If a team can overcome this issue, the creative output of the team will be higher. A team should be aware of this issue and create trustful relationships. This relates to "phychological safety" that is higher developed in a long lived team.
- We also have another problem with intuition. If people bring up ideas, the ideas are discussed in a team to validate and merge them with other ideas. If someone has a intuitive idea, it will be difficult for him to explain it. He or she just knows that it's right. But without explaining the idea, the collaborative process to build a common understanding will fail. Either the idea will be lost - and maybe it is the very best solution - or someone else can explain it. This also relates to leadership. Leaders can have intuative ideas and others may simply follow without understanding it. They trust the leader.
Well, I think that a team that is aware of the principels of design thinking can use it to create better solutions. It may also help the team to create trustful relationships quicker.
It's common sense in the agile community that design is emerging during the development. Some extremist state that modelling before coding is not necessary at all. Well, that is not true. But is it necessary to create design documents to capture the models?
Every problem solving for even a little complex problem is done by an evolutionary process. Evolutionary means that variations are created and the best variant is selected. If design is a solution for a problem, then the process to get to a design will look like this:
- The developer creates variations of possible solutions in his brain. By playing virtually with it, he finds and selects the best solution.
- Then the developer will collaborate with other developers and compare the possible solution. They select again the best solution or may even find a better one. This step creates a common understanding.
- The final step is that the solution is put into the code. That really proves it and the common understanding from the second step will help if multiple developers implement the solution. During this the solution may even be amended iteratively by going back to step 1 and 2.
First of all it's important to note, that the inital step is to create a possible solution in the brain of the developer. That requires creativity and experience. Good developers will come up with good solutions. But nobody is perfect and other developer with different experience may come up with other solutions.
In the second step the solutions can be merged together. That works best with synchronous collaboration and a modelling technique. An asynchronous review process (send review comments via email) is inefficient. Also, developers can come up with solutions that wont be selected - they are wasted. It's a nature of the evolutionary process that variants are wasted. The question is, how much effort has be invested into the variant and is it possible to create a variant with less effort.
The third step will transfer the solution - at this point it is only an idea of a solution - into the code. Only with the code the solution becomes real.
In which part of this process do we need design documents? While a developer thinks about the solution, he can write it down. That can be done with a word processor, modelling tool or paper and pen. He is maybe so bright that he actually don't need to write it down. Given that in the second step the solution variant may not be selected, he should invest not much effort. But he needs something to show the solution to others. In the ideal scenario, he has it in his head and draws it on a whiteboard by talking to others. After the second step the developers may want to fix the common understanding, because it may take some days to implement it. You could take a white board photo and store it on the wiki. After the third step the solution is in the code. The common understanding may have changed a little.
The light weight documentation (paper/pen, whiteboard, photo) may not be useful when you have large systems and bigger changes. But that would require a more high level design, aka architecture. An architecture is more long term, may change but not quickly. Also a developer cannot implement the architecture in a few days. It is more a guidance for lower level designs. Architecture and high level design documents can be reasonable. And - documentation for later maintainance is reasonable as well. It depends on the effort to what level starting from the highest level you want to create the documentation.
I have found a post on the State of the .NET Culture. It reminds me on a project which I did recently in the role of the Scrum Master. There were some .NET developers in the team. We took over a large website that was running on Sharepoint and that was developed by another supplier for a fixed price. Our goal was to get it under control and to deliver regular releases to the live environment. The code was not very good and there were no automated tests. We started with the first Sprints and could see very soon, that we cannot deliver the expected velocity.
The developers had no expecience with the XP software engineering practices. No test driven development. No refactoring. Although visual studio has some build-in refactorings, they did not use it. We were trying to find the .NET developers on the market with this background, but were not successful. What I also found difficult to handle was Sharepoint. In order to have a developer to start developing, you need a Windows Server with a Sharepoint on it. It was difficult to set up. All that lead us to the conclusion, that we should replatform the website to Java or PHP.
Yes, of course the software was bad quality in the first place. Maybe with the right skills to build good .NET software one can be a well paid developer. Microsoft makes it very hard to achieve this. With a JDK, Eclipse and Maven I can be much quicker. But maybe that's possible on .NET too?
A few weeks ago I had a nice chat with a senior manager in a large IT organization. This company is transforming to agile methods, first in R&D and now in IT. He is part of the enterprise architecture and IT governments group. His fear is that with more and more agile teams, the group wont have an influence on the project's architecture decisions.
One can argue that in the pure agile world, the architecture does no come from architects. It will be done during the project from the team. Okay. What level of architecture are we talking about? System architecture or enterprise architecture? These are different levels of planning. One build houses and the others plan the city.
But what is the goal of enterprise architecture? Is it that just the projects have to follow a plan? No. The goal is that the architecture supports the projects, that future projects have less costs assigned. In agile words: the enterprise architecture must improve the velocity of a team. Hence enterprise architects are stakeholders. They must work with the product owner to on long term goals.
I honestly think that the enterprise architecture must get its input from the business strategy. If you have a business strategy that is somehow a longterm goal of the business, you can align the architecture to it. But which company has a well defined strategy? Sometimes the business does not have a strategy. All they say is that IT must be flexible. What if in an agile company the enterprise architectures takes a main source of input from the projects. Enterprise architects work in the project with the team and the product owner to form its high level plans for the architecture. Yes - enterprise architects must get involved.