Agile Development: Minimal Doesn’t Mean No Documentation
“We embrace documentation, but not hundreds of pages of
never-maintained and rarely-used tomes”
– Jim Highsmith, History: The Agile Manifesto
The Importance of Documentation & the Consequences of Non-Documentation
A persistent myth related to Agile development is that it shuns documentation. This is due to some Agile practitioners misinterpreting the Agile Manifesto’s 2nd principle of preferring “working software over comprehensive documentation” to mean “no documentation” or “just enough documentation for the Agile team to be productive.” This is a terrible misinterpretation of the methodology.
A global survey on a company’s Agile maturity, conducted by KPMG in 2019 comes to mind. Conducted with 120 participants across 17 countries, the results revealed that 68% of respondents saw the main driver of agility in their organization as a way to provide “faster product delivery adjusted to changing customer needs,” while the biggest challenge in shifting towards agility was related to culture, according to 59% of respondents. Even in locations like the Netherlands, Belgium, and Denmark where Agile is commonly practiced, most survey respondents felt that employees were not yet ready to fully adapt to the Agile way of working. You see, human beings are creatures of habit and old habits die hard. The documentation process in software development is no exception to this.
The reality is that a key Agile practice is to do minimal documentation. Agile teams avoid extensive requirements documentation or technical documentation in a static manner, such as using a word processing tool. At the same time, Agile practices have introduced living documentations spread across multiple artifacts that collectively constitute the entire documentation of the software.
The purpose of Agile documentation practices is to spread the documentational needs across multiple artifacts so that the documentation will be done by the specific roles addressing the needs of similar resources playing the same role. For example, the minimal specification documents written by a product owner or a business analyst, inline documentation written by coders as comments in the code, and technical documents and diagrams created by the technical leads or architects collectively constitute the documentation needed for an Agile project. Further, many project stakeholders seldom consider that automated and manual test cases also play a significant part of Agile documentation.
Considering these shortfalls in the Agile process, we have created a roadmap that practitioners can take into consideration for documentation needs in their projects as follows:
A Recipe for Agile Documentation
When deciding how much and what to document, consider answering the following key questions:
1. Who is the reader?
Documentation is intended for Agile team members who have to act based on the information documented. This could be to fix a bug, further improve the software or introduce a new feature. Therefore, the author needs to have a clear idea of who is going to consume the information they document today.
2. What do they need to know?
Next the author needs to place himself in the shoes of a future reader and understand what that person would need, especially if they are new to the software or the system. Thinking from that perspective will allow the author to put in more information in the documentation that one might otherwise omit and take for granted.
3. How do I make the information audience-specific?
Next the author needs to figure out how to deliver the documented information for a specific future reader. The best way is to think about what situations would prompt the reader to consume the information and to use a tool that resonates well with the consumer’s role. For example, if the documentation is done to help a developer understand the code, chances are that they will be using a code editor, and hence inline comments in the code can make it easy to find the information needed without switching between code and a separate document. However, if the consumer is a product owner or a business stakeholder, it is highly unlikely that they would use a code editor. In this case, it would be best to use a wiki or a word processing software to document such information and incorporate diagrams and screen designs to provide easy explanations to these non-technical readers.
4. What tools do I use to produce the documentation?
How one produces the documentation has a direct connection to the delivery of the information. If one wants to prepare inline comments, then one should use a code editor. If one wants to provide JavaDoc formatted documentation, then one should use proper JavaDoc annotations and structure. Similarly, if one wants to produce UML diagrams, then a diagramming tool must be used while adhering to the standards of UML. It is important that you avoid the temptation to invent or repurpose commonly used standards known by the industry.
Best Practices for Agile Documentation
Here are some best practices to increase the agility of your documentation.
1. Executable Specifications vs. Static Documents
Traditional documentation done on word processing tools are static documents and cannot be considered as living documents. One needs to periodically update these static documents to keep them in line with the software that the documents try to explain. These traditional documents such as requirements specs, architecture specs, and design specs can be captured as executable specifications in the form of either automated or manual tests. By using methodologies like Test Driven Development (TDD) and Test First Development (TFD), you simply need to develop just-in-time (JIT) documentation in the form of tests. Since these tests are executed every time the code changes, the tests are updated, corrected, or removed as and when the code and the system changes. This is how you create living documents that work parallelly with changes made, compared to the slower periodic updates of traditional static specification documents.
2. Generate System Documentation
Modern software-based modelling tools are built to reverse-engineer existing code and present many views into it. In short, it means that you can save a significant amount of money by automatically generating much of the system documentation that you need this way.
3. Document Iteratively
Updating and enhancing Agile documentation must become a part of the iterative development process. By doing so, the Agile team, in an incremental manner, can update, enhance, or change the documentation to match the latest changes done or planned to be done in the near future.
4. Document Close to Implementation
The traditional Waterfall approach is designed to produce all the documentation at the very beginning, based on the customer’s ideas and requirements. However, when it comes time for software implementation, most of the documentation may need to be changed, because either (a) the requirement has changed or (b) there are technical or business challenges that block those requirements from being implemented or (c) the target process has evolved. On the other hand, in the Agile approach, the documentation is done closer to the point of actual implementation ensuring all dependencies, practicalities, and limitations are discussed in detail before requirements are documented. In all other cases related to technical documentation, it is done so post or parallel to development, so there is no wasteful documentation involved either by way of test case creation, commenting, or diagramming.
While Agile projects do not embrace bulky, up-front static specifications, the same level or a better level of information can be provided to customers through more practical, relevant, and easy-to-maintain means in this way.
What is important to note is that project stakeholders and Agile team members do maintain essential documentation processes. The difference is that Agile documentation eliminates the waste of managing bulky documents that are static by aligning the documentation process with the development process, in an incremental and iterative manner, making the available documentation purpose-oriented and minimal.
 KPMG Advisory N.V. (2019). Agile Transformation. From Agile experiments to operating model transformation: How do you compare to others? (Rep.).