Fighting undead documentation

I loathe maintaining documentation.

Why? Most documentation is undead. 👻

Invariably documentation becomes out of date and forgotten. It appears to be living. However, as you peel back the layers, you recoil in TRUE HORROR: the undead, zombie documentation, appearing correct, but far from the actual truth.

Try as we might, nobody has time, or frankly interest, to keep software documentation up to date. Without real, focused, care docs languish like a mummy in a spooky crypt, ready to place a curse on any misguided adventurers that stumble into its depths.

Such outdated, undead, documentation curses readers with a false illusion of knowledge.

Executable Documentation - Do X, Expect Y

Instead of dusty scrolls buried in a crypt, Keep a living spellbook 📖.

Unit tests, notebooks, and tools like doctest tell us to “do X and expect Y result”. In addition to being testable documentation, these docs present devs a very clear black-and-white promise of behavior. When devs say personum-shrinkum, they expect their buddy to shrink. If it doesn’t, you’ve given those devs a very clear fraud.

On Shopify’s search team, we had such a spell book.

We documented how to run a search relevance experiment in several Jupyter notebooks. I dislike Jupyter notebooks as a general coding environment. Yet, when it comes to code documentation, it comes with benefits. Notebooks can be constructed with a clear do-this-expect-that kind of flow. You can automate their testing. You can visualize expected results from code. You can even sprinkle assertions alongside your prose to clarify what the expected state of the universe should be to help coach users with any secondary setup or side effects you need.

Whether unit tests, doctests, or some other thing, spellbooks make easily falsifiable promises to other developers. These promises help you cleanly express the state of the world prior to the incantation, the incantation itself, and the expected outcome. Ideally these are testable, but even if they’re not, they’re still better than most docs that just draw how boxes connect with each other.

Artifacts, Not “Documentation”

Late night, coding, devs can be tempted to walk into dangerous crypts, only to find a Google Doc masquerading as living information.

Consider, though, if our devs saw such outdated documentation in its true form - moldy, yellowing pages and heard its cracked spin. Those devs might feel delighted, rather than upset. If such docs were more transparently of a different era, devs would have little expectation they represented the current state of the world.

Understanding a document as a historical artifact, not a promise about today, changes its usefulness. You can place it in context, and trust it exactly as far as you should given its age. Further, the original authors don’t feel a burden to maintain it. Unburdened people may actually write more, creating more limited-use artifacts, and less “documentation”, that attempts to describe the world.

Let’s feel OK creating artifacts, and avoid documentation. Timestamp it. Leave a note to clarify its intent. Are these meeting notes? A notional proposal? The state of the system at a point in time that’s since evolved?

Such “artifacting” really comes into play when we think about source control history. Detailed Pull Requests, commit messages, and associated documentation tied to a commit at a date can be extremely powerful forms of documentation. Let’s tie our documentation, as much as possible, to source control history rather than being flat and supposedly eternally representative of the “present”.

Even for such documentation we have good intentions to maintain, keeping a changelog can help ensure we know its provenance. As participants in a project, we can both clarify / maintain the past, while we also try to understand the present.

Documentation for wider audience - a blog, not a wiki

Undead documentation hates being brought out into the light. 🧛

A narrow problem, on your specific project, can often be turned into generally useful information to the broader public.

You’ve probably heard you should describe a problem as minimally reproducible for anyone. But the same applies to solutions too. Many technical problems aren’t really about your specific codebase, but something most developers would find useful.

Blogging, Speaking, or otherwise sharing what you’ve learned publicly can be more valuable than updating a wiki page. You can transform a narrow Wiki page read by 1-2 people, into a blog article, consumed by dozens, hundreds, or more.

This accomplishes a few really important things for your team:

1. Seeding Google - You’re more likely to search Google for a problem than your wiki’s search. I HAVE googled a problem and found my own blog articles!
2. Feedback - You open your writing up to feedback and corrections from a broader audience, resulting in better documentation. They’ll ask you to clarify what you’re talking about, ask for code that reproduces your description, give suggestions, and help you communicate better.
3. Creates an artifact - Blog posts and videos of talks typically come with a timestamp. You’re not expected to maintain them, so they’re treated as an artifact of a point in time, not a promise about today

Intentional documentation culture design

We decide how our team documents software. It’s not sufficient to just put a Wiki up somewhere and point the devs at it. It’s also not cost-effective to create complicated documentation bureacracy to test and enforce documentation.

For most teams, the best we can do, is to be thoughtful documentation creators. Put documentation in the light, not in shadows. Make it executable, and test it. Make it public, and invite scrutiny. Or clarify the document as historical, not impermenant, in nature.

Documentation, buried in shadows, just becomes undead. We may be better off not even creating it in the first place.

As always, feedback is welcome!