Design docs are one of my favourite things about software engineering. If code is the bricks and mortar, then these docs are the blueprints.
I know this is fairly controversial, but our jobs isn't just to write code. Navigating organisations and achieving consensus between a lot of teams/technologies is a huge part of it.
Design docs are a way to get all of that out of the way _before_ writing thousands of lines of code. The review process nets significantly different feedback to code reviews too.
> Navigating organisations and achieving consensus between a lot of teams/technologies is a huge part of it
I think this is a major benefit of design docs - they are a way to extend your engineering influence beyond your own individual output. If you write a design, and your design allows you and three other engineers to coordinate your efforts, then your engineering output is now "I coordinated a team to build something that any of us couldn't have written individually."
This dovetails nicely with your next point - uncovering blockers as early as possible is critical when coordinating a bunch of entities. Project plans are usually written on the assumption that every task will succeed, but there may be extra tasks added. If a task cannot be completed / you need to redesign something / etc, this will suddenly bring the work of N engineers to a halt. The earlier you successfully split the work into a series of known unknowns and implementation tasks, the better the project will go in general.
> ... uncovering blockers as early as possible is critical when coordinating a bunch of entities.
I think the challenge for me has always been that the "uncover blockers" piece means building one or more small prototypes to prove out the capabilities of the dependencies, check feasibility, etc. So the building of these prototypes occurs prior to or in parallel with the authoring of the design doc, but then at a certain point they get paused so that the design doc can be completed and reviewed, and then picked up again when it's time for the "real" implementation to occur.
But pausing there takes discipline, since it ideally happens at the exact moment when all the main blockers have been cleared away and it is maximally tempting to just step on the gas and start into the work of cobbling the prototypes together into the project.
It's also important to set clear expectations with stakeholders who have seen the prototype and may think the project is 90% done, when in fact there's still 90% more to go in making that prototype production-ready.
>If code is the bricks and mortar, then these docs are the blueprints.
That analogy falls apart quickly. Design docs aren’t specific enough to be analogous to blueprints.
You can give a set of blueprints to 3 different construction firms and get fundamentally the same building. Try giving design docs to 3 different development shops and see what happens.
The problem is that the only way to get to that level of specificity is with code.
Design docs are closer to something a city planner would produce than to something an architect or civil engineer would, and they should be treated accordingly.
I mean, you're right... but the goal of an analogy isn't to be perfect, but to be a rough mental model to quickly express a concept. I think it's fine.
Different jobs require different levels of abstraction and design docs fulfil that role for software engineers.
Yes models can be helpful even if they aren’t perfect. But not only is this one too far off to be useful, it does more harm than good. The expectations that happen when people (specifically managers) start thinking of design docs as blueprints, software architects as architects, and developers as builders are downright dangerous.
This is something that should be taught to folks while they're still in school. The "blueprint analogy" has baked in the idea that there's a "design phase" and a "construction phase" and that these are often discreet parties.
Nothing could be more incorrect when it comes to software. In software, the design is the code. The compiler/interpreter are the construction of the system.
> In software, the design is the code. The compiler/interpreter are the construction of the system.
That's IMO a pedantic view. Yes, ok, but when I design software I describe it's high-level characteristics, more like describing the plot of a book than the specific words in a book and the what the characters say to each other.
I'd say there are many levels of design. Code is the last.
Your design can look very good yet be seriously flawed when you actually write the code. Designs (blueprints) civil engineers use are based on many years of practice that are enshrined in engineering codes sometimes going back thousands of years. Key issues in the design like the way forces interplay, are amenable to mathematical analysis. The only checking we can do of UML-like diagrams and written "specs" is peer review. Not the same at all.
What happens when there is a flaw in the design? There's another higher level of design on top of that, with a platonic ideal of the design unknown to humans at the top? Turtles all the way down.
Each stakeholder has a different "design" in mind, and until you actually get specific there is no design, there's just a nebulous, incomplete list of requirements. And if you do try to get specific enough to be reproducible--you're writing code.
None of this is to say that design docs are worthless, just that they can never be specific enough to function as the actual reproducible design the way blueprints would. Thinking of them that way is harmful.
Reproducibility is just a side effect of lack of specificity, which is the major difference and the entire point of my critique. Design docs don't tell you enough to actually build the thing you want to build.
Assume you hold the view that a design document provides all the information necessary to construct a piece of software without any further design work.
Construct a software development methodology around that assumption. Now remove that assumption, and think about the practical consequences for a team who doesn't realize that the assumption is incorrect until after they have started implementing a design document.
Imagine what would happen if you hired an architectural firm to draw up some blueprints for a house, and then midway into the construction of the house it turns out that the assumptions made in the blueprints are wildly out of sync with what's possible to build.
It is significantly better for everyone involved if management goes into the process knowing the limitations of design documents and up front design.
For more concrete example of the practical problems see the history of the debate between iterative and big design up front methodologies over the last 30 years.
I think they were using "design" as akin to the blueprint to the building. I totally agree with them that in that sense, the code is the blueprint, not the building. The execution of the program is the analogy to the actual construction: the builder (computer) takes the blueprint (code) and builds the building (executes the program) based on it. The power of computers is that the "building" (execution) part is essentially free and infinitely reproducible. But the creation of the "blueprint" (code) is still labor intensive.
The right analogy for design artifacts really is to the higher-level vision definition for a project. Engineers don't just show up and start drawing blueprints; one or more people come up with a purpose and concept for a building project, brainstorm approaches, evaluate trade-offs, come to some consensus on the right direction, and only then start working with engineers to start creating and iterating on blueprints. This vision phase is where design docs fit.
"The design is the code" does not follow from the idea that design and construction are not separate things in software.
The thing to take away from that idea is that the design doc is not law written in stone, but a base for changes. It keeps your design honest on the high level. If you end up clashing with reality, and your design doesn't work out, you can backtrack to the design doc and refine it as necessary.
In short, the design doc should be a living document.
I think what you're talking about here is an "intention document" not a design document. Design is about planning. Which implies you define things in detail with steps and techniques. Intentions are about what you want to achieve. Very different.
The design doc is definitely the plan. But if the plan doesn't work when faced with reality, you need to change the plan, and therefore also the design doc.
> the design is the code.
And this is why Software development should never be considered engineering. Architecture is important and in the construction phase there are changes and revision to the design that are captured and formally reviewed.
In software, if the code is the "design" then it should be a priority for computer science to agree upon a standardized design representation (UML) that can be generated by the compiler/interpreter, along with code coverage and static analysis.
The issue here is that there has not been adoption of formal rigor of design and definition of implementation into code of proof of correctness alongside data structures and algorithms appropriate to the domain state. Add in understandable and usable build and testing tools.
The infrastructure is there, with CI/CD, and a repository of github/gitlab, team foundation, cvs code and say something like a successor to GPT3 to do the grunt work of analyzing code to determine optimal patterns of implementation vs existing code bases and development sprint branches.
Even compiling code or "making" it can be a crapshoot for extremely senior and competent developers.
The Code is the Design and the design uncovers good or bad written code. GoF design patterns were modeled after well written code. Anti-patterns (the swiss army knife is my favorite) are bad written code.
The analogy actually makes sense if you’ve worked manufacturing things from blueprints.
If you hand one complex blueprints to three random construction firms, from a distance you’d get the same result, nearly, but up close a lot would’ve changed during the project.
There is a reason why engineers are required to inspect the project at some interval and perform quality checks.
At some level sure there will be differences, but for the end users, unless something went wrong, they won't be able to tell much of a difference.
Blueprints for a house are much, much closer to being reproducible than design docs for a software project.
If you give design docs to 3 separate dev shops, the end results will be wildly different.
>There is a reason why engineers are required to inspect the project at some interval and perform quality checks.
It's true that there is still some room for interpretation because blueprints are still a model. But most of what your talking about is because people will cut corners and not follow the specifications, not because the specifications aren't there.
A well written piece of code will only tell you what it does, a good suite of tests will teach you how to use it, but only natural language documentation - be it in comments or separate documents - will explain to you why the code exists in the first place, and why it looks the way it does.
(As for "self-documenting code", unless a lot of your functions contain the word "because" in their name, the code isn't really self-documenting.)
I think -- from experience -- that in traditional IT organizations this is absolutely not true. The PMO is responsible for business requirements, and almost not time is invested in the creation of detailed DDs. You're essentially left trying to move from a PRD to writing code ... and if you have the luxury of an in-house SQA team and a relatively effective CI/CD process, you depend on quick feedback from stakeholders to certify whether the guesses and assumptions made by developers are correct or not. Since in many cases the business owner(s) had never actually considered many of the edge cases (stupid, basic example: previous company required CFO approval of all international travel requests. Tool was built with this logic. No one considered how to handle it when the CFO was unavailable for whatever reason, so the first time the CFO went on vacation it created all kinds of stupid chaos while a manual process was created.), flexible logic paths aren't designed into workflow apps. Similarly, for reporting/BI tools, there are frequently gaps between the answers executives want to glean from the data, and the business processes the ultimately result in the data creation. Because of this, nearly all reports are faulty, but unless you're close enough to the processes you don't have explainability and uninformed business decisions can result. Ditto from CRUD apps, where form validation rules can be insanely complex for stupid reasons, with the result being the data entered is impossible to use.
Apologies for the diatribe. Big fan of design docs, but an even bigger fan of software engineering cultures that focus on simplicity and usability, rather than being everything to everyone (or worse, being a pet project of an exec who changes their mind every quarter about how things should work).
Yeah that read like a lot of baggage, some of the term I'm not even familiar with (SQA, DD?)
As it stands now, we take high level feature design docs from PM, and turn them into service. Everything between that, resources, development, operations are handled entirely in team where everyones title is SDE. This placed a lot of communication, writing, designing on us but I'm not complaining.
I think this myth comes about because early in peoples' careers, the expectations of the job are a lot more focused on writing code to execute a vision defined by someone else. It is easy to get the impression from this that writing code is viewed as the most important part of the job. But in reality, it works this way because the opposite is actually true, the more important non-coding parts of the job are being entrusted to more experienced people (often to their chagrin, because writing code is way more satisfying and fun).
One thing I've noticed as I've got more experienced was that when I started out, a PM or senior engineer would give me a task to do.
I achieved the outcome, mostly with bad to average code that slowly improved over time.
When I finally understood the domain to make bigger scoped decisions, I started doing design docs beforehand and my code continued to improve greatly.
Now when assigning out work to more junior engineers I find myself giving them a high-level design doc, with some detail missing, they deliver higher quality work than I did at that stage, and they also seem to upskill faster.
This however depends on me making the right decisions at this stage, which is not always the case, so not fool-proof but an overall software quality improvement has definitely occured.
1. Start with defining the problem you want to solve and identify the stakeholders. Then meet each stakeholder 1-1 and get approval.
2. Propose an API. Then meet 1-1 again for approval.
3. Make a high level arch diagram. Seek approval again.
4. Proceed with actually writing the design doc and begin the formal and official review.
Points 1-3 could be done in a week if planned well and will make the design doc a lot less controversial and get you through the review process quicker
- Have a single person (engineer) who is responsible for making final decisions. Gather input from others but there should be one individual who is responsible for making sure that the design has a consistent vision that meets the goal. If a non-critical decision is taking too long, just make a choice and run with it.
- Be clear about the goal of the project. At Amazon they use the PRFAQ process where you need to begin by writing a costumer-centric press release that basically details why customers should be excited about the project. This helps keep design discussions focused on the customer.
- Be clear about the non-goals or things you will not do.
- At the beginning of a review meeting, have everyone read the document. The document should describe all relevant context. At the end of reading the doc, everyone should have enough context to understand and discuss the design - everyone should have the same understanding of the design (i.e. no confusion about which version of the doc we're talking about, no one who didn't have enough time to read the doc beforehand and just skimmed it).
- Have people add comments as they read through it, the authors can respond to the comments in real-time and once everyone is done reading the doc, the comments act as a list of things that need to be discussed.
- Have the doc be the absolute source of truth. If it's not in the doc, we're not planning on doing it. This prevents cases where you have a chat with someone about an idea, look into it further and decide to not do it/do it differently, but the reviewer thinks you are still going to do it, but you just haven't updated the document yet.
The review process has a human factor in it IMO. I’ve seen many designs get scratched just because “this should be done and designed by Team B rather than you”. Problem is that could come late. Sometimes it could be a career development bummer for the engineers who invest a lot into the design.
But isn’t that better than code (especially if working) that gets scratched? I think designs that are scratched indicate you don’t understand the value you contribute to your company (or that management sees in you) which should be the root concern for the career development.
Yea, designs are cheaper then actual implementation. It’s not always the case that the engineer doesn’t understand the value but most times it’s about the management/PM can’t get things finalized over a long period of time. Sometimes I feel it’s just politics in some cases. Don’t you ever need to fight to do some high visibility work?
This is a good question. It's funny, I'm the technical co-founder at my startup, so I hadn't thought to make the distinction between high and low visibility work. I guess we're small enough that I think about it as high impact/cost and low impact/cost. My non-technical co-founder will praise an engineer for their work when the product metrics are moved, which has happened for mundane tasks, too. I tend to notice the craftsmanship more, but I think it's right for attention to be given to the impact.
I have also been at a larger company, so you bring up a good point about fighting for work. I also think this is right.
1. If you win the fight for high visibility work, that's a good sign that you're the right person for the job. It wouldn't make sense for the company to have you work on something for your own benefit over the company.
2. You may opt not to fight and discover other problems to work on that have strong potential for being high visibility. Maybe that's something relegated to PMs or eng management, where the engineer might feel helpless to the given assignments, but that's more of a culture problem. I think politics increases when there's less work/budget/praise than there are people, which is more situational than cultural, in my opinion.
Agreed. On the second point, it’s definitely a bit cultural thing of the group/org. Worth note that at the higher level of the Corp ladder, there’s indeed fewer work(with the wanted scope) then the people (that’s looking to do next level work)
This often betrays an understanding of blueprints to builds that just isn't true. The blueprint of simple things is how they are built. Even complicated things, this will be roughly true for new things.
For old things and complicated things, though, they were how things could be attempted to be built. And with the builder being someone else, there had to be an audit from the build to the blueprint, if you really want confidence in that statement.
I work in an agile/scrum team. Do you have any experience in tracking the design doc work in this type of environment? It's hard to estimate how long a design doc would take since investigation into the solution and conversations with stakeholders could expand it.
But you have to estimate to know how much work you'll fit into a sprint, right? Nothing happens if the estimate is wrong really but it helps fit an amount of work into a period of time
At work, we call them blueprints :). A fantastic way to get early feedback before writing any code, especially important for junior teams who may forget to think about failure cases and operability.
What engineers actually do is determined by the incentive structure they work in. If shipping is prioritized all the time above everything else, one of the things that will suffer is documentation.
If people get promoted in spite of not doing a good job of documentation, now you know why it doesn't happen.
I know this is fairly controversial, but our jobs isn't just to write code. Navigating organisations and achieving consensus between a lot of teams/technologies is a huge part of it.
Design docs are a way to get all of that out of the way _before_ writing thousands of lines of code. The review process nets significantly different feedback to code reviews too.