> In my career, I have taken a particular approach based on building new systems from scratch (before they succumb to the three laws), but this is a lot harder than it sounds
This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them.
Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development. We have an existing system. This needs to be maintained and extended because it's our core business. We cannot just say "oh it's become too complex. Let's go and spend the next few years creating a new product, so we'll have a multi-year feature freeze".
The only feasible way I see to do anything even close to this is to split up the software into domain-like silos slowly and incrementally. At that point you can start considering rewriting silos as they become too complex/cumbersome/big. However, at that point you're already doing complex, lengthy and generally risky refactoring, and speaking of it being a write from scratch is very obviously not true.
I agree with you. Too many of us developers tend to forget that the customer only looks at the features. Clean code, unit tests, rigorous processes... they are only as valuable as the rate of features they make possible.
Complexity and technical debt are our problem, and the customer could not care less about how we solve it, or even whether we solve it. As long as we are struggling with an old piece of sh*t, that's good for us: it means the product is selling and the effort to maintain it still makes sense!
Selling without complexity and technical debt would be much better, I think we all agree on this, but it's a receipt whose success I have never personally witnessed.
While it is hard to generalise to an absolute I do largely agree with this sentiment.
The amount of times I have advised clients to not use something like Wordpress to just end up having to use a page builder like Divi hurts my soul.
At the end of the day the majority don’t care about things like carrying bloat/legacy code, rigorous testing patterns, using modern tech stacks or typesafe codebases. They simply want you to “make thing go brr”.
I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities.
I guess the main thing from a business perspective is that I’m getting paid but, at the same time, ouch, my feels.
> I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities.
I think part of the problem here is the "market for lemons" thing: The client can't tell the difference between the short-term-cheap-long-term-expensive option and the short-term-expensive-long-term-cheap option, so can't trust that the expensive option will actually save them money in the long term, rather than being a short-term-expensive-long-term-even-more-expensive option.
Consider getting quotes to re-plaster your house. One builder quotes you £2000, another £4000. Now, is the first builder giving you a fair price and the second builder charging you extra because you don't know what a fair price is? Or is the first builder a "cowboy" who's going to do a terrible job you're going to regret, and the second a craftsman who's going to do an excellent job you'll be happy you went with? Maybe you happen to know enough, but that's not where I was when we were renovating our house.
And consider that the client isn't sure they'll even need a website in 3 years; maybe this new venture will crash and burn well before then. Then we'll be doubly glad they didn't invest in a website with the equivalent of a 10-year guarantee.
> I can advise until I am blue in the face but the reality is that at least 90% of all my clients want the quickest and cheapest option, resulting in me barely even scratching the surface of my technical abilities.
Until more client gets burned with a cheap option that becomes expensive to maintain they won't care. Many clients (across the whole industry, I don't know your clients which could be very different from average) will not get burned as most problems will never get that complex. However most clients have a lot of small problems that are quick to solve meaning that most developers spend their time on the small minority of clients that have complex problems where this matters.
I agree with you though I'd state the cheapest upfront option. They tend to cost more in the long run. See Terry Pratchett's boot theory: https://en.m.wikipedia.org/wiki/Boots_theory
The problem with this theory is that the $50 boot manufacturers sell out to private equity when the founder wants to retire and then the PE firm jacks up the price to $75 and drops the quality below the $10 cardboard boots... so you're out even more money to have wet feet.
Persistent refactoring is one of those things where the cost is clear and well known (expensive developer time) while the payoff is big, but very diffuse, very hard to explain and often gets attributed to something else - sometimes even by the developers who did it.
It's not the only thing like that. Plenty of companies skimp on cost centers and fail spectacularly as a result - sometimes without even ever realizing that skimping on the relevant cost centres was the reason why they failed.
Refactoring only matters if decision makers care about long term consequences and value. One thing that makes it difficult to make these kinds of decisions wisely is that no one knows how long a piece of software will live in production.
If I could know for sure that a piece of software would exist for 50 years in production I could make a better case for refactoring. Likewise, if I knew for sure that a piece of software would get completely replaced in 1 year I would not bother with refactoring.
Never in my career have I been able to accurately predict how long software would live in production. I have had throwaway code live for years and things I thought would last forever get replaced almost instantly.
Refactoring doesn't just benefit you in 50 years it can benefit within weeks.
I agree that it's a waste on throwaway code, throwaway code is common and what ends up being throwaway code is hard to predict, but I don't think it's impossible to get a sense of the likelihood of the code being tossed, and that sense can be used to tweak your refactoring budget.
But if I spend a month refactoring and only get a weeks worth a benefit before the project is scrapped it wasn't worth it. If I get a month back before the project is scrapped it wasn't worth it (I could have been working on something else instead). Unless/until we know how to quantify how much gain will be got back, and how long it will take to get that gain back and so we cannot do a proper economic analysis if it is worth it - but that is what we need.
> Refactoring doesn't just benefit you in 50 years it can benefit within weeks.
Proving the value is the hard problem. It's almost impossible to prove the value of small changes today. Product wants things to do X and they decided the value tradeoff was good, not the engineers who are tasked with making it so.
Thats why I just do it. If somebody wants proof that my years of experience applied in a particular way will provide a quantifiable benefit that is impossible I will just avoid doing that thing.
Or more likely, move somewhere which can appreciate those non quantifiable benefits.
> Or more likely, move somewhere which can appreciate those non quantifiable benefits.
How can you find such a place? This definitely resonates with me.
There's so much value in doing things right, but it's impossible to prove. I can do things that make my team go faster, but there's no way to say that having a X% better tests resulted in us being able to release Y% more features.
Try to give off trustworthy vibes and pick customers/clients/employers who will trust you.
I'm not going to pretend it always worked for me, but it certainly got easier once I could demonstrate a wealth of experience.
There's a lot of luck, too.
In terms of refactoring, I have an iron rule that I never ask for permission. If I think it needs doing I do it. I don't justify it to product or ask for their opinion. Technical issues and priorities are on a need to know basis and they don't need to know. All the work gets baked into feature work and bugfixes.
"customer could not care less". This perspective is too narrow, or even a fallacy, often repeated to justify abandoning some good practices.
Oh yeah customer cares. If I leave my car in a workshop, I care whether mechanics have one big mess there or not, I am looking for signs of a good routine, clever process, some smart choices they made for themselves. If they did a fix for me but have a mess, I will never go back to them.
Most of customers do such choices.
In software, all that is hidden from the end customer. How do you look for the smart choices in code made by the team who implemented any of the software you download and install?
That's obviously different because at the corp level there are politics involved in choosing one vendor over another, but at the same time wheverer I'm involved in the decision process I care to avoid these products as hell.
Well at companies is much less compromise than politics. Over decaces I saw two main schemas:
- product must be purchased because its vendor is somehow connected to the upper management or the investor
- product must be purchased because the person who has chosen the product is at the moment perceived as having some kind of holy cow status and can't be criticized, so people around put that to the extreme
Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on.
> Poor choices during development reveal themselves in many ways: poor performance, awkward workflows, mandatory upgrades, one-way data conversions, inaccurate documentation, and so on.
Not necessarily. Some otherwise good systems might have one bug in each of those categories.
And, most other poorly designed and poorly executed systems just chug along fine without the customer realising anything.
I'm not sure what you mean. jagged-chisel asked how we look for smart choices in the software we download and install. I gave examples of how poor choices manifest for end users, the idea being that a lack of those issues suggests smart choices. I don't see how revenue is relevant to the question. Revenue for whom? And how did my examples miss the point exactly?
> Oh yeah customer cares. If I leave my car in a workshop, I care whether mechanics have one big mess there or not, I am looking for signs of a good routine, clever process, some smart choices they made for themselves. If they did a fix for me but have a mess, I will never go back to them.
In your car, you can tell, in software you cannot.
Add in the fact that most times even cowboy-coded systems are once off and will be replaced (and not maintained) down the line, and you really cannot tell if the system was written as a huge mess just waiting to be exploited, or as a well-crafted and resilient system, because the majority of systems are in-house only and face no adverserial threats other than the employees.
I can tell about bad development habits from using the software it because code smell (and process smell) stinks.
Besides, I was telling not about car repair per se, but about the process around car repairing, which is visible when you visit car workshop and talk to the guys, and is even more visible when you use the software.
Concrete examples:
- installation/upgrade process involving interactive scripts means that somebody thought dumbing down such process for a tiny percent of less-tech users is ok, disregarding the majority who automate everything (and interactive scripts are obviously harder to automate); it also can be a sign that vendor's decision maker (product owner) is disconnected from reality
- poor documentation; a distinct case here is when part of the product is an API and of course "there is" documentation, generated by Swagger - but the crucial part of it are some subtleties of arguments passed which are themselves undocumented, making the whole API simply unusable. The vendor has an excuse "but we have documentation" - heck yes, it's because your employees hate their superiors so they gave them something to f*ck off; and this is given to the customer. Very common practice actually, I can give you a list of vendors who notoriously do that
- painfully slow webapps are a sign of a process smell that no one thinks about the real world operations; sometimes it's not even about scalability, but bad choice/usage of libraries; i need to give here names, in rot13 so it's not googlable: Argfhvgr has a HR product for corporations where every single damn click on any active field in the browser - and you click it a lot because it is fscking calendar - triggers a series of round trips to the server, each taking over 2 seconds. The usability of this product is so braindead at so many levels that I can see a whole bunch of teams failing to stop this sh*t from being released & sold.
Long story but a good example that just by using the product you can see how rotten the vendor must be
A lot of what you're describing sounds like the innovator's dilemma. One of the things that makes legacy systems so difficult is that everything is there for a reason, but often the reason is to satisfy a small minority of customers. It's easy to look at a component and say, gee this feature is only used by 2% of customers, we should consider dropping it. But that 2% is someone at the company's customer and they will raise hell if you get rid of it. So naturally the system gets filled with all these special cases to handle niche customer use cases none of which can ever be removed. This is when a startup comes along and targets the high value basic use cases and drops most advanced functionality. Then they start acquiring bigger and bigger customers, complexity accretes to meet their needs and then eventually the startup is the legacy company and the cycle repeats.
The question as a developer is do I want to work on such a system? For me, the answer is a resounding no. If I agree to work on such a system, I will essentially be enslaved to the bad design decisions made before I joined the company. It will never be truly satisfying to work on that system. I am speaking from experience.
But that is why companies pay a salary. The established company mindset is not the startup mindset is not the hobby project mindset. It's up to you to figure out what is best for you at any time and that choice may change as life circumstances change.
Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen.
A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand.
That last part is also the most common reason why large system replacement projects fail, go over buddget, or are not even attempted.
Often the people controlling the money do not have the technical knowledge how much complexity the old system hides. The safe bet is to keep it going.
> Even systems with 15 years of legacy and eons of manhours behind them get replaced. These projects do not always succeed but I've seen it happen.
Yes. It can happen. If you can't do it with a 99.9% succes rate you cannot make it a general business practice.
> A rewrite from scratch doesn't need full parity with the old system as long as you can discover, implement and test all the critical business use cases beforehand.
Beyond my general disagreement is shouldn't need full parity (man our clients would be pissed off if we cut functionality "because it was easier to rebuild a system that was functioning quite well on your end", and they have guns!), I don't think this takes into full account how much can be hidden in years and years of commits and code. We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely". This, in spite of all handwringing about how this would never have happened if people maintained proper documentation, will happen in any legacy application. We're talking about a hundred people working side by side over decades. These things will slip through, or 'best practices' change so much that things that were a good idea then are entirely unknown now.
Even something as non-intrusive as attempting to stranglevine this will take up a lot of time and effort, if it can be done correctly.
> We have systems being maintained where the team responsible goes "look we have a million lines of SQL stored procs. We don't know what all of them do because most of them were written in the mists of time and there's no documentation, or the documentation is so obviously wrong we can ignore it entirely".
Interesting - when thinking of systems that I've personally worked on which I could never imagine being replaced, the first one that came to mind was one with exactly the same problem: a massive cache of SQL stored procedures at the core that nobody seemed to fully understand. This particular company had even implemented stuff like HTTP auth token validation and all access control with SQL scripts without any centralized access control architecture. It was outright terrifying to touch the SQL.
When I started at the company the manager advertised they "really have no upper limit" on how much they compensate developers who stick with them for years.
I guess a big problem with implementing large parts of the application logic with SQL is that it's simply not a tool fit for the purpose. Incrementally replacing, or even reasoning about, logic splintered accross assorted SQL scripts is very difficult. So nobody has the full picture of what are all the things the monolith does.
> This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them.
> Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development.
I think that what often happens, is that the manifestation of an old system crumbling under the weight of its complexity is that a "new kid on the block" (startup, etc.) eventually takes over. As long as the new thing only forgoes the old cruft very few care about and add stuff many care about, there's a possible takeover.
Entropy is a good analogy for software complexity. In that it takes more energy to decrease entropy than to increase it—is fundamental in understanding why in nature, without external intervention, systems tend to evolve towards greater disorder. As a result, a single engineer can wreck such havoc on a system that it's essentially impossible to repair.
This seems to be the a key to the issue and it's a people problem.
The issues always enter when a codebase has to be extended in some way that was not part of the original design. You actually have to do ‘the big refactor’ right at that point of extension such that it is as if the new requirements were known up front when the system was originally designed. Otherwise you end up with a complicated mess very quickly.
> What seems like an infeasible solution? The article doesn't describe the author's "particular approach".
The entire idea of "well let's rebuild this before it ever gets bad". That only works in the one tiny niche corner of greenfield development that nearly no one works in. To say "oh yeah just don't ever let software get complex" is like saying "well have you tried excising any oncogenic mutations before they cause cancer"?
I actually laughed out loud when I read that conclusion. Not only does it sound infeasible and unsustainable, it also sounds not a little arrogant and likely annoying for everyone else working with them.
Can you imagine saying to your product owner or whatever "oh yeah we're not gonna do anything new. We'll spend the next year or so rebuilding this service because the code looks ugly."
This seems like an infeasible solution in most if not all real world cases I've seen. Especially if you have a system with 15 years of legacy and some thousands of man-years of effort put into them.
Now, the obvious "don't let it get to that point" is a smug IT nerd answer I often see regarding this, but this seems like the kind of green-field startup thinking that doesn't apply to 90+% of software development. We have an existing system. This needs to be maintained and extended because it's our core business. We cannot just say "oh it's become too complex. Let's go and spend the next few years creating a new product, so we'll have a multi-year feature freeze".
The only feasible way I see to do anything even close to this is to split up the software into domain-like silos slowly and incrementally. At that point you can start considering rewriting silos as they become too complex/cumbersome/big. However, at that point you're already doing complex, lengthy and generally risky refactoring, and speaking of it being a write from scratch is very obviously not true.