After 15 Years and 500,000 Lines of Code, Complexity Was Always the Worst Enemy
Steps We Took to Simplify and Streamline Development
What’s the biggest lesson from building and maintaining web apps?
👉 Complexity is your worst enemy, and it’s caused by yourself.
As a tech lead developer and founder, it's easy to fall into the trap of thinking that you need complex and sophisticated architecture to succeed and progress. You may need the latest and greatest technology or advanced workflows to stay ahead of the competition. However, this thinking can be dangerous and lead to unnecessary overhead and complexity.
As a founder and collaborator in recent years, I’ve been part of companies’ efforts to modernize legacy solutions. Most of these processes were already underway and shared one common trait: they quickly became overwhelming and overly complex.
The reason for that is quite simple. There's too much to learn and change in a narrow timeframe, while daily business is hunting one anyway. The outcomes of these migrations can be expensive, dangerous, and simply overwhelming—something an SMB should avoid going through and experiencing.
The way into a too-complex outcome
When you are on your modernization journey, most people aim to acquire the skills to implement like:
Containerization like Docker as a new foundation for software development
Continuous Integration and Deployment (CI/CD) is a workflow automation that substitutes manual FTP uploads from the past.
Cloud is an infrastructure solution that mostly comes from on-premise infrastructure.
In general, these topics are essential and wrap up what I have experienced SMBs and Startups tend to see as a complete plan initially. But there's a catch that will inevitably lead to unwanted complexity.
People working with modern software development methods for years will notice quickly that those topics are just scratching the surface. Starting with Docker, CI/CD, and Cloud in mind is a great goal, but getting there is complicated at first:
Containerization without Orchestration is just half of the story, and orchestration can quickly become an expensive burden when it's poorly tailored to the needs and capabilities of a company.
Continuous Deployment is excellent but requires a lot of experience, maturity, and a living DevOps culture. Implementing DevOps is a process that takes time, and specific mindsets and skills are needed to develop a flexible dev culture like DevOps successfully.
Cloud is a general term, but in reality, Cloud is a large field of technologies, services, and platforms with many different providers. Do you just lift and shift into the Cloud or prefer Cloud-Native? What about Serverless? Managed Kubernetes for a smaller company, or is it still too complex when "IT-Ops" is not an in-house resource? What is the right pick for you? Early on, the wrong decision can run the company into high technical debt.
These are some examples of how the plan gets shattered early on when assuming this topic is simple. I saw frustrated CTOs and developers and was one myself a while ago. The articles of this series should help others get ideas to avoid running into possible traps themselves.
Simplicity is essential, but how do we get there?
Reducing infrastructure to a minimum is a crucial step so that, in an ideal workflow, the developer can handle infrastructure by himself or herself.
The challenge is not to avoid modern ways of development; it's to find a simple way to make complex systems easy to use in daily business. What counts for a small company is a fluent and flexible development process, which constantly adds, in small steps, to the company's outcome without becoming a problem.
When planning your development workflow, ask yourself if you need a specific step, especially when additional people with different skill sets are required to accomplish the step successfully in daily business.
Ensure you understand the "Why," which will often be the development of a product. Thus, a continuous flow is critical, and everything else should evolve around this. Reducing infrastructure to a minimum is a crucial step so that, in an ideal workflow, the developer can handle infrastructure by himself or herself.
Regarding review processes: Do you really need additional review steps? In most cases I've experienced, the problem was not the developer "misunderstanding" a task; it was the task not being precise and simple enough and leaving too much room for questions.
Lay out the shortest possible flow from a development machine to a production platform.
Use only services you really need, but consider using them managed. Especially for an SMB, GitHub Actions can be a better pick than GitLab on-premise. Remove the point of failure from the beginning.
Hint: I prefer integrated CI/CD pipelines in PaaS platforms, which do what we need in a simplified yet complete way.Reduce the necessary personnel and skills to a bare minimum. Only implement technologies and processes that can be utilized by the average developer on your team.
Understand your final goal and why you need to achieve it this way.
Grow a DevOps Culture early on. I mean the original meaning of it, not IT Ops.
Keep your pipelines and infrastructure simple and understandable.
The larger the scopes of the updates are, the more you will need to run all tests as a requirement for Deployment.
I often see small companies starting with their pipelines by setting up an on-premise Jenkins, GitLab, or TeamCity. I did it years ago, but I recommend it today only if you have a good reason. On the other hand, giving away the burden of running the infrastructure for the pipeline itself is a good relief because you still have many different challenges.
Keep the pipeline as short as possible.
For example, if you have a web app consisting of some services and a static website, make sure you build them, run your tests, and deploy the artifacts straight after the change was pushed to git, even if the deployment target is a staging environment, ensure you don't implement any manual breakpoints like reviews or unrelated tests.
If you have long-running end-to-end tests, avoid setting them as a required step during the pipeline. Instead, make sure you can fast-forward a change to production in case something goes wrong. To ensure quality, you can implement a staging environment to continuously deploy changes and run your tests. That works like a debounce and provides you enough time to deploy multiple times daily to production.
Regarding E2E-Tests: It indicates a poor development process if you depend on E2E testing during Deployment. The "E2E-Testing" should not be done during the Deployment but during the development cycle before the dev pushes the change. Thus, E2E should not be required in a pipeline!
Deploy in small badges.
The smaller the changes are, the less damage they could produce, which adds to simplicity and flexibility. With a short pipeline and minor changes, you gain performance and actionability, which will lead you after a while of practicing an actual Continuous Deployment workflow. Rule of Thumb: The larger the scopes of the updates are, the more you will need to run all tests as a requirement for Deployment.
Keep your infrastructure stupidly simple.
The worst thing your dev team can face is a broken infrastructure that the devs need help controlling or fixing quickly. Unfortunately, today's requirements to run Kubernetes or Docker Swarm clusters yourself are high. It's not done with just setting them up the first time. Maintaining and keeping them secure and reliable is not just a different skill set than development; it's a high-cost factor and risk for the business. Instead, Platform-as-a-Service solutions like Digital Oceans App Platform or AWS ECS provide highly managed platforms that developers can control.
Rule of Thumb: If you don't have a particular reason, like special scaling and performance requirements, avoid managing your infrastructure yourself, especially if you need solid IT-Ops personnel.
🍪 By “special,” I really mean special. Don’t fool yourself by thinking that yours must be special :) Done that; it doesn’t work out.
An example of a slow and risky process
As I mentioned, the idea of a fast and flexible process is to boil it down to a workflow you only need developers for. In the following example, I want to display how things are unnecessarily complex during the development cycle.
This is a typical pattern we have faced in the last few years. One thing in common: No one was pleased with it; often declared this way as a pain point.
So what is wrong then? First, there are too many steps for workflow automation, where the goal is to deploy features (values) to the client.
Put communication before development, not during
No, in most cases, we could route back the origin of this problem in an inadequate, half-baked, and too-large issue passed to the developer.
Communication in the form of reviews is a general problem. Why must you "review" your developer's work before deploying it? Don't you trust the workflow or the developer? Does the developer need help reading the requirements correctly? Is the maturity level too low?
I often hear that you should not interrupt other developers when they work. So do the pull request and the others can then, review that when they have time.
In my opinion, that's wrong, because you, who is requesting the code review, will be out of context, and the other person will be out of context, because he never were part of the context.
So, it's basically double as worse as to just ask in that moment. What is it about three to five minutes? So it should not be that much of a hassle for another developer. And in my opinion, my own experience, when someone quickly asked me something, I am not completely out of my context just because someone asked me something. Of course, this is not ideal, but on some point you need to make a compromise.
The compromise where you get things done quickly and as soon as possible, where you get features out to the client as fast as possible without stacking up a big backlog of code reviews, all this will reduce frustration and improve your flow.
Rethink how we handle pull requests and code reviews.
Transcript
No, in most cases, we could route back the origin of this problem in an inadequate, half-baked, and too-large issue passed to the developer. The task wasn't small and precise enough and was held too general. A single Task for more than 8 hours of work is too much for most developers to be sure to ship what he or she was asked for.
Solution: Make sure a task is so simple and clearly defined that a review isn't necessary to deploy, and the developer can prove that the Definition of Done is fulfilled. Could you keep the tasks as small as possible? A positive side-effect is motivational; when things get done quickly and frequently, everyone feels things are moving. If mistakes happen, they only occur in small scopes. Don't worry; you can always code-review asynchronously.
Avoid Pull-Requests
If you put too many checkpoints into your process, the developers won't build up maturity. Because in their mind, there will always be "someone" responsible before the changes get shipped. That's human nature and not desirable in a deployment workflow.
PRs are a standard method to control and review processes. But what do you actually check? If you cannot trust your developers on quality, work on it. If your definitions aren't good enough, work on that instead.
Of course, you can always refactor later, but keep overall performance as a priority. If you put too many checkpoints into your process, the developers won't build up maturity. Because in their mind, there will always be "someone" responsible before the changes get shipped. That's human nature and not desirable in a deployment workflow.
Think about an alternative for PRs
Review code later and let developers link their git commits to the issues or tickets, but push the changes already.
You will have enough time to work with your developers on quality and refactoring later. Don't branch; go for a typical "develop" branch or trunk-based development. A Developer should feel responsible for his work, and if he or she has any questions, he or she needs to decide when to ask someone. You better use Slack instead of a PR for this.
IT-Ops still provides the infrastructure stuff.
Some companies have IT-Ops or Cloud-Engineers as freelancer resources available, but I recommend putting Ops into your devs' hands. It's a core capability.
If your developers can't describe the infrastructure well, you have a problem and are not a DevOps company yet. The problem is that you have an entirely different skillset as a requirement when you have separated IT-Ops. With that, you need to have these skills in the form of personnel available. Some companies have IT-Ops or Cloud-Engineers as freelancer resources available, but I recommend putting Ops into your devs' hands. It's a core capability.
There are other reasons, like product quality. If a developer knows the Cloud-Native infrastructure well, he or she knows how to develop the services to fit into the platform better; in other words, develop the software natively for the Cloud, for the infrastructure.
Another good reason is the reaction speed. The developer will find problems like performance issues faster in the current app if the dev can overview the whole system instead of only some logs.
Finally, modern software patterns like microservices or serverless functions already depend on infrastructure know-how in the development environment. Therefore, it doesn't make sense to split this into several responsibilities.
But keep the solution easy to handle!
Great platforms provide all you need to start and progress. Platforms like Digital Oceans1 App Platform are purely developer-orientated solutions to manage your services, apps, tasks, deployment process, network, load balancing, etc. For most SMB companies I have worked with, this platform is all they need, even when growing the business. You can always switch to Kubernetes or other orchestrators since the App Platform works well with cloud standards.
Simple and fast deployment cycle. After working for a while with SMBs, I realized this is mostly the successful end result.
Conclusion - Adrian, isn't that a bit too much?
People tend to resist change when they are bound to their habits and develop safety spaces. Even when they face problems regularly, I know that change is still a challenge because I was one of them once, and I predict it always will be.
But do you think a change is necessary? Continuous enhancement leads to simple ways of work. This culture of becoming better removes obstacles, reduces risk, increases performance and output, and, most importantly, reduces mental overhead to prevent people from burning out.
I recommend aiming for the most simplistic way you can imagine. Boil it down first, and then add what's necessary to add. Never add something because it seems to be trendy.
Allow yourself to experiment and celebrate failure. But always keep the goal in mind: to stay simple and flexible.
Others may describe their ways using lean or agile approaches.
I had the best results by focusing on a solid tech culture, the culture-first approach. The idea behind that is to believe in the individuals of your teams and make it easy for them to push your company forward.
Thanks for reading and supporting! Have a great day,
Adrian
