After 15 Years and 500,000 Lines of Code, Complexity Was Always the Worst Enemy

Steps We Took to Simplify and Streamline Development

What’s the biggest lesson from building and maintaining web apps?
👉 Complexity is your worst enemy, and it’s caused by yourself.

As a tech lead developer and founder, it's easy to fall into the trap of thinking that you need complex and sophisticated architecture to succeed and progress. You may need the latest and greatest technology or advanced workflows to stay ahead of the competition. However, this thinking can be dangerous and lead to unnecessary overhead and complexity.

As a founder and collaborator in recent years, I’ve been part of companies’ efforts to modernize legacy solutions. Most of these processes were already underway and shared one common trait: they quickly became overwhelming and overly complex.

The reason for that is quite simple. There's too much to learn and change in a narrow timeframe, while daily business is hunting one anyway. The outcomes of these migrations can be expensive, dangerous, and simply overwhelming—something an SMB should avoid going through and experiencing.

The way into a too-complex outcome

When you are on your modernization journey, most people aim to acquire the skills to implement like:

• Containerization like Docker as a new foundation for software development

• Continuous Integration and Deployment (CI/CD) is a workflow automation that substitutes manual FTP uploads from the past.

• Cloud is an infrastructure solution that mostly comes from on-premise infrastructure.

In general, these topics are essential and wrap up what I have experienced SMBs and Startups tend to see as a complete plan initially. But there's a catch that will inevitably lead to unwanted complexity.

People working with modern software development methods for years will notice quickly that those topics are just scratching the surface. Starting with Docker, CI/CD, and Cloud in mind is a great goal, but getting there is complicated at first:

• Containerization without Orchestration is just half of the story, and orchestration can quickly become an expensive burden when it's poorly tailored to the needs and capabilities of a company.

• Continuous Deployment is excellent but requires a lot of experience, maturity, and a living DevOps culture. Implementing DevOps is a process that takes time, and specific mindsets and skills are needed to develop a flexible dev culture like DevOps successfully.

• Cloud is a general term, but in reality, Cloud is a large field of technologies, services, and platforms with many different providers. Do you just lift and shift into the Cloud or prefer Cloud-Native? What about Serverless? Managed Kubernetes for a smaller company, or is it still too complex when "IT-Ops" is not an in-house resource? What is the right pick for you? Early on, the wrong decision can run the company into high technical debt.

These are some examples of how the plan gets shattered early on when assuming this topic is simple. I saw frustrated CTOs and developers and was one myself a while ago. The articles of this series should help others get ideas to avoid running into possible traps themselves.

Simplicity is essential, but how do we get there?

Reducing infrastructure to a minimum is a crucial step so that, in an ideal workflow, the developer can handle infrastructure by himself or herself.

The challenge is not to avoid modern ways of development; it's to find a simple way to make complex systems easy to use in daily business. What counts for a small company is a fluent and flexible development process, which constantly adds, in small steps, to the company's outcome without becoming a problem.

When planning your development workflow, ask yourself if you need a specific step, especially when additional people with different skill sets are required to accomplish the step successfully in daily business.

Ensure you understand the "Why," which will often be the development of a product. Thus, a continuous flow is critical, and everything else should evolve around this. Reducing infrastructure to a minimum is a crucial step so that, in an ideal workflow, the developer can handle infrastructure by himself or herself.

Regarding review processes: Do you really need additional review steps? In most cases I've experienced, the problem was not the developer "misunderstanding" a task; it was the task not being precise and simple enough and leaving too much room for questions.

• Lay out the shortest possible flow from a development machine to a production platform.

• Use only services you really need, but consider using them managed. Especially for an SMB, GitHub Actions can be a better pick than GitLab on-premise. Remove the point of failure from the beginning.
  Hint: I prefer integrated CI/CD pipelines in PaaS platforms, which do what we need in a simplified yet complete way.

• Reduce the necessary personnel and skills to a bare minimum. Only implement technologies and processes that can be utilized by the average developer on your team.

• Understand your final goal and why you need to achieve it this way.

• Grow a DevOps Culture early on. I mean the original meaning of it, not IT Ops.

Keep your pipelines and infrastructure simple and understandable.

The larger the scopes of the updates are, the more you will need to run all tests as a requirement for Deployment.

I often see small companies starting with their pipelines by setting up an on-premise Jenkins, GitLab, or TeamCity. I did it years ago, but I recommend it today only if you have a good reason. On the other hand, giving away the burden of running the infrastructure for the pipeline itself is a good relief because you still have many different challenges.

Keep the pipeline as short as possible.

For example, if you have a web app consisting of some services and a static website, make sure you build them, run your tests, and deploy the artifacts straight after the change was pushed to git, even if the deployment target is a staging environment, ensure you don't implement any manual breakpoints like reviews or unrelated tests.

If you have long-running end-to-end tests, avoid setting them as a required step during the pipeline. Instead, make sure you can fast-forward a change to production in case something goes wrong. To ensure quality, you can implement a staging environment to continuously deploy changes and run your tests. That works like a debounce and provides you enough time to deploy multiple times daily to production.

Regarding E2E-Tests: It indicates a poor development process if you depend on E2E testing during Deployment. The "E2E-Testing" should not be done during the Deployment but during the development cycle before the dev pushes the change. Thus, E2E should not be required in a pipeline!

Deploy in small badges.

The smaller the changes are, the less damage they could produce, which adds to simplicity and flexibility. With a short pipeline and minor changes, you gain performance and actionability, which will lead you after a while of practicing an actual Continuous Deployment workflow. Rule of Thumb: The larger the scopes of the updates are, the more you will need to run all tests as a requirement for Deployment.

Keep your infrastructure stupidly simple.

The worst thing your dev team can face is a broken infrastructure that the devs need help controlling or fixing quickly. Unfortunately, today's requirements to run Kubernetes or Docker Swarm clusters yourself are high. It's not done with just setting them up the first time. Maintaining and keeping them secure and reliable is not just a different skill set than development; it's a high-cost factor and risk for the business. Instead, Platform-as-a-Service solutions like Digital Oceans App Platform or AWS ECS provide highly managed platforms that developers can control. 

Rule of Thumb: If you don't have a particular reason, like special scaling and performance requirements, avoid managing your infrastructure yourself, especially if you need solid IT-Ops personnel.

🍪 By “special,” I really mean special. Don’t fool yourself by thinking that yours must be special :) Done that; it doesn’t work out.

An example of a slow and risky process

As I mentioned, the idea of a fast and flexible process is to boil it down to a workflow you only need developers for. In the following example, I want to display how things are unnecessarily complex during the development cycle.

This is a typical pattern we have faced in the last few years. One thing in common: No one was pleased with it; often declared this way as a pain point.

So what is wrong then? First, there are too many steps for workflow automation, where the goal is to deploy features (values) to the client.

Simple and fast deployment cycle. After working for a while with SMBs, this is mostly the successful end result.