After 15 Years and 500,000 Lines of Code, Complexity Was Always the Worst Enemy

Steps We Took to Simplify and Streamline Development

What’s the biggest lesson from building and maintaining web apps?
👉 Complexity is your worst enemy, and it’s caused by yourself.

As a tech lead developer and founder, it's easy to fall into the trap of thinking that you need complex and sophisticated architecture to succeed and progress. You may need the latest and greatest technology or advanced workflows to stay ahead of the competition. However, this thinking can be dangerous and lead to unnecessary overhead and complexity.

As a founder and collaborator in recent years, I’ve been part of companies’ efforts to modernize legacy solutions. Most of these processes were already underway and shared one common trait: they quickly became overwhelming and overly complex.

The reason for that is quite simple. There's too much to learn and change in a narrow timeframe, while daily business is hunting one anyway. The outcomes of these migrations can be expensive, dangerous, and simply overwhelming—something an SMB should avoid going through and experiencing.

The way into a too-complex outcome

When you are on your modernization journey, most people aim to acquire the skills to implement like:

• Containerization like Docker as a new foundation for software development

• Continuous Integration and Deployment (CI/CD) is a workflow automation that substitutes manual FTP uploads from the past.

• Cloud is an infrastructure solution that mostly comes from on-premise infrastructure.

In general, these topics are essential and wrap up what I have experienced SMBs and Startups tend to see as a complete plan initially. But there's a catch that will inevitably lead to unwanted complexity.

People working with modern software development methods for years will notice quickly that those topics are just scratching the surface. Starting with Docker, CI/CD, and Cloud in mind is a great goal, but getting there is complicated at first:

• Containerization without Orchestration is just half of the story, and orchestration can quickly become an expensive burden when it's poorly tailored to the needs and capabilities of a company.

• Continuous Delivery is excellent but requires a lot of experience, maturity, and a living DevOps culture. Implementing DevOps is a process that takes time, and specific mindsets and skills are needed to develop a flexible dev culture like DevOps successfully.

• Cloud is a general term, but in reality, Cloud is a large field of technologies, services, and platforms with many different providers. Do you just lift and shift into the Cloud or prefer Cloud-Native? What about Serverless? Managed Kubernetes for a smaller company, or is it still too complex when "IT-Ops" is not an in-house resource? What is the right pick for you? Early on, the wrong decision can run the company into high technical debt.

These are some examples of how the plan gets shattered early on when assuming this topic is simple. I saw frustrated CTOs and developers and was one myself a while ago. The articles of this series should help others get ideas to avoid running into possible traps themselves.

Simplicity is essential, but how do we get there?

Reducing infrastructure to a minimum is a crucial step so that, in an ideal workflow, the developer can handle infrastructure by himself or herself.

The challenge is not to avoid modern ways of development; it's to find a simple way to make complex systems easy to use in daily business. What counts for a small company is a fluent and flexible development process, which constantly adds, in small steps, to the company's outcome without becoming a problem.

When planning your development workflow, ask yourself if you need a specific step, especially when additional people with different skill sets are required to accomplish the step successfully in daily business.

Ensure you understand the "Why," which will often be the development of a product. Thus, a continuous flow is critical, and everything else should evolve around this. Reducing infrastructure to a minimum is a crucial step so that, in an ideal workflow, the developer can handle infrastructure by himself or herself.

Regarding review processes: Do you really need additional review steps? In most cases I've experienced, the problem was not the developer "misunderstanding" a task; it was the task not being precise and simple enough and leaving too much room for questions.

• Lay out the shortest possible flow from a development machine to a production platform.

• Use only services you really need, but consider using them managed. Especially for an SMB, GitHub Actions can be a better pick than GitLab on-premise. Remove the point of failure from the beginning.
  Hint: I prefer integrated CI/CD pipelines in PaaS platforms, which do what we need in a simplified yet complete way.

• Reduce the necessary personnel and skills to a bare minimum. Only implement technologies and processes that can be utilized by the average developer on your team.

• Understand your final goal and why you need to achieve it this way.

• Grow a DevOps Culture early on. I mean the original meaning of it, not IT Ops.

Keep your pipelines and infrastructure simple and understandable.

The larger the scopes of the updates are, the more you will need to run all tests as a requirement for Deployment.

I often see small companies starting with their pipelines by setting up an on-premise Jenkins, GitLab, or TeamCity. I did it years ago, but I recommend it today only if you have a good reason. On the other hand, giving away the burden of running the infrastructure for the pipeline itself is a good relief because you still have many different challenges.

Keep the pipeline as short as possible.

For example, if you have a web app consisting of some services and a static website, make sure you build them, run your tests, and deploy the artifacts straight after the change was pushed to git, even if the deployment target is a staging environment, ensure you don't implement any manual breakpoints like reviews or unrelated tests.

If you have long-running end-to-end tests, avoid setting them as a required step during the pipeline. Instead, make sure you can fast-forward a change to production in case something goes wrong. To ensure quality, you can implement a staging environment to continuously deploy changes and run your tests. That works like a debounce and provides you enough time to deploy multiple times daily to production.

Regarding E2E-Tests: It indicates a poor development process if you depend on E2E testing during Deployment. The "E2E-Testing" should not be done during the Deployment but during the development cycle before the dev pushes the change. Thus, E2E should not be required in a pipeline!

Deploy in small badges.

The smaller the changes are, the less damage they could produce, which adds to simplicity and flexibility. With a short pipeline and minor changes, you gain performance and actionability, which will lead you after a while of practicing an actual Continuous Deployment workflow. Rule of Thumb: The larger the scopes of the updates are, the more you will need to run all tests as a requirement for Deployment.