A peek into the K8s graveyard

Another part of Avocode stack was the Continuous Deployment system for Kubernetes. It was entirely written in-house, meaning it had questionable security practices and long-forgotten dependencies. It also used a docker-based deployment system which we obsoleted with nix.

What we needed

My quest was to find a suitable replacement. In truth, we didn’t even need Kubernetes. Just anything that allowed us to quickly deploy prototypes and protect them with HTTPS and some sort of authentication.

What we got: Dokku

When I started researching our options, I was terrified. I found lots of options for CDs from around the Heroku era - like Jenkins. Then lots of dead projects based on Kubernetes, and only a handful of new ones, which might follow in the footsteps of all the rest.

After careful consideration, I decided on Dokku - a simple service deployed on one host, which turns it into a Heroku-like platform. Good old git push to deploy: it triggers Docker build and start of respective service. I used puppet bolt to configure the host and slightly modify the default service configuration to authenticate all requests via oauth2 proxy. Dokku supports Let’s Encrypt, so it checked all our requirements - and took less than a day to prepare.

It worked flawlessly for demos: I deployed demos for Illustrator parser and Photoshop parser there, and fellow devs added several others. Soon enough though we embarked on a project that required something more.

What we got, but better: Flux

It was time to revisit Kubernetes solutions. Argo CD was the most obvious candidate. After deploying it on a testing cluster (configured via GKE) I had some reservations:

• it implemented its access control layer, independent of Kubernetes - meaning any bug there automatically gave the attacker control over the cluster,
• CLI used Web proxy for all connections - meaning you needed the web part deployed (and exposed, unless you wanted to use k8s proxy),

When looking for alternatives I came upon information that Flux CD has become a CNCF Graduated project - that meant it was likely to continue having support on par with Kubernetes, Helm and Prometheus.

I quickly deployed it on another GKE cluster using the same Terraform / Helm setup I prepared for Argo. I immediately liked it a lot more than Argo - it used K8s annotations as a messaging mechanism between server and CLI, has optional UI and worked using k8s ACL - meaning it was much simpler to lock down and isolate. It became our main CD mechanism for the next half a year - until the larger Ops team took over this operation from us.