u/razr_69

Hey!

We are running with ArgoCD for multiple years. And we constantly struggle with the prmotion of changes between our different environments.

Our Setup

Most of our setups have the following environments CI, NIGHTLY, INT (sometimes), PREPROD/UAT, PROD. On top we have ephemeral environments. But let's ignore them here. They are conceptually different and we have a working setup for those.

A change is developed and tested on a ephemeral environment and merged into CI. The NIGHTLY is a (more stable) nightly copy of CI.

To bring a change in PROD, we first deploy to a PREPROD/UAT environment, that is then used by us and sometimes (depending on the change) some customer superusers to test the new version.

Our Struggle

We constantly struggle with the way we promote changes from one environment to the next. Our "ArgoCD repositories" have one folder per environment on the main branch. Each of those folders contains a folder for each app, that is deployed to that environment. We use Helm directly (not the hydrator-feature).

To promote we have some self-built pipelines today, that copy the application version from one environment-app folder to another. This works as long as the changes are only in the version of the app. Once we have other changes in the Chart config (like new versions of a chart, new or updated keys, structural changes, ...) there is no support in our pipeline. Today, the teams just merge such changes in all environments and keep the apps OutOfSync until the deployment to the next environment is planned.

This brings huge problems: constantly out-of-sync applications hide actual problems, applying hotfixes is blocked by the already made changes, accidental syncs could break production applicaitons, and may more. Nobody likes this process, but there is no better for us today.

GitOps Promoter to the Rescue?

So we started looking into proper promotion tools and we had a look into GitOps Promoter. It looks quite promising, but after testing it, we found some issues, that makes its opinionated process unusable for us.

Chain of Releases

Let's say we have delivered a feature to PREPROD/UAT and it is waiting for approval by the customer. While that is ongoing, we do not stop working on the application. So new features are still merged into main/CI all the time. And each of these merges now creates a new Promotion to CI, which then blocks us from bring the change from PREPROD/UAT to PROD.

The docs have a note about a queue-based model, which might come in the future and could be what we a re looking for.

> Once the "release latest" model is validated in production environments, we may consider adding a queue-based model for users who need it.

Source: https://gitops-promoter.readthedocs.io/en/latest/faqs/#how-does-gitops-promoter-handle-concurrent-releases

I was asking myself if maybe splitting the Promotion Strategy to a DEV (CI -> NI) and a PROD (INT -> PREPROD -> PROD) part would help. And in-between there would be a manual process to merge a dev branch into a prod branch, that backs the respective apps.

Promotion of Multiple Apps

We often have the case, that multiple ArgoCD apps have to be promoted together (they are all in the same ArgoCD repository). The Promotion strategies create one PR per ArgoCD app though. So it is the "mergers" responsibility to merge all the PRs for the environment. I could not find anything, if it is possible to track multiple apps on the same environment branch and hence have one PR for all the apps.

What other solutions are out there?

I checked the Tool Comparison and of course we could build it ourselves. But I would like to avoid that. The part for our ephemeral environments got complicated enough :D

Then there is telefonistka, which is GitHub-specific and we are running on-prem.

Kargo is a tool, that caught our eye. Though we are a bit hesitant since it is "owned" by Akuity and not in the CNCF. And today there are already some features only available in their SaaS (like ArgoCD UI integration).

CodeFresh GitOps I did not really get. But it seems this is also a SaaS.

Ideally we could figure out solutions to the issues I posted here and just use GitOps Promoter.

So how are you using it? Or which other tools are you running? Would be very interested in your experiences :)

EDIT: typos

I use Paperless for my sports club. I do the finances there and for each yearly review and aldo for our taxes I need all my receipts ready.

Today I have them all printed. Every month I print out the bank statement for the month and add a sequential number to each record. I then print all the invoices for those records, add the number, mark the amount and put them in a folder.

I would like to replicate this in Paperless. Has anyone an idea how I can do that? Is there maybe a plugin or something?

I thought about just adding the number as a custom field and then use a saved view sorted by that number. That has the problem, that I cannot add in the bank statements easily. And one major issue is, that I sometimes have one document, that is the invoice for multiple records. Like our gas company, for example, only sends out one document per year telling me what I have to pay monthly. Today I just print it 12 times and add it in for each month. How could I do that in Paperless without duplicating the document?

Looking forward to your ideas :)