So this is what it feels like to have automated tests verify the app after you deploy
For the first time in my entire software career, actual E2E tests run against my dev environment right after CI deploys to it. I wrote them with playwright and it was way less flakey than I was afraid of. I only have a retry of 3 and that seems to be the right amount. Only got 20 playwright tests right now and they run in about 1.5 mins, but I do have ~350 backend end-to-end tests that make API calls to create/execute/verify tests. These are running in about 2 mins.
I've got 4500+ unit tests across the FE and BE and those all run in under a minute.
It too 2 days to wire up and helped me fix 4 annoying bugs I'd been battling for a while the kind I could never quite pin down with unit tests because they only showed up across multiple components and caches and it covers a whole class of failures that only exist there: missing Firestore indexes, env config drift.
Anyway, this feels nice and I hope you all get to experience this one day.