Our monthly refresh was doing a lot of unnecessary work
We had one big ETL job running every month and refreshing pretty much everything.
Started looking at the numbers and realized most of the records hadn't changed.
Meanwhile, the stuff we actually cared about could already be out of date. Someone changed jobs right after the refresh and we're stuck with the useless record.
We've moved those records to change-based updates and left the slower stuff on the monthly schedule.
Coresignal has been pretty good for keeping the company and employee data fresh without having to reload everything.
Much less data moving around for no real reason.
Now we're just working out which records actually deserve the faster treatment.