spent my weekend building a salesforce to sheets pipeline and i already hate it
yeah i know data engineering is supposed to be about building robust pipelines but sometimes i wonder if we overcomplicate shit on purpose. heres the thing we needed salesforce data in google sheets for our analyst team nothing crazy just a few objects some light transformation and a daily refresh and sounds simple
so i did what any reasonable person would do spun up a heroku dyno wrote a node script to hit the salesforce rest api added some soql query logic then hooked it to google sheets api. scheduling? used a free cron job service because heroku scheduler is limited. error handling? lol we dont talk about error handling it works kinda for now. until heroku changes their free tier again or the api limits start biting or some field changes in salesforce and my script just dies silently at 3am and nobody notices until the morning meeting
i keep thinking theres gotta be a better way but every etl tool we looked at costs like 2k a month for features we dont even need. we just want a simple reliable pipe not a spaceship
what do you all do for lightweight salesforce to sheets use cases. am i overengineering this or is everyone else just silently maintaining their own fragile scripts too also bonus rant why is everything in data engineering either free but breaks constantly or enterprise priced but requires a sales call just to see a demo. is there no middle ground anymore
tbh im tired i just want to go back to building actual data models not babysitting api connections Thanks guys