u/Remarkable-Ant-2473

We’ve been running Iceberg on Spark for a while and the maintenance side keeps surprising me with how much glue code we end up writing — compaction schedules, snapshot expiration, orphan file cleanup, manifest rewrites, monitoring when small-file counts blow up etc. Can someone give me insights how are you guys doing maintenance stuff in your organisation?

P.S: Asking this on different sub reddits to gather more info

We’ve been running Iceberg on Spark for a while and the maintenance side keeps surprising me with how much glue code we end up writing — compaction schedules, snapshot expiration, orphan file cleanup, manifest rewrites, monitoring when small-file counts blow up etc. Can someone give me insights how are you guys doing maintenance stuff in your organisation?

P.S: I am asking this on various sub reddits to gather more information

We’ve been running Iceberg on Spark for a while and the maintenance side keeps surprising me with how much glue code we end up writing — compaction schedules, snapshot expiration, orphan file cleanup, manifest rewrites, monitoring when small-file counts blow up etc. Can someone give me insights how are you guys doing maintenance stuff in your organisation?

I am thinking to do cafe hopping today or tomorrow and work in cafe. Happy to have company around. Please DM if interested

P.S: My small intro - I am techie working on some idea