u/Artye10

Hi, I wanted to use Iceberg via Managed Tables to save myself from too much table maintenance, but a couple of things are not very clear.

So, to be able to query the tables directly (not via BQ) you need to export the metadata, basically the manifest files, but because this is a 'manual' operation, is it also included in the garbage collection? So when a manifest list and its files are outdated will they be deleted? Does this improve/change if you ask for auto-refresh (https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery#create-iceberg-table-snapshots)?

The objective of using this was to not have to delete files myself form the metadata folder to avoid issues and drifts, but if this still has to be manually managed I really don't know if I should go with simple REST Catalog Iceberg tables (since I have to sometimes do upserts which are better with iceberg directly, but with the amount of data I have and how is partitioned is fine to do them in BQ).

u/Artye10 — 24 days ago