Using DuckLake with Azurite (DuckDB 1.4.4 vs 1.5.2) — experience & issues
Return of experience using ducklake with Azurite.
DuckDB version tried 1.4.4(LTS) and then 1.5.2 both using duckdb ui and python library.
Setup windows laptop using azurite vscode extension stock .
Uploaded a 20M lines table (+/-350mb parquet) on Azurite blob storage.
Some early difficulty connecting to the storage.
1^(st) try: fairly simple to create a table from the blob storage and manipulate it.
Experience is quick and easy (python and duckdb -ui 1.4.4).
2^(nd) try: Ok let’s try the encrypted version (still using python and duckdb -ui 1.4.4).
Doesn’t work at all seem like it tries to make too many (first 2x64MB blocks write successfully, then crashes with Error: 409 The committed block count cannot exceed the maximum limit of 50,000 blocks). Tested on another laptop same issue.
3^(rd) try: Before complaining I figured I should try the latest version of duckdb.
So tried the same thing with 1.5.2 (python and ui).
Unencrypted works great.
Encrypted doesn’t
It doesn’t crash but is slow enough that I don’t think it’s usable (2 hours to “upload” on local azurite).
Since there are a lot of things I might have missed , a lot of friction points and unknown unknown I’m checking with the community to see if anyone got an idea of what I could improve.
Anyone got the same experience or poc that work well with similar setup ? Or any comment ?