Having read through and seeing some demos I still don’t understand if Genie Ontology is a real thing or some marketing fluff , we have been asked to compare against Palantir foundry’s ontology and I find very few comparisons apart from the data model and relationships , for example how do I show that Genie Ontology Knowledge Graph ?
Our data is quite small, 80% of the tables being less than 10 million records . I find spark to be an overkill for various transforms , a lightweight engine like dubckdb on a server less container application would be just fine. My experience shows that duckdb processes something in 5-6 seconds take up to a min in spark. How can I run this kind of load on Databricks
We have got an agency working for us migrating some pipelines and they have been writing everything as sql files , I just find this approach to be very limiting especially the inability to use external python libraries . What’s your take on it?