Agentic Workflows help
Business users want to Inspect multiple lines of business by having answers to 500-1000s of questions with complex logic. Currently, all the data is in snowflake(structured, unstructured , classification, etc) and we use databricks to orchestrate and leverage prompts engineering and answer around 10 questions that feeds a dashboard.
AI team in the org has built a framework, basically using langchain and langgaph to orchestrate agents, skills with python executable sandbox environment.
We are tasked to leverage this and come up with a solution to speed up the delivery. The agentic framework is currently planned to be hosted on Kubernetes, for us to customize and wire in different components like snowflake connectors , sharepoint files etc. to answer the questions.
Snowflake and Databricks MCP will be available soon. We are pretty clear on snowflake data input and agents/skill. The question is on how do we leverage Databricks and have most of our compute being done there rather than using K8s. Any insights or links to a similar architecture would be helpful.