r/enterpriseai

I have a pretty clear goal, and I think it makes a lot of sense regarding where AI should be heading inside companies.

I believe that, in the near future, every every small and medium-sized business will have its own centralized AI platform for all employees, but managed and controlled in a corporate way: users, roles, permissions, auditing, models, agents, tools, documentation, etc., just like we already do with all the other software that enters our companies.

The idea would be something like this:

Users access a single platform through SSO using their corporate account, whether that is Microsoft 365, Google Workspace, LDAP, etc.

Inside, they have a common chat where they ask their questions.

From there, an agent or orchestrator should decide which agent/model/tool/etc. to route the request to, based on factors such as:

• The user’s role and permissions
• The documentation they need to access
• The privacy or complexity of the question: expensive, cheaper, local model, etc
• The need to use specific tools: CRM, ERP, Microsoft 365, etc

For example, an admin user could ask ERP information, someone from IT could use an agent connected to Microsoft 365, and another department could have access to specific documentation or internal processes.

All of this seems very logical to me as an enterprise AI architecture, but I can’t seem to find a clear open-source solution, or a combination of tools, that properly covers this scenario.

I would really appreciate any ideas, experiences, or recommendations. I’m sure I won’t be the only one this could help. Has anyone built something similar, or do you know any projects that could be used as a starting point?

Any input would be very welcome 😚

We're a financial services firm running both traditional ML models in production and experimenting with agentic workflows. The traditional models are deployed on Seldon Core, it's been rock solid for real-time inference, canary rollouts, and A/B testing for years. The agentic side is newer; we're using a separate gateway layer for LLM routing, prompt management, and tool calling.

The problem is these are treated as two completely separate infrastructure stacks. Different teams, different dashboards, different governance policies. When an agent calls a traditional model via a tool, tracing that request end-to-end is a nightmare. Compliance wants a single audit trail from user query through agent reasoning to model inference output. Right now, that means stitching together logs from two systems.

We're considering merging these stacks, but that feels like a massive engineering project. Has anyone successfully unified their inference layer and gateway layer? What did it take? Are there platforms that handle both natively, or are we stuck building our own integration?