u/scott_codie

LLMs are jagged, and when you put in a prompt to do some task, you are doing some frontier state-space exploration. Once the frontier is explored (it'll make mistakes and then fix them through iteration), an optimal path can be distilled to a generalized plan. This plan can be reproducible, condensed through context ablation, and be ultimately sharable. I know that environment matters for llm plans, and that it doesn't do program synthesis, so I'm not suggesting optimizing tool call chains. But the idea is that you could share a context graph of distilled short paths, which could be opportunistically injected via cheap multi-vector models (colbert/muvera) over some window. There's a bit of handwavy stuff here for how the intents are actually mined and distilled, but that's part of the idea validation process.

This would be able to speed up LLM processing and reduce AI spend considerably for everyone. I think a crowdsourcing model could work, tho I don't see too many LLM users being the community supporting altruistic type.

Hey y'all. I work out of an AI lab and wanted to share a new method of reranking that I think has a lot of potential (all open source).

A bit of background: Relational Transformers is a new transformers architecture that is trained on relational data. You load the context with typed database cells, and then it can do prediction or classification tasks. Since the model is tiny (less than 100M parameters), it can be run very quickly over a large result set.

In this case we are trying to predict the ranking of search results. We load the context with a bunch of data like consumer preferences, buy signals, typed product data like floats for price, and condition the network to try to learn its rank. We can then rerun that conditioned network to discover what actually contributes to the reranking performance, so we can keep our context extremely lean.

But one caveat, a large part of the performance gains came from converting the query to possible database cells (to supplement the context), since RT is trained on database cells. I just have a small LLM do a single pass over it to convert it to json and then load that in as context. They don't need to be real database cells, just approximate the names and the model will figure it out. Adding some schema hints edged out some extra performance numbers. But you're not passing the entire candidate result set into an LLM so this part remains cheap, relatively speaking.

End-to-end, this gets near LLM level performance at the cost of a mid-sized reranking model.

https://relativedb.com/research/relational-reranking

If you experiment with this, let me know!

Nvidia acquired Kumo and took their offering offline. So I built the open source version.

Wondering if anyone has used relational transformers as part of their trading strategy. I've been fine tuning the one from Stanford to try to predict the influence of news on various markets, because the models encodes text well. So far I've had mild success in a couple categories: repricing jumps when news changes, getting 50/50 odds for truly random markets, creating features of past trends and being able to ablate them to find casual attribution, but these models are still very small and not tuned on this type of data. Curious if anyone else has used these new models?

You can use pagerank-like algorithms on your document graph to find extra documents after RAG search. Then your attention mechanism can find spurious correlations. But you need a lot of extra context, like slack messages that reference docs together, or documents referencing documents. Otherwise neat stuff, results have been really good for me so far, but I've been doing really researchy tasks. I've been using a modified monte carlo random-walk visit counting algorithm. How about you?