u/No_Sky9786

SALT shrinks a long document down to a fixed size before it is sent to a language model, keeping the sentences that carry the most information. It works with any model, produces a shorter plain-text prompt, and cuts the compute, memory, and wait time that long inputs cost.

I need some help with deciding and implementing a method that selects a better budget than the default 20/25% across chat. The kv cache in GPU keeps only some instructions and a retrieved % of the actual full input or conversion that is all organized in DRAM in a trie. The retrieval is insanely quick now with less than 1-2 seconds for even 100k + conversations however it is sometimes too much as it’s set by a hard % (prefill GPU men use becomes a problem at larger scale). What method could I use to decide how to adjust this % based on the question?

GitHub: oteomamo/SALT

I am looking for a dataset or benchmark for chat evaluation. What is currently available that can measure multi-turn accuracy and memory management? I have used older benchmarks like LongBench, NIAH, and RULER, but I am not sure what is currently considered SOTA or of significance to the community. Additionally, I want to use this as a way to determine more weak points in my work. Also, agents are not part of the work yet so its one long conversation.

I usually will use https://portal.core.edu.au/conf-ranks/ however there are some conferences like CLUSTER that in my department are highly regarded as good but here it says C. Is there an official website for conferences and also is there one for conferences separated by domain?

SALT shrinks a long document down to a fixed size before it is sent to a language model, keeping the sentences that carry the most information. It works with any model, produces a shorter plain-text prompt, and cuts the compute, memory, and wait time that long inputs cost. saltChat keeps the theme trie in DRAM across turns, so a document is indexed once and reused for the whole conversation instead of being re-read every message.