u/SignificantZebra5883

There's so many new techniques like RL, RL LoRA, QLoRA, CPT LoRA.

I believe i would have a usecase for them, but i don't know where to learn, youtube is filled with bad quality tutorials if i just search and the good channels (fireship, bycloud) don't cover these as they're quite new concepts, i guess?.

how can a regular joe like me learn about these concepts in a "practical depth" so i can actually fine-tune qwen 27b successfuly on lets say custom corpus? without spending 100$ figuring out that "oh i didnt even need CPT here" or "well i chose the wrong Rank count! time to start this 2 day run again!"

context and TLDR: im building a legal general purpose chatbot for context, i have a big corpus, but im a bit stuck on what to do next

thanks for reading and any pointers!

I’m building a Polish General purpose legal Model that drafts documents, answers questions using legal sources, and has enough coding ability to handle some automation. The workflow is very tool-heavy:

Question → many sequential tool calls → final answer/document

Think Claude Code/Codex-style execution, but for legal workflows. Reliable tool selection, correct arguments, and recovering from errors matter as much as writing a good final answer.

I’ve had decent results with a dense 27B Qwen 3.8 custom made fine-tune for complex legal document summarization and classification. I’m already familiar with the smaller Qwen A3B and Gemma options. What interests me is the tier above those: 50B+ total-parameter MoEs with a relatively small active parameter count.

The question is, the small dense ones are great, but slow for agentic stuff (afaik), and i wonder if theres some middle ground maybe 70-120B models that would be able to be fine-tuned for the law stuff but be MoE so the agentic ClaudeCode style inference would also be lightning fast, and also low-ish cost for fine-tuning and inference.

Basically: Does the larger-total/small-active MoE approach actually buy you meaningfully stronger reasoning and tool reliability while retaining low latency,and at what hardware cost?

I understand that small active parameter counts don’t mean small VRAM requirements: the weights still need to live somewhere, alongside context and serving overhead. I also don’t assume that more total parameters automatically means a better model. I’m interested in where that tradeoff works in practice.

There are three things I’m trying to pin down:

• Inference hardware: Ideally inference runs rented with parallel agentic loops (this is for a B2C project, not single person use, we scale based on demand)
• Fine-tuning hardware: Obviously FT LoRA will take more memory than inference, max like 4GPUs on vastai fits the budget.
• Agent performance: After it gets the prompt the tool calls and everything will be local, so imo it has no problems being blazing fast, as soon as the model calls a tool call it will be back very fast, so for this agentic use case, quick TTFT and t/s and adaptive dynamic reasoning are prefer right?

For context, fine-tuning would target Polish language, document conventions, and successful tool trajectories. The actual legal sources would remain in retrieval/tools rather than relying entirely on memorized law.

I’m not looking for someone to compile a model shortlist (althought would be nice, but i dont expect anyone to break their back over this).

I’m looking for pointers, and firsthand experience with this particular size/architecture tradeoff. A configuration like “model + quantization + GPU(s) + serving engine + context length + concurrency + measured latency,” along with whether you successfully fine-tuned it, would be much more useful than a leaderboard score.

Has moving from a ~30B model to a 50B+ low-active-parameter MoE actually improved your agent’s successful tasks per minute, or did the memory, interconnect, and training requirements erase the advantage? Thanks for reading

Hello everyone,

I am the founder of a central European AI lawyer platform. Up until now all the leads came from Google SEO "people searching lawyer AI" (in local language), or an article that blew up on a popular local news site thanks to a reddit post I made (I know inception)

Ever since the article traffic has been dying down because the product was not there yet, but now it's actually usable but all the hype is gone. I would like to do outreach to get people to start using it again more.

The obvious answer is to just make a reddit post bu the original one highlighted how I was a young founder and this is a free project and now we became a bit more corporate, still doable but I am putting it off.

Emailing our previous users, already on the table and doing in waves. People are coming over.

Other than that it's just the SEO, so I'm looking for some way to turn a dial and throw money at it and get some traffic of actual lawyers or practicioners that would be power users and work with documents a lot, I'm not expecting miracles just experiment with it and hopefully make some money.

What would be the best way to reach them? Google ads? (I've never used GA, do I have to select keywords or something?), Facebook ads?, LinkedIn?

Thanks for reading.

For the last month i've been trying to fine-tune jina-v5 (which has performed best on my corpus out of the box) on slovak law chunks, time and time again no matter what i do I can't get the model to learn nuance of slovak syntax.

here's the biggest trap chunk that keeps confusing my AI with my translation:

Query: "krádež cigariet" = theft of cigarettes

Podľa § 60 ods. 1 písm. a/ Tr. zák. súd obvinenému ukladá trest prepadnutia vecí a to:
1000 ks cigariet zn. Marlboro gold,
400 ks cigariet zn. Rothmans modré,
1000 ks cigariet zn. Rothmans červené,
400 ks cigariet zn. Bond modré,
200 ks cigariet zn. Parliament modré v celkovom množstve 3000 ks cigariet, všetky o dĺžke tabakového povrazca do 80 mm vrátane, bez platnej slovenskej kontrolnej známky. Podľa § 60 ods. 5 Tr. zák. vlastníkom prepadnutých vecí sa stáva štát. Poučenie:

you can translate it to your language, but essentialy it says, "according to paragraph 60, the court is giving a punishment of "prepadnutie". which is a synonym and could mean, mugging OR forfeiture or confiscation.

this example has been breaking every single model, because it is ambiguous but after a thorough read you can clearly tell its not theft or mugging but all of my fine-tunes consistently rank it high, higher than base jina.

I know there's a lot of moving parts and context needed to answer this question, so i will just focus on my latest run.

> i used an LLM to generate queries based on source chunks (varied personas, board short queries and long paraphrased queries [all sorts of combinations at this point])

> i used base jina to grab top 50 results based on my corpus of judicial data and legislature + i injected source chunk + it's similiar siblings (i also did a run without injecting still sucked)

> then i used qwen/qwen3.5-397b-a17b to logit mine relevance, basically "is chunk relevant, answer only yes/no" then we mined the probability for yes. humans and stronger AIs all agreed that qwen's ranking is actually good. except for some rare cases (it clearly distinguished this chunk however as NOT being theft, correctly giving it a low ranking)

> then i ran jina v5 fine-tunining LoRA on the retrival adapter (at least that's what claude opus told me xd) with these parameters:

param	value
base model	jinaai/jina-embeddings-v5-text-small (1024-dim, last-token pooling)
what's trained	built-in retrieval LoRA only — r=32, α=32, dropout=0.1, targets q/k/v/o/gate/up/down_proj
trainable params	20,185,088 / 676,790,272 = 2.98%
loss	MarginMSELoss (margin = teacher rel(pos) − rel(neg)); no Matryoshka
LR	5e-6, linear schedule, warmup_ratio 0.05
epochs	1
batch	per-device 8 × grad-accum 2 = effective 16
precision	bf16, gradient_checkpointing off
max_seq_length	2048 (v4 was 512)
optimizer	AdamW (HF default), seed 42, val_frac 0.03
data	46,001 MarginMSE triples from 2,174 Qwen-distilled queries → 44,621 train / 1,380 val → 2,789 steps
pair-mining	top-5 pos × bottom-5 neg per query, min-margin 0.2, ≤40 pairs/query, pos≥0.5 / neg≤0.3
hardware	RTX PRO 6000 Blackwell 96GB, torch 2.11+cu128, ~74 minparam valuebase model jinaai/jina-embeddings-v5-text-small (1024-dim, last-token pooling)what's trained built-in retrieval LoRA only — r=32, α=32, dropout=0.1, targets q/k/v/o/gate/up/down_projtrainable params 20,185,088 / 676,790,272 = 2.98%loss MarginMSELoss (margin = teacher rel(pos) − rel(neg)); no MatryoshkaLR 5e-6, linear schedule, warmup_ratio 0.05epochs 1batch per-device 8 × grad-accum 2 = effective 16precision bf16, gradient_checkpointing offmax_seq_length 2048 (v4 was 512)optimizer AdamW (HF default), seed 42, val_frac 0.03data 46,001 MarginMSE triples from 2,174 Qwen-distilled queries → 44,621 train / 1,380 val → 2,789 stepspair-mining top-5 pos × bottom-5 neg per query, min-margin 0.2, ≤40 pairs/query, pos≥0.5 / neg≤0.3hardware RTX PRO 6000 Blackwell 96GB, torch 2.11+cu128, ~74 min

If anyone is as invested in this as me here's the scripts i used for training:
finetune_jina.py
prepare_pairs.py

All models do get better at slovak law, but still fail these simple logical problems, i've also tried fine-tuning qwen 8b reranker in efforts of distilling it later into a bi-encoder, but these efforts also failed. qwen made same mistakes about the "prepadnutie" case.

I would be really thankful if someone highly skilled in this could eyeball this set-up and let me know if there's some architectural flaw, and if my focus should be looking for bugs in the code.

thank you very much!

For the last month i've been trying to fine-tune jina-v5 (which has performed best on my corpus out of the box) on slovak law chunks, time and time again no matter what i do I can't get the model to learn nuance of slovak syntax.

here's the biggest trap chunk that keeps confusing my AI with my translation:

Query: "krádež cigariet" = theft of cigarettes

Podľa § 60 ods. 1 písm. a/ Tr. zák. súd obvinenému ukladá trest prepadnutia vecí a to:
1000 ks cigariet zn. Marlboro gold,
400 ks cigariet zn. Rothmans modré,
1000 ks cigariet zn. Rothmans červené,
400 ks cigariet zn. Bond modré,
200 ks cigariet zn. Parliament modré v celkovom množstve 3000 ks cigariet, všetky o dĺžke tabakového povrazca do 80 mm vrátane, bez platnej slovenskej kontrolnej známky. Podľa § 60 ods. 5 Tr. zák. vlastníkom prepadnutých vecí sa stáva štát. Poučenie:

you can translate it to your language, but essentialy it says, "according to paragraph 60, the court is giving a punishment of "prepadnutie". which is a synonym and could mean, mugging or forfeiture or confiscation.

this example has been breaking every single model, because it is ambiguous but after a thorough read you can clearly tell its not theft or mugging but all of my fine-tunes consistently rank it high, higher than base jina.

I know there's a lot of moving parts and context needed to answer this question, so i will just focus on my latest run.

> i used an LLM to generate queries based on source chunks (varied personas, board short queries and long paraphrased queries [all sorts of combinations at this point])

> i used base jina to grab top 50 results based on my corpus of judicial data and legislature + i injected source chunk + it's similiar siblings (i also did a run without injecting still sucked)

> then i used qwen/qwen3.5-397b-a17b to logit mine relevance, basically "is chunk relevant, answer only yes/no" then we mined the probability for yes. humans and stronger AIs all agreed that qwen's ranking is actually good. except for some rare cases (it clearly distinguished this chunk however as NOT being theft, correctly giving it a low ranking)

> then i ran jina v5 fine-tunining LoRA on the retrival adapter (at least that's what claude opus told me xd) with these parameters:

param	value
base model	jinaai/jina-embeddings-v5-text-small (1024-dim, last-token pooling)
what's trained	built-in retrieval LoRA only — r=32, α=32, dropout=0.1, targets q/k/v/o/gate/up/down_proj
trainable params	20,185,088 / 676,790,272 = 2.98%
loss	MarginMSELoss (margin = teacher rel(pos) − rel(neg)); no Matryoshka
LR	5e-6, linear schedule, warmup_ratio 0.05
epochs	1
batch	per-device 8 × grad-accum 2 = effective 16
precision	bf16, gradient_checkpointing off
max_seq_length	2048 (v4 was 512)
optimizer	AdamW (HF default), seed 42, val_frac 0.03
data	46,001 MarginMSE triples from 2,174 Qwen-distilled queries → 44,621 train / 1,380 val → 2,789 steps
pair-mining	top-5 pos × bottom-5 neg per query, min-margin 0.2, ≤40 pairs/query, pos≥0.5 / neg≤0.3
hardware	RTX PRO 6000 Blackwell 96GB, torch 2.11+cu128, ~74 minparam valuebase model jinaai/jina-embeddings-v5-text-small (1024-dim, last-token pooling)what's trained built-in retrieval LoRA only — r=32, α=32, dropout=0.1, targets q/k/v/o/gate/up/down_projtrainable params 20,185,088 / 676,790,272 = 2.98%loss MarginMSELoss (margin = teacher rel(pos) − rel(neg)); no MatryoshkaLR 5e-6, linear schedule, warmup_ratio 0.05epochs 1batch per-device 8 × grad-accum 2 = effective 16precision bf16, gradient_checkpointing offmax_seq_length 2048 (v4 was 512)optimizer AdamW (HF default), seed 42, val_frac 0.03data 46,001 MarginMSE triples from 2,174 Qwen-distilled queries → 44,621 train / 1,380 val → 2,789 stepspair-mining top-5 pos × bottom-5 neg per query, min-margin 0.2, ≤40 pairs/query, pos≥0.5 / neg≤0.3hardware RTX PRO 6000 Blackwell 96GB, torch 2.11+cu128, ~74 min

If anyone is as invested in this as me here's the scripts i used for training:
finetune_jina.py
prepare_pairs.py

All models do get better at slovak law, but still fail these simple logical problems, i've also tried fine-tuning qwen 8b reranker in efforts of distilling it later into a bi-encoder, but these efforts also failed. qwen made same mistakes about the "prepadnutie" case.

I would be really thankful if someone highly skilled in this could eyeball this set-up and let me know if there's some architectural flaw, and if my focus should be looking for bugs in the code.

thank you very much!