fine-tuned a VLM for messy-PDF extraction, 46% → 91.1% on OmniDocBench. runs fully on your own hardware, looking for people to break it
edit:
46% to 79% (ranks 2nd) on Parsebench, on omnidocbench -> 91.1.
hey folks,
been heads-down on this for a while so figured i'd finally show it
TLDR; i've been working on document extraction, the boring-but-painful part where you take a nasty PDF (multi-column, merged-cell tables, half-scanned garbage) and try to get clean structured data out of it. took a base VLM sitting around 46% on OmniDocBench, did a bunch of LoRA + a few architecture changes, and got it to 91.1%. tables were the big unlock, that's usually where everything falls apart.
couple things someone might care about:
- it runs fully on your own hardware. no shipping documents off to some API. that was kinda the whole reason i started this.
- serves amazing on charts/tables (area I love to work on)
- near-zero hallucination, it doesn't invent rows or numbers that aren't there.
not trying to do a big pitch. i just want people to throw hard stuff at it and tell me where it breaks. so if you've got a PDF that's been the bane of your existence, the kind that makes every parser cry, drop it on me (or DM)
happy to nerd out on the training setup or the arch changes too if anyone's curious.
cheers