There was no conformance suite for org parsers, so i built one and graded four of them against org-element
Thank you to everyone who recommended or complained to me about overusing LLMs for writing this post and making responses. I admit that I am not confident enough in my English proficiency so that I usually use LLMs to filter my writing, but somehow LLM style seems soulless and robotic. Please forgive me if you feel it like an insult to you. But the effort spent on this project is real. I used to use Emacs, especially Org mode, in the past very heavily, and I know how it is far behind other technologies in popularity despite its very aesthetic and powerful syntax. That’s why I want to unchain it from Emacs’ ecosystem and make it popular in the world.
---
Every org parser i looked at tests itself against its own idea of what org does. There is one reference implementation (org-element.el) and one written spec, but nothing portable to check a parser against either. Someone in the windancer thread here said org is missing a standalone non-emacs parser, and that the lack of one is maybe why markdown is so much more common. i think the missing test suite sits upstream of that.
So the suite is the thing im showing, not the parser. 121 curated cases, each pinning one rule, plus 28 real world files and a 1,505 case generated differential corpus. Every expected answer comes out of real Emacs running org-element-parse-buffer, not out of my head. It is plain .org files, JSON and one elisp script, so a parser in any language can use it.
Then i graded four parsers with it. Strict means the tree is structurally identical to org-element's own answer, every node type, every field, every blank line count.
parser |
conf(121 |
real(28) |
sweep(1505) |
|---|---|---|---|
organism (mine) |
121 |
28 | 1505 |
uniorg 3.2.2 |
81 | 13 | 693 |
orgize 0.9.0 |
67 | 2 | 402 |
go-org v1.7.0 |
59 | 1 | 479 |
Before that table gets quoted at anyone: the corpora live in my repo and my parser was developed against them, the others were not. Each one got a thin adapter over its own AST, and what an AST simply cannot represent is counted separately from real parse bugs. go-org feeds an html renderer and orgize optimises for speed, neither of them signed up for this bar.
What i would actually like is someone running it against their own parser and telling me where i graded them wrong. Repo link in the comments.