Reading UTF-8 at GB/s
I wrote a new blog post on making UTF-8 reads fast in Perl:
Background: I maintain a UTF-8 library in C that Unicode::UTF8 uses, and I recently wired it into PerlIO::utf8_strict (a joint project with Leon Timmermans). We didn't get the throughput we hoped for, because of how Perl's read operator counts UTF-8 sequences — see Perl/perl5#24511. Karl Williamson has a WIP PR addressing it.
In the meantime I added read_utf8($fh, $buf, $length[, $offset]) to Unicode::UTF8: it reads and validates UTF-8 straight off a byte handle (no PerlIO encoding layer needed) and hits ~3.6–3.8 GB/s across scripts, versus ~0.4–1.0 GB/s for the :utf8 layer today.
Benchmark available in the Unicode::UTF8 repository.
What's next? I'm considering slurp_utf8($filename) and readline_utf8() as follow-ups — feedback on the API shape welcome.
Numbers and details are in the post.