u/christian_hansen

▲ 31 r/perl

Reading UTF-8 at GB/s

I wrote a new blog post on making UTF-8 reads fast in Perl:

Reading UTF-8 at GB/s

Background: I maintain a UTF-8 library in C that Unicode::UTF8 uses, and I recently wired it into PerlIO::utf8_strict (a joint project with Leon Timmermans). We didn't get the throughput we hoped for, because of how Perl's read operator counts UTF-8 sequences — see Perl/perl5#24511. Karl Williamson has a WIP PR addressing it.

In the meantime I added read_utf8($fh, $buf, $length[, $offset]) to Unicode::UTF8: it reads and validates UTF-8 straight off a byte handle (no PerlIO encoding layer needed) and hits ~3.6–3.8 GB/s across scripts, versus ~0.4–1.0 GB/s for the :utf8 layer today.

Benchmark available in the Unicode::UTF8 repository.

What's next? I'm considering slurp_utf8($filename) and readline_utf8() as follow-ups — feedback on the API shape welcome.

Numbers and details are in the post.

reddit.com
u/christian_hansen — 20 hours ago
▲ 26 r/perl

Time::Str 0.92: DateTime parsing at ~10.5M/sec, zero heuristics

Time::Str's DateTime format now runs on a native Ragel-generated C state machine instead of a regexp, ~20x faster than the regexp path.

Supported formats

One parser accepts ISO 8601, RFC 3339, RFC 9557, RFC 4287, ISO 9075, RFC 2822, RFC 2616, RFC 3501, and ECMAScript Date.toString, plus free-form textual dates (Monday, 24th December 2012 at 3:30 pm UTC+1 (CET), 24. XII. 2012 12PM, 24DEC2012 12:30:45.500).

No heuristics, multi-standard, single-pass

Most permissive parsers (Python's dateutil, PHP's strtotime, Ruby's Date.parse) resolve ambiguity by guessing, whether via fixed heuristics or dayfirst/yearfirst flags.Time::Str refuses it: numeric-only dates are Y-M-D only; any other ordering requires a textual or Roman-numeral month. Disambiguation is baked into the grammar's alternation, not resolved at runtime. Even separator-consistency (2024-12/24 is rejected) is enforced inline, no second pass.

Performance

The DateTime parser accepts every format listed above, yet runs within ~10-20% of the single-standard parsers beside it. On Perl v5.42 (XS), parsing 2012-12-24T11:30:45.123456Z:

                Rate   DateTime  RFC3339  RFC2822  ECMAScript
DateTime   10523558/s        --     -11%     -15%        -18%
RFC3339    11785319/s       12%       --      -5%         -8%
RFC2822    12376403/s       18%       5%       --         -3%
ECMAScript 12776079/s       21%       8%       3%          --

A purpose-built RFC 3339 parser is only ~12% faster than the permissive one. That's the payoff of baking disambiguation into the grammar — there's almost no "permissiveness tax".

Enjoy!

u/christian_hansen — 7 days ago
▲ 20 r/perl

Time::Str - Time Zones and Leap Seconds

This blog post is a follow up on my previous post, Introducing Time::Str, covered parsing and formatting. This one covers two additions, time zones and leap seconds, and ends with a note on the new C parsers.

If you have any questions or feedback, feel free to leave them here or on the blog, and I'll be happy to respond.

reddit.com
u/christian_hansen — 23 days ago