u/vorjdux

Context: Monocoque is a pure-Rust ZeroMQ-compatible runtime (ZMTP 3.1), no libzmq, no C dependency. I have not posted since 0.1.3, so this covers 0.1.4 through 0.1.7. Numbers are on an i7-1355U (12 threads), Linux 6.17, rustc 1.96, loopback TCP, sender and receiver on separate OS threads, unless noted.

The headline is the backend split in 0.1.6. It runs on either io_uring or tokio now, chosen by a Cargo feature: runtime-compio (default, native io_uring) or runtime-tokio (the same socket stack on tokio, for macOS, Windows, older kernels, or to drop into an existing tokio program). They are mutually exclusive. Adding tokio was additive because the protocol stack is already generic over the io traits, so one small runtime facade names the concrete runtime and a thin tokio stream adapter implements the same owned-buffer io traits with no extra copy on the data path. No protocol code changed, and both backends keep sockets !Send.

PUSH/PULL throughput with coalescing on, per backend against rust-zmq:

msg	compio (io_uring)	tokio (epoll)	rust-zmq
64 B	9.2M	13.6M	1.33M
256 B	5.6M	9.8M	1.09M
1 KB	2.4M	5.3M	656K
4 KB	841K	1.74M	328K
16 KB	268K	473K	117K

On these single-flow loopback microbenchmarks tokio/epoll is the faster of the two. io_uring's edge is on real network I/O and high connection counts, not loopback ping-pong, which is why compio stays the default. Both beat rust-zmq once coalescing batches the writes, but coalescing is opt-in and you flush() when you want the bytes out, so it is a throughput mode, not the default.

0.1.4 and 0.1.5 cut per-message cost in a few places. Large PUSH frames now go out with a vectored write (writev via compio's write_vectored_all) instead of copying each body into the userspace send buffer: above SocketOptions::vectored_write_threshold in eager mode, the header and the refcounted Bytes body go to the kernel as an iovec, and the header buffer and iovec list are reused so the path allocates nothing. The default threshold is 32 KB, the measured loopback crossover, where skipping the copy starts to win by about 1.1 to 1.3x on the machine I tested for it (a 4-core cloud Xeon, not the i7 above). The worker-pool PubSocket coalesces a burst of queued broadcasts into one per-subscriber vectored write while keeping the fan-out zero-copy through shared Bytes clones. On the receive side, recv_batch blocks for one message then drains every further message already decoded from the same kernel read, and recv_into writes frames into a caller-owned buffer you reuse, so a steady loop does no per-message allocation. recv_into takes 64 B from about 7.7M to 9.7M msg/s, tapering as messages grow and the path turns bandwidth-bound.

0.1.5 also added worker-pool pipelines. A plain PUSH or PULL owns one connection, so it cannot drive a pool. PushFanOut binds once, accepts N PULL workers, and round-robins sends across them, which is the ZMQ load-balancing rule. PullFanIn merges N PUSH workers into one fair-queued stream with a batched handoff, one channel hop and one await per kernel-read batch instead of per message. That roughly doubles small-message sink throughput, from about 5.25M to 9.9M msg/s at 64 B on the reference machine.

0.1.7 is correctness work. PullFanIn had a memory bug: the merge channel bounded the number of queued batches, but each batch was a whole kernel read of unbounded message count, and a frozen message pins its whole 64 KiB slab page, so a sink that fell behind its readers held a growing set of pages. Peak RSS reached about 66 MB at 32 workers and 64 B, roughly ten times a single PullSocket at the same rate. Readers now forward each read in fixed-size chunks with the channel capacity lowered to match, so queued messages and their pinned pages are bounded regardless of payload or worker count, and RSS at that cell drops to about 15 MB with throughput unchanged. Separately, TCP_NODELAY was set at connect but skipped on three socket-creation paths, automatic reconnection, XPUB accepting a subscriber, and XSUB connecting upstream, so a reconnected socket ran with Nagle's algorithm on until the process restarted, quietly raising latency on the sockets you would pick for low latency. All three now apply the same setsockopt as the initial connect. Both fixes ship with regression guards: a peak-RSS bound on PullFanIn and fd-level checks that read TCP_NODELAY off the live socket, each confirmed to fail with its fix reverted. 0.1.4 also migrated the workspace to Rust edition 2024.

Repo, full changelog, per-backend tables and IPC numbers: https://github.com/vorjdux/monocoque

Run it and tell me where it doesn't hold up.

New release is out

Context: monocoque is a pure-rust ZeroMQ-compatible runtime (ZMTP 3.1) on io_uring via compio. no libzmq, no C dependency.

One design thing up front, because it explains the numbers: monocoque doesnt auto-tune. it does what you tell it and not much more. on defaults a PUSH socket does one io_uring write per send, which is why eager mode loses to libzmq at small messages, libzmq batches for you behind its IO thread and eager mode just doesnt. that tradeoff is deliberate. you describe the workload through SocketOptions and it optimizes for that case instead of guessing for you.

The main feature across 0.1.2 and 0.1.3 is the knob you turn for throughput: write coalescing for PUSH (with_write_coalescing(true)). it accumulates encoded messages in a userspace buffer and drains them in one syscall on flush, the same batching libzmq does internally, just explicit.

loopback PUSH/PULL vs rust-zmq, coalescing on:

msg size	monocoque coalesced	rust-zmq	ratio
64 B	6.3M msg/s	971K	6.5x
256 B	3.6M	699K	5.2x
1 KB	1.5M	455K	3.3x
16 KB	120K	71K	1.7x

so the win is specifically the batching at small messages, where syscalls dominate, and it shrinks as messages get bigger. the full table including eager mode, where monocoque is slower, is in docs/performance.md. for latency you turn the knob the other way and leave it eager, since coalescing would only add delay to a round trip you asked to send now.

Methodology: sender and receiver run on separate OS threads with their own compio runtimes, so these are real kernel round-trips, not task switching in one runtime. timer is on the receiver. its loopback, so it measures stack overhead, not a network.

0.1.3 is hardening. the main fix is a buffer overflow when read_buffer_size is set above 64kb (a panic plus heap corruption), which surfaced when another pure-rust ZMQ implementation was benchmarking against monocoque and sized the read buffer up. that is exactly the kind of bug i wanted the benchmarks to flush out. also fixed a benchmark hang i traced to a stale entry left in compio's timer heap after a timeout future got dropped.

Run it and tell me where the methodology is unfair or where the numbers dont hold: https://github.com/vorjdux/monocoque/releases/tag/v0.1.3

Repo:

https://github.com/vorjdux/monocoque

Wrote this after building Monocoque, a pure-Rust ZeroMQ runtime (ZMTP 3.1) on io_uring via compio.

The argument in short:

Rust's core idea is correctness by construction, and it applies that idea to exactly one thing: memory. Owned-buffer APIs like compio and tokio-uring make the io_uring buffer-ownership problem unrepresentable, which is genuinely great. But memory safety is not cancellation safety. A multipart write cancelled mid-frame, say a timeout dropping the future between frames, leaves every buffer valid and accounted for and the wire corrupt: half a message on the stream, the peer's parser desynced. The borrow checker is happy and the runtime is happy, because the bug lives one level up in protocol state, where neither of them is looking.

The post is about where "safe" stops (memory) and where it quietly becomes your job (protocol and state under cancellation), the structural fix I went with, and why cancellation-as-Drop, which grew out of the epoll readiness world, sits awkwardly with completion I/O and stateful protocols.

https://vorjdux.com/articles/safe-rust-corrupt-wire.html

My own blog. Happy to get into any of it in the comments.

Some of you may have seen my wire-probe post here recently. It turned into a great back-and-forth with Mohit D. Patel about CharlotteOS, his from-scratch kernel in Rust, specifically about building a kernel where asynchrony is the default rather than a retrofit. I wrote up the idea.

It gets into io_uring as a completion model (and how that's Linux catching up to NT's IOCP from the 90s), why fd / HANDLE / observer-capability are the same primitive, the observer/upcall design CharlotteOS uses, and an honest look at the hard parts. Rust throughout, since the OS in question is Rust.

https://vorjdux.com/articles/hardware-is-async.html

My own blog. Happy to get into any of it in the comments.