Broadway Google Pubsub 2.0.0-rc.0 released!
Hello there!
I'm pleased to announce the release of broadway_cloud_pub_sub 2.0.0-rc.0, and looking for more testers to give feedback and try it out.
The 2.0 release adds, an set as the default, a new GRPC Streaming producer instead of the HTTP long polling. The HTTP producer can still be used, but it's not the default.
This is a significant improvement, bringing the library on par with Google's official Pub/Sub libraries. We have been using it in production for a couple of months now, with both Gun and Mint adapters, and if I had pick two main reasons to upgrade they would be what you least expect: the graceful shutdown and the improved observability.
Here is a summary of the features that the new GRPC Streaming producer provides, but for more details you can check the pull request, the 2.0 migration guide and the official google's documentation about pull methods.
Dual GRPC connection, one to receive messages and one for ack/management. Decoupling both to survive reconnects.
Server-side flow control via max_outstanding_messages and max_outstanding_bytes
Backpressure integrated with Broadway demand. Internally the producer tracks "outstanding" messages (dispatched, waiting for ack) and "buffered" messages (received from the stream, waiting for downstream demand). Outstanding, buffered and pending demand survive reconnects.
Automatic lease extension while messages are being processed, with adaptive p99-based ack deadlines.
Reconnection handling with configurable backoff.
Exactly-once delivery support, auto-detected from SubscriptionProperties at runtime. (I know real exactly-once delivery don't exist, this is what Google calls exactly-once delivery, it has different guarantees and flow requirements)
Graceful shutdown with a configurable drain_timeout_ms (default 30s). This is actually what made me develop the GRPC Streaming producer, you can find more details on the PR or ask about it =D
Telemetry events that allow for much better observability: stream lifecycle (connect, disconnect, reconnect, drain), ack batch flush timings and sizes, unary RPC spans, lease extension cycles, outstanding/buffered message gauges, pending demand and drain status. Include also a producer "telemetry_metadata" option that lets you attach static or dynamic (MFA) extra metadata to every event. This is huge for observability, you can see if your subscribers have buffere messages (it's processing slow), high pending demand (you can scale it down), shutdown drain timings and status, and much more!
If you use the library, or just want to take a look, you are totally welcome to try it, ask anything here or in the pull request :)