u/peterxsyd

https://github.com/anthropics/claude-code/issues/77136#issuecomment-5352256342

Hi everybody,

I am excited to announce the release of Lightstream, a step change capability for high-performance data transport built in Rust, that makes it essentially effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or even piped out to the terminal so an agent like Claude can watch the live batch stream in real time (example in repo).

Furthermore, Lightstream exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable). As a bonus, Lightstream is straightforward to setup with essentially zero configuration other than optional TLS certificates and your Cargo package/pip install, and endpoint addresses.

The open benchmarking methodology used - available to view and run in the repository

So what is Lightstream? It is Rust package with Python bindings, that builds directly on Minarrow ( which is in turn a high-performance implementation of the Apache Arrow memory layout in Rust, tuned for SIMD compatibility). Lightstream implements encoding, decoding, readers, writers and stream/read writers, and transport for data in Rust. But, in a manner, that is fully composable and leaves you de-coupled at any layer, to customise things architecturally. The crux then is the transport layer on top, which natively supports interchanging any of the following transport formats:

• TCP
• HTTP
• QUIC
• Websocket
• Webtransport
• UDS (pipe your data from your Rust process to Python or two Rust or Python programs plug and play )
• Stdio (pipe your data program output straight into the terminal for something else to pick it up

And finally, the (optional) Lightstream protocol, which then combines the Arrow/Proto/MsgPack and any other custom types you want to send.

An example of things you can do with it:

• setup a live stream of data batches from your program A to program B
• send typed metadata via Protobuf on the same feed
• use it for straightforward live feed delivery between server and client (though not Web JS yet)
• useful if you have a central storage server you are pulling larger than memory data over the network to churn through (though, no S3 etc. it is node to node or process to process)
• pipe data directly over UDS to Python, and then read that feed with SQL (via DuckDB).

It is not:

• Kafka or a messaging broker. There is no resiliency / vertical scalability.
• A stream processing engine like Flink. It is for sending/receiving data only. You do polars on the other end or whatever you want with the Arrow-shaped data. That is a very different back-pressure/long-lived scenario and is not that kind of large-scale streaming. --> I.e., think quick and easy Websocket, and best for settings like EKS K8 pod to pod/containers, between EC2's or between processes on the same box, "light streaming".

Examples:

Send Arrow and Protobuf data on the one connection:

use lightstream::models::protocol::connection::TcpLightstreamConnection;
use lightstream::models::protocol::LightstreamMessage;

let mut conn = TcpLightstreamConnection::from_tcp(stream);
conn.register_message("event");
conn.register_table("metrics", schema);

conn.send("event", b"user-login").await?;
conn.send_table("metrics", &table).await?;

while let Some(msg) = conn.recv().await {
    match msg? {
        // Protobuf message
        LightstreamMessage::Message { tag, payload } => { /* … */ }
        // Arrow table
        LightstreamMessage::Table { table, .. } => { /* … */ }
    }
}

Read a feed send from Rust over UDS into Python

reader = ls.read(
    "uds:///tmp/feed.sock",
    protocol="lightstream",
)

reader.register_table(
    "quotes",
    representative_table,
)
reader.register_message("health")

for frame in reader:
    if frame.is_table():
        on_quotes(frame.table)
    else:
        on_health(frame.payload)

Read/Write tables over TCP in Rust:

use lightstream::models::writers::tcp::TcpTableWriter;

let mut writer = TcpTableWriter::connect("127.0.0.1:9000", schema, None).await?;
writer.write_table(batch_1).await?;
writer.finish().await?;

use futures_util::StreamExt;
use lightstream::models::readers::tcp::TcpTableReader;

let mut reader = TcpTableReader::connect("127.0.0.1:9000").await?;

while let Some(result) = reader.next().await {
    let table = result?;
    process(table);
}

In terms of the Rust tools and techniques, Lightstream makes use of:

1. Zero-copy techniques
2. 64-byte alignment for compatibility with std-SIMD - it retains this compatibility over the wire and to/from disk.
3. Arena-based allocations (via Minarrow)
4. Mmap (the mmap reader hits 170GB/s on my laptop when warm, essentially RAM-speed)
5. Optional features with linux sys calls such as io_uring (not in the benchmark figures)
6. Manual memory layouts, and performance tricks with that.

If you have any questions about the architecture I would be happy to explain it.

Ok then "Why" do it? Basically, The outcome is that one who prefers to work with an abstraction does not need to reason about bytes on the wire. Instead, they get highly optimised data transfer, straight out of the box, with common data formats and transports. It also includes a trait for any custom data format one would like to send on that one "Lightstream connection" (which is itself optional - you can just use Arrow if you want).

The project kicked off about 12 months ago when I started standardising such patterns after working in autonomous field communication integrated with data/ML, live trading, and some other industries where there was a lot of custom work required to re-assemble from multiple components. Therefore, I have essentially aimed to package those learnings up into a tool to make data transport smoother and easier for everyone.

This r dataengineering link includes the full results, where every effort has been made to be fair (and where Lightstream wears a penalty due to stronger ordering guarantees). Unfortunately I couldn't post it here as r Rust has a limit on image content.

Please feel free to give it a run would love to know your thoughts and if you find it useful.

If you have any questions about it, or helpful suggestions please feel free to leave a comment below. If you like what you see, please consider leaving a star and/or sharing the repository, as it will help people find it easily.

Thanks a lot.

Pete

Results:

Workload Shape

Mixed

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.939	1.109	1.18x
4	3.262	4.005	1.23x
8	5.138	5.677	1.10x
16	5.142	5.784	1.12x

Numeric

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.649	1.109	1.71x
4	2.901	4.307	1.48x
8	4.851	5.693	1.17x
16	5.434	5.780	1.06x

String Heavy

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.828	1.109	1.34x
4	3.052	3.861	1.27x
8	4.899	5.782	1.18x
16	5.217	5.790	1.11x

Wide (100 cols)

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.695	1.108	1.59x
4	2.685	3.790	1.41x
8	4.549	5.725	1.26x
16	4.911	5.753	1.17x.

Hi everybody,

I am excited to announce the release of Lightstream, a step change capability for high-performance data transport, that makes it essentially effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or even piped out to the terminal so an agent like Claude can watch the live batch stream in real time (example in repo).

Furthermore, Lightstream exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable). As a bonus, Lightstream is straightforward to setup with essentially zero configuration other than optional TLS certificates and your Cargo package/pip install, and endpoint addresses.

So what is Lightstream? It is Rust package with Python bindings, that builds directly on Minarrow ( which is in turn a high-performance implementation of the Apache Arrow memory layout in Rust, tuned for SIMD compatibility). Lightstream implements Arrow IPC, Parquet encoders/decoders from scratch, up to Arrow readers/writers and IPC stream protocol, with mmap and few of these niceties. But, in a manner, that is fully composable and leaves you de-coupled at any layer, to customise things architecturally. The crux then is the transport layer on top, which natively supports interchanging any of the following transport formats:

• TCP
• HTTP
• QUIC
• Websocket
• Webtransport
• UDS (pipe your data from your Rust process to Python or two Python programs plug and play )
• Stdio (pipe your data program output straight into the terminal for something else to pick it up

And finally, the (optional) Lightstream protocol, which then combines the Arrow/Proto/MsgPack and any other custom types you want to send.

I’ve found this is really cool in practice, where you don’t need to reason about or work with bytes, or separately build your own protocol to get arrow and protobuf playing well together over the network. It is plug and play, see syntax examples attached.

In Python, every item in the stream can talk to Polars or DuckDB without any further changes - you can Duck SQL the feed or data process to your heart's content.

An example of things you can do with it:

• setup a live stream of data batches from your program A to program B
• send typed metadata via Protobuf on the same feed
• use it for straightforward live feed delivery between server and client (though not Web JS yet)
• useful if you have a central storage server you are pulling larger than memory data over the network to churn through (though, no S3 etc. it is node to node or process to process)

It is not:

• Kafka or a messaging broker. There is no resiliency / vertical scalability.
• A stream processing engine like Flink. It is for sending/receiving data only. You do polars on the other end or whatever you want with the Arrow-shaped data. That is a very different back-pressure/long-lived scenario and is not that kind of large-scale streaming. --> I.e., think quick and easy Websocket, and best for settings like EKS K8 pod to pod/containers, between EC2's or between processes on the same box, "light streaming".

Lightstream kicked off for me about 12 months ago when I started standardising patterns that have worked well for me in the past into something that reflects how I like to work when streaming data with control of both endpoints. It arose from regularly coming up against contexts requiring this capability operating in things like autonomous field communication integrated with data/ML, live trading, and some other industries where there was a lot of custom work required that I kept having to assemble from multiple components. Therefore, I have essentially aimed to package those learnings up into a tool to make data transport smoother and easier for everybody.

There are a couple of code examples attached, including the open benchmarking methodology, where every effort has been made to be fair (and where Lightstream wears a penalty due to stronger ordering guarantees).

Please feel free to give it a run would love to know your thoughts and if you find it useful.

If you have any questions about it, or helpful suggestions please feel free to leave a comment below. If you like what you see, please consider leaving a star and/or sharing the repository, as it will help people find it easily.

Thanks a lot.

Pete

Some of you may already be familiar with Minarrow, a from-scratch implementation of the Apache Arrow format in Rust. The project has grown considerably, particularly around Rust <-> Python interoperability, so I would like to share what it now enables.

What?: Apache Arrow is the columnar runtime underpinning major libraries such as Polars, DataFusion and, optionally, Pandas. Minarrow is a from-scratch implementation of the open Arrow format that now also lets you inline Python directly inside Rust.

The Pitch: Keep your application data strongly typed, SIMD-ready and native inside Rust, then connect to Python, Polars, DuckDB, scikit-learn and the wider Python data ecosystem at the boundary.

Bridge Benchmarks (Runnable in Github Repo):

Share 1 million Rows between Rust <-> Python	Time - Intel Core Ultra 7 155H, 32 GB RAM
Uncontended GIL acquisition	53 ns
Rust to Python	165 ns
Python to Rust	2.5 µs

Why?: Compatible and straightforward columnar data in Rust for running Python analytics and ML, or building custom algorithms on top of it.

Benefits - Python:

• Zero-copy bridge: Share Rust data with Python without serialising or copying it first.
• Pluggable: Pass the same zero-copy data into Polars, PyArrow, DuckDB and scikit-learn workflows through Arrow-compatible interfaces.
• Compact: pip package < 1.5 MB, comparable in size to nanoarrow, but backed by Rust rather than C.

Benefits - Rust:

• Fast SIMD : Data is automatically aligned to 64-byte boundaries, ensuring it is ready for compatible SIMD kernels and low-latency parallel processing.
• Fast Compilation: Compile times of < 2 seconds with default features.~0.15s rebuilds.
• Straightforward: The API is high-level including Pandas-style row and column selection.
• Strong typing: Columns remain strongly typed without relying on trait objects throughout the codebase, reducing runtime type checks, downcasting and manual casts. The compiler also provides a continuous feedback loop for developers and coding agents, catching type and schema mistakes directly in the IDE.

TLDR: How can I keep Rust-level performance and compile-time guarantees, make common data construction feel relatively Python-like, and still move the same data into Python analytics or machine-learning workflows without serialising and rebuilding it?

How: Minarrow keeps the application and data layer native in Rust, while allowing Python to be embedded for modelling and analytics.

For example, a Rust application can pass a Minarrow dataset directly into embedded Python, convert it to Polars without serialisation, train a scikit-learn model and return the result.

Who: Minarrow is intended for data and software engineers who are:

• Building data libraries or Python native extensions.
• Building live ingestion, streaming or off-the-wire processing systems.
• Using Rust for application, transport or data services and Python for analytics or machine learning.
• Producing data in Rust that will be consumed by Polars, DuckDB, PyArrow or pandas.
• Writing specialised SIMD-oriented native kernels.
• Building quantitative finance, risk, simulation or feature-generation systems.
• Looking for an application data model rather than a complete dataframe execution engine.
• Building embedded analytics or custom data infrastructure.

For many data engineers working primarily in Python, Minarrow may appear as the backing runtime of another library rather than something used directly.

Python Package:

pip install minarrow

That package is aimed at Rust-backed Python applications where columnar data needs to cross the language boundary cleanly while remaining usable by the broader Python data ecosystem.

Caveats:

• Minarrow currently supports flat tabular data only.
• Minarrow is not a dataframe or SQL execution engine. It is intended to provide the typed storage, native processing and interoperability layer underneath those systems.
• Minarrow is a from-scratch implementation inspired by the Apache Arrow memory layout and is not affiliated with the Apache Arrow project.

Links:

• Repository: github.com/SpaceCell/minarrow
• Rust crate: crates.io/crates/minarrow
• Rust API documentation: docs.rs/minarrow
• Python package: pypi.org/project/minarrow
• Python documentation: minarrow.org

License: Apache 2.0.

Sharing it here because I think some data engineers working on high-performance pipelines, Python/Rust bridges, embedded analytics, live data systems, or custom data infrastructure may find it useful. If you believe it is, a GitHub star is appreciated as it helps other people find the project.

Questions and feedback welcome.

Thanks everyone.

Code Examples:

Rust:

use minarrow::{fa_f64, fa_i32, fa_str32, tbl, Print};

let users = tbl!(
    "users",
    fa_i32!("id", 1, 2, 3, 4),
    fa_str32!("name", "alice", "bob", "charlie", "dan"),
    fa_f64!("price", 10.5, 20.0, 15.75, 7.25),
);

users.print();

// Pandas-style zero-copy row and column selection
let view = users
    .c(&["name", "price"])
    .r(0..3);

// With the `cast_arrow` feature
let batch = users.to_apache_arrow();

// With the `cast_polars` feature (Polars Rust)
let frame = users.to_polars();

Python (binds Rust):

import minarrow as ma

users = ma.Table(
    {
        "id": [1, 2, 3, 4],
        "name": ["alice", "bob", "charlie", "dan"],
        "price": [10.5, 20.0, 15.75, 7.25],
    },
    name="users",
)

frame = users.to_polars()
relation = users.to_duckdb()

Run a Random Forest Classifier inside Rust

// Run a Random Forest Classifier using Python inside Rust
let result = rt.with_python(&dataset, |py, obj| {
    let scope = PyDict::new(py);
    scope.set_item("table", obj)?;

    py.run(
        cr#"
import polars as pl
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

df = table.to_polars()

features = ["x0", "x1", "x2", "x3"]
X = df.select(features).to_numpy()
y = df["label"].to_numpy()

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=0,
)

model = RandomForestClassifier(
    n_estimators=100,
    random_state=0,
)
model.fit(X_train, y_train)

predicted = model.predict(X_test)

result = pl.DataFrame(
    {
        "actual": y_test.astype("int64"),
        "predicted": predicted.astype("int64"),
    }
)
"#,
        Some(&scope),
        Some(&scope),
    )?;

    scope
        .get_item("result")?
        .ok_or_else(|| pyo3::exceptions::PyKeyError::new_err("result not set"))
})?;

Hi Everyone,

Providing an update on the latest Minarrow release, which has had major updates elevating it to what I believe is now ready more widespread use.

For anyone unfamiliar, Minarrow is an original implementation of the Apache Arrow format in Rust, focused on tabular data use cases. It's key advantages are:

1. Guaranteed 64-byte SIMD alignment via a custom allocator
2. Fast - due to retaining strong typing throughout on top of zero-copy
3. Ergonomic - easy to use
4. Fast compile times < 2 seconds on the std build
5. Minimal - only 2 tiny external dependencies (num-traits + log), and Vec64 which is implemented from scratch

On top of this, it has Python bindings and PYO3 integration, which, on my consumer laptop, can send anywhere from 1m to 10m+ rows to Python in ~165ns, plus acquiring the GIL lock for ~50ns, and the return trip (Python -> Rust) is more like 3 microseconds (as it is a pointer). You can easily embed (inline) Python, for e.g., Machine learning scripts natively within Rust:

// Run a Random Forest Classifier using Python in Rust

let value = rt.with_python(&dataset, |py, obj| {
    let scope = PyDict::new(py);
    scope.set_item("table", obj)?;
    py.run(
        cr#"
        import polars as pl
        from sklearn.ensemble import RandomForestClassifier
        from sklearn.model_selection import train_test_split

        df = table.to_polars()
        features = ["x0", "x1", "x2", "x3"]
        X = df.select(features).to_numpy()
        y = df["label"].to_numpy()

        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
        model = RandomForestClassifier(n_estimators=100, random_state=0)
        model.fit(X_train, y_train)
        predicted = model.predict(X_test)

        result = pl.DataFrame(
            {
                "actual": y_test.astype("int64"),
                "predicted": predicted.astype("int64"),
            }
        )
        "#,
        Some(&scope),
        Some(&scope),
    )?;
    scope
        .get_item("result")?
        .ok_or_else(|| pyo3::exceptions::PyKeyError::new_err("result not set"))
})?;

Notably, in a production environment you might want process isolation for that kind of workload, but for non-critical and development workloads I've found it works really well.

Would love any feedback anyone has on it. If you have any questions, or if there are any particular features you would like to see or care about, please let me know, I'll consider it.

There are also a bunch of std::simd -compatible kernels in there that are relatively easy to add Rayon on top of you need some fast minimal and standard analytics, otherwise as a data landing zone. Because it's compatible with the Apache Arrow format, PyCapsule, and the C Data Interface, from services sitting in Rust it's been useful for landing data and bridging to these run-times (including Polars in Rust via .to_polars()).

I am sure someone is going to hit me up and say "Where are the Miri tests"? They are next on the list. The FFI roundtrips are fully unit tested exhaustively across types, and most unsafe you see in the codebase is for '.get_unchecked()' which is skipping bounds checks on windowed vectors with a known length.

Anyway, I am sure that's a lot of info.

Keen to hear your thoughts. if you would like to show support please consider starring the repository so that people who would find it useful can find it easily. I have spent roughly 12-15 months working on it.

There's more info and the runnable benchmarks are available in the repository.

Note this project is not affiliated with Apache Arrow who do excellent work. I built it as I wanted a lighter weight Arrow-runtime that was a closer fit for my specific requirements, despite being a massive fan.

Looking forward to your feedback.

Thanks,

Pete

https://petition.parliament.uk/petitions/764909

Lately, venues have been adding hidden preset tip values, including vendors like Dojo that hide them to automatically tip when you tap your card. These routinely add over £2 to already expensive drinks and food, during a cost of living crisis.

Please sign and share this official petition to empower the people and eradicate this untoward practice from London and UK society.

Hi all,

I built the open source columnar data library in Rust called Minarrow that implements the Apache Arrow memory format.

Here's what it supports:

• Tables ("Record Batches") and Arrays

• Stream compatible tables/arrays

    // Append chunks as they arrive 
    let mut super_table = SuperTable::new(); super_table.push_table(batch1); 
    super_table.push_table(batch2);
    // Consolidate to single table when ready let table = super_table.consolidate();
  

• Typed arrays (Integer, Float, Boolean, Categorical, Datetime). Note these are ergonomic and use Rust generics to avoid type sprawl, which can be an issue in related libraries.

    // Create arrays with macros 
    let ids = arr_i32![1, 2, 3, 4]; 
    let prices = arr_f64![10.5, 20.0, 15.75]; 
    let names = arr_str32!["alice", "bob", "charlie"]; 
    let flags = arr_bool![true, false, true];
  

• Pandas-like selection in Rust

        // Ergonomic selection
        let view = table.c(&["name", "value"]);  // columns
        let view = table.r(10..20);               // rows
        let view = table.c(&["A", "B"]).r(0..100); // both
        
        // Materialise only when needed
        let owned = view.to_table()
  

• To/From Polars and Apache Arrow + full PYO3 support

    // To Arrow
    let arrow_array = minarrow_array.to_apache_arrow();
    
    // To Polars
    let series = minarrow_array.to_polars();
    
    // FFI via Arrow C Data Interface (plus PyCapsule support)
    let (array_ptr, schema_ptr) = minarrow_array.export_to_c();
  

• Compiles in < 2 seconds

• Fully SIMD support via Vec64

• Apache Arrow Licensed

If you are the type of data engineer who likes to work directly with the metal, or are building data apps in Rust that wants your underlying data core to be solid whilst avoiding too many dependencies this could suit you.

Otherwise, if you are working with Rust for data engineering, or have been considering it, I'd love your feedback.

Thanks a lot.