Arrow as zero-serialization target

[infogulch]

Hi! I've just found iggy, and I noticed the roadmap mentions zero-copy deserialization. Have you considered using apache arrow as a message serialization format? Arrow may provide interoperability benefits, considering it can be read and written from many languages without copying.

[hubcio]

hello! to be honest, we didn't consider it. right now are intensely experimenting with multiple approaches. we tested rkyv, besides that we also cooked our own zero-copy format. Thank you for this suggestion, I will get back to you in some time with some more thoghts; i'm sure @numinnex will have more comments 🦀 .

[infogulch]

👋

I appreciate that in an experimentation phase you want to keep your options open. Arrow seems to be a practical choice since it is a pretty complete standard, supports a wide array of languages ¹, it would likely be a drop-in implementation, and it is also an apache project. (Maybe another option is spiraldb/vortex which is in a nascent stage but has some neat features.)

I just hoped to get arrow on your radar before you guys settled on something since I hadn't seen it mentioned anywhere in the repo yet.

🚀

Footnotes

1. https://arrow.apache.org/docs/status.html ↩

[numinnex]

Hello @infogulch I've took a quick peek at Apache Arrow and from what I understand it's an in memory zero-copy columnar data format. They've built some serialization infrastructure that can be used for IPC/Network transportation, but nevertheless they aren't natively a network format, rather an in-memory one, which would make it a bit problemtic, since we end up both on network and on-disk (they seem to support Parquet tho, which could help with the latter). On top of that them being strictly columnar data format makes it less desirable as we are schemaless general purpose message streaming platform. How I see that in the future, when we would support some sort of schema registry (like Avro), we could potentially create integrations with the Apache Arrow eco system that would allow to create arrows from avro and use that for streaming analytics.

[numinnex]

We attach to each message our own metadata for the internal use, by having a zero copy over the Message struct, means that we don't need to deserialize it in order to access those metadata fields and read/update those. On top of that iggy uses batching for better tput, which means that realistically you end up having Vec<Message>, rather than individual message, by employing zero copy binary schema, we can avoid the Vec allocation at all and just work with raw bytes, that either provide an "view" into the array of messages or lazily deserializes those.

[infogulch]

we don't need to deserialize it in order to access those metadata fields

Oh I see, so you're referring to avoiding copying when iggy accesses internal metadata. That makes sense to me.

iggy uses batching for better tput, which means that realistically you end up having Vec

In that case maybe you should reconsider Arrow; having all the metadata arranged in dense arrays separate from message content could be beneficial for cache locality and simd vectorization.

[numinnex]

In that case maybe you should reconsider Arrow; having all the metadata arranged in dense arrays separate from message content could be beneficial for cache locality and simd vectorization.

Yeah this is something that we are considering as well, in fact @hubcio currently is rewriting my batch based implementation to a more fine grained one, but grouping the metadata together and storing it separately from the messages data is something that we are still considering as an option, maybe not in the form of Arrow format, but rather using other zero copy layouts, for example from the already mentioned rkyv or other crates such as musli, maybe even if we'd made those arrays fixed size (which is totally doable), we could get away with some pointer cast from the struct -> bytes, either using rust native apis or bytemuck crate.

[infogulch]

Nice, I hope you find a good solution.

[numinnex]

Great discussion, thank you for your input, it actually made us go beyond our initial assumptions about how we would like to go about it, also vortex project looks really interesting, I am pretty sure we can learn a lot from their findings.

[numinnex]

I am yet to look at Vortex, which seems like a really interesting project, but I assume the argument would follow same reasoning as in case of Apache Arrow.

[infogulch]

I think Vortex has answers for most of the above arguments against Arrow:

• It's neither row- nor column-first, you can use any layout, including arbitrary chunked groupings like parquet row groups (see example at the bottom of the layout docs page)
• It is designed for disk and has a file format

[numinnex]

I see that they support different layouts, but still how we will end up using this is most likely as an additional connector, e.g exposing an endpoint that allows any data lake to connect and read our data that is transformed to parquet/vortex, afaik redpanda topics with Iceberg feature turned on, using the tiered storage architecture, migrates the avro compatible schema messages to parquet and allows to directly read data in parquet format from the borker.

[v1gnesh]

A couple of thoughts, if I may...

1. I think implementing a Kafka-compatible API is necessary for wider adoption.
2. Check out arrow-avro - https://arrow.apache.org/blog/2025/10/23/introducing-arrow-avro/

[v1gnesh]

I've also asked the Arrow question to buf (bufbuild/buf#4141).
I get that Arrow (columnar) in a streaming context means needing to wait for a chunk of data, but Arrow will be fully serde free, all the way from client-side python dataframes, over the wire (say Arrow Flight), onto query engines (Datafusion), and databases (Snowflake, etc.), data lakes (of parquet files).
In spice.ai, there is even use of moka-rs as an in-line cache that caches Arrow RecordBatches.

[spetz]

@v1gnesh Kafka-compatible API won't happen, at least not anytime soon, unless someone decides to write a proxy/shim/adapter. We can't rely on a protocol over a decade old and have such a big legacy to be supported by Iggy. If you want to innovate, provide the best performance possible, you need to come up with new schemas and APIs that will perfectly work with each other.

As for Arrow support (or any other specialized format type), we've been thinking of some extensions to the underlying Iggy storage, but it's now more on the research side.

[v1gnesh]

See how Buf did it. You only need to adopt the portions of it that make sense for Iggy, while allowing vanilla Kafka folks to see Iggy as an alternative.

[spetz]

If anyone in the community is up to doing so, we could give it a try, but adjusting the Iggy API schema to support Kafka, is not our goal. Truly, if one really needs to use Kafka, just use it instead.