Skip to content

Export Calc schemas once

Applies to: native Calc operators

Arrow C Data transfers the batch buffers without copying them, but an ArrowSchema is a separate tree of field metadata, native strings, and release callbacks. A Calc's input schema cannot change during one operator instance, yet the original bridge rebuilt and exported that identical tree for every physical batch.

The first batch still exports both its ArrowArray and ArrowSchema. Rust caches the resulting struct type alongside the compiled Calc. Later batches use Arrow Java's array-only export and Arrow Rust's from_ffi_and_data_type, so only buffer ownership and child-array layout cross JNI. This preserves the standard C Data ownership contract: the Java array's release callback is still consumed exactly once, and the schema remains owned by the native Calc handle until close.

The Q8 profile that identified the issue measured repeated Calc schema export at 120 samples in a 30-second profile (4.00 samples/second). After the change it fell to 14 samples in 25 seconds (0.56 samples/second), an 86% reduction; the remainder is the required first schema for each new job in the profile loop. Inclusive Calc CPU fell by roughly 12% after normalizing against completed jobs.

The 2-million-event exactly-once Kafka Q8 timing remained within run-to-run noise: 1.472 seconds before and 1.499 seconds after, while the adjacent Flink baseline moved from 1.617 to 1.845 seconds. The optimization is retained because it removes measured allocator and schema-tree work without changing batch contents, but no end-to-end Q8 gain is claimed from that noisy pair.