Export Calc schemas once¶
Applies to: native Calc operators
Arrow C Data transfers the batch buffers without copying them, but an ArrowSchema is a separate
tree of field metadata, native strings, and release callbacks. Most batches reuse the same schema,
yet the original bridge rebuilt and exported that identical tree for every physical batch.
The first batch of each distinct input schema exports both its ArrowArray and ArrowSchema.
The operator keeps a separate compiled handle for each schema, since a UNION ALL can mix batches
with and without the hidden row-kind column. Rust caches the resulting struct type alongside that
compiled Calc. Later batches with the same schema use Arrow Java's array-only export and
Arrow Rust's from_ffi_and_data_type, so only buffer ownership and child-array layout cross JNI.
This preserves the standard C Data ownership contract: the Java array's release callback is still
consumed exactly once, and the schema remains owned by the native Calc handle until close.
The Q8 profile that identified the issue measured repeated Calc schema export at 120 samples in a 30-second profile (4.00 samples/second). After the change it fell to 14 samples in 25 seconds (0.56 samples/second), an 86% reduction; the remainder is the required first schema for each new job in the profile loop. Inclusive Calc CPU fell by roughly 12% after normalizing against completed jobs.
The 2-million-event exactly-once Kafka Q8 timing remained within run-to-run noise: 1.472 seconds before and 1.499 seconds after, while the adjacent Flink baseline moved from 1.617 to 1.845 seconds. The optimization is retained because it removes measured allocator and schema-tree work without changing batch contents, but no end-to-end Q8 gain is claimed from that noisy pair.