Evaluate collection edits on Arrow batches¶

The existing collection extension can execute verified ARRAY_APPEND signatures without rebuilding per-row JVM containers. The kernel uses the list offsets to copy a row's visible child range and its corresponding scalar element through Arrow's mutable array builder. It dispatches storage types in Arrow rather than converting values to DataFusion scalars. A NULL container stays NULL; an empty container becomes a one-element array, and a NULL appended element is retained. Sliced offsets and DECIMAL precision/scale remain intact.

Flink's getter, string representation, NOT NULL-element and failing-expression restrictions remain enforced by the planner. This replaces evaluation inside the existing extension; its configuration is unchanged. Standalone conversion costs and the accepted regressions are recorded below; subsequent boundary improvements are measured in the collection representation ledger.

The array_edits Criterion suite invokes the production compiled Calc with independently constructed expected arrays, sliced inputs, empty batches, short and long arrays and nullable parents/elements/needles. Allocation requests and shared output buffers are recorded separately from elapsed time. Whole-job measurements retain both row/Arrow transposes, JNI, startup and the rowwise sink.

Whole-job measurements¶

Baseline revision 8798b8e6; Apple M4 Pro (14 cores), macOS arm64, JDK 17.0.12 and Flink 2.2.1; release Rust with mimalloc and no RocksDB. Parallelism one, 2 GiB heap, rowwise source and blackhole sink. Every cohort uses a fresh JVM, two warmups per engine and five alternating measured trials; both transposes, normal copies, JNI, planning and startup are included. No task builds or other tests overlap timing; desktop activity is uncontrolled and CPUs are not pinned. The baseline ran before these functions were changed. Times are medians in seconds; negative deltas mean less elapsed time. Baseline stock controls remain visible to expose drift. These are complete-job results, including startup, rather than isolated sustained kernel throughput.

Function	Rows	Prior stock	Prior SF	New stock	New SF	Versus prior SF	Versus paired stock
ARRAY_APPEND	2,000,000	0.558645	0.674472	0.522853	0.448795	-33.46%	-14.16%

ARRAY_APPEND trials and ranges, fixture and library checksums, and 90 allocation probes retain all observations. This short-array BIGINT case improves on both controls; it does not establish a gain for every type or width.

Validation: six Flink 2.2.1 runtime SQL tests cover BOOLEAN, all integer widths and exact precision-38 DECIMAL, including NULL appended elements, empty arrays, multiple batches, filters and all-filtered batches. The Criterion suite verifies 90 independent expected results; the scalar registry covers opcode 182 in all nine null/Unicode/batch profiles. Full stateful recovery tests are outside this stateless scalar change's local validation.

Prepending elements¶

ARRAY_PREPEND uses the same checked offset builder, placing the scalar before the visible child range. It retains the same basic nullable-type admission and NULL-container/element rules. The SQL and Criterion fixtures independently check the resulting element order, including sliced and empty inputs.

Removing elements¶

ARRAY_REMOVE compares matching basic elements in Arrow and gathers retained indices in original order. Unlike DataFusion/Spark removal, a NULL needle removes all NULL elements rather than producing NULL. NULL containers stay NULL. Repeated matches, sliced buffers, exact wide DECIMAL and all-filtered batches have parity fixtures.

Concatenating arrays¶

ARRAY_CONCAT copies visible child ranges in argument order and intersects parent validity. Every argument must be present, matching Flink rather than DataFusion's NULL-skipping concatenation. The builder works on Arrow storage for matching basic nullable element types; it does not reconstruct individual JVM containers. Variadic and independently NULL operands are covered by runtime SQL tests.

Union of arrays¶

ARRAY_UNION reuses DataFusion 54's released stable-union implementation for matching nullable basic types. It preserves first occurrence across the left input then the right input, including one NULL element, and returns NULL if either container is NULL. The planner retains generated evaluation for other representations and fallible operands. SQL fixtures exercise distinct left/right lengths and both operand orders rather than assuming that a set operation may reorder its result.

Stable array difference¶

ARRAY_EXCEPT dispatches once to typed integer, decimal or Boolean values. A reused hash set records the right input and then each emitted left value, so NULL is a first-class key, duplicates disappear and left occurrence order is preserved. Only the selected left values are gathered; per-row JVM objects and serialized row keys are unnecessary. Independent expected results and differing operand lengths exercise NULLs, duplicate values and stable order.

Left-ordered intersection¶

ARRAY_INTERSECT builds a reusable typed lookup set from the right input and probes the left in occurrence order, removing each matched key after emission. This preserves Flink's left order and NULL equality without another deduplication pass. DataFusion 54 instead probes the longer input, which can reorder a shorter left array; that implementation cannot be used unchanged. Tests cover both relative input lengths, duplicates, NULL keys, empty arrays and exact DECIMAL values.

Sorting verified arrays¶

ARRAY_SORT reuses DataFusion 54 for nullable integer/DECIMAL arrays whose Flink comparator is verified safe: fewer than 32 elements, or a native ARRAY_DISTINCT result. Literal Boolean options map explicitly to ASC/DESC and NULLS FIRST/LAST; the two-argument Flink form puts NULL first exactly when ascending is true.

Unrestricted arrays retain generated Flink evaluation. Flink's comparator returns -1 on equal non-NULL values; long duplicate-bearing arrays can therefore throw IllegalArgumentException: Comparison method violates its general contract! in Java TimSort. Substituting a total-order Arrow sort would change that observable behavior. A 64-element runtime SQL reproducer protects this failure path. Runtime and nullable options, strings, nested types and fallible operands also retain Flink evaluation. This is deliberately partial native admission.

Extracting a singleton¶

ELEMENT selects a nullable child index for each basic array and gathers it with Arrow. Empty and NULL arrays produce NULL; multiple elements produce the released exception class and message, carried through the existing typed-error bridge. The planner counts ELEMENT as a failure site so other failing expressions cannot be reordered across rows. The scalar registry covers empty, singleton, NULL-element and NULL-parent arrays; SQL tests cover multi-element failures, all-filtered inputs and competing arithmetic errors.

Constructing scalar arrays¶

Projected runtime ARRAY expressions reuse the existing native scalar constructor for matching nullable Boolean, integer, DECIMAL and temporal elements. The released DataFusion builder interleaves columns into flat Arrow child storage. The existing compact/expanded timestamp normalization retains Flink precision and pre-epoch behavior. No new JNI protocol or container sidecar is introduced. Fallible, STRING/nested and NOT NULL-element projections retain their previous evaluator. SQL fixtures verify typed NULL literals, exact wide decimals, DATE/TIME and TIMESTAMP/LTZ at millisecond and nanosecond precision.

Constructing maps from arrays¶

MAP_FROM_ARRAYS shares the flattened input buffers when their offsets agree and aligns visible ranges otherwise. Duplicate and NULL keys are retained, both parents must be present, and length mismatch uses Flink's exception class/message. The existing List-of-entry-Struct carrier permits nullable keys without unsafe Arrow construction or a new ownership protocol. Its marker lives on the child field so expression type inference preserves it. Only basic nullable key/value types are admitted; representation-sensitive inputs retain generated evaluation.

Shared map key views¶

MAP_KEYS reuses Arrow entry-key buffers, offsets and parent validity for ordinary maps and verified map-list carriers. It follows the released DataFusion/Comet projection pattern while accepting Flink's nullable-key representation. No per-entry gathering or JVM container reconstruction is necessary.

MAP_VALUES uses the same verified basic-map admission and shared-buffer view, selecting the value child without changing NULLs or entry order.

Released ELEMENT exception types¶

The Flink compatibility profile selects ELEMENT's released exception class without an extra per-row argument, lookup or reflective class detection. Flink 1.18 uses RuntimeException and Flink 2.2 uses TableRuntimeException; both use the exact same array kernel and message. This changes only the error path and does not alter the successful kernel or the complete-job measurements above.

ARRAY_PREPEND complete-job evidence¶

Query: ARRAY_PREPEND(arr,CAST(idx AS BIGINT)), 1,000,000 rows, BIGINT, width 64. Method and resources match the complete-job method above.

Cohort	Median seconds	Range seconds
baseline stock	0.579178	0.571242-0.585162
baseline streamfusion	1.030478	1.010875-1.038272
candidate stock	0.580153	0.567207-0.587232
candidate streamfusion	0.603231	0.594722-0.607353

Elapsed time changes by -41.46% versus the prior evaluator and +3.98% versus paired stock Flink. The stock regression remains part of this historical result; native coverage is not a universal standalone speedup.