Skip to content

Scalar function benchmarks

Question Start here
Is a function currently admitted? Calc coverage
What is the recorded fixture and timing method? Inputs, reproduction, raw data
What was measured for collection and metadata extensions? Extended coverage
Where are the SQL/JSON comparisons? SQL/JSON measurements, nested boundaries, string consumers
Which measurements are experiments or diagnostics? Evidence index, array extrema experiment, BINARY exit experiments

Each section records its own implementation and environment. Historical timings and fallback observations remain evidence for that cohort; use the coverage page for the current supported contract.

The original measurements below were made on 2026-09-09 against the production implementation in 69122d2b, using Apple M4 Pro, JDK 17, UTC, Flink 2.2.1 and DataFusion 54.0.0. The native library uses the standard Maven bench profile: release mode, mimalloc, and the development connector features.

Every query evaluates one function over 2,000,000 rows at parallelism 1. Both engines run serially in the same JVM for each scenario, alternating which engine runs first on each trial. Each engine gets two warmups and five measured trials per case; the tables show median elapsed seconds. Each scenario uses a fresh JVM. No other benchmark or test suite runs concurrently.

Times include SQL planning/execution, a rowwise DataStream source, and a rowwise blackhole sink. The harness checks that every native function and identity-control plan includes NativeCalc, RowDataToArrow, and ArrowToRowData. Ratios are Flink time divided by native time: above 1 means native is faster for that workload; below 1 means it is slower. Small differences can be run-to-run noise, and these measurements do not isolate kernel cost.

The original 48 cases cover 35 retained functions, including integer widths and literal/column search parameters. All cases run with the 264-byte ASCII/non-null and Unicode/NULL scenarios; the ten search cases also run with 8-byte ASCII padding, giving 106 Flink/native comparisons. TO_TIMESTAMP and temporal FLOOR/CEIL/CEILING were outside that measurement's coverage. Their subsequent implementation is measured in the temporal diagnostic below. See Calc / filter for the complete argument gates.

Extended function coverage

These measurements were recorded on 2026-10-17. The collection and metadata extensions are enabled by default under the maintainer's explicit acceptance of the measured regressions on 2026-10-17. Native Calc coverage includes both Rust kernels and complete Flink-generated programs through the JVM batch bridge; these are separate execution engines. Runtime UUID already uses the ordinary admitted path and is unchanged by this extension. See coverage and type gates.

Measurements use Linux x86-64 on an Intel Core i7-12650H, JDK 17, Flink 2.2.1, release/mimalloc, parallelism 1 and a 2 GiB test heap. Each case alternates stock Flink and StreamFusion, with two warmups and five measured trials per engine. Both row/Arrow transposes, planning, row source and blackhole sink remain in the elapsed time. Independent JVMs measure the candidate and previous planner 3679d312. The previous planner rejects these newly supported functions and executes the host fallback. It shares the candidate's release DSO, but those fallback jobs do not call the new kernels. No other tests or benchmarks run concurrently.

These are whole-job timings, including startup and planning; subsecond cases do not establish sustained throughput. Paired stock controls and identity controls are retained without subtraction. Stock medians can drift between JVMs, so previous/candidate differences are not a precise isolated optimization measurement. The raw trials retain all five samples and both paired stock controls.

The metadata workload uses ScalarFunctionBenchmark: two million rows, 264-byte payload budget, TYPEOF(n) or CURRENT_DATABASE() alone, and SUM(n) grouped by the same expression. Pruning removes the unused input from the standalone metadata constant, retaining row counts. Metadata aggregates improve substantially, while standalone projections remain below Flink.

Metadata case Paired Flink (s) Candidate (s) Previous StreamFusion (s) Flink / candidate
Integer identity 0.235114 0.287956 0.325156 0.816x
BIGINT identity 0.245273 0.274623 0.291892 0.893x
TYPEOF 0.315037 0.353870 0.280670 0.890x
CURRENT_DATABASE 0.305233 0.351782 0.274232 0.868x
TYPEOF with aggregate 0.676733 0.349822 0.672136 1.935x
CURRENT_DATABASE with aggregate 0.689099 0.347060 0.642492 1.986x

Collection fixtures use DynamicCollectionBenchmark: nullable BIGINT arrays at widths 8/64, NULL containers every eight rows and NULL elements every seven positions. STRING join uses width 8, a 64-byte suffix and 200,000 rows; map fixtures contain three STRING/BIGINT entries. Other cases use two million rows except width-64 sort at 500,000, plus separate ten-million-row reverse and ROW-constructor cases. Runtime constructors use the nullable input index. Inline object update projects its integer field so the sink has an ordinary SQL type.

Collection case Paired Flink (s) Candidate (s) Previous StreamFusion (s) Flink / candidate
CARDINALITY, array width 64 0.900079 1.084205 0.915184 0.830x
CARDINALITY, array width 8 0.374140 0.418935 0.362862 0.893x
CARDINALITY, map width 3 0.951833 0.923917 0.900526 1.030x
Append, width 8 0.488629 0.771157 0.456011 0.634x
ARRAY constructor 0.506142 0.611204 0.408114 0.828x
Join strings, width 8 0.245785 0.406909 0.254226 0.604x
MAP constructor 0.538187 0.782705 0.431044 0.688x
Map entries, width 3 1.291837 3.225086 1.177803 0.401x
Map keys, width 3 0.939283 2.051847 0.903832 0.458x
Inline object update, integer field 0.388066 0.476386 0.371599 0.815x
Reverse, width 64 1.260770 1.646519 1.309363 0.766x
Reverse, width 8 0.469198 0.510902 0.431597 0.918x
Reverse, width 8, 10M rows 2.138881 2.248478 1.820855 0.951x
ROW constructor 0.499804 0.801584 0.423589 0.624x
ROW constructor, 10M rows 2.285629 3.726311 1.829078 0.613x
Slice last two, width 64 0.951815 1.097785 0.926048 0.867x
Sort, width 64 1.342123 1.631030 1.349329 0.823x

The ten-million-row cases show that reverse and row construction still lose beyond the shorter runs. The 1.03x map-cardinality observation is small relative to control drift and does not establish performance admission. All other standalone collection/constructor cases in this October 17 cohort are slower than stock. The October 22 comparison records the subsequent standalone improvement pass for #392. The maintainer explicitly accepted merging these slower paths enabled by default on 2026-10-17. Both switches can be set false to restore host execution for these extensions. The conversion optimization records both the initial overhead reduction and the subsequent measured improvements; several standalone cases still lose to Flink.

The CSV labels candidate, previous and candidate_initial distinguish final code, previous production fallback and the five cases rerun after optimization. Profiled runs are excluded. The release DSO SHA-256 is 991cc2c091cf4e32ea6eb92be5d1114970bda791905ddcddabe09b973c467f26.

Case collection.expression
Reverse ARRAY_REVERSE(arr)
Slice ARRAY_SLICE(arr,-2,-1)
Append ARRAY_APPEND(arr,CAST(idx AS BIGINT))
Sort ARRAY_SORT(arr)
Join ARRAY_JOIN(arr,'|','?')
Map keys / entries MAP_KEYS(m) / MAP_ENTRIES(m)
Array constructor ARRAY[CAST(idx AS BIGINT),CAST(NULL AS BIGINT)]
Map constructor MAP[CAST(idx AS BIGINT),CAST(idx AS BIGINT)]
Row constructor ROW(idx,idx+1)
Cardinality CARDINALITY(arr) / CARDINALITY(m)

The object expression is (OBJECT_UPDATE(OBJECT_OF('tech.streamfusion.FlinkObjectFunctionsSqlHarnessTest$Item', 'a',idx,'b',CAST(idx AS STRING)),'a',idx+1)).a; the fixture class is included in the test sources. Use collection.map=true for map-source cases, collection.type=STRING and collection.bytes=64 for join, and the row counts/widths stated above. Other source cases use collection.map=false.

Reproduce an individual collection case after building the release library:

The current harness retains operator metrics and requires consumed native Calc input for admitted runs. It constructs a metrics-enabled MiniCluster before timing each job. Execution-time planning, deployment, runtime row sources, both row/Arrow transposes, production JNI and the rowwise sink remain inside timing; cluster construction, preliminary schema/explain checks and teardown are outside. The October 17 tables above used the earlier harness and are historical observations. Compare fresh stock and preceding/current StreamFusion cohorts using the same harness rather than subtracting absolute times across those methods. Per-trial logs retain warmups and native execution counters.

SF_BENCHMARK=true mvn -Pbench -pl :streamfusion-runtime -am test \
  -Dnative.build.skip=true -Dtest=DynamicCollectionBenchmark \
  -Dsurefire.failIfNoSpecifiedTests=false -Dsf.extraJvmArgs=-Xmx2g \
  -Dstreamfusion.experimental.collectionExpressions=true \
  '-Dcollection.expression=ARRAY_REVERSE(arr)' -Dcollection.map=false \
  -Dcollection.type=BIGINT -Dcollection.width=8 -Dcollection.rows=2000000 \
  -Dcollection.warmup=2 -Dcollection.runs=5

For previous-planner controls, use the same harness/library with -Dcollection.native=false. For metadata, use -Dtest=ScalarFunctionBenchmark#individualFunctions, -Dstreamfusion.experimental.metadataExpressions=true and -Dscalar.functions=BASELINE_integer,BASELINE_bigint,TYPEOF,CURRENT_DATABASE,TYPEOF_GROUP,CURRENT_DATABASE_GROUP.

SQL/JSON native-first routing (2026-09-18)

The production admission policy tries the existing native encoding first, then generates an entire JSON Calc with Flink when that fails. Existing simple paths retain their Rust kernels; slices and JSON_QUERY below use one JVM callback per Arrow batch. The scalar bridge boundary and intermediate JSON string identity restrictions still apply.

This release/mimalloc measurement ran on Apple M1 Max with the same Flink/JDK versions, 1,000,000 rows, 264-byte budget, two warmups, five alternating trials and row source/sink with both transposes as the prototype below. No other local test or benchmark ran concurrently. The unchanged simple functions act as regression controls; their small differences from the earlier native measurements are not an optimization claim.

Case Flink (s) StreamFusion (s) Flink / StreamFusion JSON evaluator
Simple-source identity 0.390342 0.655438 0.596x none
Array-source identity 0.474533 0.852896 0.556x none
JSON_VALUE, simple path 1.106959 0.796195 1.390x Rust
JSON_EXISTS, simple path 1.093376 0.739060 1.479x Rust
JSON_QUERY(s, '$.a[0:2]') 1.951007 2.823455 0.691x JVM
JSON_EXISTS(s, '$.a[0:2]') 1.744556 2.484502 0.702x JVM

The newly admitted functions are slower than stock Flink in isolation. This is a coverage route that avoids more handwritten parser semantics and permits surrounding operators to remain columnar; it is not evidence of a whole-query speedup. A larger stateful pipeline has not been benchmarked here. Identity controls are reported without subtraction. Raw trials retain every measured iteration.

Use the reproduction command below with -Dscalar.functions=JSON_VALUE,JSON_EXISTS,JSON_QUERY_SLICE,JSON_EXISTS_SLICE.

SQL/JSON JVM bridge prototype (2026-09-18)

This experiment routes a complete SQL/JSON Calc through Flink-generated JVM code using StreamFusion's existing batch UDF bridge. It preserves row evaluation order and removes Calc's native path-grammar gate. It does not remove the Rust kernels, alter JSON format connectors, or establish that a whole-query JVM replacement should ship.

On Apple Silicon, JDK 17, UTC and Flink 2.2.1, each case processes 1,000,000 rows at parallelism 1, with a 264-byte payload budget, no injected NULLs, two warmups and five measured trials. The common native library is built in release mode with mimalloc. Each route runs in a separate JVM, alternating against stock Flink within that JVM. The JVM route was measured first, then the existing native route. Both row/Arrow transposes, rowwise source and blackhole sink remain in the measured path; plans assert native Calc and both transposes. No other local tests or benchmarks ran concurrently. These are whole-job elapsed medians, including planning, not isolated JSON kernel timings.

Case Flink, native run (s) Rust route (s) Flink, JVM run (s) JVM bridge (s) JVM / Rust time
Simple-path identity 0.387916 0.659267 0.391868 0.660814 1.002x
Wildcard-source identity 0.482458 0.900425 0.472690 0.872888 0.969x
JSON_VALUE, simple path 1.108530 0.838918 1.112615 1.755095 2.092x
JSON_EXISTS, simple path 1.115314 0.748365 1.107832 1.669865 2.231x
JSON_VALUE, wildcard 4.207545 1.721668 4.243582 5.190559 3.015x
JSON_EXISTS, wildcard 3.677213 1.565452 3.683537 4.477114 2.860x

The JVM prototype costs 2.09–3.02 times the existing Rust route on these cases and 1.22–1.58 times its paired stock Flink control. Controls are not subtracted. Wildcard JSON_VALUE takes the lax EMPTY path; this is not a scalar-extraction speedup comparison. The steady paired Flink and identity controls help compare the separate JVMs, but the experiment does not measure stateful downstream pipelines or wider/more complex JSON.

The blanket replacement was rejected. Current routing retains native encoding first and uses the generated JVM Calc only when a JSON Calc exceeds that admission. The bridge adds dynamic paths, JSON_QUERY, complex selectors and exact error handling without more native parser extensions. A representative larger-pipeline benchmark remains future work. The table above records the rejected blanket replacement, not the current routing of those four cases.

Raw trials: JVM route and existing native route. In each CSV, engine=native means StreamFusion enabled; in the first file its JSON evaluator runs on the JVM. The existing route uses planner c0677de0 and the prototype's common release DSO: its only native change is filtered row-UDF output handling, which the existing JSON kernels do not call. The selected simple/wildcard grammar is shared with 286c5b88; no union/slice extension is exercised. The benchmark harness is unchanged between runs.

Build the library with cargo build --manifest-path native/Cargo.toml -p streamfusion --release --features mimalloc, then run this in each planner checkout with that same release library (using distinct output paths):

SF_BENCHMARK=true mvn -B -ntp -Pbench -pl :streamfusion-runtime -am \
  -Dnative.build.skip=true '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dscalar.functions=JSON_VALUE,JSON_EXISTS,JSON_VALUE_WILDCARD,JSON_EXISTS_WILDCARD \
  -Dscalar.rows=1000000 -Dscalar.bytes=264 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.output=target/sql-json-comparison.csv test

Earlier SQL/JSON tables below describe the Rust route before this prototype, including its former direct-projection admission. Current branch coverage is on Calc/filter.

Nested SQL/JSON batch boundaries (2026-09-19)

The generated JVM Calc now accepts recursive ARRAY/ROW inputs and results through the existing batch callback. This extends host-exact coverage inside a columnar plan; it does not establish a JSON speedup. MAP/MULTISET boundaries still use planner fallback.

On Apple M1 Max, JDK 17 and Flink 2.2.1, each case processes 2,000,000 rows at parallelism 1 with the release/mimalloc native library. The rowwise source provides arrays containing a 264-byte ASCII string, a NULL element and tail, with the whole array NULL every seventh row. The second column is {"items":[1,2,3]}. Two warmups and five measured trials per engine alternate execution order. Plans require NativeCalc and both row/Arrow transposes, with a rowwise blackhole sink. No other local tests or benchmarks ran concurrently. Whole-job elapsed medians include planning; the identity control is not subtracted.

Case Flink (s) StreamFusion (s) Flink / StreamFusion
Array identity control 1.243185 1.968439 0.632x
JSON_STRING(a) 2.774218 3.251184 0.853x
ROW(a, JSON_QUERY(s, '$.items[*]')) 2.639063 4.891670 0.539x

Both JSON cases are slower than their previous stock-Flink fallback in this row-fed comparison. The benefit is a verified batch boundary that avoids adding another JSON implementation and can compose with other columnar operators. A representative larger pipeline comparison remains future work; these results do not establish that such a pipeline is faster. The existing native JSON fast paths retain their current admission.

Raw trials: nested JSON batch comparison. engine=native means StreamFusion enabled, with Flink-generated JSON evaluation on the JVM. Build the release/mimalloc library, then reproduce with:

SF_BENCHMARK=true mvn -B -ntp -Pbench -pl :streamfusion-runtime -am \
  -Dnative.build.skip=true '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dscalar.functions=JSON_STRING_ARRAY,JSON_QUERY_ARRAY_RESULT \
  -Dscalar.rows=2000000 -Dscalar.bytes=264 -Dscalar.nullEvery=7 \
  -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.output=target/sql-json-nested.csv test

Spaced JSON paths diagnostic (2026-09-16)

Definite bracket paths with ASCII spaces now stay in native Calc instead of falling back. On Apple M4 Pro, JDK 17, UTC and Flink 2.2.1, the release/mimalloc build measured the following end-to-end medians over 2,000,000 rows at parallelism 1, with a 264-byte ASCII payload budget, NULL every eighth row, two warmups and five measured trials per engine. Both row/Arrow transposes are included and asserted in the native plan. Engines alternate trial order; the source-matched identity control is reported without subtracting it.

Case Flink (s) Native (s) Flink / native
Identity control 0.822841 1.153666 0.713x
JSON_VALUE(s, 'lax $[ ''user'' ][ ''name'' ]') 1.682775 1.248065 1.348x
JSON_EXISTS(s, 'lax $[ ''user'' ][ ''name'' ]') 1.652583 1.202643 1.374x

Reproduce with the command below, selecting -Dscalar.functions=JSON_VALUE_SPACED_PATH,JSON_EXISTS_SPACED_PATH and -Dscalar.nullEvery=8. These numbers compare newly admitted queries against their previous Flink fallback; the change normalizes literal paths during planning and does not alter native JSON parsing. They do not establish a speedup for already admitted compact paths.

Empty JSON member diagnostic (2026-09-16)

Empty quoted names now use native JSON_VALUE/JSON_EXISTS. On Apple M4 Pro, JDK 17, UTC and Flink 2.2.1, the release/mimalloc build measured these end-to-end medians over 2,000,000 rows at parallelism 1, with 264-byte ASCII padding, NULL every eighth row, two warmups and five measured trials per engine. Both row/Arrow transposes are included and asserted. Engine order alternates; the source-matched identity control is reported without subtracting it. Documents alternate between an empty key containing Alice and a space-only key, which does not match the selected path.

Case Flink (s) Native (s) Flink / native
Identity control 0.633615 1.102641 0.575x
JSON_VALUE(s, 'lax $['''']') 1.559965 1.290161 1.209x
JSON_EXISTS(s, 'lax $[""]') 1.574280 1.150832 1.368x

Reproduce with the command below, selecting -Dscalar.functions=JSON_VALUE_EMPTY_MEMBER,JSON_EXISTS_EMPTY_MEMBER and -Dscalar.nullEvery=8. These results compare newly admitted expressions with their previous Flink fallback. The change removes admission restrictions and reuses the existing native member selectors; it does not speed up previously admitted paths.

DECIMAL UDF consumer diagnostic (2026-09-16)

Measured against the retracting OFFSET implementation plus DECIMAL UDF consumer support, using Apple M4 Pro, JDK 17, UTC, Flink 2.2.1 and the release bench profile with mimalloc. The selection was UDF_DECIMAL_IS_NULL,UDF_DECIMAL_NESTED, with 2,000,000 rows, two warmups and five alternating measured trials per engine. Both transposes and native Calc substitution were asserted. The source cycles through 999.995, -999.995, 1.235, -1.235, 0 and 9.99E+8, with every eighth value NULL. The UDF declares DECIMAL(5,2), so the inputs exercise rounding and overflow as well as ordinary values.

Case Flink (s) Native (s) Flink / native
STRING identity control 0.504798 1.106419 0.456x
decimal_from_text(s) IS NULL 0.496819 1.205532 0.412x
decimal_external(decimal_from_text(s)) 0.519048 1.362920 0.381x

These expressions are slower in isolation, including the identity control. The consumer expression uses Flink-generated JVM code through the existing batch UDF bridge so that external values, internal decimal overflow and generated null flags retain Flink's exact semantics. This removes an admission blocker inside larger columnar pipelines; it is not a Rust decimal-kernel optimization or evidence of a speedup for that composition. The nested consumer returns DECIMAL(38,9). Controls are not subtracted from timings. Reproduce with the command below, selecting -Dscalar.functions=UDF_DECIMAL_IS_NULL,UDF_DECIMAL_NESTED and -Dscalar.nullEvery=8.

Floating conversion and POWER diagnostic (2026-09-16)

Measured with Apple M4 Pro, JDK 17, UTC, Flink 2.2.1 and the release bench profile with mimalloc. Each query processes 2,000,000 rows at parallelism 1, with two warmups and five alternating trials per engine. Both transposes and native Calc are asserted; the source and blackhole sink remain rowwise. No other local tests or benchmarks ran concurrently.

The scalar DECIMAL(38,9) source alternates 12345678901234567890.123456700 and -0.000000100, with every eighth value NULL. Arrays contain both values and a NULL element, with every eighth container NULL. POWER uses the existing BIGINT number fixture with a runtime base cast to DOUBLE and exponent 0.5. The three source-matched controls run in the same JVM before the functions; controls are not subtracted from timings.

Case Flink (s) Native (s) Flink / native
DECIMAL identity control 0.550296 0.906067 0.607x
ARRAY identity control 0.645095 1.420928 0.454x
BIGINT identity control 0.613385 0.858960 0.714x
CAST(n AS FLOAT) 0.825372 1.019472 0.810x
CAST(a AS ARRAY<FLOAT>) 1.682023 1.603254 1.049x
POWER(CAST(n AS DOUBLE), 0.5) 0.619829 1.071053 0.579x

The array conversion has a small advantage in this workload; the scalar conversion and POWER are slower. The conversions execute in Rust, while default POWER uses Flink-generated JVM code to preserve its exact Math.pow contract. These results establish coverage costs, not a general speedup for floating expressions or larger composed pipelines. Reproduce with -Dscalar.functions=DECIMAL_TO_FLOAT,DECIMAL_ARRAY_TO_FLOAT,POWER_EXACT and -Dscalar.nullEvery=8 in the command below. CSV output quotes the declared result type so precision/scale commas remain inside one field.

Inputs

  • Search strings add row:/other: and :match/:miss around the padding: total lengths are 274/275 bytes for the 264-byte case and 18/19 bytes for the 8-byte case. Column needles and LOCATE start positions vary independently of the source text.
  • String fixtures use a 264-byte payload budget. URL_DECODE and JSON_UNQUOTE repeat complete escape groups within that budget; JSON_UNQUOTE adds enclosing quotes. DECODE uses valid UTF-8 bytes; malformed input belongs to the semantic tests.
  • The Unicode scenario also makes every eighth source value NULL, so Unicode and nullability effects are not isolated. Integer, date and timestamp values stay the same across the two scenarios; only nullability changes. UNHEX keeps ASCII hex input in both scenarios.
  • TO_DATE reads ten-byte date strings. Calendar functions use TIMESTAMP(9) values spanning pre-epoch fractions, a leap day and a year boundary. LTRIM/RTRIM use a literal trim set.
  • GREATEST/LEAST measure BIGINT inputs. ENCODE/DECODE measure UTF-8. These cases do not claim performance for other admitted types or charsets. Source fixtures are in ScalarFunctionBenchmark and TextTimeBenchmarkInputs.

Reproduce

Use JDK 17 and run one timing process at a time:

TZ=UTC SF_BENCHMARK=true mvn -pl :streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  -Dscalar.engine=both -Dscalar.functions=SCALAR,SEARCH,ENCODING,TEXT_TIME \
  -Dscalar.rows=2000000 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.bytes=264 -Dscalar.unicode=false -Dscalar.nullEvery=0 \
  -Dscalar.output=target/scalar-ascii264.csv

Repeat in a fresh JVM with scalar.unicode=true and scalar.nullEvery=8, using a different output file. Then run scalar.functions=SEARCH, scalar.bytes=8, scalar.unicode=false and scalar.nullEvery=0 in another fresh JVM. A function name such as STARTSWITH_LITERAL can select a single case, but a different selection has a different JIT history from the full run.

Raw data

The current trials, source-matched identity controls, result medians, workload catalog and checksums are attached to PR #44 as pr44-flink-native-benchmarks.zip. The repository keeps benchmark code and final result tables; generated CSVs are not versioned. Controls are retained for checking the run, and are not subtracted from function times because their result types and lengths can differ.

IFNULL coverage diagnostic (2026-09-16)

Measured against ebe550c6 plus IFNULL support, using the release bench profile with mimalloc, JDK 17, UTC and the same rowwise source/sink methodology above. The selection was IFNULL_STRING,IFNULL_BIGINT,IFNULL_DECIMAL, with 2,000,000 rows, two warmups, five measured trials, a 264-byte ASCII string budget and every eighth input NULL. Source-matched identity controls ran in the same JVM before the functions; both transpose operators and native Calc substitution were checked for every plan.

Case Flink (s) Native (s) Flink / native
STRING identity control 0.755 1.079 0.700x
BIGINT identity control 0.259 0.419 0.618x
DECIMAL(38,9) identity control 0.290 0.639 0.453x
IFNULL(s, 'missing') 0.751 1.109 0.677x
IFNULL(n, CAST(-1 AS BIGINT)) 0.267 0.436 0.612x
IFNULL(n, CAST(0 AS DECIMAL(38,9))) 0.283 0.642 0.441x

These isolated projections are slower natively, including the identity controls. IFNULL adds coverage so that a containing filter/Top-1 island can remain columnar; this measurement does not establish an end-to-end speedup for that larger query. Treat it as a coverage prerequisite, not a standalone scalar acceleration claim. Controls are not subtracted from function times.

STRING to BOOLEAN coverage diagnostic (2026-09-16)

Measured against ebe550c6 plus STRING-to-BOOLEAN support, using JDK 17, UTC and the release bench profile with mimalloc. The source cycles through true, FALSE, t, 0, yes, n, with every eighth value NULL. The selection was STRING_TO_BOOLEAN, with 2,000,000 rows, two warmups and five measured trials. Both transpose operators, native Calc substitution and the source-matched identity control were checked.

Case Flink (s) Native (s) Flink / native
STRING identity control 0.307 0.497 0.618x
CAST(s AS BOOLEAN) 0.316 0.473 0.667x

This isolated conversion is slower natively with rowwise input/output. Its purpose is to remove a cast coverage blocker inside larger native islands, where existing Arrow batches avoid additional boundaries; these measurements do not claim a speedup for that composition. The identity control returns STRING rather than BOOLEAN and is not subtracted from the cast.

INSTR overloads diagnostic (2026-09-16)

Measured against ebe550c6 plus extended INSTR support, with JDK 17, UTC and the release bench profile with mimalloc. The selection was INSTR3_COLUMN,INSTR4_FORWARD,INSTR4_REVERSE, with 2,000,000 rows, two warmups, five measured trials, a 264-byte ASCII padding budget and every eighth source value NULL. Both transpose operators and native Calc substitutions were checked. The existing search fixtures supply runtime needles/starts to the three-argument case and repeated x padding to the forward/reverse third-occurrence cases.

Case Flink (s) Native (s) Flink / native
Runtime needle/start identity control 0.886 1.248 0.710x
Literal search identity control 0.742 1.000 0.742x
INSTR(s, needle, start_pos) 1.428 1.305 1.094x
INSTR(s, 'x', 1, 3) 0.773 0.998 0.774x
INSTR(s, 'x', -1, 3) 3.919 0.953 4.112x

Reverse search benefits from avoiding Flink's reversed-string allocations. The short forward search remains slower with rowwise input/output; these results do not establish a blanket INSTR speedup. Controls return STRING rather than INT and are not subtracted from timings.

Exact DECIMAL coverage diagnostic (2026-09-16)

Measured against ebe550c6 plus exact decimal ROUND/literal/integer-cast support, using JDK 17, UTC and the release bench profile with mimalloc. The selection was DECIMAL_ROUND_POS,DECIMAL_ROUND_NEG,DECIMAL_ROUND_EXPAND,DECIMAL_TO_BIGINT, with 2,000,000 DECIMAL(38,9) source rows, two warmups, five measured trials and every eighth value NULL. Both transpose operators and native Calc substitutions were checked. The source-matched identity control ran in the same JVM before the functions.

Case Flink (s) Native (s) Flink / native
DECIMAL(38,9) identity control 0.272 0.616 0.441x
ROUND(n, 2) 0.296 0.632 0.468x
ROUND(n, -3) 0.319 0.636 0.502x
ROUND(n, 12) 0.269 0.630 0.426x
CAST(n AS BIGINT) 0.311 0.523 0.596x

These isolated row-fed projections are slower natively, including the identity control. They remove expression blockers from larger native islands, including casts above AVG(DECIMAL), but this diagnostic does not establish an end-to-end speedup for those composed queries. Controls are not subtracted from timings. Extreme negative ROUND positions using the JVM upcall are covered by semantic tests rather than these timing cases.

STARTSWITH

STARTSWITH_LITERAL: STARTSWITH(s, 'row:'); STARTSWITH_COLUMN: STARTSWITH(s, needle)

Case Input Flink (s) Native (s) Flink / native
STARTSWITH_LITERAL ASCII, 264-byte budget 0.447 0.766 0.58x
STARTSWITH_COLUMN ASCII, 264-byte budget 0.554 0.938 0.59x
STARTSWITH_LITERAL Unicode, 264-byte budget, NULL/8 0.599 0.796 0.75x
STARTSWITH_COLUMN Unicode, 264-byte budget, NULL/8 0.700 0.981 0.71x
STARTSWITH_LITERAL ASCII, 8-byte padding 0.375 0.556 0.67x
STARTSWITH_COLUMN ASCII, 8-byte padding 0.504 0.773 0.65x

ENDSWITH

ENDSWITH_LITERAL: ENDSWITH(s, ':match'); ENDSWITH_COLUMN: ENDSWITH(s, needle)

Case Input Flink (s) Native (s) Flink / native
ENDSWITH_LITERAL ASCII, 264-byte budget 0.445 0.759 0.59x
ENDSWITH_COLUMN ASCII, 264-byte budget 0.567 0.942 0.60x
ENDSWITH_LITERAL Unicode, 264-byte budget, NULL/8 0.579 0.832 0.70x
ENDSWITH_COLUMN Unicode, 264-byte budget, NULL/8 0.700 1.003 0.70x
ENDSWITH_LITERAL ASCII, 8-byte padding 0.380 0.552 0.69x
ENDSWITH_COLUMN ASCII, 8-byte padding 0.497 0.766 0.65x

INSTR

INSTR_LITERAL: INSTR(s, ':match'); INSTR_COLUMN: INSTR(s, needle)

Case Input Flink (s) Native (s) Flink / native
INSTR_LITERAL ASCII, 264-byte budget 1.461 0.768 1.90x
INSTR_COLUMN ASCII, 264-byte budget 1.622 0.945 1.72x
INSTR_LITERAL Unicode, 264-byte budget, NULL/8 0.935 0.817 1.14x
INSTR_COLUMN Unicode, 264-byte budget, NULL/8 1.107 1.012 1.09x
INSTR_LITERAL ASCII, 8-byte padding 0.425 0.579 0.73x
INSTR_COLUMN ASCII, 8-byte padding 0.556 0.793 0.70x

LOCATE

LOCATE2_LITERAL: LOCATE(':match', s); LOCATE2_COLUMN: LOCATE(needle, s); LOCATE3_LITERAL: LOCATE(':match', s, start_pos); LOCATE3_COLUMN: LOCATE(needle, s, start_pos)

Case Input Flink (s) Native (s) Flink / native
LOCATE2_LITERAL ASCII, 264-byte budget 1.427 0.737 1.94x
LOCATE2_COLUMN ASCII, 264-byte budget 1.520 0.935 1.63x
LOCATE3_LITERAL ASCII, 264-byte budget 1.437 0.854 1.68x
LOCATE3_COLUMN ASCII, 264-byte budget 1.586 1.048 1.51x
LOCATE2_LITERAL Unicode, 264-byte budget, NULL/8 0.963 0.839 1.15x
LOCATE2_COLUMN Unicode, 264-byte budget, NULL/8 1.093 0.990 1.10x
LOCATE3_LITERAL Unicode, 264-byte budget, NULL/8 1.023 1.015 1.01x
LOCATE3_COLUMN Unicode, 264-byte budget, NULL/8 1.151 1.222 0.94x
LOCATE2_LITERAL ASCII, 8-byte padding 0.440 0.607 0.72x
LOCATE2_COLUMN ASCII, 8-byte padding 0.544 0.790 0.69x
LOCATE3_LITERAL ASCII, 8-byte padding 0.453 0.653 0.69x
LOCATE3_COLUMN ASCII, 8-byte padding 0.586 0.860 0.68x

BIN

BIN_TINYINT: BIN(n); BIN_SMALLINT: BIN(n); BIN_INTEGER: BIN(n); BIN_BIGINT: BIN(n)

Case Input Flink (s) Native (s) Flink / native
BIN_TINYINT non-null 0.335 0.472 0.71x
BIN_SMALLINT non-null 0.336 0.468 0.72x
BIN_INTEGER non-null 0.341 0.469 0.73x
BIN_BIGINT non-null 0.331 0.476 0.70x
BIN_TINYINT NULL/8 0.321 0.478 0.67x
BIN_SMALLINT NULL/8 0.324 0.468 0.69x
BIN_INTEGER NULL/8 0.332 0.464 0.71x
BIN_BIGINT NULL/8 0.325 0.468 0.69x

HEX

HEX_STRING: HEX(s); HEX_TINYINT: HEX(n); HEX_SMALLINT: HEX(n); HEX_INTEGER: HEX(n); HEX_BIGINT: HEX(n)

Case Input Flink (s) Native (s) Flink / native
HEX_TINYINT non-null 0.335 0.436 0.77x
HEX_SMALLINT non-null 0.338 0.432 0.78x
HEX_INTEGER non-null 0.344 0.430 0.80x
HEX_BIGINT non-null 0.332 0.440 0.75x
HEX_STRING ASCII, 264-byte budget 1.348 1.057 1.27x
HEX_TINYINT NULL/8 0.321 0.438 0.73x
HEX_SMALLINT NULL/8 0.325 0.434 0.75x
HEX_INTEGER NULL/8 0.328 0.434 0.76x
HEX_BIGINT NULL/8 0.314 0.428 0.73x
HEX_STRING Unicode, 264-byte budget, NULL/8 1.857 1.103 1.68x

TO_BASE64

TO_BASE64: TO_BASE64(s)

Case Input Flink (s) Native (s) Flink / native
TO_BASE64 ASCII, 264-byte budget 0.657 0.963 0.68x
TO_BASE64 Unicode, 264-byte budget, NULL/8 0.840 1.037 0.81x

UNHEX

UNHEX: UNHEX(s)

Case Input Flink (s) Native (s) Flink / native
UNHEX 528 hex bytes, non-null 1.203 1.295 0.93x
UNHEX 528 hex bytes, NULL/8 1.130 1.063 1.06x

GREATEST

GREATEST: GREATEST(n, m, 17)

Case Input Flink (s) Native (s) Flink / native
GREATEST non-null 0.308 0.471 0.65x
GREATEST NULL/8 0.287 0.453 0.63x

LEAST

LEAST: LEAST(n, m, 17)

Case Input Flink (s) Native (s) Flink / native
LEAST non-null 0.294 0.449 0.65x
LEAST NULL/8 0.276 0.453 0.61x

INITCAP

INITCAP: INITCAP(s)

Case Input Flink (s) Native (s) Flink / native
INITCAP ASCII, 264-byte budget 1.347 1.266 1.06x
INITCAP Unicode, 264-byte budget, NULL/8 1.812 1.463 1.24x

TRANSLATE

TRANSLATE: TRANSLATE(s, 'abcdef', 'ABCDEF')

Case Input Flink (s) Native (s) Flink / native
TRANSLATE ASCII, 264-byte budget 2.377 1.350 1.76x
TRANSLATE Unicode, 264-byte budget, NULL/8 3.138 1.240 2.53x

BTRIM

BTRIM: BTRIM(s)

Case Input Flink (s) Native (s) Flink / native
BTRIM ASCII, 264-byte budget 0.566 0.868 0.65x
BTRIM Unicode, 264-byte budget, NULL/8 0.728 0.937 0.78x

ELT

ELT: ELT(i, s, t)

Case Input Flink (s) Native (s) Flink / native
ELT ASCII, 264-byte budget 0.753 1.342 0.56x
ELT Unicode, 264-byte budget, NULL/8 1.112 1.575 0.71x

URL_ENCODE

URL_ENCODE: URL_ENCODE(s)

Case Input Flink (s) Native (s) Flink / native
URL_ENCODE ASCII, 264-byte budget 1.450 1.285 1.13x
URL_ENCODE Unicode, 264-byte budget, NULL/8 4.400 1.448 3.04x

OVERLAY

OVERLAY: OVERLAY(s PLACING t FROM i FOR n)

Case Input Flink (s) Native (s) Flink / native
OVERLAY ASCII, 264-byte budget 0.834 1.197 0.70x
OVERLAY Unicode, 264-byte budget, NULL/8 1.201 1.415 0.85x

URL_DECODE

URL_DECODE: URL_DECODE(s)

Case Input Flink (s) Native (s) Flink / native
URL_DECODE ASCII, 264-byte budget 1.713 1.507 1.14x
URL_DECODE Unicode, 264-byte budget, NULL/8 1.521 1.257 1.21x

ENCODE

ENCODE_UTF8: ENCODE(s, 'UTF-8')

Case Input Flink (s) Native (s) Flink / native
ENCODE_UTF8 ASCII, 264-byte budget 0.454 0.755 0.60x
ENCODE_UTF8 Unicode, 264-byte budget, NULL/8 0.900 0.821 1.10x

DECODE

DECODE_UTF8: DECODE(b, 'UTF-8')

Case Input Flink (s) Native (s) Flink / native
DECODE_UTF8 ASCII, 264-byte budget 0.442 0.657 0.67x
DECODE_UTF8 Unicode, 264-byte budget, NULL/8 0.846 0.735 1.15x

JSON_QUOTE

JSON_QUOTE: JSON_QUOTE(s)

Case Input Flink (s) Native (s) Flink / native
JSON_QUOTE ASCII, 264-byte budget 1.550 1.570 0.99x
JSON_QUOTE Unicode, 264-byte budget, NULL/8 26.048 2.972 8.77x

JSON_UNQUOTE

JSON_UNQUOTE: JSON_UNQUOTE(s)

Case Input Flink (s) Native (s) Flink / native
JSON_UNQUOTE ASCII, 264-byte budget 1.615 1.317 1.23x
JSON_UNQUOTE Unicode, 264-byte budget, NULL/8 1.719 1.312 1.31x

SPLIT

SPLIT: SPLIT(s, '|')

Case Input Flink (s) Native (s) Flink / native
SPLIT ASCII, 264-byte budget 3.241 6.581 0.49x
SPLIT Unicode, 264-byte budget, NULL/8 2.535 4.821 0.53x

SUBSTRING

SUBSTRING_DYNAMIC: SUBSTRING(s, n, len)

Case Input Flink (s) Native (s) Flink / native
SUBSTRING_DYNAMIC ASCII, 264-byte budget 0.760 0.929 0.82x
SUBSTRING_DYNAMIC Unicode, 264-byte budget, NULL/8 0.937 1.005 0.93x

LEFT

LEFT_DYNAMIC: LEFT(s, n)

Case Input Flink (s) Native (s) Flink / native
LEFT_DYNAMIC ASCII, 264-byte budget 0.615 0.880 0.70x
LEFT_DYNAMIC Unicode, 264-byte budget, NULL/8 0.800 0.967 0.83x

RIGHT_DYNAMIC: RIGHT(s, n)

Case Input Flink (s) Native (s) Flink / native
RIGHT_DYNAMIC ASCII, 264-byte budget 0.822 0.864 0.95x
RIGHT_DYNAMIC Unicode, 264-byte budget, NULL/8 0.983 0.912 1.08x

LPAD

LPAD_DYNAMIC: LPAD(s, n, p)

Case Input Flink (s) Native (s) Flink / native
LPAD_DYNAMIC ASCII, 264-byte budget 0.878 1.123 0.78x
LPAD_DYNAMIC Unicode, 264-byte budget, NULL/8 1.199 1.400 0.86x

RPAD

RPAD_DYNAMIC: RPAD(s, n, p)

Case Input Flink (s) Native (s) Flink / native
RPAD_DYNAMIC ASCII, 264-byte budget 0.815 1.173 0.70x
RPAD_DYNAMIC Unicode, 264-byte budget, NULL/8 1.189 1.410 0.84x

SPLIT_INDEX

SPLIT_INDEX_DYNAMIC: SPLIT_INDEX(s, p, n)

Case Input Flink (s) Native (s) Flink / native
SPLIT_INDEX_DYNAMIC ASCII, 264-byte budget 2.099 1.065 1.97x
SPLIT_INDEX_DYNAMIC Unicode, 264-byte budget, NULL/8 2.029 1.121 1.81x

Integer-Delimiter Split Index

The integer overload has a separate released-Flink contract and measurement. The direct SPLIT_INDEX(s, delimiter, idx) query uses a character code in 1..255 rather than the decimal rendering of an integer. Verified UTF-8 inputs and equivalent original Java UTF-16 caches use the bounded native scan. Computed operands and non-equivalent original caches retain released Flink evaluation.

Measured on 2026-10-26 on Apple M4 Pro / 24 GiB, JDK 17, released Flink 2.2.1 and 1.18.1, Rust 1.94.0 release with --no-default-features, against canonical main ffafa92b. Each engine/profile starts a fresh JVM with a 256 MiB initial and 512 MiB maximum heap, eight warmup trials and nine measured trials, processing 131,072 rows per trial. No other build, test or benchmark runs concurrently. Profiles vary batch rows (16, 1,024, 16,384), payload budget (32/264 UTF-8 bytes), ASCII/CJK/emoji, original Java cache state, and fixed/mixed field indices. Strings have cardinality 16; delimiters rotate comma, pipe and U+00E9. Source, delimiter and index are NULL every 7th, 11th and 13th row. Mixed indices include negative/out-of-range fields and invalid delimiter 256.

The portable integer_split release example measures the production compiled Calc, output allocation and checksum consumption with a timer inside Rust. Stock Flink uses its unchanged released generated expression on prepared RowData and consumes the same output bytes with a Java timer. Initial/final JNI entry, C Data import/export and perimeter transposes are outside this small-functionality boundary. The previous production Calc retains its intrinsic JVM callback, Arrow round trip and allocation inside timing. It is a separate binary linked to the unchanged main implementation, using the same driver and Java reference, with the TYPE_STRING descriptor emitted by the baseline planner. The earlier TYPE_INTERNAL control remains a separate diagnostic; only the corrected descriptor is used for final previous-native comparisons. Before timing, the candidate's full output array must equal that reference; its evaluator ID is invalid so a scalar JVM callback fails the check. Cold controls use BinaryRowData/Arrow bytes, while cached controls retain GenericRowData's Java strings and matching Arrow UTF-16 sidecars. Input setup is outside all timers.

The raw measurements retain every measured trial, medians, ranges, baseline library hashes, and earlier adaptation diagnostics. The initial 16-row implementation was slower than stock; removing generic argument adaptation, borrowing array arguments, passing only the source sidecar, reducing ASCII cache comparisons and using a bounded byte search removed that measured regression.

Values are medians with [minimum, maximum] over nine trials:

Flink Corpus / payload Java cache Batch rows Indices Stock ns/row Previous ns/row Native ns/row Stock/native
2.2.1 ascii / 32 cold 1024 1 103.43 [91.64, 108.57] 187.09 [177.73, 193.31] 13.24 [12.89, 14.37] 7.81x
2.2.1 ascii / 264 cold 1024 1 340.09 [329.18, 349.19] 479.08 [471.51, 490.86] 17.78 [17.05, 18.98] 19.13x
2.2.1 ascii / 264 present 1024 1 233.54 [222.75, 236.77] 472.54 [462.24, 486.21] 67.51 [65.10, 71.47] 3.46x
2.2.1 cjk / 264 cold 1024 1 286.61 [280.41, 324.78] 523.59 [513.30, 532.26] 17.64 [17.09, 18.69] 16.25x
2.2.1 cjk / 264 present 1024 1 190.08 [177.75, 192.98] 506.43 [501.88, 527.52] 112.34 [109.08, 117.04] 1.69x
2.2.1 emoji / 264 cold 1024 1 313.08 [303.38, 352.37] 583.21 [576.63, 598.73] 17.64 [17.14, 18.14] 17.75x
2.2.1 emoji / 264 present 1024 1 210.15 [198.55, 225.26] 534.78 [528.17, 554.40] 106.09 [105.58, 108.88] 1.98x
2.2.1 ascii / 32 present 16 1 74.34 [72.21, 84.90] 856.15 [841.49, 866.31] 69.33 [66.46, 79.41] 1.07x
2.2.1 ascii / 264 present 16384 1 245.38 [238.63, 275.09] 457.36 [440.51, 476.26] 67.99 [66.51, 69.83] 3.61x
2.2.1 ascii / 264 present 1024 mixed 194.79 [192.34, 206.07] 416.41 [405.55, 466.88] 61.86 [60.93, 66.82] 3.15x
1.18.1 ascii / 32 cold 1024 1 125.80 [114.01, 130.97] 190.10 [176.04, 233.20] 14.33 [13.18, 15.09] 8.78x
1.18.1 ascii / 264 cold 1024 1 333.38 [319.49, 391.33] 461.74 [454.26, 472.27] 18.03 [17.94, 18.26] 18.49x
1.18.1 ascii / 264 present 1024 1 225.03 [213.01, 236.39] 453.75 [447.83, 471.43] 75.49 [69.36, 78.42] 2.98x
1.18.1 cjk / 264 cold 1024 1 290.07 [280.27, 313.48] 496.14 [489.07, 501.57] 17.68 [17.46, 17.86] 16.41x
1.18.1 cjk / 264 present 1024 1 196.86 [187.04, 198.18] 495.58 [487.61, 505.17] 114.09 [111.68, 114.76] 1.73x
1.18.1 emoji / 264 cold 1024 1 285.90 [284.23, 306.30] 530.87 [526.82, 535.19] 18.45 [18.28, 18.71] 15.50x
1.18.1 emoji / 264 present 1024 1 220.33 [209.49, 250.26] 548.73 [542.43, 554.88] 107.95 [106.14, 109.60] 2.04x
1.18.1 ascii / 32 present 16 1 74.04 [72.83, 85.23] 842.07 [835.36, 884.99] 74.04 [72.34, 75.01] 1.00x
1.18.1 ascii / 264 present 16384 1 251.62 [238.71, 257.85] 449.26 [434.94, 463.80] 71.73 [70.45, 72.34] 3.51x
1.18.1 ascii / 264 present 1024 mixed 183.08 [168.34, 191.90] 405.80 [401.71, 417.17] 64.10 [60.96, 70.52] 2.86x

The 16-row 1.18 control is tied with stock within observed variability (74.04 versus 74.04 ns/row); it does not establish a small-batch speedup. The regular-batch profiles are 1.69-19.13x stock and at least 4.34x the preceding native callback path.

Reproduce the component candidate and stock controls after compiling test classes with the selected released Flink profile:

cargo build --manifest-path native/Cargo.toml --release --no-default-features \
  -p streamfusion --example integer_split
# Set SF_NATIVE_BENCH_CLASSPATH to target/test-classes, target/classes and test dependencies.
export SF_NATIVE_LIB_DIR="$PWD/native/target/release"
# This directory must contain the matching release libstreamfusion DSO for JVM callbacks.
native/target/release/examples/integer_split ascii 264 true 1024 128 false native 8 9
native/target/release/examples/integer_split ascii 264 true 1024 128 false flink 8 9

For the previous implementation, compile the same example against main ffafa92b, using its unchanged production Rust library and matching native DSO, and select baseline rather than native. IntegerSplitBenchmarkFunctions must be on its test classpath. This is a native-baseline ablation, not a Java helper pretending to be the old native implementation.

The separate IntegerSplitPipelineBenchmark diagnostic retains a row source, both production transposes and a blackhole row sink. It processes 2,000,000 rows, parallelism 1, a 264-byte ASCII payload, three delimiters and NULL every 7th row, with 2 GiB heap, two warmups and five measured jobs per engine in fresh JVMs. Plan assertions require both transposes, and runtime metrics require consumed native Calc rows. Its job timer includes query planning/submission and startup after environment setup, so it is not a sustained-throughput measurement. Output equality is established separately by the runtime-source SQL tests and component array comparison; a blackhole sink alone cannot establish equality. Run it with SF_BENCHMARK=true mvn test -Pbench -Dtest=IntegerSplitPipelineBenchmark, selecting -Dsplit.pipeline.engine=stock or native and building each production baseline in its own checkout.

Engine Job median (s) Minimum-maximum (s)
stock 1.828223 1.795710-1.863590
previous 2.460203 2.427904-2.485643
native 0.951705 0.942732-0.966414

The candidate is 1.92x stock and 2.59x the preceding native pipeline in this startup-inclusive fixture. This result applies to this final STRING-output query; it does not establish arbitrary connector or steady-state throughput.

TO_DATE

TO_DATE: TO_DATE(s)

Case Input Flink (s) Native (s) Flink / native
TO_DATE date strings, non-null 0.427 0.529 0.81x
TO_DATE date strings, NULL/8 0.435 0.544 0.80x

QUARTER

QUARTER: QUARTER(ts)

Case Input Flink (s) Native (s) Flink / native
QUARTER TIMESTAMP(9), non-null 0.368 0.434 0.85x
QUARTER TIMESTAMP(9), NULL/8 0.378 0.460 0.82x

WEEK

WEEK: WEEK(ts)

Case Input Flink (s) Native (s) Flink / native
WEEK TIMESTAMP(9), non-null 0.367 0.452 0.81x
WEEK TIMESTAMP(9), NULL/8 0.385 0.472 0.81x

DAYOFYEAR

DAYOFYEAR: DAYOFYEAR(ts)

Case Input Flink (s) Native (s) Flink / native
DAYOFYEAR TIMESTAMP(9), non-null 0.370 0.439 0.84x
DAYOFYEAR TIMESTAMP(9), NULL/8 0.373 0.464 0.80x

DAYOFWEEK

DAYOFWEEK: DAYOFWEEK(ts)

Case Input Flink (s) Native (s) Flink / native
DAYOFWEEK TIMESTAMP(9), non-null 0.364 0.420 0.87x
DAYOFWEEK TIMESTAMP(9), NULL/8 0.382 0.444 0.86x

Runtime SHA2 Widths

SHA2(s, bits) with direct STRING/INT operands uses the same RustCrypto digest and lowercase hex writer as fixed SHA-2 functions. A batch verifies non-NULL widths and reserves exactly the output space for its 224/256/384/512 digests. NULL inputs or widths produce NULL. An invalid runtime width retains released Flink evaluation and the JDK's NoSuchAlgorithmException; computed arguments keep the exact scalar evaluator. No configuration switch is introduced. Literal NULL widths are admitted, while existing valid literal fixed kernels and invalid/oversized literal fallback remain unchanged.

The measurements use canonical main ffafa92b, Rust 1.94.0 release with --no-default-features, Apple M4 Pro / 24 GiB, JDK 17, and released Flink 2.2.1/1.18.1. Each engine/profile starts a fresh JVM (256 MiB initial / 512 MiB maximum heap), with 8 warmup and 9 measured trials of 131,072 rows. Profiles vary ASCII/CJK/emoji payloads, 32/264-byte budgets, 16/1,024/16,384-row batches, and cold/cached Java strings. Logical values, widths and NULLs match: 16 keys, four rotating valid widths, source NULL every 7th row, width NULL every 11th row. Input preparation is outside timing. Engines run sequentially without concurrent builds/tests/benchmarks.

The portable runtime_sha2 example runs the production compiled Calc with a timer inside Rust, on prepared Arrow RecordBatch inputs, including allocation and checksum consumption. Stock uses unchanged released generated Flink on prepared RowData with a Java-side timer. Initial/final JNI and perimeter transposes are excluded from this small-functionality gate; the previous native production binary retains its intrinsic JVM callback, Arrow export/import and allocations. The same example is linked against the unchanged baseline Rust library for that control. Before timing, the candidate's full output array must equal the generated reference; an invalid candidate evaluator ID makes accidental JVM computation fail. Cached controls affect Java's input handling; the native digest deliberately reads bytes, just like Flink's BinaryStringData hash.

Every raw trial, median/range and library hash is retained with the complete-pipeline diagnostic. These profiles establish bounded gains for the tested direct signature, not arbitrary computed inputs or invalid-width batches that retain the host evaluator.

Values are medians with [minimum, maximum] over nine measured trials:

Flink Corpus / payload budget Java cache Batch rows Stock ns/row Previous ns/row Native ns/row Stock/native
2.2.1 ascii / 32 cold 1024 289.83 [281.41, 306.13] 358.60 [353.95, 380.94] 84.68 [80.02, 84.82] 3.42x
2.2.1 ascii / 264 cold 1024 738.92 [723.19, 816.93] 868.92 [860.50, 912.52] 171.39 [161.22, 177.45] 4.31x
2.2.1 ascii / 264 present 1024 736.94 [731.62, 754.16] 894.84 [855.14, 909.22] 163.84 [161.54, 175.59] 4.50x
2.2.1 cjk / 264 cold 1024 733.77 [727.37, 760.30] 1016.30 [1003.85, 1054.98] 159.97 [158.49, 163.85] 4.59x
2.2.1 cjk / 264 present 1024 735.07 [730.10, 752.61] 1004.26 [997.17, 1012.32] 165.00 [162.68, 172.09] 4.45x
2.2.1 emoji / 264 cold 1024 766.42 [755.58, 774.53] 1006.83 [991.28, 1023.74] 169.85 [165.28, 174.57] 4.51x
2.2.1 emoji / 264 present 1024 749.81 [729.67, 776.85] 996.59 [980.35, 1025.20] 168.10 [162.06, 171.72] 4.46x
2.2.1 ascii / 32 present 16 262.83 [253.98, 276.40] 935.46 [921.22, 949.36] 104.86 [102.90, 106.35] 2.51x
2.2.1 ascii / 264 present 16384 783.93 [768.11, 813.64] 850.41 [834.82, 860.69] 178.13 [172.19, 181.99] 4.40x
1.18.1 ascii / 32 cold 1024 285.25 [276.26, 298.58] 357.37 [350.15, 371.73] 87.18 [81.01, 89.61] 3.27x
1.18.1 ascii / 264 cold 1024 728.05 [722.41, 746.24] 881.23 [859.99, 901.85] 172.42 [170.22, 174.59] 4.22x
1.18.1 ascii / 264 present 1024 756.71 [741.79, 772.90] 869.09 [864.30, 911.75] 176.14 [168.87, 182.15] 4.30x
1.18.1 cjk / 264 cold 1024 729.19 [723.14, 756.06] 1006.41 [998.65, 1026.25] 161.39 [158.80, 167.09] 4.52x
1.18.1 cjk / 264 present 1024 748.85 [730.16, 770.20] 1027.83 [1023.35, 1041.85] 172.24 [158.60, 177.12] 4.35x
1.18.1 emoji / 264 cold 1024 770.58 [748.36, 787.40] 1000.87 [984.90, 1032.93] 159.48 [155.76, 163.07] 4.83x
1.18.1 emoji / 264 present 1024 775.41 [742.13, 801.22] 998.94 [984.40, 1026.74] 162.12 [159.17, 163.42] 4.78x
1.18.1 ascii / 32 present 16 272.49 [260.93, 282.38] 954.04 [928.45, 1049.92] 106.83 [104.47, 107.95] 2.55x
1.18.1 ascii / 264 present 16384 824.35 [781.61, 879.61] 867.03 [858.87, 876.18] 173.55 [166.10, 285.19] 4.75x

Across these profiles the candidate runs 2.51-4.83x stock and 4.10-8.93x the preceding native implementation. The raw artifact also retains the earlier borrowed-internal control separately; final comparisons use the actual baseline TYPE_STRING descriptor.

Reproduce after compiling test classes/dependencies for the selected released profile:

cargo build --manifest-path native/Cargo.toml --release --no-default-features \
  -p streamfusion --example runtime_sha2
# Set SF_NATIVE_BENCH_CLASSPATH to test classes, core classes and test dependencies.
export SF_NATIVE_LIB_DIR="$PWD/native/target/release"
# Keep the matching release libstreamfusion DSO here for intrinsic JVM callbacks.
native/target/release/examples/runtime_sha2 ascii 264 true 1024 128 native 8 9
native/target/release/examples/runtime_sha2 ascii 264 true 1024 128 flink 8 9

Compile the same example against main ffafa92b and its unchanged production native library to run the previous behavior, selecting baseline and the matching DSO. The new RuntimeSha2BenchmarkFunctions test helper is required on its test classpath.

RuntimeSha2PipelineBenchmark separately retains a row source, both production RowData/Arrow transposes and a blackhole row sink. Its plan asserts both transposes, and runtime counters require consumed native Calc rows. It uses 2M rows, parallelism 1, a 264-byte ASCII payload, four rotating digest widths, the same NULL frequencies, 2 GiB heap, two warmups and five measured jobs per engine in fresh JVMs. The job timer includes planning/submission and startup after environment creation; it is a startup-inclusive fixture rather than sustained throughput. The component and SQL tests establish value equality separately; the blackhole sink alone is insufficient. Reproduce with SF_BENCHMARK=true mvn test -Pbench -Dtest=RuntimeSha2PipelineBenchmark and -Dsha2.pipeline.engine=stock or native, building each baseline in a separate checkout.

Engine Job median (s) Minimum-maximum (s)
stock 2.301219 2.288133-2.320313
previous 2.483776 2.445620-2.545634
native 1.046509 1.017852-1.065204

The candidate is 2.20x stock and 2.37x previous native in this startup-inclusive fixture. It does not establish arbitrary connector or steady-state throughput.

LTRIM

LTRIM_LITERAL_SET: LTRIM(s, ' |ab')

Case Input Flink (s) Native (s) Flink / native
LTRIM_LITERAL_SET ASCII, 264-byte budget 0.756 1.107 0.68x
LTRIM_LITERAL_SET Unicode, 264-byte budget, NULL/8 0.851 1.072 0.79x

RTRIM

RTRIM_LITERAL_SET: RTRIM(s, ' |ab')

Case Input Flink (s) Native (s) Flink / native
RTRIM_LITERAL_SET ASCII, 264-byte budget 1.923 0.990 1.94x
RTRIM_LITERAL_SET Unicode, 264-byte budget, NULL/8 1.581 1.063 1.49x

SQL/JSON measurements

Measured on 2026-09-10 with the same release profile, JDK 17, Flink/DataFusion versions, 2,000,000 rows, parallelism 1, two warmups, five measured trials, interleaved engines and both transposes described above. Each scenario starts a fresh JVM; no other test or benchmark runs concurrently. Both functions use default admission, including synchronization with Jackson's actual recycled input-buffer capacity; no compatibility flags are enabled.

The input alternates between a document containing user.name and a document without that member. The byte budget controls a separate padding string, excluding JSON syntax and other fields. Unicode input also includes an escaped newline in the selected name; every eighth source value is SQL NULL. Each query evaluates one function, using the literal path lax $.user.name. Identity controls use the same source and are not included in the tables.

The multi-member scenarios add 16 or 64 short string members (field0: value0, and so on) between user and padding. They distinguish repeated key/value parsing from scanning one long string. Both shapes retain full input validation and last-duplicate-member semantics. The native reader selects its protected SIMD path for these multi-member scenarios and the streaming path for the padding-only scenarios. See the parsing technique.

TZ=UTC SF_BENCHMARK=true mvn -pl :streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  -Dscalar.engine=both -Dscalar.functions=JSON_VALUE,JSON_EXISTS \
  -Dscalar.rows=2000000 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.bytes=264 -Dscalar.json.fields=0 \
  -Dscalar.unicode=false -Dscalar.nullEvery=0 \
  -Dscalar.output=target/json-functions.csv

Repeat with 32 and 1024 ASCII padding bytes, and with 264 bytes plus scalar.unicode=true and scalar.nullEvery=8. For multi-member documents, use 32 padding bytes and scalar.json.fields=16 or 64; repeat the 64-member case with Unicode and NULL/8. Raw trial output stays in the local target directory; the tables contain only the final Flink/native medians, with no intermediate optimization results.

JSON_VALUE

JSON_VALUE(s, 'lax $.user.name')

Scenario Flink (s) Native (s) Flink / Native
ASCII, 32-byte padding 1.212 0.831 1.46x
ASCII, 264-byte padding 1.796 1.311 1.37x
ASCII, 1024-byte padding 3.648 2.962 1.23x
Unicode, 264-byte padding, NULL/8 1.439 1.108 1.30x
16 extra members, ASCII, 32-byte padding 2.805 1.786 1.57x
64 extra members, ASCII, 32-byte padding 8.067 4.595 1.76x
64 extra members, Unicode, 32-byte padding, NULL/8 6.429 3.321 1.94x

JSON_EXISTS

JSON_EXISTS(s, 'lax $.user.name')

Scenario Flink (s) Native (s) Flink / Native
ASCII, 32-byte padding 1.182 0.828 1.43x
ASCII, 264-byte padding 1.766 1.287 1.37x
ASCII, 1024-byte padding 3.621 3.211 1.13x
Unicode, 264-byte padding, NULL/8 1.438 1.091 1.32x
16 extra members, ASCII, 32-byte padding 2.773 1.772 1.56x
64 extra members, ASCII, 32-byte padding 8.115 4.576 1.77x
64 extra members, Unicode, 32-byte padding, NULL/8 6.464 3.315 1.95x

TO_BASE64 binary input

Measured on 2026-09-10 with the release profile, 2,000,000 rows, two warmups and five interleaved trials, including both transposes. Run with scalar.functions=TO_BASE64_BINARY. The tt_bytes fixture supplies binary columns containing the ASCII/Unicode payload bytes; encoding is the only measured function.

Scenario Flink (s) Native (s)
ASCII, 32-byte budget 0.357 0.495
ASCII, 264-byte budget 0.499 0.767
Unicode, 264-byte budget, NULL/8 0.485 0.753

ENCODE UTF-16 charsets

Measured on 2026-09-10 with the same release, interleaved 2,000,000-row method, two warmups, five measured trials and both transposes. Each charset runs separately over tt_text; select ENCODE_UTF16, ENCODE_UTF16BE, or ENCODE_UTF16LE.

Function Scenario Flink (s) Native (s)
ENCODE_UTF16 ASCII, 32-byte budget 0.440 0.627
ENCODE_UTF16BE ASCII, 32-byte budget 0.454 0.688
ENCODE_UTF16LE ASCII, 32-byte budget 0.540 0.689
ENCODE_UTF16 ASCII, 264-byte budget 0.998 1.537
ENCODE_UTF16BE ASCII, 264-byte budget 0.997 1.544
ENCODE_UTF16LE ASCII, 264-byte budget 1.599 1.484
ENCODE_UTF16 Unicode, 264-byte budget, NULL/8 1.889 1.242
ENCODE_UTF16BE Unicode, 264-byte budget, NULL/8 1.222 1.222
ENCODE_UTF16LE Unicode, 264-byte budget, NULL/8 2.063 1.227

DECODE UTF-16 charsets

Measured on 2026-09-10 with the release profile, 2,000,000 rows, two warmups, five interleaved trials and both transposes. Run each of DECODE_UTF16, DECODE_UTF16BE, and DECODE_UTF16LE independently. Source fixtures pre-encode text in the matching charset, so only DECODE is measured. The budgets describe the original text in UTF-8; actual binary input uses UTF-16 code units plus a BOM for UTF-16. Non-null benchmark inputs are valid encoded text.

Function Scenario Flink (s) Native (s)
DECODE_UTF16 ASCII, 32-byte text budget 0.423 0.526
DECODE_UTF16BE ASCII, 32-byte text budget 0.441 0.514
DECODE_UTF16LE ASCII, 32-byte text budget 0.557 0.507
DECODE_UTF16 ASCII, 264-byte text budget 1.020 1.342
DECODE_UTF16BE ASCII, 264-byte text budget 1.112 1.322
DECODE_UTF16LE ASCII, 264-byte text budget 1.318 1.389
DECODE_UTF16 Unicode, 264-byte text budget, NULL/8 1.025 0.975
DECODE_UTF16BE Unicode, 264-byte text budget, NULL/8 1.141 0.976
DECODE_UTF16LE Unicode, 264-byte text budget, NULL/8 1.311 1.001

JSON_STRING scalars

Measured on 2026-09-10 with the release profile, 2,000,000 rows, two warmups, five interleaved trials and both transposes. JSON_STRING_TEXT uses tt_text; JSON_STRING_BOOLEAN alternates boolean values; JSON_STRING_INTEGER uses the BIGINT fixture. Each query contains one JSON_STRING call. Byte budgets affect only text, so boolean/integer results list non-null and NULL/8 inputs once each.

Input scenario Flink (s) Native (s)
STRING, ASCII, 32-byte budget 0.605 0.644
STRING, ASCII, 264-byte budget 1.529 1.513
STRING, Unicode, 264-byte budget, NULL/8 1.176 1.262
BOOLEAN, non-null 0.424 0.470
BOOLEAN, NULL/8 0.386 0.464
BIGINT, non-null 0.456 0.487
BIGINT, NULL/8 0.402 0.478

JSON_OBJECT scalar values

Measured on 2026-09-10 with the release profile, 2,000,000 rows, two warmups, five interleaved trials and both transposes. Run JSON_OBJECT_NULL or JSON_OBJECT_ABSENT independently. Each query constructs one object from a text column, a BIGINT row ordinal and an alternating BOOLEAN column, using literal keys text, id and flag. The two functions differ only in NULL ON NULL versus ABSENT ON NULL. In the NULL/8 scenario each column is NULL every eighth row, at staggered positions; the byte budget describes the text column.

Function Scenario Flink (s) Native (s)
JSON_OBJECT_NULL ASCII, 32-byte text budget 1.099 0.863
JSON_OBJECT_ABSENT ASCII, 32-byte text budget 1.102 0.863
JSON_OBJECT_NULL ASCII, 264-byte text budget 2.009 1.742
JSON_OBJECT_ABSENT ASCII, 264-byte text budget 2.036 1.774
JSON_OBJECT_NULL Unicode, 264-byte text budget, NULL/8 1.644 1.484
JSON_OBJECT_ABSENT Unicode, 264-byte text budget, NULL/8 1.598 1.475

TRIM directions and literal sets

Measured on 2026-09-10 using the release profile and the interleaved, 2,000,000-row, two-warmup/five-trial method above, including both transposes. Run with scalar.functions=TRIM_LEADING,TRIM_TRAILING,TRIM_LITERAL_SET and the listed byte budgets. The queries are TRIM(LEADING FROM s), TRIM(TRAILING FROM s), and TRIM(BOTH ' |ab' FROM s). Each query measures one function independently.

Function Scenario Flink (s) Native (s) Flink / native
TRIM_LEADING ASCII, 32-byte budget 0.377 0.543 0.69x
TRIM_LEADING ASCII, 264-byte budget 0.823 1.141 0.72x
TRIM_LEADING Unicode, 264-byte budget, NULL/8 0.710 0.982 0.72x
TRIM_TRAILING ASCII, 32-byte budget 0.377 0.546 0.69x
TRIM_TRAILING ASCII, 264-byte budget 0.817 1.126 0.73x
TRIM_TRAILING Unicode, 264-byte budget, NULL/8 0.680 0.965 0.70x
TRIM_LITERAL_SET ASCII, 32-byte budget 0.845 0.579 1.46x
TRIM_LITERAL_SET ASCII, 264-byte budget 2.911 1.154 2.52x
TRIM_LITERAL_SET Unicode, 264-byte budget, NULL/8 1.605 0.985 1.63x

SHA1

SHA1(s) measured on 2026-09-10 with the same release, interleaved method as TRIM, 2,000,000 rows, two warmups and five trials, including both transposes. Run with scalar.functions=SHA1. The input is the tt_text fixture.

Scenario Flink (s) Native (s) Flink / native
ASCII, 32-byte budget 0.837 0.697 1.20x
ASCII, 264-byte budget 2.229 1.299 1.72x
Unicode, 264-byte budget, NULL/8 1.882 1.151 1.64x

IS JSON predicates

Measured on 2026-09-10 with the same release profile, 2,000,000 rows, two warmups, five interleaved trials and both transposes. Run with scalar.functions=IS_JSON_VALUE,IS_JSON_OBJECT,IS_JSON_ARRAY,IS_JSON_SCALAR. Each predicate runs independently over the same mixture: object, array, string scalar, and malformed object, in equal proportions before SQL NULL injection. Each document contains a padding string with the listed byte budget. Malformed inputs account for 25% of all rows and exercise Flink's exception-handling cost; these results are specific to that mixture. The Unicode case replaces every eighth row with SQL NULL.

Function Scenario Flink (s) Native (s) Flink / native
IS_JSON_VALUE ASCII, 32-byte padding 1.608 0.629 2.56x
IS_JSON_OBJECT ASCII, 32-byte padding 1.600 0.625 2.56x
IS_JSON_ARRAY ASCII, 32-byte padding 1.645 0.625 2.63x
IS_JSON_SCALAR ASCII, 32-byte padding 1.648 0.624 2.64x
IS_JSON_VALUE ASCII, 264-byte padding 2.224 1.156 1.92x
IS_JSON_OBJECT ASCII, 264-byte padding 2.301 1.163 1.98x
IS_JSON_ARRAY ASCII, 264-byte padding 2.284 1.133 2.02x
IS_JSON_SCALAR ASCII, 264-byte padding 2.280 1.136 2.01x
IS_JSON_VALUE Unicode, 264-byte padding, NULL/8 2.158 0.933 2.31x
IS_JSON_OBJECT Unicode, 264-byte padding, NULL/8 2.181 0.968 2.25x
IS_JSON_ARRAY Unicode, 264-byte padding, NULL/8 2.177 0.949 2.30x
IS_JSON_SCALAR Unicode, 264-byte padding, NULL/8 2.201 0.942 2.33x

JSON_VALUE RETURNING

Measured on 2026-09-10 with the release, interleaved method above: 2,000,000 rows, two warmups, five measured trials and both transposes. Run with scalar.functions=JSON_VALUE_BOOLEAN,JSON_VALUE_INTEGER,JSON_VALUE_DOUBLE. Each query projects JSON_VALUE(s, '$.v' RETURNING <type>) independently. Documents contain a selected v member and a separate padding string with the listed byte budget. BOOLEAN alternates true/false, INTEGER alternates 123456789/-234567890, and DOUBLE alternates 1.23456789/-2.3456789e12. Unicode changes the padding; every eighth row is SQL NULL in that scenario. All non-null documents are valid and contain a matching scalar.

RETURNING Scenario Flink (s) Native (s) Flink / native
BOOLEAN ASCII, 32-byte padding 0.862 0.698 1.24x
INTEGER ASCII, 32-byte padding 0.899 0.715 1.26x
DOUBLE ASCII, 32-byte padding 0.959 0.798 1.20x
BOOLEAN ASCII, 264-byte padding 1.472 1.226 1.20x
INTEGER ASCII, 264-byte padding 1.417 1.244 1.14x
DOUBLE ASCII, 264-byte padding 1.496 1.424 1.05x
BOOLEAN Unicode, 264-byte padding, NULL/8 1.331 1.158 1.15x
INTEGER Unicode, 264-byte padding, NULL/8 1.198 1.065 1.12x
DOUBLE Unicode, 264-byte padding, NULL/8 1.296 1.158 1.12x

Temporal coverage diagnostic (2026-09-15)

This run measures the new temporal expression paths on Apple M4 Pro, JDK 17, UTC, Flink 2.2.1 and DataFusion 54.0.0. It uses the release core library with mimalloc, 1,000,000 rows, parallelism 1, one warmup and three measured trials per engine/case. Cases run serially in one JVM with alternating engine order; no other tests or benchmarks run concurrently. Medians include SQL planning and execution. Every plan is checked for NativeCalc, RowDataToArrow, and ArrowToRowData.

That run enabled the former timestamp-range opt-in and used the previous nanosecond layout; it does not measure the current component layout. Inputs are non-null. Parsing uses 2000-02-29 12:34:56 and 1969-12-31 23:59:59; other cases use the existing TIMESTAMP(9) fixtures, including negative fractional epochs. These cases do not measure dynamic formats, time zones, or every temporal overload. The 264-byte payload setting is inherited from the general harness and does not pad these date/time fixtures. Identity controls use the same input sources and are reported without subtraction.

Reproduce in a fresh JVM:

TZ=UTC SF_BENCHMARK=true mvn -pl :streamfusion-runtime test -Pbench \
  -Dnative.cargo.packages='-p streamfusion' \
  '-Dnative.cargo.args=build --release --features mimalloc' \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  -Dscalar.functions=TO_TIMESTAMP,TIMESTAMP_FLOOR,TIMESTAMP_CEIL,TIMESTAMP_ADD,TIMESTAMP_DIFF \
  -Dscalar.rows=1000000 -Dscalar.warmup=1 -Dscalar.runs=3 \
  -Dscalar.output=target/scalar-temporal.csv
Case Flink (s) Native (s) Flink / native
Identity control: timestamp text 0.328 0.549 0.598x
Identity control: TIMESTAMP(9) 0.304 0.440 0.692x
TO_TIMESTAMP(s) 1.965 2.219 0.886x
FLOOR(ts TO MINUTE) 0.298 0.435 0.684x
CEIL(ts TO SECOND) 0.300 0.428 0.701x
TIMESTAMPADD(MONTH, 1, ts) 0.350 0.584 0.600x
TIMESTAMPDIFF(DAY, ts, TIMESTAMP '2024-01-01 00:00:00') 0.302 0.528 0.572x

All five standalone temporal projections are slower than Flink in this diagnostic. Even the identity controls pay more for native execution, so these measurements do not isolate kernel or upcall cost. This is coverage work: temporal expressions can now remain between native operators, and adjacent temporal calls can share one upcall. A longer pipeline needs its own benchmark before claiming a throughput improvement. The current results justify no standalone temporal speedup claim.

Byte characters and JSON extensions (2026-09-15)

These results use the implementation based on ab4a21c6, Apple M4 Pro, JDK 17, UTC, Flink 2.2.1 and DataFusion 54.0.0. Each case/scenario runs in a fresh JVM with the standard Maven bench profile (release + mimalloc), parallelism 1 and 2,000,000 rows. Flink/native run serially and alternate order, with two warmups and five measured trials per engine. Tables contain the final median elapsed seconds, including planning, the rowwise source and sink, and both native transposes. No baseline time is subtracted. Other test suites and native builds are stopped during measurements.

Small differences are subject to run-to-run variance. Standalone regressions remain admitted to preserve verified semantics and native composition; the character changes also correct existing DataFusion/Flink result differences. These are end-to-end results, not isolated kernel timings or comparisons with an earlier native implementation.

Reproduce each named case separately, then repeat with scalar.unicode=true and scalar.nullEvery=8. For integer and decimal inputs, that changes only nullability. Generated trial CSVs remain local build output and are not versioned.

TZ=UTC SF_BENCHMARK=true mvn -pl :streamfusion-runtime test -Pbench \
  -Dnative.cargo.packages='-p streamfusion' \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  -Dscalar.functions=ASCII -Dscalar.engine=both \
  -Dscalar.rows=2000000 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.bytes=264 -Dscalar.unicode=false -Dscalar.nullEvery=0 \
  -Dscalar.output=target/scalar-ascii.csv

ASCII

ASCII: ASCII(s) over 264-byte strings. ASCII inputs begin with a/c; Unicode inputs begin with a Chinese character or accented letter, exercising signed bytes.

Input Flink (s) Native (s)
ASCII, no NULLs 0.753 1.017
Unicode, NULL every eighth row 0.532 0.774

CHR

CHR: CHR(n) over BIGINT values, including negative values and repeated low bytes.

Input Flink (s) Native (s)
BIGINT, no NULLs 0.294 0.458
BIGINT, NULL every eighth row 0.292 0.431

JSON_STRING DECIMAL

JSON_STRING_DECIMAL: JSON_STRING(n) over DECIMAL(38,9), alternating 12345678901234567890.123456700 and -0.000000100 (scientific notation in JSON). The 264-byte string budget does not apply to decimal input.

Input Flink (s) Native (s)
DECIMAL(38,9), no NULLs 0.434 0.640
DECIMAL(38,9), NULL every eighth row 0.415 0.599

JSON_OBJECT DECIMAL

JSON_OBJECT_DECIMAL: JSON_OBJECT('n' VALUE n) with the same two DECIMAL(38,9) values and default NULL ON NULL policy. The 264-byte string budget does not apply.

Input Flink (s) Native (s)
DECIMAL(38,9), no NULLs 0.599 0.655
DECIMAL(38,9), NULL every eighth row 0.591 0.611

Unicode JSON member paths

JSON_VALUE_UNICODE_PATH and JSON_EXISTS_UNICODE_PATH select lax $.用户["姓.名"], each in its own query. Both scenarios use Unicode member names. Documents alternate between a selected string and a missing member, with a 264-byte padding-string budget; the nullable scenario also uses Unicode selected values/padding and NULL every eighth row.

JSON_VALUE:

Input Flink (s) Native (s)
ASCII values, no NULLs 1.718 1.224
Unicode values, NULL every eighth row 1.451 1.107

JSON_EXISTS:

Input Flink (s) Native (s)
ASCII values, no NULLs 1.703 1.226
Unicode values, NULL every eighth row 1.445 1.097

DECIMAL and VARBINARY scalar UDFs

Measured on Apple M4 Pro with JDK 17 and released Flink 2.2.1 on 2026-09-16. Each identity UDF has its own source-matched control. The source produces two million rows at parallelism 1, with NULL every eighth row; decimal input is DECIMAL(38,9) and binary input is 264 bytes. Results are medians of five measured runs after two warmups, alternating engine order. The native release build uses mimalloc, and the harness verifies both row/Arrow transposes before execution.

TZ=UTC SF_BENCHMARK=true mvn -pl streamfusion-runtime -am test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  '-Dscalar.functions=UDF_DECIMAL,UDF_BINARY' \
  -Dscalar.rows=2000000 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.bytes=264 -Dscalar.unicode=false -Dscalar.nullEvery=8 \
  -Dscalar.output=target/udf-exact-types.csv
Expression Flink (s) Native (s) Flink/native ratio
DECIMAL identity control 0.247 0.596 0.414x
Binary identity control 0.282 0.529 0.534x
DECIMAL identity UDF 0.249 0.657 0.379x
Binary identity UDF 0.275 0.601 0.457x

These simple UDFs remain slower than stock Flink. The change extends coverage so a supported exact-type UDF can stay between native operators; it does not claim a standalone speedup. The controls show the conversion cost before adding the JVM callback, and larger native islands require their own measurements before claiming an end-to-end gain.

FROM_UNIXTIME literal formats

ScalarFunctionBenchmark selects FROM_UNIXTIME_DEFAULT, FROM_UNIXTIME_LITERAL and FROM_UNIXTIME_BRIDGE. The first two use the native fixed-zone formatter; the third reads yyyyMMddHHmm from a runtime pattern column and retains the JVM bridge. All use a nullable BIGINT source, modern epoch seconds varying over one day, UTC, parallelism 1 and a rowwise blackhole sink, with both row/Arrow transposes verified in the native plan.

An M4 Pro/JDK 17 release build (-Pbench, mimalloc), 2 million rows, NULL every eighth row, two warmups and five interleaved Flink/native measurements gave:

Query Flink seconds Native seconds Flink/native
Identity control 0.801519 0.900607 0.890x
Default format 1.542533 1.303721 1.183x
Literal yyyyMMddHHmm 1.245461 1.251630 0.995x
Dynamic pattern, JVM bridge 1.204726 2.209705 0.545x

The default native format is faster in this run; the compact literal is roughly tied with Flink. The dynamic control additionally carries a pattern column, so it does not isolate callback cost.

A separate before/after run compared identical literal queries against the previous implementation at a4b9902f, using that revision's release binary and planner with the same benchmark fixtures. Each revision ran with -Dscalar.engine=native, two warmups and five measured trials, keeping all other settings unchanged:

Query Previous JVM bridge seconds Native formatter seconds Before/after
Identity control 0.655586 0.651136 1.007x
Default format 1.696912 1.003518 1.691x
Literal yyyyMMddHHmm 1.543752 0.958994 1.610x

These are sequential native-only runs, separate from the interleaved Flink comparison above. The nearly unchanged identity control supports attributing the improvement to removing the formatting handoff, but the result remains a local end-to-end measurement rather than a kernel-only speed claim.

JSON string consumers

The surrogate-identity fix evaluates a JSON string producer and its scalar consumers together in Flink-generated code within columnar Calc. This preserves correct comparisons before the final Arrow conversion; it is a correctness change, not a faster JSON parsing kernel.

Release build (-Pbench), JDK 17, UTC, parallelism 1, 2m rows, NULL every eighth row, two warmups and five interleaved measurements per engine. The runtime source alternates ASCII JSON texts "\uD800" and "?"; their lengths are fixed at eight and three bytes. The generic payload-budget flag does not pad this fixture. Both source and blackhole sink are rowwise, and the benchmark asserts both transpose operators. No other tests ran during measurement.

Query Flink seconds Native Calc with JVM expression seconds Flink/native
Source-matched identity control 0.511876 1.061221 0.482x
JSON_VALUE(s, '$') = '?' 0.771874 1.351576 0.571x
JSON_UNQUOTE(s) = '?' 0.620493 1.203020 0.516x

The fused path is slower than Flink in this short pipeline. The previous native comparisons returned incorrect answers for the surrogate row, so their timings are not a valid performance baseline for the corrected operation. Direct projections retain the existing Rust kernels. These measurements do not justify a speed claim for a larger columnar pipeline.

TZ=UTC SF_BENCHMARK=true mvn -pl streamfusion-runtime -am test -Pbench \
  -Dtest=ScalarFunctionBenchmark -Dsurefire.failIfNoSpecifiedTests=false \
  -Dscalar.functions=JSON_VALUE_IDENTITY,JSON_UNQUOTE_IDENTITY \
  -Dscalar.rows=2000000 -Dscalar.nullEvery=8 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.engine=both

Negative JSON array indexes

JSON_VALUE_NEGATIVE and JSON_EXISTS_NEGATIVE select $.a[-1] from a 32-element string array. JSON_VALUE_POSITIVE_CONTROL selects the same value with $.a[31]. Documents alternate between final values Alice and Bob, with the usual 264-byte padding-string budget. Negative indexes previously declined the native JSON kernel.

M4 Pro/JDK 17, release build (-Pbench, mimalloc), UTC, parallelism 1, 2m rows, NULL every eighth row, two warmups and five interleaved measurements per engine:

Query Flink seconds Native seconds Flink/native
Source-matched identity control 0.774887 1.496905 0.518x
JSON_VALUE(s, '$.a[-1]') 2.955556 3.951091 0.748x
JSON_VALUE(s, '$.a[31]') 3.147889 3.069012 1.026x
JSON_EXISTS(s, '$.a[-1]') 2.926447 3.831098 0.764x

Both row/Arrow transposes are present and asserted. No other local tests ran during measurement. The negative-index implementation extends native coverage, but is slower than Flink in this short pipeline. The scalar reader counts the array in an extra pass; the SIMD reader uses its existing array length. The positive-index control exposes the additional cost on the same input. These numbers do not establish a speedup for negative indexes or for a larger native pipeline.

TZ=UTC SF_BENCHMARK=true mvn -pl streamfusion-runtime -am test -Pbench \
  -Dtest=ScalarFunctionBenchmark -Dsurefire.failIfNoSpecifiedTests=false \
  -Dscalar.functions=JSON_VALUE_NEGATIVE,JSON_VALUE_POSITIVE_CONTROL,JSON_EXISTS_NEGATIVE \
  -Dscalar.rows=2000000 -Dscalar.nullEvery=8 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.engine=both

JSON_VALUE exception parity (2026-09-19)

JSON_VALUE_ERROR selects lax $.user.active with ERROR ON EMPTY ERROR ON ERROR. JSON_VALUE_ACTIVE selects the same field with the default NULL policies. Both use the unchanged tt_json row source, whose non-null documents contain that boolean field. These are successful-path measurements; the error policies still affect which evaluator runs.

Apple M1 Max, JDK 17, Flink 2.2.1, release build with mimalloc (-Pbench), parallelism 1, one million rows, 264-byte padding budget, NULL every eighth row, two warmups and five interleaved trials per engine. Each before/after run uses a fresh JVM. No other test suite or native build runs during measurement. Medians include planning, the row source and sink, and both row/Arrow transposes, which the harness asserts in the plan.

Query Before Flink (s) Before native (s) After Flink (s) After native (s)
Source-matched identity 0.354 0.608 0.355 0.592
Default NULL policies 0.994 0.812 1.006 0.799
ERROR policies 1.004 0.809 0.988 1.604

The before implementation is 762efc8, with only the two benchmark queries added. Routing ERROR policies through Flink's generated batch evaluator costs 1.98× elapsed time relative to the previous native kernel and takes 1.62× Flink's time in this short pipeline. This corrects the previous exception-class/message mismatch; it is not a performance improvement. Non-throwing policies retain the native path. The generated route performs one JVM callback per Arrow batch, with row iteration inside that callback.

SF_BENCHMARK=true mvn -pl streamfusion-runtime -am test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' \
  -Dscalar.functions=JSON_VALUE_ACTIVE,JSON_VALUE_ERROR \
  -Dscalar.rows=1000000 -Dscalar.bytes=264 -Dscalar.nullEvery=8 \
  -Dscalar.warmup=2 -Dscalar.runs=5 -Dscalar.engine=both

Integer ARRAY_DISTINCT (2026-09-28)

Measured against canonical main 665dc10d, Intel Core i7-12650H, Linux, JDK 17, Flink 2.2.1, release Rust with mimalloc, parallelism one, 2 GiB heap and 1,024-row Arrow batches. Each case uses a fresh JVM, five warmups and five alternating stock/native trials. No other build or test ran concurrently. Times include planning and the complete job, a rowwise DataStream source, both production transposes, JNI execution and a rowwise blackhole sink. The harness asserts NativeCalc and both transposes, completes the job and checks planner substitution counts; an intentionally disabled admission must have zero substitutions.

The query projects ARRAY_DISTINCT(arr) plus a TRUE anchor. Every eighth array is NULL; every seventh element is NULL; other elements are (31 * elementIndex + rowIndex) % domain - domain / 2. Width counts all elements; domain is the value generator's modulus, not the resulting distinct count.

Element type Rows Width / domain Flink median (range), s Native median (range), s Flink/native
BIGINT 200,000 64 / 64 1.047 (1.039–1.065) 0.310 (0.308–0.319) 3.38x
BIGINT 200,000 256 / 256 13.388 (13.345–13.403) 0.924 (0.911–0.942) 14.50x
BIGINT 200,000 64 / 8 0.334 (0.328–0.373) 0.243 (0.238–0.252) 1.37x
INT 200,000 64 / 64 0.989 (0.979–1.005) 0.306 (0.299–0.325) 3.23x
BIGINT 2,000,000 8 / 8 0.668 (0.621–0.672) 0.581 (0.581–0.625) 1.15x
BIGINT 20,000,000 8 / 8 5.875 (5.660–5.943) 5.215 (5.136–5.241) 1.13x
BIGINT, repeat 20,000,000 8 / 8 5.430 (5.394–5.492) 5.413 (5.358–5.468) 1.00x

Large-array gains are clear. Small-array gains are sensitive to run-to-run control movement: the repeat is effectively tied, and neither long run is evidence of a stable 13% improvement. Short jobs also include appreciable fixed startup costs. These results do not establish gains for all integer widths or data distributions.

For the previous StreamFusion behavior, an admission ablation removed only the new ARRAY_DISTINCT Java admission from the same runtime. The harness verified unsupported function/operator: ARRAY_DISTINCT and zero native substitutions. With five warmups/five trials, BIGINT 200,000 x 64/domain64 took 1.049s (1.041–1.124), versus its paired stock control 1.093s (1.031–1.154). BIGINT 2,000,000 x 8/domain8 took 0.580s (0.545–0.607), versus stock 0.564s (0.545–0.572). The candidate's 0.310s large-array result improves this prior behavior by about 70%; its 0.581s small-array result is essentially unchanged across separate JVMs. This is an admission ablation, not a rebuilt old release.

Raw trials retain the full investigation, including unfavorable implementations on baseline 4aae50a1. Released DataFusion's generic kernel took 0.724s versus Flink 0.625s for two million eight-element arrays; typed hash membership reduced native time to 0.659s versus Flink 0.623s. A bounded stack search took 0.624s versus Flink 0.601s with two warmups, and 0.592s versus Flink 0.607s with five. Its twenty-million-row result remained slower (5.393s versus 5.314s). The final implementation adds a collision-checked fingerprint to that stack search. Its first long run improved native time to 5.215s, but the Flink control also moved; the repeat above prevents attributing the entire difference to the kernel. CSV implementation/run columns distinguish these intermediate variants from the shipped typed_fingerprint measurements.

A diagnostic async-profiler CPU-timer run of the generic kernel collected 6,519 mixed Java/native samples; 683 stacks included NativeCalc/JNI, 296 the generic array-distinct kernel, 199 hash insertion, and 36 row conversion. These overlapping counts include both engines and startup. That diagnostic overlapped a native build, so its elapsed times are excluded from benchmark claims. It motivated primitive membership without encoded rows; it does not measure isolated kernel CPU percentages. See the kernel technique.

Reproduce each table scenario by changing rows, width, domain and collection.int (the default is BIGINT):

SF_BENCHMARK=true mvn -pl streamfusion-runtime -am test -Pbench \
  -Dtest=DynamicCollectionBenchmark -Dsurefire.failIfNoSpecifiedTests=false \
  -Dsf.extraJvmArgs=-Xmx2g '-Dcollection.expression=ARRAY_DISTINCT(arr)' \
  -Dcollection.map=false -Dcollection.rows=200000 -Dcollection.width=64 \
  -Dcollection.domain=64 -Dcollection.int=false -Dcollection.warmup=5

For an admission ablation, remove the Java ARRAY_DISTINCT admission and add -Dcollection.native=false and '-Dcollection.fallbackReason=unsupported function/operator: ARRAY_DISTINCT'. Restore admission before measuring the candidate. MAP_KEYS/MAP_VALUES were also investigated but remain fallback after end-to-end regressions; their raw trials and rejection rationale are retained.

Array extrema admission experiment (2026-09-28)

Direct admission of released DataFusion 54 integer ARRAY_MIN/ARRAY_MAX kernels was tested on baseline 2201e98f, Intel Core i7-12650H, Linux/JDK 17, Flink 2.2.1, release/mimalloc, 2 GiB heap, parallelism one and 1,024-row batches. Five warmups preceded five alternating whole-job trials with a row source, row blackhole sink, both transposes and verified native Calc routing.

For two million eight-element BIGINT arrays, MIN took 0.502s native versus 0.480s stock and MAX took 0.496s versus 0.465s. At twenty million rows MIN took 4.549s (4.537–4.572) versus 3.967s (3.932–4.072), a 14.7% regression. Wider arrays showed modest gains: 200,000 arrays of width 256 took 0.497s versus 0.525s for MIN and 0.498s versus 0.532s for MAX. Width-64 BIGINT and INT results were approximately tied or modestly favorable. The default admission was rejected; both functions continue on Flink.

Raw trials include every candidate scenario and previous-admission controls. The experiment note records all ranges, the input generator, reproduction commands, control drift, prototype parity results and the separate profile. No timing from the instrumented profile is included in the raw benchmark series. This investigation does not remove the remaining collection gaps in #234.

DOUBLE TRUNCATE consumer and nullable workloads

ScalarFunctionBenchmark adds DOUBLE_TRUNCATE_CASE, DOUBLE_TRUNCATE_COALESCE and DOUBLE_TRUNCATE_NESTED, all on the existing bounded signed source with runtime scale -3..3. They respectively select CASE WHEN n > 0 THEN TRUNCATE(n,s) ELSE -1E0 END, COALESCE(TRUNCATE(n,s),1E0), and TRUNCATE(TRUNCATE(n,3),s). DOUBLE_TRUNCATE_LARGE uses signed 1e12 + 0.12345 + row % 1024, explicitly outside the helper's magnitude domain, with the same runtime scales. scalar.nullEvery controls operand NULLs; new scalar.scaleNullEvery controls scale NULLs for these DOUBLE source fixtures (default zero leaves scales non-null). The configuration is captured outside the source's per-row loop. Each source has its own matching DOUBLE identity control.

The existing strict native-plan/runtime checks retain JNI, both transposes and the runtime row source/blackhole sink. Six workload smoke checks at 5,003 rows, operand NULL every eight rows and scale NULL every seven rows complete on released Flink 2.2.1. They prove execution/admission, not timing or output parity; consumer SQL parity is independently checked against released Flink. Whole-job comparisons use repeated alternating trials and the same fixture-only copies in the previous-production checkout, with fallback controls explicitly expected. An initial timing attempt overlapped an own native build and was stopped; it supplies no accepted full-run timing result. Retain its log rather than treating partial observations as complete samples.

SF_BENCHMARK=true mvn -Pbench -pl streamfusion-runtime -am test \
  -Dtest=ScalarFunctionBenchmark -Dsurefire.failIfNoSpecifiedTests=false \
  -Dscalar.functions=DOUBLE_TRUNCATE_CASE,DOUBLE_TRUNCATE_COALESCE,DOUBLE_TRUNCATE_NESTED,DOUBLE_TRUNCATE_LARGE \
  -Dscalar.rows=2000000 -Dscalar.warmup=2 -Dscalar.runs=5 \
  -Dscalar.nullEvery=8 -Dscalar.scaleNullEvery=7

Rejected manual BINARY exit projection

A direct Arrow-segment copy prototype preserved owned Flink row bytes and passed 29 targeted wire, lifetime, ownership and SQL parity checks on Flink 2.2.1. The manual per-field projection did not improve release whole-job results over the prior generated projection: ELT medians are stock/manual/prior-native 0.305/0.402/0.387 seconds; string cast medians 0.570/0.615/0.613 seconds. Both native versions still lose to stock. These use two million rows, two warmups, five alternating trials, a 2 GB heap, 1,024-row batches and both transposes. All manual trials and prior-native trials, including identity controls and changing stock timings, are retained. The prototype was removed; production still uses the generated projection. The scoped rejection records the ownership design, validation and remaining investigation. This does not close #235.

Rejected generated BINARY segment projection

A second prototype kept Flink-generated field execution while copying binary payload segments directly into owned Flink rows. It passed five wire/lifetime, fourteen fixed-BINARY SQL and ten existing ownership checks on Flink 2.2.1, but still lost to stock in the release measurements. At two million rows, stock / prototype medians are ELT 0.314 / 0.399 seconds and cast 0.567 / 0.606 seconds for the 264-byte input budget. Widening the string input to 4,096 bytes gives cast 8.496 / 8.682 seconds; the wide identity control is 9.070 / 10.345 seconds. All measurements keep BINARY(16) output, a 2 GB heap, 1,024-row batches, two warmups, five alternating trials, JNI and both transposes. The small-input trials and wide-input trials retain all observations. The prototype was removed; the original generated projection remains. The scoped rejection records validation and architecture. Wider inputs did not prove the hypothesized encoding advantage, and no issue is closed from this experiment.

MAP SQL/JSON batch boundaries (2026-09-19)

The generated Calc bridge also carries MAP values whose keys are declared non-null, and MULTISET values whose elements are declared non-null. This extends coverage through the same borrowed-input/owned-output lifetime; it adds no per-row JNI calls and no native JSON parser.

The existing scalar benchmark measured 2M rows on M1 Max/JDK 17, release + mimalloc, parallelism one, 264-byte ASCII payloads and a NULL outer map every seventh row. Each non-null map contains three string keys and one NULL value. Both transposes and the rowwise blackhole sink remain in the timed path. There were two warmups and five interleaved measured trials; medians below include all conversion and callback work.

Case Flink (s) Native pipeline (s) Flink / native
MAP identity control 1.318 2.379 0.554x
JSON_STRING(m) 2.727 4.179 0.652x
ROW(m, JSON_QUERY(s, '$.items[*]')) 2.611 5.599 0.466x

These isolated projections are slower than Flink. The purpose is to keep supported MAP-bearing JSON Calcs inside a larger native pipeline; this measurement does not establish a speedup for that composition. Existing measured native JSON fast paths remain in use. Nullable keys still fall back because Arrow's map representation cannot preserve them safely.

All measured trials are retained. Reproduce with the existing ScalarFunctionBenchmark#individualFunctions under -Pbench, SF_BENCHMARK=true, -Dscalar.functions=JSON_STRING_MAP,JSON_QUERY_MAP_RESULT -Dscalar.nullEvery=7.

A second benchmark put the same JSON map serialization inside a filter followed by grouped COUNT over 4,096 keys, retaining both transposes and the rowwise blackhole sink. It used the same 2M rows, payload sizes, NULL rate, warmups and interleaving. Flink took 3.365s and the native pipeline 4.774s (0.705x). Thus this measured composition is also slower; there is no demonstrated speedup for the MAP extension. The implementation is small because it reuses the generated host evaluator, but performance needs further work before recommending it as an optimization. Composition trials are retained; reproduce with JsonMapCompositionBenchmark under -Pbench and SF_BENCHMARK=true.

The maintainer accepted shipping the verified MAP/MULTISET boundary extension on October 16 with the measured regressions above retained and performance work deferred. The historical measurements are not a claim of improvement against current main.

MAP scalar callback costs

Measured on October 20, 2026 with released Flink 2.2.1, JDK 17, release native code, parallelism 1, 2 GiB task heap and batches of 1,024 rows. MapScalarUdfBenchmark uses a Java identity callback on nullable MAP<STRING,INT> values and a row blackhole sink. Source maps are constructed during the job. Timings include execution planning/deployment, both perimeter transposes, JNI/C Data, callbacks and the sink; initial environment/explain setup is excluded. These are complete-job cost diagnostics under the accepted UDF exception, not small-functionality admission timings.

Two warmups and five measured trials alternate stock/native-enabled order. Candidate and previous- fallback processes run serially, each with paired stock controls. Native plans require both transposes and actual NativeCalc input counts; previous fallback requires zero native operators.

Rows / entries Candidate's stock control (s) Native callback (s) Previous fallback (s) Previous stock control (s)
16,384 / 4 0.076014 0.163299 0.080867 0.080237
65,536 / 64 0.617199 1.221389 0.614807 0.607610

The callback path is about twice as slow on these fixtures. Correct external MAP coverage now defaults on without a new opt-in; this makes no standalone acceleration claim. The blackhole timings do not validate result values: independent SQL/Table API, mutable/retained Java values, error/lifecycle, separate-JVM and two-restore checks do that on both released Flink lines. The raw evidence retains all trials and ranges, paired controls, library identity, test counts, 48 transport fixtures and allocation probes.

Reproduce with SF_BENCHMARK=true, -Pbench, -Dtest=MapScalarUdfBenchmark, -Dmapbench.rows=16384 -Dmapbench.width=4 or 65536/64, and -Dsf.extraJvmArgs="-Xmx2g -Dstreamfusion.exchange.zeroCopyLocal=false". The old revision uses the same benchmark source with -Dmapbench.expectNative=false.

Final JSON_VALUE and UNION ALL (2026-10-22)

These are complete-job measurements of two filtered JSON_VALUE branches feeding a final UNION ALL and row blackhole sink. The source constructs runtime rows; both entry transposes, the exit transpose, JNI/C Data and Arrow IPC remain in the timed path. Timing starts at executeSql and ends after job completion, including execution planning and deployment. Environment creation and the separate EXPLAIN are outside timing. Each native trial asserts native Calc/Union plans, both perimeter conversion directions, positive NativeCalc input counts and no fallback. Correctness uses separate typed collectors and a sink that captures UTF-16 units before result serialization.

Apple M4 Pro, macOS/aarch64, released Flink 2.2.1, JDK 11.0.27, release/mimalloc, one task manager, parallelism one, 8 GiB maximum JVM heap and 1,024-row transpose batches. There are two warmups and five measured trials with alternating engine order. The metrics harness grants the same 48 GiB off-heap accounting budget to both engines; it is a limit, not allocated or measured resident memory.

The previous behavior is canonical main 47bbc8b080e8cece29540e2f739ddbfdc6bc7111, using its own rebuilt release library and the same benchmark source. It declines the complete query with zero substitutions. Previous and candidate processes run serially with separate stock controls; their differences must not be attributed entirely to the implementation. Medians in seconds:

Input Rows Candidate stock Native Previous stock Previous fallback
Short ASCII 2,000,000 1.042780 0.687971 0.967463 0.965541
Nine-member ASCII 2,000,000 1.537207 1.022508 1.574752 1.572480
Escaped Unicode/empty/NULL 2,000,000 1.214878 0.743581 1.132274 1.133687
Raw Chinese/supplementary/NULL 2,000,000 0.830042 0.652332 0.790804 0.787202
Unique ASCII document per row 2,000,000 1.001590 0.721988 0.982541 0.964158
Short ASCII 20,000,000 9.497192 5.962004 -- --

The five two-million-row fixtures reduce time by 21.4%-38.8% versus their paired stock controls. The larger ASCII job reduces time by 37.2%; previous-main performance was not separately measured at that size. These results establish the tested final row-sink composition, not performance or semantic admission for grouping, joins, sorting, JSON_QUERY or native columnar sinks.

The same five two-million-row fixtures also execute natively on released Flink 1.18.1/JDK 11 with the same resources, warmups, interleaving and release library. Their paired medians are:

Input Flink 1.18 stock (s) Native (s)
Short ASCII 1.050900 0.796418
Nine-member ASCII 1.697440 1.355543
Escaped Unicode/empty/NULL 1.143719 0.681589
Raw Chinese/supplementary/NULL 0.831848 0.646050
Unique ASCII document per row 1.055666 0.825663

The unique fixture constructs a distinct document and extracted value for every row inside the timed job. Its gains therefore do not depend on repeated-document encoding reuse.

Previous-main performance was not separately remeasured on 1.18. The JDK 11 correctness selection passes 85 checks on 2.2 and 74 checks with 11 explicit catalog skips on 1.18. All 23 issue-specific SQL checks execute on both lines, including duplicate/mixed branches, sink-side code-unit capture, four changelog kinds, two checkpoint restores at parallelism one/three and explicit fallback cases. A further JDK 17 selection passes 96 SQL/JNI/Arrow ownership, partial-batch close and failure-cleanup checks on 2.2. The connector-free Rust core passes all 758 library tests. These are selected JVM suites rather than a full JVM-suite run.

The initial input-cache prototype was slower than stock on short and wide ASCII. Profiling identified duplicated UTF-16 documents and entry encoding work. The final transpose omits only redundant caches for parse-only JSON inputs, retaining malformed UTF-16 and differing cached/byte values. A Unicode diagnostic with ASCII-only cache omission also regressed; retaining lazy encoding and omitting valid lazy Unicode caches removes that measured gap. These intermediate results are retained with the raw trials, ranges, native library identities and validation counts.

Reproduce with SF_BENCHMARK=true, -Pbench, -Dtest=JsonStringUnionBenchmark and -Djsonunion.fixture=ascii|wide|unicode|raw-unicode|unique. Select one fixture per invocation; use -Djsonunion.rows=20000000 for the larger job. Previous main uses the identical benchmark source with -Djsonunion.expectNative=false; add -Pflink-1.18 for the older released line. See Union coverage and parse-only entry caches.

Exact numeric ARRAY casts

The functionality gate compares released Flink 2.2.1's generated Calc with the previous StreamFusion fallback at d951a838 and the new Rust ARRAY cast. Measurements use JDK 11, Apple M4 Pro, release Rust with mimalloc, one active benchmark, 131,072 rows, five warmups and nine measured trials. The Arrow input and Flink BinaryRowData contain identical integer or DECIMAL values, NULL containers, empty arrays and NULL elements. Every trial consumes every output element; all three engines have identical checksums for each shape.

This is the repository's small-functionality boundary: stock/previous start on prepared RowData in Java, and native starts on prepared Arrow batches of 1,024 rows in Rust. Setup, input IPC loading, initial/final JNI and perimeter transposes are outside timing. Generated evaluation, native Calc evaluation, allocation and equivalent output consumption are inside. These measurements do not establish an end-to-end job speedup. DECIMAL rescale is DECIMAL(18,2) to DECIMAL(18,1); integer-to-DECIMAL targets DECIMAL(18,2).

Median milliseconds; speedup is stock divided by Rust:

Conversion / array width Stock Previous fallback Rust Speedup
INT to BIGINT / 4 10.003 6.681 0.992 10.08x
INT to BIGINT / 32 17.828 17.958 6.629 2.69x
INT to BIGINT / 128 58.470 60.420 24.851 2.35x
INT to DECIMAL / 4 10.589 13.155 2.322 4.56x
INT to DECIMAL / 32 50.672 37.684 16.713 3.03x
INT to DECIMAL / 128 203.839 186.003 68.721 2.97x
DECIMAL rescale / 4 10.629 10.119 3.426 3.10x
DECIMAL rescale / 32 49.437 45.040 24.884 1.99x
DECIMAL rescale / 128 178.626 152.197 98.925 1.81x

Raw warmup/measured times, ranges and checksums retain the JVM trial variation; short stock trials are noisier than the Rust batch loop. The initial sliced-input prototype repeatedly converted the complete shared child buffer. Restricting conversion to each slice's referenced values removes that amplification, including at nested array levels.

Run the Java half with:

SF_BENCHMARK=true mvn -pl streamfusion-runtime -Pbench \
  -Dtest=ArrayCastAdmissionBenchmark -Darray.cast.mode=stock \
  -Darray.cast.shape=int-bigint -Darray.cast.rows=131072 -Darray.cast.width=32 \
  -Darray.cast.warmups=5 -Darray.cast.runs=9 -Darray.cast.export=/tmp/array-cast.arrow test

Use int-decimal and decimal-rescale for the other shapes. Copy the same benchmark test to a separate checkout of the baseline and use -Darray.cast.mode=previous to require the old planner's whole-Calc ARRAY cast fallback. The new benchmark class is disabled unless SF_BENCHMARK=true. Build and run the native half on the exported input:

cd native
cargo build -p streamfusion --release --no-default-features --features mimalloc --example array_cast
target/release/examples/array_cast /tmp/array-cast.arrow 5 9

The native example uses the production Calc decoder and cast, times inside Rust and verifies the same output digest. Ordinary SQL regressions separately establish released-Flink result and type parity and positive consumed native work on both supported Flink lines. Numeric ARRAY TRY_CAST invokes the identical total conversion kernel; these measurements use CAST SQL and do not claim a separate TRY_CAST measurement.

Standalone collection programs

Standalone ARRAY/MAP/ROW casts and dynamic or empty-separator SPLIT can now select the existing generated collection evaluator without another collection function in the Calc. This is coverage under the accepted collection-extension tradeoff; these new standalone forms are slower than stock Flink and previous-main fallback. Existing numeric ARRAY kernels remain unchanged.

Measured on 2026-10-23 with Linux x86-64/WSL2, Intel Core i7-12650H, OpenJDK 17.0.20.1, Flink 2.2.1 and release/mimalloc. Each query uses 500,000 runtime-source rows, parallelism 1, an 8 GiB test heap, two warmups and three alternating measurements per engine. Source, blackhole sink, both row/Arrow transposes, planning and submission remain timed. Initial explain and cluster setup are outside timing. No other tests or benchmarks ran concurrently. Previous main is 1bd9daa8, in a separate JVM using the same native DSO and benchmark source; each new expression must have zero native Calc substitutions there. Identity controls still require native Calc.

DECIMAL arrays contain two DECIMAL(38,9) values and a NULL element. Every eighth container is NULL. SPLIT uses a 32-byte Unicode payload budget and every eighth input is NULL; the dynamic separator is LEFT(s,1). Empty separators produce UTF-16-unit elements, including isolated surrogate halves, whose Java caches must survive the represented Arrow result.

Median seconds; ranges and both paired stock controls are retained in the trial/profile summary.

Expression Stock Generated collection Previous fallback Stock / generated
DECIMAL ARRAY to STRING ARRAY 0.397833 0.681770 0.400672 0.584x
SPLIT with dynamic separator 0.304137 0.575999 0.294347 0.528x
SPLIT with empty separator 0.661662 1.967336 0.650776 0.336x

These complete jobs include startup; the subsecond results do not establish sustained throughput. The initial DECIMAL-array-to-FLOAT diagnostic was already native on previous main and is retained as a diagnostic, not evidence of newly admitted coverage or an improvement from this change.

Optimization investigation was limited to two eight-second CPU captures, one per SPLIT form. The empty-separator capture has 10,331 samples across all JVM threads, including 2,309 inclusive samples in Flink's array copy and 3,666 in represented array writing; the dynamic capture has 13,333 total samples, with 943 and 1,286 in those paths. Counts overlap, include compilation and planning, and do not measure allocations or copied bytes. These paths preserve released array copy categories and UTF-16 caches; no shortcut was enabled without a verified representation contract. Further standalone optimization remains in #392.

Reproduce with a released native build:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' -Dsf.testForks=1 \
  -Dscalar.functions=DECIMAL_ARRAY_TO_STRING,SPLIT_DYNAMIC,SPLIT_EMPTY \
  -Dscalar.rows=500000 -Dscalar.bytes=32 -Dscalar.unicode=true -Dscalar.nullEvery=8 \
  -Dscalar.warmup=2 -Dscalar.runs=3

The previous checkout copies only the same benchmark source and adds -Dscalar.native.expected=false. SPLIT is absent from released Flink 1.18.1; standalone cast parity is checked on both supported lines, with NativeCalc consumption required for successful runtime queries and ordinary fallback retained at unverified collection boundaries.

These timings and CPU profiles precede the JVM bridge/type extension in #469. The rebased change passes 250 focused checks on Flink 2.2.1 and 213 passes plus 37 host-function skips on 1.18.1, including a temporary numeric UDF named SPLIT. The historical timings were not repeated after rebasing.

Collection character casts

Standalone ARRAY/MAP CAST and TRY_CAST to STRING, VARCHAR and CHAR now use the existing generated collection evaluator; ROW-to-character casts already used whole-Calc JVM evaluation. This is coverage under the accepted collection-extension tradeoff. These standalone jobs are slower than stock Flink and previous-main fallback; no acceleration is claimed.

The 2026-10-23 release/mimalloc comparison uses Flink 2.2.1, OpenJDK 17.0.20.1 and Linux x86-64/WSL2 on an Intel Core i7-12650H. Each engine processes 500,000 runtime-source rows, parallelism 1, two warmups and three alternating trials with an 8 GiB test heap. Complete-job timing retains planning, submission, source, rowwise blackhole sink and both row/Arrow transposes; cluster setup and initial explain are outside timing. No other tests or benchmarks run concurrently. Previous main 8eb2b004 uses the identical benchmark source and released native DSO in a separate JVM, with fallback required for the new expressions. Source-matched identity controls still require native Calc. Subsecond complete-job results include startup and do not establish sustained throughput.

DECIMAL arrays have three elements: two DECIMAL(38,9) values and a NULL. Text arrays have a 32-byte-budget Unicode string, NULL and tail; maps have three ordered STRING entries, including a NULL value. Every eighth container is NULL. Trial values, variability, both paired stock controls, identity controls and CPU counts are retained in the measurement summary.

Median seconds (minimum–maximum):

Expression Stock Flink Generated collection Previous fallback
DECIMAL ARRAY → STRING 0.406602 (0.397808–0.410599) 0.547139 (0.546228–0.568279) 0.401366 (0.395287–0.406225)
STRING ARRAY → CHAR(32), TRY_CAST 0.355477 (0.352671–0.357062) 0.577292 (0.573435–0.592959) 0.355399 (0.338167–0.361236)
STRING MAP → STRING 0.561181 (0.553266–0.564460) 0.899699 (0.887600–0.900220) 0.559865 (0.557814–0.569516)

Optimization was limited to two eight-second CPU captures, one each for ARRAY and MAP formatting. The ARRAY capture has 15,151 samples across all JVM threads, with 3,660 inclusive samples in the JVM callback and 2,521 in generated row evaluation. The MAP capture has 13,381 total samples, with 3,479 in the callback, 2,278 in generated evaluation and 1,247 in represented-string writing. Counts overlap and include compilation/planning; they do not measure allocation or copied bytes. Formatting, materializing strings and preserving Java caches remain visible costs. No obvious safe shortcut removed those costs within this increment; further work stays in #392.

Reproduce with a released native build:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' -Dsf.testForks=1 \
  -Dscalar.functions=ARRAY_FORMAT_STRING,TEXT_ARRAY_FORMAT_CHAR,MAP_FORMAT_STRING \
  -Dscalar.rows=500000 -Dscalar.bytes=32 -Dscalar.unicode=true -Dscalar.nullEvery=8 \
  -Dscalar.warmup=2 -Dscalar.runs=3

The previous checkout copies the same benchmark source and adds -Dscalar.native.expected=false. For each of ARRAY_FORMAT_STRING and MAP_FORMAT_STRING, select #profileFunction, set SF_PROFILE_SCALAR=true, -Dscalar.warmup=1 -Dscalar.seconds=8 and -Dprofile.asprof=/path/to/asprof -Dprofile.outputDir=/path/to/output. Runtime parity on both supported Flink lines independently requires consumed NativeCalc input, including bounded UTF-16 results, legacy formatting and all four changelog kinds. NOT NULL fixed BINARY MAP leaves remain guarded on 1.18; the 2.2 admission has separate measurements. Unverified state/native-writer boundaries retain fallback. Fixed-BINARY MULTISET elements use the separate verified boundary.

Fixed BINARY map values

Nullable BINARY(n) map values now enter the existing generated collection evaluator. A direct final MAP_VALUES of an input-reference map with VARCHAR keys reuses the Rust shared-child view; other consumers retain Flink evaluation. NOT NULL fixed BINARY keys and values remain guarded on 1.18; the 2.2 admission has separate measurements. Nullable fixed BINARY keys are covered below. NOT NULL values have an observed NULL-formatting difference on released Flink 1.18.

Release/mimalloc measurements on 2026-10-24 use Flink 2.2.1, OpenJDK 17.0.20.1 and Linux x86-64/WSL2 on an Intel Core i7-12650H. Each job has 500,000 runtime-source rows, parallelism 1, two warmups and three alternating trials per engine, with an 8 GiB test heap. The three-entry map has fixed ASCII STRING keys and two BINARY values of the selected width plus a NULL value; every eighth parent is NULL. Timing includes planning, submission, processing, both row/Arrow transposes and the blackhole row sink; cluster setup and initial explain are outside timing. No other tests, benchmarks or native compilation run concurrently with these measurements.

Previous main 01b24b62 uses identical benchmark code and the same native DSO in a separate JVM. The new expressions must fall back there; source-matched identity controls still require NativeCalc. The initial generated route precedes the bounded optimization; the final route uses Rust for the direct MAP_VALUES projection. MAP-to-STRING continues to use generated Flink evaluation. All trial values, both paired stock controls, identity controls and CPU/allocation diagnostics are in the measurement summary.

Median seconds (minimum–maximum), retaining complete-job startup costs:

Expression / bytes per value Paired stock, final run Previous fallback Initial generated route Final native route
MAP_VALUES / 4 0.312171 (0.310226–0.327523) 0.291862 (0.289075–0.299226) 0.581776 (0.561526–0.597252) 0.366429 (0.352071–0.367083)
MAP → STRING / 4 0.462423 (0.450366–0.471487) 0.438994 (0.437611–0.451575) 0.737274 (0.728585–0.741667) 0.756175 (0.752077–0.835100)
MAP_VALUES / 256 0.400533 (0.377027–0.426528) 0.387280 (0.383559–0.407149) 0.685394 (0.681593–0.699388) 0.448989 (0.446273–0.480963)
MAP → STRING / 256 1.167681 (1.161664–1.181050) 1.173208 (1.138053–1.187564) 1.651487 (1.616399–1.669336) 1.763396 (1.744110–1.808813)

The Rust view reduces MAP_VALUES native medians by 34.5–37.0% versus the generated route. Every final standalone case remains slower than stock Flink and previous fallback, under the existing accepted collection-extension tradeoff. These subsecond/second complete jobs do not establish sustained throughput, and this is not a standalone acceleration claim.

Optimization stopped after two passes. An eight-second CPU capture of generated MAP_VALUES has 12,125 samples, with 2,297 inclusive samples in the JVM callback and 1,122 in generated evaluation. The bounded change selects the existing Rust child view and avoids that callback for the verified projection. The second eight-second capture, for MAP-to-STRING at 256 bytes, has 12,044 samples, including 4,961 in the callback, 2,686 in generated evaluation and 2,056 in represented-string writing. Inclusive counts overlap and include compilation/planning; they do not measure allocations or copied bytes. Formatting, byte/string materialization and both perimeter conversions remain costs for #392.

Six release Criterion smokes independently assert shared child identity, offsets and parent validity on sliced maps at 16/1024 rows and 0/4/256-byte payloads with NULL/empty parents and NULL values. The untimed System-allocation probe makes 104 allocation requests for 6,481 bytes per case and reports zero newly owned Arrow output-buffer bytes. This includes first schema binding and is not a steady-state allocation or copied-byte count. The complete output buffers share input storage; the SQL benchmarks above still retain all perimeter conversion costs.

Reproduce each width with a released native build:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' -Dsf.testForks=1 \
  -Dscalar.functions=FIXED_BINARY_MAP_VALUES,FIXED_BINARY_MAP_STRING \
  -Dscalar.rows=500000 -Dscalar.binary.width=256 -Dscalar.nullEvery=8 \
  -Dscalar.warmup=2 -Dscalar.runs=3

Also run with -Dscalar.binary.width=4. The previous checkout copies the same benchmark and input fixture sources and adds -Dscalar.native.expected=false. For CPU captures, select #profileFunction, set SF_PROFILE_SCALAR=true, choose one function, and use -Dscalar.warmup=1 -Dscalar.seconds=8 -Dprofile.asprof=/path/to/asprof. Rust fixtures run with cargo bench -p streamfusion --bench collection_expressions --features mimalloc --locked -j2 -- binary_map_values --test from native/.

Fixed BINARY map keys

Nullable BINARY(n) map keys use the existing length-delimited generated collection bridge. A direct final MAP_KEYS of an input-reference map with nullable basic values reuses the Rust shared-child view; other consumers retain generated Flink evaluation. NOT NULL fixed BINARY keys and values remain guarded on 1.18; see the 2.2 measurements. External STRUCTURED inputs and unverified state/native-writer boundaries remain guarded. Released Flink 1.18 cannot initialize CASE returning BINARY arrays; tests retain its exact compiler failure and planning fallback for that form.

Release/mimalloc measurements on 2026-10-24 use Flink 2.2.1, OpenJDK 17.0.20.1 and Linux x86-64/WSL2 on an Intel Core i7-12650H. Each job has 500,000 runtime-source rows, parallelism 1, an 8 GiB test heap, two warmups and three alternating trials per engine. The ordered internal RowData source supplies three-entry maps: two binary keys of width4/256 and a NULL key, with INT values 1/NULL/-1. Every eighth parent is NULL. Public InternalTypeInfo registration avoids an external byte-array-key conversion that could reorder entries by object identity. Timing includes planning, submission, both row/Arrow transposes and a blackhole row sink; cluster setup and initial explain are outside timing. No other tests, benchmarks or native compilation run concurrently with these measurements.

Previous main dbb9cffd uses identical benchmark/input code and the same release native DSO in a separate JVM. New expressions must fall back there, while source-matched identity controls still require NativeCalc. The initial generated route precedes the bounded optimization; MAP-to-STRING remains generated throughout. All trials and diagnostics retain both stock controls, identity controls, allocation probes and CPU profile summaries.

Median seconds (minimum–maximum), retaining complete-job startup costs:

Expression / bytes per key Paired stock, final run Previous fallback Initial generated route Final native route
MAP_KEYS / 4 0.243699 (0.234586–0.247416) 0.224914 (0.224026–0.225689) 0.383493 (0.378825–0.386779) 0.253453 (0.252776–0.257386)
MAP → STRING / 4 0.340207 (0.336850–0.350045) 0.339059 (0.338770–0.354235) 0.487764 (0.482822–0.492686) 0.482662 (0.478601–0.498011)
MAP_KEYS / 256 0.274349 (0.263257–0.276442) 0.275038 (0.265149–0.280884) 0.490600 (0.485984–0.510844) 0.304727 (0.298132–0.439454)
MAP → STRING / 256 0.982406 (0.978934–0.987015) 0.968631 (0.958710–0.969560) 1.081767 (1.076390–1.102349) 1.070686 (1.040013–1.078272)

The Rust view reduces MAP_KEYS native medians by 33.9–37.9% versus generated evaluation. All final standalone cases remain slower than paired stock and previous fallback under the accepted collection-extension tradeoff. The wide-key native range also contains a slower trial. These short complete jobs do not establish sustained throughput or standalone acceleration.

Optimization stopped after two passes. The first eight-second CPU capture of generated MAP_KEYS has 14,036 samples, including 2,494 inclusive callback samples, 1,038 in generated evaluation and 822 in array writing. Reusing the existing Rust view avoids that callback for the verified projection. The second eight-second capture, for MAP-to-STRING at 256 bytes, has 11,930 samples, including 4,263 in the callback, 2,907 in generated evaluation and 898 in represented-string writing. Inclusive counts overlap and include planning and compilation; they do not measure allocations or copied bytes. Formatting, map serialization and both perimeter conversions remain costs for #392.

Six release Criterion smokes assert shared key-child identity, offsets and parent validity on sliced maps at 16/1024 rows and 0/4/256-byte keys with duplicate key contents, NULL values and NULL/empty parents. The untimed System-allocation probe reports 104 allocation requests for 6,475 bytes per case and zero newly owned Arrow output-buffer bytes, including first schema binding. Buffer sharing and allocation requests do not measure copied bytes or end-to-end speedup.

Reproduce each width with a released native build:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' -Dsf.testForks=1 \
  -Dscalar.functions=FIXED_BINARY_MAP_KEYS,BINARY_KEY_MAP_STRING \
  -Dscalar.rows=500000 -Dscalar.binary.width=256 -Dscalar.nullEvery=8 \
  -Dscalar.warmup=2 -Dscalar.runs=3

Also run with -Dscalar.binary.width=4. The previous checkout copies the same benchmark/input sources and adds -Dscalar.native.expected=false. For the two CPU captures, select #profileFunction, set SF_PROFILE_SCALAR=true, choose one function, and use -Dscalar.warmup=1 -Dscalar.seconds=8 -Dprofile.asprof=/path/to/asprof. From native/, run cargo bench -p streamfusion --bench collection_expressions --features mimalloc --locked -j2 -- binary_map_keys --test for the Rust fixtures.

MULTISET casts

Standalone MULTISET-to-MULTISET and MULTISET-to-STRING/VARCHAR/CHAR CAST and TRY_CAST use the existing generated collection evaluator and map-shaped Arrow transport. Nullable elements, counts, nested ROW/ARRAY elements, DECIMAL conversion, legacy formatting and whole-container TRY_CAST failures retain released Flink semantics. Direct fixed BINARY elements, external STRUCTURED inputs and unverified representation boundaries retain fallback. COLLECT and ARRAY_AGG are separate remaining coverage; updating retained MAP/MULTISET payload work in #437 stays deferred.

Release/mimalloc measurements on 2026-10-24 use Flink 2.2.1, OpenJDK 17.0.20.1 and Linux x86-64/WSL2 on an Intel Core i7-12650H. Each complete job has 250,000 runtime-source rows, parallelism 1, an 8 GiB test heap, two warmups and three alternating trials per engine. Inputs have 4/32 entries: INT keys i*257 with counts i%3+1, followed by a NULL key with count 3. Every eighth parent is NULL. An ordered internal GenericMapData source shares the prepared, immutable key/count map across its rows and registers it using public InternalTypeInfo.

Timing includes planning, submission, source, both row/Arrow transposes and a blackhole row sink; cluster setup and initial explain are outside timing. Measurements run without concurrent tests, native compilation or other benchmarks. The previous-code checkout is the PR #476 tree measured at d48fba98, with identical benchmark/input code and the same native DSO. Both new expressions must fall back there; source-matched identity controls still require a native plan. The full trial summary retains both stock controls, identity controls and CPU diagnostics. CSV multiset_entries records the selected cardinality; text-fixture defaults such as payload_bytes=264 are unused for this input.

Median seconds (minimum–maximum), retaining complete-job startup costs:

Expression / entries Paired stock Previous fallback Final native route
INT → BIGINT elements / 4 0.185376 (0.182766–0.187421) 0.191836 (0.188395–0.199897) 0.253451 (0.246239–0.261849)
MULTISET → STRING / 4 0.226101 (0.219468–0.252986) 0.224224 (0.216410–0.229614) 0.287859 (0.286316–0.290994)
INT → BIGINT elements / 32 0.466856 (0.462920–0.470641) 0.456071 (0.454004–0.466342) 0.756280 (0.752158–0.756718)
MULTISET → STRING / 32 0.491673 (0.479175–0.542994) 0.493186 (0.480225–0.510621) 0.718291 (0.713210–0.727207)

All final native cases are slower than stock and previous fallback under the accepted collection tradeoff. These short complete jobs do not establish sustained throughput or standalone acceleration. No new performance-only opt-in is added.

Optimization stops after two profiling/optimization passes. The eight-second numeric-cast CPU capture has 12,872 samples, including 3,793 inclusive callback samples, 2,209 in generated evaluation, 2,812 in map serialization and 2,277 in Arrow map writing. The second eight-second capture, for formatting, has 15,115 samples, including 4,007 in the callback, 3,303 in generated evaluation, 1,907 in map serialization and 908 in Arrow map writing. Inclusive counts overlap and include planning and compilation; they do not measure allocation counts or copied bytes. No safe small shortcut was identified that preserves released cast/copy semantics. Further serialization, formatting and conversion work stays in #392. No production Rust kernel or JNI ownership protocol changes in this increment.

Reproduce both cardinalities with a release native build:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' -Dsf.testForks=1 \
  -Dscalar.functions=MULTISET_CAST,MULTISET_FORMAT_STRING \
  -Dscalar.rows=250000 -Dscalar.multiset.entries=32 -Dscalar.nullEvery=8 \
  -Dscalar.warmup=2 -Dscalar.runs=3

Also run with -Dscalar.multiset.entries=4. The previous checkout copies the same benchmark and input sources and adds -Dscalar.native.expected=false. For each of the two CPU captures, select #profileFunction, set SF_PROFILE_SCALAR=true, choose one function, and use -Dscalar.warmup=1 -Dscalar.seconds=8 -Dprofile.asprof=/path/to/asprof.

Fixed BINARY MULTISET elements

The verified generated boundary admits nullable fixed-BINARY elements on both released lines and NOT NULL elements on 2.2. Flink 1.18 NOT NULL elements retain the observed unchecked-NULL getter/formatting fallback. Fixed-binary MULTISET CASE retains released implicit branch casts, including 1.18 padding. No state, ownership or Rust operation changes. The local matrix passes 235 checks on 2.2.1 and 191 with 44 version skips on 1.18.1. This combines the broad eight-class matrix and final 24-case multiset reruns after CASE routing and deterministic-control corrections; the other 211 checks are unchanged. Benchmark smokes are excluded from correctness totals.

Release/mimalloc measurements on the Intel Core i7-12650H Linux x86-64/WSL2 host use Flink 2.2.1, OpenJDK 17.0.20.1, parallelism one and an 8 GiB heap. Runtime internal RowData contains an ordered four-entry multiset with 4-byte keys at 500,000 rows, then 16 entries with 256-byte keys at 200,000 rows. Each has one NULL key with count 3; other key counts cycle through 1/2/3. Every eighth parent is NULL. CAST changes BINARY(width) elements to VARBINARY(width); the second query formats the original multiset as STRING.

Timing includes planning, submission, runtime source, key conversions, output, both row/Arrow transposes and the blackhole row sink. Fresh per-job metrics-cluster/schema setup and plan assertions are outside timing. Each admitted job requires positive consumed Calc and both transpose counters; stock/prior-fallback jobs require no native operators. Two warmups precede three alternating trials per engine in each separate cohort JVM. Prior candidate 7391128b uses identical benchmark/input sources and the same frozen release DSO. Source-matched identity controls remain native on both checkouts and are retained without subtraction. Their variability and all independent prior stock controls remain in the full evidence.

A two-second sampler observed no overlapping external Java/Cargo workloads. Two matrix observations and one profile observation are own Maven launch shells, identified by their unique CSV/profile-output arguments within 0.1 seconds of phase launch before JVM replacement. The committed summary retains those classifications; raw local reports retain every observation.

Median seconds (minimum–maximum):

Expression Bytes per key / entries Paired stock Previous fallback Native route
CAST 4/4 0.211763 (0.204312–0.309692) 0.205085 (0.200423–0.211794) 0.422545 (0.412186–0.430073)
STRING 4/4 0.337583 (0.331511–0.347192) 0.357177 (0.339805–0.377967) 0.490561 (0.481319–0.500198)
CAST 256/16 0.382414 (0.379815–0.387833) 0.379241 (0.377375–0.386449) 1.460521 (1.444518–1.474352)
STRING 256/16 2.529796 (2.512200–2.538575) 2.769301 (2.761990–2.785375) 4.928001 (4.909214–5.198685)

All four native medians exceed both paired stock and prior fallback. These complete-job costs ship under the accepted collection tradeoff; they establish no acceleration claim. The shorter jobs retain startup and trial variation rather than implying sustained throughput.

The bounded investigation closes after two eight-second target CPU captures, after one unprofiled native job each. Both use 200,000 rows, 256-byte keys and 16 entries. CAST has 11,711 samples: 2,929 inclusive in the callback, 822 in generated evaluation, 3,957 in Arrow map writing and 2,088 in map serialization. STRING has 13,087 samples: 7,623 in the callback, 2,563 in evaluation,895 in map writing and 4,830 in represented-string writing. UTF16-to-UTF8 encoding contributes 1,595 self samples, with 527 in UTF8-to-UTF16 decoding. Counts overlap and include fresh-cluster setup, planning, compilation and all JVM threads. They are not allocation or copied-byte counts, nor a causal isolation of the entire slowdown against stock. Map serialization/writing, callback conversions and represented string encoding remain costs. No safe small shortcut preserving released key conversion and representation was found in the two passes; further optimization belongs to #392.

Reproduce the small shape after a release build:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' -Dsf.testForks=1 \
  -Dscalar.functions=BINARY_MULTISET_CAST,BINARY_MULTISET_STRING \
  -Dscalar.rows=500000 -Dscalar.binary.width=4 -Dscalar.multiset.entries=4 \
  -Dscalar.nullEvery=8 -Dscalar.warmup=2 -Dscalar.runs=3

Also run rows 200000, width 256, entries 16. The prior checkout copies the same benchmark/input sources and adds -Dscalar.native.expected=false. For each of the two profiles, select #profileFunction, set SF_BENCHMARK=true SF_PROFILE_SCALAR=true, choose one function, and retain the wide shape with -Dscalar.warmup=1 -Dscalar.seconds=8 plus an async-profiler path through profile.asprof. Profiled timings do not enter the comparison table.

NOT NULL fixed BINARY map leaves

On Flink 2.2, fixed BINARY(n) NOT NULL map keys and values now enter the existing generated collection evaluator. The Rust MAP_KEYS/MAP_VALUES child-view whitelist is unchanged: these new NOT NULL-element projections retain generated evaluation. Flink 1.18 keeps its fallback because unchecked NULL getters can change released map formatting. No native operation, transport schema, state serializer or ownership protocol changes.

Release/mimalloc measurements on 2026-10-24 use Flink 2.2.1, OpenJDK 17.0.20.1 and Linux x86-64/WSL2 on an Intel Core i7-12650H. Every job has 500,000 rows, parallelism one, an 8 GiB heap, two warmups and three alternating trials per engine. Widths are four and 256 bytes. The MAP_VALUES fixture has three ASCII STRING keys and three non-null binary values; the MAP_KEYS fixture has three non-null binary keys with INT values 1/NULL/-1. Every eighth parent is NULL. Performance inputs respect declared widths and nullability; separate correctness checks include real NULL leaves and out-of-declaration lengths in both generic and binary maps.

Previous candidate 25fe176a uses identical benchmark/input sources and the same frozen release DSO. The current parent's later merge adds only a unit-test fixture repair and suite documentation; production and measured benchmark sources are unchanged. Each admitted measured job requires consumed NativeCalc and both row/Arrow transpose counters. Stock and previous-fallback jobs require no native operators. Same-source identity controls still consume native work on both checkouts and are retained without subtraction. Timing includes planning/submission, processing, all intrinsic conversions and the blackhole row sink. Fresh per-job metrics-cluster/schema setup and plan assertions are outside timing. No other local Java/Cargo work overlaps measurements. All trials, both stock controls, identities and profiles retain the unfavorable results.

Median seconds (minimum–maximum), including complete-job startup:

Expression / bytes per leaf Paired stock Previous fallback Generated native route
NOT_NULL_BINARY_MAP_VALUES / 4 0.295370 (0.288900–0.307176) 0.301189 (0.289556–0.302601) 0.620183 (0.619322–0.622486)
NOT_NULL_BINARY_MAP_STRING / 4 0.458551 (0.455308–0.476524) 0.454731 (0.442781–0.461344) 0.796819 (0.796102–0.801216)
NOT_NULL_BINARY_MAP_KEYS / 4 0.254220 (0.253209–0.262033) 0.242558 (0.240891–0.245612) 0.463470 (0.462984–0.464247)
NOT_NULL_BINARY_KEYS_STRING / 4 0.402373 (0.397256–0.413330) 0.382396 (0.380785–0.383945) 0.570521 (0.569917–0.572501)
NOT_NULL_BINARY_MAP_VALUES / 256 0.381256 (0.375123–0.382389) 0.379787 (0.377312–0.388174) 0.846614 (0.836081–0.851432)
NOT_NULL_BINARY_MAP_STRING / 256 1.484237 (1.480179–1.501203) 1.464445 (1.449250–1.486186) 3.074439 (3.047926–3.095101)
NOT_NULL_BINARY_MAP_KEYS / 256 0.319906 (0.318649–0.339938) 0.310305 (0.305981–0.321606) 0.675674 (0.658897–0.679109)
NOT_NULL_BINARY_KEYS_STRING / 256 1.383292 (1.335081–1.385634) 1.379093 (1.372284–1.394037) 2.419619 (2.377253–2.434042)

Optimization stopped after exactly two eight-second CPU captures at 256-byte width. MAP_VALUES has 13,987 samples across all JVM threads, including 2,494 inclusive callback samples, 1,120 in generated row evaluation, 1,033 in map serialization and 350 in array serialization. Binary-key map formatting has 14,113 samples, including 6,115 in the callback, 2,426 in generated evaluation and 3,329 in represented-string writing. Both retain copying/materialization and perimeter costs. Counts overlap and include fresh-cluster setup, planning and compilation; they are not allocation or copied-byte measurements and do not isolate the whole slowdown. No safe small shortcut preserving released NULL and representation behavior was identified inside the timebox. No further profiling was performed for this increment.

Every new native median exceeds paired stock and previous fallback. These short complete jobs ship under the accepted collection-coverage tradeoff and do not establish standalone acceleration or sustained throughput. Further performance work stays in #392.

Reproduce each width with a released native build:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=ScalarFunctionBenchmark#individualFunctions' -Dsf.testForks=1 \
  -Dscalar.functions=NOT_NULL_BINARY_MAP_VALUES,NOT_NULL_BINARY_MAP_STRING,NOT_NULL_BINARY_MAP_KEYS,NOT_NULL_BINARY_KEYS_STRING \
  -Dscalar.rows=500000 -Dscalar.binary.width=256 -Dscalar.nullEvery=8 \
  -Dscalar.warmup=2 -Dscalar.runs=3

Also run with -Dscalar.binary.width=4. The previous checkout copies both benchmark/input sources and adds -Dscalar.native.expected=false. For each of NOT_NULL_BINARY_MAP_VALUES and NOT_NULL_BINARY_KEYS_STRING, select #profileFunction, set SF_PROFILE_SCALAR=true, and use -Dscalar.native=true -Dscalar.warmup=1 -Dscalar.seconds=8, with -Dprofile.asprof=/path/to/asprof -Dprofile.outputDir=/path/to/output.

Runtime ARRAY_SORT options (2026-10-27)

The runtime Boolean-option subset now executes in Rust for nullable integer and DECIMAL arrays already proven shorter than 32 elements or produced by native ARRAY_DISTINCT. Constant non-NULL options keep the existing DataFusion sort. Unrestricted long duplicate arrays, representation-sensitive types and fallible operands retain generated Flink evaluation. ARRAY_SORT is absent from 1.18.1. This is one increment of #505.

The small-function comparison uses Flink 2.2.1, HotSpot 17.0.12, Rust 1.94.0 release/mimalloc without RocksDB, macOS 26.5 arm64, an Apple M4 Pro with 14 logical cores and 24 GiB RAM. The JVM heap is 1 GiB; execution is serial, without CPU pinning. No task builds or other tests overlap measurements; desktop activity is uncontrolled. Baseline eca8f717 precedes this change. The subsequently merged integer SPLIT_INDEX increment does not change the measured ARRAY_SORT paths.

Every engine evaluates the same query and logical inputs:

SELECT ARRAY_SORT(ARRAY_SLICE(a,1,31),ascending_order,null_first) FROM inputs

Stock starts on prepared GenericRowData or BinaryRowData in Java. Previous StreamFusion and the candidate start on prepared Arrow RecordBatch input in Rust. Setup and initial expression compilation are outside timers. Stock times released generated evaluation and complete typed-array consumption. Both native timers run inside Rust around production compiled Calc evaluation and equivalent typed Arrow consumption. Previous StreamFusion retains its whole-Calc kind-38 JVM callback, including the callback's Arrow import/export and conversions. The candidate uses the production ARRAY_SLICE and typed ARRAY_SORT kernels, without a JVM callback. Initial native entry, final output export and perimeter row/Arrow transposes are outside this small-function boundary only.

Arrays contain repeated exact values; BIGINT values exceed 2^53 and decimals are DECIMAL(38,9). Nullable profiles include NULL parents, elements and independently NULL option flags. Ascending and NULL-order flags vary by row. Output schema and complete arrays match the previous generated evaluator before timing; every stock/previous/native trial also checks matching consumed-value checksums. Each profile uses eight warmups and nine measured trials, rotating engine order per trial. Batches of 1,024 rows run 32 iterations; batches of 16 run 1,024.

Median nanoseconds per row (minimum-maximum); the binary/generic column describes stock's prepared input, and every StreamFusion input is already Arrow:

Type / width / batch rows / stock input / NULL profile Stock Flink Previous StreamFusion Rust runtime options Stock/native
INT / 8 / 1024 / generic / nullable 92.65 (90.81-143.65) 247.91 (240.87-320.22) 30.79 (29.15-33.61) 3.01x
INT / 31 / 1024 / generic / nullable 469.91 (457.15-476.93) 781.34 (735.84-813.48) 93.28 (90.74-98.95) 5.04x
BIGINT / 8 / 1024 / binary / nullable 139.24 (126.27-270.01) 276.10 (255.95-325.14) 32.58 (29.15-35.66) 4.27x
BIGINT / 31 / 1024 / binary / non-null 817.17 (799.83-858.47) 1178.60 (1113.14-1253.36) 82.44 (76.79-92.76) 9.91x
DECIMAL / 8 / 1024 / binary / nullable 254.94 (243.62-312.33) 640.27 (609.27-668.54) 33.67 (32.30-37.39) 7.57x
DECIMAL / 31 / 1024 / generic / non-null 1180.24 (1163.51-1233.21) 2983.79 (2839.01-3053.41) 127.77 (124.37-133.96) 9.24x
INT / 8 / 16 / binary / nullable 125.21 (121.63-133.57) 1647.62 (1455.35-1853.17) 93.03 (84.16-102.95) 1.35x
DECIMAL / 31 / 16 / generic / nullable 633.60 (624.04-659.60) 3119.82 (3051.92-3340.86) 160.04 (152.57-165.47) 3.96x

All eight measured medians improve on stock and previous StreamFusion. The previous/native ratios range from 8.05x to 23.35x. These measurements establish the applicable small-function gate; they do not measure complete-job throughput. All 408 trials, configuration and fingerprints retain warmups, ranges and the slower observations.

A separate conversion-inclusive diagnostic uses 500,000 nullable INT rows of width eight, parallelism one, a 2 GiB heap, one warmup and three measured trials in a fresh JVM per engine. It retains a row source, both row/Arrow transposes and a blackhole row sink. Native plans require consumed NativeCalc work; the candidate must have no bound JVM expression, while the previous planner must retain its generated evaluator. Planning/submission and execution are timed; MiniCluster/environment construction and EXPLAIN checks are outside timing. The same release library and fixture run stock, previous, then candidate.

Complete job Median seconds Minimum-maximum
Stock Flink 0.224650 0.219570-0.225222
Previous StreamFusion 0.277142 0.274509-0.286544
Rust runtime options 0.197072 0.193343-0.204127

The candidate takes 12.3% less time than stock and 28.9% less than previous in this short job. This single profile does not establish sustained performance for every width/type, nor remove general row/Arrow perimeter costs.

The 27 focused Flink 2.2 SQL tests pass, including positive consumed native-work checks for all five admitted element types, independent nullable flags, a 66-element distinct array, filtering, empty input and lazy failing branches. The unchanged released ARRAY_SORT specification also passes all 56 invocations under stock and the injected native planner. Its 54 executing queries retain their expected custom-type-inference fallback, with zero native-work credit; the other two are validation cases. Native sort evidence comes from the focused SQL contracts and the compiled-Calc test that runs without a JVM. On 1.18, ARRAY_SORT is N/A; shared array search/distinct regressions pass on both lines.

After normal Java test compilation, construct a test classpath containing streamfusion-runtime/target/test-classes, its production classes and released dependencies, and supply it as SF_NATIVE_BENCH_CLASSPATH. Set SF_NATIVE_LIB_DIR to the matching release library directory and select JDK 17 with JAVA_HOME. Run one small profile from the repository root:

cargo +1.94.0 run --manifest-path native/Cargo.toml --release --no-default-features \
  -p streamfusion --example runtime_array_sort -- int 8 true false 1024 32 8 9

Arguments are element (int, bigint, decimal), width, nullable, binary stock input, batch rows, iterations, warmups and measured trials. The complete-job diagnostic is optional and excluded from ordinary test runs:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  -Dtest=RuntimeArraySortPipelineBenchmark -Dsf.testForks=1 \
  -Darray-sort.pipeline.engine=native -Darray-sort.pipeline.rows=500000 \
  -Darray-sort.pipeline.warmups=1 -Darray-sort.pipeline.runs=3

Use engine=flink for stock. For engine=previous, compile the same fixture against the preceding planner and require its bound generated ARRAY_SORT route; the candidate planner deliberately fails that previous-route assertion.

Integral SIGN

Measured on 2026-10-27, canonical baseline 7a6a790e, Apple M4 Pro (14 logical CPUs), macOS aarch64, Flink 2.2.1, JDK 17.0.12, Rust 1.94.0 release with default features disabled and the system allocator. This replaces generated computation in an already-admitted Calc; it adds no operator, type admission or configuration switch.

integral_sign prepares equal integer values, minima and NULLs outside timing. Stock Flink uses its released generated expression over prebuilt GenericRowData or BinaryRowData in Java. The previous and candidate controls use production native Calc over prebuilt Arrow batches; the previous generated JVM callback, its conversions and dispatch remain inside the Rust timer. The candidate uses the typed SIGN kernel. Every output value participates in a checksum, with full Arrow output/schema equality checked before timing. JNI entry/input import, final export and perimeter row transposes are outside this prepared-input timer.

Both logical schemas are nullable; the nullable flag controls actual data (every seventh row is NULL when true, none when false). Each case has 16 warmups and 11 measured trials, 131,072 rows per trial, rotating the three engine orders. Values are ns/row, median [min, max]; speedups divide stock/previous by candidate.

Type, input NULLs, stock row format, batch rows Stock Previous callback Candidate vs stock vs previous
BIGINT, no NULLs, binary, 1024 6.426 [6.258, 6.815] 42.782 [41.370, 44.467] 0.620 [0.578, 0.699] 10.37x 69.05x
BIGINT, with NULLs, generic, 1024 4.819 [4.381, 5.007] 43.542 [41.267, 59.324] 1.222 [1.035, 1.296] 3.94x 35.62x
BIGINT, with NULLs, binary, 16 6.275 [6.061, 7.140] 773.391 [755.702, 841.681] 22.712 [21.311, 26.172] 0.28x 34.05x
INT, no NULLs, binary, 1024 6.701 [6.256, 6.776] 42.707 [41.813, 44.414] 0.536 [0.512, 0.636] 12.50x 79.68x
INT, with NULLs, generic, 1024 4.554 [4.367, 4.623] 42.878 [41.013, 55.608] 1.428 [1.335, 1.481] 3.19x 30.03x
INT, with NULLs, binary, 16 5.650 [5.397, 5.821] 749.547 [740.193, 768.904] 22.887 [22.644, 23.201] 0.25x 32.75x

At 1,024 rows the candidate wins these prepared-input controls. At 16 rows it remains slower than stock despite removing most of the preceding callback cost. A separate native-only five-second sample identifies generic DataFusion argument/field setup and destruction, allocation, batch construction and consumption as remaining small-batch costs. Inclusive frames overlap; they are not independent percentages or a universal causal estimate. The implementation bypasses the generic scalar adapter for integer arrays/scalars. A production Calc allocation probe still requests 20 allocations / 1,255 bytes for a nullable 16-row INT batch and 20 / 5,287 bytes for 1,024 rows: result values are new, validity is shared. Further Calc batching/argument overhead remains part of #491.

A separate complete INT job retains the bounded 1M-row Flink source, both transposes and rowwise blackhole sink, with parallelism one, two warmups and five measured jobs in separate sequential JVMs for stock, previous and candidate. The native plan contains both boundary transposes and consumed NativeCalc work is required. Elapsed seconds are median [min, max]:

Stock Previous callback Candidate
0.129245 [0.119284, 0.135270] 0.168748 [0.159775, 0.187790] 0.144245 [0.134325, 0.155320]

The complete candidate job improves the preceding implementation but still loses to stock. These short local jobs include submission/JIT/startup effects; they do not establish a sustained end-to-end acceleration. Do not infer a standalone win from the prepared-input results. Full measured trials, configuration and allocation observations are retained in the raw result record.

The example is native/engine/examples/integral_sign.rs; the stock/previous helper is IntegralSignBenchmarkFunctions and the full job is IntegralSignPipelineBenchmark. Build the Java test classes, prepare their dependency classpath and set JAVA_HOME, SF_NATIVE_BENCH_CLASSPATH and SF_NATIVE_LIB_DIR to the release library directory. Then run:

cargo build --manifest-path native/Cargo.toml -p streamfusion --release \
  --no-default-features --lib --example integral_sign
native/target/release/examples/integral_sign INT true false 1024 128 16 11

The arguments are type (INT/BIGINT), with NULLs, stock BinaryRowData, rows/batch, loops, warmups and trials. SF_NATIVE_BENCH_ENGINE=native optionally isolates the candidate for profiling. The full job is opt-in with SF_BENCHMARK=true and the bench Maven profile; use -Dtest=IntegralSignPipelineBenchmark -Dsign.pipeline.engine=flink or native. Run the same benchmark class with the baseline planner/native library for the preceding implementation.

Correctness checks pass on Flink 2.2.1 and 1.18.1 with JDK 11 and 17: 12 focused SQL cases and three existing numeric/floating cases per combination. The focused runtime witnesses consume 5,003 rows for direct INT/BIGINT and explicitly widened narrow inputs, with no JVM evaluator. Released unchanged ScalarFunctionsTest.testSign passes on both lines; it tests generated expressions and is oracle evidence, while the focused runtime witnesses establish executed native work. The release Rust core suite passes 813 tests with default features disabled. Full JVM/upstream suites are left to CI.

Integral ABS

October 27, 2026; Apple M4 Pro/macOS aarch64, JDK 17, released Flink 2.2.1, Rust 1.94.0, release build without default features and with the system allocator. This replaces the JVM evaluator inside an already-admitted Calc for the verified integer ABS subset; it does not expand the set of admitted SQL islands.

The functionality control starts stock Flink on prepared RowData and the native engines on matching prepared Arrow RecordBatch inputs. The stock timer is inside Java and the native timers are inside Rust. The preceding control uses the existing generated whole-Calc callback; its intrinsic JNI, Arrow handoffs and output writing remain timed. The candidate executes the compiled production Calc, including result allocation and checksum consumption. Setup, initial input import, final output export and perimeter transposes are outside this small functionality timer. Checksums, output types and NULL positions agree before timing.

All four widths include mixed signs and minimum negative values. Nullable inputs insert NULL every seventh row. Each engine has 16 warmups and 11 measured trials of 131,072 rows, rotating engine order. The allocation probe runs outside timing; its counting allocator remains inactive during the measured iterations. Table entries are median nanoseconds per row; all trials and minimum/maximum variability are in the raw results.

Type / input profile Stock Flink Previous callback Rust Calc Stock / Rust
TINYINT / nullable GenericRowData, batch 1024 4.798 43.584 0.915 5.24x
SMALLINT / nullable GenericRowData, batch 1024 4.524 41.101 0.907 4.99x
INT / nullable GenericRowData, batch 1024 4.557 40.236 0.916 4.98x
BIGINT / nullable GenericRowData, batch 1024 4.797 41.975 0.960 5.00x
TINYINT / no-NULL BinaryRowData, batch 1024 6.486 43.763 0.529 12.27x
SMALLINT / no-NULL BinaryRowData, batch 1024 6.508 42.487 0.529 12.30x
INT / no-NULL BinaryRowData, batch 1024 5.988 43.962 0.536 11.17x
BIGINT / no-NULL BinaryRowData, batch 1024 6.363 43.394 0.605 10.51x
TINYINT / nullable BinaryRowData, batch 16 5.836 739.484 21.764 0.27x
SMALLINT / nullable BinaryRowData, batch 16 5.759 745.591 23.342 0.25x
INT / nullable BinaryRowData, batch 16 5.613 784.946 23.013 0.24x
BIGINT / nullable BinaryRowData, batch 16 6.055 764.878 22.401 0.27x

The default-size batches are 4.98-12.30x faster than stock and 43.72-82.78x faster than the preceding callback. Sixteen-row batches improve 31.94-34.15x over the old callback but remain slower than stock. Direct scalar/array dispatch removes the generic scalar adapter's temporary vectors, but fixed Calc overhead remains: the final INT probe records 20 Rust allocation requests at both batch sizes, requesting 1,255 bytes for 16 rows and 5,287 bytes for 1,024 rows. The output value buffer is new and the input validity buffer is shared. The operation still constructs a projected input and an output batch per call. These numbers do not claim that Rust beats stock for tiny batches.

A separate standalone INT query uses one million runtime rows, parallelism one, a bounded row source, a blackhole row sink, default batches, two warmups and five measured runs per engine in separate sequential JVM processes. Native runs require consumed NativeCalc work and plans containing both RowDataToArrow and ArrowToRowData. Median complete-job seconds (minimum-maximum), including submission/startup, intrinsic JNI and both transposes:

Stock Flink Previous callback Rust Calc
0.127333 (0.123515-0.129871) 0.160102 (0.157433-0.180064) 0.144527 (0.136700-0.155362)

The candidate improves the old standalone median but remains slower than stock. These short jobs and sequential process order do not establish a sustained end-to-end throughput win. Only the default-size prepared-input functionality comparison establishes a speedup here.

Reproduce after compiling test classes and resolving their released Flink classpath:

export SF_NATIVE_BENCH_CLASSPATH="streamfusion-runtime/target/test-classes:streamfusion-runtime/target/classes:$(cat /path/to/test-classpath.txt)"
export SF_NATIVE_LIB_DIR="$PWD/native/target/release"
cargo build --manifest-path native/Cargo.toml -p streamfusion --release \
  --no-default-features --lib --example integral_abs
native/target/release/examples/integral_abs INT true false 1024 128 16 11

The arguments are type, inject-NULLs, use-BinaryRowData, rows per batch, loops, warmups and trials. Select each of TINYINT/SMALLINT/INT/BIGINT; also run false true 1024 128 16 11 and true true 16 8192 16 11. The complete-job diagnostic is SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench -Dtest=IntegralAbsPipelineBenchmark -Dabs.pipeline.engine=native; use flink for stock. For the previous complete-job control, use the parent planner and matching native library with the same benchmark class and inputs.

Integral unary minus

Measured on 2026-10-27, canonical baseline 7a6a790e, Apple M4 Pro (14 logical CPUs), macOS aarch64, Flink 2.2.1, JDK 17.0.12, Rust 1.94.0 release with default features disabled and the system allocator. This removes generated computation from an already-admitted Calc; it introduces no operator/type admission or configuration switch.

The integral_negate example prepares identical integer widths, values, minima and NULLs. Stock executes the released generated expression over prebuilt GenericRowData or BinaryRowData with a Java timer. Previous/candidate execute production Calc over prebuilt Arrow RecordBatch with timers inside Rust. The preceding intrinsic JVM callback and its conversions remain in timing. The candidate uses DataFusion's physical negative expression for the input column. Full output/schema equality is checked before measurement; every output participates in the checksum. Setup, JNI entry/input import, final export and perimeter transposes are outside this prepared-input timer. Both schemas are nullable; the NULL flag controls actual data (every seventh row is NULL when true, none when false).

Each case has 16 warmups, 11 measured trials and 131,072 rows per trial. The three engine orders rotate each trial. Values are ns/row, median [min, max]; speedups divide stock/previous by candidate.

Type, input NULLs, stock row format, batch rows Stock Previous callback Candidate vs stock vs previous
BIGINT, no NULLs, binary, 1024 6.568 [6.366, 6.880] 43.653 [41.321, 46.587] 0.503 [0.476, 0.535] 13.07x 86.86x
BIGINT, with NULLs, generic, 1024 4.648 [4.501, 4.868] 42.628 [40.340, 55.610] 0.840 [0.809, 0.865] 5.53x 50.75x
BIGINT, with NULLs, binary, 16 5.954 [5.890, 6.078] 755.616 [750.959, 773.752] 18.546 [18.170, 18.993] 0.32x 40.74x
INT, no NULLs, binary, 1024 6.356 [6.096, 6.550] 41.874 [41.222, 55.153] 0.456 [0.432, 0.490] 13.94x 91.86x
INT, with NULLs, generic, 1024 4.529 [4.438, 4.743] 40.935 [39.514, 55.468] 0.828 [0.788, 0.874] 5.47x 49.45x
INT, with NULLs, binary, 16 5.523 [5.186, 5.776] 741.295 [730.931, 763.478] 16.898 [16.172, 17.647] 0.33x 43.87x
SMALLINT, no NULLs, binary, 1024 6.682 [6.150, 6.839] 39.565 [38.405, 41.207] 0.413 [0.395, 0.436] 16.19x 95.89x
SMALLINT, with NULLs, generic, 1024 4.593 [4.550, 4.630] 40.504 [40.053, 60.173] 0.818 [0.797, 0.836] 5.62x 49.52x
SMALLINT, with NULLs, binary, 16 5.770 [5.651, 5.978] 771.427 [760.795, 786.748] 18.223 [17.536, 18.751] 0.32x 42.33x
TINYINT, no NULLs, binary, 1024 6.705 [6.390, 6.922] 41.736 [40.644, 44.806] 0.437 [0.421, 0.447] 15.33x 95.41x
TINYINT, with NULLs, generic, 1024 4.732 [4.596, 5.028] 41.661 [40.280, 60.471] 0.813 [0.783, 0.882] 5.82x 51.27x
TINYINT, with NULLs, binary, 16 5.821 [5.596, 5.913] 755.487 [747.510, 766.201] 17.883 [17.567, 18.380] 0.33x 42.25x

The 1,024-row prepared-input cases beat stock; the 16-row cases still lose to stock despite removing most of the preceding callback cost. A five-second native-only CPU sample exposed scalar-function argument/field cloning, dispatch and destruction in the initial adapter. The final planner reuses DataFusion's physical negative expression for verified input-reference chains and reserves the wrapping adapter for literal-rooted chains. A nullable 16-row INT allocation probe falls from 20 requests / 1,255 bytes to 16 / 838 bytes. Output values are new and validity is shared. These are allocation traffic counts, not peak memory or copied bytes. The remaining profile includes Calc batch/schema construction, allocation, Arrow numeric output and checksum consumption; inclusive stacks overlap. Generic small-batch costs remain part of #491.

A separate complete 1M-row INT job retains the bounded Flink row source, both transposes and rowwise blackhole sink. Parallelism is one, with two warmups and five measured jobs per engine in separate sequential stock/previous/candidate JVMs. Native plans require both boundary transposes and consumed NativeCalc work. Elapsed seconds are median [min, max]:

Stock Previous callback Candidate
0.128052 [0.119442, 0.132488] 0.162798 [0.161175, 0.181275] 0.146477 [0.134687, 0.153037]

The complete candidate job improves the preceding implementation but still loses to stock. These short jobs include submission, JIT and startup costs; the results do not demonstrate sustained standalone acceleration. The prepared-input win does not remove the row/Arrow perimeter cost. Full repeated measurements, configuration and allocation observations are in the raw result record.

The example is native/engine/examples/integral_negate.rs, with Java helper IntegralNegateBenchmarkFunctions and opt-in complete-job fixture IntegralNegatePipelineBenchmark. Build Java test classes and their dependency classpath, set JAVA_HOME, SF_NATIVE_BENCH_CLASSPATH and SF_NATIVE_LIB_DIR to the matching release library directory:

cargo build --manifest-path native/Cargo.toml -p streamfusion --release \
  --no-default-features --lib --example integral_negate
native/target/release/examples/integral_negate INT true false 1024 128 16 11

Arguments are integer type (TINYINT/SMALLINT/INT/BIGINT), with NULLs, stock BinaryRowData, rows/batch, loops, warmups and measured trials. SF_NATIVE_BENCH_ENGINE=native isolates the candidate for profiling. The complete job uses SF_BENCHMARK=true, Maven -Pbench, -Dtest=IntegralNegatePipelineBenchmark and -Dnegate.pipeline.engine=flink or native. Run the same fixture against the baseline planner/native library for the preceding control.

Validation passes on Flink 2.2.1/1.18.1 with JDK 11/17: 14 focused SQL cases, one encoding case and two existing numeric cases per combination. Runtime witnesses consume 5,003 rows per integer width with a recorded native minus route, no JVM evaluator and no fallback. Minima, widths, NULLs, filtering, empty batches, lossless widening, generated producer guards and selected/unselected failure behavior are covered. Unchanged released ScalarOperatorsTest.testUnaryPlusMinus passes on both lines; it exercises generated expressions and supplies oracle evidence, while the focused witnesses establish executed native work. All 814 release Rust core tests pass without default features, including a compiled-Calc literal-minimum test that runs without a JVM. Full JVM/upstream suites are left to CI.

Integer FLOOR

Measured on 2026-10-27 against canonical main 1604a3aa: Apple M4 Pro (14 logical CPUs), macOS aarch64, Flink 2.2.1, JDK 17.0.12, Rust 1.94.0 release, default features disabled, system allocator. The change removes generated computation from an already-admitted Calc; it does not add operator/type admission or a configuration switch.

integral_rounding prepares identical typed integer values, minima, BIGINTs above 2^53 and NULLs outside timing. Stock uses released generated Java over prepared GenericRowData or BinaryRowData. Previous and candidate use production Calc over prepared Arrow; the previous JVM callback, its conversion and dispatch remain inside the Rust timer. Candidate integer FLOOR emits its operand directly. Full Arrow/schema equality is checked before timing; every output value participates in an equal checksum in its own runtime. JNI entry/import, final export and perimeter transposes are outside this small-function timer.

All schemas are nullable; the input NULL flag controls actual values (NULL every seventh row when true). Each case has 16 warmups and 11 measured trials of 131,072 rows, rotating stock/previous/native order. Values are ns/row, median [min, max].

Type, input NULLs, stock format, batch rows Stock Previous callback Candidate vs stock vs previous
BIGINT, no NULLs, binary, 1024 6.403 [6.145, 6.606] 43.082 [41.289, 44.238] 0.337 [0.325, 0.340] 19.00x 127.85x
BIGINT, with NULLs, generic, 1024 4.740 [4.649, 17.621] 40.934 [39.657, 42.302] 0.711 [0.699, 0.836] 6.67x 57.59x
BIGINT, with NULLs, binary, 16 6.242 [5.819, 6.345] 766.428 [754.580, 802.290] 14.398 [13.828, 15.580] 0.43x 53.23x
INT, no NULLs, binary, 1024 6.459 [6.215, 6.724] 42.038 [40.380, 64.610] 0.326 [0.295, 0.353] 19.78x 128.76x
INT, with NULLs, generic, 1024 4.638 [4.491, 5.035] 42.349 [40.091, 59.744] 0.714 [0.702, 0.789] 6.50x 59.34x
INT, with NULLs, binary, 16 5.905 [5.735, 6.150] 741.679 [736.464, 775.826] 14.805 [14.082, 15.093] 0.40x 50.10x
SMALLINT, no NULLs, binary, 1024 6.331 [5.972, 6.491] 41.370 [41.097, 43.408] 0.336 [0.294, 0.351] 18.84x 123.12x
SMALLINT, with NULLs, generic, 1024 4.580 [4.271, 4.637] 41.599 [39.527, 52.664] 0.714 [0.611, 0.746] 6.41x 58.26x
SMALLINT, with NULLs, binary, 16 5.690 [5.347, 6.019] 782.506 [765.244, 791.278] 14.865 [13.527, 15.567] 0.38x 52.64x
TINYINT, no NULLs, binary, 1024 6.172 [5.958, 6.412] 40.221 [38.532, 41.907] 0.329 [0.303, 0.407] 18.76x 122.25x
TINYINT, with NULLs, generic, 1024 4.679 [4.580, 4.944] 41.606 [40.318, 56.980] 0.690 [0.630, 0.742] 6.79x 60.34x
TINYINT, with NULLs, binary, 16 5.444 [5.244, 5.704] 752.234 [742.954, 769.962] 14.164 [13.976, 14.453] 0.38x 53.11x

The 1,024-row candidate wins these prepared-input controls. The 16-row controls still lose to stock, despite sharing all output value/validity buffers and eliminating the callback. The production INT Calc allocation probe requests 13 allocations / 606 bytes at either batch size, including RecordBatch/result bookkeeping; no new output buffer bytes are requested. A separate five-second native-only sample shows Calc evaluation, RecordBatch projection, allocation/free, Arc ownership and checksum consumption as remaining costs. The sample is diagnostic; overlapping frames do not establish independent percentages. This experiment does not claim that pure Rust or columnar execution always wins.

A separate complete 1M-row INT job retains a bounded Flink row source, both transposes and a rowwise blackhole sink. Parallelism is one, with two warmups and five measured jobs in separate sequential stock/previous/native JVMs. Previous planning uses the unchanged main encoder and requires a FLOOR JVM route; the candidate requires jvm=[]. Both native jobs require NativeCalc, both transposes and positive consumed native work. Seconds are median [min, max]:

Stock Previous callback Candidate
0.131442 [0.125461, 0.149769] 0.158871 [0.157430, 0.172846] 0.138411 [0.132027, 0.139610]

The candidate improves the preceding implementation but remains slower than stock in this complete job. These short jobs include startup, submission and JIT effects; they do not prove sustained end-to-end acceleration. Full trials/checksums and allocation observations are in the raw result record. Broader Calc/perimeter costs remain in #491.

The released Java control is IntegralRoundingBenchmarkFunctions; the complete job is IntegralRoundingPipelineBenchmark, enabled by SF_BENCHMARK=true and the bench profile. After building Java test classes, prepare the dependency classpath and set JAVA_HOME, SF_NATIVE_BENCH_CLASSPATH and SF_NATIVE_LIB_DIR to the release library directory:

cargo build --manifest-path native/Cargo.toml -p streamfusion --release \
  --no-default-features --lib --example integral_rounding
SF_INTEGRAL_ROUNDING_FUNCTION=FLOOR \
  native/target/release/examples/integral_rounding INT true false 1024 128 16 11

For the complete job, use -Drounding.pipeline.function=FLOOR and -Drounding.pipeline.engine=flink|previous|native. The previous control requires the preceding encoder first on the classpath; merely changing the engine label is insufficient. SF_NATIVE_BENCH_ENGINE=native selects only native trials for a separate profiling diagnostic.

Integer CEIL / CEILING

Measured separately on 2026-10-27 against canonical main 1604a3aa, on the same Apple M4 Pro, Flink 2.2.1, JDK 17.0.12 and Rust 1.94.0 release configuration as Integer FLOOR. The input values, all four widths, NULL patterns, stock GenericRowData/BinaryRowData and Rust-side production Calc timing boundaries are identical. The query calls CEIL; CEILING resolves to the same admitted operator and is additionally covered by runtime parity tests. Previous generated JVM computation/conversions remain inside its Rust timer; candidate integer rounding reuses the verified operand. Setup, JNI entry/import/export and perimeter transposes are excluded from prepared-input timing. Complete schema/Arrow equality and checksums verify every output. No operator/type admission or configuration switch is added.

All input schemas are nullable; the flag denotes actual NULL values. Each case rotates stock/previous/native for 16 warmups and 11 measured trials of 131,072 rows. Values are ns/row, median [min, max].

Type, input NULLs, stock format, batch rows Stock Previous callback Candidate vs stock vs previous
BIGINT, no NULLs, binary, 1024 6.556 [6.260, 6.740] 43.084 [41.326, 47.007] 0.344 [0.329, 0.382] 19.06x 125.26x
BIGINT, with NULLs, generic, 1024 4.680 [4.479, 4.859] 41.885 [39.602, 55.228] 0.699 [0.676, 0.774] 6.70x 59.95x
BIGINT, with NULLs, binary, 16 6.058 [5.987, 6.602] 756.261 [746.306, 797.435] 13.754 [13.474, 14.605] 0.44x 54.99x
INT, no NULLs, binary, 1024 6.361 [6.240, 6.480] 40.662 [38.870, 41.578] 0.311 [0.280, 0.325] 20.48x 130.92x
INT, with NULLs, generic, 1024 4.534 [4.463, 4.759] 41.025 [39.273, 54.750] 0.692 [0.669, 0.814] 6.55x 59.28x
INT, with NULLs, binary, 16 5.335 [5.174, 5.623] 753.331 [740.801, 781.752] 14.322 [13.754, 14.870] 0.37x 52.60x
SMALLINT, no NULLs, binary, 1024 6.587 [6.408, 6.717] 41.383 [40.316, 43.505] 0.336 [0.300, 0.353] 19.60x 123.16x
SMALLINT, with NULLs, generic, 1024 4.627 [4.476, 5.068] 41.518 [39.421, 54.906] 0.693 [0.678, 0.739] 6.67x 59.88x
SMALLINT, with NULLs, binary, 16 5.530 [5.313, 7.826] 741.658 [735.667, 765.638] 15.277 [14.481, 15.898] 0.36x 48.55x
TINYINT, no NULLs, binary, 1024 6.353 [6.011, 6.534] 39.769 [38.562, 40.987] 0.367 [0.337, 0.393] 17.32x 108.41x
TINYINT, with NULLs, generic, 1024 4.728 [4.625, 4.823] 41.698 [39.988, 54.603] 0.675 [0.666, 0.692] 7.01x 61.78x
TINYINT, with NULLs, binary, 16 5.651 [5.270, 5.855] 747.532 [735.846, 774.913] 14.199 [13.643, 15.871] 0.40x 52.65x

The candidate beats both controls at 1,024 rows. At 16 rows it remains slower than stock, with the same shared-operand Calc path and generic batch/result costs diagnosed for FLOOR. The INT allocation probe again requests 13 allocations / 606 bytes at both batch sizes; all output value/validity buffers are shared. This is a measured improvement to the existing native Calc path, without a universal small-batch acceleration claim.

The independent complete job uses the same 1M-row INT source, parallelism one, rowwise blackhole sink, two warmups and five measured jobs in sequential stock/previous/native JVMs. Both transposes stay in native plans, and positive consumed NativeCalc work is required. The previous unchanged encoder must report a CEIL JVM route; candidate must report jvm=[]. Seconds are median [min, max]:

Stock Previous callback Candidate
0.127985 [0.118868, 0.129826] 0.154719 [0.149769, 0.156016] 0.139331 [0.128387, 0.145281]

The candidate improves previous StreamFusion but remains slower than stock. Startup/JIT and submission are included; this is not evidence of sustained complete-job acceleration. All measured trials, checksums, configuration and allocation observations are retained. Generic Calc/perimeter costs remain in #491.

Use the shared integral_rounding release example and IntegralRoundingPipelineBenchmark from the FLOOR instructions, selecting SF_INTEGRAL_ROUNDING_FUNCTION=CEIL for prepared controls or -Drounding.pipeline.function=CEIL for complete jobs. The explicit previous encoder and route checks apply to both functions.

Single-argument integer ROUND

Measured on 2026-10-27 against canonical main d98dc2f2, on Apple M4 Pro, JDK 17.0.12, Rust 1.94.0 release without default features. Both released Flink 1.18.1 and 2.2.1 are measured independently. Only ROUND(value) on verified integer operands is changed; explicit scale arguments and unverified producers retain generated evaluation.

Equal prepared values, resolved widths and NULL patterns start on RowData for stock Flink and Arrow RecordBatch for previous/candidate native Calc. Stock executes its released generated expression on prepared GenericRowData/BinaryRowData. Native timers start and stop inside Rust around production Calc and equivalent checksum consumption; preceding JVM computation and its intrinsic conversions remain timed. Setup, initial JNI entry, Arrow import/export and perimeter transposes are excluded from prepared measurements. Exact output schema/Arrow equality is checked before timing, and every timed output contributes to matching checksums. All schemas are nullable; the NULL flag denotes actual NULL values every seventh row. Stock/previous/native order rotates for 16 warmups and 11 measured trials of 131,072 rows each. Values are median [min, max] ns/row.

Flink, type, NULL values, stock format, batch rows Stock ns/row Previous ns/row Native ns/row vs stock vs previous
1.18.1, BIGINT, false, binary, 1024 8.986 [6.941, 18.408] 54.478 [47.632, 64.772] 0.348 [0.306, 0.363] 25.79x 156.36x
1.18.1, BIGINT, true, generic, 1024 6.400 [5.808, 16.941] 47.064 [42.686, 50.074] 0.701 [0.681, 0.875] 9.13x 67.11x
1.18.1, BIGINT, true, binary, 16 5.628 [5.491, 6.113] 768.974 [761.583, 784.511] 14.967 [14.371, 15.448] 0.38x 51.38x
1.18.1, INT, false, binary, 1024 5.838 [5.432, 5.972] 42.728 [40.682, 57.784] 0.305 [0.292, 0.317] 19.17x 140.30x
1.18.1, INT, true, generic, 1024 4.519 [4.452, 18.006] 37.391 [35.634, 39.007] 0.717 [0.693, 0.829] 6.30x 52.14x
1.18.1, INT, true, binary, 16 5.409 [5.286, 5.889] 762.968 [757.441, 772.750] 14.406 [14.010, 14.858] 0.38x 52.96x
1.18.1, SMALLINT, false, binary, 1024 5.689 [5.467, 5.932] 41.973 [39.931, 59.374] 0.328 [0.315, 0.346] 17.33x 127.82x
1.18.1, SMALLINT, true, generic, 1024 5.853 [5.665, 6.021] 42.039 [40.251, 62.638] 0.693 [0.672, 0.741] 8.44x 60.63x
1.18.1, SMALLINT, true, binary, 16 5.446 [5.280, 5.825] 742.352 [737.850, 752.496] 14.085 [13.723, 14.905] 0.39x 52.71x
1.18.1, TINYINT, false, binary, 1024 7.581 [7.238, 17.433] 42.084 [41.233, 57.969] 0.336 [0.313, 0.375] 22.56x 125.25x
1.18.1, TINYINT, true, generic, 1024 8.117 [5.692, 8.867] 43.509 [38.539, 50.779] 0.745 [0.706, 0.932] 10.90x 58.41x
1.18.1, TINYINT, true, binary, 16 5.629 [5.530, 5.820] 772.427 [763.950, 783.090] 14.762 [13.350, 15.756] 0.38x 52.33x
2.2.1, BIGINT, false, binary, 1024 10.198 [8.557, 20.903] 50.490 [45.352, 60.357] 0.346 [0.334, 0.364] 29.46x 145.85x
2.2.1, BIGINT, true, generic, 1024 6.886 [6.348, 18.278] 55.496 [48.014, 63.099] 0.696 [0.677, 0.750] 9.89x 79.75x
2.2.1, BIGINT, true, binary, 16 6.246 [6.150, 6.558] 773.582 [756.754, 793.155] 14.002 [13.630, 16.256] 0.45x 55.25x
2.2.1, INT, false, binary, 1024 7.436 [6.917, 14.206] 56.088 [49.785, 72.955] 0.330 [0.317, 0.359] 22.53x 169.98x
2.2.1, INT, true, generic, 1024 4.766 [4.670, 4.946] 48.693 [44.799, 61.810] 0.692 [0.672, 0.851] 6.88x 70.33x
2.2.1, INT, true, binary, 16 6.060 [6.009, 6.960] 781.974 [769.627, 816.086] 15.437 [13.803, 18.351] 0.39x 50.66x
2.2.1, SMALLINT, false, binary, 1024 6.265 [6.026, 6.478] 48.445 [43.458, 62.665] 0.338 [0.329, 0.347] 18.52x 143.23x
2.2.1, SMALLINT, true, generic, 1024 4.925 [4.675, 5.293] 41.861 [40.380, 56.673] 0.710 [0.676, 0.787] 6.93x 58.94x
2.2.1, SMALLINT, true, binary, 16 5.575 [5.347, 5.802] 773.666 [761.840, 789.940] 14.704 [14.331, 15.127] 0.38x 52.62x
2.2.1, TINYINT, false, binary, 1024 8.385 [7.735, 19.151] 44.980 [41.002, 64.401] 0.343 [0.319, 0.370] 24.47x 131.26x
2.2.1, TINYINT, true, generic, 1024 6.457 [5.727, 15.747] 42.013 [39.691, 56.503] 0.696 [0.676, 0.845] 9.28x 60.37x
2.2.1, TINYINT, true, binary, 16 6.280 [6.077, 6.752] 732.857 [728.134, 750.648] 15.533 [14.994, 16.240] 0.40x 47.18x

At 1,024 rows this shared-operand path improves 6.30-29.46x over stock and 52.14-169.98x over the preceding callback. Sixteen-row controls remain slower than stock, despite removing most callback overhead. The candidate has the same input-expression Calc path as FLOOR/CEIL, without a rounding loop to optimize. Its allocation probe requests 13 allocations / 606 bytes per batch and shares all output value/validity buffers; generic Calc/RecordBatch ownership, result handling and checksum consumption retain the fixed costs identified in the FLOOR profile. These are sampled batch improvements, not a universal small-batch acceleration claim.

The independent complete job uses 1M INT rows, parallelism one, a row source and a rowwise blackhole sink. Native plans retain both transposes, require consumed NativeCalc work and distinguish the old JVM route from candidate jvm=[]. The previous encoder comes from d98dc2f2; relabeling an unchanged candidate is rejected. Separate sequential JVMs run two warmups and five measured jobs per engine. Seconds are median [min, max]:

Stock Previous callback Candidate
0.138810 [0.133661, 0.148813] 0.162969 [0.156567, 0.183940] 0.141323 [0.134961, 0.155040]

The complete candidate improves previous StreamFusion but remains slightly slower than stock by median, with overlapping ranges. Submission, startup and JIT remain included; this does not establish sustained full-job acceleration. All raw trials, checksums, configuration and allocation observations are in the result record.

Use the shared integral_rounding release example from the FLOOR instructions, selecting SF_INTEGRAL_ROUNDING_FUNCTION=ROUND. For the complete job, select -Drounding.pipeline.function=ROUND with -Drounding.pipeline.engine=flink|previous|native; the previous control requires the preceding encoder first on the classpath. Runtime parity/route tests cover all four integer widths on both supported Flink lines and JDK 11/17, including NULLs, signed minima, nested/widened calls and failing unselected branches. The unchanged released testRound fixtures pass on both lines.

Single-argument integer TRUNCATE

Measured on 2026-10-27 against canonical main d98dc2f2, using the same machine, release configuration, timing boundaries, NULL patterns and output checks as single-argument integer ROUND. Both Flink 1.18.1 and 2.2.1 are measured independently. INT and BIGINT single-argument TRUNCATE reuse the verified operand instead of invoking Flink's DECIMAL(38,18) intermediate. This is exact at the default zero scale, including signed minima and large BIGINTs. Explicit scales, unverified producers, DECIMAL/floating contracts and direct TINYINT/SMALLINT compilation failures retain their preceding paths.

Stock runs the released generated expression on equal prepared RowData; previous and candidate run production Calc on prepared Arrow inside Rust. Intrinsic callback and conversion costs stay timed; setup/import/export and perimeter transposes are excluded. Every output contributes to a checksum, with full schema/Arrow equality checked before timing. Sixteen warmups and eleven measured 131,072-row trials rotate engine order. All input schemas are nullable; actual NULL values occur every seventh row when enabled. Values are median [min, max] ns/row.

Flink, type, NULL values, stock format, batch rows Stock ns/row Previous ns/row Native ns/row vs stock vs previous
1.18.1, BIGINT, false, binary, 1024 114.692 [108.152, 121.879] 163.985 [155.083, 185.194] 0.357 [0.305, 0.395] 321.56x 459.76x
1.18.1, BIGINT, true, generic, 1024 94.695 [88.018, 99.201] 152.532 [148.545, 156.392] 0.719 [0.698, 0.833] 131.75x 212.22x
1.18.1, BIGINT, true, binary, 16 102.172 [94.620, 103.305] 851.419 [832.244, 868.287] 14.344 [13.178, 14.791] 7.12x 59.36x
1.18.1, INT, false, binary, 1024 50.861 [48.534, 54.994] 100.856 [94.868, 109.976] 0.345 [0.308, 0.409] 147.60x 292.68x
1.18.1, INT, true, generic, 1024 46.233 [44.280, 49.321] 91.235 [87.666, 102.056] 0.720 [0.684, 0.777] 64.21x 126.71x
1.18.1, INT, true, binary, 16 42.947 [40.513, 45.516] 811.769 [806.045, 815.572] 14.427 [13.648, 14.712] 2.98x 56.27x
2.2.1, BIGINT, false, binary, 1024 112.933 [103.880, 117.159] 167.469 [157.077, 173.877] 0.357 [0.310, 0.408] 316.07x 468.70x
2.2.1, BIGINT, true, generic, 1024 99.859 [94.567, 104.457] 147.552 [141.161, 153.150] 0.703 [0.610, 0.746] 142.01x 209.84x
2.2.1, BIGINT, true, binary, 16 98.121 [88.774, 101.313] 839.763 [827.729, 851.937] 15.164 [14.409, 16.540] 6.47x 55.38x
2.2.1, INT, false, binary, 1024 64.964 [59.286, 73.217] 95.684 [94.034, 98.344] 0.318 [0.294, 0.347] 204.56x 301.30x
2.2.1, INT, true, generic, 1024 45.872 [44.496, 53.871] 91.028 [85.339, 94.128] 0.700 [0.622, 0.741] 65.53x 130.04x
2.2.1, INT, true, binary, 16 55.065 [51.269, 60.222] 834.556 [821.474, 875.742] 14.462 [13.581, 15.038] 3.81x 57.70x

All sampled cases beat stock and the previous callback. The 1,024-row cases improve 64.21-321.56x over stock and 126.71-468.70x over previous StreamFusion. Sixteen-row controls improve 2.98-7.12x over stock and 55.38-59.36x over previous StreamFusion. The native path remains a shared-operand Calc, requesting 13 allocations / 606 bytes per batch with no new output values/validity buffers. Flink's decimal conversion and rounding work account for the difference from FLOOR/CEIL/ROUND controls; these batch ratios must not be applied to complete jobs.

A separate Flink 2.2.1 complete job retains a bounded 1M-row INT source, both transposes, parallelism one and a rowwise blackhole sink. Positive consumed NativeCalc work and the expected previous/candidate JVM route are required. Previous uses the canonical main encoder. Each engine runs in a separate sequential JVM with two warmups and five measured jobs. Seconds are median [min, max]:

Stock Previous callback Candidate
0.188721 [0.185241, 0.200471] 0.261994 [0.256899, 0.264275] 0.132958 [0.129006, 0.147995]

The measured candidate reduces elapsed time by about 29.5% versus stock and 49.3% versus the preceding implementation in this workload. Startup, submission and JIT remain included; this is not proof of sustained acceleration for arbitrary inputs or jobs. The raw result record retains every trial, checksum, allocation observation and benchmark configuration.

Use the shared release example with SF_INTEGRAL_ROUNDING_FUNCTION=TRUNCATE and INT or BIGINT. Complete-job controls select -Drounding.pipeline.function=TRUNCATE; the prior encoder must be first on the classpath for the previous control. Runtime parity and consumed-work witnesses cover both types on Flink 1.18.1/2.2.1 and JDK 11/17. Explicit widening of TINYINT/SMALLINT is verified, while direct calls retain stock compilation failures. Unchanged released testTruncate fixtures pass on both lines.

DECIMAL ABS

Measured on 2026-10-28 against canonical main d89e03b5, with released Flink 1.18.1 and 2.2.1, JDK 17, Apple M4 Pro, Rust 1.94.0 and a release/no-default-features native build with the system allocator. All core Java production sources and compatibility sources are compiled from the candidate with --release 11. This replaces generated JVM computation inside an already-admitted Calc without changing operator/type admission.

Stock executes the released generated expression over prepared GenericRowData or BinaryRowData. Previous and candidate execute production Calc over equal Arrow values, types and NULLs, with the timer inside Rust. Setup, Arrow import/export and perimeter transposes are excluded from this prepared control. The previous control uses the scalar generated-expression callback emitted by main (kind 17), including decimal argument and result marshalling; the earlier whole-row callback control is superseded. Intrinsic conversions, result allocation and equivalent signum consumption remain timed. Full schema/Arrow output equality is checked before timing, and each measured output contributes to a checksum. Schemas are nullable; enabled NULL values occur every seventh row. Inputs alternate signs, include precision maxima, and include values beyond 2^53 for noncompact types. Sixteen warmups and eleven measured trials of 65,536 rows rotate engine order. No other test or benchmark runs concurrently. Values are median [min, max] ns/row:

Flink, type, NULL values, stock format, batch rows Stock ns/row Previous ns/row Native ns/row vs stock vs previous
1.18.1, DECIMAL(18,3), false, binary, 1024 6.771 [6.255, 33.129] 38.748 [31.810, 46.170] 1.277 [1.254, 1.657] 5.30x 30.34x
1.18.1, DECIMAL(18,3), true, generic, 1024 6.641 [5.821, 31.708] 46.051 [41.422, 67.759] 1.778 [1.691, 1.986] 3.74x 25.91x
1.18.1, DECIMAL(18,3), true, binary, 16 5.556 [4.824, 8.092] 596.312 [564.630, 711.404] 25.133 [23.392, 26.421] 0.22x 23.73x
1.18.1, DECIMAL(19,3), false, binary, 1024 31.134 [23.839, 47.973] 66.718 [48.780, 139.852] 1.256 [1.146, 1.330] 24.78x 53.11x
1.18.1, DECIMAL(19,3), true, generic, 1024 8.194 [7.320, 9.511] 96.540 [75.018, 127.890] 1.782 [1.696, 1.970] 4.60x 54.17x
1.18.1, DECIMAL(19,3), true, binary, 16 18.159 [17.847, 23.510] 685.521 [571.056, 760.549] 23.739 [22.762, 24.487] 0.76x 28.88x
1.18.1, DECIMAL(38,18), false, binary, 1024 25.438 [22.809, 48.136] 58.722 [44.867, 91.820] 1.297 [1.263, 1.371] 19.61x 45.27x
1.18.1, DECIMAL(38,18), true, generic, 1024 8.318 [7.112, 30.991] 89.708 [78.290, 132.299] 1.781 [1.743, 2.115] 4.67x 50.37x
1.18.1, DECIMAL(38,18), true, binary, 16 18.058 [16.403, 22.636] 594.976 [570.112, 623.375] 24.618 [24.240, 25.860] 0.73x 24.17x
1.18.1, DECIMAL(38,38), false, binary, 1024 27.431 [24.631, 49.947] 66.619 [46.130, 134.900] 1.331 [1.214, 1.488] 20.60x 50.04x
1.18.1, DECIMAL(38,38), true, generic, 1024 7.809 [6.671, 30.333] 94.542 [83.144, 120.974] 1.834 [1.743, 4.465] 4.26x 51.56x
1.18.1, DECIMAL(38,38), true, binary, 16 19.852 [17.552, 23.687] 624.911 [583.037, 760.024] 24.169 [23.059, 28.229] 0.82x 25.86x
2.2.1, DECIMAL(18,3), false, binary, 1024 8.650 [6.227, 34.993] 53.272 [44.177, 86.080] 1.297 [1.142, 1.580] 6.67x 41.07x
2.2.1, DECIMAL(18,3), true, generic, 1024 7.817 [6.538, 14.962] 64.180 [36.743, 212.652] 2.127 [1.836, 3.219] 3.67x 30.17x
2.2.1, DECIMAL(18,3), true, binary, 16 5.786 [4.878, 8.431] 596.074 [567.899, 718.964] 22.915 [22.058, 24.011] 0.25x 26.01x
2.2.1, DECIMAL(19,3), false, binary, 1024 26.063 [24.449, 52.284] 66.973 [49.121, 126.535] 1.317 [1.204, 1.621] 19.79x 50.86x
2.2.1, DECIMAL(19,3), true, generic, 1024 8.820 [7.537, 10.936] 93.368 [72.261, 150.984] 1.766 [1.605, 2.389] 4.99x 52.86x
2.2.1, DECIMAL(19,3), true, binary, 16 19.715 [18.974, 20.285] 596.653 [570.852, 694.953] 24.459 [23.104, 25.412] 0.81x 24.39x
2.2.1, DECIMAL(38,18), false, binary, 1024 25.969 [25.345, 49.030] 71.037 [49.401, 127.594] 1.300 [1.154, 1.346] 19.97x 54.64x
2.2.1, DECIMAL(38,18), true, generic, 1024 7.465 [7.186, 34.662] 75.359 [47.146, 168.208] 1.760 [1.705, 2.176] 4.24x 42.81x
2.2.1, DECIMAL(38,18), true, binary, 16 19.304 [17.955, 27.007] 581.214 [573.262, 902.620] 24.179 [23.502, 26.571] 0.80x 24.04x
2.2.1, DECIMAL(38,38), false, binary, 1024 23.404 [22.887, 24.978] 66.765 [54.471, 96.620] 1.295 [1.149, 1.497] 18.07x 51.55x
2.2.1, DECIMAL(38,38), true, generic, 1024 7.948 [7.013, 10.184] 66.031 [48.108, 108.056] 1.802 [1.633, 1.963] 4.41x 36.65x
2.2.1, DECIMAL(38,38), true, binary, 16 18.920 [17.187, 22.478] 587.500 [567.618, 703.150] 26.159 [24.940, 27.522] 0.72x 22.46x

At 1,024 rows the candidate is 3.67-24.78x faster than stock and 25.91-54.64x faster than the preceding callback. Sixteen-row controls still lose to stock (0.22-0.82x), while improving the previous native-Calc route by 22.46-28.88x. These are sampled prepared-batch gains; they do not establish universal small-batch acceleration. The nullable 1,024-row allocation probe records 20 allocations / 17,575 requested bytes, with 16,384 new value bytes and 128 shared validity bytes. Generic Calc/batch/result handling retains fixed costs alongside the allocation-free per-element transform. Remaining small-batch and row/Arrow perimeter work stays in #491.

The independent complete job uses 1M DECIMAL(18,3) rows, parallelism one, a bounded row source, both transposes and a rowwise blackhole sink. The previous control loads the actual d89e03b5 encoder ahead of candidate classes and requires its JVM route; candidate requires jvm=[]. Both require positive consumed NativeCalc work. Each engine runs two warmups and five measured jobs in separate sequential JVMs. Seconds are median [min, max]:

Stock Previous callback Candidate
0.138626 [0.123898, 0.143039] 0.183183 [0.176360, 0.212639] 0.139391 [0.137642, 0.155442]

The complete candidate improves previous StreamFusion by 23.9% but remains 0.6% slower than stock by median in this workload. Startup, submission, JIT, source conversion and both transposes remain included; their individual contributions are not isolated here. The prepared-batch gain is not a whole-job acceleration claim. The result record retains every trial, checksum, environment and allocation observation.

After test-compiling with the bench Maven profile, set SF_NATIVE_BENCH_CLASSPATH to test classes, production classes and Maven dependencies, and SF_NATIVE_LIB_DIR to the release library directory. Build/run the shared control from native/:

cargo build --release -p streamfusion --no-default-features --example decimal_unary
./target/release/examples/decimal_unary 18 3 true false 1024 64 16 11

Arguments are precision, scale, actual NULL values, BinaryRowData input, batch rows, loops, warmups and measured trials. Complete controls use DecimalUnaryPipelineBenchmark with SF_BENCHMARK=true and -Ddecimal.unary.pipeline.engine=flink|previous|native; previous requires the baseline encoder first on the classpath. Runtime tests verify results, resolved types, consumed native work and no JVM binding on both Flink lines and JDK 11/17. Rust/encoding checks cover all legal precision/scale pairs. Unchanged released testAbs fixtures pass on both lines.

DECIMAL unary minus

Measured on 2026-10-28 against canonical main d89e03b5, with released Flink 1.18.1 and 2.2.1, JDK 17, Apple M4 Pro, Rust 1.94.0 and a release/no-default-features native build with the system allocator. All core Java production sources and compatibility sources are compiled from the candidate with --release 11. This replaces generated JVM computation inside an already-admitted Calc without changing operator/type admission.

Stock executes the released generated expression over prepared GenericRowData or BinaryRowData. Previous and candidate execute production Calc over equal Arrow values, types and NULLs, with the timer inside Rust. Setup, Arrow import/export and perimeter transposes are excluded from this prepared control. The previous control uses the scalar generated-expression callback emitted by main (kind 17), including decimal argument and result marshalling; the earlier whole-row callback control is superseded. Intrinsic conversions, result allocation and equivalent signum consumption remain timed. Full schema/Arrow output equality is checked before timing, and each measured output contributes to a checksum. Schemas are nullable; enabled NULL values occur every seventh row. Inputs alternate signs, include precision maxima, and include values beyond 2^53 for noncompact types. Sixteen warmups and eleven measured trials of 65,536 rows rotate engine order. No other test or benchmark runs concurrently. Values are median [min, max] ns/row:

Flink, type, NULL values, stock format, batch rows Stock ns/row Previous ns/row Native ns/row vs stock vs previous
1.18.1, DECIMAL(18,3), false, binary, 1024 5.608 [5.397, 30.964] 38.780 [36.360, 48.179] 1.424 [1.360, 1.448] 3.94x 27.23x
1.18.1, DECIMAL(18,3), true, generic, 1024 5.138 [4.994, 5.221] 39.243 [30.175, 62.565] 2.113 [2.067, 2.254] 2.43x 18.57x
1.18.1, DECIMAL(18,3), true, binary, 16 5.169 [5.025, 9.325] 589.500 [563.921, 610.549] 25.894 [24.883, 27.327] 0.20x 22.77x
1.18.1, DECIMAL(19,3), false, binary, 1024 25.869 [24.083, 45.725] 60.425 [53.040, 101.635] 1.488 [1.422, 1.619] 17.39x 40.62x
1.18.1, DECIMAL(19,3), true, generic, 1024 17.958 [16.544, 35.588] 120.014 [115.727, 146.039] 2.097 [2.040, 2.342] 8.56x 57.24x
1.18.1, DECIMAL(19,3), true, binary, 16 18.345 [17.713, 19.267] 584.371 [578.275, 610.158] 25.778 [25.321, 27.777] 0.71x 22.67x
1.18.1, DECIMAL(38,18), false, binary, 1024 34.512 [21.738, 44.704] 64.531 [43.767, 72.120] 1.458 [1.367, 1.554] 23.67x 44.26x
1.18.1, DECIMAL(38,18), true, generic, 1024 7.675 [7.338, 7.944] 41.761 [39.220, 59.480] 2.070 [1.995, 2.111] 3.71x 20.17x
1.18.1, DECIMAL(38,18), true, binary, 16 18.198 [16.834, 23.837] 576.080 [568.844, 607.250] 26.079 [25.583, 26.886] 0.70x 22.09x
1.18.1, DECIMAL(38,38), false, binary, 1024 30.949 [22.371, 51.205] 51.428 [45.383, 66.168] 1.465 [1.390, 1.631] 21.13x 35.11x
1.18.1, DECIMAL(38,38), true, generic, 1024 7.643 [7.118, 8.465] 42.615 [39.193, 86.474] 2.089 [2.033, 2.347] 3.66x 20.40x
1.18.1, DECIMAL(38,38), true, binary, 16 18.907 [17.902, 19.407] 581.264 [573.115, 596.482] 27.077 [26.356, 28.397] 0.70x 21.47x
2.2.1, DECIMAL(18,3), false, binary, 1024 8.586 [7.102, 8.956] 90.221 [35.983, 179.781] 1.551 [1.473, 1.753] 5.54x 58.18x
2.2.1, DECIMAL(18,3), true, generic, 1024 6.987 [6.547, 7.708] 51.341 [36.867, 91.749] 2.126 [2.028, 2.342] 3.29x 24.15x
2.2.1, DECIMAL(18,3), true, binary, 16 6.000 [5.145, 6.451] 620.616 [588.384, 971.621] 25.933 [25.349, 34.006] 0.23x 23.93x
2.2.1, DECIMAL(19,3), false, binary, 1024 23.132 [21.831, 42.864] 46.033 [42.233, 72.575] 1.542 [1.408, 1.836] 15.00x 29.85x
2.2.1, DECIMAL(19,3), true, generic, 1024 9.666 [8.183, 11.278] 44.669 [40.233, 67.673] 2.143 [2.030, 2.563] 4.51x 20.84x
2.2.1, DECIMAL(19,3), true, binary, 16 19.288 [18.641, 25.727] 585.044 [571.290, 591.088] 27.367 [26.751, 28.982] 0.70x 21.38x
2.2.1, DECIMAL(38,18), false, binary, 1024 23.328 [22.225, 42.695] 48.661 [43.836, 67.923] 1.458 [1.418, 1.911] 15.99x 33.36x
2.2.1, DECIMAL(38,18), true, generic, 1024 8.416 [7.359, 14.885] 51.033 [42.275, 80.820] 2.165 [2.050, 2.892] 3.89x 23.57x
2.2.1, DECIMAL(38,18), true, binary, 16 24.755 [18.713, 33.524] 585.119 [570.680, 1058.025] 27.826 [26.473, 78.137] 0.89x 21.03x
2.2.1, DECIMAL(38,38), false, binary, 1024 23.077 [21.238, 45.155] 47.891 [43.574, 75.425] 1.476 [1.308, 1.616] 15.63x 32.44x
2.2.1, DECIMAL(38,38), true, generic, 1024 8.402 [7.579, 13.434] 57.577 [43.598, 75.661] 2.181 [2.028, 2.316] 3.85x 26.39x
2.2.1, DECIMAL(38,38), true, binary, 16 19.016 [18.543, 19.412] 599.046 [577.167, 607.373] 25.196 [24.239, 26.623] 0.75x 23.78x

At 1,024 rows the candidate is 2.43-23.67x faster than stock and 18.57-58.18x faster than the preceding callback. Sixteen-row controls still lose to stock (0.20-0.89x), while improving the previous native-Calc route by 21.03-23.93x. These are sampled prepared-batch gains; they do not establish universal small-batch acceleration. The nullable 1,024-row allocation probe records 20 allocations / 17,575 requested bytes, with 16,384 new value bytes and 128 shared validity bytes. Generic Calc/batch/result handling retains fixed costs alongside the allocation-free per-element transform. Remaining small-batch and row/Arrow perimeter work stays in #491.

The independent complete job uses 1M DECIMAL(18,3) rows, parallelism one, a bounded row source, both transposes and a rowwise blackhole sink. The previous control loads the actual d89e03b5 encoder ahead of candidate classes and requires its JVM route; candidate requires jvm=[]. Both require positive consumed NativeCalc work. Each engine runs two warmups and five measured jobs in separate sequential JVMs. Seconds are median [min, max]:

Stock Previous callback Candidate
0.131800 [0.119555, 0.133355] 0.174760 [0.167390, 0.180106] 0.150979 [0.145502, 0.164335]

The complete candidate improves previous StreamFusion by 13.6% but remains 14.6% slower than stock by median in this workload. Startup, submission, JIT, source conversion and both transposes remain included; their individual contributions are not isolated here. The prepared-batch gain is not a whole-job acceleration claim. The result record retains every trial, checksum, environment and allocation observation.

Use the shared release example with SF_DECIMAL_UNARY_FUNCTION=NEGATE. Complete-job controls select -Ddecimal.unary.pipeline.function=NEGATE and the engine property above. Previous requires the baseline encoder first on the classpath; relabeling an unchanged candidate is rejected. Runtime regressions cover precision/scale endpoints, extrema, NULL/non-NULL data, widening/nesting, string/Boolean consumers, unselected failures and empty filtered batches, with positive consumed NativeCalc work on both Flink lines and JDK 11/17. Rust/encoding tests cover every legal decimal precision/scale pair. Unchanged released ScalarOperatorsTest.testUnaryPlusMinus and DecimalTypeTest.testUnaryPlusMinus fixtures pass on both lines.

External STRUCTURED collection inputs

Source STRUCTURED values now reuse the existing generated collection evaluator and represented ROW boundary, including nested array/map/multiset containers and ROW/STRUCTURED casts. NativeCalc consumption establishes this route; these Java casts are not described as Rust kernels. Coverage gives the exact built-in-program, callback, state/order and writer limits.

The 2026-10-28 release/mimalloc comparison uses Flink 2.2.1/1.18.1, OpenJDK 17.0.20.1, Linux x86-64/WSL2, parallelism one and an 8 GiB heap. Every job has 50,000 rowwise source records and a rowwise blackhole sink, with two warmups and three alternating trials per engine. The source array elements remain declared STRUCTURED with nullable INT/string fields. Widths are 3/16, parent NULLs occur every eighth row, element NULLs every seventh position, and INT/string field NULLs every eleventh/thirteenth row. Strings use 16 seeds plus a 32-character suffix. The single projection constructs a ROW containing both an ARRAY<ROW> cast and character formatting of the original array; it measures their combined production boundary.

Candidate and actual previous 86c9d244 use identical fixture sources and the same frozen native release library. The previous planner must fall back without NativeCalc. Each candidate native job requires 50,000 consumed NativeCalc inputs and jvm=[ROW, CAST] descriptors. Both perimeter transposes, JNI, generated evaluation and planning/submission remain timed; environment setup and the initial explain probe remain outside timing. Local exchange caching stays disabled. No competing Java/native build, test or benchmark process was observed by the process monitor.

All 80 jobs, 20 native Calc witnesses and paired controls retain source/library hashes, raw trials, commands and ranges. Median seconds (min–max):

Flink Array width Previous paired stock Previous fallback Candidate paired stock NativeCalc
2.2 3 0.194330 (0.170898–0.262270) 0.191025 (0.175968–0.222784) 0.177382 (0.175108–0.182358) 0.384402 (0.365160–0.412428)
2.2 16 0.370867 (0.356476–0.390086) 0.372421 (0.365365–0.372573) 0.383198 (0.380443–0.386313) 0.735325 (0.727280–0.855846)
1.18 3 0.178257 (0.174778–0.197603) 0.183170 (0.182411–0.217055) 0.184813 (0.178814–0.220786) 0.382043 (0.360289–0.391352)
1.18 16 0.375509 (0.362542–0.478923) 0.378216 (0.367642–0.418500) 0.383667 (0.374440–0.396060) 0.779504 (0.709350–0.798988)

Every native median loses to both stock and the preceding fallback. The new route is roughly twice paired stock on these workloads. This ships as correctness-verified collection coverage under the existing accepted collection-extension tradeoff, without an acceleration claim or an additional performance-only switch. Standalone optimization remains in #392.

Exactly two eight-second CPU captures on Flink 2.2, one at each width, close the bounded investigation. Each has one unprofiled native warmup followed by native-only complete jobs; profiled timings are excluded from the table. Width 3/16 captures have 20,683/15,380 execution samples. Row serialization appears in 349/800 samples, Arrow field writing in 476/789, generated evaluation in 385/513 and the existing JVM batch callback in 309/374. Categories are inclusive and overlap. They include JIT, job setup and all process threads, and do not measure allocations, copied bytes or isolated kernel costs. No small safe change to this existing serialization/representation boundary was selected. The report retains capture hashes, scope and the remaining conversion/serialization costs; no third tuning pass follows.

Reproduce the complete-job comparison after building the release library:

SF_BENCHMARK=true mvn -pl streamfusion-runtime test -Pbench \
  '-Dtest=DynamicCollectionBenchmark#compareExpressions' \
  '-Dcollection.expression=ROW(CAST(arr AS ARRAY<ROW<a BIGINT,b STRING>>),CAST(arr AS STRING))' \
  -Dcollection.map=false -Dcollection.type=STRUCTURED -Dcollection.width=3 \
  -Dcollection.rows=50000 -Dcollection.bytes=32 -Dcollection.domain=16 \
  -Dcollection.warmup=2 -Dcollection.runs=3 -Dcollection.native=true -Dsf.testForks=1

Repeat at width 16 and add flink-1.18 to the profile list for the second released line. The previous checkout copies the identical benchmark fixture, sets collection.native=false and loads the same release library. For the CPU diagnostic, additionally set SF_PROFILE_COLLECTION=true, select DynamicCollectionBenchmark#profileNativeExpressions and set collection.profileSeconds=12. After its collection-profile-ready message, attach asprof -d 8 -e cpu -i 1ms -f structured-width3.jfr PID; repeat once at width 16. The diagnostic requires positive native work in every job and does not supply timing results for comparison.

DECIMAL SIGN

The typed Decimal128 kernel replaces the scalar generated callback for verified direct/literal, same-scale lossless-widening and nested SIGN inputs. Ordinary p > s returns scaled -1/0/1; noncompact p = s returns NULL for nonzero values and scaled zero otherwise. Compact p = s remains generated because its out-of-precision internal value has consumer-specific behavior. DECIMAL CASE/IF containing that compact intermediate retains the released branch conversion. This bounded implementation does not complete #491.

Prepared operation

Apple M4 Pro, macOS aarch64, JDK 17.0.12, Rust 1.94.0, release/no-default-features, system allocator with the counting wrapper; released Flink 1.18.1 and 2.2.1. Each trial consumes 65,536 logical rows: 64 batches of 1,024 or 4,096 batches of 16. Each configuration has 16 warmups and 11 measured trials, rotating stock/previous/native order. Data include signed extremes, values beyond 2^53, actual zeros every eleventh row, and NULL every seventh row in the nullable-value configurations. All logical schemas are nullable; the NULL-values column below describes actual values. GenericRowData and BinaryRowData stock formats are both retained.

Stock starts in Java on prepared RowData, evaluates the released generated projection and consumes every decimal result. Previous and candidate start in Rust on equivalent prepared Arrow batches, run the production Calc and consume each output through an equivalent checksum. Their timers start and stop inside Rust. Setup, Arrow import/export and perimeter transposes are outside this small-operation timer. The previous control uses the actual main d89e03b5 kind-17 scalar evaluator with one decimal argument and the matching precision/scale result code; its JVM callback and both intrinsic decimal marshalling boundaries stay inside timing. The baseline encoding is independently checked on both released lines. The candidate asserts opcode 65 and empty JVM descriptors. Before timing, every candidate Arrow output and schema is compared exactly with the preceding evaluator; each timed engine also checks its checksum.

Medians and observed min/max are ns/row. Ratios above 1 favor native.

Flink, type, NULL values, stock format, batch rows Stock ns/row Previous ns/row Native ns/row vs stock vs previous
1.18.1, DECIMAL(18,3), false, binary, 1024 8.788 [7.560, 32.969] 31.644 [29.802, 35.288] 1.726 [1.422, 1.775] 5.09x 18.34x
1.18.1, DECIMAL(18,3), true, generic, 1024 5.188 [5.105, 5.262] 39.490 [37.286, 64.526] 2.113 [2.028, 2.317] 2.45x 18.69x
1.18.1, DECIMAL(18,3), true, binary, 16 5.563 [4.834, 6.092] 573.643 [558.766, 590.951] 25.682 [23.939, 28.564] 0.22x 22.34x
1.18.1, DECIMAL(19,19), false, binary, 1024 36.277 [33.051, 60.411] 49.828 [45.473, 70.459] 3.698 [3.549, 3.969] 9.81x 13.47x
1.18.1, DECIMAL(19,19), true, generic, 1024 17.546 [16.832, 29.591] 50.177 [47.459, 73.354] 3.915 [3.748, 3.991] 4.48x 12.82x
1.18.1, DECIMAL(19,19), true, binary, 16 27.656 [27.335, 34.155] 588.833 [575.139, 716.778] 29.204 [28.835, 30.251] 0.95x 20.16x
1.18.1, DECIMAL(19,3), false, binary, 1024 31.000 [28.632, 49.302] 53.205 [44.348, 117.357] 1.711 [1.563, 1.806] 18.12x 31.10x
1.18.1, DECIMAL(19,3), true, generic, 1024 9.205 [8.858, 10.038] 44.694 [41.683, 68.454] 2.144 [1.930, 2.423] 4.29x 20.85x
1.18.1, DECIMAL(19,3), true, binary, 16 19.688 [19.352, 20.637] 592.555 [583.083, 611.048] 25.354 [24.656, 26.271] 0.78x 23.37x
1.18.1, DECIMAL(38,18), false, binary, 1024 28.429 [27.130, 49.131] 46.403 [43.915, 85.672] 1.654 [1.457, 1.750] 17.19x 28.06x
1.18.1, DECIMAL(38,18), true, generic, 1024 8.948 [8.448, 26.579] 42.668 [40.415, 60.392] 2.248 [2.197, 2.358] 3.98x 18.98x
1.18.1, DECIMAL(38,18), true, binary, 16 20.622 [19.639, 25.847] 585.923 [563.824, 598.157] 23.598 [23.005, 24.824] 0.87x 24.83x
1.18.1, DECIMAL(38,38), false, binary, 1024 43.238 [34.046, 66.534] 52.330 [45.084, 97.438] 3.705 [3.527, 3.743] 11.67x 14.12x
1.18.1, DECIMAL(38,38), true, generic, 1024 19.147 [18.496, 42.937] 72.809 [54.204, 98.392] 3.871 [3.808, 4.006] 4.95x 18.81x
1.18.1, DECIMAL(38,38), true, binary, 16 29.189 [28.351, 33.110] 576.757 [569.712, 624.387] 31.391 [30.214, 32.533] 0.93x 18.37x
2.2.1, DECIMAL(18,3), false, binary, 1024 5.978 [5.406, 34.654] 38.769 [30.835, 45.469] 1.563 [1.343, 1.754] 3.82x 24.80x
2.2.1, DECIMAL(18,3), true, generic, 1024 7.204 [6.912, 7.796] 32.211 [29.266, 59.492] 2.220 [1.803, 2.257] 3.24x 14.51x
2.2.1, DECIMAL(18,3), true, binary, 16 5.570 [5.396, 6.739] 557.027 [543.966, 583.133] 25.658 [24.828, 26.042] 0.22x 21.71x
2.2.1, DECIMAL(19,19), false, binary, 1024 35.250 [34.651, 74.614] 76.201 [48.690, 124.938] 3.799 [3.780, 3.902] 9.28x 20.06x
2.2.1, DECIMAL(19,19), true, generic, 1024 15.188 [14.617, 34.014] 71.803 [50.053, 157.637] 3.953 [3.872, 4.104] 3.84x 18.17x
2.2.1, DECIMAL(19,19), true, binary, 16 28.546 [27.047, 32.573] 590.797 [561.872, 627.825] 29.456 [28.050, 30.531] 0.97x 20.06x
2.2.1, DECIMAL(19,3), false, binary, 1024 27.291 [26.423, 46.593] 45.663 [42.999, 72.019] 1.730 [1.412, 1.883] 15.78x 26.40x
2.2.1, DECIMAL(19,3), true, generic, 1024 8.733 [7.748, 11.492] 46.698 [41.372, 63.026] 2.172 [1.893, 2.428] 4.02x 21.50x
2.2.1, DECIMAL(19,3), true, binary, 16 20.555 [19.496, 23.326] 574.708 [558.461, 636.387] 24.882 [23.509, 30.155] 0.83x 23.10x
2.2.1, DECIMAL(38,18), false, binary, 1024 28.486 [27.484, 49.393] 51.687 [46.701, 74.498] 1.746 [1.603, 1.935] 16.32x 29.61x
2.2.1, DECIMAL(38,18), true, generic, 1024 9.229 [8.596, 14.255] 49.003 [42.676, 66.046] 2.261 [2.113, 2.501] 4.08x 21.67x
2.2.1, DECIMAL(38,18), true, binary, 16 20.540 [19.787, 26.115] 606.761 [588.124, 715.747] 24.800 [23.598, 25.590] 0.83x 24.47x
2.2.1, DECIMAL(38,38), false, binary, 1024 34.427 [32.860, 68.352] 52.185 [46.946, 90.398] 3.644 [3.595, 3.922] 9.45x 14.32x
2.2.1, DECIMAL(38,38), true, generic, 1024 15.423 [14.765, 34.475] 46.836 [43.168, 89.345] 3.973 [3.814, 4.130] 3.88x 11.79x
2.2.1, DECIMAL(38,38), true, binary, 16 30.945 [28.578, 46.801] 596.071 [584.038, 614.315] 30.892 [29.867, 31.805] 1.00x 19.30x

At 1,024 rows the candidate is 2.45–18.12x stock and 11.79–31.10x previous across these configurations. At 16 rows it is 0.22–1.00x stock, although 18.37–24.83x previous. Small-batch Calc/result overhead remains in #491; these controls do not establish a stock speedup. The allocation probe records 20 calls for an ordinary SIGN Calc at either batch size, with 256 new output value-buffer bytes at 16 rows versus 16,384 at 1,024. Noncompact p=s also derives a new validity bitmap (22 recorded allocation calls). These are per-invocation observations, not a complete causal CPU profile or an allocation-free execution claim.

Complete row-source job

Flink 2.2.1, parallelism one, one million nullable DECIMAL(18,3) values with mixed signs, zero and extremes, rowwise blackhole sink. Each engine has two warmups and five measured jobs in separate sequential JVMs. Previous uses the unchanged main encoder; native plans require both RowDataToArrow and ArrowToRowData and positive consumed NativeCalc work. Timings include submission, full source/sink processing and both perimeter transposes.

Engine Median seconds Observed min/max seconds
Stock Flink 0.135526792 0.125703833–0.148966750
Previous scalar callback 0.174364125 0.168311458–0.186801792
Typed native SIGN 0.151711958 0.141338042–0.159535541

The candidate is 0.89x stock and 1.15x previous by these medians. It still loses to stock in this standalone job, with overlapping stock/native ranges; the prepared-kernel gain is not an end-to-end acceleration claim. Both transposes remain a material pipeline cost.

Validation and reproduction

The focused matrix passes 324 JVM test invocations across both Flink lines and JDK 11/17, including NULL/NOT NULL results, released sink failures, consumed native batches, compact character and DECIMAL conditional consumers, integer/floating SIGN controls, rounding metadata, unselected failures and all-filtered batches. A released-method oracle separately checks 2,337 values and precision/scale pairs per Flink line, including compact and noncompact p=s. The unchanged released ScalarFunctionsTest.testSign passes on both lines as a result oracle; separate runtime-source witnesses establish native execution over 5,003 rows per admitted profile. Rust tests cover every supported precision/scale, scalar/array results, slices, empty/all-NULL inputs and input ownership; the release core suite passes 834 tests.

The helper is DecimalSignBenchmarkFunctions, the full job is DecimalSignPipelineBenchmark, and all measured trials/allocation probes are in the raw results. With the ordinary Maven test classpath in SF_NATIVE_BENCH_CLASSPATH, JDK 17 in JAVA_HOME, and SF_NATIVE_LIB_DIR pointing at the release library directory:

cargo build --manifest-path native/Cargo.toml -p streamfusion --release \
  --no-default-features --lib --example decimal_sign
native/target/release/examples/decimal_sign 18 3 true false 1024 64 16 11

Use 19 19 or 38 38 for noncompact zero-integer-width controls; 16 4096 as the batch/loop arguments retains the same row count for the small control. The pipeline is opt-in: SF_BENCHMARK=true mvn test -Pbench -pl streamfusion-core -Dtest=DecimalSignPipelineBenchmark with -Ddecimal.sign.pipeline.engine=flink or native; previous requires the independently compiled baseline encoder at the front of the classpath. Defaults retain one million rows, two warmups and five measured runs. Do not compare a debug library with these release results.

Fixed binary Base64 encoding

Measured on 2026-10-28 against canonical main d89e03b5, with released Flink 1.18.1 and 2.2.1, JDK 17, Apple M4 Pro, Rust 1.94.0 and release/no-default-features native code using the system allocator. The fixed-BINARY overload now uses the same byte encoder as VARBINARY. Both Arrow Binary and FixedSizeBinary are consumed directly without a preliminary byte copy. Legacy actual byte lengths remain unchanged; casts own padding/truncation.

Stock runs the released generated expression over prepared GenericRowData or BinaryRowData. Previous and candidate production Calc start on identical prepared Arrow values/types/NULLs, with timers inside Rust. The previous control retains the actual main scalar callback (kind 17, BINARY argument and STRING result), including its intrinsic marshalling. Setup, initial JNI import/export and perimeter transposes are excluded from this small-function control. Full schema/Arrow output equality is checked before timing; each trial consumes equal length/first/last-byte checksums. All schemas are nullable; enabled NULL values occur every seventh row. Sixteen warmups and eleven measured 65,536-row trials rotate engine order. Configurations execute serially, without concurrent tests or builds. Values are median [min, max] ns/row:

Flink, bytes, NULL values, stock format, Arrow type, batch rows Stock ns/row Previous ns/row Native ns/row vs stock vs previous
1.18.1, 16, false, binary, FixedSizeBinary, 1024 25.670 [24.779, 47.560] 84.452 [78.779, 115.376] 7.149 [7.073, 8.103] 3.59x 11.81x
1.18.1, 16, true, generic, Binary, 1024 27.252 [16.367, 42.290] 97.043 [76.092, 127.300] 6.220 [5.834, 6.392] 4.38x 15.60x
1.18.1, 16, true, binary, FixedSizeBinary, 16 26.383 [23.383, 36.641] 612.534 [604.555, 641.434] 35.286 [34.123, 36.072] 0.75x 17.36x
1.18.1, 2, false, binary, FixedSizeBinary, 1024 18.431 [17.228, 34.025] 59.009 [51.335, 86.955] 6.527 [6.195, 6.877] 2.82x 9.04x
1.18.1, 2, true, generic, Binary, 1024 12.836 [10.766, 26.017] 57.096 [48.562, 76.233] 5.573 [4.941, 6.486] 2.30x 10.25x
1.18.1, 2, true, binary, FixedSizeBinary, 16 14.723 [13.955, 18.323] 566.812 [554.206, 580.385] 32.871 [32.306, 34.672] 0.45x 17.24x
1.18.1, 256, false, binary, FixedSizeBinary, 1024 45.441 [43.418, 72.034] 182.394 [172.059, 195.187] 30.001 [29.427, 30.876] 1.51x 6.08x
1.18.1, 256, true, generic, Binary, 1024 29.374 [28.728, 54.389] 164.123 [153.769, 187.140] 25.521 [25.085, 26.051] 1.15x 6.43x
1.18.1, 256, true, binary, FixedSizeBinary, 16 37.827 [35.375, 40.659] 735.647 [717.014, 813.629] 47.815 [47.017, 50.003] 0.79x 15.39x
2.2.1, 16, false, binary, FixedSizeBinary, 1024 26.119 [24.993, 43.158] 91.498 [83.882, 115.524] 7.357 [7.231, 7.710] 3.55x 12.44x
2.2.1, 16, true, generic, Binary, 1024 16.402 [15.439, 34.462] 77.600 [71.830, 98.326] 6.156 [5.679, 6.345] 2.66x 12.61x
2.2.1, 16, true, binary, FixedSizeBinary, 16 24.954 [22.902, 29.020] 610.266 [605.935, 625.162] 36.786 [35.926, 37.446] 0.68x 16.59x
2.2.1, 2, false, binary, FixedSizeBinary, 1024 18.450 [17.279, 34.189] 52.354 [50.027, 75.937] 6.508 [6.390, 6.891] 2.83x 8.04x
2.2.1, 2, true, generic, Binary, 1024 10.519 [10.345, 13.292] 53.988 [48.670, 71.146] 5.329 [4.779, 5.461] 1.97x 10.13x
2.2.1, 2, true, binary, FixedSizeBinary, 16 14.802 [13.870, 20.783] 556.320 [547.763, 559.575] 32.548 [31.918, 32.867] 0.45x 17.09x
2.2.1, 256, false, binary, FixedSizeBinary, 1024 43.749 [42.463, 53.862] 179.748 [170.487, 216.596] 29.687 [27.475, 30.474] 1.47x 6.05x
2.2.1, 256, true, generic, Binary, 1024 30.261 [28.229, 40.974] 158.586 [156.895, 174.400] 26.091 [25.161, 26.950] 1.16x 6.08x
2.2.1, 256, true, binary, FixedSizeBinary, 16 36.395 [35.843, 41.303] 742.716 [732.299, 774.394] 47.683 [46.579, 48.219] 0.76x 15.58x

The 1,024-row configurations improve 1.15-4.38x over stock and 6.05-15.60x over the preceding callback. Sixteen-row controls remain slower than stock (0.45-0.79x) while improving 15.39-17.36x over the preceding callback: batch setup/allocation is significant for tiny inputs. These controls are retained and do not establish universal acceleration.

The initial scalar codec lost to stock on 256-byte inputs. A native CPU sample placed 1,864 of 2,442 main-thread samples in that codec. The released SIMD encoder removed much of that cost, after which redundant UTF-8 validation still accounted for 413 of 2,433 samples. Checked offsets, exact-size initialized row buffers and the encoder's ASCII-only output permit omitting that rescan. Runtime CPU dispatch retains a portable fallback; regression tests compare against the independent standard codec and fully validate Arrow outputs at vector/padding boundaries, slices, NULLs and empty batches. See the allocation technique.

The separate Flink 2.2.1 complete job uses a bounded 1M-row BINARY(16) source, parallelism one, both row/Arrow transposes and a rowwise blackhole sink. Candidate requires jvm=[] and positive consumed NativeCalc work. The previous control loads main's actual encoder first on the classpath and requires its JVM evaluator. Each engine runs two warmups and five measured jobs in separate sequential JVMs. Seconds are median [min, max]:

Stock Previous callback Candidate
0.186338 [0.175739, 0.196376] 0.270293 [0.265321, 0.280235] 0.170568 [0.162968, 0.171469]

This workload reduces elapsed time by about 8.5% versus stock and 36.9% versus previous StreamFusion. Startup, submission, JIT and both transposes remain included; these subsecond jobs do not establish sustained throughput for arbitrary pipelines. The result record retains all trials, checksums, allocation observations and the measured intermediate codec controls.

After compiling test helpers, set SF_NATIVE_BENCH_CLASSPATH to test/production classes and released Maven dependencies, and SF_NATIVE_LIB_DIR to the release library directory. Build/run the prepared control from native/:

cargo build --release -p streamfusion --no-default-features --example fixed_binary_base64
./target/release/examples/fixed_binary_base64 16 true false false 1024 64 16 11

Arguments are byte width, actual NULL values, BinaryRowData stock input, FixedSizeBinary native input, batch rows, loops, warmups and measured trials. Complete jobs use FixedBinaryBase64PipelineBenchmark, SF_BENCHMARK=true and -Dfixed.binary.base64.pipeline.engine=flink|previous|native; previous requires the baseline encoder first on the classpath. Both released lines and JDK 11/17 retain runtime parity and consumed native-work witnesses. The subsequent FROM_BASE64 decoder is measured separately.

FROM_BASE64

Measured on 2026-10-28 with canonical main 5dce7f04, released Flink 1.18.1/2.2.1, JDK 17, Apple M4 Pro, Rust 1.94.0 and release/no-default-features native code using the system allocator. The previous implementation is PR #549's preceding commit ea0268e5. Only direct final projections enter the native decoder; the sensitive-string consumer and relational restrictions in the Calc coverage page remain unchanged.

Stock generated evaluation starts on prebuilt GenericRowData or BinaryRowData. The native and preceding production Calc start on equivalent prepared Arrow buffers, with their timers inside Rust. Initial import/export and perimeter transposes are outside this small-function measurement. The preceding encoder is compiled independently and loaded first on the classpath: STRING asserts its actual kind-44 original-string evaluator, BYTES asserts kind 17, and both return raw Arrow Binary. The candidate asserts kind 6/opcode 230. Intrinsic callbacks and argument/result marshalling remain timed in the preceding implementation.

Each encoder runs in its own sequential JVM with a stock control and rotating two-engine order. Both stock controls are retained: their JIT/GC variability is visible, and the speedup column conservatively uses the smaller of their medians. Each process uses sixteen warmups and eleven measured trials of 65,536 consumed rows. Output length and first/last-byte checksums must match. Values contain deterministic arbitrary bytes before encoding; nullable schemas include actual NULLs every seventh row when enabled. No tests or builds run concurrently with measurements. Width is decoded byte length; reported values are median [min, max] ns/row:

Flink, bytes, NULLs, SQL argument, stock format, batch rows Stock/current process Stock/previous process Previous Native vs stock, conservative vs previous
1.18.1, 16, false, BYTES, binary, 1024 26.901 [26.489, 40.339] 31.277 [27.017, 49.859] 62.787 [56.459, 83.304] 8.784 [8.691, 9.063] 3.06x 7.15x
1.18.1, 16, true, STRING, generic, 1024 30.517 [28.866, 61.686] 36.214 [34.405, 64.553] 74.523 [66.104, 90.952] 8.684 [8.558, 9.497] 3.51x 8.58x
1.18.1, 16, true, BYTES, binary, 16 22.761 [22.219, 36.091] 23.008 [22.319, 28.255] 601.892 [596.954, 617.264] 34.293 [33.745, 34.853] 0.66x 17.55x
1.18.1, 2, false, BYTES, binary, 1024 24.943 [22.899, 37.237] 23.944 [22.483, 46.245] 48.325 [43.325, 71.232] 6.770 [5.990, 7.114] 3.54x 7.14x
1.18.1, 2, true, STRING, generic, 1024 41.164 [27.367, 57.690] 27.501 [27.240, 56.787] 66.313 [60.933, 105.553] 6.821 [6.140, 7.190] 4.03x 9.72x
1.18.1, 2, true, BYTES, binary, 16 20.885 [19.041, 32.747] 19.783 [19.346, 23.593] 560.979 [554.251, 586.925] 32.129 [31.499, 32.871] 0.62x 17.46x
1.18.1, 256, false, BYTES, binary, 1024 65.030 [63.217, 75.474] 77.693 [68.219, 116.290] 104.519 [98.938, 123.337] 33.895 [33.569, 34.896] 1.92x 3.08x
1.18.1, 256, true, STRING, generic, 1024 73.407 [68.149, 111.362] 75.641 [73.624, 103.769] 126.862 [118.484, 181.338] 31.077 [29.801, 33.537] 2.36x 4.08x
1.18.1, 256, true, BYTES, binary, 16 52.860 [51.834, 65.730] 52.813 [52.046, 57.954] 647.031 [641.312, 659.692] 53.922 [53.396, 54.974] 0.98x 12.00x
2.2.1, 16, false, BYTES, binary, 1024 35.700 [27.991, 54.179] 29.100 [27.502, 46.365] 57.854 [51.655, 78.010] 8.886 [8.677, 9.521] 3.27x 6.51x
2.2.1, 16, true, STRING, generic, 1024 39.433 [29.179, 43.844] 33.691 [31.814, 55.526] 71.623 [66.146, 91.021] 8.807 [8.603, 9.258] 3.83x 8.13x
2.2.1, 16, true, BYTES, binary, 16 29.550 [22.308, 40.774] 23.141 [21.571, 27.645] 600.082 [579.973, 628.577] 34.034 [33.515, 35.280] 0.68x 17.63x
2.2.1, 2, false, BYTES, binary, 1024 43.046 [24.864, 45.284] 24.703 [23.446, 43.276] 46.487 [43.537, 66.571] 6.715 [5.974, 6.949] 3.68x 6.92x
2.2.1, 2, true, STRING, generic, 1024 29.752 [26.725, 40.999] 28.487 [26.877, 39.600] 64.948 [60.902, 84.585] 6.152 [6.048, 6.826] 4.63x 10.56x
2.2.1, 2, true, BYTES, binary, 16 36.005 [18.298, 41.290] 19.275 [18.974, 24.094] 557.683 [550.521, 575.581] 34.298 [31.134, 36.540] 0.56x 16.26x
2.2.1, 256, false, BYTES, binary, 1024 75.551 [63.354, 113.574] 68.417 [64.597, 81.800] 102.013 [99.084, 115.237] 34.513 [33.751, 35.754] 1.98x 2.96x
2.2.1, 256, true, STRING, generic, 1024 81.366 [67.916, 121.744] 73.105 [67.038, 89.697] 121.209 [114.834, 145.936] 30.738 [29.545, 31.277] 2.38x 3.94x
2.2.1, 256, true, BYTES, binary, 16 52.263 [51.912, 64.269] 58.423 [52.698, 60.040] 665.132 [654.268, 689.308] 57.742 [57.051, 60.036] 0.91x 11.52x

The 1,024-row configurations improve 1.92-4.63x over stock and 2.96-10.56x over the preceding scalar path. Sixteen-row controls improve over the preceding callback but remain slower than stock (0.56-0.98x using the conservative stock median). Calc/batch allocation and dispatch are significant at tiny sizes; these results do not establish a universal speedup. The implementation sizes checked offsets, writes directly into final Binary buffers and decodes complete groups with runtime SIMD dispatch. A short scalar tail preserves the JDK's accepted nonzero unused bits; invalid-input diagnostics stay in Rust. See the allocation technique.

The separate Flink 2.2.1 complete job uses one million encoded STRING rows, representing 16 arbitrary decoded bytes, parallelism one, a bounded row source and a rowwise blackhole sink. Source encoding, submission, JIT, JNI and both row/Arrow transposes remain included. Candidate requires jvm=[], both transposes and positive consumed NativeCalc work. The previous control loads the actual preceding encoder and requires the FROM_BASE64 evaluator. Each engine runs two warmups and five measured jobs in a separate sequential JVM. Seconds are median [min, max]:

Stock Previous callback Native
0.243187 [0.235530, 0.244118] 0.344742 [0.341944, 0.353287] 0.210599 [0.201220, 0.212980]

Elapsed time falls about 13.4% versus stock and 38.9% versus previous StreamFusion in this bounded workload. These subsecond jobs include startup costs and do not establish sustained throughput for other pipelines. The result record retains both stock controls, every timing trial, checksums and untimed allocation observations.

Validation includes 542 focused JVM invocations across Flink 1.18.1/2.2.1 and JDK 11/17, 841 release Rust core tests, and the unchanged released testFromBase64/testToBase64 fixtures on both lines. A deterministic 46,443-case oracle covers short encodings, missing and malformed padding, unused bits, nonalphabet/signed bytes and mutated longer encodings. Both Flink overloads on both JDKs/versions agree, and the production Rust scalar matches every case. SQL tests require consumed native work and no JVM evaluator for STRING, CHAR, BYTES and fixed-BINARY inputs. Raw decoded bytes, nullable slices, vector boundaries, legacy casts, all-filtered batches, lazy branches and retained consumer/sink restrictions remain covered.

After compiling FromBase64BenchmarkFunctions, set SF_NATIVE_BENCH_CLASSPATH to test and production classes plus released dependencies, and SF_NATIVE_LIB_DIR to the release library directory. Build and run from native/:

cargo build --release --locked -p streamfusion --no-default-features --example from_base64
./target/release/examples/from_base64 16 true false false 1024 64 16 11
java -cp "$SF_NATIVE_BENCH_CLASSPATH" tech.streamfusion.planner.FromBase64BenchmarkFunctions > oracle.tsv
./target/release/examples/from_base64 --oracle oracle.tsv

Prepared arguments are decoded width, NULL values, binary SQL argument, BinaryRowData stock input, batch rows, loops, warmups and measured trials. The previous control sets SF_FROM_BASE64_PREVIOUS=true and loads the independently compiled preceding encoder first on the classpath. Complete jobs use FromBase64PipelineBenchmark, SF_BENCHMARK=true and -Dfrom.base64.pipeline.engine=flink|previous|native, with the same previous-classpath rule.

Runtime SQL TRIM

The sql_trim release driver compares SQL TRIM in all three directions using identical nullable values on prebuilt Java RowData and Arrow batches. Stock Flink and the previous StreamFusion route are the same generated Flink operation: runtime SQL trim previously caused planner fallback. A third generated batch-callback measurement is diagnostic only; it is not labelled as the previous route. All three consume output byte lengths.

The native timer runs inside Rust around production compiled Calc, including the ASCII/cache certificate, set membership, output allocation and consumption. Setup, initial JNI/Arrow import and final export are outside this prepared-function timer. Java times the released generated expression and equivalent consumption on its prepared rows. Both retain nullable strings and the original decoded Java caches. The native driver uses an invalid evaluator id for admitted ASCII controls, so an unexpected callback fails instead of silently being counted as native computation. Unicode controls retain the exact evaluator.

Measurements on Apple M4 Pro/24 GiB, JDK 17.0.12 and Rust 1.95.0 use released Flink 1.18.1/2.2.1, release Rust with mimalloc, Arrow 58.4.1 and DataFusion 54.0.0. Each profile runs eight warmups and nine measured trials of 128 batches, sequentially without concurrent tests or builds. Fresh JVMs use 256/512 MiB initial/maximum heaps. Four runtime trim sets (ab, ab, tab-plus-space and []+) provide cardinality four. Source NULLs occur every seventh row, trim-set NULLs every eleventh row. Width is the middle payload in bytes; each source also contains the trim set on both ends.

Median [minimum-maximum] ns/row:

Flink Payload bytes / batch rows Direction Stock / previous fallback Native Stock/native
1.18.1 8 / 16 BOTH 184.31 [162.33-334.27] 56.97 [54.38-58.59] 3.24x
1.18.1 8 / 16 LEADING 102.86 [96.42-157.27] 56.82 [53.79-57.64] 1.81x
1.18.1 8 / 16 TRAILING 112.35 [104.13-180.09] 56.54 [50.05-59.65] 1.99x
1.18.1 8 / 1024 BOTH 147.08 [143.19-152.81] 20.96 [19.73-21.70] 7.02x
1.18.1 8 / 1024 LEADING 84.72 [82.31-88.87] 19.66 [18.52-20.77] 4.31x
1.18.1 8 / 1024 TRAILING 103.82 [101.80-104.36] 20.19 [19.19-20.80] 5.14x
1.18.1 264 / 16 BOTH 301.21 [248.94-554.30] 96.92 [89.56-100.54] 3.11x
1.18.1 264 / 16 LEADING 159.28 [113.87-240.48] 90.50 [85.53-95.24] 1.76x
1.18.1 264 / 16 TRAILING 237.90 [196.82-1063.74] 92.88 [86.26-96.46] 2.56x
1.18.1 264 / 1024 BOTH 260.56 [252.13-263.20] 58.98 [56.48-61.01] 4.42x
1.18.1 264 / 1024 LEADING 109.44 [107.03-110.84] 55.22 [53.59-59.03] 1.98x
1.18.1 264 / 1024 TRAILING 209.08 [204.29-214.20] 55.34 [54.65-57.76] 3.78x
2.2.1 8 / 16 BOTH 163.17 [148.21-313.42] 57.96 [52.06-67.89] 2.82x
2.2.1 8 / 16 LEADING 174.99 [97.17-218.97] 54.85 [49.82-59.86] 3.19x
2.2.1 8 / 16 TRAILING 202.90 [110.41-1007.57] 57.58 [56.80-58.92] 3.52x
2.2.1 8 / 1024 BOTH 152.31 [150.98-156.82] 21.17 [20.34-21.66] 7.19x
2.2.1 8 / 1024 LEADING 86.67 [83.49-94.38] 20.29 [19.06-20.98] 4.27x
2.2.1 8 / 1024 TRAILING 119.16 [115.61-120.89] 19.76 [18.67-20.59] 6.03x
2.2.1 264 / 16 BOTH 321.49 [250.08-1077.96] 92.20 [90.82-104.19] 3.49x
2.2.1 264 / 16 LEADING 130.45 [110.03-1226.44] 90.86 [87.22-92.90] 1.44x
2.2.1 264 / 16 TRAILING 212.24 [201.80-260.23] 90.17 [82.52-92.26] 2.35x
2.2.1 264 / 1024 BOTH 269.05 [266.16-272.72] 57.91 [57.15-60.14] 4.65x
2.2.1 264 / 1024 LEADING 111.92 [108.80-113.16] 56.65 [55.57-58.46] 1.98x
2.2.1 264 / 1024 TRAILING 215.36 [212.93-218.58] 55.40 [54.48-57.55] 3.89x

The 24 measured medians improve 1.44-7.19x over stock, which is also the previous fallback route. This does not establish a win for every input, computed expression, Unicode evaluator batch or downstream pipeline. The result record retains every warmup and measurement, checksums, ranges, callback diagnostics and initial implementation. The raw CSV contains all prepared trials. The callback is deliberately a separate diagnostic and cannot serve as the previous-production baseline.

The initial cache comparison checked one UTF-16 unit at a time. On 264-byte, 1,024-row LEADING profiles it lost to stock: 121.10 versus 106.99 ns/row on 1.18.1, and 123.13 versus 109.35 on 2.2.1. Profiling localized the expensive certificate frame. Comparing eight widened byte/unit lanes together preserves the complete check and removes that measured regression. Both the unfavorable initial samples and final ranges are retained; their source delta is the cache-comparison loop.

A separate complete job uses one million nullable rows, 64-byte payloads, BOTH, parallelism one and a rowwise blackhole sink. It retains the row source, both perimeter transposes, native JNI and planning/submission through completion. Each Flink version runs two warmups and five measured trials per engine in one JVM, alternating engine order, with the same 1 GiB maximum heap. Native plans and positive consumed Calc work are required. Seconds are median [minimum-maximum]:

Flink Stock / previous fallback Native
1.18.1 0.427211 [0.423715-0.438213] 0.325314 [0.318184-0.330477]
2.2.1 0.531298 [0.524301-0.537697] 0.319176 [0.317447-0.327937]

These short jobs include startup and do not establish sustained connector throughput. Blackhole timing does not verify output values; separate SQL tests compare values, logical types, NULLs, lazy branches, empty filtered batches, Unicode and isolated surrogates. Validation passes 94 focused JVM tests with 14 explicit version skips across both Flink lines and JDK 11/17, 857 release Rust core tests, and unchanged released ScalarFunctionsTest.testTrim on each Flink line. Rust tests use an invalid evaluator ID to prove each admitted trim direction can execute without a JVM callback.

Reproduction after compiling runtime production/test classes and obtaining their test classpath:

cd native
cargo build --locked --release -p streamfusion --no-default-features --features mimalloc --example sql_trim
SF_NATIVE_BENCH_CLASSPATH="$TEST_CLASSPATH" SF_NATIVE_LIB_DIR="$NATIVE_LIB_DIR" \
  target/release/examples/sql_trim ascii 264 true 1024 128 0 all 8 9

Arguments are corpus (ascii or unicode), payload bytes, decoded-cache presence, batch rows, repeated batches, direction (0 BOTH, 1 LEADING, 2 TRAILING), engine filter, warmup trials and measured trials. These prepared measurements do not establish complete-job throughput; a row-source/row-sink job retains both perimeter transposes.

Run the separate bounded job from the repository root, with the release library directory in NATIVE_LIB_DIR:

SF_BENCHMARK=true mvn -B -ntp -pl streamfusion-runtime -am test -Pbench \
  -Dnative.build.skip=true -Dsurefire.failIfNoSpecifiedTests=false \
  -Dsf.testForks=1 -Dtest=SqlTrimPipelineBenchmark \
  "-Dsf.extraJvmArgs=-Xmx1g -Djava.library.path=$NATIVE_LIB_DIR" \
  -Dtrim.pipeline.rows=1000000 -Dtrim.pipeline.width=64 \
  -Dtrim.pipeline.direction=BOTH -Dtrim.pipeline.warmups=2 -Dtrim.pipeline.runs=5

Add flink-1.18 to the Maven profiles for the other supported line. Complete-job trials and their engine order are retained in the JSON result record.

Integer ROUND with a literal position

Measured on 2026-10-28 against canonical main d89e03b5: Apple M4 Pro, macOS aarch64, JDK 17.0.12, Rust 1.94.0 release without default features, system allocator with counting allocation wrapper. Released Flink 1.18.1 and 2.2.1 are measured independently. The query is ROUND(v, literal_position) on each signed integer width. No option is required for admitted positions in -38..38 or typed NULL; runtime/extreme positions stay generated.

The shared integral_rounding example compares equal prepared logical values, widths, NULLs, signed minima and BIGINTs above 2^53. Stock uses the released generated expression over GenericRowData/BinaryRowData. Main and candidate start on Arrow in production Calc, with timers inside Rust. Main's actual encoding was independently checked on both lines: kind 17, one integer input, a matching primitive result code and the literal embedded in the generated expression. Its scalar JVM callback and intrinsic conversions remain timed; a whole-row evaluator is not substituted for that baseline. Setup, initial JNI entry, Arrow import/export and perimeter transposes are excluded. Full Arrow/schema equality precedes timing; every measured output contributes to matching wrapping checksums.

All schemas are nullable; the NULL flag means actual NULLs every seventh row. Profiles include a native-width divisor (-1), a widened divisor (-19), shared-buffer identity (+2), 1,024-row bulk and 16-row controls. Stock/main/candidate order rotates over 16 warmups and 11 measured trials of 65,536 rows per trial. Runs are sequential without concurrent builds or tests. Values are median [min, max] ns/row.

Flink / type / NULL values / stock format / rows / position Stock ns/row Main ns/row Native ns/row Stock/native Main/native
1.18.1 / BIGINT / false / binary / 1024 / -1 20.295 [16.401, 34.218] 63.417 [42.317, 97.714] 1.054 [0.970, 1.186] 19.25x 60.16x
1.18.1 / BIGINT / true / generic / 1024 / -1 25.187 [15.237, 44.253] 57.227 [40.222, 115.171] 1.401 [1.374, 1.504] 17.98x 40.86x
1.18.1 / BIGINT / true / generic / 1024 / 2 21.570 [19.807, 38.154] 48.349 [42.130, 79.212] 0.776 [0.706, 0.835] 27.79x 62.28x
1.18.1 / BIGINT / true / binary / 1024 / -19 21.341 [19.769, 45.342] 64.600 [48.355, 104.913] 1.433 [1.362, 1.626] 14.89x 45.08x
1.18.1 / BIGINT / true / binary / 16 / -1 8.309 [8.127, 8.578] 522.553 [509.412, 531.985] 26.217 [25.069, 27.737] 0.32x 19.93x
1.18.1 / INT / false / binary / 1024 / -1 14.713 [11.123, 16.694] 56.767 [51.779, 101.699] 1.043 [0.915, 1.096] 14.11x 54.44x
1.18.1 / INT / true / generic / 1024 / -1 9.023 [8.751, 10.166] 32.519 [29.354, 98.593] 1.403 [1.362, 1.588] 6.43x 23.18x
1.18.1 / INT / true / generic / 1024 / 2 9.085 [8.963, 12.779] 39.531 [34.158, 76.625] 0.813 [0.800, 0.884] 11.17x 48.61x
1.18.1 / INT / true / binary / 1024 / -19 10.394 [9.523, 29.420] 44.711 [41.842, 96.408] 1.404 [1.354, 1.561] 7.40x 31.85x
1.18.1 / INT / true / binary / 16 / -1 8.261 [8.142, 12.412] 506.146 [501.726, 549.752] 23.167 [22.579, 23.507] 0.36x 21.85x
1.18.1 / SMALLINT / false / binary / 1024 / -1 12.840 [12.487, 34.390] 50.510 [40.183, 100.056] 1.024 [0.963, 1.071] 12.54x 49.31x
1.18.1 / SMALLINT / true / generic / 1024 / -1 10.525 [9.774, 28.827] 32.763 [31.300, 39.398] 1.408 [1.360, 1.470] 7.48x 23.28x
1.18.1 / SMALLINT / true / generic / 1024 / 2 14.768 [11.401, 28.476] 47.838 [35.615, 119.448] 0.830 [0.793, 0.968] 17.79x 57.61x
1.18.1 / SMALLINT / true / binary / 1024 / -19 19.641 [14.444, 36.503] 37.951 [33.329, 54.068] 1.392 [1.289, 1.594] 14.11x 27.27x
1.18.1 / SMALLINT / true / binary / 16 / -1 6.919 [6.827, 27.100] 523.215 [515.179, 527.838] 23.293 [22.640, 23.792] 0.30x 22.46x
1.18.1 / TINYINT / false / binary / 1024 / -1 11.786 [11.625, 12.552] 51.721 [46.646, 98.059] 1.037 [1.020, 1.140] 11.37x 49.88x
1.18.1 / TINYINT / true / generic / 1024 / -1 10.710 [9.556, 29.348] 41.841 [31.691, 72.752] 1.429 [1.249, 1.666] 7.49x 29.27x
1.18.1 / TINYINT / true / generic / 1024 / 2 16.310 [14.906, 31.041] 38.218 [35.713, 56.314] 0.826 [0.773, 0.844] 19.75x 46.27x
1.18.1 / TINYINT / true / binary / 1024 / -19 21.159 [15.428, 37.910] 58.338 [39.677, 113.485] 1.396 [1.350, 1.724] 15.15x 41.78x
1.18.1 / TINYINT / true / binary / 16 / -1 9.501 [8.812, 10.990] 569.044 [549.186, 615.807] 24.003 [23.064, 25.495] 0.40x 23.71x
2.2.1 / BIGINT / false / binary / 1024 / -1 17.445 [16.772, 31.464] 38.129 [36.368, 54.123] 1.046 [1.013, 1.104] 16.68x 36.46x
2.2.1 / BIGINT / true / generic / 1024 / -1 18.519 [18.015, 36.332] 34.995 [32.736, 55.193] 1.427 [1.368, 1.521] 12.97x 24.52x
2.2.1 / BIGINT / true / generic / 1024 / 2 34.648 [19.582, 43.114] 65.889 [49.659, 140.938] 0.837 [0.698, 1.000] 41.38x 78.69x
2.2.1 / BIGINT / true / binary / 1024 / -19 20.863 [20.085, 64.760] 65.118 [54.057, 106.282] 1.440 [1.410, 1.745] 14.49x 45.22x
2.2.1 / BIGINT / true / binary / 16 / -1 8.451 [7.376, 13.141] 571.144 [539.469, 584.906] 26.101 [22.622, 28.975] 0.32x 21.88x
2.2.1 / INT / false / binary / 1024 / -1 12.973 [10.591, 17.623] 54.145 [47.759, 98.277] 1.066 [0.959, 1.113] 12.17x 50.81x
2.2.1 / INT / true / generic / 1024 / -1 12.401 [9.130, 14.297] 49.821 [34.920, 102.674] 1.415 [1.298, 1.521] 8.77x 35.22x
2.2.1 / INT / true / generic / 1024 / 2 12.624 [10.127, 27.711] 66.395 [42.700, 91.185] 0.837 [0.818, 0.850] 15.09x 79.35x
2.2.1 / INT / true / binary / 1024 / -19 15.711 [9.849, 37.292] 105.385 [38.283, 164.042] 1.377 [1.354, 1.432] 11.41x 76.53x
2.2.1 / INT / true / binary / 16 / -1 8.342 [7.921, 8.502] 516.578 [501.617, 538.036] 23.314 [21.689, 25.815] 0.36x 22.16x
2.2.1 / SMALLINT / false / binary / 1024 / -1 14.891 [12.455, 33.859] 60.584 [44.174, 101.692] 1.100 [1.064, 1.170] 13.54x 55.08x
2.2.1 / SMALLINT / true / generic / 1024 / -1 13.615 [10.628, 30.334] 55.721 [51.214, 106.354] 1.415 [1.271, 1.623] 9.62x 39.39x
2.2.1 / SMALLINT / true / generic / 1024 / 2 13.747 [11.222, 32.217] 49.999 [41.403, 97.153] 0.832 [0.741, 0.855] 16.53x 60.12x
2.2.1 / SMALLINT / true / binary / 1024 / -19 14.741 [14.492, 31.559] 35.318 [33.274, 53.660] 1.404 [1.383, 1.635] 10.50x 25.16x
2.2.1 / SMALLINT / true / binary / 16 / -1 7.176 [6.625, 8.462] 557.781 [533.161, 611.802] 24.355 [22.543, 28.458] 0.29x 22.90x
2.2.1 / TINYINT / false / binary / 1024 / -1 24.363 [11.980, 32.704] 57.568 [33.740, 84.532] 1.095 [0.927, 1.188] 22.25x 52.58x
2.2.1 / TINYINT / true / generic / 1024 / -1 11.126 [9.336, 39.951] 43.566 [31.790, 70.793] 1.387 [1.241, 1.590] 8.02x 31.40x
2.2.1 / TINYINT / true / generic / 1024 / 2 18.447 [15.104, 29.708] 68.672 [44.312, 97.975] 0.820 [0.750, 0.860] 22.51x 83.80x
2.2.1 / TINYINT / true / binary / 1024 / -19 16.088 [15.526, 35.040] 54.562 [38.261, 114.259] 1.413 [1.254, 1.724] 11.38x 38.61x
2.2.1 / TINYINT / true / binary / 16 / -1 9.165 [8.950, 9.549] 498.255 [490.307, 538.081] 22.830 [22.391, 24.350] 0.40x 21.82x

The 1,024-row candidate improves 6.43-41.38x over stock and 23.18-83.80x over main. The 16-row candidate remains slower than stock (0.29-0.40x), while improving 19.93-23.71x over main. Native-width remainder avoids per-element i128 division; large powers use widened comparisons and nonnegative positions share the input. A three-second native-only sample on the slower INT/16-row profile observes scalar-function argument/result setup and destruction, primitive-array allocation, Calc projection and checksum consumption. Overlapping sampled frames do not establish independent CPU percentages. The raw allocation probes distinguish shared output buffers from newly requested values buffers. These are bounded batch gains, not a universal small-batch or end-to-end claim.

An independent complete job processes 1M INT rows using ROUND(v,-1), parallelism one, a bounded Flink row source and a rowwise blackhole sink. Native plans retain both transposes and require positive consumed NativeCalc work. Main uses its actual encoder first on the classpath and requires the ROUND JVM route; candidate requires jvm=[]. Two warmups precede five measured jobs in each sequential engine JVM. Seconds are median [min, max]:

Stock Main scalar callback Candidate
0.143764 [0.139465, 0.157385] 0.164713 [0.157161, 0.173648] 0.139733 [0.130734, 0.143993]

The candidate improves main on this job, with overlapping stock/candidate ranges. Startup, submission and JIT remain included; these short jobs do not establish sustained complete-job acceleration. All raw trials, checksums, configuration and allocation probes remain available. Generic small-batch and perimeter work stays in #491.

After building current Java test classes, set JAVA_HOME, SF_NATIVE_BENCH_CLASSPATH and SF_NATIVE_LIB_DIR to the release library directory. Build and run:

cargo build --manifest-path native/Cargo.toml -p streamfusion --release \
  --no-default-features --lib --example integral_rounding
SF_INTEGRAL_ROUNDING_FUNCTION=ROUND SF_INTEGRAL_ROUNDING_POSITION=-1 \
  native/target/release/examples/integral_rounding INT true false 1024 64 16 11

Arguments are type, actual NULL flag, stock BinaryRowData flag, batch rows, loops, warmups and measured trials. Without the position environment variable the shared example retains its historical single-argument mode. Complete-job properties are rounding.pipeline.function=ROUND, rounding.pipeline.position=-1 and rounding.pipeline.engine=flink|previous|native, enabled by SF_BENCHMARK=true and the bench profile. The previous mode requires the preceding encoder first on the classpath; relabeling the unchanged candidate is insufficient.

Runtime parity covers every admitted position and all widths on both supported Flink lines and JDK 11/17. Witnesses consume 5,003 rows per width with no ROUND JVM route, check exact expected values/NULLs, and require positive native work and no fallback. Nested/widened expressions, scalar consumers, filters, empty batches, selected/unselected failures and extreme-position host errors are covered. The unchanged released upstream ScalarFunctionsTest.testRound passes on both lines; it supplies oracle evidence, while the runtime witnesses prove consumed native work. Full JVM/upstream suites remain CI checks.

Integer TRUNCATE with a literal position

Measured independently on 2026-10-28 against canonical main d89e03b5, using the same Apple M4 Pro, JDK 17.0.12, Rust 1.94.0 release/no-default-features configuration, system/counting allocator and boundaries as literal-position ROUND. The query is TRUNCATE(v, literal_position) on INT/BIGINT. Direct narrow-integer signatures preserve released Flink's generated-compilation failure; explicit widening is supported. Admission covers INT literal positions -38..38 or typed NULL and verified producers only; runtime/extreme positions and other producers stay generated.

Stock executes the released generated projection over prepared GenericRowData/BinaryRowData. Main's actual kind-17 scalar expression has one integer argument/result and embeds the literal position; its callback and intrinsic conversion stay inside the Rust timer. Candidate production Calc invokes the Arrow toward-zero kernel. Values, widths, NULLs, minima and BIGINTs above 2^53 match. Full output/schema equality precedes timing and all measured outputs contribute to matching checksums in their own runtime. Input/operator setup, JNI import/export and perimeter transposes are excluded only from this prepared functionality measurement. No concurrent tests/builds run during measurement.

Every schema is nullable; actual NULLs appear every seventh row when the flag is true. Profiles cover -1, -19 and +2 positions, native-width/large divisors, shared-buffer identity and 1,024-/16-row batches. Each case rotates stock/main/candidate order for 16 warmups and 11 measured trials of 65,536 rows. Values are median [min, max] ns/row.

Flink / type / NULL values / stock format / rows / position Stock ns/row Main ns/row Native ns/row Stock/native Main/native
1.18.1 / BIGINT / false / binary / 1024 / -1 82.793 [80.556, 96.215] 105.433 [99.404, 160.713] 1.088 [1.046, 1.149] 76.06x 96.86x
1.18.1 / BIGINT / true / generic / 1024 / -1 73.506 [70.045, 119.251] 99.348 [92.960, 154.620] 1.449 [1.300, 1.569] 50.73x 68.57x
1.18.1 / BIGINT / true / generic / 1024 / 2 79.189 [66.412, 109.428] 96.375 [88.223, 119.227] 0.942 [0.796, 0.998] 84.04x 102.28x
1.18.1 / BIGINT / true / binary / 1024 / -19 29.352 [28.196, 60.632] 49.988 [47.063, 72.699] 0.969 [0.942, 1.118] 30.29x 51.59x
1.18.1 / BIGINT / true / binary / 16 / -1 66.630 [63.771, 71.818] 622.942 [606.855, 636.044] 24.017 [22.496, 30.481] 2.77x 25.94x
1.18.1 / INT / false / binary / 1024 / -1 63.423 [61.859, 95.424] 88.022 [80.719, 137.247] 1.005 [0.931, 1.125] 63.10x 87.57x
1.18.1 / INT / true / generic / 1024 / -1 56.305 [54.621, 109.278] 81.130 [74.706, 99.076] 1.493 [1.376, 1.634] 37.70x 54.32x
1.18.1 / INT / true / generic / 1024 / 2 54.448 [52.867, 97.235] 96.239 [73.837, 135.696] 0.849 [0.821, 0.868] 64.10x 113.30x
1.18.1 / INT / true / binary / 1024 / -19 26.170 [24.514, 49.325] 62.735 [46.422, 148.438] 1.012 [0.946, 1.991] 25.86x 61.98x
1.18.1 / INT / true / binary / 16 / -1 52.738 [50.178, 59.952] 595.444 [586.147, 614.468] 24.014 [22.436, 26.831] 2.20x 24.80x
2.2.1 / BIGINT / false / binary / 1024 / -1 81.130 [77.929, 119.018] 106.023 [100.971, 168.556] 1.126 [1.043, 1.234] 72.05x 94.16x
2.2.1 / BIGINT / true / generic / 1024 / -1 71.206 [68.321, 89.049] 93.903 [89.711, 114.658] 1.430 [1.390, 1.520] 49.80x 65.67x
2.2.1 / BIGINT / true / generic / 1024 / 2 74.601 [61.907, 107.140] 93.374 [87.193, 154.513] 0.860 [0.745, 0.956] 86.72x 108.55x
2.2.1 / BIGINT / true / binary / 1024 / -19 29.249 [27.757, 61.291] 66.074 [50.031, 106.862] 1.003 [0.959, 1.086] 29.17x 65.90x
2.2.1 / BIGINT / true / binary / 16 / -1 66.425 [64.204, 75.006] 623.986 [616.370, 649.518] 24.160 [23.481, 25.192] 2.75x 25.83x
2.2.1 / INT / false / binary / 1024 / -1 63.797 [60.988, 116.205] 88.085 [79.557, 121.353] 1.036 [0.984, 1.175] 61.60x 85.05x
2.2.1 / INT / true / generic / 1024 / -1 55.975 [53.712, 95.767] 82.354 [73.753, 118.947] 1.420 [1.291, 1.466] 39.41x 57.98x
2.2.1 / INT / true / generic / 1024 / 2 65.570 [53.518, 104.253] 85.697 [76.602, 140.537] 0.852 [0.741, 0.964] 76.96x 100.59x
2.2.1 / INT / true / binary / 1024 / -19 25.875 [24.856, 50.336] 51.790 [44.918, 96.118] 0.957 [0.909, 1.131] 27.04x 54.13x
2.2.1 / INT / true / binary / 16 / -1 52.284 [50.070, 56.420] 586.700 [566.703, 602.521] 28.638 [27.205, 29.861] 1.83x 20.49x

At 1,024 rows the recorded candidate improves 25.86-86.72x over stock and 51.59-113.30x over main. At 16 rows it improves 1.83-2.77x over stock and 20.49-25.94x over main. The kernel avoids released Flink's DECIMAL(38,18) intermediate, uses native integer remainder for negative positions, writes zero when the divisor exceeds the input width, and shares buffers for nonnegative positions. There is no HALF_UP increment in TRUNCATE. Allocation probes in the raw record distinguish shared validity/identity buffers from new result values. These measurements do not imply that every Calc or workload accelerates.

The independent complete job retains the bounded row source, both transposes and a rowwise blackhole sink: Flink 2.2.1, 1M INT rows, TRUNCATE(v,-1), parallelism one, two warmups and five measured jobs per engine in sequential JVMs. The actual main encoder is loaded first for the previous control; route checks require its TRUNCATE JVM evaluator and candidate jvm=[]. Both native modes require consumed NativeCalc work and both transposes. Seconds are median [min, max]:

Stock Main scalar callback Candidate
0.210510 [0.200087, 0.216174] 0.215636 [0.205634, 0.227845] 0.131335 [0.127314, 0.152458]

The recorded complete workload improves 1.60x over stock and 1.64x over main by median. Submission, startup, JIT and both transposes remain included; this does not establish sustained throughput or a universal gain. All raw trials, checksums and allocation probes retain variability and the exact configuration. Generic Calc/perimeter costs remain in #491.

Use the shared release example and complete-job fixture from ROUND, selecting SF_INTEGRAL_ROUNDING_FUNCTION=TRUNCATE / rounding.pipeline.function=TRUNCATE, with the corresponding position setting. Without a position setting the historical single-argument mode is unchanged. The previous mode requires its actual encoder first on the classpath.

The final combined regression passes 396 JVM test invocations across 13 classes, Flink 1.18.1/2.2.1 and JDK 11/17, plus all 835 release Rust core tests. Fresh all-core production compilation targets Java 11 on both lines. TRUNCATE parity covers all 77 positions, declared nullability, typed NULL positions, signed extrema, exact BIGINTs, scalar consumers, filters and ROUND/TRUNCATE nesting. INT/BIGINT witnesses verify exact outputs and consumed native work over 5,003 rows per type with no evaluator or fallback. Selected/unselected failures, empty batches, extreme-position Underflow and narrow-input compilation behavior remain checked. Unchanged released upstream testRound and testTruncate pass on both lines; those supply oracle evidence, while the focused witnesses establish execution of the native kernels. Full JVM/upstream suites remain CI checks.

After sharing the ROUND/TRUNCATE input handling, independent ROUND INT/-1 and BIGINT/-19 1,024-row controls recheck output/schema/checksums and timing against stock and actual main. Their full measured trials are included as shared_round_kernel_rechecks in the TRUNCATE raw record; the original 40-case ROUND matrix records the preceding ROUND-only implementation.

DECIMAL to STRING

The direct-input CAST(d AS STRING) comparison uses released Flink 1.18.1 and 2.2.1, JDK 17, one thread and a release/mimalloc native build. Stock Flink starts with prepared GenericRowData or BinaryRowData; StreamFusion starts with the same logical decimals in prepared Arrow Decimal128 buffers. Setup and input preparation are outside both timers. Stock uses the released generated expression and consumes UTF-8 result lengths; native times the production Calc and the same length checksum inside Rust. Output allocation and the previous implementation's JVM callbacks remain inside the timer. Perimeter JNI entry, export and row/Arrow transposes are excluded from this small-functionality comparison.

Inputs cover DECIMAL(18,3), DECIMAL(38,0) and DECIMAL(38,38), with signed varying coefficients, maxima, zero, coefficients above 2^53 and optional NULL every seven rows. Batch sizes are 16 and 1,024 rows. Each independent JVM runs 16 warmups and nine measured trials, alternating engine order. Each timed trial processes 131,072 rows. The previous control loads the actual preceding encoder from 2441db79 first on the classpath and verifies its JVM expression binding; it uses the unchanged callback implementation in the same release engine build. Candidate and previous controls each retain their paired stock trials. Benchmarks run serially.

On Apple M4 Pro with JDK 17.0.12, the measured median ranges are:

Flink line Batch rows Flink / native Previous StreamFusion / native
1.18.1 16 1.008–2.362x 19.776–23.101x
1.18.1 1,024 2.861–6.438x 5.725–8.238x
2.2.1 16 1.087–1.964x 20.088–22.412x
2.2.1 1,024 2.816–6.363x 5.743–8.234x

The near-1x small-batch observation is within timing variability, not evidence of a reliable speedup. Separate-JVM stock controls drift between 0.786x and 1.380x, so previous/candidate ratios do not isolate only the implementation change. The prepared raw trials retain every warmup/sample, paired stock control and checksum. The result record retains min/median/max for every configuration, the release DSO hash and all complete-job trials.

The separate bounded job compares one million nullable DECIMAL(18,3) rows at parallelism 1, with a rowwise sequence/map source and blackhole STRING sink. Two warmups and five measured trials alternate engine order. Planning, startup, production JNI and both perimeter transposes remain included. Each native job must consume NativeCalc rows, and its plan and evaluator descriptors must match the expected native or previous JVM route.

Flink line Candidate's stock (s) Native (s) Previous's stock (s) Previous StreamFusion (s)
1.18.1 0.205514 0.174687 0.206497 0.237319
2.2.1 0.203710 0.176602 0.198950 0.232253

Native job medians improve by 15.0% and 13.3% versus their paired stock controls, and 26.4% and 24.0% versus previous StreamFusion. These subsecond bounded jobs include startup and do not establish sustained throughput for other pipelines or configurations.

The released DecimalData oracle checks 28,823 boundary and deterministic random values across every precision 1–38 and every valid scale against released DecimalData's plain formatting. The focused SQL matrix additionally checks resolved types, legacy/modern mode, consumers, nullable runtime inputs, unselected failures, empty filtered results and retained bounded and computed-DECIMAL paths. The unchanged released DecimalTypeTest literal, field-access and CASE fixtures supplement that oracle; their local expression tests are not native-work evidence. The runtime SQL tests separately require native execution and empty JVM evaluator descriptors. All 352 focused JVM invocations pass across Flink 1.18.1/2.2.1 and JDK 11/17 with no skips; 855 release Rust core tests also pass. Each of the four version/JDK oracle runs matches all 28,823 values. This does not replace the full upstream integration suite or CI.

After compiling DecimalStringBenchmarkFunctions, use the released test/runtime classpath as SF_NATIVE_BENCH_CLASSPATH and the release library directory as SF_NATIVE_LIB_DIR:

cargo build --manifest-path native/Cargo.toml --release --locked -p streamfusion \
  --no-default-features --features mimalloc --example decimal_string
native/target/release/examples/decimal_string 18 3 true false 1024 128 16 9
java -cp "$SF_NATIVE_BENCH_CLASSPATH" tech.streamfusion.planner.DecimalStringBenchmarkFunctions > oracle.csv
native/target/release/examples/decimal_string --oracle oracle.csv

Arguments are precision, scale, NULL values, BinaryRowData stock input, batch rows, loops, warmups and measured trials. The previous control also sets SF_DECIMAL_STRING_PREVIOUS=true and prepends the independently compiled preceding encoder to the classpath. Complete jobs use DecimalStringPipelineBenchmark, SF_BENCHMARK=true and the corresponding sf.decimal.string.previous property/classpath pair.

Binary STARTSWITH

Released Flink 2.2.1, Oracle JDK 17.0.12 and a release/mimalloc engine are compared on an Apple M4 Pro, at one thread. Stock executes the released generated expression over prepared GenericRowData or BinaryRowData. Native executes the production Calc over equivalent prepared Arrow Binary or FixedSizeBinary inputs. Both consume a Boolean checksum including NULLs. Inputs, compilation and setup are outside the timers; native timing starts and stops inside Rust. Output allocation, validity work and the previous implementation's JVM callback remain inside timing. Initial JNI entry, Arrow export and row/Arrow transposes are excluded only from this prepared-function gate.

The 40 configurations cover 16/1,024-row batches, widths 8/64/4,096, fixed/variable bytes, Generic/BinaryRowData and NULL/non-NULL inputs. Needles include matching and non-matching bytes, empty needles and needles longer than the input. Fixed-width needles contain four bytes; padding is retained. Each independent JVM alternates engine order for 24 warmups and nine measured trials, processing 131,072 rows per trial. Previous controls prepend the independently compiled encoder from canonical main 3af56f9a, assert its JVM binding and retain that callback inside the same production Calc. They each retain their own stock control. All runs are serial.

Ranges of medians across the matching configurations, in nanoseconds per row:

Stock input Batch rows Stock Flink Previous StreamFusion Native
GenericRowData 16 1.521–2.047 525.025–776.607 19.858–29.297
GenericRowData 1,024 2.043–3.137 36.469–220.812 3.028–4.568
BinaryRowData 16 10.491–183.693 554.351–839.167 19.851–29.036
BinaryRowData 1,024 15.052–190.540 37.113–213.782 3.033–4.622

For 1,024-row BinaryRowData inputs, matching stock/native ratios are 3.878–62.816x and previous/native ratios are 8.338–70.478x. The stock generated binary getter copies bytes from BinaryRowData; native compares existing Arrow buffers. GenericRowData already holds byte arrays, so its cheap getter avoids that copying: those controls still beat native, whose stock/native ratios are 0.474–0.829x at 1,024 rows. Sixteen-row native batches also lose to stock for all GenericRowData and several BinaryRowData cases. This is a qualified bulk binary comparison improvement, not a universal scalar or complete-job acceleration claim.

The raw trials distinguish each control and include all warmups, samples and checksums. The record includes per-case min/median/max and the release library SHA-256.

The separate million-row job uses nullable 64-byte inputs, parallelism 1, a rowwise sequence/map source and a blackhole BOOLEAN sink. Two warmups and five measured trials alternate engine order. JNI, planning/startup and both transposes remain included. Native operators must consume rows, and evaluator descriptors must match the intended native or previous JVM route.

Job control Stock median (min–max), seconds StreamFusion median (min–max), seconds
Candidate 0.170384 (0.162449–0.181901) 0.187063 (0.185032–0.198267)
Previous 0.166628 (0.157360–0.177254) 0.223773 (0.215104–0.239078)

Native improves 16.4% over previous StreamFusion but takes 9.8% longer than its paired stock control. Separate-JVM stock medians drift 2.3%; these short bounded jobs do not establish sustained throughput. All complete-job trials retain the unfavorable results as well as the gains over the callback.

After compiling the test helper, set JAVA_HOME to the supported JDK and provide the matching runtime/test classpath through SF_NATIVE_BENCH_CLASSPATH, plus SF_NATIVE_LIB_DIR:

cargo build --manifest-path native/Cargo.toml --release --locked -p streamfusion \
  --features mimalloc --lib --example binary_boundary
native/target/release/examples/binary_boundary 64 false true true 1024 128 24 9

Arguments are width, fixed-width input, nullable input, BinaryRowData stock input, batch rows, loops, warmups and measured trials. Previous controls additionally set SF_BINARY_BOUNDARY_PREVIOUS=true and prepend the actual preceding encoder's compiled classes. An optional final ends argument selects ENDSWITH; the default remains STARTSWITH. Complete jobs use BinaryBoundaryPipelineBenchmark with SF_BENCHMARK=true; its test-only sf.binary.boundary.previous property must agree with that classpath. Set sf.binary.boundary.suffix=true for the suffix workload.

Binary ENDSWITH

The suffix comparison uses the same released Flink 2.2.1/JDK 17.0.12, release/mimalloc, single-thread machine, input matrix, checksums and timer boundaries as the binary STARTSWITH comparison. A separate 40-configuration matrix compares the shared suffix kernel with the released generated expression and the actual preceding encoder from 3af56f9a. Matching needles are drawn from the tail of the input. Arbitrary non-text bytes, NULLs, mismatches, empty/long needles and fixed padding remain represented. Each trial processes 131,072 rows; 24 warmups and nine measured trials alternate engine order. Benchmarks run serially.

Ranges of medians, in nanoseconds per row:

Stock input Batch rows Stock Flink Previous StreamFusion Native
GenericRowData 16 1.971–2.325 534.839–813.729 19.512–27.349
GenericRowData 1,024 2.515–4.505 35.695–215.287 3.015–4.722
BinaryRowData 16 11.218–194.367 535.303–804.324 19.411–26.987
BinaryRowData 1,024 14.347–196.462 35.084–214.879 3.100–4.623

At 1,024 rows with BinaryRowData stock inputs, native improves 3.677–62.521x over stock and 8.218–68.381x over previous StreamFusion. GenericRowData stock/native ratios range from 0.571 to 1.223x: its existing byte arrays avoid the generated BinaryRowData getter's copying, and most of those cheap controls still beat native. Sixteen-row controls also retain Calc setup and output-allocation costs. These results establish a bulk binary comparison improvement, not a universal speedup across input representations or batch sizes.

Raw prepared trials retain every control, warmup, sample and checksum. The result record includes all min/median/max values and the release library SHA-256. Use the shared binary_boundary example with a final ends argument to reproduce the suffix comparison.

The corresponding million-row job retains the same nullable 64-byte source, blackhole sink, parallelism 1, both transposes and JNI as the prefix diagnostic. Two warmups and five measured trials alternate engines; consumed native work and the exact evaluator route are required.

Job control Stock median (min–max), seconds StreamFusion median (min–max), seconds
Candidate 0.177847 (0.163777–0.195478) 0.198512 (0.193328–0.208405)
Previous 0.180443 (0.172684–0.201166) 0.237657 (0.226938–0.247604)

Native takes 16.5% less time than previous StreamFusion, but 11.6% more than its paired stock control. Separate-JVM stock medians drift 1.5%; the diagnostic includes startup and does not establish sustained throughput. All complete-job trials retain these limits. The shared-kernel change also rechecks representative STARTSWITH prepared configurations; those trials are included in the suffix result record.

On Flink 2.2.1, the focused prefix/suffix and adjacent STRING/BOOLEAN runtime suites pass 42 tests each on JDK 11 and 17. They include arbitrary bytes, fixed casts, filters, conditional branches and empty filtered batches. Separate 5,004-row witnesses consume native Calc output without JVM evaluator bindings or fallback. The Rust core suite without default features passes 865 tests, including sliced value/validity buffers, all 256 byte values, empty arrays, scalar NULLs and byte-for-byte comparison oracles. The default-feature release suite also passes 1,004 tests with two existing ignores. CI supplies the broader version matrix.

Binary to STRING

The 2026-11-01 comparison uses Apple M4 Pro / 24 GiB, released Flink 1.18.1 and 2.2.1, JDK 17.0.12 and a Rust 1.95.0 release/mimalloc build with default features. JDK 11 correctness is checked separately. Stock executes the released generated cast over prepared GenericRowData or BinaryRowData. Native executes the complete production Calc over equivalent prepared Arrow Binary or FixedSizeBinary batches; the timer starts and stops inside Rust. Setup, initial JNI entry, final export and perimeter transposes are outside this small-functionality timer. Allocation and all conversions intrinsic to the expression remain inside it.

Both timers consume NULL as -1, empty as zero, otherwise the first UTF-16 unit plus one. The cast still computes its complete result. Complete outputs are separately verified against released generated casts over 90,404 deterministic inputs per Flink/JDK pair. Earlier full-length consumption added a UTF-16 scan on Arrow that Java String.length does not perform; those diagnostics are retained separately, not pooled with the final results.

The actual preceding encoder from main 3af56f9a is independently compiled and loaded first for the previous control. It must reject the binary cast, so its prepared operation is the resulting generated Flink fallback, rather than a simulated JVM callback. Each candidate JVM alternates stock/native order; the previous control runs in a separate JVM with the same values and resources. All runs are serial, with no concurrent builds or tests, 256/512 MiB initial/max heap, 16 warmups, nine measured trials and 65,536 processed rows per trial.

The 71 configurations include widths 8/64/4096, batches 16/1024, fixed/variable bytes, both stock RowData forms, NULL/non-NULL controls, ASCII, supplementary/CJK text, malformed surrogate tails, mixed tails and dense deterministic arbitrary bytes. NULL appears every seventh row when enabled. Flink 2.2 has the full 55-configuration matrix; Flink 1.18 has 16 representative controls including dense invalid input. At 1,024 rows with BinaryRowData:

Flink Stock / native median Previous fallback / native median
1.18.1 1.225-27.153x 1.206-26.926x
2.2.1 1.220-27.143x 1.181-26.864x

Selected variable-width nullable inputs, ns/row, median [minimum-maximum]:

Flink Bytes / content Stock Previous fallback Native
2.2.1 8 / ASCII 18.90 [18.69-19.16] 17.46 [16.40-18.01] 1.98 [1.92-2.08]
2.2.1 64 / supplementary+CJK 55.18 [50.66-91.12] 54.81 [53.80-84.05] 21.44 [21.19-27.80]
2.2.1 4096 / malformed tail 3541.72 [3488.84-3632.33] 3505.34 [3437.55-3646.49] 130.49 [126.02-131.81]
2.2.1 4096 / dense arbitrary 3187.90 [3090.52-3303.64] 3085.71 [3070.70-3224.00] 2612.54 [2536.02-2682.08]
1.18.1 4096 / supplementary+CJK 2202.27 [2169.05-2249.56] 2182.91 [2153.31-2273.44] 1264.74 [1201.06-1324.22]

Small batches remain a limit. Across the Flink 2.2 16-row controls, stock/native median ratios range from 0.504x to 20.828x, and previous/native from 0.528x to 20.673x. One 1,024-row GenericRowData case also loses to its independent previous fallback control despite beating paired stock. Independent stock/fallback control medians drift by factors 0.625-1.736. Retain both controls and their variability; these measurements do not isolate a universal throughput improvement.

The complete record includes all configuration ranges, the production library hash, validation and full-job trials. All prepared trials retain warmups and checksums. The implementation shares valid buffers, copies validated prefixes, writes directly into Arrow and batches independent invalid-byte replacements; see the decoder technique.

Complete row-source jobs

Each Flink line also runs a one-million-row job at parallelism one, with 64-byte inputs, a malformed tail every seventh row and independent NULL every eleventh row. A rowwise sequence/map source feeds the cast and a blackhole STRING sink. Planning, startup, production JNI and both RowData/Arrow transposes stay timed. The candidate must show both transposes, no evaluator/fallback and consumed NativeCalc work; the old encoder must show stock fallback and zero native work. Each independent JVM uses the same 2 GiB maximum heap, two warmups and five measured jobs with alternating engine order. Seconds, median [minimum-maximum]:

Flink Candidate's stock Native Previous's stock Previous fallback
2.2.1 0.228064 [0.225555-0.241543] 0.210187 [0.209547-0.220651] 0.213727 [0.206077-0.250102] 0.212407 [0.206951-0.222358]
1.18.1 0.239151 [0.219020-0.254718] 0.204987 [0.202946-0.213079] 0.212720 [0.205804-0.217317] 0.213624 [0.204449-0.216186]

Native is 7.8%/14.3% lower than paired stock medians, but the independent stock controls drift and ranges overlap. Gains versus the actual preceding fallback are only 1.0%/4.0%, also with overlapping ranges. These short jobs do not establish a reliable whole-job speedup or sustained connector throughput.

Profiling and validation

Earlier experiments are retained as diagnostics with their original consumption method. The initial partial matrix, prefix-validation experiment and UTF-8-chunks experiment consume full UTF-16 length. Profiling first located most long-ASCII samples in that extra result scan. After changing to constant output consumption, the prefix/manual decoder still lost on dense invalid bytes. The direct-builder version measured 3412 ns/row versus stock 3164 for 4096-byte dense inputs. Batching independent replacement writes reduces that profile to 2596 versus stock 3129 in the final preflight. Only the subsequent frozen 71-configuration matrix supplies the reported ranges.

Final validation passes 142 focused JVM test invocations across both released lines and JDK 11/17, without skips, and 1004 release Rust core tests with two existing ignores. Each of the four generated-expression oracles matches all 90,404 inputs, including every single/two-byte input, surrogate continuation boundaries and deterministic longer arbitrary bytes. SQL coverage adds modern and legacy modes, fixed padding, composed consumers, TRY_CAST operand failures, lazy CASE, all-filtered batches, multi-batch ownership and retained fallbacks. Separate fixed/variable witnesses consume 5,003 rows and require jvm=[] plus positive native work. Full CI and unchanged upstream suites remain required.

Reproduce

Compile the test helper with the released runtime/test classpath, then use JDK 17:

mvn -Pbench -pl streamfusion-runtime -am -Dnative.build.skip=true -DskipTests test-compile
mvn -pl streamfusion-runtime dependency:build-classpath -Dmdep.outputFile=target/binary-character.classpath
export SF_NATIVE_BENCH_CLASSPATH="$PWD/streamfusion-runtime/target/test-classes:$PWD/streamfusion-runtime/target/classes:$(cat streamfusion-runtime/target/binary-character.classpath)"
export SF_NATIVE_LIB_DIR="$PWD/native/target/release"
cargo build --manifest-path native/Cargo.toml --release --locked -p streamfusion \
  --features mimalloc --lib --example binary_character
native/target/release/examples/binary_character 64 4 false true true 1024 64 16 9
java -cp "$SF_NATIVE_BENCH_CLASSPATH" tech.streamfusion.planner.BinaryCharacterBenchmarkFunctions > oracle.csv
native/target/release/examples/binary_character --oracle oracle.csv

Arguments are byte width, content mode 0-4, fixed-width input, nullable values, BinaryRowData stock input, batch rows, loops, warmups and measured trials. The previous control prepends the independently compiled main encoder and sets SF_BINARY_CHARACTER_PREVIOUS=true; it fails if that encoder admits the cast. Use the flink-1.18 Maven profile for the other released line. Complete jobs use BinaryCharacterPipelineBenchmark, SF_BENCHMARK=true and the matching sf.binary.character.previous property/classpath pair.

Modern bounded CHAR/VARCHAR targets

Modern CAST/TRY_CAST(binary AS VARCHAR(n)/CHAR(n)) extends the same kernel with Flink's UTF-16 truncation and CHAR padding. The preceding encoder at c0d1a509 rejects these targets, so its control verifies that rejection and measures released Flink's generated fallback. The prepared comparison uses released Flink 1.18.1/2.2.1 on JDK 17.0.12, Apple M4 Pro, one thread and a release/mimalloc build, with the binary-character harness above: nullable inputs, GenericRowData/BinaryRowData, 16/1,024-row batches, 131,072 rows per trial, 16 warmups and nine measurements per engine. Targets are VARCHAR(4), VARCHAR(64) and CHAR(16) over 8/64-byte ASCII, supplementary+CJK and malformed-tail content, plus 16-byte fixed BINARY. The stock consumer maps the lone high surrogate of a split pair to ?, the character Flink writes for it.

Selected variable-width nullable BinaryRowData inputs at 1,024 rows, ns/row median:

Flink Target Bytes / content Stock Previous fallback Native
2.2.1 VARCHAR(4) 8 / ASCII 18.42 18.56 5.10
2.2.1 VARCHAR(4) 64 / supplementary+CJK 58.09 62.03 29.79
2.2.1 VARCHAR(64) 64 / ASCII 15.02 14.09 3.48
2.2.1 CHAR(16) 8 / ASCII 23.91 24.79 8.69
2.2.1 CHAR(16) 64 / supplementary+CJK 59.35 57.26 38.37
1.18.1 VARCHAR(4) 8 / supplementary+CJK 27.20 27.09 11.94
1.18.1 CHAR(16) 64 / ASCII 25.19 23.85 5.96
1.18.1 CHAR(16) 64 / supplementary+CJK 50.34 56.44 37.36

Across all 42 1,024-row configurations per line, stock/native median ratios are 1.357-8.168x on Flink 2.2.1 and 1.331-6.179x on 1.18.1; previous/native ratios are 1.370-8.457x and 1.198-7.167x. Supplementary and CJK text gains least. Sixteen-row batches remain a limit: 30 of 36 such configurations on Flink 2.2.1 and 32 of 36 on 1.18.1 are slower than stock, with stock/native ratios down to 0.404x and 0.394x. These results do not establish acceleration for small batches.

An initial Flink 2.2.1 matrix built every bounded row through a string builder and scanned whole rows for ASCII. Seven of its 42 1,024-row configurations lost to stock (down to 0.818x), all truncating short ASCII outputs. The measured version writes values and offsets directly, reuses input validity and checks only the truncated prefix for ASCII. All final trials retain warmups, checksums and the target column. Before timing, the generated-expression oracle matches all 90,404 inputs on both lines for STRING, VARCHAR(1), VARCHAR(2), VARCHAR(5), CHAR(1), CHAR(3) and CHAR(8).

Complete one-million-row jobs reuse the row-source pipeline above, with a bounded blackhole column of the same type. Candidate runs require both transposes, consumed NativeCalc work and no evaluator or fallback. Two warmups and five measured jobs alternate engine order in one JVM per target. Seconds, median [minimum-maximum]:

Flink Target Stock Native
2.2.1 VARCHAR(16) 0.231831 [0.229030-0.237508] 0.207224 [0.200973-0.220357]
2.2.1 CHAR(16) 0.239948 [0.232398-0.260705] 0.212151 [0.203249-0.215048]
1.18.1 VARCHAR(16) 0.235405 [0.223152-0.248219] 0.211334 [0.201674-0.211897]
1.18.1 CHAR(16) 0.237634 [0.235295-0.243123] 0.202265 [0.198970-0.212588]

Native medians are 10.2-14.9% lower. The previous encoder falls back to the same generated stock path, so these jobs have no separate previous column. They are short and use one stock control per target, so they do not establish sustained connector throughput.

Reproduce with the commands above, setting SF_BINARY_CHARACTER_TARGET, for example VARCHAR(4) or CHAR(16), for the prepared runs and oracle. The Java oracle also reads -Dsf.binary.character.target. Complete jobs pass the target with -Dsf.extraJvmArgs="-Dsf.binary.character.target=CHAR(16) -Xmx2g".

Latest-main bounded-cast requalification (2026-11-03)

The fresh record repeats the prepared comparison on canonical main 5d628b6a with the same final kernel and released lines, using release/mimalloc for both the executable and JNI DSO (verified from their allocator symbols). The actual preceding main encoder is compiled independently against the current runtime and fixture. This isolates admission/kernel changes, not an entire historical library replay.

Twelve 1,024-row configurations per line cover VARCHAR(4), VARCHAR(64), CHAR(16), ASCII, supplementary/CJK, malformed tails, dense arbitrary bytes, NULLs and a non-null fixed-BINARY GenericRowData control. Stock/native median ratios are 1.107-5.306x on 2.2 and 1.105-6.429x on 1.18; previous/native ratios are 1.111-4.950x and 1.194-5.524x. All 24 bulk cells improve against both controls. Four of eight 16-row controls lose to stock; the lowest stock/native ratio is 0.705x. All trials retain warmups, checksums and those slower cells.

Complete-job stock/native medians, seconds, with five measured jobs per engine:

Flink Target Paired stock Native Independent stock Previous fallback
2.2.1 VARCHAR(16) 0.234476 0.211839 0.211336 0.213190
2.2.1 CHAR(16) 0.231994 0.209202 0.216906 0.214053
1.18.1 VARCHAR(16) 0.247202 0.210752 0.211955 0.209260
1.18.1 CHAR(16) 0.233629 0.208619 0.205848 0.209841

All four candidate medians beat paired stock, but independent stock/fallback controls drift. One native median loses to previous fallback and measured ranges overlap. These short jobs do not establish a reliable full-job throughput win. The linked JSON retains all complete-job trials and configuration ranges.

Fresh validation passes 292 focused JVM invocations across Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Twelve existing fixed-BINARY controls abort on Flink 1.18 under their version assumptions; the bounded-cast tests all execute. Twenty-eight generated-expression oracle runs each match 90,404 complete outputs, covering seven targets across all four release/JDK pairs. The Rust core suite passes 892 tests without ignores. Runtime witnesses consume 5,003 rows and require empty evaluator descriptors plus consumed NativeCalc work; lazy branches, all-filtered failures, operand exceptions and represented consumers have explicit SQL regressions. Full hosted CI remains a separate check.

DECIMAL to STRING TRY_CAST

Direct-input TRY_CAST(d AS STRING) reuses the qualified native formatter above. Its previous encoder at 3af56f9a rejects this signature, so the previous control verifies that rejection and measures released Flink's generated fallback; it does not substitute a JVM callback that the old query would never execute. Modern and legacy modes share the same plain formatting. Bounded targets and computed DECIMAL operands retain their existing rules.

The prepared comparison uses released Flink 1.18.1/2.2.1 on JDK 17.0.12, Apple M4 Pro, one thread, and a release/default-feature/mimalloc build. It retains the preceding comparison's RowData and Arrow timer boundaries, UTF-8 length checksum, three DECIMAL types, signed coefficients, optional NULLs, GenericRowData/BinaryRowData and 16/1,024-row batches. Each of the 48 configurations processes 131,072 rows per trial, with 16 warmups and nine measurements per engine. Separate candidate/previous processes retain both stock controls; all timings run serially. Median ratio ranges are:

Flink line Batch rows Stock / native Previous fallback / native
1.18.1 16 1.084–2.549x 1.215–3.082x
1.18.1 1,024 3.103–7.309x 3.451–7.837x
2.2.1 16 1.089–2.716x 1.082–3.022x
2.2.1 1,024 2.830–7.505x 3.104–7.949x

The small-batch results closest to 1x do not establish a reliable speedup. Independent stock controls drift by 0.994–1.491x, so the previous/candidate ratios also contain run variation. The raw trials keep every warmup, measurement and checksum; the result record includes per-configuration min/median/max, build hash and complete-job samples.

The separate complete job processes one million nullable DECIMAL(18,3) rows with a rowwise sequence/map source and blackhole STRING sink at parallelism one. Two warmups and five measurements alternate paired engines. Planning, startup, production JNI and both candidate perimeter transposes remain included. Candidate plans require consumed NativeCalc work and jvm=[]; previous plans require the actual unsupported-TRY_CAST fallback and no native work.

Flink line Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous fallback median (range), s
1.18.1 0.210033 (0.198174–0.341704) 0.178884 (0.175655–0.193307) 0.173271 (0.171626–0.178455) 0.176192 (0.171641–0.181437)
2.2.1 0.200191 (0.189743–0.214374) 0.176218 (0.172520–0.180682) 0.180489 (0.175176–0.203107) 0.171967 (0.163684–0.176673)

Candidate medians improve over their paired stock controls, but the previous fallback is 1.5%/2.5% faster than native. The independent stock drift, overlapping ranges and subsecond startup cost prevent a reliable complete-job improvement claim. These observations do not establish sustained throughput for other jobs.

The focused suite passes 91 tests on each released line/JDK 11/17 pair, 364 invocations with no skips. It retains the CAST regressions and adds TRY_CAST to the all-precision/scale encoding checks, runtime multi-batch witnesses, consumers, NULLs, lazy branches and empty filtered results. Computing a failing operand still fails; bounded TRY_CAST remains fallback. The unchanged formatting kernel keeps its preceding released-DecimalData oracle coverage; this admission change does not replace the full upstream integration suite or CI.

Use the preceding decimal_string commands with SF_DECIMAL_STRING_TRY_CAST=true. For the old control also set SF_DECIMAL_STRING_PREVIOUS=true and prepend an independently compiled 3af56f9a encoder. Complete jobs set SF_BENCHMARK=true and -Dsf.decimal.string.tryCast=true, plus the matching previous property/classpath when required.

STRING IS_ALPHA

The native classifier uses the actual worker JDK's immutable BMP letter bitmap. It copies 8 KiB once during expression compilation, outside the prepared-operation timer; batch and row execution do not call Java. ASCII letters take a byte fast path. The UTF-16-char contract rejects supplementary letters, combining marks and isolated surrogates, and returns nonnullable FALSE for NULL or empty strings.

The original alpha matrix uses nullable generated Java/encoder signatures for both data populations; its nullable flag selects whether rows contain NULL. It does not establish the cost of a declared NOT NULL signature. The separate matching NOT NULL controls below qualify that signature explicitly. Original trials remain available with their source-commit metadata.

Prepared controls use released Flink 1.18.1/2.2.1, JDK 17.0.12, Apple M4 Pro, one thread and a release/default-feature/mimalloc native build. Flink starts with prepared GenericRowData or BinaryRowData and times its released generated expression in Java. StreamFusion starts with equivalent Arrow strings and times the production Calc and a TRUE-count checksum inside Rust. Both engines consume the same logical Boolean results. Perimeter JNI/import/export and row/Arrow transposes are outside this small-operation timer; output allocation and previous behavior's intrinsic JVM callback remain inside.

The 96 configurations cover ASCII letter bodies of 16/128 characters, mixed ASCII, BMP letters, mixed Unicode and supplementary-letter rejection with 16-character bodies, varying prefixes and optional NULL every seven rows. Each independently prepared JVM runs 16 warmups and nine measurements, 131,072 rows per trial, with alternating paired engines. The previous control prepends the actual 3af56f9a encoder and requires its generated evaluator binding. Candidate and previous processes retain separate stock controls. All measurements run serially.

Flink line Batch rows Stock / native median range Previous / native median range
1.18.1 16 0.612–2.389x 9.915–31.988x
1.18.1 1,024 1.321–13.424x 3.189–25.839x
2.2.1 16 0.628–2.359x 10.343–33.583x
2.2.1 1,024 1.312–12.425x 3.058–26.485x

All 1,024-row cells improve over their paired stock controls, including cached GenericRowData. Twenty-three 16-row cells lose to stock: small batches and early-rejecting strings expose fixed Calc/output costs. These losses remain in the raw trials and result record. Independent stock controls drift by 0.761–1.774x; previous/candidate ratios include that variability. The strongest supplementary-string ratios do not describe the cost of scanning an entire letter string.

The separate complete job processes one million nullable mixed-Unicode strings with a rowwise sequence/map source and BOOLEAN NOT NULL blackhole sink at parallelism one. Two warmups and five measurements alternate paired engines. The measured job retains planning, startup, production JNI and both perimeter transposes. Candidate and previous plans both require consumed NativeCalc work, with empty versus present JVM evaluator descriptors respectively.

Flink line Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 0.209614 (0.204382–0.218415) 0.213378 (0.205885–0.222676) 0.218010 (0.205615–0.228056) 0.334182 (0.320680–0.358193)
2.2.1 0.233453 (0.226726–0.262184) 0.213483 (0.209738–0.218223) 0.220989 (0.211911–0.226113) 0.315859 (0.307981–0.323943)

The 2.2 native median improves 8.6% over its paired stock and 32.4% over previous StreamFusion. The 1.18 native median is 1.8% slower than paired stock, with overlapping ranges, while improving 36.2% over previous StreamFusion. These startup-heavy bounded jobs do not establish a universal or sustained whole-job speedup.

Focused SQL checks every BMP unit, supplementary letters, NULL/empty inputs, CHAR casts, consumers, filters, CASE, grouping, selected failures and empty filtered batches on each released Flink/JDK 11/17 pair. The multi-batch witness checks every input ID exactly once, positive native work and jvm=[]. Released upstream ScalarFunctionsTest.testIsAlpha assertions also run unchanged on all four pairs; those local expression tests are not native-work evidence. Noncharacter and sensitive-expression evaluator rules remain separate.

After compiling StringClassificationBenchmarkFunctions, supply the corresponding released runtime/test classpath as SF_NATIVE_BENCH_CLASSPATH and release library directory as SF_NATIVE_LIB_DIR:

cargo build --manifest-path native/Cargo.toml --release --locked -p streamfusion \
  --features mimalloc --lib --example string_classification
native/target/release/examples/string_classification unicode 16 true true 1024 128 16 9

Arguments are corpus, body characters, NULL inputs, BinaryRowData stock input, batch rows, loops, warmups and measured trials. Available corpora are ascii, mixed, bmp, unicode and supplementary. The old control also sets SF_CLASSIFICATION_PREVIOUS=true and prepends an independently compiled 3af56f9a encoder. Complete jobs use StringClassificationPipelineBenchmark, SF_BENCHMARK=true and the matching sf.classification.previous property/classpath pair.

STRING IS_DIGIT

The digit classifier uses the same worker-JDK snapshot lifecycle and Arrow/RowData timing boundaries as IS_ALPHA. A separate 8 KiB BMP digit bitmap preserves Character.isDigit(char); NULL/empty strings, supplementary digits, signs, decimal points, superscripts and Roman numerals return nonnullable FALSE. ASCII decimal digits have a specialized byte loop. There are no batch/row JVM calls in the classifier.

As in the original alpha matrix, generated Java/encoder signatures are nullable in all original digit cells; the flag selects data NULLs. See the separate declared NOT NULL controls for that signature.

The prepared controls cover ASCII digit bodies of 16/128 characters, mixed ASCII, BMP digits, mixed Unicode digits and supplementary-digit rejection with 16-character bodies, varying prefixes, NULL every seven rows, GenericRowData/BinaryRowData and 16/1,024-row batches. Flink 1.18.1/2.2.1, JDK 17.0.12, Apple M4 Pro, one thread and a release/default-feature/ mimalloc build match the alpha controls. Every trial consumes 131,072 Boolean results, with 16 warmups and nine measurements, alternating paired engines and retaining both independent stock controls. The previous process uses the actual 3af56f9a encoder and its generated JVM callback.

Flink line Batch rows Stock / native median range Previous / native median range
1.18.1 16 0.527–3.017x 10.051–32.780x
1.18.1 1,024 1.160–13.203x 2.605–25.935x
2.2.1 16 0.530–2.556x 9.431–31.182x
2.2.1 1,024 1.172–12.236x 2.471–25.662x

Every 1,024-row configuration improves over paired stock, including cached GenericRowData. Twenty-three 16-row cells lose to stock: fixed Calc/output costs remain visible on tiny batches. Independent stock controls drift by 0.710–1.553x. Supplementary-digit rejection does not measure the cost of scanning a full digit string. The raw trials and result record retain these limitations and all losses.

The separate one-million-row jobs use nullable mixed Unicode digits, a rowwise source and BOOLEAN NOT NULL blackhole, parallelism one, two warmups and five measurements. Production JNI, planning/startup and both transposes stay in the job path. Both candidates require consumed NativeCalc work; only the candidate requires empty JVM evaluator descriptors.

Flink line Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 0.208241 (0.206801–0.219251) 0.214863 (0.209152–0.225335) 0.207409 (0.202933–0.213170) 0.331346 (0.322421–0.333293)
2.2.1 0.220444 (0.216505–0.229146) 0.217703 (0.213208–0.220951) 0.214789 (0.209489–0.223645) 0.315420 (0.306975–0.349680)

The 2.2 native median is 1.2% faster than paired stock and 31.0% faster than previous StreamFusion. The 1.18 median is 3.2% slower than paired stock and 35.2% faster than previous StreamFusion. Stock/native ranges overlap on both lines. These startup-heavy jobs do not establish a reliable whole-job win over stock or a universal sustained throughput gain.

Focused classification checks pass 56 JVM invocations across released Flink 1.18.1/2.2.1 and JDK 11/17, including both alpha/digit kernels after shared initialization was factored. Every BMP unit, supplementary characters, NULL/empty inputs, CHAR casts, consumers, grouping, lazy and selected failures, and multi-batch IDs are checked against stock. The unchanged released ScalarFunctionsTest.testIsDigit also passes on all four pairs; these local expression tests do not establish native work. Default-feature release/mimalloc Rust core: 1,004 pass, two pre-existing ignores.

Eight 1,024-row alpha controls after sharing initialization cover ASCII 16/128, nullable mixed Unicode BinaryRowData and BMP GenericRowData on both lines. All remain faster than paired stock, by 1.350–4.314x. Their summaries are in the digit result record and raw trials retain the separate controls.

Reproduce with the IS_ALPHA commands and arguments, additionally setting SF_CLASSIFICATION_DIGIT=true for the native example. For the complete-job launcher set -Dsf.classification.digit=true instead. Both are benchmark controls, not production switches. The old control keeps its matching SF_CLASSIFICATION_PREVIOUS=true or -Dsf.classification.previous=true and independently compiled encoder classpath.

STRING IS_DECIMAL

Flink's predicate tries Java integer, long and double parsing. Native ASCII lexical classification therefore accepts signed integers/fractions, decimal/hexadecimal exponents, floating suffixes, signed NaN/Infinity and magnitude overflow. It trims only U+0000..U+0020 on the double path. A separate worker-JDK snapshot supplies packed BMP radix-10 digit values for Unicode integers, which require signed-long bounds and no trimming. No row/batch Java parser, exception allocation or floating conversion remains in the classifier.

Controls use released Flink 1.18.1/2.2.1, JDK 17.0.12, Apple M4 Pro, one thread and a release/default-feature/mimalloc build. The generated released RowData evaluator and production Arrow Calc consume matching Boolean TRUE counts in their own runtimes. Native timers are inside Rust after Arrow preparation and before export. Perimeter JNI and both transposes are outside this small-operation timer; intrinsic previous JVM callbacks remain inside. Candidate and actual 3af56f9a encoder processes retain independent stock controls.

The 160 prepared configurations cover ASCII digit bodies of 1/6/16/128 characters, fractions, decimal/hexadecimal exponents, mixed invalid/empty strings, BMP/Unicode digits, Unicode integer overflow, special values and supplementary-digit rejection. Each has GenericRowData/ BinaryRowData and nullable/nonnullable controls at 1,024 rows. Six representative corpora also have 16-row controls. Trials process 32,768 rows with 16 warmups and nine measurements, alternating paired engines, with all timing jobs run serially.

Flink line Batch rows Stock / native median range Previous / native median range
1.18.1 16 0.653–54.114x 35.620–81.912x
1.18.1 1,024 2.310–229.729x 10.439–256.494x
2.2.1 16 1.369–59.370x 28.012–84.348x
2.2.1 1,024 3.261–203.293x 9.438–215.964x

All 1,024-row cells improve over paired stock, including cached GenericRowData and actual NOT NULL declarations. Four 16-row 1.18 cells lose to stock; fixed Calc/output cost remains visible on short integer strings. The strongest gains avoid released Flink's caught parser exceptions for fractional, invalid or non-int numeric syntax; they do not describe arbitrary numeric arithmetic or every string input. Independent stock controls drift by 0.423–2.257x, and previous/native ratios include that variability. The result record and raw trials retain every qualified cell and trial.

Separate one-million-row jobs retain production JNI, startup/planning, a rowwise source, BOOLEAN NOT NULL blackhole and both transposes. They use nullable ASCII bodies of six characters or mixed Unicode bodies of sixteen, parallelism one, two warmups and five measurements. Both native modes require consumed Calc work; the candidate additionally requires empty JVM evaluator descriptors. The independent previous process keeps its own stock control.

Flink line / corpus Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / ASCII 6 0.204180 (0.188094–0.212755) 0.192536 (0.189526–0.208814) 0.190818 (0.186239–0.199758) 0.252382 (0.245369–0.265800)
2.2.1 / ASCII 6 0.199403 (0.196239–0.213949) 0.183029 (0.177659–0.187342) 0.211075 (0.209459–0.220351) 0.259388 (0.247681–0.320371)
1.18.1 / Unicode 16 1.459045 (1.447974–1.466384) 0.250724 (0.242249–0.255112) 1.468166 (1.455251–1.487260) 1.802646 (1.780039–1.815647)
2.2.1 / Unicode 16 1.311750 (1.308548–1.336238) 0.237144 (0.225340–0.255713) 1.340659 (1.302556–1.365580) 1.691473 (1.621389–1.739532)

The ASCII native medians improve 5.7%/8.2% over paired stock and 23.7%/29.4% over previous StreamFusion on 1.18/2.2 respectively. Stock/native ranges overlap on 1.18; independent stock controls also vary. Unicode medians improve 5.82x/5.53x over paired stock on these exact jobs: longer Unicode integer strings cause the released predicate to allocate and catch integer-parser failures before trying long parsing. These bounded jobs do not establish a universal or sustained whole-job gain, and neither the source string construction nor the sink/transpose costs are removed.

Focused classification checks pass 76 JVM invocations across released Flink 1.18.1/2.2.1 and JDK 11/17, including the three classifiers after shared initialization changes. The decimal witnesses compare all 65,536 BMP single units and 82,556 embedded-unit, grammar-combination and seeded random strings, plus positive/negative signed-long bounds for every BMP digit, against the released parser and stock runtime SQL. Unique source IDs, complete output counts, consumed native work, no fallback and empty JVM descriptors are required. Unchanged released upstream ScalarFunctionsTest.testIsDecimal passes on all four pairs; local expression assertions are separate from the native witnesses. Default-feature release/mimalloc Rust core: 1,007 pass, two pre-existing ignores.

Reproduce with the classification example and SF_CLASSIFICATION_DECIMAL=true instead of the digit flag. The complete-job launcher takes -Dsf.classification.decimal=true, plus -Dclassification.pipeline.corpus=ascii -Dclassification.pipeline.width=6 or -Dclassification.pipeline.corpus=unicode -Dclassification.pipeline.width=16. These are benchmark-only controls. The example additionally accepts fraction, exponent, hex, special and unicode-overflow corpora. Previous controls prepend the independently compiled main encoder and enable the existing previous-mode flag.

Declared NOT NULL classification controls

These additional controls declare STRING NOT NULL in the actual encoder and generated released Java input signature and use a matching nonnullable Arrow field with no NULL data. They separate that signature from the original alpha/digit matrices, whose no-NULL data populations still used nullable Java signatures. StringClassificationBenchmarkFunctions requires the fixture's data setting to match its signature. The native example passes that setting before creating the JVM; it is not a production option.

The 176 configurations cover 48 alpha, 48 digit and 80 decimal cells on both releases, with the same corpora, prepared timing boundaries, one thread, release/default/mimalloc, 16 warmups and nine measurements. Alpha/digit trials process 131,072 rows and decimal trials 32,768 rows. Every 1,024-row cell beats its paired stock control. Twelve alpha, eleven digit and two decimal 16-row cells lose to stock; no small-batch win is assumed. The decimal NOT NULL cells are included in its 160-cell qualified matrix, not extra independent evidence for that total.

The result record and raw trials retain both stock controls, all variability and each slower case. Existing nullable-signature data remains in the original alpha/digit records with its scope and source commits explicitly identified.

Integer ABS over arithmetic

The admitted producer tree contains only integer inputs, typed literals, recursive lossless widening, nested ABS, addition, subtraction and multiplication. Every arithmetic node wraps to its resolved Java width. Division, remainder, narrowing and other unverified producers retain the complete generated outer expression.

IntegralExpressionBenchmarkFunctions parses real SQL through each released planner and uses the actual encoder, including the independently compiled preceding main encoder. The candidate must have no JVM descriptors; the previous control must bind one. Stock uses the released generated evaluator on prepared GenericRowData/BinaryRowData. The native example prepares matching Arrow input and times production Calc plus equal output checksum consumption inside Rust. Perimeter JNI/import/export and both row/Arrow transposes are outside this small-operation timer; previous intrinsic callbacks stay inside.

The release/default-feature/mimalloc controls use JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups, nine measurements and 131,072 rows per trial. Input declarations and Arrow metadata match the selected nullable/NOT NULL data. The matrix covers all four integer widths, varying values including signed minima, optional NULL every seven rows and both RowData representations. Both ABS(v+1) and ABS((v+1)*(v-1)) use typed literals in 1,024-row batches; INT/BIGINT addition also has 16-row controls. Independent paired-stock processes, checksums and all unfavorable timings are retained.

The initial correct native implementation still lost to stock in twelve bulk cells per release. Result casts after every narrow arithmetic node widened already-correct arrays to BIGINT and reconstructed their original width. Sharing only actual same-width integer arrays removes that work without changing true narrowing or scalar materialization.

Flink line Batch rows Initial stock/native median range Qualified stock/native median range Qualified previous/native median range
1.18.1 16 0.054–0.124x 0.067–0.128x 9.650–10.806x
1.18.1 1,024 0.369–4.570x 1.477–7.808x 11.416–41.864x
2.2.1 16 0.057–0.126x 0.070–0.129x 10.010–11.365x
2.2.1 1,024 0.369–4.637x 1.498–8.087x 10.858–50.934x

All 64 qualified bulk cells beat their paired stock control. All sixteen small-batch cells remain slower than stock: fixed Calc/result costs dominate this tiny computation. Independent stock controls drift by 0.611–1.638x. Initial/qualified processes are separate measurements, so their timing ratios do not isolate a causal effect from that variability. The result record and raw trials retain both matrices, including the rejected initial bulk path and every small-batch loss.

Single-batch allocation probes use the same production operation after compilation, with 1,024 nullable GenericRowData-equivalent inputs on 2.2.1. They count requests/bytes, not retained memory or copied bytes. The accompanying prepared timing cells are reported separately; allocation probes themselves are not elapsed-time evidence.

Input / expression Initial allocation requests / bytes Qualified allocation requests / bytes Initial / qualified median ns per row
INT / addition 52 / 23,876 41 / 10,820 4.151 / 1.259
INT / nested 129 / 61,704 96 / 22,536 11.249 / 2.711
BIGINT / addition 39 / 18,728 39 / 18,728 1.320 / 1.306
BIGINT / nested 76 / 37,399 76 / 37,399 2.471 / 2.473

The BIGINT controls already lack the narrow result casts and retain their allocation counts. Rust tests require pointer-identical same-width slices/NULLs/empty arrays, unchanged scalar materialization and real narrowing. The default-feature release/mimalloc Rust suite passes 1,003 tests with two pre-existing ignores. Eighty focused JVM invocations pass across released 1.18.1/2.2.1 and JDK 11/17, including width/overflow/NULL parity, multi-batch IDs, consumed native work and empty evaluator descriptors. The independently compiled preceding encoder fails the new route regression. Released upstream testAbs passes unchanged on all four pairs; those local assertions are not native-work evidence.

The allocation executable retains its counting wrapper with counts disabled during timing; link-symbol inspection confirms its System calls resolve through the enabled engine aliases (malloc=mi_malloc, free=sf_free). JVM/C++/other-thread allocations are outside the probe.

Separate million-row INT source/blackhole jobs retain production JNI, planning/startup and both transposes. They use nullable inputs, parallelism one, two warmups and five measurements. Both native modes require consumed Calc work; the candidate additionally requires empty evaluator descriptors. Each previous process has its own paired stock.

Flink line / expression Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / addition 0.132738 (0.125002–0.135948) 0.148356 (0.143165–0.164692) 0.125945 (0.121515–0.144187) 0.164539 (0.153559–0.179181)
1.18.1 / nested 0.134089 (0.123711–0.144396) 0.147029 (0.144632–0.161026) 0.124107 (0.120619–0.128548) 0.164153 (0.154318–0.174387)
2.2.1 / addition 0.134425 (0.128763–0.179637) 0.147232 (0.142800–0.159292) 0.128637 (0.124836–0.164790) 0.168241 (0.153037–0.189463)
2.2.1 / nested 0.129607 (0.121807–0.147824) 0.142487 (0.139940–0.152310) 0.128585 (0.121173–0.152629) 0.171499 (0.160844–0.192563)

Native medians improve 9.8–16.9% over previous StreamFusion, but remain 9.5–11.8% slower than paired stock. The 2.2 stock/native ranges overlap; independent stock controls also drift. Source/sink and perimeter conversion costs remain in these startup-heavy bounded jobs. This qualifies the bulk small-operation gate, not a standalone-job or universal sustained throughput win over Flink. Generic small-batch and perimeter costs remain in the numeric follow-up.

Reproduce after compiling the Java fixture and providing its matching released runtime/test classpath as SF_NATIVE_BENCH_CLASSPATH and the release library directory as SF_NATIVE_LIB_DIR:

cargo build --manifest-path native/Cargo.toml --release --locked -p streamfusion \
  --features mimalloc --lib --example integral_expression
SF_INTEGRAL_EXPRESSION='ABS((v+CAST(1 AS $TYPE))*(v-CAST(1 AS $TYPE)))' \
  native/target/release/examples/integral_expression INT true false 1024 128 16 9

The example arguments are SQL integer type, NULL inputs, BinaryRowData input, batch rows, loops, warmups and measurements. The previous control sets SF_INTEGRAL_PREVIOUS=true and prepends the actual preceding encoder to the classpath. Complete jobs use IntegralExpressionPipelineBenchmark, SF_BENCHMARK=true and the corresponding sf.integral.expression / sf.integral.previous properties. These are benchmark controls, not production switches.

Integer unary minus over arithmetic

Unary minus now admits the same recursively verified integer addition, subtraction and multiplication producers as ABS. Every intermediate retains its resolved integer width and wrapping overflow. An input-containing tree uses the physical negative expression; a constant-only tree uses the wrapping scalar kernel because DataFusion's ordinary scalar negative checks overflow. Division, remainder, narrowing and unverified producers retain the complete generated expression.

The prepared controls use the actual preceding main encoder and the production Calc, with the same timing boundaries, release/default-feature/mimalloc build and matching nullable/NOT NULL declarations as the ABS arithmetic matrix. Stock runs released generated Java on prepared GenericRowData/BinaryRowData; native runs matching Arrow with both timing and equal checksum consumption inside Rust. Intrinsic previous JVM work remains timed; entry JNI/import/export and both perimeter transposes remain outside this small-operation timer. The candidate requires an intended native function and empty JVM descriptors; the previous control must bind a JVM evaluator.

JDK 17.0.12 on Apple M4 Pro, one thread, 16 warmups, nine measurements and 131,072 rows per trial cover all four integer widths, NULL/NOT NULL, both row representations, typed addition and nested multiplication. INT/BIGINT addition also has 16-row controls.

Flink line Batch rows Stock/native median range Previous/native median range
1.18.1 16 0.073-0.323x 10.525-29.251x
1.18.1 1,024 1.483-8.502x 12.874-57.345x
2.2.1 16 0.073-0.337x 10.747-30.518x
2.2.1 1,024 1.507-9.339x 13.313-74.468x

All 64 bulk cells beat paired stock; all sixteen tiny-batch cells remain slower. Fixed Calc/result costs dominate tiny computations. Independent paired-stock controls drift by 0.663-1.353x. The result record and raw trials retain every measurement, including slower controls and allocation probes.

Separate million-row nullable INT source/blackhole jobs keep production JNI, startup, planning and both transposes, with parallelism one, two warmups and five measurements. Both native controls require consumed Calc work; the candidate requires empty evaluators.

Flink line / expression Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / addition 0.126943 (0.123006-0.136017) 0.145248 (0.141566-0.159738) 0.130385 (0.120768-0.146977) 0.166026 (0.154087-0.170880)
1.18.1 / nested 0.127378 (0.119838-0.136126) 0.142236 (0.138390-0.162559) 0.129773 (0.123038-0.134026) 0.168336 (0.155652-0.171366)
2.2.1 / addition 0.137106 (0.130255-0.147236) 0.149406 (0.140301-0.164516) 0.138002 (0.126575-0.175749) 0.174674 (0.159422-0.202962)
2.2.1 / nested 0.127712 (0.123361-0.134710) 0.145771 (0.142729-0.152221) 0.125265 (0.122946-0.132947) 0.163662 (0.158853-0.182695)

Native medians improve 10.9-15.5% over previous StreamFusion, but remain 9.0-14.4% slower than paired stock. The 2.2 addition ranges overlap. This qualifies prepared bulk computation, not whole-job acceleration: fixed Calc, source/sink and perimeter costs remain in these bounded jobs. Tiny-batch/perimeter optimization stays in the numeric follow-up.

Correctness checks pass 160 focused JVM invocations across released Flink 1.18.1/2.2.1 and JDK 11/17, plus eight strengthened encoder checks. Consumed runtime jobs compare every ID, value, type and NULL across batches and require empty JVM descriptors and no fallback. Sixteen dedicated JVM-started JNI invocations additionally test four signed widths, both constant operand orders and zero/one/1,023 rows: all 96 batches preserve cardinality, types and wrapping signed minima after the input closes. The unchanged released upstream unary-plus/minus method passes on all four pairs. These local upstream assertions are separate from native-work evidence. Java compiles for release 11. No new Rust production code is added beyond the separately tested ABS identity-cast optimization.

Reproduce with the same integral_expression fixture and preceding-encoder controls as the ABS matrix, using SF_INTEGRAL_EXPRESSION='-((v+CAST(1 AS $TYPE))*(v-CAST(1 AS $TYPE)))'. Complete jobs use the corresponding sf.integral.expression property.

Integer SIGN over arithmetic

The existing native SIGN implementation now admits recursively verified integer addition, subtraction and multiplication over its input/literal, lossless-widening and nested-function whitelist. Each arithmetic node keeps its resolved signed width and Java overflow wrapping. Division, remainder, narrowing, DECIMAL arithmetic and other unverified producers retain the complete generated expression. No configuration switch is added.

The result record and raw trials retain all paired controls, warmups, measurements, allocation probes and complete-job samples. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups, nine measurements and 131,072 rows per trial compare released generated Java on prepared GenericRowData/BinaryRowData with production Calc on matching Arrow. Native timers and equivalent checksum consumption are inside Rust; initial JNI/import/export and both perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation remains timed. The actual independently compiled preceding main encoder is required to bind an evaluator; the candidate requires the intended native function and empty JVM descriptors. Nullable/NOT NULL declarations match both inputs, including actual NULLs every seven rows where selected.

Typed addition and nested multiplication cover every supported integer width and both row representations at 1,024 rows; INT/BIGINT addition also has 16-row controls. SIGN and TRUNCATE retain released Flink's direct narrow-integer compilation failures and use INT/BIGINT here.

Flink line Batch rows Stock/native median range Previous/native median range Slower than stock
1.18.1 16 0.071-0.128x 9.673-11.315x 8 / 8
1.18.1 1,024 1.540-5.768x 10.535-34.869x 0 / 16
2.2.1 16 0.072-0.127x 10.139-11.226x 8 / 8
2.2.1 1,024 1.500-5.587x 10.989-41.784x 0 / 16

The bulk matrix has 0 slower cells out of 32; the tiny-batch matrix has 16 slower cells out of 16. Fixed Calc/result costs remain visible on small computations. Independent stock-control drift ranges from 0.443 to 1.453x; cross-process ratios do not isolate variability.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes. Parallelism is one, with two warmups and five measurements. Both native controls require consumed Calc work; the candidate requires empty descriptors.

Flink line / expression Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / add 0.130256 (0.124880-0.139337) 0.143118 (0.137992-0.154661) 0.136756 (0.134128-0.150880) 0.169804 (0.165546-0.210110)
1.18.1 / nested 0.136361 (0.123144-0.192691) 0.154559 (0.147487-0.170228) 0.122213 (0.119685-0.153817) 0.165679 (0.156633-0.174462)
2.2.1 / add 0.135250 (0.126187-0.152305) 0.147599 (0.141288-0.175990) 0.131027 (0.121526-0.137569) 0.164077 (0.151389-0.210137)
2.2.1 / nested 0.129468 (0.122160-0.140752) 0.156528 (0.143648-0.181162) 0.136065 (0.122717-0.150573) 0.159781 (0.159538-0.207609)

Native/paired-stock elapsed-time ratios range from 1.091 to 1.209; native/previous ratios range from 0.843 to 0.980. Ratios above one are slower. Retain overlapping ranges, independent stock drift and unfavorable complete jobs. Prepared bulk qualification does not establish a universal or complete-job throughput improvement. Remaining small-batch and perimeter costs stay in the numeric follow-up.

All 100 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17. Runtime witnesses compare complete values, types, NULLs, cardinality and unique IDs across multiple batches, and require consumed native work, no fallback and empty JVM descriptors. Existing selected/unselected failures, empty filtered batches and unverified-producer guards remain covered. The unchanged released upstream method also passes on all four pairs; its local expression assertions supplement the independent runtime native-work witnesses. Both Java lines compile for release 11. No new Rust production code is introduced by this producer-admission change.

Reproduce with integral_expression and IntegralExpressionPipelineBenchmark, setting SF_INTEGRAL_EXPRESSION='SIGN((v+CAST(1 AS $TYPE))*(v-CAST(1 AS $TYPE)))' or the matching sf.integral.expression property. As in the ABS controls, the actual previous encoder is prepended to its own classpath and qualified with SF_INTEGRAL_PREVIOUS=true.

Integer FLOOR over arithmetic

The existing native FLOOR implementation now admits recursively verified integer addition, subtraction and multiplication over its input/literal, lossless-widening and nested-function whitelist. Each arithmetic node keeps its resolved signed width and Java overflow wrapping. Division, remainder, narrowing, DECIMAL arithmetic and other unverified producers retain the complete generated expression. No configuration switch is added.

The result record and raw trials retain all paired controls, warmups, measurements, allocation probes and complete-job samples. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups, nine measurements and 131,072 rows per trial compare released generated Java on prepared GenericRowData/BinaryRowData with production Calc on matching Arrow. Native timers and equivalent checksum consumption are inside Rust; initial JNI/import/export and both perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation remains timed. The actual independently compiled preceding main encoder is required to bind an evaluator; the candidate requires the intended native function and empty JVM descriptors. Nullable/NOT NULL declarations match both inputs, including actual NULLs every seven rows where selected.

Typed addition and nested multiplication cover every supported integer width and both row representations at 1,024 rows; INT/BIGINT addition also has 16-row controls. SIGN and TRUNCATE retain released Flink's direct narrow-integer compilation failures and use INT/BIGINT here.

Flink line Batch rows Stock/native median range Previous/native median range Slower than stock
1.18.1 16 0.073-0.385x 10.848-33.897x 8 / 8
1.18.1 1,024 1.526-11.036x 11.725-85.888x 0 / 32
2.2.1 16 0.078-0.377x 11.285-34.749x 8 / 8
2.2.1 1,024 1.568-12.272x 10.653-90.189x 0 / 32

The bulk matrix has 0 slower cells out of 64; the tiny-batch matrix has 16 slower cells out of 16. Fixed Calc/result costs remain visible on small computations. Independent stock-control drift ranges from 0.592 to 1.601x; cross-process ratios do not isolate variability.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes. Parallelism is one, with two warmups and five measurements. Both native controls require consumed Calc work; the candidate requires empty descriptors.

Flink line / expression Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / add 0.123600 (0.122369-0.129598) 0.151265 (0.137397-0.159724) 0.126829 (0.118631-0.138255) 0.164821 (0.157293-0.189417)
1.18.1 / nested 0.125440 (0.118526-0.154963) 0.140254 (0.138623-0.154078) 0.127278 (0.122852-0.136469) 0.163857 (0.157304-0.179982)
2.2.1 / add 0.143825 (0.125135-0.170857) 0.149311 (0.146698-0.163415) 0.132833 (0.129114-0.141480) 0.167962 (0.162157-0.214726)
2.2.1 / nested 0.128320 (0.122758-0.130398) 0.145575 (0.143623-0.154209) 0.137148 (0.122207-0.147096) 0.166432 (0.160759-0.177441)

Native/paired-stock elapsed-time ratios range from 1.038 to 1.224; native/previous ratios range from 0.856 to 0.918. Ratios above one are slower. Retain overlapping ranges, independent stock drift and unfavorable complete jobs. Prepared bulk qualification does not establish a universal or complete-job throughput improvement. Remaining small-batch and perimeter costs stay in the numeric follow-up.

All 132 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17. Runtime witnesses compare complete values, types, NULLs, cardinality and unique IDs across multiple batches, and require consumed native work, no fallback and empty JVM descriptors. Existing selected/unselected failures, empty filtered batches and unverified-producer guards remain covered. The unchanged released upstream method also passes on all four pairs; its local expression assertions supplement the independent runtime native-work witnesses. Both Java lines compile for release 11. No new Rust production code is introduced by this producer-admission change.

Reproduce with integral_expression and IntegralExpressionPipelineBenchmark, setting SF_INTEGRAL_EXPRESSION='FLOOR((v+CAST(1 AS $TYPE))*(v-CAST(1 AS $TYPE)))' or the matching sf.integral.expression property. As in the ABS controls, the actual previous encoder is prepended to its own classpath and qualified with SF_INTEGRAL_PREVIOUS=true.

Integer CEIL over arithmetic

The existing native CEIL implementation now admits recursively verified integer addition, subtraction and multiplication over its input/literal, lossless-widening and nested-function whitelist. Each arithmetic node keeps its resolved signed width and Java overflow wrapping. Division, remainder, narrowing, DECIMAL arithmetic and other unverified producers retain the complete generated expression. No configuration switch is added.

The result record and raw trials retain all paired controls, warmups, measurements, allocation probes and complete-job samples. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups, nine measurements and 131,072 rows per trial compare released generated Java on prepared GenericRowData/BinaryRowData with production Calc on matching Arrow. Native timers and equivalent checksum consumption are inside Rust; initial JNI/import/export and both perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation remains timed. The actual independently compiled preceding main encoder is required to bind an evaluator; the candidate requires the intended native function and empty JVM descriptors. Nullable/NOT NULL declarations match both inputs, including actual NULLs every seven rows where selected.

Typed addition and nested multiplication cover every supported integer width and both row representations at 1,024 rows; INT/BIGINT addition also has 16-row controls. SIGN and TRUNCATE retain released Flink's direct narrow-integer compilation failures and use INT/BIGINT here.

Flink line Batch rows Stock/native median range Previous/native median range Slower than stock
1.18.1 16 0.077-0.375x 10.891-34.160x 8 / 8
1.18.1 1,024 1.555-11.353x 11.566-82.999x 0 / 32
2.2.1 16 0.078-0.363x 11.241-34.910x 8 / 8
2.2.1 1,024 1.553-12.186x 11.711-112.828x 0 / 32

The bulk matrix has 0 slower cells out of 64; the tiny-batch matrix has 16 slower cells out of 16. Fixed Calc/result costs remain visible on small computations. Independent stock-control drift ranges from 0.417 to 1.414x; cross-process ratios do not isolate variability.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes. Parallelism is one, with two warmups and five measurements. Both native controls require consumed Calc work; the candidate requires empty descriptors.

Flink line / expression Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / add 0.130198 (0.122790-0.137458) 0.157354 (0.138733-0.168121) 0.121527 (0.119802-0.135765) 0.163856 (0.155947-0.169186)
1.18.1 / nested 0.129851 (0.120823-0.152799) 0.141774 (0.138176-0.156186) 0.128809 (0.127996-0.153699) 0.165188 (0.161602-0.178558)
2.2.1 / add 0.126569 (0.122766-0.131987) 0.148518 (0.143178-0.160139) 0.126695 (0.121754-0.140802) 0.162624 (0.157716-0.181336)
2.2.1 / nested 0.126593 (0.121725-0.168759) 0.147910 (0.141753-0.154536) 0.126083 (0.119509-0.155248) 0.166177 (0.162577-0.186416)

Native/paired-stock elapsed-time ratios range from 1.092 to 1.209; native/previous ratios range from 0.858 to 0.960. Ratios above one are slower. Retain overlapping ranges, independent stock drift and unfavorable complete jobs. Prepared bulk qualification does not establish a universal or complete-job throughput improvement. Remaining small-batch and perimeter costs stay in the numeric follow-up.

All 152 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17. Runtime witnesses compare complete values, types, NULLs, cardinality and unique IDs across multiple batches, and require consumed native work, no fallback and empty JVM descriptors. Existing selected/unselected failures, empty filtered batches and unverified-producer guards remain covered. The unchanged released upstream method also passes on all four pairs; its local expression assertions supplement the independent runtime native-work witnesses. Both Java lines compile for release 11. No new Rust production code is introduced by this producer-admission change.

Reproduce with integral_expression and IntegralExpressionPipelineBenchmark, setting SF_INTEGRAL_EXPRESSION='CEIL((v+CAST(1 AS $TYPE))*(v-CAST(1 AS $TYPE)))' or the matching sf.integral.expression property. As in the ABS controls, the actual previous encoder is prepended to its own classpath and qualified with SF_INTEGRAL_PREVIOUS=true.

Integer ROUND over arithmetic

The existing native ROUND implementation now admits recursively verified integer addition, subtraction and multiplication over its input/literal, lossless-widening and nested-function whitelist. Each arithmetic node keeps its resolved signed width and Java overflow wrapping. Division, remainder, narrowing, DECIMAL arithmetic and other unverified producers retain the complete generated expression. No configuration switch is added.

The result record and raw trials retain all paired controls, warmups, measurements, allocation probes and complete-job samples. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups, nine measurements and 131,072 rows per trial compare released generated Java on prepared GenericRowData/BinaryRowData with production Calc on matching Arrow. Native timers and equivalent checksum consumption are inside Rust; initial JNI/import/export and both perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation remains timed. The actual independently compiled preceding main encoder is required to bind an evaluator; the candidate requires the intended native function and empty JVM descriptors. Nullable/NOT NULL declarations match both inputs, including actual NULLs every seven rows where selected.

Typed addition and nested multiplication cover every supported integer width and both row representations at 1,024 rows; INT/BIGINT addition also has 16-row controls. ROUND covers all four integer widths.

Additional nullable Generic/Binary cells apply literal INT positions -1, -38 and 38 to the nested arithmetic expression over every admitted width. NULL-position cells are retained separately as constant-result controls and excluded from function-execution qualification.

Flink line Batch rows Stock/native median range Previous/native median range Slower than stock
1.18.1 16 0.084-0.384x 11.029-35.381x 8 / 8
1.18.1 1,024 1.805-35.430x 10.527-98.619x 0 / 56
2.2.1 16 0.091-0.397x 11.018-35.726x 8 / 8
2.2.1 1,024 1.832-35.412x 11.169-105.774x 0 / 56

The bulk matrix has 0 slower cells out of 112; the tiny-batch matrix has 16 slower cells out of 16. Fixed Calc/result costs remain visible on small computations. Independent stock-control drift ranges from 0.284 to 6.389x; cross-process ratios do not isolate variability.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes. Parallelism is one, with two warmups and five measurements. Both native controls require consumed Calc work; the candidate requires empty descriptors. The position-add/position-nested job rows apply the explicit literal position -1.

Flink line / expression Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / add 0.138509 (0.128871-0.141295) 0.151582 (0.149946-0.187205) 0.135664 (0.129138-0.147661) 0.179961 (0.172285-0.213695)
1.18.1 / nested 0.131959 (0.126554-0.138073) 0.148946 (0.141745-0.190673) 0.133393 (0.127564-0.136626) 0.175208 (0.171869-0.184764)
1.18.1 / position-add 0.143669 (0.135566-0.165214) 0.146206 (0.144089-0.168085) 0.141263 (0.139604-0.155608) 0.175963 (0.164515-0.233102)
1.18.1 / position-nested 0.155569 (0.146503-0.156909) 0.157583 (0.143202-0.174439) 0.158029 (0.141994-0.164224) 0.193834 (0.177136-0.201926)
2.2.1 / add 0.138633 (0.135453-0.148245) 0.149119 (0.143823-0.155219) 0.136070 (0.132937-0.150031) 0.178963 (0.170773-0.199273)
2.2.1 / nested 0.132262 (0.128172-0.159622) 0.143343 (0.136777-0.161370) 0.136466 (0.126677-0.165135) 0.172319 (0.167299-0.197420)
2.2.1 / position-add 0.146133 (0.142373-0.166690) 0.147639 (0.145069-0.161272) 0.147190 (0.132771-0.152431) 0.170306 (0.165477-0.186386)
2.2.1 / position-nested 0.145609 (0.142876-0.159408) 0.143521 (0.142514-0.155381) 0.146946 (0.140844-0.160657) 0.187572 (0.177320-0.211928)

Native/paired-stock elapsed-time ratios range from 0.986 to 1.129; native/previous ratios range from 0.765 to 0.867. Ratios above one are slower. Retain overlapping ranges, independent stock drift and unfavorable complete jobs. Prepared bulk qualification does not establish a universal or complete-job throughput improvement. Remaining small-batch and perimeter costs stay in the numeric follow-up.

All 240 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17. Runtime witnesses compare complete values, types, NULLs, cardinality and unique IDs across multiple batches, and require consumed native work, no fallback and empty JVM descriptors. Existing selected/unselected failures, empty filtered batches and unverified-producer guards remain covered. The unchanged released upstream method also passes on all four pairs; its local expression assertions supplement the independent runtime native-work witnesses. Both Java lines compile for release 11. No new Rust production code is introduced by this producer-admission change.

Reproduce with integral_expression and IntegralExpressionPipelineBenchmark, setting SF_INTEGRAL_EXPRESSION='ROUND((v+CAST(1 AS $TYPE))*(v-CAST(1 AS $TYPE)))' or the matching sf.integral.expression property. As in the ABS controls, the actual previous encoder is prepended to its own classpath and qualified with SF_INTEGRAL_PREVIOUS=true.

The 16 constant-NULL fold controls have paired stock/native median ratios from 2.425 to 23.015x. They measure production Calc result materialization and consumption after a bounded proof excludes operand failures; they do not establish executed function-kernel work. Division, remainder, narrowing and other unverified producers retain evaluation even when the position is NULL.

The initial eager NULL-position implementation lost to stock in all eight diagnostic cells (0.65-0.71x stock/native). Eliminating the verified failure-free operand tree moves matched diagnostic medians from 2.50-2.72 to 0.62-0.72 ns/row and the paired ratios to 2.48-2.80x, with independent process/stock variation retained. Untimed Rust allocation requests drop from 79-99 to 16 per batch; these exclude JVM/other-thread allocations and are not retained memory. The initial trials and folded trials supplement the main result record.

All 116 additional shared typed-NULL encoding/SQL invocations pass across the same four version/JDK pairs, covering precision, nested fields, temporal types, unknown-type guards and partial final batches.

Integer TRUNCATE over arithmetic

The existing native TRUNCATE implementation now admits recursively verified integer addition, subtraction and multiplication over its input/literal, lossless-widening and nested-function whitelist. Each arithmetic node keeps its resolved signed width and Java overflow wrapping. Division, remainder, narrowing, DECIMAL arithmetic and other unverified producers retain the complete generated expression. No configuration switch is added.

The result record and raw trials retain all paired controls, warmups, measurements, allocation probes and complete-job samples. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups, nine measurements and 131,072 rows per trial compare released generated Java on prepared GenericRowData/BinaryRowData with production Calc on matching Arrow. Native timers and equivalent checksum consumption are inside Rust; initial JNI/import/export and both perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation remains timed. The actual independently compiled preceding main encoder is required to bind an evaluator; the candidate requires the intended native function and empty JVM descriptors. Nullable/NOT NULL declarations match both inputs, including actual NULLs every seven rows where selected.

Typed addition and nested multiplication cover every supported integer width and both row representations at 1,024 rows; INT/BIGINT addition also has 16-row controls. TRUNCATE retains released Flink's direct narrow-integer compilation failures and uses INT/BIGINT here.

Additional nullable Generic/Binary cells apply literal INT positions -1, -38 and 38 to the nested arithmetic expression over every admitted width. NULL-position cells are retained separately as constant-result controls and excluded from function-execution qualification.

Flink line Batch rows Stock/native median range Previous/native median range Slower than stock
1.18.1 16 0.978-8.364x 12.298-41.801x 1 / 8
1.18.1 1,024 10.545-277.690x 19.304-317.218x 0 / 28
2.2.1 16 0.970-7.944x 12.495-40.313x 1 / 8
2.2.1 1,024 10.417-281.861x 19.904-333.712x 0 / 28

The bulk matrix has 0 slower cells out of 56; the tiny-batch matrix has 2 slower cells out of 16. Fixed Calc/result costs remain visible on small computations. Independent stock-control drift ranges from 0.271 to 1.448x; cross-process ratios do not isolate variability.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes. Parallelism is one, with two warmups and five measurements. Both native controls require consumed Calc work; the candidate requires empty descriptors. The position-add/position-nested job rows apply the explicit literal position -1.

Flink line / expression Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / add 0.193199 (0.183309-0.200168) 0.148608 (0.138675-0.165526) 0.190651 (0.188763-0.206507) 0.261375 (0.241094-0.283575)
1.18.1 / nested 0.200089 (0.190741-0.203405) 0.151694 (0.145246-0.160465) 0.200865 (0.188948-0.208953) 0.251374 (0.248945-0.270095)
1.18.1 / position-add 0.196940 (0.188253-0.201866) 0.151558 (0.143435-0.181935) 0.195267 (0.192303-0.203604) 0.219569 (0.216203-0.231150)
1.18.1 / position-nested 0.200262 (0.191257-0.219927) 0.153747 (0.147628-0.165354) 0.207347 (0.203120-0.215704) 0.236552 (0.231091-0.247184)
2.2.1 / add 0.192395 (0.188942-0.201869) 0.152104 (0.145143-0.156501) 0.189003 (0.181838-0.199292) 0.268802 (0.263646-0.283940)
2.2.1 / nested 0.206748 (0.196520-0.222838) 0.144713 (0.142744-0.166039) 0.205656 (0.198518-0.212245) 0.250286 (0.236342-0.257092)
2.2.1 / position-add 0.209153 (0.200628-0.220100) 0.147064 (0.143148-0.172021) 0.206633 (0.199655-0.210218) 0.222729 (0.217287-0.230073)
2.2.1 / position-nested 0.215521 (0.212657-0.222801) 0.149474 (0.144635-0.154722) 0.213070 (0.206850-0.220141) 0.244035 (0.233602-0.270349)

Native/paired-stock elapsed-time ratios range from 0.694 to 0.791; native/previous ratios range from 0.566 to 0.690. Ratios above one are slower. These eight complete-job medians improve by 20.9-30.6% over paired stock and 31.0-43.4% over previous StreamFusion. Retain overlapping ranges and independent stock drift; the measured workloads do not establish a universal complete-job throughput improvement. Remaining small-batch and perimeter costs stay in the numeric follow-up.

All 236 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17. Runtime witnesses compare complete values, types, NULLs, cardinality and unique IDs across multiple batches, and require consumed native work, no fallback and empty JVM descriptors. Existing selected/unselected failures, empty filtered batches and unverified-producer guards remain covered. The unchanged released upstream method also passes on all four pairs; its local expression assertions supplement the independent runtime native-work witnesses. Both Java lines compile for release 11. No new Rust production code is introduced by this producer-admission change.

Reproduce with integral_expression and IntegralExpressionPipelineBenchmark, setting SF_INTEGRAL_EXPRESSION='TRUNCATE((v+CAST(1 AS $TYPE))*(v-CAST(1 AS $TYPE)))' or the matching sf.integral.expression property. As in the ABS controls, the actual previous encoder is prepended to its own classpath and qualified with SF_INTEGRAL_PREVIOUS=true.

The 8 constant-NULL fold controls have paired stock/native median ratios from 2.392 to 7.203x. They measure production Calc result materialization and consumption after a bounded proof excludes operand failures; they do not establish executed function-kernel work. Division, remainder, narrowing and other unverified producers retain evaluation even when the position is NULL.

Numeric IS_ALPHA

Released Flink receives boxed primitive integers or internal DecimalData in this overload. All admitted numeric arguments return FALSE, including runtime/literal NULLs. Direct inputs/literals keep a native type-based predicate. Only the recursively verified failure-free integer producer scope folds away unused arithmetic; division, remainder, narrowing, UDFs and other unverified producers retain the complete generated call, including failures and state changes. Floating signatures remain generated. This preserves the released behavior rather than inferring it from Java's public BigDecimal API.

The final admission guard additionally keeps unverified producers generated. An untimed comparison on each Flink line confirms that all 28 distinct measured direct/add/nested signatures retain identical encoded programs and constant pools; the result record keeps both measured/final source hashes and final-guard correctness validation.

The result record and raw trials retain 80 configurations and 8,000 main trials. Direct runtime arguments retain the native call; arithmetic fold controls are explicitly excluded from predicate-kernel qualification. The direct matrix covers four integer widths and DECIMAL(18,2)/(38,18), nullable/NOT NULL and GenericRowData/BinaryRowData at 1,024 rows. Tiny INT/BIGINT controls use 16 rows. Typed addition/nested multiplication provide separate fold diagnostics. No input values are converted to strings.

Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial compare actual released generated Java on prepared RowData with production Calc on matching Arrow. Native operation and equivalent Boolean result consumption are timed inside Rust; initial JNI/import/export and perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation stays inside. The actual independently compiled preceding-main encoder must bind an evaluator; the candidate requires empty descriptors and the intended function before the bounded fold. Both sides retain exact types, widths, signed boundaries and matching NULLs.

Flink line Rows / measured work Stock/native median range Previous/native median range Slower than stock
1.18.1 1,024 / predicate 4.338-75.846x 57.267-83.709x 0 / 24
1.18.1 16 / predicate 0.132-0.177x 29.727-32.708x 8 / 8
1.18.1 1,024 / fold control 5.214-14.863x 80.878-156.067x 0 / 8
2.2.1 1,024 / predicate 4.378-74.153x 56.493-174.781x 0 / 24
2.2.1 16 / predicate 0.132-0.167x 29.850-33.778x 8 / 8
2.2.1 1,024 / fold control 5.669-10.663x 82.630-211.918x 0 / 8

Ratios above one favor native. The type-based predicate returns a Boolean scalar which production Calc materializes to the batch length; the measurements do not claim a per-row character scan. Small-batch fixed Calc/result costs and every unfavorable sample remain in the record. Independent paired-stock drift ranges from 0.474 to 4.622x; cross-process ratios are not causal isolation.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes, with parallelism one and two warmups/five measurements. Direct jobs retain the predicate. Add/nested jobs exercise the bounded fold and must not be cited as predicate-kernel work. All require consumed native Calc work, no fallback and empty candidate evaluator descriptors.

Flink line / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / direct 0.144208 (0.139951-0.185509) 0.158264 (0.155906-0.180063) 0.150093 (0.140484-0.155497) 0.190045 (0.184045-0.214745)
1.18.1 / add 0.158886 (0.138947-0.184109) 0.153646 (0.148057-0.187208) 0.144336 (0.143366-0.162540) 0.180045 (0.173658-0.223902)
1.18.1 / nested 0.147116 (0.143477-0.157420) 0.153543 (0.149247-0.156138) 0.148057 (0.140475-0.163178) 0.179666 (0.174594-0.195992)
2.2.1 / direct 0.171778 (0.156760-0.184056) 0.163535 (0.162832-0.170514) 0.155110 (0.150512-0.163980) 0.192114 (0.179358-0.216947)
2.2.1 / add 0.159357 (0.151531-0.164622) 0.151333 (0.150413-0.174848) 0.152845 (0.150406-0.165735) 0.190004 (0.185241-0.206299)
2.2.1 / nested 0.155059 (0.149082-0.160268) 0.150045 (0.143815-0.165674) 0.170340 (0.152518-0.174406) 0.191185 (0.188699-0.243760)

Across these jobs, native/stock elapsed-time ratios range from 0.950 to 1.097; native/previous ratios range from 0.785 to 0.855. Ratios above one are slower. Retain overlapping ranges and independent stock drift; prepared bulk qualification does not establish universal whole-job acceleration.

All 56 focused JVM invocations and unchanged released IS_ALPHA expression tests pass on Flink 1.18.1/2.2.1 with JDK 11/17. Eight numeric declarations, 5,003 rows, nullable and NOT NULL inputs, literal NULLs and Boolean consumers compare complete values, resolved types, cardinality and unique IDs across multiple batches. Direct runtime witnesses require consumed work, no fallback and empty descriptors. Separate failure/empty-batch tests and four additional shared-state invocations prove that arguments are not discarded. Java compiles for release 11; the release/default/mimalloc Rust core suite passes 1,017 tests with two pre-existing ignores.

Reproduce with integral_expression, for example SF_INTEGRAL_EXPRESSION='IS_ALPHA(v)', and IntegralExpressionPipelineBenchmark with sf.integral.expression=IS_ALPHA(v). Prepended independently compiled previous-encoder classes and SF_INTEGRAL_PREVIOUS=true select the preceding callback control. Remaining coverage stays in the classification follow-up.

The initial nullable INT arithmetic probe loses to stock (0.882x stock/native). The bounded fold moves the matched native median from 2.352 to 0.381 ns/row, with a paired stock/native ratio of 5.815x after the change. The initial and folded probe trials retain direct/tiny controls, allocation probes and independent process variation. They are optimizer diagnostics, separate from the direct predicate qualification.

Numeric IS_DIGIT

Released Flink receives boxed primitive integers or internal DecimalData in this overload. Integral arguments return TRUE regardless of sign or runtime/literal NULL; internal DecimalData arguments return FALSE. Direct inputs/literals keep a native type-based predicate. Only the recursively verified failure-free integer producer scope folds away unused arithmetic; division, remainder, narrowing, UDFs and other unverified producers retain the complete generated call, including failures and state changes. Floating signatures remain generated. This preserves the released behavior rather than inferring it from Java's public BigDecimal API.

The result record and raw trials retain 80 configurations and 8,000 main trials. Direct runtime arguments retain the native call; arithmetic fold controls are explicitly excluded from predicate-kernel qualification. The direct matrix covers four integer widths and DECIMAL(18,2)/(38,18), nullable/NOT NULL and GenericRowData/BinaryRowData at 1,024 rows. Tiny INT/BIGINT controls use 16 rows. Typed addition/nested multiplication provide separate fold diagnostics. No input values are converted to strings.

Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial compare actual released generated Java on prepared RowData with production Calc on matching Arrow. Native operation and equivalent Boolean result consumption are timed inside Rust; initial JNI/import/export and perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation stays inside. The actual independently compiled preceding-main encoder must bind an evaluator; the candidate requires empty descriptors and the intended function before the bounded fold. Both sides retain exact types, widths, signed boundaries and matching NULLs.

Flink line Rows / measured work Stock/native median range Previous/native median range Slower than stock
1.18.1 1,024 / predicate 4.327-76.960x 56.537-83.467x 0 / 24
1.18.1 16 / predicate 0.146-0.207x 29.999-32.504x 8 / 8
1.18.1 1,024 / fold control 5.655-9.284x 86.747-124.336x 0 / 8
2.2.1 1,024 / predicate 4.403-52.638x 48.010-73.082x 0 / 24
2.2.1 16 / predicate 0.138-0.187x 30.394-32.411x 8 / 8
2.2.1 1,024 / fold control 5.822-9.055x 68.054-78.744x 0 / 8

Ratios above one favor native. The type-based predicate returns a Boolean scalar which production Calc materializes to the batch length; the measurements do not claim a per-row character scan. Small-batch fixed Calc/result costs and every unfavorable sample remain in the record. Independent paired-stock drift ranges from 0.585 to 9.993x; cross-process ratios are not causal isolation.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes, with parallelism one and two warmups/five measurements. Direct jobs retain the predicate. Add/nested jobs exercise the bounded fold and must not be cited as predicate-kernel work. All require consumed native Calc work, no fallback and empty candidate evaluator descriptors.

Flink line / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / direct 0.147362 (0.143816-0.187192) 0.165473 (0.163363-0.191847) 0.155385 (0.145714-0.159703) 0.188236 (0.181853-0.326542)
1.18.1 / add 0.152047 (0.144916-0.158812) 0.159139 (0.153293-0.174958) 0.150465 (0.142756-0.192243) 0.186863 (0.182414-0.205804)
1.18.1 / nested 0.152921 (0.146317-0.182007) 0.155172 (0.151606-0.171286) 0.148784 (0.140122-0.150412) 0.181575 (0.178331-0.194795)
2.2.1 / direct 0.152754 (0.144340-0.183355) 0.167731 (0.157047-0.171134) 0.154388 (0.149800-0.160568) 0.177077 (0.176169-0.203367)
2.2.1 / add 0.159082 (0.157465-0.169012) 0.158240 (0.151098-0.166858) 0.158511 (0.149533-0.166617) 0.180761 (0.176107-0.188447)
2.2.1 / nested 0.154696 (0.144169-0.162562) 0.152258 (0.149800-0.169870) 0.155161 (0.145043-0.165499) 0.185097 (0.180521-0.209116)

Across these jobs, native/stock elapsed-time ratios range from 0.984 to 1.123; native/previous ratios range from 0.823 to 0.947. Ratios above one are slower. Retain overlapping ranges and independent stock drift; prepared bulk qualification does not establish universal whole-job acceleration.

All 96 focused JVM invocations and unchanged released IS_DIGIT expression tests pass on Flink 1.18.1/2.2.1 with JDK 11/17. Eight numeric declarations, 5,003 rows, nullable and NOT NULL inputs, literal NULLs and Boolean consumers compare complete values, resolved types, cardinality and unique IDs across multiple batches. Direct runtime witnesses require consumed work, no fallback and empty descriptors. Separate failure/empty-batch tests and four additional shared-state invocations prove that arguments are not discarded. Java compiles for release 11; the release/default/mimalloc Rust core suite passes 1,017 tests with two pre-existing ignores.

Reproduce with integral_expression, for example SF_INTEGRAL_EXPRESSION='IS_DIGIT(v)', and IntegralExpressionPipelineBenchmark with sf.integral.expression=IS_DIGIT(v). Prepended independently compiled previous-encoder classes and SF_INTEGRAL_PREVIOUS=true select the preceding callback control. Remaining coverage stays in the classification follow-up.

Numeric IS_DECIMAL

Released Flink receives boxed primitive integers or internal DecimalData in this overload. Integral arguments return TRUE regardless of sign or runtime/literal NULL; internal DecimalData arguments return FALSE. Direct inputs/literals keep a native type-based predicate. Only the recursively verified failure-free integer producer scope folds away unused arithmetic; division, remainder, narrowing, UDFs and other unverified producers retain the complete generated call, including failures and state changes. Floating signatures remain generated. This preserves the released behavior rather than inferring it from Java's public BigDecimal API.

The result record and raw trials retain 80 configurations and 8,000 main trials. Direct runtime arguments retain the native call; arithmetic fold controls are explicitly excluded from predicate-kernel qualification. The direct matrix covers four integer widths and DECIMAL(18,2)/(38,18), nullable/NOT NULL and GenericRowData/BinaryRowData at 1,024 rows. Tiny INT/BIGINT controls use 16 rows. Typed addition/nested multiplication provide separate fold diagnostics. No input values are converted to strings.

Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial compare actual released generated Java on prepared RowData with production Calc on matching Arrow. Native operation and equivalent Boolean result consumption are timed inside Rust; initial JNI/import/export and perimeter transposes are outside this small-operation timer. Intrinsic previous JVM computation stays inside. The actual independently compiled preceding-main encoder must bind an evaluator; the candidate requires empty descriptors and the intended function before the bounded fold. Both sides retain exact types, widths, signed boundaries and matching NULLs.

Flink line Rows / measured work Stock/native median range Previous/native median range Slower than stock
1.18.1 1,024 / predicate 4.444-64.321x 55.610-91.331x 0 / 24
1.18.1 16 / predicate 0.155-0.190x 29.598-32.049x 8 / 8
1.18.1 1,024 / fold control 5.805-14.961x 80.176-100.769x 0 / 8
2.2.1 1,024 / predicate 4.555-58.821x 59.944-78.244x 0 / 24
2.2.1 16 / predicate 0.143-0.187x 30.265-32.526x 8 / 8
2.2.1 1,024 / fold control 6.007-11.253x 74.172-127.370x 0 / 8

Ratios above one favor native. The type-based predicate returns a Boolean scalar which production Calc materializes to the batch length; the measurements do not claim a per-row character scan. Small-batch fixed Calc/result costs and every unfavorable sample remain in the record. Independent paired-stock drift ranges from 0.558 to 1.749x; cross-process ratios are not causal isolation.

Separate million-row nullable INT source/blackhole jobs retain production JNI, planning, startup and both transposes, with parallelism one and two warmups/five measurements. Direct jobs retain the predicate. Add/nested jobs exercise the bounded fold and must not be cited as predicate-kernel work. All require consumed native Calc work, no fallback and empty candidate evaluator descriptors.

Flink line / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / direct 0.152290 (0.143413-0.157737) 0.164787 (0.151287-0.177813) 0.143699 (0.140306-0.158073) 0.182686 (0.176609-0.207314)
1.18.1 / add 0.150613 (0.146701-0.159775) 0.158213 (0.153216-0.160860) 0.146302 (0.140714-0.157664) 0.187217 (0.173755-0.193327)
1.18.1 / nested 0.143415 (0.139697-0.151992) 0.147550 (0.145604-0.167166) 0.143714 (0.141247-0.143864) 0.183689 (0.171484-0.196876)
2.2.1 / direct 0.163562 (0.152313-0.169804) 0.164748 (0.160193-0.185271) 0.165385 (0.151227-0.184964) 0.193048 (0.190476-0.220503)
2.2.1 / add 0.162638 (0.155080-0.175627) 0.148933 (0.144973-0.164875) 0.160004 (0.145057-0.171511) 0.189043 (0.184488-0.221272)
2.2.1 / nested 0.156228 (0.154461-0.169524) 0.158591 (0.154876-0.165304) 0.148495 (0.145414-0.173559) 0.187562 (0.184164-0.199794)

Across these jobs, native/stock elapsed-time ratios range from 0.916 to 1.082; native/previous ratios range from 0.788 to 0.902. Ratios above one are slower. Retain overlapping ranges and independent stock drift; prepared bulk qualification does not establish universal whole-job acceleration.

All 136 focused JVM invocations and unchanged released IS_DECIMAL expression tests pass on Flink 1.18.1/2.2.1 with JDK 11/17. Eight numeric declarations, 5,003 rows, nullable and NOT NULL inputs, literal NULLs and Boolean consumers compare complete values, resolved types, cardinality and unique IDs across multiple batches. Direct runtime witnesses require consumed work, no fallback and empty descriptors. Separate failure/empty-batch tests and four additional shared-state invocations prove that arguments are not discarded. Java compiles for release 11; the release/default/mimalloc Rust core suite passes 1,017 tests with two pre-existing ignores.

Reproduce with integral_expression, for example SF_INTEGRAL_EXPRESSION='IS_DECIMAL(v)', and IntegralExpressionPipelineBenchmark with sf.integral.expression=IS_DECIMAL(v). Prepended independently compiled previous-encoder classes and SF_INTEGRAL_PREVIOUS=true select the preceding callback control. Remaining coverage stays in the classification follow-up.

BOOLEAN CAST to character

Direct BOOLEAN inputs/literals use a native formatter for STRING and bounded VARCHAR/CHAR. Modern mode emits uppercase text, truncates overlength values and pads bounded CHAR; legacy mode emits lowercase text without target-length enforcement. NULL remains NULL. Unverified computed operands retain generated Flink evaluation; independent TRY_CAST and logical-operand qualification follow below. No production switch is added. Both modes use Flink's existing cast configuration.

The result record and raw trials retain 152 configurations and 15,200 main trials. Modern bulk cells cover STRING, VARCHAR(1/3/4/5/8) and CHAR(1/3/4/5/8/32); legacy cells cover STRING, VARCHAR(3) and CHAR(8), including their ignored target lengths. Both released lines use matching nullable/NOT NULL declarations, alternating true/false values and actual NULLs every seven rows where selected, with GenericRowData and BinaryRowData controls. Tiny STRING/CHAR(8) cells retain both modes.

Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial compare released generated Java on prebuilt RowData with production Calc on matching Arrow buffers. Both consume UTF-8 length and boundary bytes; Java uses BinaryStringData access without copying to a byte array. The Rust timer covers the native operation and equivalent consumption, excluding initial JNI/import/export and both perimeter transposes. Intrinsic previous JVM computation stays inside timing. The actual independently compiled preceding encoder must bind a JVM evaluator; the candidate requires the intended function and empty descriptors. Every checksum agrees.

Flink line Mode Batch rows Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 modern 1024 48 1.626-11.810x 9.879-19.183x 0
1.18.1 modern 16 8 0.190-1.836x 22.304-24.939x 4
1.18.1 legacy 1024 12 6.949-17.607x 12.056-20.007x 0
1.18.1 legacy 16 8 0.825-1.385x 20.465-24.113x 4
2.2.1 modern 1024 48 1.551-10.159x 10.185-19.556x 0
2.2.1 modern 16 8 0.198-1.764x 21.865-23.653x 4
2.2.1 legacy 1024 12 8.237-11.477x 12.273-17.424x 0
2.2.1 legacy 16 8 0.838-1.031x 22.531-25.358x 6

Ratios above one favor native. All 120 bulk cells pass the prepared-operation gate; 18 of 32 tiny cells remain slower than stock. Paired stock-control drift ranges from 0.597 to 1.844x. Keep every range and unfavorable case rather than treating cross-process ratios as causal isolation.

The initial row writer loses the modern cached-STRING probe. The bitmap block technique reduces the matched native median from 6.162 to 2.446 ns/row; paired stock/native moves from 0.657x to 1.601x. The initial and optimized diagnostics retain eight cells each, their stock drift and the untimed allocation/buffer observations. They remain separate from the main matrix. The output validity is shared; masked NULL positions can carry payload, so fewer requested bytes do not imply smaller output buffers.

Separate million-row nullable BOOLEAN source/blackhole jobs retain production JNI, planning/startup and both transposes, parallelism one, two warmups and five measurements. They require consumed NativeCalc work, no fallback and empty candidate descriptors.

Flink line / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-string 0.158820 (0.157408-0.167913) 0.159093 (0.154768-0.173915) 0.157794 (0.144974-0.175266) 0.191205 (0.188041-0.197415)
1.18.1 / modern-varchar3 0.173374 (0.170471-0.180415) 0.158684 (0.156780-0.167511) 0.173464 (0.172478-0.178372) 0.219246 (0.216342-0.222373)
1.18.1 / modern-char8 0.198373 (0.190325-0.201441) 0.166334 (0.157919-0.168544) 0.190850 (0.184823-0.195628) 0.230782 (0.227496-0.245033)
1.18.1 / legacy-string 0.183357 (0.172087-0.197666) 0.169239 (0.158804-0.179719) 0.186869 (0.176384-0.253488) 0.190473 (0.182322-0.195269)
1.18.1 / legacy-char8 0.175425 (0.173670-0.219901) 0.167376 (0.157756-0.171659) 0.178774 (0.171553-0.242362) 0.200614 (0.194548-0.215857)
2.2.1 / modern-string 0.161098 (0.151663-0.162549) 0.162494 (0.156651-0.167074) 0.155657 (0.150932-0.157231) 0.193431 (0.183526-0.199987)
2.2.1 / modern-varchar3 0.181900 (0.180263-0.197655) 0.161911 (0.158735-0.177116) 0.181335 (0.174057-0.184622) 0.220254 (0.214121-0.229879)
2.2.1 / modern-char8 0.194581 (0.188651-0.204043) 0.162995 (0.158961-0.171878) 0.193474 (0.183651-0.200218) 0.227816 (0.222634-0.248081)
2.2.1 / legacy-string 0.170059 (0.169012-0.170231) 0.161957 (0.159721-0.165669) 0.178505 (0.172566-0.190951) 0.189525 (0.187156-0.201872)
2.2.1 / legacy-char8 0.179689 (0.173855-0.195317) 0.160113 (0.159534-0.168144) 0.171615 (0.170076-0.182689) 0.182692 (0.178134-0.195459)

Across these jobs, native/stock elapsed ratios range from 0.838 to 1.009, and native/previous ratios range from 0.715 to 0.889. Ratios above one are slower. These short full jobs retain startup and overlapping ranges; prepared bulk qualification does not establish universal complete-job acceleration.

The initial 172 focused JVM invocations pass, including existing numeric and decimal-cast regressions. After the bitmap optimization, all 40 focused invocations pass on released Flink 1.18.1/2.2.1 with JDK 11/17, including added string consumers. Runtime-source witnesses compare complete 5,003-row outputs, resolved types, NULLs, cardinality and unique IDs across batches, and require consumed native work, no fallback and empty descriptors. The unchanged released CastRulesTest supplies 535 passing stock-oracle cases on each Flink/JDK pair; the separate runtime witnesses establish native execution. Four additional shared-state invocations pass. Twelve existing integer/predicate fixture smokes retain their types and checksums. Java compiles for release 11; release/default/mimalloc Rust core passes 1,019 tests with two pre-existing ignores, including every bitmap word and unaligned tail. Strict docs, formatting and source/privacy checks pass.

Reproduce the prepared path with integral_expression, BOOLEAN input and SF_INTEGRAL_EXPRESSION='CAST(v AS STRING)'; set SF_INTEGRAL_LEGACY=true for the legacy control. Complete jobs use IntegralExpressionPipelineBenchmark, SF_BENCHMARK=true, sf.integral.booleanInput=true, sf.integral.expression=CAST(v AS CHAR(8)) and sf.integral.legacy=true where selected. These are benchmark controls, not production options. Remaining cast coverage stays in #492.

BOOLEAN TRY_CAST to character

Direct BOOLEAN inputs/literals now use the same native STRING/VARCHAR/CHAR formatter for TRY_CAST as for CAST. Formatting cannot produce a conversion error. Modern and legacy spelling, truncation/padding and NULL behavior match the released generated code. Unverified computed operands keep the complete generated expression; TRY_CAST must not swallow failures while evaluating an operand or drop shared stateful calls. An unconfigured encoder also keeps generated evaluation. No production option is added.

The independent result record and raw trials retain 152 configurations and 15,200 trials for TRY_CAST itself. The baseline is the actual preceding CAST encoder, which still binds a JVM evaluator for TRY_CAST; the candidate requires empty descriptors and the intended native function. Both configure the same released cast mode. The Rust kernel, release DSO and benchmark executable are unchanged from CAST qualification; source and DSO hashes are checked rather than rerunning the identical Rust suite.

The prepared boundary and matching BOOLEAN/NULL/cached-or-binary RowData inputs are the same as CAST above: release/default/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial. Rust times the production operation and equivalent UTF-8 boundary consumption. Initial JNI/import/export and both perimeter transposes are outside this timer; intrinsic previous JVM calls remain timed. All checksums agree. Tiny controls and independent stock drift remain separate from bulk qualification.

Flink line Mode Batch rows Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 modern 1024 48 1.646-8.972x 8.592-18.641x 0
1.18.1 modern 16 8 0.194-2.023x 22.171-25.598x 4
1.18.1 legacy 1024 12 7.273-11.839x 16.625-24.167x 0
1.18.1 legacy 16 8 0.807-1.134x 23.742-26.719x 4
2.2.1 modern 1024 48 1.604-13.670x 8.472-18.768x 0
2.2.1 modern 16 8 0.198-1.783x 22.358-26.250x 4
2.2.1 legacy 1024 12 8.882-12.173x 15.719-21.836x 0
2.2.1 legacy 16 8 0.846-1.052x 23.919-27.109x 6

All 120 bulk cells pass the prepared-operation gate; 18 of 32 tiny cells are slower than stock. Ratios above one favor native. Paired stock-control drift spans 0.674-1.940x; cross-process ratios are not causal isolation.

Ten million-row nullable BOOLEAN source/blackhole configurations retain production JNI, planning/startup and both transposes, parallelism one, two warmups/five measurements. They require consumed NativeCalc work, no fallback and empty candidate descriptors.

Flink line / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-string 0.164568 (0.152968-0.178889) 0.155482 (0.154097-0.169737) 0.163534 (0.150881-0.166770) 0.178076 (0.175590-0.190067)
1.18.1 / modern-varchar3 0.172382 (0.168989-0.187415) 0.163684 (0.158259-0.172744) 0.182536 (0.172397-0.188274) 0.205375 (0.204545-0.218700)
1.18.1 / modern-char8 0.199662 (0.192069-0.280178) 0.159020 (0.156592-0.169004) 0.193263 (0.188125-0.200050) 0.224147 (0.213284-0.226749)
1.18.1 / legacy-string 0.181813 (0.172095-0.202011) 0.161006 (0.159682-0.172754) 0.173408 (0.164714-0.192122) 0.208225 (0.194695-0.216245)
1.18.1 / legacy-char8 0.179340 (0.176745-0.217252) 0.166424 (0.164905-0.172886) 0.183752 (0.171472-0.191367) 0.205981 (0.200621-0.224427)
2.2.1 / modern-string 0.167586 (0.155133-0.190176) 0.164438 (0.157190-0.166998) 0.155230 (0.150007-0.166039) 0.181543 (0.179630-0.195766)
2.2.1 / modern-varchar3 0.176160 (0.168682-0.181995) 0.163558 (0.160969-0.169159) 0.174333 (0.167268-0.188351) 0.207104 (0.199274-0.218739)
2.2.1 / modern-char8 0.195855 (0.191255-0.209562) 0.158579 (0.157120-0.160678) 0.192746 (0.189712-0.197713) 0.216276 (0.211593-0.218344)
2.2.1 / legacy-string 0.171966 (0.168650-0.194207) 0.160860 (0.154903-0.172927) 0.169731 (0.160435-0.179396) 0.206011 (0.197393-0.216290)
2.2.1 / legacy-char8 0.175236 (0.169376-0.186207) 0.159021 (0.156173-0.166410) 0.174426 (0.166370-0.183576) 0.193287 (0.187956-0.213551)

Native/stock elapsed ratios range from 0.796 to 0.981; native/previous ratios range from 0.709 to 0.906. Ratios above one are slower. Keep overlapping ranges and unfavorable outcomes; prepared qualification does not prove universal full-job acceleration.

All 68 focused JVM invocations pass across Flink 1.18.1/2.2.1 and JDK 11/17, including CAST regression controls. Complete 5,003-row outputs, resolved types, cardinality, unique IDs and NULLs match stock across batches; direct runtime witnesses require consumed native work with no fallback or evaluator descriptor. Selected operand division errors, lazy branches, empty filtered batches, string consumers and unconfigured encoder behavior are covered. Four additional shared-state invocations pass. The unchanged released CastRules stock oracle (535 cases per pair) and 1,019 Rust core passes/two pre-existing ignores are reused from the identical kernel qualification; they are separate from the new native TRY_CAST witnesses. Java compiles for release 11. Strict docs, diff and privacy checks pass.

Reproduce with SF_INTEGRAL_EXPRESSION='TRY_CAST(v AS STRING)' and BOOLEAN input for integral_expression; complete jobs use sf.integral.booleanInput=true and sf.integral.expression=TRY_CAST(v AS CHAR(8)). The existing benchmark-only legacy control selects the corresponding mode. Remaining casts stay in #492.

BOOLEAN CAST over logical operands

The formatter accepts recursively verified BOOLEAN-only NOT, AND, OR, IS NULL, IS NOT NULL, IS TRUE, IS NOT TRUE, IS FALSE and IS NOT FALSE over BOOLEAN references and literals. These operations cannot fail or mutate shared state. At this logical-only qualification, numeric comparisons, division, UDFs and CASE kept their generated or short-circuit fallback routes. The proof never discards a computed operand.

The result record and raw trials retain 76 configurations and 7,600 trials. Representative producers are NOT(v) and nullable v IS TRUE. The initial Flink 2.2 NOT NULL v IS TRUE control simplifies to v before encoding and is already native in the actual preceding encoder. NOT NULL truth-test cells are excluded from new-feature performance coverage on both lines. The initial strict previous-route assertion exposed that simplification rather than allowing it to count as callback removal. Its independent candidate-only diagnostic retains all 50 samples separately. Correctness witnesses still cover both declarations.

Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary described above. Modern targets are STRING/VARCHAR(3)/CHAR(8); legacy targets are STRING/CHAR(8). Cached and binary RowData controls match logical values and NULLs. The actual preceding encoder must bind a JVM evaluator; candidate encoding must have none. Equivalent UTF-8 boundary consumption and every checksum agree. The identical production Rust kernel/DSO is reused, with source and DSO hashes checked.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 logical bulk 30 1.456-11.667x 11.497-20.919x 0
1.18.1 tiny NOT control 8 0.175-0.918x 19.655-21.435x 8
2.2.1 logical bulk 30 1.460-10.751x 12.686-17.764x 0
2.2.1 tiny NOT control 8 0.180-0.896x 19.915-22.033x 8

All 60 bulk cells pass the prepared-operation gate; 16 of 16 tiny cells are slower than stock. Ratios above one favor native. Independent stock-control drift spans 0.573-1.499x. These measurements qualify the representative logical producers; they do not assert a speedup for every expression tree.

Six million-row source/blackhole configurations retain planning/startup, production JNI and both transposes. Modern NOT-to-STRING, modern truth-test-to-CHAR(8) and legacy NOT-to-STRING run at parallelism one with two warmups/five measurements. Runtime witnesses require consumed native Calc work, no fallback and empty candidate evaluator descriptors.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-not 0.155199 (0.152203-0.180638) 0.157988 (0.153672-0.165963) 0.156686 (0.155055-0.184734) 0.187435 (0.182388-0.190953)
1.18.1 / modern-truth 0.239740 (0.234626-0.249306) 0.193730 (0.190997-0.201901) 0.240761 (0.235625-0.248413) 0.275991 (0.274064-0.291803)
1.18.1 / legacy-not 0.172859 (0.170668-0.216929) 0.165043 (0.159584-0.165952) 0.193202 (0.175767-0.262754) 0.191391 (0.189054-0.218440)
2.2.1 / modern-not 0.162671 (0.154038-0.169849) 0.155008 (0.153929-0.162233) 0.159649 (0.156157-0.168375) 0.196254 (0.186756-0.199247)
2.2.1 / modern-truth 0.251202 (0.245248-0.265937) 0.206193 (0.203454-0.212622) 0.245309 (0.244001-0.258694) 0.269516 (0.266690-0.276784)
2.2.1 / legacy-not 0.177856 (0.172548-0.199941) 0.163611 (0.160215-0.168126) 0.176808 (0.172573-0.188163) 0.191048 (0.186868-0.204655)

Native/stock elapsed ratios span 0.808-1.018; native/previous ratios span 0.702-0.862. Ratios above one are slower. Retain overlapping ranges and all unfavorable cases; bulk qualification does not establish universal full-job acceleration.

All 84 focused JVM invocations and four additional shared-state invocations pass on released Flink 1.18.1/2.2.1 with JDK 11/17. Runtime witnesses compare complete 5,003-row outputs, resolved types, NULLs, cardinality and unique IDs across batches; independent BOOLEAN columns cover AND/OR combinations and nesting. They require consumed native work, no fallback and empty descriptors for every admitted logical query. Failing comparisons under logical wrappers, lazy/empty branches and stateful producers retain their exact prior route and successful/exception behavior. The unchanged released ScalarFunctionsTest.testIsTrueIsFalse and ScalarOperatorsTest.testFunctionWithBooleanExpression also pass on all four pairs; they are separate from the new native execution witnesses. Java targets release 11. The unchanged kernel retains its prior 1,019 Rust core passes, two pre-existing ignores and released CastRules stock oracle. Strict docs, diff, source and privacy checks pass.

Reproduce prepared qualification with integral_expression, BOOLEAN input and SF_INTEGRAL_EXPRESSION='CAST(NOT v AS STRING)'. Complete jobs use sf.integral.booleanInput=true and sf.integral.expression=CAST(v IS TRUE AS CHAR(8)). No production option is introduced. Remaining conversions are tracked in #492.

BOOLEAN TRY_CAST over logical operands

The formatter accepts recursively verified BOOLEAN-only NOT, AND, OR, IS NULL, IS NOT NULL, IS TRUE, IS NOT TRUE, IS FALSE and IS NOT FALSE over BOOLEAN references and literals. These operations cannot fail or mutate shared state. At this logical-only qualification, numeric comparisons, division, UDFs and CASE kept their generated or short-circuit fallback routes. The proof never discards a computed operand.

The result record and raw trials retain 76 configurations and 7,600 trials. Representative producers are NOT(v) and nullable v IS TRUE. The initial Flink 2.2 NOT NULL v IS TRUE control simplifies to v before encoding and is already native in the actual preceding encoder. NOT NULL truth-test cells are excluded from new-feature performance coverage on both lines. The preceding CAST strict previous-route assertion exposed that simplification rather than allowing it to count as callback removal. Its independent candidate-only diagnostic retains all 50 samples separately. Correctness witnesses still cover both declarations.

Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary described above. Modern targets are STRING/VARCHAR(3)/CHAR(8); legacy targets are STRING/CHAR(8). Cached and binary RowData controls match logical values and NULLs. The actual preceding encoder must bind a JVM evaluator; candidate encoding must have none. Equivalent UTF-8 boundary consumption and every checksum agree. The identical production Rust kernel/DSO is reused, with source and DSO hashes checked.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 logical bulk 30 1.429-18.184x 8.887-21.064x 0
1.18.1 tiny NOT control 8 0.175-0.939x 19.482-25.024x 8
2.2.1 logical bulk 30 1.453-12.490x 9.313-21.843x 0
2.2.1 tiny NOT control 8 0.175-1.104x 19.154-27.388x 7

All 60 bulk cells pass the prepared-operation gate; 15 of 16 tiny cells are slower than stock. Ratios above one favor native. Independent stock-control drift spans 0.632-1.992x. These measurements qualify the representative logical producers; they do not assert a speedup for every expression tree.

Six million-row source/blackhole configurations retain planning/startup, production JNI and both transposes. Modern NOT-to-STRING, modern truth-test-to-CHAR(8) and legacy NOT-to-STRING run at parallelism one with two warmups/five measurements. Runtime witnesses require consumed native Calc work, no fallback and empty candidate evaluator descriptors.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-not 0.192990 (0.170395-0.289754) 0.188779 (0.175542-0.276829) 0.168986 (0.155901-0.175384) 0.190369 (0.185188-0.193414)
1.18.1 / modern-truth 0.240194 (0.235374-0.249937) 0.189541 (0.177280-0.190945) 0.218437 (0.205282-0.244150) 0.235512 (0.227074-0.263892)
1.18.1 / legacy-not 0.211747 (0.189535-0.228928) 0.179852 (0.169331-0.181138) 0.185579 (0.179722-0.194138) 0.224362 (0.216911-0.230450)
2.2.1 / modern-not 0.164041 (0.156972-0.166769) 0.162113 (0.158810-0.163989) 0.178893 (0.172315-0.193451) 0.206872 (0.204361-0.213012)
2.2.1 / modern-truth 0.234089 (0.214258-0.237513) 0.168890 (0.167765-0.186218) 0.234878 (0.214046-0.243389) 0.250744 (0.240538-0.263417)
2.2.1 / legacy-not 0.226982 (0.200094-0.244595) 0.205476 (0.184333-0.210453) 0.198807 (0.191128-0.219500) 0.244790 (0.222760-0.250576)

Native/stock elapsed ratios span 0.721-0.988; native/previous ratios span 0.674-0.992. Ratios above one are slower. Retain overlapping ranges and all unfavorable cases; bulk qualification does not establish universal full-job acceleration.

All 92 focused JVM invocations and four additional shared-state invocations pass on released Flink 1.18.1/2.2.1 with JDK 11/17. Runtime witnesses compare complete 5,003-row outputs, resolved types, NULLs, cardinality and unique IDs across batches; independent BOOLEAN columns cover AND/OR combinations and nesting. They require consumed native work, no fallback and empty descriptors for every admitted logical query. Failing comparisons under logical wrappers, lazy/empty branches and stateful producers retain their exact prior route and successful/exception behavior. The unchanged released ScalarFunctionsTest.testIsTrueIsFalse and ScalarOperatorsTest.testFunctionWithBooleanExpression retain their unchanged preceding CAST qualification on all four pairs; the new TRY_CAST runtime witnesses establish the changed native route. Java targets release 11. The unchanged kernel retains its prior 1,019 Rust core passes, two pre-existing ignores and released CastRules stock oracle. Strict docs, diff, source and privacy checks pass.

Reproduce prepared qualification with integral_expression, BOOLEAN input and SF_INTEGRAL_EXPRESSION='TRY_CAST(NOT v AS STRING)'. Complete jobs use sf.integral.booleanInput=true and sf.integral.expression=TRY_CAST(v IS TRUE AS CHAR(8)). No production option is introduced. Remaining conversions are tracked in #492.

Bounded DECIMAL CAST to character

Direct DECIMAL inputs/literals reuse the exact native plain-text formatter. Modern VARCHAR truncates through LEFT; modern CHAR truncates or space-pads through RPAD. DECIMAL text is ASCII, so these existing kernels preserve released Flink character widths without a Unicode approximation. Legacy mode ignores target length; maximum-INT CHAR does not pad. This direct-input qualification does not measure computed decimals; the later unary-operand qualification is recorded separately below. Other computed decimals and their metadata guards retain existing admission. Bounded TRY_CAST remains a separate qualification.

The result record and raw trials retain 96 configurations/9600 trials. Release/default-feature/mimalloc, Oracle JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. DECIMAL(18,2)/(38,18), signed extrema and large signed coefficients match across GenericRowData/BinaryRowData and nullable/NOT NULL schemas. Modern targets are VARCHAR(3)/CHAR(8)/CHAR(44); legacy targets VARCHAR(3)/CHAR(8). All output checksums match. Candidate descriptors must be empty; the independently compiled preceding encoder must retain its actual JVM route. Tiny 16-row CHAR(8) controls remain separate from 1,024-row bulk qualification.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 40 3.076-9.964x 4.247-11.757x 0
1.18.1 tiny control 8 1.421-3.349x 10.676-13.234x 0
2.2.1 bulk 40 3.073-10.201x 4.239-11.687x 0
2.2.1 tiny control 8 1.427-3.294x 10.489-12.713x 0

All 80 bulk cells beat stock. 0 of 16 tiny controls are slower. Independent stock-control drift spans 0.731-1.130x; all ranges remain in the record.

Six complete million-row jobs retain source generation, planning/startup, production JNI and both perimeter transposes, using a rowwise source and blackhole at parallelism one. Each has two warmups/five measurements. Compact modern VARCHAR(3), wide modern CHAR(44) and wide legacy CHAR(8) require consumed NativeCalc, no fallback and no candidate evaluator.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-compact 0.310117 (0.303876-0.311701) 0.215852 (0.211178-0.227059) 0.315230 (0.311484-0.321128) 0.375995 (0.373839-0.383962)
1.18.1 / modern-wide 0.566128 (0.550808-0.568562) 0.270693 (0.266628-0.276478) 0.554017 (0.540817-0.565504) 0.621521 (0.619297-0.633610)
1.18.1 / legacy-wide 0.278262 (0.263808-0.285093) 0.218404 (0.214724-0.225595) 0.272596 (0.267901-0.296278) 0.310646 (0.304137-0.321228)
2.2.1 / modern-compact 0.316695 (0.315670-0.330976) 0.215533 (0.210773-0.216891) 0.315654 (0.311846-0.325644) 0.355857 (0.346658-0.382724)
2.2.1 / modern-wide 0.565624 (0.549676-0.577944) 0.263300 (0.249329-0.267397) 0.555790 (0.545149-0.560500) 0.622109 (0.613828-0.638969)
2.2.1 / legacy-wide 0.277581 (0.270206-0.302031) 0.216288 (0.210198-0.222025) 0.280246 (0.269209-0.293993) 0.308279 (0.306989-0.325054)

Native/stock elapsed ratios span 0.466-0.785; native/previous ratios span 0.423-0.703. Ratios above one are slower. Overlapping ranges and unfavorable whole-job results remain; prepared qualification does not establish universal end-to-end acceleration.

All 132 focused encoding/runtime invocations (including existing unbounded/failure controls) pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row outputs verify declared types, NULLs, unique IDs and cardinality across batches with consumed native work and empty evaluator descriptors. Additional consumers, truncation/padding, lazy and empty-batch cases, selected operand failures and materialized aggregate results match stock. Flink 1.18 does not support MAX(CHAR); that boundary witness first widens the padded CHAR to STRING before MAX. Twelve existing integer/predicate fixture controls pass. The unchanged released CastRulesTest oracle retains its prior 535 stock cases per pair; the unchanged Rust DSO retains 1,019 core passes/two pre-existing ignores. Neither reused oracle substitutes for the new native execution witnesses. Source/DSO hashes and independently recompiled preceding encoder classes are retained.

Reproduce prepared qualification with integral_expression, DECIMAL input and SF_INTEGRAL_EXPRESSION='CAST(v AS CHAR(8))'. Complete jobs use sf.integral.decimalPrecision=38, sf.integral.decimalScale=18 and the same expression. These are benchmark-only controls; no production option is introduced. Remaining work is tracked in #492.

Bounded DECIMAL TRY_CAST to character

Direct DECIMAL inputs/literals reuse the exact native plain-text formatter. Modern VARCHAR truncates through LEFT; modern CHAR truncates or space-pads through RPAD. DECIMAL text is ASCII, so these existing kernels preserve released Flink character widths without a Unicode approximation. Legacy mode ignores target length; maximum-INT CHAR does not pad. This direct-input TRY_CAST qualification does not measure computed decimals; the later unary-operand qualification is recorded separately below. Other computed decimals and their metadata guards retain existing admission. TRY_CAST uses the same total formatting operation and does not catch operand failures.

The result record and raw trials retain 96 configurations/9600 trials. Release/default-feature/mimalloc, Oracle JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. DECIMAL(18,2)/(38,18), signed extrema and large signed coefficients match across GenericRowData/BinaryRowData and nullable/NOT NULL schemas. Modern targets are VARCHAR(3)/CHAR(8)/CHAR(44); legacy targets VARCHAR(3)/CHAR(8). All output checksums match. Candidate descriptors must be empty; the independently compiled preceding encoder must reject bounded DECIMAL TRY_CAST. Its prepared control runs the released evaluator in Java on matching RowData; operation and equivalent consumption are timed in Java. Complete previous jobs require the real planner fallback reason, zero native substitutions and zero consumed native work. The candidate uses Rust-side timers and empty evaluator descriptors. Tiny 16-row CHAR(8) controls remain separate from 1,024-row bulk qualification.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 40 3.251-9.827x 3.488-10.748x 0
1.18.1 tiny control 8 1.412-3.401x 1.544-3.745x 0
2.2.1 bulk 40 3.104-10.154x 3.422-10.276x 0
2.2.1 tiny control 8 1.428-3.434x 1.746-3.500x 0

All 80 bulk cells beat stock. 0 of 16 tiny controls are slower. Independent stock-control drift spans 0.759-0.999x; all ranges remain in the record.

Six complete million-row configurations use a rowwise source and blackhole at parallelism one, with two warmups/five measurements. Candidate native jobs retain source generation, planning/startup, production JNI and both perimeter transposes; the preceding stock fallback retains its actual path. Compact modern VARCHAR(3), wide modern CHAR(44) and wide legacy CHAR(8) require consumed NativeCalc, no fallback and no candidate evaluator.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-compact 0.323108 (0.317246-0.325959) 0.215176 (0.214526-0.218355) 0.272784 (0.264374-0.280154) 0.272559 (0.259251-0.276117)
1.18.1 / modern-wide 0.554577 (0.544595-0.577801) 0.249330 (0.245439-0.250792) 0.521098 (0.515807-0.523225) 0.520737 (0.517170-0.526215)
1.18.1 / legacy-wide 0.288225 (0.274643-0.316488) 0.218552 (0.208366-0.222127) 0.240750 (0.233640-0.246038) 0.249669 (0.236538-0.253221)
2.2.1 / modern-compact 0.317752 (0.315881-0.329789) 0.216294 (0.213674-0.223117) 0.268792 (0.265014-0.273145) 0.281972 (0.268092-0.286282)
2.2.1 / modern-wide 0.555897 (0.533242-0.559481) 0.252509 (0.245032-0.255392) 0.520144 (0.518186-0.537838) 0.520204 (0.515485-0.531938)
2.2.1 / legacy-wide 0.285559 (0.268036-0.291094) 0.218255 (0.212956-0.227082) 0.244965 (0.238019-0.249634) 0.237997 (0.235115-0.245426)

Native/stock elapsed ratios span 0.450-0.764; native/previous ratios span 0.479-0.917. Ratios above one are slower. Overlapping ranges and unfavorable whole-job results remain; prepared qualification does not establish universal end-to-end acceleration.

All 184 focused encoding/runtime invocations (including existing unbounded/failure controls) pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row outputs verify declared types, NULLs, unique IDs and cardinality across batches with consumed native work and empty evaluator descriptors. Additional consumers, truncation/padding, lazy and empty-batch cases, selected operand failures and materialized aggregate results match stock. Flink 1.18 does not support MAX(CHAR); that boundary witness first widens the padded CHAR to STRING before MAX. Twelve existing integer/predicate fixture controls pass. The unchanged released CastRulesTest oracle retains its prior 535 stock cases per pair; the unchanged Rust DSO retains 1,019 core passes/two pre-existing ignores. Neither reused oracle substitutes for the new native execution witnesses. Source/DSO hashes and independently recompiled preceding encoder classes are retained.

Reproduce prepared qualification with integral_expression, DECIMAL input and SF_INTEGRAL_EXPRESSION='TRY_CAST(v AS CHAR(8))'. Complete jobs use sf.integral.decimalPrecision=38, sf.integral.decimalScale=18 and the same expression. These are benchmark-only controls; no production option is introduced. Remaining work is tracked in #492.

The preceding fallback control launches IntegralExpressionBenchmarkFunctions with sf.integral.previous=true and sf.integral.fallbackControl=true, using the same expression, legacy mode, row format and trial counts. Complete previous jobs use the same fallback control. These controls never replace the actual rejected encoding with a synthetic native or callback program.

BOOLEAN CAST over integer comparisons

Integer =, <>, <, <=, > and >= comparisons reuse the native BOOLEAN formatter. The recursive proof admits verified integer references/literals, wrapping addition, subtraction and multiplication, lossless widening and the existing integral rounding forms. BOOLEAN logical/null/truth wrappers remain native; integer IS NULL/IS NOT NULL also cover the optimizer's expansion of nullable comparisons. The proof excludes failing and stateful producers. DECIMAL, character, floating and temporal comparisons, division, remainder and other unverified producers keep their existing routes. Modern and legacy text/width rules are unchanged. No production switch is introduced.

The result record and raw trials retain 64 configurations/6,400 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. INT/BIGINT extrema and values beyond 2^53 match across GenericRowData/BinaryRowData and nullable/NOT NULL declarations. Representative bulk expressions are modern v>=0 to STRING, modern wrapping v+1<0 to CHAR(8), and legacy v<0 to CHAR(8). Sixteen-row modern comparison-to-STRING controls remain separate. All checksums agree. The independently recompiled actual preceding encoder must bind a JVM evaluator; the candidate must have no evaluator. The unchanged Rust formatter, Calc and comparison kernels retain their exact DSO.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 1.571-9.760x 11.270-14.952x 0
1.18.1 tiny control 8 0.123-0.162x 14.109-15.029x 8
2.2.1 bulk 24 1.581-9.791x 11.132-15.929x 0
2.2.1 tiny control 8 0.123-0.163x 14.003-15.089x 8

All 48 bulk cells beat stock; 16 of 16 tiny controls are slower. Independent stock-control drift spans 0.650-1.393x. All ranges are retained. For the Flink 2.2 NOT NULL INT/GenericRowData comparison, untimed native allocation probes report 29 requests for both batch sizes: 10,534 requested bytes at 1,024 rows and 1,816 at 16 rows. Native medians rise from 2.315 to 33.023 ns/row while stock remains near 4.7 ns/row. Fixed per-batch work remains a small-batch limitation; removing the preceding JVM evaluator does not eliminate that cost.

Six complete million-row source/blackhole configurations retain source generation, planning/startup, production JNI and both transposes. Modern INT comparison-to-STRING, modern BIGINT arithmetic comparison-to-CHAR(8), and legacy INT comparison-to-CHAR(8) run at parallelism one with two warmups/five measurements. Each native query requires consumed NativeCalc, no fallback and empty candidate evaluator descriptors.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-comparison 0.188399 (0.186934-0.194068) 0.191475 (0.190601-0.197079) 0.192767 (0.183027-0.197884) 0.217869 (0.215422-0.238049)
1.18.1 / modern-arithmetic 0.235702 (0.231248-0.249417) 0.199672 (0.196615-0.223385) 0.242474 (0.237011-0.247843) 0.273772 (0.270471-0.284878)
1.18.1 / legacy-comparison 0.208412 (0.200292-0.227803) 0.188325 (0.185272-0.198546) 0.212331 (0.208584-0.216080) 0.223269 (0.220450-0.231086)
2.2.1 / modern-comparison 0.185857 (0.181895-0.190989) 0.187063 (0.184722-0.196945) 0.187830 (0.180328-0.192330) 0.219429 (0.215679-0.226802)
2.2.1 / modern-arithmetic 0.225332 (0.219251-0.228684) 0.190797 (0.190280-0.194885) 0.229180 (0.218205-0.244467) 0.258519 (0.251868-0.277756)
2.2.1 / legacy-comparison 0.205362 (0.204444-0.214410) 0.192941 (0.189956-0.201685) 0.204743 (0.194486-0.218907) 0.212190 (0.207976-0.218663)

Native/stock elapsed ratios span 0.847-1.016; native/previous ratios span 0.729-0.909. Ratios above one are slower. All unfavorable results and overlapping ranges remain. Prepared qualification does not establish universal end-to-end acceleration.

All 268 focused JVM invocations, four additional shared-state invocations and 12 existing fixture smokes pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row witnesses compare all values, schema, NULLs, IDs and cardinality across batches for TINYINT/SMALLINT/INT/BIGINT, independent input columns, nullable/NOT NULL declarations, modern/legacy mode, signed extrema and BIGINT values beyond 2^53. They require consumed native work, no fallback and no evaluator. String consumers, grouping, lazy branches and empty filtered batches match stock. Failing and stateful inputs keep their released behavior and exact previous routes. Unchanged released boolean-expression/truth tests, 535 CastRules stock-oracle cases per pair, and the identical DSO's 1,019 core passes/two pre-existing ignores retain their preceding qualification; new runtime witnesses establish the changed route. Source/DSO hashes and independently recompiled baseline classes are retained.

Reproduce prepared qualification with integral_expression, INT/BIGINT input and SF_INTEGRAL_EXPRESSION='CAST(v>=CAST(0 AS $TYPE) AS STRING)'. Complete jobs use sf.integral.expression=CAST(v>=0 AS STRING); BIGINT controls additionally use sf.integral.bigintInput=true. Remaining conversions are tracked in #492.

BOOLEAN TRY_CAST over integer comparisons

Integer =, <>, <, <=, > and >= comparisons reuse the native BOOLEAN formatter. The recursive proof admits verified integer references/literals, wrapping addition, subtraction and multiplication, lossless widening and the existing integral rounding forms. BOOLEAN logical/null/truth wrappers remain native; integer IS NULL/IS NOT NULL also cover the optimizer's expansion of nullable comparisons. The proof excludes failing and stateful producers. DECIMAL, character, floating and temporal comparisons, division, remainder and other unverified producers keep their existing routes. Modern and legacy text/width rules are unchanged. No production switch is introduced.

The result record and raw trials retain 64 configurations/6,400 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. INT/BIGINT extrema and values beyond 2^53 match across GenericRowData/BinaryRowData and nullable/NOT NULL declarations. Representative bulk expressions are modern v>=0 to STRING, modern wrapping v+1<0 to CHAR(8), and legacy v<0 to CHAR(8). Sixteen-row modern comparison-to-STRING controls remain separate. All checksums agree. The independently recompiled actual preceding encoder must bind a JVM evaluator; the candidate must have no evaluator. The unchanged Rust formatter, Calc and comparison kernels retain their exact DSO.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 1.502-16.474x 10.559-18.565x 0
1.18.1 tiny control 8 0.124-0.162x 14.250-15.230x 8
2.2.1 bulk 24 1.531-10.600x 10.074-19.601x 0
2.2.1 tiny control 8 0.122-0.163x 14.110-14.918x 8

All 48 bulk cells beat stock; 16 of 16 tiny controls are slower. Independent stock-control drift spans 0.756-1.905x. All ranges are retained.

Six complete million-row source/blackhole configurations retain source generation, planning/startup, production JNI and both transposes. Modern INT comparison-to-STRING, modern BIGINT arithmetic comparison-to-CHAR(8), and legacy INT comparison-to-CHAR(8) run at parallelism one with two warmups/five measurements. Each native query requires consumed NativeCalc, no fallback and empty candidate evaluator descriptors.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-comparison 0.188606 (0.183451-0.213412) 0.194802 (0.194257-0.200845) 0.193022 (0.181021-0.197639) 0.212879 (0.212570-0.221866)
1.18.1 / modern-arithmetic 0.240461 (0.234930-0.242738) 0.202671 (0.191075-0.207119) 0.240431 (0.227648-0.241450) 0.244998 (0.240404-0.252949)
1.18.1 / legacy-comparison 0.210634 (0.209291-0.225512) 0.193515 (0.189962-0.200454) 0.215743 (0.197997-0.227196) 0.245962 (0.234360-0.359305)
2.2.1 / modern-comparison 0.189071 (0.184989-0.199581) 0.189496 (0.185077-0.194949) 0.192146 (0.185846-0.196981) 0.207776 (0.204363-0.213585)
2.2.1 / modern-arithmetic 0.229806 (0.221967-0.238040) 0.198334 (0.195886-0.204989) 0.224424 (0.220635-0.238883) 0.247177 (0.244628-0.259436)
2.2.1 / legacy-comparison 0.195756 (0.192557-0.200052) 0.185488 (0.183502-0.196643) 0.202816 (0.201091-0.213579) 0.228695 (0.224209-0.232692)

Native/stock elapsed ratios span 0.843-1.033; native/previous ratios span 0.787-0.915. Ratios above one are slower. All unfavorable results and overlapping ranges remain. Prepared qualification does not establish universal end-to-end acceleration.

All 292 focused JVM invocations, four additional shared-state invocations and 12 existing fixture smokes pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row witnesses compare all values, schema, NULLs, IDs and cardinality across batches for TINYINT/SMALLINT/INT/BIGINT, independent input columns, nullable/NOT NULL declarations, modern/legacy mode, signed extrema and BIGINT values beyond 2^53. They require consumed native work, no fallback and no evaluator. String consumers, grouping, lazy branches and empty filtered batches match stock. Failing and stateful inputs keep their released behavior and exact previous routes. Unchanged released boolean-expression/truth tests, 535 CastRules stock-oracle cases per pair, and the identical DSO's 1,019 core passes/two pre-existing ignores retain their preceding qualification; new runtime witnesses establish the changed route. Source/DSO hashes and independently recompiled baseline classes are retained.

Reproduce prepared qualification with integral_expression, INT/BIGINT input and SF_INTEGRAL_EXPRESSION='TRY_CAST(v>=CAST(0 AS $TYPE) AS STRING)'. Complete jobs use sf.integral.expression=TRY_CAST(v>=0 AS STRING); BIGINT controls additionally use sf.integral.bigintInput=true. Remaining conversions are tracked in #492.

DECIMAL unary CAST to character

Verified DECIMAL ABS and unary minus, same-operator nesting, and same-scale precision widening reuse the native plain-text formatter and existing ASCII-safe LEFT/RPAD width kernels. Released Flink retains precision/scale in these unary operations. Modern/legacy spelling, trailing zeros, truncation/padding and NULLs remain unchanged. Arithmetic CAST/TRY_CAST is qualified separately below. Other arithmetic, rounding metadata, SIGN, mixed unary trees, stateful/failing producers and unknown configuration retain their existing routes. No native kernel, JNI boundary, ownership contract or production switch changes.

The result record and raw trials retain 64 configurations/6,400 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. DECIMAL(18,2)/(38,18) extrema and large signed coefficients match across GenericRowData/BinaryRowData and nullable/NOT NULL declarations. Representative bulk expressions are modern ABS(v) to STRING, modern -v to CHAR(8), and legacy ABS(v) to CHAR(44). Sixteen-row modern ABS-to-STRING controls remain separate. All checksums agree. The independently recompiled actual preceding encoder must bind a JVM evaluator; the candidate must have no evaluator. The unchanged Rust decimal formatter, Calc and unary kernels retain their exact DSO.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 3.527-8.304x 5.151-8.657x 0
1.18.1 tiny control 8 1.174-3.390x 16.101-19.433x 0
2.2.1 bulk 24 3.242-8.637x 4.963-8.754x 0
2.2.1 tiny control 8 1.221-3.017x 17.392-19.115x 0

All 48 bulk cells beat stock; 0 of 16 tiny controls are slower. Independent stock-control drift spans 0.792-1.170x. All ranges are retained.

Six complete million-row source/blackhole configurations retain source generation, planning/startup, production JNI and both transposes. Modern DECIMAL(18,2) ABS-to-STRING, modern DECIMAL(38,18) negate-to-CHAR(8), and legacy DECIMAL(38,18) ABS-to-CHAR(44) run at parallelism one with two warmups/five measurements. Each native query requires consumed NativeCalc, no fallback and empty candidate evaluator descriptors.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-abs 0.259279 (0.245537-0.298334) 0.212230 (0.207713-0.220957) 0.257366 (0.249812-0.266154) 0.271891 (0.264303-0.274019)
1.18.1 / modern-negate 0.388762 (0.378809-0.394126) 0.231802 (0.224440-0.236891) 0.382977 (0.378330-0.387905) 0.448462 (0.438670-0.452394)
1.18.1 / legacy-abs 0.290965 (0.277537-0.297425) 0.219611 (0.217127-0.231349) 0.282531 (0.277297-0.308506) 0.303356 (0.296926-0.312935)
2.2.1 / modern-abs 0.260823 (0.248016-0.278068) 0.210224 (0.202756-0.216913) 0.260424 (0.255985-0.281087) 0.271364 (0.267967-0.273161)
2.2.1 / modern-negate 0.365805 (0.361973-0.375923) 0.224516 (0.218071-0.230758) 0.367469 (0.361109-0.375109) 0.420946 (0.414307-0.428079)
2.2.1 / legacy-abs 0.294697 (0.279771-0.298122) 0.227533 (0.225032-0.228816) 0.287236 (0.281952-0.329673) 0.293164 (0.290418-0.302692)

Native/stock elapsed ratios span 0.596-0.819; native/previous ratios span 0.517-0.781. Ratios above one are slower. All unfavorable results and overlapping ranges remain. Prepared qualification does not establish universal end-to-end acceleration.

All 248 focused JVM invocations, four additional shared-state invocations and 12 existing fixture smokes pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row witnesses compare all values, schema, NULLs, IDs and cardinality across batches for eight DECIMAL precision/scale pairs from (1,1) through (38,38), nullable/NOT NULL declarations, modern/legacy mode, signed extrema, zeros and unit coefficients. They require consumed native work, no fallback and no evaluator. String consumers, grouping, lazy branches and empty filtered batches match stock. Failing and stateful inputs keep their released behavior and exact previous routes. The unchanged released DecimalData formatting oracle, 535 CastRules stock-oracle cases per pair, and the identical DSO's 1,019 core passes/two pre-existing ignores retain their preceding qualification; new runtime witnesses establish the changed route. Source/DSO hashes and independently recompiled baseline classes are retained.

Reproduce prepared qualification with integral_expression, DECIMAL input and SF_INTEGRAL_EXPRESSION='CAST(ABS(v) AS STRING)'. Complete jobs use sf.integral.expression=CAST(ABS(v) AS STRING) and sf.integral.decimalPrecision=18, sf.integral.decimalScale=2. Remaining conversions are tracked in #492.

DECIMAL unary TRY_CAST to character

Verified DECIMAL ABS and unary minus, same-operator nesting, and same-scale precision widening reuse the native plain-text formatter and existing ASCII-safe LEFT/RPAD width kernels. Released Flink retains precision/scale in these unary operations. Modern/legacy spelling, trailing zeros, truncation/padding and NULLs remain unchanged. Arithmetic CAST/TRY_CAST is qualified separately below. Other arithmetic, rounding metadata, SIGN, mixed unary trees, stateful/failing producers and unknown configuration retain their existing routes. No native kernel, JNI boundary, ownership contract or production switch changes.

The result record and raw trials retain 64 configurations/6,400 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. DECIMAL(18,2)/(38,18) extrema and large signed coefficients match across GenericRowData/BinaryRowData and nullable/NOT NULL declarations. Representative bulk expressions are modern ABS(v) to STRING, modern -v to CHAR(8), and legacy ABS(v) to CHAR(44). Sixteen-row modern ABS-to-STRING controls remain separate. All checksums agree. The independently recompiled actual preceding encoder rejects this signature. Its prepared controls use released Java on RowData with Java-side timing, preserving the actual fallback reason. No JVM callback baseline is fabricated; the candidate must have no evaluator. The unchanged Rust decimal formatter, Calc and unary kernels retain their exact DSO.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 3.215-8.503x 3.728-8.758x 0
1.18.1 tiny control 8 1.302-3.462x 1.356-3.734x 0
2.2.1 bulk 24 3.215-8.064x 3.849-9.161x 0
2.2.1 tiny control 8 1.157-3.159x 1.475-3.526x 0

All 48 bulk cells beat stock; 0 of 16 tiny controls are slower. Independent stock-control drift spans 0.721-1.218x. All ranges are retained.

Six complete million-row source/blackhole configurations retain source generation, planning/startup, production JNI and both transposes. Modern DECIMAL(18,2) ABS-to-STRING, modern DECIMAL(38,18) negate-to-CHAR(8), and legacy DECIMAL(38,18) ABS-to-CHAR(44) run at parallelism one with two warmups/five measurements. Each native query requires consumed NativeCalc, no fallback and empty candidate evaluator descriptors. Previous jobs require actual planner fallback with zero native substitutions or work.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-abs 0.264686 (0.250927-0.269670) 0.209733 (0.206909-0.211031) 0.209105 (0.203733-0.212599) 0.210178 (0.207587-0.217686)
1.18.1 / modern-negate 0.378330 (0.370410-0.386081) 0.224428 (0.222141-0.229540) 0.342058 (0.334649-0.342707) 0.344431 (0.338319-0.346211)
1.18.1 / legacy-abs 0.291547 (0.280389-0.335874) 0.216920 (0.214796-0.218376) 0.239804 (0.237483-0.253204) 0.244747 (0.235303-0.253585)
2.2.1 / modern-abs 0.259245 (0.252922-0.268575) 0.206518 (0.201711-0.211899) 0.204024 (0.195745-0.214377) 0.215092 (0.194208-0.217394)
2.2.1 / modern-negate 0.376024 (0.372521-0.391604) 0.227104 (0.225205-0.233658) 0.344694 (0.340141-0.350055) 0.354460 (0.345592-0.365853)
2.2.1 / legacy-abs 0.279563 (0.275328-0.306825) 0.217126 (0.211098-0.219148) 0.235848 (0.231293-0.247688) 0.236468 (0.234912-0.251507)

Native/stock elapsed ratios span 0.593-0.797; native/previous ratios span 0.641-0.998. Ratios above one are slower. All unfavorable results and overlapping ranges remain. Prepared qualification does not establish universal end-to-end acceleration.

The modern ABS-to-STRING jobs overlap the previous fallback control: native/previous elapsed ratios are 0.998 on Flink 1.18 and 0.960 on Flink 2.2. Do not infer a robust whole-job improvement from these subsecond differences. Independent stock-process controls also differ; all paired samples remain available instead of selecting the more favorable control.

All 300 focused JVM invocations, four additional shared-state invocations and 12 existing fixture smokes pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row witnesses compare all values, schema, NULLs, IDs and cardinality across batches for eight DECIMAL precision/scale pairs from (1,1) through (38,38), nullable/NOT NULL declarations, modern/legacy mode, signed extrema, zeros and unit coefficients. They require consumed native work, no fallback and no evaluator. String consumers, grouping, lazy branches and empty filtered batches match stock. Failing and stateful inputs keep their released behavior and exact previous routes. The unchanged released DecimalData formatting oracle, 535 CastRules stock-oracle cases per pair, and the identical DSO's 1,019 core passes/two pre-existing ignores retain their preceding qualification; new runtime witnesses establish the changed route. Source/DSO hashes and independently recompiled baseline classes are retained.

Reproduce prepared qualification with integral_expression, DECIMAL input and SF_INTEGRAL_EXPRESSION='TRY_CAST(ABS(v) AS STRING)'. Complete jobs use sf.integral.expression=TRY_CAST(ABS(v) AS STRING) and sf.integral.decimalPrecision=18, sf.integral.decimalScale=2. Remaining conversions are tracked in #492.

Nullable DECIMAL arithmetic CAST to character

Nullable DECIMAL-only addition, subtraction and multiplication trees reuse the existing typed arithmetic kernels and exact character formatter. Each node retains its resolved precision/scale and overflow NULL. The proof also admits previously verified ABS/negate inputs; it excludes division, remainder, mixed numeric types, stateful/failing producers and unverified conversions. Both the computed source and the character result must be nullable. NOT NULL CAST retains the exact generated value/default/null-flag contract; arithmetic TRY_CAST is qualified separately below. No Rust kernel, JNI/ownership boundary or production option changes.

The result record and raw trials retain 32 configurations/3,200 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. DECIMAL(18,2)/(38,18) extrema and large signed coefficients match across GenericRowData/BinaryRowData and nullable declarations. Representative bulk expressions are modern v+v to STRING, modern vv to CHAR(8), and legacy (v+v)v to CHAR(44). Sixteen-row modern v*v-to-STRING controls remain separate. All checksums agree. The independently recompiled actual preceding encoder must bind a JVM evaluator; the candidate must have no evaluator. The unchanged Rust decimal formatter, Calc and typed arithmetic kernels retain their exact DSO.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 12 1.832-8.545x 1.819-10.077x 0
1.18.1 tiny control 4 1.430-2.923x 9.921-13.647x 0
2.2.1 bulk 12 1.799-8.606x 1.772-10.408x 0
2.2.1 tiny control 4 1.459-2.968x 9.659-13.076x 0

All 24 bulk cells beat stock; 0 of 8 tiny controls are slower. Independent stock-control drift spans 0.899-1.085x. All ranges are retained.

Six complete million-row source/blackhole configurations retain source generation, planning/startup, production JNI and both transposes. Modern DECIMAL(18,2) v+v-to-STRING, modern DECIMAL(38,18) vv-to-CHAR(8), and legacy DECIMAL(38,18) (v+v)v-to-CHAR(44) run at parallelism one with two warmups/five measurements. Each native query requires consumed NativeCalc, no fallback and empty candidate evaluator descriptors.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-add 0.256403 (0.240670-0.266953) 0.219047 (0.215546-0.224134) 0.254911 (0.244312-0.263866) 0.285142 (0.278247-0.293991)
1.18.1 / modern-multiply 0.378419 (0.360666-0.380856) 0.244122 (0.239227-0.254304) 0.366048 (0.359645-0.377322) 0.346962 (0.340675-0.351597)
1.18.1 / legacy-nested 0.331110 (0.329966-0.337205) 0.261276 (0.254386-0.275251) 0.327974 (0.316690-0.341677) 0.308758 (0.299839-0.322865)
2.2.1 / modern-add 0.247919 (0.246311-0.257852) 0.217974 (0.212238-0.226258) 0.260530 (0.251712-0.287248) 0.285710 (0.282034-0.291648)
2.2.1 / modern-multiply 0.374700 (0.372850-0.383431) 0.250588 (0.242618-0.255825) 0.385305 (0.381401-0.389040) 0.326326 (0.320013-0.336114)
2.2.1 / legacy-nested 0.337211 (0.328946-0.361897) 0.252416 (0.249894-0.256505) 0.344956 (0.331128-0.350719) 0.303112 (0.294999-0.315260)

Native/stock elapsed ratios span 0.645-0.879; native/previous ratios span 0.704-0.846. Ratios above one are slower. All unfavorable results and overlapping ranges remain. Prepared qualification does not establish universal end-to-end acceleration.

All 640 focused JVM invocations (including the shared-state suite) and 12 existing fixture smokes pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row witnesses compare all values, schema, NULLs, IDs and cardinality across batches for eight DECIMAL precision/scale pairs from (1,1) through (38,38), nullable declarations, modern/legacy mode, signed extrema, zeros and unit coefficients. They require consumed native work, no fallback and no evaluator. String consumers, grouping, lazy branches and empty filtered batches match stock. NOT NULL CAST and failing/stateful producers retain their preceding generated/fallback routes, with the overflow repair also revalidated. Arithmetic TRY_CAST has a separate qualification below. The unchanged released DecimalData formatting oracle, 535 CastRules stock-oracle cases per pair, and the identical DSO's 1,019 core passes/two pre-existing ignores retain their preceding qualification; new runtime witnesses establish the changed route. Source/DSO hashes and independently recompiled baseline classes are retained.

Reproduce prepared qualification with integral_expression, DECIMAL input and SF_INTEGRAL_EXPRESSION='CAST(v+v AS STRING)'. Complete jobs use sf.integral.expression=CAST(v+v AS STRING) and sf.integral.decimalPrecision=18, sf.integral.decimalScale=2. Remaining conversions are tracked in #492.

DECIMAL arithmetic TRY_CAST to character

DECIMAL-only addition, subtraction and multiplication trees reuse the existing typed arithmetic kernels and exact character formatter. Each node retains its resolved precision/scale and overflow NULL. The proof also admits previously verified ABS/negate inputs; it excludes division, remainder, mixed numeric types, stateful/failing producers and unverified conversions. TRY_CAST keeps a nullable character result with nullable or NOT NULL arithmetic sources, retaining overflow NULL instead of a non-nullable CAST projection default. NOT NULL CAST keeps its generated value/default/null-flag contract. No Rust kernel, JNI/ownership boundary or production option changes.

The result record and raw trials retain 64 configurations/6,400 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial use the prepared Rust/Arrow versus released Java/RowData boundary. DECIMAL(18,2)/(38,18) extrema and large signed coefficients match across GenericRowData/BinaryRowData and nullable/NOT NULL declarations. Representative bulk expressions are modern v+v to STRING, modern vv to CHAR(8), and legacy (v+v)v to CHAR(44). Sixteen-row modern v*v-to-STRING controls remain separate. All checksums agree. The independently recompiled actual preceding encoder rejects this signature. Its prepared controls use released Java on RowData with Java-side timing, preserving the actual fallback reason. No JVM callback baseline is fabricated; the candidate must have no evaluator. The unchanged Rust decimal formatter, Calc and typed arithmetic kernels retain their exact DSO.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 1.787-10.422x 2.035-11.123x 0
1.18.1 tiny control 8 1.416-3.495x 1.482-3.738x 0
2.2.1 bulk 24 1.890-10.314x 1.983-10.945x 0
2.2.1 tiny control 8 1.417-3.648x 1.464-3.773x 0

All 48 bulk cells beat stock; 0 of 16 tiny controls are slower. Independent stock-control drift spans 0.824-1.030x. All ranges are retained.

Six complete million-row source/blackhole configurations retain source generation, planning/startup, production JNI and both transposes. Modern DECIMAL(18,2) v+v-to-STRING, modern DECIMAL(38,18) vv-to-CHAR(8), and legacy DECIMAL(38,18) (v+v)v-to-CHAR(44) run at parallelism one with two warmups/five measurements. Each native query requires consumed NativeCalc, no fallback and empty candidate evaluator descriptors. Previous jobs require actual planner fallback with zero native substitutions or work.

Flink / input Candidate stock median (range), s Native median (range), s Previous stock median (range), s Previous median (range), s
1.18.1 / modern-add 0.261276 (0.245271-0.299600) 0.221004 (0.217863-0.234701) 0.221582 (0.212394-0.230538) 0.222617 (0.218642-0.236744)
1.18.1 / modern-multiply 0.370796 (0.351150-0.375298) 0.250625 (0.241326-0.258752) 0.328880 (0.314737-0.335114) 0.323991 (0.320824-0.335213)
1.18.1 / legacy-nested 0.337627 (0.323339-0.343045) 0.270040 (0.264440-0.275955) 0.302010 (0.300327-0.305214) 0.299326 (0.290850-0.303809)
2.2.1 / modern-add 0.247882 (0.239912-0.266768) 0.220313 (0.214473-0.228999) 0.222404 (0.210382-0.229336) 0.214866 (0.207088-0.232275)
2.2.1 / modern-multiply 0.382778 (0.373021-0.391164) 0.235763 (0.231195-0.237817) 0.339623 (0.332366-0.347701) 0.338182 (0.324840-0.350138)
2.2.1 / legacy-nested 0.332702 (0.329976-0.352564) 0.253855 (0.247578-0.268084) 0.296810 (0.288384-0.300062) 0.296049 (0.287876-0.301089)

Native/stock elapsed ratios span 0.616-0.889; native/previous ratios span 0.697-1.025. Ratios above one are slower. All unfavorable results and overlapping ranges remain. Prepared qualification does not establish universal end-to-end acceleration. The simple-addition jobs overlap their previous fallback controls: native/previous is 0.993 on Flink 1.18 and 1.025 on 2.2, with the latter median 2.5% slower. Neither simple-addition comparison establishes a stable complete-job improvement.

All 680 focused JVM invocations (including the shared-state suite) and 12 existing fixture smokes pass on released Flink 1.18.1/2.2.1 with JDK 11/17 (Java release 11). Complete 5,003-row witnesses compare all values, schema, NULLs, IDs and cardinality across batches for eight DECIMAL precision/scale pairs from (1,1) through (38,38), nullable/NOT NULL declarations, modern/legacy mode, signed extrema, zeros and unit coefficients. They require consumed native work, no fallback and no evaluator. String consumers, grouping, lazy branches and empty filtered batches match stock. NOT NULL CAST and failing/stateful producers retain their preceding generated/fallback routes, with the overflow repair also revalidated. Mixed numeric arithmetic TRY_CAST retains an explicit fallback witness. The unchanged released DecimalData formatting oracle, 535 CastRules stock-oracle cases per pair, and the identical DSO's 1,019 core passes/two pre-existing ignores retain their preceding qualification; new runtime witnesses establish the changed route. Source/DSO hashes and independently recompiled baseline classes are retained.

Reproduce prepared qualification with integral_expression, DECIMAL input and SF_INTEGRAL_EXPRESSION='TRY_CAST(v+v AS STRING)'. Complete jobs use sf.integral.expression=TRY_CAST(v+v AS STRING) and sf.integral.decimalPrecision=18, sf.integral.decimalScale=2. Remaining conversions are tracked in #492.

DECIMAL ABS over arithmetic

Verified DECIMAL-only addition, subtraction and multiplication now stay in Arrow through ABS, including nested ABS and same-scale precision widening. Each arithmetic node retains its resolved precision/scale and overflow NULL. Division, remainder, mixed numeric operands, rescaling, narrowing, mixed ABS/negation and unverified producers keep generated evaluation. This extends producer coverage using the existing typed arithmetic and ABS rules. A typed DECIMAL-array loop removes per-row input-type and legacy integer-error dispatch, retaining exact iterator capacity for the output. There is no JNI/ownership or option change. The separate NOT NULL projection-default repairs remain active for affected basic-type consumers; they are not claimed as JVM-free acceleration. Character CAST/TRY_CAST keep their own producer whitelist in #492; this ABS extension does not claim every enclosing character conversion is JVM-free.

The result record and raw trials retain 64 configurations and 6,400 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial compare released generated Java on prepared GenericRowData/BinaryRowData with production Calc on Arrow. Native timing stays inside Rust; initial JNI/C Data and perimeter transposes are excluded, while intrinsic previous callbacks remain inside. The independently recompiled actual preceding encoder includes the correctness repair; its retained preceding executable and DSO bind the real generated evaluator. The candidate requires empty evaluator descriptors and the intended native function work. Checksums match.

DECIMAL(18,2)/(38,18), nullable/NOT NULL inputs, signed extrema, large coefficients, addition, multiplication and nested producers cover 1,024-row bulk batches; 16-row multiplication controls remain separate. Every expression consumes ABS through >0 OR CAST(NULL AS BOOLEAN), retaining overflow NULL even for NOT NULL sources. These timings measure arithmetic ABS through a BOOLEAN consumer, not final DECIMAL output. Untimed allocation probes retain requested bytes and output buffers separately.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 1.086-4.339x 3.732-8.435x 0
1.18.1 tiny control 8 0.193-1.198x 7.920-12.712x 6
2.2.1 bulk 24 1.065-3.979x 3.684-7.591x 0
2.2.1 tiny control 8 0.171-1.171x 8.148-12.954x 6

The initial 64-cell investigation retained 11 slower bulk cells; the initial raw trials and full initial results remain in the record. A ten-second native stack sample located repeated arithmetic input/error dispatch and output-buffer growth. The typed-array loop removes those costs without changing arithmetic rules. Final native bulk timings improve by 1.111-1.982x over the initial native attempt, with independent stock-process drift retained.

All 48 final bulk cells beat stock and the actual preceding path; 12 of 16 tiny controls are slower than stock. Independent stock-process drift spans 0.643-2.211x; every raw trial is retained.

Six million-row row-source/blackhole configurations retain source generation, planning/startup, production JNI and both transposes. Nullable DECIMAL(18,2) addition and DECIMAL(38,18) multiplication/nested producers use the same BOOLEAN consumers, parallelism one, two warmups/five measurements. Both versions require consumed NativeCalc work and no fallback; the candidate requires no evaluator and the actual preceding version requires its generated route. Seconds are median (min-max):

Flink / producer Candidate stock Native Previous stock Previous
1.18.1 / add 0.156515 (0.149577-0.168454) 0.181894 (0.177573-0.183620) 0.172555 (0.170005-0.179224) 0.206288 (0.201754-0.210439)
1.18.1 / multiply 0.226769 (0.220124-0.239322) 0.214279 (0.211862-0.224900) 0.229178 (0.220161-0.240174) 0.278883 (0.271642-0.300167)
1.18.1 / nested 0.252536 (0.245212-0.256080) 0.213737 (0.210284-0.232305) 0.250923 (0.233308-0.260183) 0.319663 (0.302313-0.334127)
2.2.1 / add 0.156260 (0.149138-0.169494) 0.179437 (0.175440-0.200563) 0.163660 (0.155227-0.168385) 0.211092 (0.204884-0.224814)
2.2.1 / multiply 0.221233 (0.216625-0.238267) 0.202235 (0.196252-0.205395) 0.221499 (0.215712-0.227944) 0.283277 (0.277864-0.290121)
2.2.1 / nested 0.248131 (0.242918-0.252392) 0.221560 (0.216796-0.230291) 0.246680 (0.242554-0.248168) 0.311119 (0.309095-0.319681)

Native/stock elapsed-time ratios span 0.846-1.162; native/previous ratios span 0.669-0.882. Ratios above one are slower. Unfavorable results, overlapping ranges and independent stock controls remain visible; prepared qualification does not establish universal complete-job acceleration.

All 856 focused JVM invocations and 12 existing fixture smokes pass across released Flink 1.18.1/2.2.1 with JDK 11/17, compiled for Java release 11. Complete 5,003-row witnesses compare values, schema, NULLs, IDs and cardinality over 11 precision/scale pairs, including nullable extrema/overflow and safe NOT NULL values. They require consumed native work, no fallback and empty evaluators for admitted arithmetic ABS. Filters, grouping, lazy CASE branches, empty batches, selected failures, shared state and the #577/#581 projection-default regressions pass. Direct overflowing DECIMAL outputs retain stock ERROR/DROP behavior. The new release DSO passes a fresh 1,019 core tests with two pre-existing ignores; source/DSO hashes, allocator aliases and independently recompiled baseline classes are recorded.

Reproduce prepared qualification using integral_expression, DECIMAL inputs and SF_INTEGRAL_EXPRESSION='ABS(v*v)>0 OR CAST(NULL AS BOOLEAN)'. Complete jobs use the same sf.integral.expression with sf.integral.decimalPrecision=38 and sf.integral.decimalScale=18. Remaining unary/rounding producers are tracked in #491.

DECIMAL negation over arithmetic

DECIMAL-only addition, subtraction and multiplication now retain native unary minus, including same-operator nesting and same-scale precision widening. Each arithmetic node keeps its resolved precision/scale and overflow NULL. Negation uses the existing Arrow numeric kernel for arrays and exact i128 scalar dispatch; all legal DECIMAL coefficients have a representable negation. The batching structure follows Comet's Arrow negation; released Flink supplies the semantic rules. Rust, the release DSO, JNI/ownership and options are unchanged.

Division, remainder, mixed numeric inputs, rescaling, narrowing, mixed ABS/negation and unverified producers retain generated evaluation. The existing projection-default repairs remain active for NOT NULL basic-type consumers and are not claimed as JVM-free acceleration. Direct overflowing DECIMAL outputs retain ERROR/DROP enforcement. Character CAST/TRY_CAST retain their separate producer whitelist; admitting arithmetic negation does not extend every enclosing conversion.

The result record and raw trials retain 64 configurations and 6,400 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial compare released generated Java on matching prepared GenericRowData/BinaryRowData with production Calc on Arrow. Native timing stays inside Rust; initial JNI/C Data import/export and perimeter transposes are excluded, while intrinsic previous JVM callbacks remain inside. The independently recompiled actual preceding encoder includes the explicit CAST correctness repair and binds its real generated evaluator. Both versions use the identical, byte-verified preceding executable and DSO. The candidate requires empty descriptors and intended native work; checksums match.

DECIMAL(18,2)/(38,18), nullable/NOT NULL sources, signed extrema and large coefficients cover -(v+v), -(v*v) and -((v+v)*v) through >0 OR CAST(NULL AS BOOLEAN). This nullable consumer retains overflow NULL even for NOT NULL inputs. Timings measure composed arithmetic negation through a BOOLEAN consumer, not final DECIMAL output. Bulk batches contain 1,024 rows; 16-row multiplication controls remain separate. Untimed allocation probes retain requested bytes and output buffers separately.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 1.132-3.867x 3.654-9.141x 0
1.18.1 tiny control 8 0.224-1.174x 7.753-10.926x 6
2.2.1 bulk 24 1.089-4.076x 3.694-8.399x 0
2.2.1 tiny control 8 0.228-1.135x 7.319-10.605x 6

All 48 bulk cells beat stock and actual previous behavior; 12/16 tiny controls remain slower than stock. Independent stock-process drift spans 0.565-2.730x; every trial is retained.

Six million-row row-source/blackhole controls retain planning/startup, source generation, production JNI and both transposes. Nullable DECIMAL(18,2) addition and DECIMAL(38,18) multiplication/nested producers use the same BOOLEAN consumers, parallelism one, two warmups/five measurements. Both versions require consumed NativeCalc and no fallback; the candidate requires empty evaluators while the actual preceding control retains its generated route. Seconds are median (min-max):

Flink / producer Candidate stock Native Previous stock Previous
1.18.1 / add 0.159210 (0.154452-0.162179) 0.183171 (0.182496-0.189683) 0.165552 (0.158069-0.180885) 0.211707 (0.201569-0.221333)
1.18.1 / multiply 0.220525 (0.214375-0.243314) 0.204681 (0.204143-0.217983) 0.219037 (0.212691-0.241882) 0.273960 (0.270173-0.283251)
1.18.1 / nested 0.249028 (0.239493-0.252058) 0.218595 (0.209154-0.219580) 0.252982 (0.237527-0.265619) 0.325243 (0.317213-0.337302)
2.2.1 / add 0.156649 (0.152608-0.161492) 0.177944 (0.177173-0.185652) 0.166901 (0.157939-0.180192) 0.216225 (0.209954-0.223065)
2.2.1 / multiply 0.229529 (0.223492-0.241392) 0.207194 (0.202259-0.210548) 0.228994 (0.211501-0.237611) 0.281756 (0.274132-0.290865)
2.2.1 / nested 0.236939 (0.224113-0.247618) 0.206135 (0.204783-0.215894) 0.236026 (0.229298-0.253815) 0.308385 (0.304602-0.325335)

Native/stock elapsed-time ratios span 0.870-1.151; native/previous ratios span 0.668-0.865. Ratios above one are slower. Overlapping ranges, unfavorable samples and independent stock controls remain visible; prepared qualification does not establish universal complete-job acceleration.

All 1088 focused JVM invocations and 12 fixture smokes pass on released Flink 1.18.1/2.2.1 and JDK 11/17, compiled for Java release 11. Complete 5,003-row witnesses compare values, types, NULLs, IDs and cardinality over eleven precision/scale pairs. Admitted arithmetic negation requires consumed native work, no fallback and empty evaluator descriptors. Filters, grouping, CASE, empty batches, selected failures, shared state, ABS controls and #577/#581/#582 defaults remain verified. Safe numeric CAST and retained NULL-aware consumers preserve their native routes. The unchanged release DSO retains the earlier fresh 1,019 Rust core passes and two pre-existing ignores; that run is reused rather than presented as a new run. Source/runtime hashes, allocator aliases and independently recompiled baseline classes are recorded.

Reproduce prepared qualification using integral_expression and SF_INTEGRAL_EXPRESSION='-(v*v)>0 OR CAST(NULL AS BOOLEAN)'. Complete jobs use the same sf.integral.expression, sf.integral.decimalPrecision=38 and sf.integral.decimalScale=18. Further unary/sign/rounding coverage remains in #491.

DECIMAL SIGN over arithmetic

DECIMAL-only addition, subtraction and multiplication now retain native SIGN, including verified SIGN nesting and same-scale precision widening. Each arithmetic node preserves its resolved precision/scale and overflow NULL. SIGN reuses the existing typed Decimal128 kernel: ordinary decimals return scaled -1, zero or one; noncompact DECIMAL(p,p), p > 18, returns NULL for nonzero coefficients and scaled zero for zero. Released Flink supplies these rules; Comet's Arrow unary batching is a structural reference, not a source of Spark's result-type semantics. Production Rust, the DSO, JNI/ownership and options are unchanged.

SIGN retains a separate recursive proof so compact DECIMAL(p,p), p <= 18, intermediates stay generated even under arithmetic or widening. Division, remainder, mixed numeric operands, ABS/negation, rescaling, narrowing and unverified producers also retain generated evaluation. The #584 final-projection-default repair remains active for affected NOT NULL basic-type consumers; it is not claimed as JVM-free acceleration. Direct DECIMAL ERROR/DROP enforcement remains intact. Multiplication of DECIMAL(38,38) inputs resolves to scale 37 in released Flink, so its SIGN can represent one; input type alone does not determine whether SIGN returns NULL. Character CAST/TRY_CAST keep their separate producer proofs.

The result record and raw trials retain 128 configurations and 12,800 trials. Release/default-feature/mimalloc, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows/trial compare released generated Java on matching prepared GenericRowData/BinaryRowData with production Calc on Arrow. Native timing stays inside Rust; initial JNI/C Data import/export and perimeter transposes are excluded, while intrinsic previous JVM callbacks remain inside. The independently recompiled actual preceding encoder at 69e4198f includes the SIGN projection-default repair and binds its real generated evaluator. Both paths share the expanded four-type benchmark fixture and identical DSO. The executable changes only to add the two zero-integral-width corpora; it is not claimed byte-identical to the historical two-type executable. Separate old/new fixture smokes verify unchanged original-type checksums. The candidate requires empty descriptors and intended native work; every trial has a matching checksum.

DECIMAL(18,2)/(19,19)/(38,18)/(38,38), nullable/NOT NULL sources, signed extrema and large coefficients cover SIGN(v+v), SIGN(v*v) and SIGN((v+v)*v) through >0 OR CAST(NULL AS BOOLEAN). This nullable consumer retains overflow NULL even for NOT NULL inputs. Timings measure composed arithmetic SIGN through a BOOLEAN consumer, not final DECIMAL output. Bulk batches contain 1,024 rows; 16-row multiplication controls remain separate. Untimed allocation probes retain requested bytes and output buffers separately.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 48 1.495-7.225x 3.745-9.152x 0
1.18.1 tiny control 16 0.217-1.630x 8.005-13.690x 12
2.2.1 bulk 48 1.514-7.059x 3.636-9.193x 0
2.2.1 tiny control 16 0.221-1.644x 8.235-13.466x 11

All 96 bulk cells beat stock and actual previous behavior; 23/32 tiny controls remain slower than stock. Independent stock-process drift spans 0.721-2.134x; every trial is retained.

Eight million-row row-source/blackhole configurations retain planning/startup, source generation, production JNI and both transposes. Nullable DECIMAL(18,2) addition, DECIMAL(38,18) multiplication/nested and DECIMAL(38,38) multiplication use the same BOOLEAN consumers, parallelism one, two warmups/five measurements. Both versions require consumed NativeCalc and no fallback; the candidate requires empty evaluators while the actual preceding control retains its generated route. Seconds are median (min-max):

Flink / producer Candidate stock Native Previous stock Previous
1.18.1 / add 0.158934 (0.153983-0.176404) 0.179950 (0.178487-0.193787) 0.159119 (0.156121-0.168153) 0.212601 (0.202957-0.214075)
1.18.1 / multiply 0.226859 (0.223039-0.230969) 0.205721 (0.198910-0.217739) 0.232498 (0.219782-0.235035) 0.279492 (0.276106-0.288636)
1.18.1 / nested 0.260404 (0.253915-0.266903) 0.216729 (0.212598-0.222201) 0.263372 (0.247378-0.275634) 0.314720 (0.310512-0.334009)
1.18.1 / zero-integer 0.204853 (0.201356-0.214965) 0.195758 (0.191983-0.201677) 0.204023 (0.193139-0.204758) 0.261931 (0.246114-0.269036)
2.2.1 / add 0.164609 (0.159899-0.179596) 0.178191 (0.173004-0.184767) 0.158762 (0.157087-0.171613) 0.211523 (0.206122-0.228402)
2.2.1 / multiply 0.240034 (0.229098-0.243295) 0.204904 (0.196789-0.211282) 0.227246 (0.221138-0.231929) 0.294640 (0.287326-0.313013)
2.2.1 / nested 0.259564 (0.251532-0.264922) 0.213077 (0.209736-0.222779) 0.262493 (0.255530-0.272066) 0.331459 (0.315043-0.340202)
2.2.1 / zero-integer 0.205501 (0.198043-0.216030) 0.196261 (0.187968-0.209492) 0.205637 (0.195450-0.218607) 0.262806 (0.252995-0.285873)

Native/stock elapsed-time ratios span 0.821-1.132; native/previous ratios span 0.643-0.846. Ratios above one are slower. Overlapping ranges, unfavorable samples and independent stock controls remain visible; prepared qualification does not establish universal complete-job acceleration.

All 480 focused JVM invocations pass on released Flink 1.18.1/2.2.1 and JDK 11/17, compiled for Java release 11. All 36 fixture smokes pass on both Flink lines using JDK 17. Complete 5,003-row witnesses compare values, types, NULLs, IDs and cardinality over eleven precision/scale pairs, crossing the default 1,024-row Arrow batch limit. Admitted arithmetic SIGN requires consumed native work, no fallback and empty evaluator descriptors. Filters, grouping, CASE, empty batches, selected failures, shared state and SIGN/CAST projection defaults remain verified. The unchanged released ScalarFunctionsTest.testSign stock oracle also passes on each release/runtime pair; separate SQL witnesses establish native work. The unchanged release DSO retains the earlier fresh 1,019 Rust core passes and two pre-existing ignores; that run is reused rather than presented as a new run. Source/runtime hashes, allocator aliases and independently recompiled baseline classes are recorded. The fixture example changes; no production Rust source changes.

Reproduce prepared qualification using integral_expression and SF_INTEGRAL_EXPRESSION='SIGN(v*v)>0 OR CAST(NULL AS BOOLEAN)'. Complete jobs use the same sf.integral.expression, sf.integral.decimalPrecision=38 and sf.integral.decimalScale=38. Further unary/sign/rounding coverage remains in #491.

Mixed exact GREATEST

GREATEST over verified direct/literal integer and DECIMAL operands now casts each argument to Flink's resolved DECIMAL precision/scale before the existing Arrow extrema kernel. Integer capacity must be sufficient and no DECIMAL may lose scale. A DECIMAL whose metadata changes must be NOT NULL: released Flink can dereference it before checking its NULL flag. Nullable metadata changes, scale reduction and computed/unverified producers retain generated evaluation. These measurements qualify GREATEST; LEAST has its own qualification. No production Rust, JNI/ownership or configuration changes.

The result record and raw trials retain 112 configurations/11,200 trials against canonical main b0eb507c. The actual preceding encoder is independently compiled from that committed source; 1,390 unchanged Java source hashes match the complete previous build. Both paths use the same production DSO and expanded two-column fixture. A fresh release benchmark executable links the current production Rust. Separate old/new fixture smokes confirm that adding the second input leaves existing one-column corpora unchanged.

Release/default-feature, JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows per trial compare stock generated Java over prepared GenericRowData/BinaryRowData with production Calc over equal Arrow batches. Native timing stays inside Rust; initial JNI/C Data entry/export and perimeter transposes are excluded. Intrinsic previous JVM callbacks remain included. Every trial checksum matches. The new path requires empty evaluators and intended native work.

DECIMAL(18,2)/(19,19)/(38,18) with INT/BIGINT, both operand orders, a three-argument control, NULL/NOT NULL inputs, signed extrema, large coefficients and both row representations use GREATEST(v,n)>0 OR CAST(NULL AS BOOLEAN). This measures numeric extrema through a nullable BOOLEAN consumer, not final DECIMAL output. Bulk batches contain 1,024 rows; 16-row controls remain separate. Untimed allocation probes retain their requested-byte/output-buffer counts.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 38 1.959-14.317x 5.929-16.405x 0
1.18.1 tiny control 18 0.078-0.546x 6.045-8.433x 18
2.2.1 bulk 38 1.943-13.777x 5.827-17.625x 0
2.2.1 tiny control 18 0.081-0.580x 6.844-8.663x 18

All 76 bulk cells beat stock and the actual preceding implementation. All 36 tiny controls remain slower than stock, although they improve on the preceding JVM route.

Eight million-row row-source/blackhole configurations retain planning/startup, source generation, production JNI and both transposes, at parallelism one with two warmups/five measurements. Compact controls use DECIMAL(18,2) with INT/BIGINT, wide uses DECIMAL(38,18) with BIGINT, and fractional uses NOT NULL DECIMAL(19,19) with BIGINT. Both versions require consumed NativeCalc and no fallback; only the candidate requires empty evaluators. Seconds are median (min-max); independent stock controls and unfavorable samples are retained.

Flink / inputs Candidate stock Native Previous stock Previous
1.18.1 / compact-int 0.216645 (0.198815-0.224089) 0.217573 (0.214520-0.243753) 0.207151 (0.202257-0.211784) 0.247769 (0.244072-0.264842)
1.18.1 / compact-bigint 0.198637 (0.188381-0.220365) 0.209963 (0.201012-0.225706) 0.204936 (0.193470-0.217870) 0.250919 (0.240080-0.262518)
1.18.1 / wide-bigint 0.218562 (0.214016-0.223888) 0.233461 (0.219373-0.241594) 0.223502 (0.216588-0.238021) 0.260326 (0.259093-0.304801)
1.18.1 / fractional-bigint 0.210791 (0.210339-0.241606) 0.215612 (0.212575-0.223834) 0.224164 (0.219900-0.236927) 0.276282 (0.272253-0.284695)
2.2.1 / compact-int 0.203020 (0.195792-0.208881) 0.210581 (0.209462-0.214752) 0.195421 (0.192371-0.206530) 0.257562 (0.243652-0.261182)
2.2.1 / compact-bigint 0.198064 (0.197332-0.206831) 0.218062 (0.209673-0.236910) 0.207929 (0.203583-0.210888) 0.268514 (0.257670-0.275773)
2.2.1 / wide-bigint 0.219288 (0.214332-0.220702) 0.229656 (0.224162-0.231757) 0.219867 (0.212633-0.229906) 0.276054 (0.268153-0.318449)
2.2.1 / fractional-bigint 0.231148 (0.228257-0.240856) 0.224373 (0.220700-0.246401) 0.230612 (0.228576-0.238758) 0.294351 (0.283615-0.319577)

Native/stock elapsed-time ratios span 0.971-1.101; native/previous ratios span 0.762-0.897. Ratios above one are slower. These measurements do not establish universal complete-job acceleration.

All 796 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17, compiled for Java release 11. Complete 5,003-row witnesses verify values, types, NULLs, IDs, cardinality and actual native consumption, plus consumers, grouping, lazy/empty batches, nullable-metadata failures and the current DECIMAL projection-default repairs. The unchanged released GreatestLeastFunctionsITCase passes 146 stock-oracle invocations; 44 fixture smokes pass separately. Existing native production code and its DSO are unchanged; this is not a claim of a new Rust core-suite run. Further mixed numeric, rounding and producer scope remains in #491.

Mixed exact LEAST

Verified mixed integer/DECIMAL LEAST reuses the lossless casts and existing Arrow minimum kernel described for GREATEST. Nullable DECIMAL metadata changes, scale reduction, insufficient integral capacity and unverified/computed producers retain generated evaluation. This preserves released Flink's nullable-conversion failures. No production Rust, JNI, ownership or option changes.

The result record and raw trials retain 112 configurations/11,200 trials. The actual preceding encoder is independently recompiled from a7eaea10, the separate GREATEST commit on canonical main b0eb507c. Both paths use the same release/default-feature DSO, fresh benchmark executable and two-column fixture. JDK 17.0.12, Apple M4 Pro, one thread, 16 warmups/nine measurements and 131,072 rows per trial match the GREATEST methodology. Prepared GenericRowData/BinaryRowData and Arrow contain equal values/types/NULLs. Rust timing excludes initial JNI/import/export and perimeter transposes; intrinsic previous callbacks remain. Checksums match, with consumed native function work and empty candidate evaluator descriptors.

DECIMAL(18,2)/(19,19)/(38,18) with INT/BIGINT, both orders, a three-argument control, NULL/NOT NULL, signed extrema and large coefficients use LEAST(v,n)>0 OR CAST(NULL AS BOOLEAN). These timings consume a nullable BOOLEAN, not final DECIMAL output. Bulk batches contain 1,024 rows and tiny controls 16. Requested-byte/output-buffer allocation probes remain in the result record.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 38 1.911-13.370x 6.227-15.783x 0
1.18.1 tiny control 18 0.070-0.552x 6.984-8.190x 18
2.2.1 bulk 38 1.200-13.535x 5.905-15.711x 0
2.2.1 tiny control 18 0.073-0.532x 6.956-8.294x 18

All 76 bulk medians beat stock and previous. All 36 tiny controls remain slower than stock. Eight million-row row-source/blackhole controls retain startup/planning, source generation, production JNI and both transposes at parallelism one, with two warmups/five measurements. Seconds are median (min-max); independent stock controls and unfavorable samples are retained.

Flink / inputs Candidate stock Native Previous stock Previous
1.18.1 / compact-int 0.280502 (0.261990-0.367126) 0.315172 (0.205658-0.333249) 0.210022 (0.193419-0.293259) 0.272373 (0.242126-0.282771)
1.18.1 / compact-bigint 0.216120 (0.200640-0.235634) 0.229793 (0.212071-0.291695) 0.222998 (0.195001-0.241038) 0.343002 (0.275064-0.491480)
1.18.1 / wide-bigint 0.286441 (0.224139-0.691903) 0.308327 (0.235885-0.503705) 0.227179 (0.208461-0.231006) 0.280126 (0.269329-0.381201)
1.18.1 / fractional-bigint 0.219772 (0.213547-0.227718) 0.218514 (0.212634-0.229046) 0.216152 (0.211148-0.221203) 0.285881 (0.277306-0.305001)
2.2.1 / compact-int 0.204871 (0.198575-0.211131) 0.209832 (0.204694-0.221375) 0.204786 (0.196269-0.207228) 0.250877 (0.243003-0.270677)
2.2.1 / compact-bigint 0.205481 (0.203301-0.217226) 0.214091 (0.207014-0.227696) 0.201066 (0.197392-0.217963) 0.258565 (0.248192-0.292048)
2.2.1 / wide-bigint 0.215578 (0.211774-0.219300) 0.217707 (0.214002-0.225476) 0.214131 (0.210309-0.233518) 0.272811 (0.254320-0.355432)
2.2.1 / fractional-bigint 0.298122 (0.272389-0.354822) 0.255497 (0.245776-0.353708) 0.308293 (0.245968-0.486820) 0.474460 (0.336639-0.607842)

Native/stock elapsed ratios span 0.857-1.124 and native/previous 0.539-1.157. Several short jobs have wide sample ranges and differing stock controls; these are retained, not discarded or interpreted as universal complete-job acceleration.

All 264 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17, compiled for Java release 11, plus 44 fixture smokes. Complete 5,003-row runtime witnesses compare values/types/NULLs/IDs/cardinality and require consumed NativeCalc work with no fallback and empty evaluators. Consumers, grouping, selected failures, empty batches, nullable metadata failures and current DECIMAL projection defaults remain covered. The 146 unchanged GreatestLeastFunctionsITCase stock-oracle invocations recorded for GREATEST cover both extrema functions; they are not claimed as a second new run. Production Rust is unchanged. Further mixed numeric and producer scope remains in #491.

Mixed exact IF

Direct/literal mixed integer/DECIMAL IF branches use a typed Arrow selector when both conversions are lossless and the resolved DECIMAL precision exceeds 18. NULL conditions select ELSE; selected NULL values stay NULL. The selector writes one output array without filtered batches, separate cast arrays or a merge. Matching branch types retain the existing native CASE path. Computed/unverified mixed branches, scale reduction, insufficient integral capacity, stateful/volatile calls and compact-result mixed normalization retain their existing routes. No JNI, ownership or configuration change is introduced.

The first generic CASE investigation had 16/48 bulk cells slower than stock. Process sampling identified CaseBody::expr_or_expr, arrow_select::merge and MutableArrayData::extend. A representative 1,024-row compact-result control went from 109 allocations/68,528 requested bytes to 56/20,418 after fusion. Its fused median was still 7.84 ns/row against stock's 6.78 ns/row, so compact mixed normalization stays generated. The result record preserves this failed matrix and pilot; initial raw trials are separate from the final raw trials.

The final noncompact matrix retains 72 configurations/7,200 trials against independently recompiled actual preceding commit 289b69f0, on canonical main b0eb507c. The candidate uses the new release/default-feature DSO; previous controls use their independently preserved preceding DSO/executable and the same expanded logical corpus. Apple M4 Pro, JDK 17.0.12, one thread, 16 warmups/nine measurements, 131,072 rows/trial and matching prepared GenericRowData/BinaryRowData/Arrow inputs follow the preceding extrema method. Timing stays inside Rust and includes intrinsic JVM callbacks, allocations, conversions and output consumption; initial JNI/import/export and perimeter transposes are outside this small-function timer. Every trial checksum agrees and the admitted candidate requires empty evaluators.

DECIMAL(4,2)/(18,2)/(19,19)/(38,18) with INT/BIGINT, both branch orders, NULL/NOT NULL, signed extrema and large coefficients use IF(n>=0,v,n)>0 OR CAST(NULL AS BOOLEAN). Every selected bulk combination resolves to precision greater than 18. This measures a nullable BOOLEAN consumer, not final DECIMAL output. Bulk batches have 1,024 rows and tiny controls 16.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 24 1.051-5.253x 6.298-12.165x 0
1.18.1 tiny control 12 0.065-0.326x 6.936-8.202x 12
2.2.1 bulk 24 1.523-5.820x 5.314-14.075x 0
2.2.1 tiny control 12 0.066-0.319x 7.117-8.761x 12

All 48 bulk medians beat stock and previous. All 24 tiny controls remain slower than stock. Eight complete million-row source/blackhole jobs retain startup, JNI and both transposes, at parallelism one with two warmups/five measurements. fractional-int uses DECIMAL(19,19)/INT; compact-bigint uses DECIMAL(18,2)/BIGINT (the resolved result is noncompact); wide-bigint uses DECIMAL(38,18)/BIGINT; fractional-bigint uses DECIMAL(19,19)/BIGINT. Seconds are median (min-max); independent stock controls and unfavorable samples remain recorded.

Flink / inputs Candidate stock Native Previous stock Previous
1.18.1 / fractional-int 0.207477 (0.201241-0.208401) 0.223703 (0.215123-0.231077) 0.202862 (0.196273-0.206931) 0.275605 (0.272918-0.285621)
1.18.1 / compact-bigint 0.215457 (0.205784-0.220610) 0.224521 (0.213762-0.234266) 0.202049 (0.192554-0.209908) 0.258307 (0.256854-0.272890)
1.18.1 / wide-bigint 0.202431 (0.198542-0.209724) 0.220264 (0.214817-0.227628) 0.205333 (0.200273-0.229048) 0.293870 (0.281837-0.301714)
1.18.1 / fractional-bigint 0.213749 (0.211021-0.221769) 0.232539 (0.227125-0.249843) 0.206994 (0.200390-0.297613) 0.293367 (0.268386-0.505663)
2.2.1 / fractional-int 0.204839 (0.202637-0.214151) 0.235065 (0.228285-0.237179) 0.212130 (0.208221-0.220079) 0.282727 (0.272990-0.290860)
2.2.1 / compact-bigint 0.199342 (0.197950-0.216623) 0.217757 (0.211355-0.227652) 0.191884 (0.185234-0.202888) 0.253936 (0.243814-0.262679)
2.2.1 / wide-bigint 0.206973 (0.199895-0.215854) 0.235268 (0.226704-0.252028) 0.202695 (0.195838-0.210014) 0.277684 (0.274433-0.317087)
2.2.1 / fractional-bigint 0.213100 (0.206608-0.223398) 0.230963 (0.227135-0.242107) 0.205177 (0.198572-0.223956) 0.275922 (0.266477-0.288107)

Native/stock elapsed ratios span 1.042-1.148: every measured complete job remains slower than stock. Native/previous ratios span 0.750-0.869. This establishes improved function work and previous-path costs, not whole-job acceleration.

The new Rust selector passes scalar, nullable array, sliced-offset, full-BIGINT and empty-input regressions; 883 core tests pass with --no-default-features. The released upstream ScalarFunctionsTest.testIf stock oracle passes on Flink 1.18.1/2.2.1 with JDK 11/17. All 538 final focused JVM invocations pass (two released-1.18 negative-TIME controls abort as expected), plus 48 fixture smokes. Runtime SQL witnesses compare complete 5,003-row outputs, schemas, NULLs, IDs/cardinality, consumers, grouping, lazy branches, selected failures, empty filtered batches and current DECIMAL projection defaults, requiring consumed native work and empty evaluators for admitted paths. Final source hashes and focused results accompany the measurement record. A final admission-only restriction preserves matching-type CASE; measured mixed-signature lowering is unchanged. Further producers, compact mixed normalization and rounding remain in #491.

Mixed selection latest-main regression

The six GREATEST, LEAST, IF and DATE HOUR/MINUTE/SECOND commits rebase onto canonical main be842d2d with six equal patches in git range-diff. Historical measurements keep their original source hashes, encoders, DSOs and pre-rebase baseline IDs; they are not relabeled as new timings. The regression record maps those commits into the rebased history and records 768 fresh passing JVM invocations (192 per released Flink 1.18.1/2.2.1 and JDK 11/17 pair), compiled for Java release 11. They cover complete mixed-selection/DATE result maps and native consumption, shared state, DECIMAL defaults, calendar/timestamp controls and current nested bridge admission/fallback.

UTC LTZ calendar FLOOR

The FLOOR qualification snapshot admits direct/literal TIMESTAMP_LTZ to MONTH/YEAR in fixed zero-offset session zones using the existing calendar kernel. Released Flink uses the same Julian-date arithmetic there, including signed division, int overflow and the Gregorian cutover. Regional zones, nonzero offsets, computed inputs and other units retain their existing routes. CEIL is qualified separately below. Production Rust, JNI, ownership and configuration are unchanged.

The result record and 4,000 raw trials retain stock controls, an independently recompiled actual preceding encoder at 11e13486, source/DSO hashes and unfavorable results. Both paths use the same unchanged release/default-feature DSO and the same expanded timestamp fixture/executable; this is not relabeled as the preceding historical executable. On Apple M4 Pro, JDK 17.0.12, one thread, 16 warmups/nine measurements and 131,072 rows/trial, TIMESTAMP_LTZ(3)/(9), NULL/NOT NULL and GenericRowData/BinaryRowData/Arrow controls consume YEAR(FLOOR(v TO MONTH|YEAR))>0 OR CAST(NULL AS BOOLEAN). Inputs include full signed millisecond extrema, pre-epoch fractions, cutover and year boundaries. The timer starts/stops inside Rust, including intrinsic work and output consumption; initial JNI/import/export and perimeter transposes are outside this small-function timer. Checksums agree in every trial, and admitted candidate expressions require empty JVM evaluators.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 16 1.182-1.551x 2.286-2.821x 0
1.18.1 tiny control 4 0.193-0.253x 8.358-8.609x 4
2.2.1 bulk 16 1.224-1.632x 2.312-3.178x 0
2.2.1 tiny control 4 0.195-0.247x 8.123-8.654x 4

All 32 bulk medians beat stock and previous. All eight 16-row controls remain slower than stock. Eight complete million-row source/blackhole configurations retain startup, JNI and both transposes, at parallelism one with two warmups/five measurements. Seconds are median (min-max), with independent stock controls retained.

Flink / inputs Candidate stock Native Previous stock Previous
1.18.1 / millis-month 0.188040 (0.186749-0.205863) 0.221072 (0.214229-0.226698) 0.193261 (0.185584-0.202123) 0.245723 (0.235856-0.248278)
1.18.1 / millis-year 0.195539 (0.186410-0.203306) 0.220040 (0.211578-0.263018) 0.180761 (0.175119-0.191061) 0.238446 (0.236861-0.267134)
1.18.1 / nanos-month 0.180207 (0.177152-0.188744) 0.212674 (0.211109-0.219351) 0.185782 (0.179583-0.188839) 0.235585 (0.235366-0.246597)
1.18.1 / nanos-year 0.181047 (0.176542-0.182064) 0.216663 (0.211255-0.223228) 0.177336 (0.172870-0.186011) 0.229550 (0.228119-0.237410)
2.2.1 / millis-month 0.179602 (0.175361-0.193702) 0.212927 (0.209005-0.226058) 0.183578 (0.179895-0.192829) 0.246426 (0.234361-0.263776)
2.2.1 / millis-year 0.183784 (0.182589-0.197169) 0.231608 (0.228845-0.251674) 0.180244 (0.179069-0.201296) 0.239005 (0.228666-0.274734)
2.2.1 / nanos-month 0.185797 (0.180933-0.190350) 0.218771 (0.216324-0.227968) 0.189567 (0.181581-0.202551) 0.239713 (0.235645-0.246484)
2.2.1 / nanos-year 0.178197 (0.172020-0.185764) 0.218076 (0.212453-0.226665) 0.176826 (0.174746-0.191225) 0.237412 (0.233253-0.262087)

Native/stock elapsed ratios span 1.125-1.260: every measured complete job is slower than stock. Native/previous ratios span 0.864-0.969. The small-function improvement does not establish whole-job acceleration, and tiny batches remain an unfavorable workload.

All 68 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17, compiled for Java release 11. Complete 5,003-row runtime maps compare values, declared precision, NULLs, unique IDs/cardinality and full-range inputs across eight session zones; only normalized fixed-zero zones require empty evaluators. Negative admission controls at this snapshot cover other units, CEIL and computed inputs. The unchanged released ScalarFunctionsTest.testTemporalFloor stock oracle passes on all four pairs, separately from native-route witnesses. All 28 fixture smokes pass, including unchanged original-corpus checksums. The unchanged production DSO reuses the recorded 886 Rust core passes; the expanded release benchmark executable is rebuilt. Further LTZ scope remains in #504.

UTC LTZ calendar CEIL

CEIL/CEILING to MONTH/YEAR admits the same direct/literal LTZ and fixed-zero session-zone subset, independently of FLOOR qualification. Released Flink drops submillisecond fractions before its calendar calculation; the existing kernel reproduces this behavior at exact month boundaries and for negative/full-range milliseconds. Fixed-unit LTZ CEIL is different and remains outside this admission, together with nonzero/regional zones and computed inputs. No production Rust, DSO, JNI, ownership or option changes are needed.

The result record and 4,000 raw trials compare stock with independently recompiled actual preceding encoder bc9156b3 and candidate. They use the same timestamp fixture, production DSO and release/default-feature executable as FLOOR, with YEAR(CEIL(v TO MONTH|YEAR))>0 OR CAST(NULL AS BOOLEAN). The prepared-input Rust-side timer, types, signed extrema, NULLs, GenericRowData/BinaryRowData controls, batch sizes, 16 warmups/nine measurements and 131,072 rows/trial are unchanged. Every checksum agrees; candidate native expressions have empty JVM evaluators. All unfavorable controls and source/runtime hashes remain public.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 16 1.177-1.672x 2.539-4.442x 0
1.18.1 tiny control 4 0.203-0.249x 8.403-8.773x 4
2.2.1 bulk 16 1.157-1.665x 2.380-3.261x 0
2.2.1 tiny control 4 0.201-0.258x 8.495-8.830x 4

All 32 bulk medians beat stock and previous; all eight tiny controls lose to stock. The eight million-row source/blackhole configurations retain startup, JNI and both transposes, two warmups/five measurements and independent stock controls. Seconds are median (min-max).

Flink / inputs Candidate stock Native Previous stock Previous
1.18.1 / millis-month 0.186132 (0.178269-0.186337) 0.208164 (0.206946-0.214929) 0.182699 (0.181872-0.216983) 0.243392 (0.241391-0.262012)
1.18.1 / millis-year 0.188171 (0.175289-0.197888) 0.219593 (0.209214-0.226519) 0.185199 (0.178695-0.188247) 0.250715 (0.231815-0.271578)
1.18.1 / nanos-month 0.190383 (0.182234-0.201164) 0.209850 (0.205938-0.228390) 0.180241 (0.176636-0.205446) 0.249955 (0.244625-0.276685)
1.18.1 / nanos-year 0.196046 (0.178773-0.199058) 0.217153 (0.208887-0.223411) 0.196122 (0.187562-0.199845) 0.268520 (0.251645-0.291934)
2.2.1 / millis-month 0.180832 (0.177707-0.195561) 0.208945 (0.200662-0.229296) 0.188054 (0.184059-0.194765) 0.245933 (0.238274-0.261214)
2.2.1 / millis-year 0.184349 (0.181639-0.193377) 0.216672 (0.213815-0.233547) 0.187517 (0.181014-0.198702) 0.248477 (0.234889-0.258829)
2.2.1 / nanos-month 0.188166 (0.177473-0.191057) 0.215282 (0.212135-0.229300) 0.196054 (0.183815-0.204254) 0.244518 (0.239370-0.254048)
2.2.1 / nanos-year 0.183150 (0.180889-0.203244) 0.218773 (0.213459-0.236328) 0.189499 (0.181699-0.192151) 0.240628 (0.237761-0.265100)

Native/stock elapsed ratios span 1.102-1.195; every measured complete job still loses to stock. Native/previous ratios span 0.809-0.909. These results qualify bulk function work and quantify the remaining complete-job cost, without asserting whole-job acceleration.

All 68 new focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17, compiled for Java release 11. Full 5,003-row maps cover eight zones, NULLs, declared precision, millisecond extrema and nanoseconds, with consumed intended native work and empty evaluators only in qualified zones. Other units and computed round operands remain negative admission controls. The unchanged released temporal FLOOR/CEIL stock oracle is reused from FLOOR's four passing release/JDK controls; these are not claimed as a second new oracle run. All 28 fixture smokes pass. The production DSO and its recorded 886 Rust core passes are unchanged. A further 152 passing JVM invocations (38 per release/JDK pair) exercise the final FLOOR/CEIL implementation together with existing calendar/interval controls; that source/result record accompanies the CEIL artifact. Broader calendar work remains in #504.

BIGINT epoch millisecond conversion

Direct/literal TO_TIMESTAMP_LTZ(n,3) with BIGINT input reuses the existing epoch-millisecond kernel in ordinary Calc. Released bounds are inclusive -62167219200000..253402300799999; NULL and out-of-range values stay NULL. The timestamp shares the immutable input value buffer, uses zero nanoseconds, and places combined input/range validity on its parent. A packed range mask plus wordwise validity intersection replaces per-row Option handling. At this qualification, computed ordinary-Calc inputs, seconds and other numeric signatures retain existing routes; the later seconds qualification extends the direct BIGINT subset.

The initial unchanged kernel's nullable Flink 1.18 IS NULL pilot loses: stock 3.160 versus native 4.490 ns/row. CPU samples identify conversion, Vec growth/realloc and Vec validity copying. Sharing the millisecond buffer reduces a 1,024-row probe from 55 allocation requests/28,575 requested bytes to 29/6,047, but one isolated pilot still measures 0.990x stock/native. The final packed-mask pass uses 31 requests/6,231 bytes and measures 1.269 ns/row in that final pilot. These are request counts, not peak memory or bytes copied. The BOOLEAN output itself owns a new bitmap; timestamp buffer sharing is independently asserted for sliced inputs after input owners are dropped.

The result record preserves the failed initial pilot, first-pass matrix/jobs, allocation/profile observations and final qualification; 4,000 final raw trials retain all unfavorable controls. The release/default-feature prepared executable and both core DSOs use System, verified against retained Mach-O allocator symbols; these are not mimalloc-specific measurements. The actual preceding encoder/runtime at 871f269e is preserved separately, with its own DSO/executable hashes; both sides use the same expanded logical BIGINT corpus. Inputs include signed extrema, both exact conversion boundaries and adjacent invalid values, cutover and modern dates, NULL/NOT NULL and GenericRowData/BinaryRowData/Arrow. Apple M4 Pro, JDK 17.0.12, one thread, 16 warmups/nine measurements and 131,072 rows/trial follow the prepared Rust-side method: production operation and output consumption are timed inside Rust; initial JNI/import/ export and perimeter transposes are excluded. Intrinsic callback costs remain included in previous controls. Candidate expressions require empty evaluators and every checksum agrees. Consumers are YEAR/MONTH/SECOND comparisons and IS NULL, each planned independently.

Flink Workload Cells Stock/native median range Previous/native median range Slower than stock
1.18.1 bulk 16 1.343-3.910x 5.438-42.181x 0
1.18.1 tiny control 4 0.078-0.117x 9.571-23.935x 4
2.2.1 bulk 16 1.405-5.014x 5.860-40.166x 0
2.2.1 tiny control 4 0.111-0.124x 9.757-24.940x 4

All 32 bulk medians beat stock and previous. All eight 16-row controls remain slower than stock. Eight complete million-row source/blackhole configurations retain startup, JNI and both transposes, with two warmups/five measurements and independent stock controls. Seconds are median (min-max).

Flink / consumer Candidate stock Native Previous stock Previous
1.18.1 / year 0.172933 (0.160338-0.174794) 0.185525 (0.180171-0.207699) 0.173175 (0.163452-0.180049) 0.208390 (0.206822-0.222593)
1.18.1 / month 0.162458 (0.157122-0.182099) 0.186845 (0.182849-0.190759) 0.166398 (0.163162-0.172407) 0.212426 (0.204017-0.217323)
1.18.1 / second 0.161529 (0.157477-0.173762) 0.176890 (0.173200-0.186999) 0.166245 (0.163281-0.172074) 0.224440 (0.221124-0.231372)
1.18.1 / null 0.185503 (0.176530-0.189883) 0.195437 (0.187705-0.200961) 0.180688 (0.175215-0.185593) 0.227803 (0.224631-0.240018)
2.2.1 / year 0.178724 (0.165996-0.183075) 0.182469 (0.181150-0.194421) 0.165492 (0.157553-0.174396) 0.221700 (0.213897-0.251977)
2.2.1 / month 0.171128 (0.163341-0.172744) 0.177046 (0.174130-0.185138) 0.171559 (0.166280-0.172917) 0.216314 (0.211774-0.223302)
2.2.1 / second 0.161398 (0.159844-0.170333) 0.181007 (0.176693-0.183741) 0.166190 (0.159351-0.171686) 0.208888 (0.204120-0.223793)
2.2.1 / null 0.192193 (0.185861-0.195798) 0.199313 (0.196734-0.209371) 0.186166 (0.180513-0.191239) 0.231576 (0.230858-0.242957)

Native/stock elapsed ratios span 1.021-1.150: every measured complete job still loses to stock. Native/previous ratios span 0.788-0.890. Bulk function work improves; these short complete jobs do not establish whole-job acceleration.

All 148 focused JVM invocations pass across released Flink 1.18.1/2.2.1 and JDK 11/17, compiled for Java release 11. Complete 5,003-row maps verify values, types, NULLs, unique IDs/cardinality, both nullabilities, four zones, calendar consumers, typed conditional selection, all-filtered batches and the then-generated seconds/computed controls. Direct BIGINT seconds are admitted by the separate qualification below. Admitted paths require consumed NativeCalc work and empty evaluators. Existing UNNEST temporal bounds, zones and failure contracts remain covered. All 888 Rust core tests pass without default features, zero ignored, including sliced buffer ownership, parent/child NULL contracts, scalars and empty arrays. The unchanged released TemporalTypesTest Shanghai, UTC, boundary and invalid TO_TIMESTAMP_LTZ methods pass in 16 stock-oracle invocations, separately from native-route witnesses. All 28 fixture smokes pass, including old/new original-corpus checksum controls. Broader conversions remain in #503.

Function allocator provenance and matched IF controls

The artifact audit corrects blanket mimalloc labels on this change's mixed-selection and UTC-rounding records. Prepared Rust executables and JNI core DSOs are separate link products. CountingAllocator declares System, but only inspecting the final linked artifact establishes whether allocator symbols were rebound. The retained extrema executable imports System malloc while its shared ef29d700 JNI DSO uses mimalloc. The retained 4a10d804 UTC-rounding core DSO uses System. Original IF candidate and calendar executable artifacts are not retained for independent symbol reinspection; their hashes/raw measurements remain, without inventing an allocator attribution. These historical numbers are not a mimalloc-specific prepared-function qualification.

A fresh matched IF record and 7,200 raw trials use the same current release/System executable and core DSO on both sides. The independently compiled actual pre-feature encoder 289b69f0 keeps its generated evaluator; candidate uses the typed selector. This isolates the encoder/selection change on the same current primitive evaluator/transport; it is not a replay of the complete historical DSO. Input values/checksums, DECIMAL signatures, branch orders, nullabilities, GenericRowData/BinaryRowData, nullable BOOLEAN consumer, timing boundary, warmup/repetition and complete-job perimeter match the original IF qualification. Every checksum agrees and candidate native expressions require empty evaluators. Reused fixture and released-oracle controls are identified as reused, not newly executed IF oracles.

Flink Workload Cells Stock/native median range Prior encoder/native median range Slower than stock
1.18.1 bulk 24 1.243-6.803x 7.068-10.927x 0
1.18.1 tiny control 12 0.064-0.293x 6.613-8.869x 12
2.2.1 bulk 24 1.067-4.861x 6.423-10.811x 0
2.2.1 tiny control 12 0.066-0.289x 6.625-8.397x 12

All 48 bulk medians beat stock and the prior-encoder control. All 24 tiny controls still lose to stock. Complete million-row source/blackhole jobs retain startup, JNI and both transposes, two warmups/five measurements and independent stock controls. Seconds are median (min-max).

Flink / inputs Candidate stock Native Prior-control stock Prior encoder
1.18.1 / fractional-int 0.211896 (0.205774-0.214337) 0.226388 (0.214241-0.227146) 0.201043 (0.194222-0.210099) 0.270145 (0.267369-0.273343)
1.18.1 / compact-bigint 0.199994 (0.194616-0.210764) 0.224322 (0.211926-0.235112) 0.200780 (0.199217-0.201742) 0.266821 (0.257370-0.278869)
1.18.1 / wide-bigint 0.208675 (0.202621-0.213879) 0.225573 (0.219125-0.241540) 0.214507 (0.204583-0.217570) 0.291849 (0.284361-0.306561)
1.18.1 / fractional-bigint 0.215004 (0.209909-0.229445) 0.228181 (0.221224-0.245202) 0.208205 (0.202990-0.223996) 0.282070 (0.271877-0.284726)
2.2.1 / fractional-int 0.206288 (0.203484-0.206633) 0.225171 (0.223702-0.236765) 0.202783 (0.201772-0.209552) 0.287532 (0.285755-0.299882)
2.2.1 / compact-bigint 0.204214 (0.197145-0.210660) 0.226605 (0.215100-0.238641) 0.197689 (0.193144-0.209770) 0.256399 (0.251695-0.270723)
2.2.1 / wide-bigint 0.209169 (0.201884-0.212275) 0.226093 (0.221882-0.236119) 0.200475 (0.194004-0.205439) 0.278284 (0.276462-0.285379)
2.2.1 / fractional-bigint 0.201508 (0.196561-0.211933) 0.222281 (0.220729-0.230879) 0.207573 (0.204575-0.210746) 0.279755 (0.270325-0.294065)

Native/stock elapsed ratios span 1.061-1.122 and native/prior-encoder 0.773-0.884. Every measured complete job remains slower than stock. These matched controls replace an assumption of matched historical allocator settings; original trials/hashes remain available.

Canonical main advances to 76b1e46f through CI-status-only PR #586. All ten implementation/ record patches rebase equally in git range-diff; git diff 14f4c6dc HEAD -- src native is empty at the rebased implementation head recorded in the audit. No runtime source changes or new performance timings are inferred from that CI-only rebase. Historical base IDs remain attached to the measurements they actually produced.

BIGINT epoch second conversion

Direct/literal TO_TIMESTAMP_LTZ(n,0) with BIGINT input reuses the epoch conversion structure. Both released Flink jars check original seconds against inclusive [-62167219200,253402300799] before multiplying by 1000. Invalid values stay NULL even if their scaled long would wrap into a valid timestamp. The new millisecond buffer and zero nanoseconds use the existing component schema; input/range validity stays on the parent. Numeric conversion is zone independent. Computed ordinary-Calc/UNNEST inputs and other numeric, string, format or precision signatures keep their existing routes. There is no new switch.

The record and 4,000 raw trials compare the independently compiled actual preceding encoder 5d78fd4a with candidate on one current release/default-feature/System runtime, allocator, transport and expanded seconds fixture. Both final linked artifacts import System malloc and have no mimalloc allocator symbols. This isolates the encoder/kernel route change; it is not a replay of a complete historical DSO. The preceding encoder still binds its generated evaluator, while candidate requires empty evaluators. All checksums agree.

On Apple M4 Pro, Oracle JDK 17.0.12, one thread, 16 warmups/nine measurements and 131,072 rows/trial, equal GenericRowData/BinaryRowData/Arrow contain exact seconds bounds and adjacent invalid values, signed extrema and multiplication-overflow controls, cutover/modern/pre-epoch values and NULL/NOT NULL. Production work/output consumption are timed inside Rust with intrinsic callback costs retained in the preceding control. Initial JNI/C Data entry/export and perimeter transposes are excluded only from this prepared-function timer. Independently planned nullable BOOLEAN consumers are YEAR/MONTH/SECOND comparisons and IS NULL.

Flink Workload Cells Stock/native median range Prior encoder/native median range Slower than stock
1.18.1 bulk 16 1.342-3.448x 5.967-39.244x 0
1.18.1 tiny control 4 0.078-0.106x 9.399-21.834x 4
2.2.1 bulk 16 1.350-4.511x 5.777-37.158x 0
2.2.1 tiny control 4 0.099-0.110x 9.369-22.675x 4

All 32 bulk medians beat stock and the preceding encoder; all eight 16-row controls lose to stock. Complete million-row source/blackhole jobs retain startup, JNI, production conversions and both transposes, with two warmups/five measurements and independent stock controls. Seconds are median (min-max).

Flink / consumer Candidate stock Native Prior-control stock Prior encoder
1.18.1 / year 0.178340 (0.170356-0.219641) 0.184620 (0.178544-0.200105) 0.162921 (0.161419-0.182398) 0.213423 (0.206437-0.226048)
1.18.1 / month 0.172563 (0.166091-0.180995) 0.188816 (0.184118-0.202078) 0.185944 (0.168941-0.196921) 0.220979 (0.212758-0.236827)
1.18.1 / second 0.169010 (0.163062-0.184871) 0.185623 (0.175359-0.191390) 0.162442 (0.157704-0.167774) 0.216820 (0.208115-0.234500)
1.18.1 / null 0.178418 (0.169744-0.192398) 0.203734 (0.197714-0.234369) 0.176524 (0.171268-0.184801) 0.230295 (0.224920-0.239758)
2.2.1 / year 0.178993 (0.168456-0.192806) 0.191535 (0.186340-0.223990) 0.173104 (0.168654-0.181658) 0.216082 (0.213046-0.241996)
2.2.1 / month 0.181048 (0.166665-0.188743) 0.183546 (0.177635-0.202007) 0.171645 (0.162908-0.181974) 0.223246 (0.208871-0.239910)
2.2.1 / second 0.159443 (0.155857-0.194190) 0.184733 (0.170795-0.189403) 0.162390 (0.159129-0.170958) 0.210622 (0.208602-0.224377)
2.2.1 / null 0.194371 (0.178394-0.201297) 0.201201 (0.196232-0.232690) 0.191489 (0.185210-0.207430) 0.229590 (0.226761-0.258798)

Native/stock elapsed ratios span 1.014-1.159; all eight measured complete jobs remain slower than stock. Native/prior-encoder ratios span 0.822-0.886. These prepared bulk function wins do not establish complete-job acceleration.

All 212 focused JVM invocations pass on released Flink 1.18.1/2.2.1 x JDK 11/17 with Java release 11. Complete 5,003-row maps compare values, resolved types, NULLs, IDs/cardinality, exact seconds/millisecond bounds, signed extrema, four zones, calendar consumers, typed conditional selection, all-filtered batches and explicit generated controls for computed/ other-numeric inputs. The intended NativeCalc must consume runtime work and admitted functions have no JVM evaluator. Existing millisecond and UNNEST coverage stays covered. All 890 Rust core tests pass without default features and zero ignored, including parent/child validity, sliced/scalar/empty inputs and pre-scaling range checks. Sixteen freshly executed unchanged released TemporalTypesTest stock-oracle invocations and 28 fixture smokes pass separately; original-corpus controls compare the retained preceding executable with the expanded fixture. Broad temporal parsing/formatting and other conversions stay in #503.

Retained millisecond controls

A first seconds implementation passed its bulk gate but added redundant buffer-reference clones and selected range bounds at runtime. A retained Flink 1.18 millisecond IS NULL pilot then took 1.274 versus preceding-native 1.151 ns/row, with nonoverlapping measured ranges. The final implementation moves the scaled buffer directly and specializes fixed bounds by unit. Both complete seconds qualifications, the first 4,000 trials and all 800 first/final retained-millisecond raw trials remain in the result record. These diagnostics compare the retained actual previous native executable/DSO with candidate; their meaning differs from the preceding-encoder-only seconds control above.

Final millisecond medians (min-max) in ns/row:

Flink / consumer Retained preceding native Candidate native Candidate / retained median
1.18.1 / year 7.597 (7.446-7.771) 7.710 (7.575-9.072) 1.015
1.18.1 / null 1.251 (1.197-1.298) 1.221 (1.194-1.271) 0.976
2.2.1 / year 7.704 (7.678-7.897) 7.678 (7.516-7.769) 0.997
2.2.1 / null 1.161 (1.119-1.183) 1.238 (1.208-1.483) 1.067

The original 1.18 NULL regression is not stable in the final control, but the final 2.2 NULL pilot is still 6.7% slower than retained native, also with nonoverlapping ranges. The other controls overlap. These isolated short probes do not establish a universal unchanged-path speedup or absence of regressions; remaining small-operation overhead stays visible.

Narrow integer epoch conversion

Direct TINYINT/SMALLINT/INT TO_TIMESTAMP_LTZ(v,0/3) widens signed input losslessly with the existing Arrow integer cast, then reuses the qualified BIGINT epoch kernel. The two released Flink lines use the same long-based conversion. Computed inputs, non-integer signatures and existing UNNEST restrictions retain their preceding routes. No production Rust implementation, JNI contract, timestamp schema or switch changes.

The 2026-11-03 record compares released Flink 1.18.1/2.2.1 on JDK 17.0.12, Apple M4 Pro, one thread, and release/mimalloc-linked artifacts. Allocator symbols are checked for both executable and JNI DSO. Stock uses prepared BinaryRowData; Rust uses the same signed values in prepared Arrow, with and without NULLs, over all three input widths, precision 0/3 and YEAR/IS NULL consumers. Each engine processes 131,072 rows per trial, with 16 warmups and nine measurements. The Rust timer includes widening, allocation, result consumption and intrinsic callbacks; perimeter import/export is excluded only for this prepared function comparison.

The actual preceding encoder 490f458a is compiled against the same current runtime, fixture and allocator. It retains a JVM evaluator. Both its independent stock control and the candidate's paired stock control are retained separately, avoiding an overwritten stock baseline. This does not replay an entire historical native library.

Flink Bulk cells Stock / native median Previous / native median
2.2.1 24 2.313-4.193x 6.536-61.804x
1.18.1 24 1.682-3.062x 6.133-44.228x

All 48 1,024-row cells improve against both controls. All twelve 16-row controls lose to stock, with stock/native ratios 0.246-0.277x. Keep these small-batch limits: replacement of a JVM callback alone is not evidence of a stock-Flink speedup. All trials include warmups, checksums, paired/independent stock and prior evaluator measurements.

Complete jobs use the existing INT row source and blackhole sink, one million rows, UTC, two warmups and five measurements per engine with alternating order. Planning, startup, production JNI and both transposes remain timed. Candidates require consumed NativeCalc input and empty evaluators; previous controls require the generated route. Seconds, median, for the same YEAR/IS NULL consumers:

Flink Unit / consumer Paired stock Native Independent stock Previous evaluator
2.2.1 0 / YEAR 0.168406 0.174265 0.182519 0.217596
2.2.1 0 / NULL 0.180299 0.192963 0.177445 0.218229
2.2.1 3 / YEAR 0.169664 0.175953 0.171438 0.206674
2.2.1 3 / NULL 0.184774 0.197030 0.183020 0.227416
1.18.1 0 / YEAR 0.169060 0.178559 0.177995 0.204989
1.18.1 0 / NULL 0.168649 0.190953 0.168474 0.216864
1.18.1 3 / YEAR 0.172733 0.180505 0.166589 0.205832
1.18.1 3 / NULL 0.173585 0.194221 0.173924 0.223086

All eight complete native medians remain slower than stock (native/stock elapsed 1.035-1.132), while improving over prior generated execution (native/prior elapsed 0.801-0.884). The JSON retains every trial and range. These measurements qualify the bulk function path; they do not establish complete-job acceleration or timestamp serialization throughput.

A before-change runtime control matches values but fails six narrow-integer pure-route assertions with jvm=[TO_TIMESTAMP_LTZ]; BIGINT and computed controls pass. The final 44 focused JVM invocations pass on both releases and JDK 11/17, targeting Java release 11, with no skips. Full 5,003-row maps compare values, schema, NULLs and unique IDs; UTC/regional sessions, signed extrema, both precisions, CASE and all-filtered batches have runtime work/empty-evaluator assertions. Computed operands retain explicit generated-route controls. The 892-test Rust core result uses the unchanged production Rust implementation; the new example metadata checks actual BIGINT result types even when input is narrow. Full hosted CI is separate. Sixteen freshly executed, unchanged released TemporalTypesTest stock-oracle invocations pass separately across both Flink lines and JDKs. They confirm released behavior and do not count as native-runtime coverage.

UTC LTZ fixed-unit FLOOR

Direct/literal LTZ FLOOR in a fixed-zero session zone reuses the existing plain-timestamp fixed-width floor kernel. DAY/HOUR qualify on both released lines; MINUTE/SECOND/MILLISECOND qualify only on 2.2. Released 1.18 generates invalid Java casts for those sub-hour LTZ forms; explicit failure controls preserve that behavior. Regional/nonzero zones, computed operands and fixed-unit LTZ CEIL retain their preceding routes. No production Rust, JNI, timestamp schema or runtime switch changes.

The 2026-11-03 record uses released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc-linked artifacts. Equal prepared GenericRowData/BinaryRowData and Arrow contain signed-long extrema, normal/pre-epoch values, LTZ(3/9) fractional inputs and nullable/NOT NULL columns. The Rust-side timer retains production work, output consumption and intrinsic callbacks; initial import/export and perimeter transposes are excluded only from this prepared comparison. Each trial processes 131,072 rows with 16 warmups and nine measurements. The consumer is YEAR(FLOOR(v TO unit))>0 OR CAST(NULL AS BOOLEAN), not timestamp output serialization.

The independently compiled actual preceding encoder db05513e retains generated FLOOR on the shared current runtime, fixture and allocator. Candidate paired stock and independent prior-stock controls remain distinct; this is not a complete historical DSO replay. Symbol checks confirm both the prepared executable and JNI DSO resolve malloc to mimalloc.

Flink Bulk cells Stock / native median range Previous / native median range
2.2.1 40 1.099-1.612x 3.520-4.682x
1.18.1 16 1.160-1.479x 3.504-4.551x

All 56 bulk cells improve against stock and the previous encoder. All fourteen 16-row controls lose to stock (stock/native 0.146-0.175x); their overhead remains visible. All 7,000 raw trials retain warmups, checksums and both stock controls. Reproduce with the existing integral_expression fixture, TIMESTAMP_LTZ(3/9), the expression above, and each admitted unit.

Ten complete million-row jobs use the existing row source/blackhole fixture, UTC, LTZ(3/9), two warmups/five measurements and alternating engine order. Startup, JNI and both transposes remain in the timing. Candidate runs require consumed native Calc work and empty evaluators; previous runs require generated FLOOR. Seconds, median:

Flink Precision / unit Paired stock Native Independent stock Previous evaluator
2.2.1 3 / DAY 0.176608 0.211898 0.182302 0.241057
2.2.1 3 / HOUR 0.171679 0.204881 0.174518 0.231262
2.2.1 3 / MILLISECOND 0.170209 0.215291 0.170087 0.242998
2.2.1 9 / DAY 0.180133 0.205462 0.173028 0.235460
2.2.1 9 / HOUR 0.177703 0.207438 0.179124 0.231418
2.2.1 9 / MILLISECOND 0.174788 0.211231 0.174412 0.238524
1.18.1 3 / DAY 0.174992 0.209729 0.178677 0.242289
1.18.1 3 / HOUR 0.177254 0.209395 0.175687 0.231452
1.18.1 9 / DAY 0.175201 0.216809 0.178319 0.241498
1.18.1 9 / HOUR 0.184392 0.211330 0.178049 0.258221

Native/stock elapsed ratios span 1.141-1.265; native/previous ratios span 0.818-0.905. These complete-job results remain separate from bulk function qualification. The JSON retains every trial and range; prepared gains do not establish complete-job acceleration.

A before-change runtime control matches complete values/schema, then fails six zero-offset pure-route assertions with jvm=[FLOOR]. The final 44 focused JVM invocations and 136 adjacent calendar FLOOR/CEIL/encoder regressions pass across both releases and JDK 11/17, targeting Java release 11, with no skips. The runtime matrix compares full 5,003-row maps, types, NULLs, IDs/cardinality, signed-long extrema, fractions, precisions 0/3/9, zero-offset aliases, nonzero/regional generated controls, CASE and all-filtered batches. Admitted paths require consumed NativeCalc work and empty evaluator descriptors; computed/CEIL/unsupported-unit encoder controls remain explicit. Released 1.18's three smaller LTZ units fail independently under both stock and native-enabled fallback with the same code-generation cause/phase. Four freshly executed unchanged released ScalarFunctionsTest.testTemporalFloor stock-oracle invocations pass separately, not as native-runtime coverage. The existing 892-test Rust core result covers the unchanged production Rust; hosted CI remains separate. Remaining temporal work stays in #504.

UTC LTZ fixed-unit CEIL

A separate Rust kernel preserves released LTZ signed-remainder CEIL, including ignored submillisecond fractions and final long overflow. The ordinary TIMESTAMP CEIL kernel stays unchanged. Direct/literal inputs qualify only in normalized fixed-zero zones, for DAY/HOUR on both releases and MINUTE/SECOND/MILLISECOND on 2.2. The latter three forms retain the released 1.18 code-generation failure. Computed inputs, nonzero/regional zones and other units retain their existing routes. Component and primitive Arrow timestamps are accepted without changing JNI, timestamp transport or configuration.

The 2026-11-03 record compares released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc-linked artifacts. Both executable and JNI DSO have malloc/mi_malloc symbol identity checks. Prepared GenericRowData, BinaryRowData and Arrow contain identical signed-long extrema, normal and pre-epoch values, LTZ(3/9), submillisecond fractions and nullable/NOT NULL columns. Each trial processes 131,072 rows with 16 warmups and nine measurements. The Rust-side timer retains production operation, output consumption and intrinsic prior callbacks; initial import/export and perimeter transposes are excluded only from this function comparison. The consumer is YEAR(CEIL(v TO unit))>0 OR CAST(NULL AS BOOLEAN), not timestamp output serialization.

Actual preceding encoder e7b2a8e4 runs generated CEIL on the same current runtime, fixture and allocator. Its independent stock control remains distinct from candidate paired stock; this is not an entire historical DSO replay. The prior native artifacts are retained separately for provenance; this qualification isolates the encoder/kernel route change.

Flink Bulk cells Stock / native median range Previous / native median range
2.2.1 40 1.208-1.632x 3.798-7.901x
1.18.1 16 1.237-1.581x 3.862-4.969x

All 56 bulk cells improve against stock and the prior evaluator. All fourteen 16-row controls lose to stock (stock/native 0.150-0.179x). The 7,000 raw trials retain warmups, checksums and both stock controls. Reproduce with integral_expression, LTZ(3/9), the expression above and each admitted unit.

Ten complete million-row jobs retain startup, JNI and both transposes, using the existing row source/blackhole fixture in UTC with LTZ(3/9), two warmups, five measurements and alternating engine order. Candidate jobs require consumed NativeCalc work and empty evaluators; prior jobs require generated CEIL. Seconds, median:

Flink Precision / unit Paired stock Native Independent stock Previous evaluator
2.2.1 3 / DAY 0.182005 0.200002 0.181019 0.235959
2.2.1 3 / HOUR 0.185597 0.203753 0.172905 0.230548
2.2.1 3 / MILLISECOND 0.178166 0.208781 0.174501 0.227337
2.2.1 9 / DAY 0.175395 0.206331 0.177940 0.234895
2.2.1 9 / HOUR 0.180320 0.203949 0.181615 0.237350
2.2.1 9 / MILLISECOND 0.170617 0.206994 0.173575 0.235976
1.18.1 3 / DAY 0.173835 0.206550 0.184450 0.245386
1.18.1 3 / HOUR 0.174190 0.202289 0.181930 0.233150
1.18.1 9 / DAY 0.185248 0.206611 0.173145 0.227567
1.18.1 9 / HOUR 0.173767 0.207287 0.178209 0.232716

All ten complete native medians remain slower than stock: native/stock elapsed 1.098-1.213, native/previous 0.842-0.918. Every trial, range and independent control is retained. The prepared function gate does not establish complete-job acceleration.

All 896 release Rust core tests pass without default features, zero ignores. New tests exercise full signed millis, fractions, final-wrap overflow, distinct ordinary-TIMESTAMP behavior, primitive/component timestamps without coercion, scalar broadcasts, slices, NULL widths, empty/all-NULL arrays, ownership and rejected signatures. All 268 focused JVM invocations pass on both released lines and JDK 11/17, targeting Java release 11, with no skips. Complete 5,003-row maps compare values/types/NULLs/IDs/cardinality, precisions 0/3/9, zero-offset aliases, nonzero/regional generated controls, CASE and all-filtered batches. Admitted paths require consumed NativeCalc work and empty evaluators; computed/unsupported shapes and 1.18 code-generation failures retain explicit expectations. Adjacent fixed/calendar FLOOR/CEIL and narrow integer epoch routes are included. The actual before-change control matches values/schema but fails six zero-offset pure-route assertions with jvm=[CEIL]. Four fresh unchanged released ScalarFunctionsTest.testTemporalFloor oracle invocations pass separately; that upstream method contains both FLOOR and CEIL assertions and does not provide native-runtime coverage. Hosted CI remains separate. Further calendar/timezone work stays in #504.

Computed integer epoch conversion

TO_TIMESTAMP_LTZ(...,0/3) now reuses the existing exact epoch kernels for recursively verified integer arithmetic, lossless widening and rounding trees in ordinary Calc. The child retains its resolved integer width before any widening, preserving intermediate wrap and the released epoch bounds. Division/remainder, narrowing, unknown/stateful producers, other signatures and existing UNNEST restrictions retain their preceding routes. No native implementation, JNI, transport, dependency or configuration changes.

The 2026-11-03 record compares released Flink 1.18.1/2.2.1 on Oracle JDK 17.0.12, Apple M4 Pro, one thread, with release/mimalloc-linked artifacts. The executable and DSO retain verified malloc/mi_malloc identity. Equal prepared GenericRowData, BinaryRowData and Arrow contain nullable/NOT NULL TINYINT/SMALLINT/INT/BIGINT, signed extrema and released epoch boundaries. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. Rust times production operation and equivalent result consumption after import and before export, including the prior evaluator's intrinsic callbacks. Initial JNI/import/ export and perimeter transposes are excluded only from this prepared function comparison.

The add consumer is YEAR(TO_TIMESTAMP_LTZ(v+CAST(1 AS $TYPE),unit)); composed controls use TO_TIMESTAMP_LTZ(ROUND((v+CAST(1 AS $TYPE))*v,-2),unit) IS NULL, at precision 0 and 3. This is not a timestamp output serialization benchmark. Actual preceding encoder 998df4fd runs against the shared current runtime, fixture and allocator; it is not a whole historical DSO replay. Its independent stock control remains distinct from candidate paired stock.

Flink Bulk cells Stock / native median range Previous / native median range
2.2.1 40 1.683-9.329x 5.203-15.737x
1.18.1 40 1.584-5.455x 5.411-14.469x

All 80 bulk cells improve against stock and previous. All sixteen 16-row controls lose to stock (stock/native 0.146-0.178x). The 9,600 raw qualification trials retain warmups, checksums and independent controls. The separate 800 initial pilot trials are also retained, with their summaries in the result record. Reproduce with integral_expression, the types and expressions above, and matching epoch-seconds/millis corpora.

Twelve complete million-row jobs use the existing row source/blackhole fixture in UTC, with two warmups, five measurements and alternating engine order. They retain startup, JNI and both transposes. Candidate requires consumed NativeCalc work and empty evaluators; prior requires generated epoch conversion. Seconds, median:

Flink / input / precision / shape Paired stock Native Independent stock Previous evaluator
2.2.1 / INT-0-add 0.170323 0.173181 0.175443 0.207233
2.2.1 / INT-3-add 0.176385 0.176725 0.171943 0.207876
2.2.1 / BIGINT-0-add 0.166357 0.184379 0.164921 0.213727
2.2.1 / BIGINT-3-add 0.165382 0.176141 0.167736 0.221302
2.2.1 / BIGINT-0-round-product 0.195209 0.193531 0.205385 0.227742
2.2.1 / BIGINT-3-round-product 0.199279 0.194703 0.206845 0.229284
1.18.1 / INT-0-add 0.165017 0.175784 0.172271 0.207979
1.18.1 / INT-3-add 0.174836 0.175597 0.174498 0.205013
1.18.1 / BIGINT-0-add 0.175806 0.179489 0.173366 0.206188
1.18.1 / BIGINT-3-add 0.165088 0.184378 0.170795 0.214139
1.18.1 / BIGINT-0-round-product 0.196274 0.196922 0.196744 0.221564
1.18.1 / BIGINT-3-round-product 0.191578 0.192440 0.198439 0.222426

Native/stock elapsed spans 0.977-1.117; native/previous spans 0.796-0.889. Every trial and variability range is retained; the prepared function gate does not establish complete-job acceleration.

All 380 focused JVM invocations pass without skips across Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Complete 5,003-row maps compare values, resolved types, NULLs, IDs and cardinality for arithmetic overflow, epoch bounds and adjacent values, rounding and widening, nullable/NOT NULL inputs, CASE and all-filtered batches. Admitted paths require consumed NativeCalc work and empty evaluators. Narrowing, division, failing lazy/empty producers and recurring-DST extraction retain explicit generated controls. Adjacent direct epochs, UNNEST and fixed-unit LTZ rounding are included. The actual before control matches values/schema then fails all eight new pure-route assertions with jvm=[TO_TIMESTAMP_LTZ]; its unverified-producer control passes. Sixteen fresh unchanged released TemporalTypesTest oracle invocations pass separately, providing stock semantics, not native-runtime evidence. No Rust implementation changes from the preceding CEIL commit, which passed 896 release core tests without ignores. Hosted CI remains separate. Broader parsing/conversion work stays in #503.

Integer ROUND with a bounded runtime position

The planner proves an INT CASE position always returns -38..38 or NULL, with only failure-free BOOLEAN/logical/integer-comparison conditions. A typed selector chooses the first TRUE branch in one Arrow pass, retaining NULL conditions and selected NULLs. Integer ROUND consumes that position array, preserves resolved widths, signed overflow and HALF_UP ties, and uses no JVM evaluator. Nested bounded CASE and verified integer value producers are covered. Arbitrary position columns, extreme results and failing/stateful/unverified producers remain generated. TRUNCATE was literal-position only in this qualification; its later bounded-runtime extension is qualified separately below.

The 2026-11-03 record retains released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc-linked artifacts. Equal prepared GenericRowData, BinaryRowData and Arrow contain nullable/NOT NULL TINYINT/SMALLINT/INT/BIGINT, extrema and NULLs. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. Rust times production operation and equivalent output consumption after input import and before export. Prior intrinsic JVM callbacks remain timed. Initial JNI/import/export and perimeter transposes are excluded only from this small-function gate. Complete jobs retain those costs.

The short shape selects -2/-1; the wide-null shape selects -38/NULL/0; extra computed INT/BIGINT controls add one in the resolved input width before rounding. The secondary position-condition input is INT. Actual preceding encoder 7c8ffe00 is independently compiled against the shared current runtime, fixture and allocator. This isolates encoder and kernel changes, not a whole historical DSO replay. The independent prior-stock control remains separate from the candidate's paired stock control.

Flink Bulk cells Stock / native median range Previous / native median range
2.2.1 36 1.417-3.557x 3.872-10.091x
1.18.1 36 1.020-4.796x 3.963-8.528x

All 72 bulk cells beat paired stock and the previous generated route. Independent prior-stock/native spans 0.928-3.531x; 2 bulk cells have a lower independent stock median than native. These separate-process stock controls are retained rather than normalized to the paired controls. The smallest paired gain is modest and measurement ranges can overlap. Sixteen 16-row controls retain stock/native 0.126-0.291x; these small-function results do not establish complete-job acceleration. The 8,800 raw trials retain warmups, checksums and independent stock measurements.

Two failed/intermediate implementations remain public: the initial generic CASE pilot loses on every nullable wide-null width, and the selector-only partial matrix still loses on nullable INT/short GenericRowData (9.433 stock versus 10.452 native ns/row). The selector removes generic filtering/scattering; precomputed signed-64-bit powers then remove per-row i128 exponentiation/remainder, retaining an exact separate 10^19 wrapping case. Initial and intermediate artifacts/classes are hashed. Raw initial pilot, selector pilot, partial selector qualification, final pilot and retained literal controls prevent replacing failed measurements with only favorable final numbers. The literal controls replay the saved preceding executable/DSO independently; their variation is retained.

Eight complete million-row INT/BIGINT jobs use the bounded row source and blackhole sink, two warmups, five measurements and alternating engine order. They retain startup, JNI and both transposes. Candidate requires consumed NativeCalc and empty evaluator descriptors; the actual preceding encoder requires the ROUND evaluator. Seconds, median:

Flink / type / position shape Paired stock Native Independent stock Previous evaluator
2.2.1 / INT-short 0.182302 0.195996 0.183552 0.228636
2.2.1 / INT-wide-null 0.188430 0.200061 0.186238 0.227620
2.2.1 / BIGINT-short 0.196292 0.213548 0.193110 0.231555
2.2.1 / BIGINT-wide-null 0.197680 0.201215 0.186568 0.231936
1.18.1 / INT-short 0.188607 0.198215 0.195201 0.228789
1.18.1 / INT-wide-null 0.187114 0.203781 0.183182 0.229998
1.18.1 / BIGINT-short 0.216962 0.223382 0.197413 0.237526
1.18.1 / BIGINT-wide-null 0.196454 0.207811 0.195804 0.249493

Native/stock elapsed spans 1.018-1.089; native/previous spans 0.833-0.940. Every trial and full range remains in the record. No universal job speedup is claimed.

All 268 focused JVM invocations pass without skips across both releases and JDK 11/17, targeting Java release 11. Complete 5,003-row maps compare values, schema, NULLs, IDs and cardinality for every integer width, arithmetic/widening/nesting, NULL/negative/extreme positions, consumers and all-filtered batches. Admitted routes require actual NativeCalc consumption and empty evaluators. Unverified positions, out-of-range branches and failing conditions retain generated controls. The actual before encoder matches complete values and schema, then fails eight new pure-route assertions; its retained control passes. Four fresh unchanged released ScalarFunctionsTest.testRound stock-oracle invocations pass separately. All 900 release Rust core tests pass without ignores, including dynamic positions, 10^19 ties, slices, ownership, scalar broadcasts and empty/rejected inputs. Hosted CI is separate. Broader runtime/extreme numeric work remains in #491.

Reproduce prepared measurements with the release integral_expression example, SF_INTEGRAL_SECONDARY_TYPE=INT and expressions from the record; supply the independently compiled preceding encoder for SF_INTEGRAL_PREVIOUS=true. Complete-job controls use IntegralExpressionPipelineBenchmark, sf.integral.secondaryType=INT, the matching sf.integral.expression, and sf.integral.bigintInput=true for BIGINT.

Integer TRUNCATE with a bounded runtime position

INT/BIGINT TRUNCATE reuses the committed bounded-position selector and toward-zero integer kernel when the planner proves every CASE result is an INT literal in -38..38 or NULL and all conditions/value producers are failure-free. No new Rust implementation, opcode, JNI, transport, dependency or configuration switch is required. Direct narrow signatures retain released Flink's generated-compilation failure; explicit widening is supported. Arbitrary columns, extreme/out-of-range results and unverified/failing/stateful producers retain their preceding routes, including empty-batch and lazy-failure behavior.

The 2026-11-03 record measures released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro and one thread. Release/mimalloc-linked executable and DSO hashes are identical to the preceding ROUND qualification; allocator symbols remain verified. Equal prepared GenericRowData, BinaryRowData and Arrow contain nullable/NOT NULL INT/BIGINT, signed extrema and NULLs. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. Rust times production Calc and equivalent output consumption after import and before export. Intrinsic prior evaluator callbacks remain timed. Initial JNI/import/export and perimeter transposes are excluded only from this small-function gate; complete jobs retain those costs.

The short CASE chooses -2/-1; the wide-null CASE chooses -38/NULL/0. Extra computed controls add one at the original input width before truncation. Secondary condition input is INT. Actual preceding encoder 71b821c4 is independently compiled against the unchanged current runtime/fixture/allocator; it still evaluates runtime TRUNCATE in the JVM. This is an encoder comparison, not a whole historical DSO replay. Independent stock remains distinct.

Flink Bulk cells Stock / native median range Previous / native median range
2.2.1 20 3.801-15.378x 7.653-19.165x
1.18.1 20 3.915-16.393x 6.796-20.044x

All 40 bulk cells beat paired stock and the previous evaluator. Independent prior-stock/ native spans 3.970-16.296x; all independent controls and variability remain in the record. Eight 16-row controls retain stock/native 0.565-1.466x. The 4,800 raw trials include warmups and checksums; 400 initial pilot trials are separate. Reproduce with the release integral_expression example, SF_INTEGRAL_SECONDARY_TYPE=INT and the expressions above; the prior control supplies its independently compiled encoder and SF_INTEGRAL_PREVIOUS=true.

Eight complete million-row jobs retain the bounded row source, blackhole sink, startup, JNI and both transposes. Two warmups and five measurements alternate engine order. Candidate requires consumed NativeCalc work and empty evaluators; previous requires the TRUNCATE evaluator. Seconds, median:

Flink / type / position shape Paired stock Native Independent stock Previous evaluator
2.2.1 / INT-short 0.251914 0.192503 0.254554 0.285888
2.2.1 / INT-wide-null 0.219562 0.203798 0.211600 0.253925
2.2.1 / BIGINT-short 0.266054 0.212577 0.266745 0.299863
2.2.1 / BIGINT-wide-null 0.231545 0.210317 0.233702 0.272629
1.18.1 / INT-short 0.247101 0.189082 0.239740 0.280074
1.18.1 / INT-wide-null 0.226415 0.198152 0.212048 0.256564
1.18.1 / BIGINT-short 0.269613 0.217554 0.262311 0.316432
1.18.1 / BIGINT-wide-null 0.236126 0.214385 0.239261 0.274413

Native/stock elapsed spans 0.764-0.928; native/previous spans 0.673-0.803. Every trial and range is retained. The prepared function gate does not establish a universal job speedup. Reproduce complete jobs with IntegralExpressionPipelineBenchmark, sf.integral.secondaryType=INT, matching sf.integral.expression, and sf.integral.bigintInput=true for BIGINT.

All 344 focused JVM invocations pass without skips on Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Complete 5,003-row maps compare values, schema, NULLs, IDs and cardinality for bounded runtime positions, toward-zero behavior, arithmetic/widening/nesting, STRING/epoch consumers and all-filtered batches. Existing literal TRUNCATE and ROUND encoding/rounding controls are included, preserving the narrow-input compilation contract. Admitted paths require actual consumed NativeCalc and empty evaluator descriptors. The actual preceding encoder matches full values/schema then fails the four new pure-route assertions; its retained unverified-position control passes. Four fresh unchanged released ScalarFunctionsTest.testTruncate stock-oracle invocations pass separately. The exact unchanged Rust runtime already passed 900 release core tests without ignores, including dynamic-position TRUNCATE, slices, NULLs, ownership and empty arrays. Hosted CI remains separate; broader runtime/extreme numeric scope stays in #491.

Integer PRINTF with a literal format

On released Flink 2.2.1, verified integer inputs/literals and failure-free arithmetic, lossless widening or admitted rounding trees use an evaluator-free Arrow formatter with an ASCII literal format of at most 512 characters. Conversions are %s/%S, %d, %x/%X, %o and %%; width/precision are at most 256. The planner validates flags and indexing during planning using the running JDK's Formatter(Locale.US). Explicit index zero stays on its preceding route because JDK 11 accepts it and JDK 17 rejects it. Every producer must be failure-free, including unused arguments. Other formats/types and sensitive string consumers retain their existing routes. Flink 1.18 has no PRINTF and retains SQL rejection.

The new path borrows the literal format, reuses an output scratch String, and appends into one Arrow builder rather than creating an output String and materialized format per row. The existing compatibility formatter remains shared; this does not substitute Rust format syntax for Java Formatter. Parsing and per-conversion strings are still rowwise. Actual preceding encoder f910c352 is compiled independently against the shared current runtime, fixture and allocator. Its direct compatible ASCII batches can already execute certified Rust with a retained evaluator; computed producers retain generated evaluation. The previous control is not assumed to perform callbacks on every batch, nor is it a whole historical DSO replay.

The 2026-11-03 record uses released Flink 2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc-linked artifacts. Both executable and DSO verify identical malloc/mi_malloc addresses. Prepared GenericRowData, BinaryRowData and Arrow contain the same values, all signed integer widths, nullable/NOT NULL, extrema and NULLs. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. Rust times production Calc and equivalent output consumption after input import and before output export. Intrinsic callbacks remain timed; initial JNI/import/export and perimeter transposes are excluded only from the prepared function gate. Complete jobs retain those costs.

Short formatting uses PRINTF('%d/%s/%1$x',v,v); padded formatting uses PRINTF('%+020d:%2$-12.3S:%1$08X',v,v). Additional INT/BIGINT controls add one at the original input width in the first formatter argument. Secondary input is not needed.

Bulk cells Paired stock / native median range Previous / native median range Independent stock / native
40 2.532-5.180x 1.072-5.250x 2.462-5.094x

All 40 bulk cells beat paired stock and the actual preceding encoder. Independent stock has 0 lower medians than native; all independent measurements and variability remain. The eight 16-row controls retain paired stock/native 2.174-4.008x, including 0 controls slower than stock. The 4,800 raw trials include checksums and warmups.

The initial implementation merely removed the descriptor and reused certified formatting; its previous/native pilot spans 0.968-1.033x and loses several controls. Its 800 pilot trials, source and artifact hashes remain separate from the 800 optimized pilot trials. For the 1,024-row nullable INT short probe, scratch/builder reuse reduces allocation requests from 4,136/195,476 requested bytes to 3,102/91,842. Retained width-300 formats replay the saved pre-refactor executable/DSO and current artifacts, both retaining evaluators; initial/current median ratios span 0.987-1.005x. These 400 controls retain shared-formatter variation separately from the admitted scope.

Eight complete million-row INT/BIGINT jobs retain a bounded row source, blackhole sink, startup, JNI and both transposes; two warmups and five measurements alternate engine order. Candidate requires consumed NativeCalc work and empty evaluators. Previous requires its retained evaluator, which does not prove callbacks for every compatible direct batch. Seconds, median:

Flink / type / format shape Paired stock Native Independent stock Previous route
2.2.1 / INT-short 0.512502 0.282118 0.528230 0.308422
2.2.1 / INT-padded 0.888427 0.326280 0.970378 0.331879
2.2.1 / INT-short-computed 0.511739 0.288623 0.483461 0.507033
2.2.1 / INT-padded-computed 0.983899 0.327239 0.933144 0.967021
2.2.1 / BIGINT-short 0.566439 0.293006 0.580639 0.325968
2.2.1 / BIGINT-padded 0.962960 0.319097 0.932796 0.343919
2.2.1 / BIGINT-short-computed 0.556859 0.298003 0.558454 0.609614
2.2.1 / BIGINT-padded-computed 0.925704 0.330283 0.969479 1.019794

Native/stock elapsed spans 0.331-0.564; native/previous spans 0.324-0.983. All trials/ranges are retained; prepared function gains do not imply a universal complete-job speedup.

All 392 focused JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Runtime tests compare complete 5,003-row maps, resolved schemas, IDs, cardinality, all signed widths/extrema, NULLs, indexing, padding, precision, computed wrapping producers and all-filtered batches. Generated controls retain unverified formats/producers, JDK-dependent index zero and CHAR_LENGTH representation guards. Existing certified/PRINTF tests qualify the shared formatter on Flink 2.2. Before controls match full maps/schema then fail all eight new pure-route assertions; the generated control passes. Two unchanged released StringFunctionsITCase.printfTestCases runs pass all 47 original assertions each, separately as stock oracles. All 902 release Rust core tests pass without ignores, including slices, ownership, empty arrays and rejected contracts. Hosted CI remains separate; broader formats/types/consumers stay in #497.

Reproduce prepared controls with the release integral_expression example and SF_INTEGRAL_EXPRESSION above; supply independently compiled previous encoder and SF_INTEGRAL_PREVIOUS=true for that control. Complete jobs use IntegralExpressionPipelineBenchmark, matching sf.integral.expression, and sf.integral.bigintInput=true for BIGINT. Other resources/settings remain identical.

Integer TRY_CAST

Verified TINYINT/SMALLINT/INT/BIGINT TRY_CAST reuses exact native primitive conversion. Both modern and legacy Flink modes wrap integer narrowing overflow; they do not yield NULL on out-of-range integer input. NULL input stays NULL and resolved schemas remain unchanged. Inputs/literals, failure-free integer arithmetic, lossless widening and admitted rounding producers qualify; division, narrowing producers and other unverified trees retain fallback. Errors while computing an operand are not conversion errors. Same-type planner rewrites are covered by parity but excluded from performance claims.

The first implementation only admitted the existing CAST kernel. All 112 bulk cells beat paired stock, but four independent previous medians remain slightly lower than native. The initial 12,400 trials, 600 pilot trials, source and runtime hashes remain. The integer kernel used an intermediate i64 column and nullable row iteration. Following Comet's legacy integer-cast structure, the final implementation maps the original typed values in one Arrow unary pass and retains the immutable NULL buffer. Flink's own overflow rules apply in both modes; Spark's ANSI/try semantics are not imported. DECIMAL/floating paths remain separate. A nullable INT-to-TINYINT 1,024-row allocation probe drops from 28 requests/11,007 requested bytes to 21/2,279.

The final record measures released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc-linked artifacts, with verified allocator symbols. Equal prepared GenericRowData, BinaryRowData and Arrow contain all signed integer source/destination widths, nullable/NOT NULL, extrema and NULLs. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. Native timing starts/stops inside Rust, after import and before export, retaining production Calc and equivalent output consumption. Only initial JNI/import/export and perimeter transposes are excluded from this small-function gate. Complete jobs retain them.

Actual preceding encoder 04e0cf86 rejects these signatures. Its Java prepared control verifies the exact fallback reason, then measures the released generated expression on RowData without artificial native overhead. Independent stock and previous remain distinct samples of that stock fallback. Both controls use the same final fixture, which checks and consumes the resolved output width. Short/wide destination casts and extra computed inputs v+CAST(1 AS source) are qualified. No historical DSO callback is invented for the prior.

Flink Bulk cells Paired stock / native Previous fallback / native Independent stock / native
2.2.1 56 2.468-15.672x 2.435-13.883x 2.468-13.879x
1.18.1 56 2.463-12.864x 2.440-12.919x 2.445-12.987x

All 112 bulk cells beat paired stock and the actual preceding fallback. Twelve 16-row controls retain stock/native 0.276-0.368x, including 12 slower controls. The 12,400 final trials retain warmups, checksums and variability; 600 optimized pilot trials remain separate. Ten ordinary CAST controls replay saved initial executable/DSO and optimized artifacts on the same final encoder/fixture. Eight integer initial/current median ratios span 4.190-8.023x; two separate unchanged DECIMAL controls span 0.987-1.040x. All variation is retained.

Eight complete million-row INT/BIGINT jobs retain the bounded row source, blackhole sink, startup, JNI and both native transposes. Two warmups and five measurements alternate order. Candidate requires consumed NativeCalc work and empty evaluators; actual previous must fall back with the exact expected reason and zero native substitutions/operators. Seconds, median:

Flink / source / destination Paired stock Native Independent stock Previous fallback
2.2.1 / INT-TINYINT 0.162741 0.172488 0.146897 0.148155
2.2.1 / INT-BIGINT 0.160180 0.174861 0.143499 0.144603
2.2.1 / BIGINT-TINYINT 0.154210 0.164889 0.147629 0.148952
2.2.1 / BIGINT-INT 0.154995 0.168157 0.147347 0.152415
1.18.1 / INT-TINYINT 0.165774 0.170204 0.150125 0.144074
1.18.1 / INT-BIGINT 0.163677 0.172033 0.151377 0.149134
1.18.1 / BIGINT-TINYINT 0.174743 0.177714 0.156687 0.154429
1.18.1 / BIGINT-INT 0.163483 0.180986 0.168767 0.164599

Native/stock elapsed spans 1.017-1.107; native/previous spans 1.100-1.209. Unfavorable complete jobs and independent controls remain; prepared gains do not establish universal complete-job acceleration.

All 716 focused JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Complete 5,003-row maps/schema compare every integer source/destination width, NULLs, extrema, modern/legacy modes, computed overflow, filters, CASE and all-filtered batches. Failing/unverified producers retain explicit fallback and lazy/empty/failure controls. Shared ARRAY CAST and adjacent rounding/epoch/ PRINTF paths are rechecked. Sixteen actual-before cases match full maps/schema, then fail the old fallback assertion; two old filter controls stop at missing native substitution before their final map assertion. Four unchanged released CastRulesTest integer-oracle runs each pass 41 original assertions, separately as stock semantics. All 903 release Rust core tests pass, including six integer narrowing pairs, signed wrapping, slices, empty arrays, NULL-buffer sharing, scalar materialization and input ownership. Hosted CI remains separate; broader scalar conversion work stays in #492.

Reproduce native prepared controls with release integral_expression, SF_INTEGRAL_EXPRESSION='TRY_CAST(v AS target)', equal source type and nullable/row-layout arguments. The preceding control uses IntegralExpressionBenchmarkFunctions on the actual prior encoder with sf.integral.previous=true, sf.integral.fallbackControl=true and an exact sf.integral.fallbackReason (for example unsupported TRY_CAST BIGINT followed by U+2192 and INTEGER). Complete jobs use IntegralExpressionPipelineBenchmark, matching sf.integral.expression, BIGINT input flag when needed, and those previous fallback controls.

Integer boolean PRINTF formats

On released Flink 2.2.1, %b/%B treats any non-NULL integer object, including zero, as true/TRUE; NULL becomes false/FALSE. Bounded literal ASCII formats use the existing evaluator-free integer formatter, including precision, space padding, explicit indices and argument reuse. This does not admit BOOLEAN-object or other argument types. Invalid flags and unverified producers keep Flink's NULL/error and lazy/empty behavior. Flink 1.18.1 has no PRINTF and retains its released SQL rejection.

The record uses released Flink 2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc-linked artifacts. Equal prepared GenericRowData, BinaryRowData and Arrow include all four integer widths, nullable/NOT NULL, signed extrema and zero. Each trial consumes 131,072 rows, with 16 warmups/nine measurements. The 48 cells cover short and padded formats, plus computed INT/BIGINT arithmetic. Timers start/stop inside Rust after import and before export, retaining production Calc, equivalent output consumption and intrinsic callbacks. Stock starts on prepared RowData; only initial JNI/import/export and perimeter transposes are outside the small-function gate. Complete jobs retain them.

The actual preceding encoder and formatter-admission helper, plus saved whole DSO and executable from 5d46dde1, run against the same final fixture/JDK/data. Both artifacts retain verified mimalloc aliases. This preserves the original integer boolean format's JDK route instead of accidentally giving the preceding certified path the new kernel. Existing compatible certified formats are not described as universal Java callbacks.

The four numeric-format controls and reverse-order replay retain whole-runtime before/after differences for existing numeric formats. Both BIGINT controls have lower preceding-runtime medians in both orders, by approximately 3-6%. The new-format gains below do not establish an improvement for those unchanged numeric signatures. An attempted non-inlined default helper does not recover them (four diagnostic cells, initial/current 0.938-1.011x); that extra indirection is discarded. Its rejected patch, source/artifact hashes and successful 904-test Rust result remain separately recorded. The exact originally measured production source, DSO and executable are restored, checked by hash, then requalified with the final SQL tests including independently nullable indexing arguments. The residual numeric-format cost differences still need attribution/profile work; no cause is inferred from a source branch alone.

Scope Cells Paired stock / native Actual previous / native Independent stock / native
Bulk (1,024 rows) 40 4.947-7.411x 5.723-8.108x 5.346-7.519x
Tiny (16 rows) 8 3.789-5.423x 11.212-12.205x 3.853-5.508x

All bulk cells beat paired stock and the actual previous implementation. Tiny controls retain 0 stock losses and 0 previous losses. The 4,800 trials retain warmups, checksums and variability. Four numeric-format controls replay initial/current artifacts on the same final encoder and fixture. Their initial/current median ratios span 0.945-1.024x; all variation remains.

Eight complete million-row INT/BIGINT jobs include direct/computed short/padded formats. They retain the bounded row source, blackhole sink, startup, JNI and both transposes. Two warmups and five measurements alternate stock/native. Each native control requires consumed NativeCalc work; new paths require empty evaluators and previous paths require their original retained evaluators. Seconds, median:

Flink / source / shape Paired stock Native Independent stock Actual previous
2.2.1 / INT-short 0.621235 0.259344 0.606269 0.634951
2.2.1 / INT-padded 0.750707 0.273240 0.784988 0.820727
2.2.1 / INT-short-computed 0.597576 0.266733 0.607545 0.619766
2.2.1 / INT-padded-computed 0.809255 0.270421 0.795140 0.804340
2.2.1 / BIGINT-short 0.671868 0.266082 0.578773 0.597283
2.2.1 / BIGINT-padded 0.828991 0.269570 0.783403 0.819152
2.2.1 / BIGINT-short-computed 0.673516 0.257944 0.571059 0.568660
2.2.1 / BIGINT-padded-computed 0.828161 0.274525 0.751099 0.753517

Native/stock elapsed spans 0.325-0.446; native/previous spans 0.329-0.454. These are measured workloads; the prepared gate does not establish universal complete-job acceleration.

All 716 focused JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Complete 5,003-row runtime maps/schema cover integer widths, NULLs, signed extrema/zero, computed producers, boolean format precision/ padding/indexing with independently nullable arguments, all-filtered batches, remaining generated controls and adjacent scalar/ARRAY CAST paths. Pure claims require consumed NativeCalc work and empty evaluators. The actual preceding control compares full maps/schema then fails eight intended pure-route assertions; its generated control passes. Two fresh unchanged released PRINTF oracle runs pass 47 original assertions each, separately as stock semantics. All 904 release Rust core tests pass, including sliced/empty/NULL inputs and input ownership. Hosted CI remains separate. Remaining format/type/consumer scope stays in #497.

Reproduce with release integral_expression, SF_INTEGRAL_EXPRESSION set to PRINTF('%b/%1$B/%1$.2b',v,v), and matching signed type/nullability/row-layout arguments. The previous control supplies its saved DSO/executable, separately compiled preceding encoder/helper and SF_INTEGRAL_PREVIOUS=true. Complete jobs use IntegralExpressionPipelineBenchmark with matching sf.integral.expression, sf.integral.bigintInput=true for BIGINT and previous route settings. All resources and data otherwise match.

BOOLEAN-to-integer TRY_CAST

Verified BOOLEAN input/literal, logical, NULL-check and integer-comparison trees now use the existing primitive native cast for TRY_CAST to TINYINT/SMALLINT/INT/BIGINT. TRUE is 1, FALSE is 0 and NULL stays NULL in modern/legacy modes. Conversion cannot fail; failing or unverified producers keep their preceding route. Ordinary CAST already becomes Flink's CASE expression during released planning and needs no admission change. This follows Flink's BooleanToNumeric rule and Comet's DataFusion-compatible primitive conversion. There is no new native opcode, JNI entry point, setting or Rust production change.

The record retains 48 cells and 4,800 raw trials, including warmups/checksums/ranges. Released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc artifacts are used. Equal prepared GenericRowData/BinaryRowData and Arrow contain true, false and NULL with nullable/NOT NULL schemas and four target integer widths. Computed controls use NOT v OR v IS NULL. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. The native timer runs inside Rust after import and before export, covering production Calc and equivalent output consumption. Stock runs on prepared RowData; setup and perimeter conversion are outside only the small-function gate. Intrinsic costs remain timed. The actual previous encoder from 8ccec369 verifies each exact fallback reason and executes released generated Java without invented native overhead. Current and previous controls use identical logical data/resources. The existing release runtime and its mimalloc aliases are checked by hash and reused unchanged.

Scope Cells Paired stock / native Actual previous / native Independent stock / native
Bulk (1,024 rows) 40 1.092-2.819x 1.075-2.135x 1.071-2.147x
Tiny (16 rows) 8 0.281-0.301x 0.277-0.302x 0.277-0.303x

All bulk cells beat stock and actual previous fallback. Tiny controls retain 8 stock losses and 8 previous losses; small batches are not described as accelerated when they lose.

Eight million-row complete jobs retain bounded row input, blackhole output, startup, JNI and both transposes. Two warmups/five measurements alternate order. Candidate controls require consumed NativeCalc work and empty evaluator descriptors. Actual previous controls require exact fallback, zero substitutions and zero native operators. Seconds, median:

Flink / source / target Paired stock Native Independent stock Actual previous
2.2.1 / BOOLEAN-TINYINT 0.170140 0.167920 0.143392 0.145854
2.2.1 / BOOLEAN-SMALLINT 0.158451 0.162561 0.142876 0.142270
2.2.1 / BOOLEAN-INT 0.167679 0.173542 0.146190 0.146495
2.2.1 / BOOLEAN-BIGINT 0.161557 0.169256 0.140627 0.157842
1.18.1 / BOOLEAN-TINYINT 0.155952 0.170372 0.141720 0.140429
1.18.1 / BOOLEAN-SMALLINT 0.165398 0.164696 0.136228 0.137099
1.18.1 / BOOLEAN-INT 0.157851 0.164143 0.149238 0.144749
1.18.1 / BOOLEAN-BIGINT 0.157443 0.170163 0.139337 0.143281

Native/stock elapsed spans 0.987-1.092; native/previous spans 1.072-1.213. All slower results remain; prepared-kernel qualification does not establish universal complete-job acceleration. Six of eight complete jobs lose to paired stock; all eight lose to the independently measured actual previous fallback.

All 740 focused JVM invocations pass without skips across both released Flink lines and JDK 11/17, targeting Java release 11. Complete 5,003-row maps/schema, both cast modes, NULLs/logical producers, filters, CASE, character consumers, all-filtered batches, selected producer failures and adjacent function regressions pass. Admitted pure paths require consumed NativeCalc work and empty evaluators. Sixteen actual previous positive cases compare maps/schema then fail only the expected native-route assertion; eight control methods stop earlier at their positive filter assertion, so their later negative checks are not claimed as previous result comparisons. The unchanged released CastRules oracle passes eight original BOOLEAN-to-integer assertions for each release/JDK pair. No Rust code changes; the prior 904-test accepted-runtime result retains its historical provenance. Other type/producer coverage remains in #492.

Reproduce with release integral_expression, BOOLEAN input, the matching nullable/binary/ batch arguments and SF_INTEGRAL_EXPRESSION=TRY_CAST(v AS INT). Previous uses the frozen encoder, IntegralExpressionBenchmarkFunctions, sf.integral.previous=true, sf.integral.fallbackControl=true and the exact fallback reason. Complete jobs use IntegralExpressionPipelineBenchmark with sf.integral.booleanInput=true and the same expression/previous settings; all resources and data otherwise match.

Integer-to-BOOLEAN TRY_CAST

TRY_CAST of verified TINYINT/SMALLINT/INT/BIGINT inputs and failure-free arithmetic, widening/rounding producers uses the existing native comparison with a zero of the input's resolved width. Nonzero values, including signed minima, become TRUE; zero is FALSE and NULL stays NULL in both modes. This mirrors Flink's NumericToBoolean rule. Comet also treats integer-to-BOOLEAN as a DataFusion-compatible conversion; using the existing comparison avoids a new cast target/opcode, JNI entry point or Rust kernel. Ordinary CAST already lowers to this comparison through released planning. Failing, stateful or unverified producers keep their previous route. At this qualification revision, character formatting of the new boolean TRY_CAST result remained generated; the later character-consumer qualification extends that producer proof.

The record retains 48 cells and 4,800 raw trials, including warmups/checksums/ranges. Released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc artifacts are used. Equal prepared GenericRowData/BinaryRowData and Arrow include all four signed integer source widths, nullable/NOT NULL, extrema and zero. Computed controls use v+CAST(1 AS source). Each trial consumes 131,072 rows, with 16 warmups and nine measurements. The native timer runs inside Rust after import and before export, retaining production Calc and equivalent output consumption. Stock uses prepared Java RowData. Setup and perimeter conversions are outside only the small-function gate; intrinsic costs remain. The actual previous encoder from 49f269d9 rejects each exact integer-to-BOOLEAN TRY_CAST and its control executes released generated Java without synthetic native costs. Current and previous inputs/resources match. The release runtime is unchanged and verified by hash, including mimalloc malloc aliases.

Scope Cells Paired stock / native Actual previous / native Independent stock / native
Bulk (1,024 rows) 40 2.658-20.095x 2.656-6.831x 2.645-6.790x
Tiny (16 rows) 8 0.259-0.272x 0.257-0.277x 0.255-0.274x

The largest paired ratio comes from the Flink 1.18.1 NOT NULL GenericRowData INT cell: paired stock is 12.271 ns/row (3.524-17.732), versus independent stock 2.088 ns/row. A diagnostic replay with 64 warmups/nine measurements gives stock/native 3.524x and previous/native 3.352x. This retains all 292 replay samples separately and does not overwrite the primary data or prove the cause of the stock drift. The original 20x ratio is not a stable speedup claim.

All bulk cells beat stock and actual previous fallback. Tiny controls retain 8 stock losses and 8 previous losses. Small batches are not called accelerated when they lose.

Eight complete million-row jobs cover direct/computed INT/BIGINT inputs. They retain the bounded row source, blackhole output, startup, JNI and both transposes. Two warmups/five measurements alternate order. Candidate requires consumed NativeCalc work and empty JVM evaluator descriptors. Actual previous controls require exact fallback, zero substitutions and zero native operators. Seconds, median:

Flink / source / shape Paired stock Native Independent stock Actual previous
2.2.1 / INT-BOOLEAN 0.162491 0.166682 0.147023 0.148354
2.2.1 / INT-BOOLEAN-computed 0.155750 0.167884 0.149599 0.156696
2.2.1 / BIGINT-BOOLEAN 0.158948 0.167414 0.155004 0.155063
2.2.1 / BIGINT-BOOLEAN-computed 0.161442 0.179230 0.147702 0.153635
1.18.1 / INT-BOOLEAN 0.167345 0.173842 0.144691 0.145597
1.18.1 / INT-BOOLEAN-computed 0.163659 0.167112 0.141206 0.142495
1.18.1 / BIGINT-BOOLEAN 0.169389 0.180325 0.161704 0.161217
1.18.1 / BIGINT-BOOLEAN-computed 0.169641 0.184478 0.152336 0.159917

Native/stock elapsed spans 1.021-1.110; native/previous spans 1.071-1.194. 8 of eight complete jobs lose to paired stock; 8 lose to independently measured actual previous fallback. All slower results remain. Prepared-function gains do not establish universal complete-job acceleration.

All 528 focused JVM invocations pass without skips across both released Flink lines and JDK 11/17, targeting Java release 11. The new complete 5,003-row maps/schema cover four integer widths, modern/legacy modes, NULLs, signed extrema and zero, arithmetic, widening, rounding, logic, CASE and empty filtered batches. Real filters and failing/lazy producer controls pass; at this qualification revision, character consumers explicitly retain their generated route. The following qualification removes those evaluators for the separately verified character-consumer subset. Related integer/BOOLEAN TRY_CAST, temporal parsing and BOOLEAN CASE regressions also pass. Admitted pure claims require consumed NativeCalc work and empty evaluators. Sixty-four actual previous positive cases compare full maps/schema before the expected fallback assertion; their empty/character follow-ups are not reached. Eight control methods stop at the positive filter route assertion before their later negative checks. Four unchanged released CastRules oracle runs pass 12 original integer-to-BOOLEAN assertions each. No Rust production code changes; the previously qualified accepted runtime's 904-test result retains its historical provenance. Other producers, types and unverified character consumers remain in #492.

Reproduce with release integral_expression, matching integer type/nullability/row-layout/ batch arguments and SF_INTEGRAL_EXPRESSION=TRY_CAST(v AS BOOLEAN). Previous uses its frozen encoder, IntegralExpressionBenchmarkFunctions, sf.integral.previous=true, sf.integral.fallbackControl=true and the exact fallback reason. Complete jobs use IntegralExpressionPipelineBenchmark, sf.integral.bigintInput=true for BIGINT, the same expression and previous settings, preserving resources/data and perimeter costs.

Character consumers of integer truth

CAST/TRY_CAST of the verified integer-to-BOOLEAN TRY_CAST result to STRING/VARCHAR/CHAR reuse the existing BOOLEAN formatter. The same total-integer producer proof governs the inner nonzero comparison and its outer character/logical consumers. Integer-target casts retain their separate preceding proof. This follows Flink's NumericToBoolean and BooleanToString rules, including mode-dependent spelling, NULL, CHAR padding and VARCHAR truncation. Comet's compatible numeric/boolean conversion paths are consulted; no Rust kernel, opcode, JNI entry point, setting or dependency changes. Division, narrowing and unverified/failing/stateful producers retain their preceding routes. Ordinary integer-to-BOOLEAN CAST already has a released native comparison rewrite.

The record retains 96 prepared cells and 9,600 raw samples, including warmups, checksums and variability. Released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc artifacts are used. Equal GenericRowData/BinaryRowData and Arrow contain four signed integer widths, nullable/NOT NULL, extrema and zero. Short STRING and padded CHAR(8) forms use direct or addition producers. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. Rust times production Calc and equivalent output consumption after import/before export; stock uses equal prepared RowData. Setup and perimeter conversions are excluded only from this small-function timer; intrinsic costs remain. The actual preceding encoder 45cdf27a runs on the exact same unchanged DSO/executable and fixture. It retains its registered evaluators and any production shortcuts. A retained descriptor alone does not prove Java executes on every row. Artifact hashes and mimalloc aliases match.

Scope Cells Paired stock / native Actual previous / native Independent stock / native
Bulk (1,024 rows) 80 1.301-10.373x 9.137-18.033x 1.399-7.896x
Tiny (16 rows) 16 0.113-0.962x 13.991-15.825x 0.112-0.990x

All bulk cells beat stock and the actual preceding behavior. Tiny controls retain 16 stock losses and 0 previous losses. All independent samples remain; the function gate does not establish universal job gains.

Sixteen complete million-row INT/BIGINT jobs cover direct/computed short/padded forms. They retain bounded row input, blackhole output, startup, JNI and both transposes. Two warmups/five measurements alternate order. New controls require consumed NativeCalc work and empty evaluators; actual preceding controls require retained evaluators and consumed work. Seconds, median:

Flink / source / shape Paired stock Native Independent stock Actual previous
2.2.1 / INT-short 0.188565 0.190101 0.190131 0.222412
2.2.1 / INT-padded 0.219376 0.191142 0.231627 0.248017
2.2.1 / INT-short-computed 0.186346 0.195479 0.191713 0.223141
2.2.1 / INT-padded-computed 0.220819 0.192531 0.224752 0.242662
2.2.1 / BIGINT-short 0.191168 0.197340 0.190325 0.220678
2.2.1 / BIGINT-padded 0.236820 0.198431 0.244500 0.255661
2.2.1 / BIGINT-short-computed 0.203478 0.202725 0.190485 0.226375
2.2.1 / BIGINT-padded-computed 0.237860 0.198294 0.233832 0.250998
1.18.1 / INT-short 0.192091 0.203404 0.190491 0.216337
1.18.1 / INT-padded 0.216036 0.193191 0.219586 0.238627
1.18.1 / INT-short-computed 0.190646 0.194826 0.186903 0.218633
1.18.1 / INT-padded-computed 0.228368 0.196992 0.220991 0.245635
1.18.1 / BIGINT-short 0.201844 0.210436 0.202028 0.239229
1.18.1 / BIGINT-padded 0.242544 0.208591 0.240534 0.258406
1.18.1 / BIGINT-short-computed 0.206283 0.214572 0.203259 0.234902
1.18.1 / BIGINT-padded-computed 0.229412 0.210455 0.233978 0.255656

Native/stock elapsed spans 0.834-1.059, with 7 stock losses; native/previous spans 0.771-0.940, with 0 previous losses. Every slower control remains; prepared-function gains are distinct from complete-job gains.

All 384 focused JVM invocations pass without skips across both released Flink lines and JDK 11/17, targeting Java release 11. Complete 5,003-row maps/schema cover all integer widths, signed extrema/zero, nullable/NOT NULL, modern/legacy modes, both outer cast kinds, STRING/VARCHAR(1/3)/CHAR(4/8), arithmetic/logical producers, explicitly retained nested-integer fallback and all-filtered batches. Real filters, CONCAT/length, selected failures and untaken branches also pass, with explicit retained-route assertions. Related boolean CASE, integer casts and the original BOOLEAN formatter regressions pass. Pure claims require consumed native work and empty evaluator descriptors.

The actual previous 64 type/mode/nullability cases compare the first CAST query's full maps/schema, then fail only the new evaluator-free expectation. Their empty query and later outer TRY_CAST loop are not reached; candidate tests verify those separately. All sixteen original compound/failing/empty controls across both outer operators pass. Four freshly executed unchanged released CastRules oracle runs pass eight original BOOLEAN-to-character assertions each, separately from native work evidence. No Rust production changes; the accepted runtime's prior 904-test result remains historical. Other type/producer/consumer gaps stay in #492.

Reproduce with release integral_expression, matching signed input/nullability/row-layout/ batch arguments and SF_INTEGRAL_EXPRESSION=CAST(TRY_CAST(v AS BOOLEAN) AS STRING) or TRY_CAST(TRY_CAST(v AS BOOLEAN) AS CHAR(8)). Previous supplies the frozen encoder and SF_INTEGRAL_PREVIOUS=true on the same release runtime. Complete jobs use IntegralExpressionPipelineBenchmark, the corresponding expression, bigint input flag and sf.integral.previous=true for the preceding control; other resources/data match.

DECIMAL-to-BOOLEAN TRY_CAST

Direct DECIMAL inputs/literals use the existing native <> comparison against a zero with identical precision/scale. Every nonzero coefficient is TRUE, zero is FALSE and NULL remains NULL in both cast modes. This follows Flink's NumericToBoolean rule and Comet's same-type Decimal128 zero comparison. Ordinary CAST already rewrites to a comparison when precision exceeds scale; released ordinary DECIMAL(p,p) CAST fails during planning and remains unchanged. Computed/metadata-sensitive/failing/stateful producers retain their previous routes and character formatting retains its separate proof. No new Rust production kernel, setting, dependency or JNI contract is added.

The record retains all 50 prepared configurations and 5,000 raw samples, including warmups, checksums and variability. Released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread, and release/mimalloc are used. Five compact/noncompact input types are DECIMAL(4,2), (18,2), (19,19), (38,18) and (38,38), each nullable/NOT NULL and GenericRowData/BinaryRowData. Matching Arrow and Java fixtures include signed extrema, both signs and periodic zeros. The explicit SF_INTEGRAL_CORPUS=decimal-truth corpus adds the same zeros to both fixtures; the earlier nonzero-only pilot is retained separately and is not the final qualification.

Each trial consumes 131,072 rows with 16 warmups/nine measurements. The Rust timer encloses production Calc and equivalent consumption after import/before export; Flink starts on equal prepared RowData. Setup/perimeter costs alone are outside this function timer; intrinsic costs remain. Actual preceding encoder 907c078c rejects TRY_CAST and its real Java fallback is measured, not an invented callback. Both use the exact same unchanged accepted DSO and final fixtures. Only the benchmark executable changes to add the explicit zero corpus. Artifact/source hashes and matching mimalloc aliases remain.

Scope Cells Paired stock / native Actual previous / native Independent stock / native
Bulk (1,024 rows) 40 4.245-52.674x 4.250-45.661x 4.222-45.962x
Tiny (16 rows) 10 0.341-1.487x 0.406-1.842x 0.390-1.810x

All bulk cells beat stock and actual previous behavior. Tiny cells retain 5 stock losses and 5 previous losses. Independent stock ranges remain alongside paired controls.

Ten complete million-row configurations cover all five input types on both released Flink lines. They retain bounded row input, blackhole output, startup, JNI and both transposes, with two warmups/five measurements in alternating order. Candidate controls require consumed NativeCalc work and empty evaluators; preceding controls require their actual fallback and zero native substitutions. Seconds, median:

Flink / source / target Paired stock Native Independent stock Actual previous
2.2.1 / DECIMAL(4,2)-BOOLEAN 0.169193 0.184344 0.162222 0.157377
2.2.1 / DECIMAL(18,2)-BOOLEAN 0.161567 0.177438 0.151877 0.156473
2.2.1 / DECIMAL(19,19)-BOOLEAN 0.160679 0.194553 0.151313 0.161298
2.2.1 / DECIMAL(38,18)-BOOLEAN 0.169855 0.193467 0.148120 0.160680
2.2.1 / DECIMAL(38,38)-BOOLEAN 0.157564 0.192809 0.159520 0.148599
1.18.1 / DECIMAL(4,2)-BOOLEAN 0.173802 0.178297 0.154189 0.154342
1.18.1 / DECIMAL(18,2)-BOOLEAN 0.159733 0.180253 0.149595 0.154122
1.18.1 / DECIMAL(19,19)-BOOLEAN 0.167480 0.186420 0.146684 0.153933
1.18.1 / DECIMAL(38,18)-BOOLEAN 0.157908 0.190552 0.156317 0.147475
1.18.1 / DECIMAL(38,38)-BOOLEAN 0.163782 0.187075 0.154887 0.149671

Native/stock elapsed spans 1.026-1.224, retaining 10 stock losses; native/previous spans 1.134-1.298, retaining 10 previous losses. Every unfavorable result remains; function qualification is distinct from full-job gains.

All 456 focused JVM invocations pass without skips on released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Full 5,003-row maps/schema cover eleven precision/scale pairs from DECIMAL(1,0)/(1,1) to (38,38), nullable/NOT NULL, both cast modes, zero, signed extrema, logical/NULL/CASE consumers and all-filtered batches. At this qualification revision, the character control retains its generated evaluator; the following qualification extends that proof. Filters, computed/failing producers and lazy branches also pass. Admitted paths require actual consumed native work and empty evaluators. The four ordinary DECIMAL(p,p) planning failures are separately verified in both modes. Adjacent integer/BOOLEAN casts and DECIMAL projection/default regressions pass.

Actual-before runs compare 44 positive invocations' first NOT NULL/modern normal maps/schema before the expected old-fallback assertion; later modes/nullability/empty queries are not claimed as reached. Eight before character-consumer controls also compare maps/schema before their fallback assertion, with later filters/negative checks unreached. All sixteen ordinary CAST planning-failure invocations pass, each covering both modes. Four fresh unchanged released CastRules oracles pass two original DECIMAL-to-BOOLEAN assertions each. A fresh release core run passes 904 Rust tests, no ignores; only the benchmark fixture changes in Rust, and the production DSO hash stays identical.

Reproduce with release integral_expression, matching DECIMAL input/nullability/row-layout/ batch arguments, SF_INTEGRAL_CORPUS=decimal-truth and SF_INTEGRAL_EXPRESSION=TRY_CAST(v AS BOOLEAN). Actual previous uses its frozen encoder with the same final Java fixture and explicit sf.integral.fallbackControl=true / sf.integral.fallbackReason=unsupported TRY_CAST DECIMAL→BOOLEAN control. Complete jobs use IntegralExpressionPipelineBenchmark and matching sf.integral.decimalPrecision/decimalScale; the preceding control must keep its fallback. Remaining producers, consumers and source/ target families stay in #492.

Character consumers of decimal truth

CAST/TRY_CAST of verified direct DECIMAL-to-BOOLEAN TRY_CAST to STRING/VARCHAR/CHAR reuse the existing native BOOLEAN formatter. The producer proof permits verified logical and NULL wrappers around that conversion. Same-precision/scale decimal zero comparison, modern/legacy spelling, NULLs, VARCHAR truncation and CHAR padding remain unchanged. The structure follows Flink's NumericToBoolean/BooleanToString rules and Comet's same-type zero comparison. No native kernel, setting, dependency or JNI changes. Computed/failing/stateful/metadata-sensitive producers retain their preceding routes. Integer-target and bounded-position condition proofs remain separate. At this qualification revision ordinary DECIMAL CAST uses a different comparison tree and retains its character evaluator; the following qualification extends that direct-comparison proof. Ordinary DECIMAL(p,p) CAST retains the released planning failure.

The record retains 100 prepared configurations and 10,000 raw samples, including warmups, checksums and variability. Released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc are used. Equal GenericRowData/BinaryRowData and Arrow contain DECIMAL(4,2), (18,2), (19,19), (38,18) and (38,38), nullable/NOT NULL, signed extrema, zero and both signs. SF_INTEGRAL_CORPUS=decimal-truth selects the same fixture. Shapes are outer CAST to STRING and outer TRY_CAST to CHAR(8), each over the direct inner TRY_CAST.

Each trial consumes 131,072 rows, with 16 warmups/nine measurements. Rust times production Calc and equivalent consumption after import/before export; Flink starts on equal prepared RowData. Setup and perimeter costs alone are excluded from this function timer; intrinsic costs remain. Actual preceding encoder 8b42f7c1 retains its character evaluator and any production shortcuts on the exact same unchanged DSO/executable/fixture. A retained descriptor alone is not proof of a JVM callback on every row. Hashes/mimalloc aliases match.

Scope Cells Paired stock / native Actual previous / native Independent stock / native
Bulk (1,024 rows) 80 2.109-13.089x 11.307-18.114x 1.854-11.668x
Tiny (16 rows) 20 0.211-1.308x 13.644-15.236x 0.204-1.244x

All bulk cells beat stock and actual previous behavior. Tiny cells retain 14 stock losses and 0 previous losses. All independent measurements and unfavorable controls remain.

Twenty complete million-row configurations cover five DECIMAL types and both shapes on both released Flink lines. They retain bounded row input, blackhole output, startup, JNI and both transposes. Two warmups/five measurements alternate order. Candidate controls require consumed NativeCalc work and empty evaluators; previous controls require consumed work and their retained evaluators. Seconds, median:

Flink / source / shape Paired stock Native Independent stock Actual previous
2.2.1 / DECIMAL(4,2)-short 0.207693 0.205985 0.211752 0.238406
2.2.1 / DECIMAL(4,2)-padded 0.250289 0.213052 0.246135 0.267123
2.2.1 / DECIMAL(18,2)-short 0.210433 0.206001 0.201563 0.232634
2.2.1 / DECIMAL(18,2)-padded 0.240361 0.207615 0.244654 0.276751
2.2.1 / DECIMAL(19,19)-short 0.206595 0.201382 0.208084 0.237834
2.2.1 / DECIMAL(19,19)-padded 0.244125 0.222056 0.254456 0.282303
2.2.1 / DECIMAL(38,18)-short 0.208986 0.211712 0.208985 0.241863
2.2.1 / DECIMAL(38,18)-padded 0.242238 0.211111 0.249556 0.281209
2.2.1 / DECIMAL(38,38)-short 0.215642 0.211295 0.207752 0.243736
2.2.1 / DECIMAL(38,38)-padded 0.247046 0.217456 0.250570 0.282227
1.18.1 / DECIMAL(4,2)-short 0.207685 0.209153 0.210604 0.236102
1.18.1 / DECIMAL(4,2)-padded 0.250418 0.211956 0.243160 0.262985
1.18.1 / DECIMAL(18,2)-short 0.210772 0.206240 0.220348 0.247438
1.18.1 / DECIMAL(18,2)-padded 0.245597 0.208913 0.243005 0.266683
1.18.1 / DECIMAL(19,19)-short 0.207183 0.228833 0.210799 0.240593
1.18.1 / DECIMAL(19,19)-padded 0.249136 0.216391 0.240292 0.270399
1.18.1 / DECIMAL(38,18)-short 0.205375 0.216000 0.215327 0.232637
1.18.1 / DECIMAL(38,18)-padded 0.246364 0.232057 0.240270 0.273388
1.18.1 / DECIMAL(38,38)-short 0.218105 0.214802 0.216118 0.249770
1.18.1 / DECIMAL(38,38)-padded 0.246352 0.215345 0.241897 0.273894

Native/stock elapsed spans 0.846-1.104, retaining 4 stock losses; native/previous spans 0.750-0.951, retaining 0 previous losses. Prepared function gains are distinct from complete-job gains; no slower control is removed.

All 304 focused JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Complete 5,003-row maps/schema cover eleven precision/scale pairs from DECIMAL(1,0)/(1,1) to (38,38), both outer cast kinds, nullable/ NOT NULL, modern/legacy modes, STRING/VARCHAR(1/3)/CHAR(4/8), signed extrema/zero, logical/ NULL wrappers and empty filtered batches. Filter/length/CONCAT consumers, selected errors, untaken branches and explicitly retained integer targets pass. Related direct decimal, integer truth/character and BOOLEAN/integer regressions pass. Admitted claims require consumed NativeCalc work and empty evaluator descriptors.

Actual-before runs compare 44 first NOT NULL/modern outer-CAST queries' complete maps/ schema before the evaluator-free assertion; their subsequent modes/nullability/empty and outer TRY_CAST queries are not claimed as reached. All sixteen preceding compound/failing/ empty/integer-target controls pass. Four fresh unchanged released CastRules oracles pass eight original BOOLEAN-to-character assertions each. No Rust source/artifact changes; the immediately preceding qualification's fresh 904-test Rust result remains historical.

Reproduce with release integral_expression, matching DECIMAL input/nullability/row-layout/ batch arguments, SF_INTEGRAL_CORPUS=decimal-truth and SF_INTEGRAL_EXPRESSION=CAST(TRY_CAST(v AS BOOLEAN) AS STRING) or TRY_CAST(TRY_CAST(v AS BOOLEAN) AS CHAR(8)). Actual previous supplies the frozen encoder and SF_INTEGRAL_PREVIOUS=true on the same runtime/fixture. Complete jobs use IntegralExpressionPipelineBenchmark with matching sf.integral.decimalPrecision/ decimalScale and sf.integral.previous=true for previous. Other resources/data match. Remaining type/producer/consumer gaps stay in #492.

Character consumers of direct decimal predicates

CAST/TRY_CAST to STRING/VARCHAR/CHAR reuse the native BOOLEAN formatter for direct same-precision/scale DECIMAL comparisons, direct DECIMAL NULL checks and verified logical/ NULL wrappers. The NULL proof also covers released nullable-comparison rewrites. The ordinary DECIMAL-to-BOOLEAN CAST rewrite consequently qualifies when precision exceeds scale; ordinary DECIMAL(p,p) retains released planning failure. Computed/failing/stateful/ mixed-type operands, integer-target TRY_CAST and bounded-position proofs stay separate. No Rust kernel, setting, dependency or JNI/transport change is introduced by this extension.

The record retains 60 prepared configurations and 6,000 raw samples, including warmups, checksums and variability. Released Flink 1.18.1/2.2.1, Oracle JDK 17.0.12, Apple M4 Pro, one thread and release/mimalloc are used. Equal GenericRowData/BinaryRowData and Arrow contain DECIMAL(4,2), (18,2), and (38,18), nullable/NOT NULL, signed extrema, periodic zero and both signs. Shapes are CAST(CAST(v AS BOOLEAN) AS STRING) and TRY_CAST(CAST(v AS BOOLEAN) AS CHAR(8)); the inner ordinary CAST produces a verified same-type zero comparison in released planning.

Each trial consumes 131,072 rows, with 16 warmups/nine measurements. Rust times production Calc and equivalent consumption after import/before export; stock starts on equal prepared RowData. Setup/perimeter conversion alone is excluded; intrinsic costs remain timed. Actual preceding encoder 6305c6f9 retains its character evaluator and any production shortcuts on the exact same newly rebuilt DSO/executable/fixture. A descriptor alone is not proof of a Java callback on every row. Allocator aliases and artifact hashes match.

Scope Cells Paired stock / native Actual previous / native Independent stock / native
Bulk (1,024 rows) 48 1.834-15.438x 10.918-33.003x 1.604-12.815x
Tiny (16 rows) 12 0.169-1.127x 13.296-14.487x 0.167-1.185x

All bulk cells beat paired stock and actual previous behavior. Tiny cells retain 10 paired-stock losses and 0 previous losses; all independent measurements and unfavorable controls remain.

Eight complete million-row configurations cover compact DECIMAL(18,2) and noncompact DECIMAL(38,18), both shapes and both released lines. Bounded row input, blackhole output, startup, JNI and both transposes remain. Two warmups/five measurements alternate order. Candidate needs consumed NativeCalc work and empty evaluators; preceding controls retain consumed work and their character evaluator. Seconds, median:

Flink / input / shape Paired stock Native Independent stock Actual previous
2.2.1 / DECIMAL(18,2)-short 0.200580 0.192215 0.206523 0.233667
2.2.1 / DECIMAL(18,2)-padded 0.246108 0.208744 0.243367 0.260601
2.2.1 / DECIMAL(38,18)-short 0.201272 0.208937 0.205316 0.237216
2.2.1 / DECIMAL(38,18)-padded 0.246396 0.215287 0.245370 0.277956
1.18.1 / DECIMAL(18,2)-short 0.205336 0.203482 0.210707 0.244583
1.18.1 / DECIMAL(18,2)-padded 0.242385 0.207672 0.233590 0.251610
1.18.1 / DECIMAL(38,18)-short 0.203797 0.222972 0.194975 0.237246
1.18.1 / DECIMAL(38,18)-padded 0.238009 0.210415 0.246277 0.265138

Native/stock elapsed spans 0.848-1.094, retaining 2 stock losses; native/previous spans 0.775-0.940, retaining 0 previous losses. Prepared function gains are distinct from complete-job gains; no slower control is removed.

All 260 focused JVM invocations pass without skips on released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java release 11. Complete 5,003-row maps/schema cover eleven precision/ scale pairs through (38,38), all six comparisons, independently varying DECIMAL columns and NULL positions, NULL/logical wrappers, both outer cast kinds, STRING/VARCHAR(1/3)/ CHAR(4/8), modern/legacy modes and empty filtered batches. Computed operands retain their generated controls; integer targets retain fallback. Filters/length, selected division failures and untaken branches pass. Ordinary DECIMAL(p,p) planning failures remain covered. Pure claims require consumed NativeCalc work and empty evaluator descriptors.

Actual-before runs compare all 44 first NOT NULL/modern outer-CAST queries' maps/schema before failing the new evaluator-free assertion. Later modes/nullability/empty/outer-TRY_CAST queries are not claimed as reached by those invocations. All eight retained-route controls pass. Four fresh unchanged released DecimalTypeTest oracles each execute three original comparison methods with their setup/teardown and original assertions. These are stock oracles; the independent runtime-source SQL witnesses establish native parity and execution.

After rebasing onto merged main 2bb52ca2, fresh full runtime Maven compilations for both lines, 904 release Rust core tests with no ignores and 304 preceding-character JVM controls pass on the rebuilt artifacts. A separate 260 invocations separately qualify this comparison extension. Another 80 targeted metadata regressions pass across the four release/JDK pairs for NOT NULL arithmetic, narrowing CAST overflow and noncompact SIGN character/default consumers. Eight fresh compact/noncompact direct-TRY_CAST character prepared replays also beat stock and the actual latest-main encoder on the rebuilt runtime; their raw samples are retained in the linked record. Earlier character measurements retain their original revision/DSO provenance; none is relabeled as a fresh measurement of this rebuilt runtime. The 800 fresh replay samples are separate prepared controls; they do not relabel the earlier twenty complete-job measurements.

Reproduce using release integral_expression, matching DECIMAL type/nullability/row layout/ batch, SF_INTEGRAL_CORPUS=decimal-truth and SF_INTEGRAL_EXPRESSION set to either shape. Actual previous supplies the frozen encoder with SF_INTEGRAL_PREVIOUS=true on the same runtime/fixture. Complete jobs use IntegralExpressionPipelineBenchmark with the matching expression/precision/scale and sf.integral.previous=true for previous controls. Remaining type/producer/consumer gaps stay in #492.

Runtime integer epoch precision

Ordinary Calc admits verified INT CASE precision trees with literal 0, 3 or NULL results for integer TO_TIMESTAMP_LTZ. The native kernel reads conditions/branches directly, checks original seconds/milliseconds before scaling, and writes one timestamp column. Literal millisecond-or-NULL branches share immutable input values and alter parent validity only. Nested branches reuse the total integer selector. Computed integer children retain their resolved widths before widening; unproven/failing/stateful inputs and UNNEST keep their routes.

The record retains all 80 final configurations and 19,800 raw samples across all development stages and the final matrix. Apple M4 Pro, macOS arm64, JDK 17.0.12, one thread, release/mimalloc, Flink 1.18.1/2.2.1. Matched prepared GenericRowData/BinaryRowData and Arrow cover all four integer widths, nullable/NOT NULL inputs, extrema and epoch bounds. Each trial consumes 131,072 rows; 16 warmups/nine measurements alternate engines. Native timing starts inside Rust after import and stops before export, retaining production Calc/consumption and intrinsic costs. Stock starts on equivalent prepared RowData. Actual preceding encoder a90183ec retains its evaluator on the same current native runtime/fixture, with independent stock controls.

The two measured shapes consume YEAR over seconds/milliseconds selected by the sign of v, or IS NULL over millisecond/NULL selection. SQL strings, all medians/ranges and artifacts are retained in the record. The timings do not qualify every nested composition.

Scope Cells Stock / native Actual previous / native Independent stock / native
Bulk, 1,024 rows 64 1.145-4.399x 6.227-38.014x 1.372-5.028x
Tiny, 16 rows 16 0.042-0.197x 10.826-17.692x 0.044-0.161x

Retain 0 paired-stock bulk medians below 1x, 0 independent-stock bulk medians below 1x and 16 tiny paired-stock losses. These are measured diagnostics; native execution evidence alone does not establish a speedup.

The first timestamp-CASE implementation loses despite correctness. CPU samples identify CASE batch partitioning and Struct timestamp merging, including child/null-buffer copies. A first fused kernel removes that merging but still materializes an INT precision column; another sample identifies selector/materialization work. Direct branch selection removes that intermediate, and millisecond-or-NULL buffer sharing removes the remaining value copy. The shared-buffer version still loses bulk cases. Packed Arrow condition/validity masks replace per-row selection for scalar precision leaves, retain first-true CASE selection and scale only selected in-range seconds. Final masks exclude NULL predicates. All development stages, actual source/artifact hashes and slower samples are retained rather than replaced by the final matrix. Inclusive sample counts overlap; they are not wall-time shares.

Eight complete million-row configurations retain the bounded row source, blackhole sink, startup, JNI and both transposes. Two warmups/five measurements alternate stock/native; candidate requires consumed NativeCalc work and empty evaluators, previous keeps its evaluator. Median seconds:

Flink / input / shape Paired stock Native Independent stock Actual previous
2.2.1 / INT-seconds-millis 0.176752 0.177871 0.172852 0.212247
2.2.1 / INT-millis-null 0.177497 0.188569 0.183063 0.234874
2.2.1 / BIGINT-seconds-millis 0.169195 0.183573 0.162459 0.218152
2.2.1 / BIGINT-millis-null 0.181030 0.190784 0.182086 0.236146
1.18.1 / INT-seconds-millis 0.166785 0.178363 0.170580 0.206144
1.18.1 / INT-millis-null 0.178303 0.187838 0.175389 0.233447
1.18.1 / BIGINT-seconds-millis 0.159494 0.179954 0.165135 0.219383
1.18.1 / BIGINT-millis-null 0.179089 0.191463 0.177121 0.223125

Native/stock elapsed ranges 1.006-1.128, retaining 8 stock losses; native/previous ranges 0.803-0.865, retaining 0 previous losses. No universal complete-job gain is claimed.

All 432 final JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, Java release 11: 36 new precision invocations, 136 preceding epoch/rounding controls and 260 preceding decimal/integer-character controls. Complete 5,003-row maps/schema cover six precision trees, nested/NULL conditions, all integer widths, resolved overflow, original epoch bounds, UTC/New York and empty filtered batches. New pure routes require consumed native work, no fallback and empty evaluator descriptors. Fallible/unknown precision and epoch producers retain explicit generated controls. Recorded preceding-encoder controls compare all 32 first-query maps/schema before the expected new route assertion fails; their subsequent empty-query comparisons are not reached. Four retained-route methods pass six controls each.

All 910 release Rust core tests pass without ignores. New tests cover original-unit bounds, pre-scaling overflow, independently sliced validity, scalar/array combinations, first-true/NULL conditions, independently sliced conditions across 64-bit boundaries, empty arrays, invalid internal signatures and immutable buffer ownership. Sixteen unchanged released TemporalTypesTest method invocations pass separately; these stock oracles do not replace the independent native SQL/evidence witnesses.

Reproduce with release integral_expression, SF_INTEGRAL_CORPUS=epoch-millis and the exact expression/type/NULL/layout/batch from the record. Actual previous supplies the frozen encoder and SF_INTEGRAL_PREVIOUS=true on the same final native runtime. Complete jobs use IntegralExpressionPipelineBenchmark with matching expression and INT/BIGINT input settings. Remaining signatures and timezone/parser work stay in #503.

Boolean-object and mixed integer PRINTF

On released Flink 2.2.1, verified total BOOLEAN producers and mixed signed-integer/BOOLEAN arguments use the evaluator-free literal ASCII formatter. BOOLEAN %b observes true/false; NULL is false. BOOLEAN %s uses true/false words and null for NULL. A non-NULL integer, including zero, still formats as true with %b. Argument indices/reuse, uppercasing, width, precision, NULLs and invalid-format routes retain Java Formatter semantics. The running JDK validates each actual argument type at planning time. Explicit index zero stays on the preceding route because JDK 11 and 17 disagree. Bounds remain 512 ASCII format characters and width/precision at most 256. Other types/formats, unverified/failing/stateful producers and representation-sensitive consumers keep their previous routes. Flink 1.18 has no PRINTF and retains released SQL rejection.

The record includes all final cells, initial slower versions, source/artifact hashes and 16,000 raw trials. Apple M4 Pro, macOS arm64, one thread, Oracle JDK 17.0.12, released Flink 2.2.1 and release/mimalloc-linked DSO/executable with verified malloc aliases. Equal prepared GenericRowData/BinaryRowData and Arrow cover true/false, nullable/NOT NULL and mixed INT/BIGINT signed extrema/NULLs. Each trial consumes 131,072 rows, with 16 warmups and nine measurements. Native timing starts/stops inside Rust after import and before export, retaining production Calc, output consumption and intrinsic costs. Initial JNI/import/export and perimeter transposes are excluded only from this small-function timer. Stock starts on matching prepared RowData.

The actual preceding encoder/helper 1346ed57 runs with its saved whole DSO/executable and the identical final fixture/JDK/data. It does not receive the new certified formatter. New BOOLEAN controls retain their preceding evaluator; eight existing integer native controls require empty evaluators on both versions. A retained descriptor alone does not prove a Java callback on every batch. Independent stock measurements remain separate.

Shapes are PRINTF('%b/%1$B',v), PRINTF('%1$8.2S:%1$-8.3b',v) and PRINTF('%1$+08d:%2$8.2B:%1$x:%2$s',n,v) with INT or BIGINT n. Both row layouts and NULL modes are retained, plus 16-row controls.

Scope Cells Stock / native Actual previous / native Independent stock / native
New bulk, 1,024 rows 16 4.930-7.627x 5.632-8.099x 4.849-7.460x
New tiny, 16 rows 4 3.382-5.248x 10.339-12.905x 3.367-5.101x

All new bulk cells beat paired stock and the actual preceding implementation. Retain 0 tiny stock losses and 0 tiny previous losses. Eight retained nullable INT/BIGINT %d/%+08d:%1$x native controls have previous/current median ratios 1.285-1.440x, with all ranges and checksums retained. Eight additional unchanged certified DECIMAL(4,2)/DECIMAL(38,18) short/padded controls retain their evaluators on both whole runtimes; previous/current spans 1.103-1.214x. Their matching output digests and allocation probes are retained separately. The new pure scope does not include these DECIMAL arguments.

The first shared-loop implementation passes correctness but adds a stable BIGINT-simple slowdown in the recorded order (previous/current 0.931x with nonoverlapping ranges). CPU samples locate format parsing and integer conversion/allocation; inclusive counts overlap and do not attribute wall time to a source branch. Separately compiling Boolean-capable and integer-only loops recovers that simple-format probe, but padded BIGINT retains lower preceding medians in both orders. The final implementation also reuses one conversion String across rows/conversions and writes decimal integer digits through the existing itoa buffer. Boolean/NULL words and hexadecimal/octal/character output append into this buffer. General certified DECIMAL formatting retains its owned result without an additional copy. Parsing is still rowwise. The intermediate buffered version also enables Boolean dispatch in general certified formatting; DECIMAL controls expose previous/current as low as 0.927x. The final general formatter uses its preceding non-Boolean specialization, with Boolean-object evaluation retained outside the planner-verified literal path. An ownership-only experiment also retains slower padded medians and is discarded. When every format in a certified batch is equal and non-NULL, the final evaluator borrows it once and reuses the Arrow builder/ row/conversion buffers. This removes per-row output String materialization; varying/NULL formats retain row handling. Existing complete-batch ASCII/cache checks still precede formatting, and a failed formatter still rejects the whole certificate. All slower stages and reverse-order numeric probes remain recorded. For the nullable GenericRowData BIGINT padded control, one warmed 1,024-row operation drops from 2,489 allocation requests / 80,164 requested bytes on the actual preceding whole runtime to 29 / 48,687; output buffer bytes remain 35,159. Allocation traffic is diagnostic, not peak memory or proof of a complete-job gain.

Four million-row jobs retain the bounded row source, blackhole sink, startup, JNI and both transposes. Two warmups and five measurements alternate stock/native. New paths require consumed NativeCalc work and empty evaluators; previous retains its original evaluator. Median seconds:

Shape Paired stock Native Independent stock Actual previous
truth 0.449773 0.287959 0.459998 0.460954
padded 0.709251 0.303006 0.704850 0.753436
mixed-int 1.046717 0.340149 1.021757 1.049034
mixed-bigint 1.065773 0.329840 1.059488 1.135357

Native/stock elapsed spans 0.309-0.640; native/previous spans 0.291-0.625. All trials and variability remain, including any losses; prepared gains do not establish universal complete-job acceleration.

All 112 final focused JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java 11: 24 new BOOLEAN/absence-control invocations, 20 existing integer-PRINTF/absence controls, 12 format-helper invocations, 36 preceding epoch invocations and 20 existing PRINTF/shared-certificate invocations on Flink 2.2. Complete 5,003-row maps/schema cover true/false/NULL, all signed widths, logical/comparison producers, mixed independently nullable arguments, literals, padding/index/reuse/precision, empty filtered batches and explicit invalid/dynamic/failing/lazy/protected-consumer controls. Pure-route witnesses require consumed native work, no fallback and empty evaluator descriptors. The actual preceding Flink 2.2 controls each compare ten complete first-query maps/schema before failing the intended pure-route assertions; subsequent empty-query checks in those failing invocations are not reached. Retained routes and both 1.18 absence controls pass.

All 912 release Rust core tests pass without ignores, including Boolean/NULL/signed-width formatting, independently sliced/empty arrays, rejected contracts and input ownership. Two unchanged released StringFunctionsITCase.printfTestCases stock-oracle runs each pass all 47 original assertions. These stock oracles are separate from the native SQL and execution witnesses. Earlier decimal/epoch commit timing records retain their own runtime provenance; this rebuilt DSO is not substituted into historical measurements. Hosted CI remains separate.

Reproduce with release integral_expression, the exact expression/type/secondary-type/ NULL/layout/batch from the record and the saved previous encoder/helper plus whole runtime. New preceding routes set SF_INTEGRAL_PREVIOUS=true; retained old-native integer controls leave it false. Certified DECIMAL controls set it true on both runtimes. Complete jobs use IntegralExpressionPipelineBenchmark, sf.integral.booleanInput=true, matching expression and secondary type. Remaining scope stays in #497.

Numeric Boolean producers for PRINTF

The literal primitive formatter also accepts the exact numeric BOOLEAN producer proof already used by character CAST/TRY_CAST: integer truth TRY_CAST over total integer trees, direct DECIMAL truth TRY_CAST, same-precision/scale direct DECIMAL comparisons, and their verified logical/NULL wrappers. This keeps PRINTF('%b',0) as true while PRINTF('%b',TRY_CAST(0 AS BOOLEAN)) is false. Ordinary DECIMAL-to-BOOLEAN CAST benefits from its released comparison rewrite where supported; DECIMAL(p,p) retains the released planning failure. Computed/mixed decimal predicates, failing/stateful inputs, protected consumers and integer-target/rounding-position proofs retain their preceding scopes. Only released Flink 2.2 has PRINTF; 1.18 retains its SQL rejection.

This is a Java admission extension with no native source/build change. The exact preceding encoder/helper 46a1f487 and its saved whole runtime execute the same fixture/data/JDK. Candidate native artifacts are byte-identical. Final prepared and SQL-test encoder/context bytecode matches within Flink 2.2, and before-control bytecode matches the saved sources. The record retains sources, hashes, all cells and 4,050 raw prepared trials.

Apple M4 Pro, macOS arm64, one thread, JDK 17.0.12, Flink 2.2.1 and the same verified release/mimalloc DSO/executable as the preceding BOOLEAN formatter qualification. Equal prepared GenericRowData/BinaryRowData and Arrow cover nullable/NOT NULL DECIMAL(4,2)/DECIMAL(38,18), extrema/zero, independently nullable mixed INT and signed INT/BIGINT truth controls. Each trial consumes 131,072 rows with 16 warmups and nine measurements. The production Calc/output-consumption timer starts/stops inside Rust after import and before export, retaining intrinsic allocation, conversions and callbacks. Only this prepared gate excludes initial JNI/import/export and perimeter transposes. Stock starts on matching prepared RowData. New paths require empty evaluators; actual preceding paths retain their original evaluator.

Scope Cells Stock / native Actual previous / native Independent stock / native
Bulk, 1,024 rows 32 4.646-8.479x 5.200-8.644x 4.495-8.516x
Tiny, 16 rows 8 2.603-4.874x 8.343-10.742x 2.168-4.687x

All bulk cells beat paired stock and actual preceding behavior. Retain 0 tiny stock losses and 0 tiny previous losses. The initial candidate run completed 50 paired stock/native samples with matching digests, but the local parser failed on the comma in the existing unquoted DECIMAL type field before launching its preceding control. Those samples remain a separate parser-attempt stage; they are not paired previous evidence. Normalizing that field before parsing and quoting public CSV preserves the engine outputs without an engine/runtime modification.

Four million-row jobs retain the bounded row source, blackhole sink, job submission/startup, JNI and both perimeter transposes. Environment setup and surrounding JVM process startup are outside the job timer. Two warmups and five measurements alternate stock/native. NativeCalc must consume work and the new routes require empty evaluators; preceding routes keep their original evaluators. Median seconds:

Shape Paired stock Native Independent stock Actual previous
truth-DECIMAL(4,2) 0.707504 0.258533 0.698759 0.696531
mixed-DECIMAL(4,2) 0.836147 0.324166 0.819374 0.819529
truth-DECIMAL(38,18) 0.691274 0.258629 0.661155 0.712982
mixed-DECIMAL(38,18) 0.835070 0.323816 0.816792 0.830866

Native/stock elapsed spans 0.365-0.388 (0 stock losses); native/previous spans 0.363-0.396 (0 previous losses). All variability and independent stock measurements remain. These prepared and complete-job results do not establish universal job acceleration.

All 76 fresh focused JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java 11. Complete 5,003-row maps/schema cover eight decimal precision/scale pairs, direct and comparison truth, independently nullable inputs, modern/legacy modes, string precision/padding, mixed integer arguments, all-filtered batches and retained invalid/failing/lazy/consumer controls. Signed-width cases also compare a boxed integer's %b with its explicit BOOLEAN truth conversion. Admitted paths require consumed NativeCalc work, no fallback and empty evaluator descriptors.

Actual preceding controls on each JDK compare sixteen complete first-query maps/schema before the intended new pure-route assertion fails. Subsequent queries/modes/empty checks in those failing invocations are not counted. Four other dynamic tests per JDK pass, covering existing pure BOOLEAN routes and retained generated routes. The unchanged native artifacts retain the preceding 912 release Rust core tests and two unchanged released PRINTF stock-oracle runs with 47 assertions each; these are not new Rust/oracle runs. Focused current JVM verification is separate from full hosted CI on the exact pushed head.

Reproduce with integral_expression using the record's expression/type, secondary type, NULL/layout/batch and SF_INTEGRAL_CORPUS=decimal-truth; previous controls set SF_INTEGRAL_PREVIOUS=true and use the saved actual encoder/helper and whole runtime. Complete jobs use IntegralExpressionPipelineBenchmark, matching decimal precision/scale, secondary type and expression. General STRING/DECIMAL printf arguments, remaining formats, producers and consumers remain in #497.

Length of primitive ASCII PRINTF

On 2026-11-04, direct released built-in CHAR_LENGTH/CHARACTER_LENGTH over the already-qualified primitive ASCII formatter reuse the native length kernel. ASCII has identical byte, character and UTF-16 lengths, so this consumer does not need Java String identity. The proof includes total integer/BOOLEAN and admitted exact numeric truth producers. Operator identity prevents a same-name catalog UDF from being substituted. Generic Unicode, invalid/dynamic formats, unverified producers, other consumers and relational representation guards remain unchanged. No native runtime or transport change.

The record retains source/artifact hashes, every configuration, allocation diagnostics and 3,000 raw trials. The actual preceding encoder/helper is bbada54b, replayed on its saved whole runtime; candidate DSO and prepared executable are byte-identical. Apple M4 Pro, macOS arm64, one thread, JDK 17.0.12, released Flink 2.2.1, Java --release 11, release/mimalloc build.

Stock starts on prepared GenericRowData/BinaryRowData and native starts inside Rust on equal prepared Arrow batches. The timer retains production Calc, consumption, allocation, intrinsic conversions and callbacks after input import and before output export. Only initial JNI/import/export and perimeter transposes are outside this small-function timer. Each cell uses 131,072 logical rows, 16 warmups and nine alternating measured trials. Both NULL modes and row layouts cover INT/BIGINT extrema, BOOLEAN/mixed INT, compact/wide DECIMAL truth/comparisons, indexed formats and padding. Digests match for all engines.

Scope Cells Stock / native Actual previous / native Independent stock / native
Bulk, 1,024 rows 24 2.224-6.884x 2.718-7.269x 2.159-6.921x
Tiny, 16 rows 6 1.352-4.703x 8.432-10.516x 1.406-5.079x

Four complete million-row jobs retain job submission/startup, JNI and both RowData/Arrow transposes, with bounded row sources and blackhole sinks. Each engine has two warmups and five alternating measured jobs; the surrounding JVM startup and setup remain outside the job timer. Median seconds:

Shape Paired stock Native Independent stock Actual previous
int-length 0.319771 0.210095 0.318141 0.343298
boolean-mixed 0.762079 0.277136 0.779459 0.808125
decimal-truth 0.608789 0.226033 0.658311 0.658383
decimal-comparison 0.780619 0.298393 0.755788 0.810500

Native/stock elapsed spans 0.364-0.657 and native/previous 0.343-0.612. All measured bulk, tiny and complete-job cells improve; these bounded measurements do not establish universal job acceleration. Raw trials retain variability and independent stock controls.

All 114 final focused JVM invocations pass, zero skips, across released Flink 1.18.1/2.2.1 and JDK 11/17. Full 5,003-row maps and resolved types cover all integer widths, BOOLEAN, numeric truth, decimal extremes/NULLs, aliases, modern/legacy casts, empty filtered batches, invalid/dynamic/non-ASCII formats and failing/lazy controls. An executed same-name catalog UDF keeps its implementation. Pure routes require consumed NativeCalc work, no fallback and empty evaluator descriptors. Each JDK's actual preceding control compares fourteen complete first-query maps/schema before the intended pure-route assertion fails; later queries in those failing invocations are not counted. Its retained route and catalog UDF controls pass. An initial local UDF control needed quoted SQL identifiers; final controls use the corrected test with no production-code change.

Four unchanged released ScalarFunctionsTest.testCharLength stock-oracle runs pass separately, one per release/JDK pair. The byte-identical native artifacts retain their earlier 912-core-test and PRINTF-oracle qualifications; these are not new Rust/PRINTF oracle runs. Full hosted suites must qualify the exact pushed head separately.

Reproduce with release integral_expression, the record's expression/type/secondary type, NULL/layout/batch and SF_INTEGRAL_CORPUS=decimal-truth. Previous controls set SF_INTEGRAL_PREVIOUS=true and use the saved actual preceding encoder/helper and runtime. Complete jobs use IntegralExpressionPipelineBenchmark with matching input and expression properties. Remaining consumers/representations stay in #497.

Comparisons of primitive ASCII PRINTF

On 2026-11-04, direct built-in =, <>, <, <=, >, >= comparisons use native kernels when both sides are qualified primitive ASCII formatter results or ASCII/NULL character literals, with at least one formatter. Cached Java UTF-16 and serialized UTF-8 have identical order on this proven subset. Native filters can consume the predicates. This Java-only admission proof preserves built-in identity and the complete existing formatter producer proof. Arbitrary STRING inputs, non-ASCII literals, unknown/invalid formats, other consumers and relational representation guards retain their prior paths.

The record retains every cell, source/ artifact hashes, variability, route evidence and 3,000 raw trials. Apple M4 Pro, macOS arm64, one thread, JDK 17.0.12, released Flink 2.2.1, Java release 11, release/mimalloc. The DSO and prepared executable are byte-identical to the qualified preceding runtime; the actual previous encoder/helper is 40f26521.

Prepared stock starts on equal GenericRowData/BinaryRowData and native starts inside Rust on prebuilt Arrow. Each cell has 131,072 logical rows, 16 warmups and nine alternating measurements. The Rust timer retains production Calc, consumption, allocation, intrinsic conversions and callbacks after import and before export. Only initial JNI/import/export and perimeter transposes are outside this small-function timer. Signed INT/BIGINT extrema, BOOLEAN/mixed INT, compact/wide DECIMAL truth/comparisons, both NULL modes and row layouts are covered; all engine digests match.

The previous route is verified per shape. Equality/inequality retain a native Calc with the generated evaluator on the saved whole runtime. Ordered prepared controls verify the actual previous whole-Calc stock fallback and exact representation reason, then time the released generated evaluator on prepared RowData using the existing fallback fixture. They do not substitute a hypothetical native callback for a rejected signature.

Scope Cells Stock / native Actual previous / native Independent stock / native
Bulk, 1,024 rows 24 1.420-6.704x 2.192-7.800x 1.464-7.000x
Tiny, 16 rows 6 0.774-4.163x 3.509-9.424x 0.627-4.129x

All bulk cells pass the prepared bulk-function gate. The 16-row INT equality control loses to stock; its timing and independent stock control remain in the record. The other five tiny controls improve. No universal small-batch improvement is claimed.

Four complete million-row jobs retain job submission/startup, JNI and both transposes, using bounded external Row sources and blackhole sinks. The external source proves Java-backed inputs to the preceding planner, so its ordered complete-job controls retain the actual native Calc with generated evaluation. This differs from the conservative prepared/internal-RowData fallback. Both runtimes must consume native work, while the new route additionally requires empty evaluators. Each engine has two warmups and five alternating measured jobs; surrounding JVM startup and setup are outside timing. Median seconds:

Shape Paired stock Native Independent stock Actual previous
int-order 0.218030 0.197722 0.202463 0.232047
boolean-mixed-order 0.795598 0.299646 0.748803 0.744022
decimal-truth 0.567530 0.232338 0.595066 0.585824
decimal-comparison-order 0.707602 0.300004 0.710702 0.776674

Native/stock elapsed spans 0.377-0.907, native/previous 0.386-0.852. All four measured complete-job medians improve; independent controls and variability remain, and the bounded qualification does not imply every job will improve.

All 148 final focused JVM invocations pass, zero skips, on released Flink 1.18.1/2.2.1 and JDK 11/17. Complete 5,003-row maps/schema cover all six comparisons, operand reversal, two formatter operands, all integer widths, BOOLEAN, DECIMAL truth, NULLs, padding, empty filtered batches and the retained Unicode/format/failing/lazy paths. An executed filter compares every surviving ID/value with stock. Admitted paths require consumed NativeCalc work, no fallback and empty evaluators. The previous encoder compares fourteen complete first-query projection maps/schema plus the filter result per JDK before fifteen expected route failures. Later queries/modes in those failing invocations are not counted. The unverified-path control passes with its explicit fallback/generated expectations. An initial residual test expected generated Calc where the existing guard correctly used whole-Calc fallback; only that test expectation changed.

Four unchanged released stock-oracle invocations each pass ScalarOperatorsTest.testCompareOperator and testMoreEqualAndNonEqual, separately from native qualification. The unchanged native artifacts keep their earlier 912-core-test/PRINTF-oracle qualification; those are not new Rust/PRINTF runs. Hosted suites must qualify the exact pushed head separately.

Reproduce prepared cells with release integral_expression and the record's type/expression, NULL/layout/batch/secondary type and SF_INTEGRAL_CORPUS=decimal-truth. Previous generated controls set SF_INTEGRAL_PREVIOUS=true on the saved encoder/helper/runtime. Ordered previous controls run IntegralExpressionBenchmarkFunctions with sf.integral.previous=true, sf.integral.fallbackControl=true and the recorded exact fallback reason. Complete jobs use IntegralExpressionPipelineBenchmark with matching inputs and expressions. Remaining scope stays in #497 and #508.

Integer text producers for PRINTF

On 2026-11-04, unbounded STRING CAST/TRY_CAST of a proven total integer producer can feed the primitive formatter without materializing intermediate text. The planner first validates conversions against the logical STRING argument. Only then does it emit the underlying integer: %s/%S and %b/%B have the same result for its decimal text, including NULLs, precision, padding and indexing. Numeric conversions remain invalid for STRING; their NULL-specific released behavior stays on the preceding route. Bounded character targets, BOOLEAN/DECIMAL text and unverified/failing producers are excluded. In particular, BOOLEAN false formatted with %b differs from its non-NULL text false.

This Java-only elision reuses the byte-identical native runtime and its buffered formatter. Existing direct length/comparison consumers reuse the same ASCII proof; it does not widen general STRING/representation admission. The record retains all configurations, hashes, allocation diagnostics and 3,500 raw trials. Apple M4 Pro, macOS arm64, one thread, JDK 17.0.12, released Flink 2.2.1, Java release 11, release/mimalloc. Actual preceding encoder/helper 5e18ac8d runs on its saved whole DSO/executable, including its generated evaluator; measured and validated bytecode match.

Prepared stock starts on GenericRowData/BinaryRowData and native starts inside Rust on equal prebuilt Arrow. Each cell has 131,072 logical rows, 16 warmups and nine alternating measurements. Production Calc, output consumption, allocation, intrinsic conversions and callbacks remain inside the Rust timer after import and before export. Initial JNI/import/ export and perimeter transposes are outside this small-function timer only. Four signed integer widths, both NULL modes/layouts, computed wrapping, mixed BOOLEAN/integer arguments, general string/Boolean formats, indexing, precision and padding are covered; all digests match.

Scope Cells Stock / native Actual previous / native Independent stock / native
Bulk, 1,024 rows 28 2.059-7.947x 3.023-8.367x 2.102-8.084x
Tiny, 16 rows 7 1.184-5.362x 8.791-12.787x 1.138-5.676x

Four complete million-row jobs use bounded external Row sources and blackhole sinks, retaining job submission/startup, JNI and both transposes. Each engine has two warmups and five alternating measured jobs; surrounding JVM startup and setup are outside timing. Candidate and preceding native operators must consume work; only the new route requires empty evaluators. Median seconds:

Shape Paired stock Native Independent stock Actual previous
int-text 0.267157 0.225393 0.268707 0.292513
bigint-text 0.691436 0.250366 0.622890 0.655572
boolean-and-text 0.816311 0.279250 0.753133 0.738243
computed-text 0.668061 0.249539 0.682569 0.675910

Native/stock elapsed spans 0.342-0.844 and native/previous 0.369-0.771. All measured cells improve, with raw variability and independent stock controls retained. These gains qualify this bounded feature, not universal job acceleration or the earlier 16-row comparison regression, which remains in its own record.

All 172 final focused JVM invocations pass without skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java 11. Full 5,003-row maps/schema cover CAST and TRY_CAST, signed extrema/NULLs, modern/legacy modes, arithmetic wrapping, mixed BOOLEAN, length/comparison consumers, all-filtered batches, invalid STRING numeric conversions, bounded/BOOLEAN/DECIMAL exclusions and failing/lazy/unused arguments. Admitted paths require consumed NativeCalc work, no fallback and empty evaluators. On each JDK the actual previous encoder compares eight first-query maps/schema, then fails the intended route assertions; later legacy/empty queries in those invocations are not counted. Those broad queries include ordered consumers and encounter the preceding internal-RowData whole-Calc fallback. Its residual dynamic control passes; prepared direct-PRINTF controls independently verify and time its generated native Calc route.

The unchanged DSO/executable retain their earlier 912-core-test and released PRINTF stock- oracle qualification; these are not new Rust/oracle runs. Flink 1.18's lack of PRINTF remains an absence control, not a native gap. Hosted suites must qualify the exact pushed head.

Reproduce prepared measurements with release integral_expression and the record's type/expression, secondary type, NULL/layout/batch and SF_INTEGRAL_CORPUS=decimal-truth. Previous controls set SF_INTEGRAL_PREVIOUS=true with the actual saved encoder/helper/runtime. Complete jobs use IntegralExpressionPipelineBenchmark, matching boolean/BIGINT input, secondary type and expression properties. Remaining direct STRING/DECIMAL arguments, formats and consumers stay in #497.

Primitive text UPPER

On 2026-11-04, built-in UPPER executes in Rust for verified total integer/BOOLEAN casts to unbounded STRING on both released lines. Qualified primitive ASCII PRINTF also participates on Flink 2.2 when its literal contains no lowercase i. Primitive conversions cannot write i, so this domain has identical Java-backed/binary-only and worker-locale behavior. Integer text has no letters and reuses the native CAST result; BOOLEAN text writes uppercase directly. Only formatted text needs the existing Arrow case kernel. General STRING/Unicode, bounded casts, unverified/failing producers, dependent consumers and existing relational guards retain their previous routes.

The record and 10,500 raw samples retain 7,000 final trials and 3,500 trials from the initial redundant-case implementation. Apple M4 Pro, macOS arm64, one thread, JDK 17.0.12, released Flink 1.18.1/2.2.1, Java release 11, release/mimalloc. Actual preceding encoder/helper 534099a2 uses its saved whole runtime, including the generated evaluator. Native artifacts are byte-identical; measured/validated encoder bytecode and source/artifact hashes match.

Prepared stock starts on equal GenericRowData/BinaryRowData and native starts inside Rust on equal Arrow. Each cell processes 131,072 logical rows with sixteen warmups and nine alternating measurements. Production Calc, output consumption, allocation and intrinsic conversions/callbacks remain inside timing; initial JNI/import/export and perimeter transposes are outside this small-function gate only. Four integer widths, NULL modes/layouts, modern/legacy BOOLEAN text and padded/indexed PRINTF are covered.

Flink Batch rows Cells Stock / native Actual previous / native Independent stock / native
2.2 1,024 32 1.569-18.254x 2.964-27.255x 1.592-15.203x
1.18 1,024 24 1.511-21.388x 2.916-27.811x 1.569-18.239x
2.2 16 8 0.489-4.960x 10.057-26.178x 0.471-5.085x
1.18 16 6 0.519-1.293x 11.934-24.137x 0.502-1.142x

All 56 bulk cells beat stock and previous. Seven of fourteen tiny controls remain slower than stock. A three-second native-only INT/16-row sample identifies scalar dispatch, casts and case conversion. Removing the redundant integer case scan/output reduces a representative batch from 38 allocations/2,760 requested bytes to 33/2,208. BOOLEAN text also writes its final case directly. The original 2.2 tiny BOOLEAN control improves from 0.824x to 1.205x stock; simple small integers retain fixed Calc/CAST overhead and losses. All initial measurements and allocation diagnostics remain; bulk qualification does not establish universal small-batch acceleration.

Eight complete million-row configurations use bounded external Row sources and blackhole sinks, retaining job submission/startup, JNI and both transposes. Each engine has two warmups and five alternating measured jobs. Candidate and previous must consume native work; only the candidate requires empty evaluators. Median seconds:

Flink/shape Paired stock Native Independent stock Actual previous
22-int-text 0.231095 0.215401 0.234017 0.257369
22-boolean-text 0.209379 0.250789 0.210564 0.259598
22-int-format 0.713247 0.257157 0.677347 0.699764
22-boolean-format 0.680982 0.297569 0.654967 0.707587
22-boolean-legacy 0.210135 0.197164 0.231739 0.299726
118-int-text 0.231382 0.204040 0.227603 0.253104
118-boolean-text 0.220297 0.247613 0.209224 0.241832
118-boolean-legacy 0.237388 0.257726 0.228105 0.289418

Native/stock elapsed spans 0.361-1.198 and native/previous 0.367-1.024. Three complete-job medians lose to stock. The 1.18 modern BOOLEAN median also loses 2.4% to previous, with overlapping ranges. All losses, stock-process drift and raw variability remain; the prepared bulk gate is distinct from complete-job acceleration.

All 238 final focused JVM invocations pass without skips on released Flink 1.18.1/2.2.1 and JDK 11/17, compiling with Java release 11: 106 case/existing-function invocations plus 132 BOOLEAN CAST regression invocations. Full 5,003-row maps/schema cover signed extrema, NULLs, total wrapping/BOOLEAN producers, modern/legacy modes, empty batches, invalid formats, bounded/DECIMAL exclusions, failing/lazy paths and same-name catalog UDFs. Four runtime locales and exhaustive single-byte ASCII validate both released string representations. Admitted paths require consumed NativeCalc work, no fallback and empty evaluators. Four unchanged released testUpperCase stock-oracle invocations pass separately.

Per JDK, the previous encoder compares thirteen full first-query maps/schema on 2.2 and ten on 1.18, plus the first locale query, before the intended evaluator-route failures. Later modes/queries in failing invocations are not counted. Its two residual/ catalog controls per version/JDK pass. The unchanged native artifacts retain their archived 912-core-test qualification; these are not new Rust runs. Hosted suites must qualify the exact pushed head separately.

Reproduce prepared cells with release integral_expression, the record's type/expression, NULL/layout/batch and SF_INTEGRAL_CORPUS=decimal-truth. Legacy controls also set SF_INTEGRAL_LEGACY=true. Previous controls set SF_INTEGRAL_PREVIOUS=true with the saved encoder/helper/runtime. Complete jobs use IntegralExpressionPipelineBenchmark with matching input, expression and sf.integral.legacy properties. General Unicode casing and other residual contracts remain in #494.

Primitive text LOWER

On 2026-11-04, LOWER reuses the primitive producer proof and native text writers qualified for UPPER. Total integer/BOOLEAN casts to unbounded STRING are admitted on both released lines. Qualified primitive ASCII PRINTF participates on Flink 2.2 when its literal contains no uppercase I: cached Java strings otherwise lowercase it differently in Turkish/Azeri locales. Primitive conversions cannot introduce I. Integer text reuses its existing native CAST result, BOOLEAN writes lowercase directly, and formatted text uses the existing Arrow case kernel. General STRING/Unicode and consumers keep their exact paths.

The record and 7,000 raw trials retain all cells, allocation diagnostics, variability and actual source/artifact hashes. Apple M4 Pro, macOS arm64, one thread, JDK 17.0.12, released Flink 1.18.1/2.2.1, Java release 11, release/mimalloc. Actual preceding encoder/helper 60d1e655 includes native UPPER but retains generated LOWER evaluation. Both whole runtimes are byte-identical; measured and validated scalar bytecode match. No new configuration or transport contract is introduced.

Prepared stock starts on GenericRowData/BinaryRowData and native starts inside Rust on equal Arrow. Each cell processes 131,072 logical rows, sixteen warmups and nine alternating measurements. Production Calc, result consumption, allocation and intrinsic conversions/ callbacks remain inside timing; initial JNI/import/export and perimeter transposes are outside this small-function gate only. Four integer widths, both NULL modes/layouts, modern/legacy BOOLEAN text and padded/indexed formats are covered; all digests match.

Flink Batch rows Cells Stock / native Actual previous / native Independent stock / native
2.2 1,024 32 1.514-16.338x 2.959-28.997x 1.687-14.814x
1.18 1,024 24 1.508-21.269x 3.020-30.765x 1.638-15.777x
2.2 16 8 0.514-4.932x 10.132-26.523x 0.525-4.992x
1.18 16 6 0.498-1.158x 12.343-24.451x 0.543-1.212x

All 56 bulk medians beat stock and previous; nine of fourteen tiny controls still lose to stock. This extension already uses the integer identity and final-case BOOLEAN writer profiled in the UPPER qualification. Remaining small-batch Calc/CAST costs, independent stock drift and the slightly slower legacy BOOLEAN tiny control are retained.

Eight complete million-row configurations use bounded external Row sources and blackhole sinks, retaining job submission/startup, JNI and both transposes. Each engine has two warmups and five alternating measured jobs. Both candidate/previous require consumed native work; only the candidate requires empty evaluators. Median seconds:

Flink/shape Paired stock Native Independent stock Actual previous
22-int-text 0.240047 0.201903 0.231463 0.261967
22-boolean-text 0.210758 0.208022 0.212839 0.264560
22-int-format 0.645009 0.263799 0.631092 0.650668
22-boolean-format 0.645045 0.286106 0.661133 0.700037
22-boolean-legacy 0.230676 0.244194 0.211327 0.241696
118-int-text 0.233692 0.214742 0.234665 0.260876
118-boolean-text 0.233153 0.247825 0.218829 0.265762
118-boolean-legacy 0.229295 0.238191 0.212077 0.288261

Native/stock elapsed spans 0.409-1.063, native/previous 0.405-1.010. Three complete-job medians lose to stock; 2.2 legacy BOOLEAN also loses 1.0% to previous, with overlapping ranges. These losses remain and are distinct from the bulk gate.

All 196 fresh focused JVM invocations pass without skips on released Flink 1.18.1/2.2.1 and JDK 11/17, compiling with Java release 11: 164 shared case/existing-function invocations plus 32 DECIMAL-truth/predicate composition invocations. The shared tests revalidate UPPER and LOWER over full 5,003-row maps/schema, signed extrema/NULLs, total wrapping, modern/legacy casts, empty batches, locale-sensitive/Unicode exclusions, retained consumers, failing/lazy producers and same-name catalog UDFs. Additional compact DECIMAL(4,2) and noncompact DECIMAL(38,18) controls feed both case functions through verified truth/predicate text. Admitted routes require consumed NativeCalc work, no fallback and empty evaluator descriptors. Four unchanged released testLowerCase stock oracles pass separately. The earlier 132-invocation BOOLEAN CAST qualification is retained in the UPPER record, not relabeled as a new test run.

Per JDK, preceding LOWER controls compare thirteen first-query maps/schema on 2.2 and ten on 1.18, plus the first locale query, before expected evaluator-route failures. Later modes/queries in those failing invocations are not counted. Its residual/catalog controls pass. In the additional DECIMAL matrix, all four preceding UPPER controls pass; four preceding LOWER controls compare results and then fail the intended route assertion. The unchanged native artifacts retain the archived 912-core-test qualification; these are not new Rust runs. Hosted suites must qualify the exact pushed head.

Reproduce prepared cells with release integral_expression, the record's type/expression, NULL/layout/batch and SF_INTEGRAL_CORPUS=decimal-truth; legacy cells also set SF_INTEGRAL_LEGACY=true. Previous controls set SF_INTEGRAL_PREVIOUS=true on the saved encoder/helper/runtime. Complete jobs use IntegralExpressionPipelineBenchmark with matching inputs, expression and sf.integral.legacy. Remaining general Unicode casing and other signatures stay in #494.

Integer ROUND with clamped runtime positions

This independent extension admits total INT position trees bounded by built-in GREATEST, LEAST and CASE. For example, ROUND(v,GREATEST(-38,LEAST(n,38))) stays inside the existing verified -38..38 range. Range propagation is conservative across wrapping arithmetic; every condition/argument must be total, including unselected and NULL-dominated arguments. Only all signed integer widths are admitted. Unproven ranges/producers and custom functions keep their preceding routes. Epoch precision remains on its separate 0/3/NULL proof. No runtime opcode, JNI, representation or switch is added.

The public record and 8,800 raw trials retain full ranges, independent stock controls, allocation probes, checksums and source/bytecode/artifact hashes. The actual preceding encoder is 2539fabb; it binds the ROUND JVM evaluator for these clamps. Both encoders run on byte-identical frozen release DSOs/executables with verified mimalloc aliases. This isolates Java admission on the same production kernels; it is not a replay of a different historical whole runtime. Archived 912-core-test runtime qualification is retained, rather than relabeled as new Rust tests.

Prepared operation

Stock starts on equal prepared GenericRowData or BinaryRowData; native starts on prepared Arrow buffers with its timer inside Rust, after import and before export. Input/setup, initial JNI/C Data entry and perimeter transposes are outside this small-functionality timer only. Production Calc evaluation, output consumption, allocations and intrinsic conversions/callbacks remain inside. One thread, JDK 17, Java release 11, both released Flink lines, 131,072 logical rows per trial, 16 warmups and nine measured repetitions. Both nullable modes, independent position NULLs, signed extrema and BIGINT values beyond 2^53 are included. Each comparison checks matching consumed-output digests.

Each type has GREATEST(-38,LEAST(n,38)) and GREATEST(-3,LEAST(n,-1)) shapes. The wide prepared corpus mostly saturates at -38/38; its results alone would overrepresent zero or identity shortcuts. The separate negative clamp executes nontrivial -3/-1 rounding. Functional SQL sources additionally walk all interior positions through id-2500, with nested/NULL CASE, arithmetic, STRING/epoch consumers and empty batches.

All 64 bulk cells (1,024 rows/batch) beat paired stock by 2.052-20.494x and previous by 6.389-29.439x. Ratios of paired medians:

Released Flink Input Stock/native throughput Previous/native throughput
2.2.1 TINYINT 2.424-19.783x 7.466-29.439x
2.2.1 SMALLINT 2.412-17.210x 7.726-27.877x
2.2.1 INT 3.298-17.027x 6.685-24.739x
2.2.1 BIGINT 2.593-19.992x 7.215-26.252x
1.18.1 TINYINT 2.052-20.494x 6.389-27.700x
1.18.1 SMALLINT 2.322-18.250x 7.877-26.777x
1.18.1 INT 2.710-14.844x 7.842-23.883x
1.18.1 BIGINT 3.020-18.607x 7.274-26.679x

Of 16 sixteen-row controls, 12 lose to stock. Their stock/native ratios range from 0.211 to 1.189x. These are retained diagnostics, not removed from the data or used to claim universal tiny-batch acceleration. Independent prior-JVM stock timings remain alongside candidate-JVM stock, with no subtraction or normalization.

Complete jobs and correctness

Eight complete configurations process 1M rows each: INT/BIGINT, both clamp shapes and both Flink releases. Runtime rowwise source to blackhole, parallelism one, two warmups, five alternating stock/native measurements per engine, submission/startup, JNI and both RowData/Arrow transposes remain in the job timer. Seconds, median (minimum-maximum):

Release/shape Stock Native Independent stock Previous
22-int-wide 0.211375 (0.208757-0.216321) 0.188357 (0.186368-0.215522) 0.214924 (0.211917-0.232876) 0.246277 (0.243375-0.281274)
22-int-negative 0.192676 (0.189297-0.204379) 0.194442 (0.191757-0.353703) 0.187509 (0.184368-0.193174) 0.220544 (0.217093-0.246181)
22-bigint-wide 0.236759 (0.234225-0.242784) 0.214993 (0.201897-0.233321) 0.232573 (0.225093-0.236360) 0.272229 (0.264901-0.288339)
22-bigint-negative 0.199538 (0.192411-0.207178) 0.209872 (0.200907-0.212597) 0.201076 (0.200125-0.215894) 0.238390 (0.229101-0.250444)
118-int-wide 0.215960 (0.204185-0.230945) 0.189556 (0.189021-0.207103) 0.218499 (0.210813-0.220883) 0.255599 (0.250429-0.270002)
118-int-negative 0.209546 (0.188340-0.211879) 0.199818 (0.194083-0.222225) 0.190139 (0.185749-0.218386) 0.239198 (0.221494-0.266854)
118-bigint-wide 0.235236 (0.223393-0.250472) 0.216688 (0.209132-0.226110) 0.240505 (0.236140-0.253585) 0.283365 (0.273492-0.312494)
118-bigint-negative 0.206729 (0.200961-0.242730) 0.218570 (0.215633-0.229089) 0.205106 (0.196390-0.219563) 0.253952 (0.247074-0.277491)

Native/stock elapsed is 0.878-1.057; 3 configurations lose to paired stock. Native/previous is 0.742-0.882, with 0 previous-path losses. Whole-job variability remains in the raw record; prepared-operation speedups do not establish universal full-job acceleration.

240 fresh focused JVM invocations pass with no skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java 11. Complete maps/schema compare 5,003-row sources, widths, signed overflow, HALF_UP/toward-zero behavior as applicable, independent NULLs, lazy/failing paths and empty batches. Admitted queries require consumed NativeCalc work, no fallback and empty evaluators. Original literal/bounded CASE and temporal-precision controls pass. Four unchanged released ROUND stock-oracle invocations pass separately. The preceding encoder compares every positive first-query map/schema and then fails the intended pure-route assertion; later queries in those failing invocations are not counted. Both retained/generated and same-name catalog-UDF controls pass. These local qualifications do not stand in for the hosted suites on the exact pushed PR head.

Reproduce prepared controls with the existing release integral_expression example, SF_INTEGRAL_SECONDARY_TYPE=INT and SF_INTEGRAL_EXPRESSION='ROUND(v,GREATEST(-3,LEAST(n,-1)))', selecting each input width, nullable/row-representation/batch configuration and the preceding encoder first on its classpath. Complete jobs use IntegralExpressionPipelineBenchmark, the same expression and -Dsf.integral.secondaryType=INT; the prior control sets sf.integral.previous=true. The records retain the exact source/artifact hashes and trial configuration.

Integer TRUNCATE with clamped runtime positions

This independent extension admits total INT position trees bounded by built-in GREATEST, LEAST and CASE. For example, TRUNCATE(v,GREATEST(-38,LEAST(n,38))) stays inside the existing verified -38..38 range. Range propagation is conservative across wrapping arithmetic; every condition/argument must be total, including unselected and NULL-dominated arguments. Only INT/BIGINT are admitted. Unproven ranges/producers and custom functions keep their preceding routes. Epoch precision remains on its separate 0/3/NULL proof. No runtime opcode, JNI, representation or switch is added.

The public record and 4,000 raw trials retain full ranges, independent stock controls, allocation probes, checksums and source/bytecode/artifact hashes. The actual preceding encoder is 0acdd9b7; it binds the TRUNCATE JVM evaluator for these clamps. Both encoders run on byte-identical frozen release DSOs/executables with verified mimalloc aliases. This isolates Java admission on the same production kernels; it is not a replay of a different historical whole runtime. Archived 912-core-test runtime qualification is retained, rather than relabeled as new Rust tests.

Prepared operation

Stock starts on equal prepared GenericRowData or BinaryRowData; native starts on prepared Arrow buffers with its timer inside Rust, after import and before export. Input/setup, initial JNI/C Data entry and perimeter transposes are outside this small-functionality timer only. Production Calc evaluation, output consumption, allocations and intrinsic conversions/callbacks remain inside. One thread, JDK 17, Java release 11, both released Flink lines, 131,072 logical rows per trial, 16 warmups and nine measured repetitions. Both nullable modes, independent position NULLs, signed extrema and BIGINT values beyond 2^53 are included. Each comparison checks matching consumed-output digests.

Each type has GREATEST(-38,LEAST(n,38)) and GREATEST(-3,LEAST(n,-1)) shapes. The wide prepared corpus mostly saturates at -38/38; its results alone would overrepresent zero or identity shortcuts. The separate negative clamp executes nontrivial -3/-1 rounding. Functional SQL sources additionally walk all interior positions through id-2500, with nested/NULL CASE, arithmetic, STRING/epoch consumers and empty batches.

All 32 bulk cells (1,024 rows/batch) beat paired stock by 6.593-28.118x and previous by 11.913-36.111x. Ratios of paired medians:

Released Flink Input Stock/native throughput Previous/native throughput
2.2.1 INT 6.746-26.683x 11.959-33.213x
2.2.1 BIGINT 8.351-28.118x 12.890-34.283x
1.18.1 INT 6.593-26.033x 11.913-36.111x
1.18.1 BIGINT 9.130-26.918x 12.208-35.569x

Of 8 sixteen-row controls, 3 lose to stock. Their stock/native ratios range from 0.826 to 1.613x. These are retained diagnostics, not removed from the data or used to claim universal tiny-batch acceleration. Independent prior-JVM stock timings remain alongside candidate-JVM stock, with no subtraction or normalization.

Complete jobs and correctness

Eight complete configurations process 1M rows each: INT/BIGINT, both clamp shapes and both Flink releases. Runtime rowwise source to blackhole, parallelism one, two warmups, five alternating stock/native measurements per engine, submission/startup, JNI and both RowData/Arrow transposes remain in the job timer. Seconds, median (minimum-maximum):

Release/shape Stock Native Independent stock Previous
22-int-wide 0.216388 (0.213630-0.223902) 0.190361 (0.185447-0.204954) 0.220159 (0.213937-0.223426) 0.258516 (0.248854-0.274577)
22-int-negative 0.248733 (0.244749-0.253861) 0.190804 (0.190657-0.208462) 0.237455 (0.233748-0.249631) 0.276953 (0.272416-0.291678)
22-bigint-wide 0.235817 (0.230376-0.243834) 0.208476 (0.202216-0.217662) 0.243563 (0.239602-0.250967) 0.276771 (0.260991-0.290315)
22-bigint-negative 0.263149 (0.255959-0.270185) 0.201221 (0.196207-0.217436) 0.254803 (0.250489-0.266496) 0.293021 (0.287375-0.307211)
118-int-wide 0.222041 (0.216154-0.231507) 0.197143 (0.194934-0.198736) 0.216478 (0.215683-0.242135) 0.253723 (0.248861-0.282366)
118-int-negative 0.240669 (0.230602-0.243193) 0.197705 (0.187929-0.199502) 0.234460 (0.230710-0.239697) 0.269294 (0.262753-0.283474)
118-bigint-wide 0.242870 (0.237979-0.257082) 0.210705 (0.209571-0.218985) 0.249933 (0.234538-0.263448) 0.275957 (0.267568-0.288913)
118-bigint-negative 0.252105 (0.246873-0.266810) 0.214418 (0.206610-0.221217) 0.265535 (0.260429-0.270674) 0.301647 (0.293013-0.315090)

Native/stock elapsed is 0.765-0.888; 0 configurations lose to paired stock. Native/previous is 0.687-0.777, with 0 previous-path losses. Whole-job variability remains in the raw record; prepared-operation speedups do not establish universal full-job acceleration.

264 fresh focused JVM invocations pass with no skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java 11. Complete maps/schema compare 5,003-row sources, widths, signed overflow, HALF_UP/toward-zero behavior as applicable, independent NULLs, lazy/failing paths and empty batches. Admitted queries require consumed NativeCalc work, no fallback and empty evaluators. Original literal/bounded CASE and temporal-precision controls pass. Four unchanged released TRUNCATE stock-oracle invocations pass separately. The preceding encoder compares every positive first-query map/schema and then fails the intended pure-route assertion; later queries in those failing invocations are not counted. Both retained/generated and same-name catalog-UDF controls pass. These local qualifications do not stand in for the hosted suites on the exact pushed PR head.

Reproduce prepared controls with the existing release integral_expression example, SF_INTEGRAL_SECONDARY_TYPE=INT and SF_INTEGRAL_EXPRESSION='TRUNCATE(v,GREATEST(-3,LEAST(n,-1)))', selecting each input width, nullable/row-representation/batch configuration and the preceding encoder first on its classpath. Complete jobs use IntegralExpressionPipelineBenchmark, the same expression and -Dsf.integral.secondaryType=INT; the prior control sets sf.integral.previous=true. The records retain the exact source/artifact hashes and trial configuration.

Length of proven primitive case results

Direct built-in CHAR_LENGTH and CHARACTER_LENGTH now consume qualified native UPPER/LOWER results without a JVM evaluator. Their existing producer proof establishes locale-independent ASCII from total integer/BOOLEAN text, direct DECIMAL truth/predicate text and qualified Flink 2.2 primitive PRINTF. UTF-8 and Java UTF-16 lengths therefore agree. Integer casing still shares its CAST result; BOOLEAN writes its final case directly. The existing Arrow length kernel consumes those results. This introduces no Rust source, transport, opcode or switch change. General Unicode, bounded casts, locale-sensitive formats, fallible/unverified producers and further dependent expressions keep prior routes.

The record and 9,400 raw trials preserve full ranges, independent stock controls, pilot provenance, checksums, allocations and source/ bytecode/artifact hashes. Actual preceding encoder 519f221f already supports primitive UPPER/LOWER but delegates these length consumers. Both versions run on identical release/mimalloc runtime artifacts; archived 912-core-test qualification is retained rather than called a new Rust run. Final validated and measured scalar bytecode matches.

Functionality gate

One thread on Apple M4 Pro/macOS arm64, JDK 17.0.12, Java release 11, released Flink 1.18.1/2.2.1. Stock starts on equal prepared GenericRowData/BinaryRowData; native starts inside Rust on prepared Arrow. Input/setup, initial JNI/import/export and perimeter transposes are outside this small-function timer only. Production Calc, output consumption, allocation and intrinsic conversions/callbacks remain inside. Every cell processes 131,072 logical rows with sixteen warmups and nine measured alternating trials; all digests match.

The matrix covers four integer widths, BOOLEAN, compact DECIMAL(4,2) truth and noncompact DECIMAL(38,18) predicates, both NULL modes and row layouts, modern/legacy text and indexed/ padded formatter output. PRINTF is absent from Flink 1.18 and its cells run only on 2.2. Every successful encoder control now uses the production context-aware entry, passing the real TableConfig even when the output is INT. The old output-type heuristic missed inner character casts in numeric length outputs and failed registration before any timing. The corrected common fixture is used by stock/actual-prior/native, with equal fixture bytecode for both encoders. Setup/configuration remains outside timing; rejected-signature controls keep their diagnostic path. No timings from the failed registration are used.

Flink Batch rows Cells Stock/native throughput Previous/native throughput Independent stock/native
2.2.1 1,024 40 1.580-16.236x 2.898-21.935x 1.551-16.865x
2.2.1 16 10 0.424-4.561x 8.989-19.484x 0.430-4.827x
1.18.1 1,024 32 1.437-16.119x 2.626-21.305x 1.629-15.155x
1.18.1 16 8 0.413-0.903x 10.027-18.291x 0.427-0.934x

All 72 bulk cells beat stock by 1.437-16.236x and previous by 2.626-21.935x. Of 18 sixteen-row controls, 15 lose to stock. Those losses, independent stock drift and the four-cell pilot are retained. The final common fixture removes only unreachable rejected- signature diagnostic branches after that pilot; pilot source/bytecode hashes remain in the record. There is no broad small-batch acceleration claim.

Complete jobs and correctness

Twelve complete 1M-row configurations retain the runtime rowwise source, blackhole sink, parallelism one, submission/startup, JNI and both RowData/Arrow transposes. Each engine has two warmups and five alternating measured trials. Shapes include integer and BOOLEAN text, compact/noncompact DECIMAL producers, modern/legacy text and applicable formatters. Seconds, median (minimum-maximum):

Release/shape Stock Native Independent stock Previous
22-int-text 0.196109 (0.186710-0.216379) 0.180605 (0.175743-0.197065) 0.184473 (0.181079-0.204818) 0.221067 (0.216402-0.246059)
22-boolean-text 0.194744 (0.173471-0.206837) 0.167066 (0.161800-0.183659) 0.174838 (0.163094-0.179601) 0.205228 (0.201143-0.219627)
22-int-format 0.672973 (0.655778-0.678616) 0.230926 (0.226071-0.246572) 0.631091 (0.618354-0.633107) 0.620273 (0.603027-0.631882)
22-boolean-format 0.664518 (0.664071-0.673431) 0.218722 (0.212579-0.230046) 0.668366 (0.661009-0.669023) 0.639350 (0.630945-0.648802)
22-decimal-truth 0.202152 (0.194988-0.217200) 0.189457 (0.182427-0.207775) 0.191021 (0.186779-0.209982) 0.234690 (0.232336-0.238282)
22-decimal-predicate 0.190472 (0.185858-0.201505) 0.189945 (0.185299-0.199012) 0.187950 (0.186691-0.195766) 0.245734 (0.236047-0.250349)
22-boolean-legacy 0.187480 (0.180378-0.200136) 0.171613 (0.164190-0.181044) 0.180563 (0.175631-0.187807) 0.211343 (0.200118-0.223612)
118-int-text 0.204004 (0.191636-0.216645) 0.188930 (0.181423-0.206237) 0.190948 (0.187601-0.198028) 0.220120 (0.217568-0.234642)
118-boolean-text 0.168852 (0.162029-0.199493) 0.168579 (0.161640-0.182174) 0.174042 (0.168368-0.176570) 0.207406 (0.197169-0.213397)
118-decimal-truth 0.186952 (0.181923-0.200563) 0.178256 (0.173796-0.191289) 0.195962 (0.187249-0.206562) 0.224311 (0.213567-0.235870)
118-decimal-predicate 0.190382 (0.181827-0.206279) 0.190054 (0.184949-0.242657) 0.186770 (0.182483-0.198041) 0.251478 (0.233013-0.253193)
118-boolean-legacy 0.175627 (0.167726-0.193887) 0.174471 (0.164651-0.196198) 0.174645 (0.161150-0.194922) 0.215897 (0.210575-0.231338)

Native/stock elapsed ranges from 0.329 to 0.998; 0 configurations lose to paired stock. Native/previous elapsed is 0.342-0.858, with 0 previous-path losses. Variability and independent controls remain in the record. Two independent prior-JVM stock medians beat native in DECIMAL predicate jobs, and several paired ranges overlap. No stable complete-job gain is claimed for those near-parity cases. Function-level gains are not universal job gains.

190 fresh focused JVM invocations pass with no skips across both released lines and JDK 11/17, targeting Java 11. They revalidate UPPER/LOWER and compare complete 5,003-row maps/schema for integer/BOOLEAN text, compact/noncompact DECIMAL truth/predicates, NULLs, wrapping, both cast modes, four JVM locales, lazy failures and empty batches. Positive routes require consumed NativeCalc work, no fallback and empty JVM evaluators. The actual prior encoder compares every positive first-query map/schema before the intended route assertion fails; later queries in those failing invocations are not counted. Generated/consumer controls and same-name catalog UDFs pass. Four unchanged released testCharLength stock-oracle invocations pass separately. Hosted suites still need the exact pushed PR head; earlier per-function records keep their original provenance.

Use the existing release integral_expression example with, for example, SF_INTEGRAL_EXPRESSION='CHAR_LENGTH(UPPER(CAST(v AS STRING)))', then select the input width/NULL/layout/batch and candidate or actual preceding encoder on its classpath. IntegralExpressionPipelineBenchmark uses the same expression through sf.integral.expression; previous controls set sf.integral.previous=true. Records retain exact hashes/settings.

UTC FLOOR over integer epochs

FLOOR(TO_TIMESTAMP_LTZ(n,p) TO unit) composes the existing Rust integer, epoch and rounding kernels in fixed-zero session zones, after proving the genuine built-in conversion and total integer/precision producers. YEAR/MONTH/DAY/HOUR qualify on both released lines; smaller verified units remain 2.2-only. Other zones, fallible/unverified inputs and invalid precision keep their correct preceding routes. See the coverage contract. The fixed-unit FLOOR kernel now writes directly to the existing capacity-sized timestamp builder. This removes a temporary result vector and repeated growth/splitting, while retaining Flink arithmetic and the existing JNI/transport/configuration contracts.

The final matrix passes 348 fresh JVM invocations with zero skips across released Flink 1.18.1/2.2.1 and JDK 11/17, targeting Java 11. Full 5,003-row maps/schema cover all signed integer widths, NULLs, epoch limits, wrapping arithmetic, runtime 0/3/NULL precision, YEAR/NULL consumers, three zero-zone aliases and all-filtered batches. Direct LTZ FLOOR/CEIL, nanoseconds and released unsupported sub-hour forms are revalidated. Catalog functions and fallible/unknown producers retain their existing contracts. Four unchanged released testTemporalFloor stock-oracle invocations pass separately. Actual preceding encoder b6cccbf1 matches each positive invocation's first complete map/schema before the intended pure-route assertion fails. Subsequent queries in those failing invocations are not reached. Admitted SQL consumes NativeCalc with no fallback and empty evaluators.

The rebuilt release runtime passes 914 fresh Rust core tests, including sliced parent validity, independent NULL operands, empty results and fractional rounding in the new builder regression. The preceding runtime retains its separate archived 912-test qualification. Measured and validated encoder, build-selected compatibility and fixture hashes are recorded. Setup additionally requires FLOOR in candidate native descriptors and previous JVM descriptors, preventing the outer YEAR consumer from disguising a generated or removed round operation.

Prepared timing starts stock on equal prebuilt GenericRowData/BinaryRowData and native inside Rust on equal Arrow batches. It retains production epoch, rounding and YEAR, NULL handling, result consumption, allocation and intrinsic conversions/callbacks; initial JNI/import/export and perimeter transposes remain outside this small-function timer. One thread, 131,072 rows/trial, 16 warmups and nine measurements, both NULL modes, batches 1,024 and 16. Four shapes consume seconds/YEAR, milliseconds/MONTH, INT milliseconds/DAY, and runtime CASE precision/HOUR through YEAR. Epoch corpora include released range limits, invalid values and signed extrema; INT uses the identical signed-width generator in both runtimes. Candidate and actual preceding encoders use identical fixtures/TableConfig and their recorded release native artifacts; stock is independently measured in both JVMs.

Prepared controls Cells Stock/native throughput Previous/native throughput
1,024 rows/batch 32 1.017-1.985x 2.552-5.131x
16 rows/batch 8 0.112-0.321x 8.137-11.642x

All 32 final bulk medians beat stock and previous. The two smallest paired gains are only about 2%, with overlapping ranges; they do not establish a stable speedup. 8/8 tiny controls lose to stock; they remain in the record. Independent stock medians and min/max ranges remain visible rather than being replaced with paired controls.

Complete jobs use a bounded million-row source and blackhole sink, one thread, two warmups and five alternating measured trials. Submission/startup, JNI and both perimeter transposes stay inside the job timer; surrounding JVM startup is outside. Six configurations span the three BIGINT shapes on both releases. Values are median elapsed seconds:

Release-shape Paired stock Native Independent stock Previous
22-seconds-year 0.174415 0.194423 0.174084 0.235397
22-millis-month 0.179176 0.188011 0.174567 0.222483
22-runtime-hour 0.171393 0.188866 0.168582 0.221171
118-seconds-year 0.181120 0.192063 0.178531 0.216032
118-millis-month 0.177334 0.194292 0.180732 0.225815
118-runtime-hour 0.174516 0.188233 0.173405 0.216282

Native/stock elapsed is 1.049-1.115, with 6/6 paired medians slower than stock; native/previous is 0.826-0.889, with 0/6 slower than previous. 6/6 independent stock medians also beat native. Overlapping ranges and slower complete jobs remain limitations; prepared bulk gains do not establish universal pipeline acceleration.

The initial 32 bulk cells include one nullable runtime-precision/HOUR cell at 0.978x stock. A five-second process sample locates timestamp_array and vector growth under TemporalRound (272 sampled inclusive frames); counts are not elapsed percentages. The capacity-sized builder follows the same Arrow construction pattern as Comet's temporal casts, reusing StreamFusion's existing two-component builder rather than Spark semantics. Builder-only reduces a representative allocation probe from 122/86,917 bytes to 99/48,253 bytes, but still has slower bulk cells (0.871x on one 1.18 control). Disassembly identifies a runtime sdiv inside the per-row loop. The final scalar-width specialization makes the five supported widths constant to the compiler and avoids materializing width/direction arrays; arbitrary widths and NULL arguments retain the generic path. This retains all released signed-overflow and nanosecond rules. All five specialized implementations have no integer division instruction in the release binary. The same final allocation probe is 92 requests and 39,541 bytes. Both initial stages, their twelve complete jobs and source/artifact provenance remain in the record alongside the final optimization, including every slower result.

All 12,000 prepared raw samples, allocation probes, complete-job trials, independent controls, source/bytecode/artifact hashes and exact expressions are in the portable JSON and compressed raw CSV. The existing prepared integral_expression fixture selects these shapes through SF_INTEGRAL_EXPRESSION, SF_INTEGRAL_CORPUS=epoch-seconds|epoch-millis and SF_INTEGRAL_REQUIRED_FUNCTION=FLOOR; the complete-job fixture uses its corresponding epoch input properties. The record identifies the actual preceding encoder; an arbitrary older main/runtime is not an equivalent previous control. Full hosted checks qualify the exact pushed head separately. Broad #504 remains open for other producers/zones, units and recurring-DST extraction.