Skip to content

Benchmarks

Use the evidence index to find complete comparisons, component measurements, profiles and superseded experiments. Each recorded result applies to its stated revision and environment; current support is documented on the operator and connector pages.

Methodology

  • Small functionality admission. Flink starts in Java on prebuilt RowData; StreamFusion starts in Rust on prebuilt Arrow RecordBatch buffers. Match logical rows, types, NULLs, widths, cardinalities, resources and equivalent output consumption. Prepare inputs/operator setup outside timing. The StreamFusion clock starts and stops inside Rust, after initial Arrow import and before final export; Java-side JNI/C Data entry/exit and perimeter transposes are excluded. Intrinsic operation/state work and JVM/UDF callbacks remain timed. A Java timer around JNI is a separate wrapper diagnostic. Use release builds, warmup, repeated trials, parity checks and verified native execution. Correct UDF coverage may be slower than Flink: this accepted tradeoff does not require a performance-only opt-in or justify a speedup claim. Nexmark, connector, complete-pipeline and end-to-end measurements retain their full production boundaries and costs.
  • Nexmark, end to end. The headline numbers below are an end-to-end, exactly-once Kafka pipeline — not a blackhole sink. Stock Flink and StreamFusion run at parallelism 4, read the same 2M-event Kafka JSON corpus from a four-partition topic (one split per source subtask), and publish each query's result to a fresh Kafka topic with a one-second checkpoint interval. Append-only queries use kafka; updating queries use upsert-kafka with the result's actual primary key. Each timed run includes source consumption, query execution, the keyed shuffle, serialization, Kafka writes, checkpoints, and the bounded job's final transaction commit. q6 is omitted because Flink SQL itself cannot run it. Both engines receive the same properties.max.poll.records=8192 source setting and the same producer settings: batch.size=524288 and linger.ms=20.
  • Local shuffle handles are disabled. The bench Maven profile sets streamfusion.exchange.zeroCopyLocal=false, so StreamFusion serializes columnar shuffle records with Arrow IPC just as it would across TaskManagers. Stock Flink likewise serializes records on same-JVM network edges; neither side receives a process-local object-handoff advantage.
  • The native Kafka boundaries are asserted. Stock Flink uses its normal rowwise format decode. StreamFusion retains Flink's Kafka enumerator, partition assignment, offsets, checkpointing, and client, but its split-aware reader batches Kafka bytes and decodes them directly to Arrow in Rust. Sink key/value/tombstone encoding is likewise Rust, feeding Flink's exactly-once KafkaSink. An unsupported shape falls back explicitly rather than being credited as native.
  • Every cell asserts the plan shape. The native plan — including Kafka poll/decode, every supported operator, and sink key/value/tombstone serialization — is asserted for every cell, so a silent fallback can't masquerade as a native number.
  • Release builds only, always. Debug Rust is roughly an order of magnitude slower than release; every number here comes from the bench Maven profile (mvn test -Pbench ...), which builds and loads the release native library. Reporting a benchmark from a debug build is a standing mistake this project checks for explicitly — one early Parquet-copy number silently regressed from 3.19× to 0.45× before this was caught.
  • Micro-benchmarks. Per-operator Criterion benchmarks (cd native && cargo bench) measure each native operator's steady-state hot loop over an in-memory Arrow batch, isolated from the JVM bridge and Flink's scheduling — these are what the entries in Optimizations cite for a specific technique's speedup.

The format-decode microbenchmark compares the exact Nexmark Kafka value payloads across JSON, bare Avro, and protobuf. It times only bytes to the engine's destination representation: Flink's format decoder materializes one RowData per message, while the StreamFusion format decoder materializes one Arrow batch per 8,192 messages. Corpus construction, Kafka polling, SQL operators, checkpoints, and sinks are outside the timed region. Each decoder is warmed independently, every trial processes whole batches for a fixed wall-clock interval, and the best trial is reported in rows/s and ns/row.

TZ=UTC SF_BENCHMARK=true SF_DECODE_BATCH_ROWS=8192 \
  SF_DECODE_WARMUP_SECONDS=1 SF_DECODE_SECONDS=3 SF_DECODE_RUNS=3 \
  mvn -pl :streamfusion-runtime test -Pbench \
  -Dtest=NexmarkFormatDecodeBenchmark

For matched Java and native CPU flame graphs of one format, enable NexmarkFormatDecodeBenchmark#profileDecode. The harness warms both decoders, records them separately for -Dprofile.seconds=20, and writes JFR files below -Dprofile.outputDir.

TZ=UTC SF_BENCHMARK=true SF_PROFILE_DECODE=true \
  mvn -pl :streamfusion-runtime test -Pbench \
  -Dtest='NexmarkFormatDecodeBenchmark#profileDecode' \
  -Dprofile.format=protobuf -Dprofile.seconds=20 \
  -Dprofile.outputDir=target/profiles/protobuf-decode

The experimental Flink 1.18 build has a separate row-fed Nexmark compatibility baseline. Its generator/blackhole results are not comparable to the Kafka headline workload below.

Nexmark, parallelism 4

Apple M1 Max, release + mimalloc, measured across all four backend/mode combinations. The memory columns use one measured run per cell (2026-08-25); the disk columns use one warmup and the best of two measured runs (2026-09-05, after the pinned batched-read and column-state-codec work). The memory columns compare Flink's default heap state against StreamFusion's memory state; the disk columns compare stock Flink RocksDB against StreamFusion's native RocksDB backend. The Kafka harness fixes the table session time zone at UTC so TIMESTAMP_LTZ window coverage does not depend on the benchmark host. Mini-batching ("on") uses the same production-style configuration on both engines (allow-latency=2s, size=50000). Each cell is StreamFusion throughput divided by Flink throughput within the same backend and mode.

Query Memory, off Memory, on Disk, off Disk, on
q0 1.69× 1.39× 1.36× 1.40×
q1† 1.40× 1.40× 1.38× 1.41×
q2 1.26× 1.08× 1.11× 1.08×
q3 1.03× 1.16× 1.08× 1.14×
q4 1.83× 1.60× 8.92× 15.11×
q5 1.41× 1.23× 4.38× 4.38×
q7 1.30× 1.47× 7.41× 11.93×
q8 1.27× 1.12× 2.33× 2.43×
q9 1.22× 1.80× 17.83× 67.95×
q10† 1.45× 1.94× 1.42× 1.40×
q11 1.47× 1.50× 9.84× 10.03×
q12 1.09× 1.13× 1.45× 1.33×
q13 1.20× 1.14× 1.15× 1.22×
q14† 1.47× 1.25× 1.49× 1.53×
q15† 1.47× 1.22× 1.61× 1.32×
q16† 1.17× 1.41× 1.49× 1.28×
q17† 1.20× 1.18× 1.92× 1.65×
q18 1.07× 1.37× 6.01× 8.92×
q19 0.99× 2.82× 2.44× 5.09×
q20 1.22× 1.56× 37.43× 68.20×
q21† 1.23× 1.31× 1.26× 1.27×
q22 1.33× 1.25× 1.30× 1.22×
q23 1.69× 3.32× 3.15× 12.10×
geomean 1.31× 1.44× 2.75× 3.41×

† The benchmark's existing opt-in expression variants are used for these headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and q21 use native datetime or regex/case behavior that can differ from Flink at documented edge cases.

Apple M1 Max; numbers are comparable only within a machine.

Reproducing

SF_BENCHMARK=true mvn -pl :streamfusion-runtime test -Pbench

runs the end-to-end suites (ThroughputBenchmark, NexmarkBenchmark, NexmarkKafkaBenchmark, NexmarkMatrixBenchmark); the -Pbench profile is required. The Criterion micro-benchmarks run independently with cd native && cargo bench.

To reproduce the two persistent-backend columns above in one run:

SF_BENCHMARK=true SF_MATRIX_STATE_BACKENDS=true SF_STATE_BACKENDS_MINI_BATCH=both \
  SF_ROWS=2000000 SF_PARALLELISM=4 SF_KAFKA_PARTITIONS=4 SF_WARMUP=1 SF_RUNS=2 \
  mvn -pl :streamfusion-runtime test -Pbench \
  -Dtest='NexmarkMatrixBenchmark#stateBackendComparison'

To capture matched async-profiler CPU recordings for every exactly-once Kafka query with the memory backend and mini-batching disabled, run exactlyOnceKafkaSinkProfileAll with SF_PROFILE_ALL_KAFKA_SINK=true. The harness reuses one broker and input corpus, performs one warmup per engine/query, and writes flink-q*.jfr and streamfusion-q*.jfr under -Dprofile.outputDir=.... It invokes asprof from PATH by default; override that executable with -Dprofile.asprof=....

Parquet, Delta, and Paimon sink diagnostics

These diagnostics use the readme-like 2M-event Kafka JSON workload with four input partitions, parallelism four, memory state, mini-batching disabled, one warmup, and the best of three measured runs. They cover q0–q5 and q7–q23; q6 is omitted because stock Flink cannot execute it. Unlike the headline table, these runs measure local data-file output rather than Kafka output.

Apple M1 Max, release + mimalloc; Parquet and Delta measured 2026-08-22, Paimon append tables 2026-09-07 and primary-key tables 2026-09-09:

Sink Completed Suite geomean
Parquet physical changelog 23/23 1.535×
Delta (MOR for updating queries) 23/23 1.522×
Combined 46/46 1.529×
Paimon append, bucket-unaware (16 append-only queries) 16/16 1.47×
Paimon append, 4 fixed buckets (16 append-only queries) 16/16 1.47×
Paimon primary key, 4 fixed buckets, in-job compaction (7 updating queries) 7/7 1.64×

For the Parquet diagnostic, set SF_MATRIX_PARQUET_SINK=true and run NexmarkMatrixBenchmark#changelogParquetSinkComparison. This mode always disables Flink logical mini-batching. The normal filesystem/Parquet table sink is append-only, so the harness uses a benchmark-only changelog connector that writes every physical change and prepends _row_kind (+I, -U, +U, or -D) to the Parquet schema. The Flink baseline maps each change to a row and uses parquet-mr; StreamFusion keeps the query output as Arrow, materializes the four-value row-kind column natively, and feeds the batch directly to parquet-rs. Both paths retain Flink's checkpoint-aware filesystem writer and each sink subtask produces its own part files. Set SF_PARQUET_OUTPUT to retain them at a chosen path; otherwise the harness prints its temporary output root. These results are intentionally separate from the headline Kafka table because they measure local Parquet IO rather than Kafka IO.

TZ=UTC SF_BENCHMARK=true SF_MATRIX_PARQUET_SINK=true SF_MATRIX_KAFKA=false \
  SF_MATRIX_GENERATOR=false SF_MATRIX_PARQUET=false SF_ROWS=2000000 \
  SF_PARALLELISM=4 SF_KAFKA_PARTITIONS=4 SF_WARMUP=1 SF_RUNS=3 \
  mvn -pl :streamfusion-runtime test -Pbench \
  -Dtest='NexmarkMatrixBenchmark#changelogParquetSinkComparison'

Add SF_MATRIX_QUERIES=q0,q4 for a focused run or set SF_PARQUET_OUTPUT to retain the part files instead of deleting the temporary output root.

Set SF_PROFILE_PARQUET_SINK=true and run changelogParquetSinkProfile for matched q0 CPU recordings of the Flink and StreamFusion writers. The harness performs an unprofiled warmup, then loops each writer for -Dprofile.seconds=20 by default and writes both JFR files below -Dprofile.outputDir=....

The Delta diagnostic compares the published Delta 4.4 connector with StreamFusion's data-file acceleration. Queries with an updating changelog use the result's real primary key and Delta 4.4 merge-on-read upserts; append-only queries use append mode. Delta Kernel owns table metadata, deletion vectors, statistics, actions, and commits on both sides. The harness pre-creates each table with deletion vectors enabled because delta.enableDeletionVectors is a Delta table property, not a SQL connector option accepted by the published connector. Set SF_DELTA_OUTPUT to retain the tables.

TZ=UTC SF_BENCHMARK=true SF_MATRIX_DELTA_SINK=true SF_ROWS=2000000 \
  SF_PARALLELISM=4 SF_KAFKA_PARTITIONS=4 SF_WARMUP=1 SF_RUNS=3 \
  mvn -Pdelta -pl :streamfusion-delta -am test -Pbench \
  -Duser.timezone=UTC -Dnative.build.skip=true \
  -Dtest='NexmarkDeltaSinkBenchmark#mergeOnReadUpsertComparison' \
  -Dsurefire.failIfNoSpecifiedTests=false

The explicit UTC setting is required for the timestamp-window queries: the native TIMESTAMP_LTZ window path accepts zones with a fixed offset for the full timestamp range, while a host-local DST zone is an intentional planner fallback.

The Paimon diagnostic compares the published Paimon 2.0.0 Flink 2.2 connector with StreamFusion's Paimon sink in three variants. The two append variants cover the 16 queries whose result is insert-only: every table is a Parquet append table that Paimon creates from the sink DDL; the bucket-unaware variant keeps Paimon's in-job compaction topology, and the fixed-bucket variant uses four buckets keyed on the result's first column. Both engines run Paimon's writer, committer, and manifests unchanged, so the row counts read back through Paimon's snapshots agree on every query except q12: that historical run used a native processing-time window that was incorrectly flushed at bounded-input completion. The engine now leaves unfinished processing-time windows open, including when it receives the terminal watermark; only the processing-time clock fires them. Per query the speed-up ranges from 1.08× (q3, a join that emits under a thousand rows) to 2.04× (q23, which writes 5.5 M joined rows); queries that write most of their input land at 1.4–1.8×.

The primary-key variant covers the seven updating queries (q4, q9, q15–q19) with a deduplicate primary-key table on the result's key, four fixed buckets, and Paimon's default in-job compaction; Flink's upsert materializer is disabled for both engines because Paimon refuses it. Stock Paimon takes the changelog one row at a time into its sort buffer, while StreamFusion buckets the Arrow batches natively, merges each bucket's changelog by key in Rust, and writes the level-0 files itself before Paimon's own writer compacts them. Per query the speed-up ranges from 1.18× (q9, whose changelog is dominated by updates of the same 120 K keys) to 2.38× (q16, eight keys rewritten a million times), and the merged row counts read back through Paimon agree on every query. Set SF_PAIMON_OUTPUT to retain the tables, one directory per variant.

TZ=UTC SF_BENCHMARK=true SF_MATRIX_PAIMON_SINK=true SF_ROWS=2000000 \
  SF_PARALLELISM=4 SF_KAFKA_PARTITIONS=4 SF_WARMUP=1 SF_RUNS=3 \
  mvn -Ppaimon -pl :streamfusion-paimon -am test -Pbench \
  -Duser.timezone=UTC -Dtest='NexmarkPaimonSinkBenchmark' \
  -Dsurefire.failIfNoSpecifiedTests=false

NexmarkPaimonSinkBenchmark runs all three variants; select unawareBucketAppendComparison, fixedBucketAppendComparison, or primaryKeyComparison for one of them.

To run the headline disk workload with only the sink changed to Paimon, set SF_LAKE_STATE_BACKEND=disk, SF_WARMUP=1, and SF_RUNS=2, and select NexmarkPaimonSinkBenchmark#unawareBucketAppendComparison+primaryKeyComparison. This covers all 23 executable headline queries with 2M Kafka JSON events, four partitions, parallelism four, and mini-batching disabled. Stock Flink uses RocksDB and StreamFusion uses native RocksDB; q4 asserts that both persistent backends engage during the measured jobs. The default lake-sink state backend remains memory.

For Flink 1.18.1, run with Java 17 and select the matching module. Set SF_BENCH_MANAGED_MEMORY to provision equal managed memory for both engines; the local Flink 1.18 cluster's small default can force RocksDB to flush tiny memtables continuously. With this override, the shared cluster's slot count matches SF_PARALLELISM (default four), so unused slots do not dilute the workload's managed-memory share.

TZ=UTC SF_BENCHMARK=true SF_MATRIX_PAIMON_SINK=true SF_LAKE_STATE_BACKEND=disk \
  SF_BENCH_MANAGED_MEMORY='512 mb' \
  SF_ROWS=2000000 SF_PARALLELISM=4 SF_KAFKA_PARTITIONS=4 SF_WARMUP=1 SF_RUNS=2 \
  mvn -Pflink-1.18,paimon,bench -pl :streamfusion-paimon-flink1.18 -am test \
  -Duser.timezone=UTC \
  -Dsf.extraJvmArgs=-Xmx3g \
  -Dtest='NexmarkPaimonSinkBenchmark#unawareBucketAppendComparison+primaryKeyComparison' \
  -Dsurefire.failIfNoSpecifiedTests=false

The test dependencies pin Testcontainers core to the same version as its Kafka and MinIO modules, including on the Flink 1.18 line, to prevent Flink's older transitive core from breaking startup.

The disk/Paimon matrix reads output counts after the final measured repetition. CSV count fields are blank for repetitions whose output was not counted; timings retain every warmup and measured repetition.

The example caps the shared test JVM's heap at 3 GiB for both engines. The default 8 GiB Surefire heap can exceed a small benchmark host's physical memory during state-heavy queries.

Unmodified Flink 1.18 cannot generate the mixed TIMESTAMP_LTZ(3)/TIMESTAMP(3) window-boundary comparison in q7. The installed planner supplies the timestamp comparison coercion used by newer released Flink versions, while retaining the fixed SQL and schema. Both benchmark engines use this compatibility step: the baseline disables native substitutions and executes Flink's rowwise operators. Accordingly, q7 compares against Flink 1.18 execution with that shared planner compatibility fix, rather than its unmodified failing compiler.

The Q7/Q12 verification on a WSL2 Linux x86-64 host (7.6 GiB visible RAM) used the configuration above. Q7 produced 4,000 rows per engine: measured Flink trials were 15.363/40.813 s and native trials 8.176/5.694 s (2.70× by the suite's best-of-two rule). Q12 produced zero unfinished-window rows per engine: 1.066/1.107 s Flink and 0.828/0.801 s native (1.33×). These are targeted verification results, not a replacement headline table; Q7 has substantial timing variability. All trials, including warmups are retained. Q23's selective disk-state implementation subsequently measured 11.339/13.016 s Flink and 4.055/4.084 s StreamFusion (2.80× best), with 5,520,000 rows per engine. The complete rerun below verifies a timing win for all 23 queries; Q3 remains a narrow win, checked in a separate five-run verification below. All Q23 trials include the original 12.709 s Flink / 22.114 s native best times and an idle rerun's 23.484 s / 21.838 s; native stayed around 22 s while Flink varied substantially. The idle rerun's 1.08× is insufficient evidence of a robust acceleration win.

The configuration above completed all 23 queries with native substitutions and matching final-repetition output counts. The geometric-mean speedup is 2.46× by the unchanged best-of-two rule. Q3 is the narrowest full-suite win (1.12× best, 1.09× by mean time). A separate verification with one warmup and five measured runs per engine, keeping all other settings unchanged, measured 0.939–1.350 s Flink and 0.786–0.929 s native: 1.20× best and 1.29× by mean time. All Q3 verification trials are retained. Its CPU profile points mainly to Kafka fetching and JSON decoding; join execution contributes a small share. These sub-second native timings leave startup and I/O as material parts of the end-to-end comparison. Flink Q7, Q8, Q18, and Q23 show large variability; all warmups and measured times are retained in the trial CSV. The Q7 baseline includes the shared compiler compatibility step described above.

Query Flink best (s) StreamFusion best (s) Speedup
q0 1.196 0.720 1.66×
q1 1.248 0.686 1.82×
q2 0.851 0.656 1.30×
q3 0.833 0.747 1.12×
q4 8.274 5.305 1.56×
q5 51.404 4.466 11.51×
q7 22.817 3.204 7.12×
q8 31.567 0.899 35.11×
q9 7.900 6.670 1.18×
q10 1.505 1.093 1.38×
q11 5.371 0.670 8.02×
q12 0.959 0.684 1.40×
q13 1.411 0.839 1.68×
q14 1.785 1.346 1.33×
q15 9.394 5.640 1.67×
q16 7.443 4.689 1.59×
q17 2.933 1.312 2.24×
q18 19.978 3.631 5.50×
q19 5.975 3.467 1.72×
q20 4.907 4.181 1.17×
q21 1.231 0.675 1.82×
q22 1.498 0.698 2.15×
q23 19.911 4.026 4.95×

Set SF_PROFILE_PAIMON_SINK=true and run unawareBucketAppendProfile for matched q0 CPU and wall-clock recordings of the stock and native paths. A word of caution that this profile taught us: stock Paimon's parquet-mr wraps its record consumer in per-value debug logging whenever its logger has DEBUG enabled, so a benchmark JVM with an unconfigured log4j 1.x binding makes the stock sink look 15–60× slower than it is. The module's test classpath excludes Hadoop's log4j 1.x bindings for exactly this reason; check the stock profile before trusting a sink number that far out of line with the rest of this table.

The September 30 Q23 controlled record-layout/cache experiment retains all cache policies, unfavorable results, and library hashes. It uses the existing shared-cluster 48 GiB task off-heap ceiling, equal for all policies, alongside 512 MiB managed memory and a 3 GiB heap. That ceiling is not an allocation and these timings do not establish performance under a 512 MiB total native pool.

Optional RocksDB io_uring isolation experiment

A generated-row/blackhole experiment at 2M events, parallelism 4, native RocksDB, mini batching off and one-second checkpoints compared the best of three trials with io_uring OFF and ON. The measured geometric-mean throughput change was +3.93%, median +0.80%; using median trial times reduced these to +1.08% and +0.59%. This is a different workload from the Kafka headline and does not establish a broad, repeatable speedup. Method, baseline disclosure, regressions and all individual trials document the optional integration.

Native operation profiling

Use the native Criterion suites to isolate Rust execution, allocation requests, and Arrow buffer sharing before optimizing an end-to-end regression. python3 bin/bench-native.py --list discovers every suite; --smoke checks fixtures in a release build. These measurements complement the Flink comparisons on this page.

The Fluss-to-Fluss suite uses the same queries and event corpus with released Fluss 1.0.0 as both input and output. The harness also supports the full state/mini-batch matrix via SF_FLUSS_STATE_BACKEND=memory|rocksdb and SF_FLUSS_MINI_BATCH=false|true: four input/output buckets, primary-key outputs only for updating queries, and identical two-second/50,000-row mini-batch settings on stock Flink and StreamFusion. Use Java 17 and the release bench profile. The 2026-10-08 four-mode results retain 90 valid cells and 540 measured durations, with one warmup and three trials per engine. Common-22-query median-speedup geomeans are 4.66× memory/off, 5.07× memory/on, 4.83× disk/off and 5.71× disk/on. All 23 queries pass with mini-batching off; both original on sweeps failed q23 with a native regular-join changelog correctness error. The bug is fixed: q23-only release reruns pass exact parity at 2M events, with 6.61× memory/on and 4.06× disk/on median speedups. The original failed cells and aggregates remain visible as pre-fix evidence; these reruns add 12 durations rather than replace the full matrix. All seven primary-key output queries pass in all modes. Memory/off q4 and memory/on q16 regress on their medians. Configuration, absolute times, variability and plans are retained; short-job lifecycle and host pressure limit sustained-throughput claims. This sweep uses exact SQL rather than Kafka's incompatible variants.

The earlier memory-state sweep of 16 append-only queries has a 2.02× geomean of median speedups on memory state over stock Fluss, with one warmup and three measured pairs each, including teardown. q2/q20/q23 have substantial variance and short-job lifecycle costs dominate many queries. Primary-key outputs retain the stock writer and have separate single-pair correctness validation. That sweep references the published Kafka README results. The subsequent matched local comparison reruns both transports with RocksDB and exact expressions. Primary-key outputs are excluded from transport comparisons. Full trials and profiles live on the connector page.

The earlier RocksDB Fluss matrix has a 2.34× append-only geomean over the same 16 queries, with one warmup and three measured pairs, mini-batching off and fixed 128 MiB state pools per slot. Both real backends are checked before timing, all 23 queries pass their output/native plan checks, and all disk trials are retained on the connector page.

The subsequent matched local Kafka/Fluss sweep uses 2M events, parallelism four, RocksDB, mini-batching off and exact expressions. Median-speedup geomeans over the 16 append-only queries are 8.56× for Fluss and 1.57× for Kafka; native Fluss is 2.12× faster geometrically in absolute runtime. All 192 measured samples are retained. Removing the SDK's two-second shutdown quiet period improves bounded-job lifecycle, rather than establishing a uniform sustained-throughput improvement. q23 remains slower than native Kafka on its median, and stateful outliers remain in the results. Delivery is exactly-once for Kafka and at-least-once for Fluss. The connector page contains the absolute per-query table, baseline sweep, controlled shutdown experiment and profiles.

Iceberg streaming comparison

The fresh Kafka-to-Iceberg RocksDB/off suite completes all 23 queries and 120 paired validations: repeated-median geometric means are 1.44× exact SQL and 1.47× with the README variants over stock Flink. It uses four Kafka partitions, four-bucket output tables, 512 MiB managed memory across four slots, one warmup and three alternating measured pairs. All regressions, ranges and configuration differences from historical runs are retained.

The separate Kafka JSON → Iceberg comparison uses the unchanged Kafka corpus and views with four topic partitions, parallelism four and four-bucket Iceberg output tables. Select the input with SF_ICEBERG_SOURCE=kafka; for the RocksDB, mini-batching-off suite run:

SF_ICEBERG_SOURCE=kafka SF_ICEBERG_MODES=disk-off \
  SF_BENCH_MANAGED_MEMORY='512 mb' \
  SF_ROWS=2000000 SF_PARALLELISM=4 SF_WARMUP=1 SF_RUNS=3 \
  bin/bench-iceberg.sh

Use JDK 17. This command gives both engines 512 MiB managed memory across four slots (128 MiB per slot), matching the historical disk/Paimon diagnostic. Both engines run in streaming mode with the same earliest-to-latest finite Kafka offsets, 8,192-record poll setting, UTC IANA session zone, one-second checkpoints and Iceberg writer settings. Timings include planning/submission, consumption, query/state/shuffle work and the final Iceberg commit; input production and committed-output readback are untimed. Updating queries use identifier-field upserts and Java delete encoding. Every native graph must have the native Kafka source, native Iceberg writer and no row/Arrow transposes; data-file footers must identify parquet-rs. q4 checks that each requested RocksDB backend actually opens. All warmups and alternating measured pairs are retained. Exact SQL and the existing labelled expression variants are reported separately. Processing-time q12 leaves unfinished windows open, as the historical Kafka suites do; its emitted per-bidder counts are checked against input bounds rather than requiring identical clock-dependent windows. The Iceberg-input comparison below has a different completion procedure.

For connector development, the fast reader/writer loop isolates production file tasks against stock Flink Iceberg in seconds. It complements the full streaming comparison below; it does not replace query, shuffle, checkpoint or commit measurements.

The latest four-partition Iceberg Nexmark matrix passes all 120 cells / 480 paired validations. Using the README query variants, geometric means of repeated median speedups are 1.22× / 1.81× on memory and 2.80× / 2.94× on RocksDB (mini-batching off/on). Exact-SQL ratios are 1.23× / 1.69× and 2.51× / 2.74×. The report retains every range, remaining state-heavy regressions and the mixed comparison with the previous complete matrix. The optional connector accelerates supported streaming contracts by default when installed; the Kafka headline is unchanged.

Paimon streaming comparison

The fresh Kafka→Paimon RocksDB/off comparison completes all 23 queries and 120 paired validations on this same host: 1.43× for exact SQL and the README selection. Exact append / primary-key geomeans are 1.24× / 1.97×, versus 1.45× / 1.40× for the matched Kafka→Iceberg setup. The whole-suite point estimates are almost identical (Iceberg exact: 1.44×). Paimon retains q5/q20/q23 losses and every wide timing range. The detailed report contains per-query comparisons, physical write-volume witnesses and an independent plan/table/footer audit. This fresh Flink 2.2 run does not reuse the historical Flink 1.18 best-of-two samples.

The matched Kafka→Paimon runner, bin/bench-paimon.sh, uses the unchanged Kafka JSON corpus and Paimon SQL fixtures with four Kafka partitions and four fixed output buckets. Run with JDK 17: bin/bench-paimon.sh. It builds and stages release/mimalloc libraries, defaults to 2M events, RocksDB, mini-batching off, 512 MiB managed memory across four slots and a 4 GiB job JVM heap. Each exact SQL cell and existing labelled expression variant gets one warmup and three measured pairs, alternating engine order. SF_ROWS, SF_WARMUP, SF_RUNS, SF_MATRIX_QUERIES, SF_PAIMON_VARIANTS=false, and SF_PAIMON_ARTIFACTS support smaller iteration runs and choosing the evidence directory. Production acceleration remains enabled by default.

The runner times streaming executeSql through final end-of-input commit; finite Kafka offsets bound the corpus. Corpus production, plan inspection and stock committed-table readback are untimed. Every deterministic pair must match full-value multiset fingerprints; processing-time q12 instead checks emitted per-bidder counts against input bounds. Native plans require native Kafka decode and Paimon routing without row/Arrow transposes, and nonempty native outputs must contain Rust Parquet files. q4 observes both real RocksDB backends. It retains all trial times, plans, sink DDL, file footers, table payloads/metadata and fingerprints. bin/report-paimon.py <artifacts> audits all pairs and regenerates the median/range report, metrics CSV and geometric means from the retained observations. Test logging uses WARN, matching the Iceberg comparison. Paimon's released default file settings (recorded from the released connector) and normal in-job compaction apply to both engines; these differ from the explicit Iceberg writer settings and delete files, so a cross-connector speedup gap is a diagnostic signal rather than proof of its cause. The original best-of Paimon runner remains available unchanged.

Historical Nexmark report review (2026-09-08)

The report submitted in #354 records canonical commit a9c6ebdc462100266c7b70b57a1fbbef1c5c4dbd, two million events, parallelism four, four Kafka partitions, release/mimalloc, one warmup and the best of two measurements. Its 23-query columns omit q6 under the existing harness policy. The PDF records all 92 cells, requires each column's stage validation before publishing its geometric mean, and carries the existing arithmetic/datetime/regex/case opt-in caveats.

An independent recomputation from the displayed rounded ratios gives geometric means 1.72017, 1.85400, 3.62848 and 4.10587 for memory off/on and disk off/on, matching the printed 1.72, 1.85, 3.63 and 4.11 after rounding. The transcribed ratios and recomputation make that arithmetic check reviewable. This verifies the report's summary arithmetic; its ratios alone cannot verify raw timing distributions, runtime operator consumption, backend assignment, identical inputs/resources or exact semantic parity.

Follow-up measurement should retain every warmup and repetition with durations, input/output counts, final validated backend/mode metadata, plans and actual native consumption. Use the unchanged queries and both transposes, run repeated stock/native trials in alternating order, and report medians and variability alongside any existing harness headline. Repeat the large disk gains with matched backend configuration, managed memory, cache settings and checkpoint frequency; profile state I/O and allocations. Repeat at a larger sustained workload and on current canonical main with the same build/resources. Keep exact-parity results separate from explicitly approximate expression variants. These checks require additional evidence beyond the historical PDF and do not establish a fresh current-main throughput claim.