Benchmark evidence index¶
Start with the benchmark method and headline results. This index connects complete comparisons, component measurements and retained investigations. Each report applies to its recorded revision, environment and measurement boundary; a historical result is not a measurement of the current main branch.
Choose the right document¶
| Question | Source |
|---|---|
| Does this query accelerate, and why does it fall back? | Operator coverage, connector coverage and configuration |
| How does a current optimization work? | Optimization ledger |
| How do I reproduce a native kernel measurement? | Criterion suites |
| How do I compare complete Flink jobs? | Benchmark methodology and performance audit |
| What did a particular SQL audit execute? | SQL invocation inventory and upstream-suite guide |
Coverage pages describe the current supported contract. Recorded timings, profiles and SQL inventories are evidence for their stated run, including its restrictions.
Complete comparisons¶
These reports retain the corresponding workload, completion protocol, all trial times and validation evidence. Compare within a report before comparing across hosts, connectors, backends, exact SQL and explicitly labelled expression variants.
| Workload | Report | Scope |
|---|---|---|
| Kafka Nexmark headline | Benchmark table and reproduction | Historical headline and its sampling convention |
| Iceberg to Iceberg | After boundary optimization | Complete repeated matrix, with previous-matrix comparison |
| Earlier Iceberg matrix | Before boundary optimization | Earlier complete cohort; its previous I/O comparison is separate from the stock-Flink matrix |
| Kafka to Iceberg | Repeated comparison | Separate q0/q19 CPU diagnostics are excluded from its aggregate |
| Kafka to Paimon | Repeated comparison | Includes a matched Iceberg comparison with connector-setting differences retained |
| Scalar functions | Recorded function workloads | Dated cohorts with full-job conversion costs and current coverage links |
| Basic ARRAY searches | Native search and decimal-entry comparison | Original/generated versus Rust kernels, large-decimal controls and remaining standalone losses; all 260 measured trials retained |
| Integer/DECIMAL ARRAY extrema | Released reduction kernels | Generated versus released DataFusion MIN/MAX with both transposes; 120 measured trials, including tied cases and wide-decimal standalone losses |
| Collection materialization and metadata | Two-line standalone comparison | Actual preceding StreamFusion and paired stock controls; shared map views, BIGINT entry capacity and scalar string exit, including remaining losses |
| Basic collection edits | Offset-based collection edits | Prior generated evaluator and stock Flink versus Rust, with both transposes and complete measured trials |
| Decimal expressions | Exact arithmetic comparison | Recorded expression workloads and argument/type restrictions |
| Other focused workloads | Append Top-1, basic types, floating GROUP BY, Flink 1.18 | Separate fixtures and environments; not interchangeable headline cells |
Component measurements and diagnostics¶
Certified scalar functions retains prepared Java/RowData versus Rust/Arrow measurements, whole-job previous/stock controls, all trials and execution limits.
Rust feature admission, 2026-10-19 starts the clock inside Rust on prepared Arrow and compares matching stock Flink Java/RowData execution. The earlier Java-wrapper audit retains its five-switch inventory, CPU profiles and trials as diagnostics; it includes JNI/C Data handoffs.
File-task loops and isolated routing measurements exclude parts of a streaming job. Profiled durations and one-pair preflights do not replace repeated, unprofiled complete-job observations. The linked reports explain which comparisons are valid.
| Investigation | Evidence and interpretation |
|---|---|
| Iceberg file-task development loop | Method, first boundary investigation and its retained cohorts |
| Iceberg selection ownership and dynamic routing | Second boundary investigation; includes an abandoned combined-size prototype and earlier generator references corrected in the checked comparison |
| Iceberg writer-only v2/v3 diagnostics | Format v2 and format v3; these are not Nexmark matrix cells |
| Iceberg state/checkpoint and I/O investigations | Profile guide and cohort reports; profiled, compressed and previous-I/O runs stay separate |
| Iceberg correctness validation | Recorded test inventory and reports; Criterion smoke checks fixtures, not throughput |
| Superseded Iceberg completion protocol | Interrupted canonical-export sweep; excluded from final aggregates |
Experimental and rejected approaches¶
These materials preserve the reasoning and reproduction inputs for experiments.
Their presence in the documentation does not enable a production path. Current
admission remains on the operator pages; rejected decisions also live in the
repository's .claude/wontdos/ directory.
| Experiment | Outcome and evidence |
|---|---|
| Hyperbolic compiler specialization | Prototype comparison: isolated kernel gains did not establish broad complete-job admission |
| Reserved primitive entry | Rejected entry-only experiment: mixed full-job changes did not resolve typed math regressions |
| Primitive and binary boundary experiments | Criterion diagnostics and scalar BINARY exit experiments retain their unfavorable results |
Maintaining the evidence¶
- Link a new report from its topic page or this index, and link each cohort's human-readable result from its parent report. Global navigation contains the topic entry points rather than every repetition.
- Record the revision and library hashes, released dependencies, host, resources, input, measured boundary, warmups, repetitions and engine order. State any difference between the stock and preceding-StreamFusion controls.
- Keep raw observations, ranges, native-execution witnesses, validation results and reproduction inputs alongside the conclusion. Retain regressions, failed attempts and superseded cohorts with an explicit interpretation.
- Keep profile runs, changed configurations and incomplete sweeps out of a primary aggregate unless its stated method includes them. Preserve the original cohort when a method or implementation changes.
- Before moving or deleting evidence, update document links and any verifier that reads its paths. Large artifacts need a durable versioned location, checksums and retrieval instructions before their repository copy is removed; expiring CI artifacts are not a sole archive.