Deployment¶

StreamFusion currently supports exactly Flink 2.2.0 and 2.2.1. The loader fails closed for an unknown or unversioned planner ABI. Install the loader and core JARs into Flink's lib directory — never into a job JAR — and it accelerates ordinary streaming SQL jobs with no application-side NativePlanner.install(...) call and no query rewriting. Planning itself loads the native library (the planner compiles each Calc's expressions to verify their result types before admitting it), so the JARs must be present on the submission client or JobManager as well as the TaskManagers — which installing them into lib on one shared image already ensures.

Builds record the target Flink line and module identity in every payload's manifest. The loader checks its embedded core and installed StreamFusion JARs before creating the planner classloader, including renamed extensions. A different line or a missing marker fails loader initialization with an explicit compatibility error; rebuild or upgrade the loader, core and extensions together. The artifact coordinates for 2.2 remain unchanged. The flink-1.18 development profile produces separate streamfusion-*-flink1.18 coordinates and admits only Flink 1.18.1; release support remains gated by the outstanding validation in #182. See Flink line compatibility for build commands, dependency selections and known host differences. Builds and deployments support JDK 11 and JDK 17. Core and the existing connector/format artifacts target Java 11; the optional Iceberg adapter targets Java 17 and requires JDK 17 throughout its deployment. Use the same JDK version on the client, JobManager and TaskManagers. The Java compatibility matrix describes build checks and version-dependent runtime behavior.

Images built by bin/build-flink-image.sh perform the same checks before starting a JobManager, TaskManager, standalone application or history server. The entrypoint checks the host ABI, the loader's embedded core, and installed payload identities before handing control to Flink's original entrypoint. An incompatible image exits with a message naming the conflicting lines; it does not wait for a SQL query. The normal Flink configuration and command handling remain owned by the original entrypoint. Bare-metal installations and custom images retain the checks at loader initialization.

The image suite injects conflicting identities into packaged loader, core and renamed extension JARs on both supported build lines, and verifies nonzero exit before the JobManager starts. These are startup-admission checks, not cross-version state recovery tests. Cross-line savepoint upgrade and downgrade validation remain pending in #188; no upgrade direction is announced as supported yet. Testcontainers selects the container runtime from its normal configuration, including a configured Podman endpoint, without a hard-coded socket path.

Release artifacts are available from Maven Central and already contain the optimized native libraries. Fetch the loader and the separate runtime-visible core payload directly into a Flink distribution; installing StreamFusion does not require a source checkout, Rust, or a local build:

STREAMFUSION_VERSION=0.1.0-rc4
curl --fail --location \
  "https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-loader/$STREAMFUSION_VERSION/streamfusion-loader-$STREAMFUSION_VERSION.jar" \
  --output "$FLINK_HOME/lib/00-streamfusion-loader.jar"
curl --fail --location \
  "https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-core/$STREAMFUSION_VERSION/streamfusion-core-$STREAMFUSION_VERSION-runtime.jar" \
  --output "$FLINK_HOME/lib/streamfusion-core.jar"

Optional modules use their directory names as artifact IDs, for example tech.streamfusion:streamfusion-kafka:0.1.0-rc4 and tech.streamfusion:streamfusion-json:0.1.0-rc4. Install the matching stock Flink connector and format JARs alongside them as described below.

The GitHub release archive includes all optional StreamFusion modules, including Fluss, Delta and Paimon. Install only the modules your jobs use and their stock connector dependencies; the archive does not bundle the stock connectors.

Kubernetes or Docker¶

Generated Calc/filter bindings contain runtime-only functions; code generation remains in the planner process. TaskManagers do not need the planner or Calcite to restore or execute these bindings. An ExprCodeGenerator or CodeGeneratorContext linkage failure while restoring a factory through StreamConfig.getStreamOperatorFactory indicates a planner/runtime dependency leak. Upgrade matching compiler and runtime payloads and regenerate the job graph; adding the whole planner to worker job resources is not the fix. See the isolated-worker regression on the Calc/filter page.

Create a job-neutral image directly from the Maven Central artifacts:

ARG FLINK_IMAGE=flink:2.2.1-scala_2.12-java17
FROM ${FLINK_IMAGE}

ARG STREAMFUSION_VERSION=0.1.0-rc4
ADD https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-loader/${STREAMFUSION_VERSION}/streamfusion-loader-${STREAMFUSION_VERSION}.jar /opt/flink/lib/00-streamfusion-loader.jar
ADD https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-core/${STREAMFUSION_VERSION}/streamfusion-core-${STREAMFUSION_VERSION}-runtime.jar /opt/flink/lib/streamfusion-core.jar

ENV GLIBC_TUNABLES=glibc.rtld.optional_static_tls=131072 \
    ROCKSDB_MUSL_LIBC=false

Use that image as spec.image in a Flink Kubernetes Operator FlinkDeployment, or as kubernetes.container.image.ref for Flink's native Kubernetes deployment. It works for either mode:

Session — run the JobManager, TaskManagers, and the SQL/client process from the StreamFusion image; submit job JARs through your normal REST, SQL Gateway, or FlinkSessionJob path.
Application — derive a job image from the StreamFusion base image, place the job JAR in /opt/flink/usrlib, and use that image in the Application deployment. Remote job-artifact delivery remains supported too.

Build this image for linux/amd64. The current runner-built release JARs contain Linux x86_64 and macOS Apple Silicon payloads; Linux ARM64 is not part of the published binary set yet.

Layering connectors and formats¶

The optional Doris adapter builds with the doris Maven profile. Install streamfusion-doris and the released flink-doris-connector-2.2-26.3.0.jar with the loader and core. It requires JDK 17 because the stock Doris connector targets Java 17; the adapter itself targets Java 11. The Flink 2.2 release archive includes this Java-only adapter; it adds no connector DSO or separate format JAR. The base image does not include it. Stock Arrow sink fallback also requires the matching Flink flink-python JAR from opt/ at runtime.

The base image contains the loader and self-contained core runtime JAR and remains connector- and format-neutral: every optional connector or format is its own streamfusion-* artifact, matching Flink's own connector/format module split. Derive a small image and install Flink's connector and format JARs, the matching StreamFusion connector JAR, and only the StreamFusion format JARs your jobs actually use into /opt/flink/lib — use that same image for the JobManager, TaskManagers, and submission client. For example, JSON on Kafka needs four JARs:

FROM registry.example/streamfusion-flink:dev
ARG STREAMFUSION_VERSION=0.1.0-rc4
ADD https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-kafka/${STREAMFUSION_VERSION}/streamfusion-kafka-${STREAMFUSION_VERSION}.jar /opt/flink/lib/streamfusion-kafka.jar
ADD https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-json/${STREAMFUSION_VERSION}/streamfusion-json-${STREAMFUSION_VERSION}.jar /opt/flink/lib/streamfusion-json.jar
COPY flink-connector-kafka-5.0.0-2.2.jar flink-json-2.2.1.jar /opt/flink/lib/

Replace streamfusion-json with streamfusion-csv, streamfusion-raw, streamfusion-avro, or streamfusion-protobuf and add Flink's like-named format JAR — see Connectors for the full per-format breakdown. avro-confluent uses both streamfusion-avro (the shared native Avro codec) and streamfusion-avro-confluent-registry with Flink's flink-avro-confluent-registry. Use flink-parquet with streamfusion-parquet, or flink-orc with streamfusion-orc, the same way. Paimon needs paimon-flink-2.2-2.0.0.jar, the selected streamfusion-parquet or streamfusion-orc format module, and streamfusion-paimon installed as 01-streamfusion-paimon.jar — Paimon takes the first matching format factory it finds and Flink loads lib/ in sorted name order, so the StreamFusion JAR must sort before paimon-flink-*; see Apache Paimon. A missing optional module is always a normal planner fallback to stock Flink, never a linkage failure — the core image doesn't require any of them. Iceberg on Flink 2.2/JDK 17 uses the released iceberg-flink-runtime-2.2 JAR plus streamfusion-iceberg; its data encoder and reader live in that module's own native library. Rewrite maintenance also links DataFusion’s physical external sort and its utility dependencies into this library; the SQL planner, session and StreamFusion engine remain separate. It does not require streamfusion-parquet. See Iceberg.

Bare metal¶

For a local Flink distribution instead:

STREAMFUSION_VERSION=0.1.0-rc4
curl --fail --location \
  "https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-loader/$STREAMFUSION_VERSION/streamfusion-loader-$STREAMFUSION_VERSION.jar" \
  --output "$FLINK_HOME/lib/00-streamfusion-loader.jar"
curl --fail --location \
  "https://repo1.maven.org/maven2/tech/streamfusion/streamfusion-core/$STREAMFUSION_VERSION/streamfusion-core-$STREAMFUSION_VERSION-runtime.jar" \
  --output "$FLINK_HOME/lib/streamfusion-core.jar"

Restart Flink after installation, then submit ordinary streaming SQL jobs as usual.

Contributing from source¶

For local development, mvn compile is Java-only and does not invoke Cargo. mvn test builds the host debug native libraries from the Cargo workspace in native/. Each extension Maven module builds its matching Cargo package; the runtime test assembly builds the workspace. Tests load libstreamfusion and the individual libstreamfusion_<extension> libraries from native/target/debug, just as a deployment loads each library from its owning JAR. A missing local extension library cannot bind its methods to the engine library.

Debug builds are roughly an order of magnitude slower than release, so use mvn test -Pbench for benchmarks. Build the portable optimized artifacts when developing or preparing a release:

bin/build-release.sh

The Flink 2.2 release reactor and deployment archive include the optional Iceberg JAR and its platform-specific native payload. Host, macOS and container staging, artifact isolation checks and Maven publication all include it. Flink 1.18 builds omit Iceberg because the released adapter targets 2.2/JDK 17. The bytecode check retains the Java 11 limit for core and other modules, with a Java 17 limit only for Iceberg. Ordinary PR CI runs the Iceberg/upstream streaming selection on JDK 17; the native workspace check also verifies Iceberg's package and JNI export ownership.

The release build enables mimalloc by default. Its thread-local allocator state uses dynamic TLS because the JVM loads JNI libraries after startup, potentially through more than one classloader. Linux artifact validation rejects libraries that require a reserved static TLS block; deployment does not require preloading libraries or changing glibc tunables.

The ORC module builds its released orc-rust reader through the normal Cargo/Maven lifecycle on macOS and Linux. Writing uses the Java ORC library supplied by Flink or Paimon, with a shared Arrow-to-Hive-vector converter. The ORC C++ adapter, CMake bootstrap and bundled C++ codec archives have been removed; no ORC/Arrow C++ installation is needed. Other modules retain their normal native build requirements. See ORC for deployment and configuration details.

Native workspace¶

Directory	Responsibility
`native/engine`	Operators, planner bridge, Rust state, and the core `Native` JNI entry points; produces `libstreamfusion`.
`native/bridge`	JNI guards, Arrow C Data import/export, handle accounting, Flink numeric/text semantics, and format ABI types. No JNI exports of its own.
`native/format-support`	Shared decoder lifecycle, parse-error isolation, key/value composition, CDC gathering, and format facade macros. No engine or third-party format implementation.
`native/kafka`	Kafka-specific JNI entry points, source implementation and existing sink encoders.
`native/fluss`	Fluss compacted bucket-key encoding, bucket grouping and Arrow batch statistics, bundled only in `streamfusion-fluss`. Connection management and statistics wire serialization remain in the Java SDK.
`native/parquet`	Parquet format encoding and decoding, including reads through host FileIO.
`native/orc`	orc-rust decoding, with host-owned I/O and Arrow C Data. Java ORC writing lives in the format/connector JARs.
`native/iceberg`	Optional Iceberg task decode, Parquet data-file encode and external rewrite sort; Java retains file/table lifecycle and delete encoding.
`native/paimon`	Released paimon-rust source decoding and supported snapshot merging over Java FileIO, bundled in either Paimon adapter. Upstream includes its formats in this DSO; writer codecs remain separate.
`native/json`, `native/csv`, `native/raw`, `native/avro`, `native/protobuf`	One decoder library per format JAR. Avro and Avro-Confluent-Registry continue to share the Avro native library.
`native/native-build`	Shared build dependency for library-local mimalloc aliases and the checked free/realloc shim.
`native/integration-tests`	Rust round trips that exercise both connector encoding and format decoding.

Shared crates are statically linked into each library. Handles, JVM references, and allocator state stay local to their owning library; Arrow release callbacks preserve ownership across the C Data boundary. The Java leak sentinel queries every loaded library's handle registry. The engine enables rocksdb-state by default; cargo build -p streamfusion --no-default-features builds an engine without persistent native state. No connector or format feature selects JNI exports anymore.

cd native
cargo test --workspace
cargo build -p streamfusion-json
cargo bench -p streamfusion --bench operators
cargo bench -p streamfusion-kafka --bench kafka_sink
cargo bench -p streamfusion-json --bench json_codecs

From the repository root, python3 bin/check-native-workspace.py --libraries native/target/debug checks dependency isolation and the built libraries' JNI exports. native.cargo.args controls the Cargo command/profile, and native.cargo.packages controls Maven's package selection.

The split removes engine dependencies from standalone extension builds: Parquet's normal dependency graph contains 79 packages instead of 253. Release validation did not reproduce the historical binary-size reduction proposed in issue #51; the previous export gating already let the linker discard unused engine code.

Deployment JVM flags¶

Run the TaskManager JVM with Arrow's safety checks off, as Comet/Spark do — profiling showed roughly a third of the transpose CPU was per-accessor bounds/refcount checks:

-Darrow.enable_unsafe_memory_access=true -Darrow.enable_null_check_for_get=false

See Configuration for the full -Dstreamfusion.* runtime flag surface, including off-heap sizing for Arrow batches and native operator state.

Experimental Flink 1.18 images¶

The development image builder selects the matching official Flink base and qualified payloads:

bin/build-flink-image.sh --flink-line 1.18 --tag streamfusion-flink:1.18-dev --load --platform linux/amd64
mvn -Pimage-it,flink-1.18 -pl streamfusion-image-it verify \
  -Dstreamfusion.image.name=streamfusion-flink:1.18-dev

For an unpacked official distribution, use bin/install-flink.sh --flink-line 1.18 <FLINK_HOME>. The installer checks the supported distribution version and both payload manifests before copying either JAR. Its default remains the 2.2 line.

Both CI image jobs build optimized native libraries and submit a normal thin user JAR to a Session cluster. They verify native Calc and grouped aggregation with the native RocksDB backend, and load each packaged format/connector extension in its own JVM. Flink 1.18 uses the state.backend configuration key; 2.2 uses state.backend.type. The 1.18 line remains experimental: these smoke jobs do not establish cross-line savepoint upgrade support, which is tracked in #188.

Host logging ownership¶

Flink owns the process logging API and binding. StreamFusion compiles against Flink's SLF4J 1.7.36 API as a provided dependency and does not package SLF4J classes in its deployment JARs. This prevents Arrow's transitive SLF4J 2 API from shadowing Flink's binding and silently selecting a no-operation logger. Artifact validation rejects bundled logging API classes, and the image SQL smoke test requires a real logging provider before running native SQL.

Connector build classpath ownership¶

The released Paimon 2.0 connector bundles compatibility copies of Flink RowData and ArrayData, including the newer variant methods even in its Flink 1.18 artifact. These classes must not select the API used to compile StreamFusion's shared source. The Paimon module declares the target host's flink-table-common directly before the connector bundle; normal dependency version mediation alone cannot detect duplicate classes inside that bundle. Maven's dependency ordering can expose this difference, so reusing a successful local cache is not evidence that a fresh image reactor will select the same classes.

python3 bin/test_paimon_host_classpath.py resolves the production dependency declarations against canonical released artifacts on both supported Flink lines. It checks the loaded RowData/ArrayData origins and variant APIs and compiles the actual shared PrunedRowData with each compatibility adapter. Its negative control puts the fat connector first and reproduces the Flink 1.18 getVariant(int) compilation failure. Image CI runs this lightweight Java-only regression and records the Maven version before the native image build.

Experimental Fluss connector¶

Source builds and the release archive include the optional streamfusion-fluss extension. Install it alongside the matching core payload and Apache Fluss's released org.apache.fluss:fluss-flink-2.2:1.0.0 JAR. It reuses the core Arrow bridge and bundles libstreamfusion_fluss for columnar bucket-key hashing and log-batch statistics. Verified planning substitutions are enabled by default when the module is installed. Set -Dstreamfusion.fluss.enabled=false to retain the stock endpoints; see the experimental coverage described in Fluss. Primary-key writes remain on Flink's normal Fluss connector.

Shared source ownership¶

Shared Java sources also require explicit module ownership during compilation. Reproducible JAR timestamps can precede a fresh checkout's source timestamps; javac then prefers the newer source of a referenced core class and can implicitly emit that class into a connector's output. The compiler configuration inherited by every module uses -implicit:none: javac may resolve shared source symbols, but emits classes only for the module's explicit source includes. This keeps the core runtime and optional connector/format JARs separate without hiding duplicate classes from artifact validation. Clean rebuilds remove classes emitted by an older configuration.

python3 bin/test_module_source_ownership.py recreates an old-timestamp dependency JAR and newer shared source. Its negative control demonstrates implicit duplication; the configured compiler flag preserves only the connector's explicit class. A runtime probe executes the connector and verifies its dependency resolves from the core JAR. Image CI runs the regression before packaging, and the unchanged artifact checker verifies ownership in the real bundle.