Skip to content

Aggregate specialization fast paths

Applies to: the two-phase local aggregate's numeric MIN/MAX, and mini-batch group-aggregate DISTINCT (q15/q16/q17-shaped queries)

Two of the local aggregate's hot leaves were paying for generality their actual input doesn't need: an insert-only MIN/MAX carrying full retraction support, and a DISTINCT accumulator boxing every probe value into a ScalarValue. Specializing each to what its input actually requires turned into two of the larger single-operator wins in the ledger.

Append-only local numeric MIN/MAX keeps one running extreme

The two-phase local aggregate had been giving every numeric MIN/MAX group a retractable BTreeMap<value, count>, even though the local half of an insert-only plan can only ever add values — it never needs to know what to fall back to when the current extreme is retracted. It now uses the existing scalar running MIN/MAX state when no row-kind column is present; retracting input, strings, decimals, and the global merge still retain the counted tree and its delete semantics, since those genuinely need multiset bookkeeping.

Criterion's 4096-row, 64-key MIN/MAX logical bundle rose from 9.50 to 33.89 M rows/s (3.57x, +258%). A contemporaneous release+mimalloc q17 mini-batch A/B rose from 1.535 to 1.661 M events/s (+8.2%); the immediate path, which does not use the local pre-aggregate, remained approximately flat at 1.750 versus 1.745 M events/s. The matched 25-second CPU profile completed 180 iterations versus 163 before and removed the local aggregate's 87-sample tree search, 68-sample tree destruction, and 37-sample aggregate-state destruction leaves; GroupAggState::accumulate fell from 55 to 31 samples. The few remaining tree samples come from the downstream global aggregate, whose input is retracting partial updates and so still needs the tree.

Group-aggregate DISTINCT folds primitives; the changelog emit reads its cache

The multi-DISTINCT day/channel GROUP BY aggregates (q15/q16/q17) owned the largest native islands, and their hot leaves were ScalarValue construct/hash/clone/drop: every row built a scalar per distinct agg call just to probe the distinct sets, and each emit materialized the group's full output tuple twice — the pre-update value for the changelog -U, the post-update value for the +U.

Distinct sets are now typed — a BIGINT distinct column keys a plain i64 map read straight off the array, no scalar involved — and each group caches its last-emitted tuple, so the pre-update value is a take-from-cache (recomputed only after restore) and the -U moves it out instead of cloning it. The emit protocol stays byte-identical, including the unchanged-result suppression Flink itself applies. Measured on the generator profile loop: q16 +17%, q17 +4%, q15 +3% — q16, long the floor of the Parquet/Kafka tables, gains the most. The cached tuple's own size accounting is covered on Memory accounting designed off the hot path.