Skip to content

Streams, tables, and joins

Joining events with changing reference data

How a join attaches a plan to a usage event, why current and historical prices give different answers, and how time, version history, and input progress determine the result.

Learn with an AI assistant

Copy the prompt and paste it into a chat with an AI assistant.

You are a tutor for the course "Stream Processing".
Topic: "Joining events with changing reference data"
In short: How a join attaches a plan to a usage event, why current and historical prices give different answers, and how time, version history, and input progress determine the result.
Topic text in Markdown: https://stream-processing-systems.github.io/tables/joining-events-with-changing-reference-data.md
Base your explanations on the topic text. First ask how I want to study: a short summary, a step-by-step walkthrough, or a deep dive with questions that check my understanding. Use the text's example and terms. If you add anything the text does not say, point it out.

At 14:20 on February 2, acme changes from team to business in our plans database. Its price drops from 0.40 to 0.30 per thousand tokens. The usage processor keeps a current copy of the table, updated by CDC, and uses it to estimate the cost of requests for the usage page.

A request completes at 14:19:58 and uses 1,000 tokens. Its usage record arrives at 14:20:03, after the processor has applied the new plan. The account ID finds acme’s row, but the row now says 0.30. Which price should the processor use?

A join combines related records using a matching condition. Here the condition is that a usage record’s account_id equals the plan row’s account_id. The processor attaches the plan and price to the usage record and calculates tokens / 1,000 × price_per_1k_tokens.

The plan table has at most one current row for each account, so a lookup finds at most one price. If no row is present, the processor has no price to apply. The row might still be on its way from the snapshot, or the account might have no plan. Treating that absence as a zero price would silently make the request free. For our usage estimate, the processor holds the record for retry and reports an unresolved price if the row still cannot be found.

Looking up the row for each usage record is called enrichment: the record gains context from reference data. Keeping a local copy avoids asking the plans database for every request. That copy, however, has to keep receiving the changes captured from the database.

Suppose the next request completes at 14:20:02 and also uses 1,000 tokens. Both records reach the processor after it applies the new plan:

Request completion timePrice currently in the local tablePrice effective at completion
14:19:580.300.40
14:20:020.300.30
Estimated cost of both requests0.600.70

The company wants to estimate usage under the price effective when each request completed. In this example, the plan change takes effect when it is committed at 14:20:00, including requests completed at exactly that time. The first request therefore uses the old price. A report asking what the same usage would cost at the current price would use 0.30 for both requests.

With this lookup, the processor takes the price from the current plan row in its local table. Once a record has been enriched, a later plan change does not by itself cause that lookup to run again. Replaying old usage against the latest plan can consequently produce a different result. If every past request must keep the price it originally selected, the system needs that decision or the history needed to reproduce it.

An event-time temporal join finds the reference row valid at an event’s time. A versioned table keeps earlier values for a key as well as the times at which they changed. Flink supports this kind of join over a versioned table.1

For acme, the price history is:

Valid fromValid until, exclusivePrice per thousand tokens
Before these requests14:20:000.40
14:20:00Until the next price change0.30

The request at 14:19:58 selects the first interval even if its record arrives after 14:20. The request at 14:20:02 selects the second. The limit correction at 14:22 produces another version of the plan row, with the same price.

Keep the versions that events need The request completed while the price was 0.40 and arrived after the change to 0.30; the temporal join selects the version at completion. 14:20:00 Plan price per 1,000 tokens 0.40 0.30 Request completion 1,000 tokens 14:19:58 Record arrival the same request 14:20:03 TIME

The version time must match the business rule. We chose database commit time as the effective time. A scheduled change taking effect at midnight needs an explicit effective time; its CDC arrival time would tell us something else. A rule that fixes the price when a request starts needs the start time, or the selected price version, in the input. The join cannot infer this policy from timestamps alone.

History also has to survive long enough. A compacted plans topic can restore the latest row but may have discarded the 0.40 version. A new processor cannot reconstruct February’s prices from the current rows in March. Historical replay needs retained versions covering that period, or usage records that already carry the selected version and price.

Keeping history solves a delayed usage record only if the needed plan changes are available too. Suppose the usage at 14:20:02 reaches the processor before the plan change. The only price it has seen is 0.40. An immediate lookup would select it, even though the request should use 0.30.

The processor needs to know how far both inputs have progressed. The usage input’s watermark says nothing about a plan change waiting in the CDC path. Flink’s event-time temporal join waits on watermarks from both sides to select the relevant reference version.2 Waiting allows an out-of-order plan change to reach the join before it resolves the usage record. A stalled plan input can delay the answer even while usage keeps arriving.

A plan change that arrives behind the progress already declared can violate the join’s assumptions. Handling it requires an explicit correction path or recalculation; we cannot assume that an event-time join revisits every old result. Once version history has been removed from state, keeping the latest row cannot recover it.

There is a third useful question: keep a report of all represented usage at the current price. When the price changes to 0.30, even the earlier 1,000 tokens should be revalued. A maintained join between changing tables does this: changes on either side affect the joined result.3 The processor must retain enough information about usage to update the affected rows.

CalculationPrice selectedEffect of a later plan change
Enrich each record using a current lookupThe row available at lookup timeFuture lookups see it; past outputs stay as they were
Event-time temporal joinThe version valid at the request’s completion timeA later effective price does not reprice earlier requests
Maintained join at current pricesThe current plan rowAlready represented usage is revalued too

Our live cost estimate uses the second rule. It can store the chosen version and amount with each request so a correction is a deliberate change. This still leaves the external billing effect to be handled safely; choosing the right price does not make a write idempotent.

The completed request gave the processor both its token count and completion time. To measure how long it ran, the processor also needs its start. Starts and completions travel on separate paths, and either can arrive first. There is no current plan row to look up in that problem: the matching event may still be on its way.

  • A matching key finds related data. The calculation must also specify which version it needs.
  • Current lookup, event-time temporal join, and a maintained join at current prices answer different questions.
  • Historical selection needs meaningful effective times, retained versions, and progress from both inputs.
  • A late reference change needs a correction policy. The latest row alone cannot reproduce a historical price decision.
  1. Apache Flink 2.3: Versioned Tables.

  2. Apache Flink 2.3: Event Time Temporal Join.

  3. Apache Flink 2.3: Regular Joins.

  • Tyler Akidau, Slava Chernyak, and Reuven Lax, Streaming Systems (O’Reilly Media, 2018), chap. 9, “Streaming Joins.” The section on temporal validity explains how reference values define intervals and how an out-of-order update changes those intervals.

© 2026 Grig Alex · CC BY-NC-SA 4.0