Streams, tables, and joins
Joining events with changing reference data
How a join attaches a plan to a usage event, why current and historical prices give different answers, and how time, version history, and input progress determine the result.
Learn with an AI assistant
Copy the prompt and paste it into a chat with an AI assistant.
You are a tutor for the course "Stream Processing".Topic: "Joining events with changing reference data"In short: How a join attaches a plan to a usage event, why current and historical prices give different answers, and how time, version history, and input progress determine the result.Topic text in Markdown: https://stream-processing-systems.github.io/tables/joining-events-with-changing-reference-data.mdBase your explanations on the topic text. First ask how I want to study: a short summary, a step-by-step walkthrough, or a deep dive with questions that check my understanding. Use the text's example and terms. If you add anything the text does not say, point it out.At 14:20 on February 2, acme changes from team to business in our
plans database.
Its price drops from 0.40 to 0.30 per thousand tokens. The usage processor
keeps a current copy of the table,
updated by CDC, and uses it to estimate the cost of requests for the usage
page.
A request completes at 14:19:58 and uses 1,000 tokens. Its usage record
arrives at 14:20:03, after the processor has applied the new plan. The account
ID finds acme’s row, but the row now says 0.30. Which price should the
processor use?
The key finds the row
Section titled “The key finds the row”A join combines related records using a matching condition. Here the
condition is that a usage record’s account_id equals the plan row’s
account_id. The processor attaches the plan and price to the usage record
and calculates tokens / 1,000 × price_per_1k_tokens.
The plan table has at most one current row for each account, so a lookup finds at most one price. If no row is present, the processor has no price to apply. The row might still be on its way from the snapshot, or the account might have no plan. Treating that absence as a zero price would silently make the request free. For our usage estimate, the processor holds the record for retry and reports an unresolved price if the row still cannot be found.
Looking up the row for each usage record is called enrichment: the record gains context from reference data. Keeping a local copy avoids asking the plans database for every request. That copy, however, has to keep receiving the changes captured from the database.
Current lookup and historical price
Section titled “Current lookup and historical price”Suppose the next request completes at 14:20:02 and also uses 1,000 tokens. Both records reach the processor after it applies the new plan:
| Request completion time | Price currently in the local table | Price effective at completion |
|---|---|---|
| 14:19:58 | 0.30 | 0.40 |
| 14:20:02 | 0.30 | 0.30 |
| Estimated cost of both requests | 0.60 | 0.70 |
The company wants to estimate usage under the price effective when each request completed. In this example, the plan change takes effect when it is committed at 14:20:00, including requests completed at exactly that time. The first request therefore uses the old price. A report asking what the same usage would cost at the current price would use 0.30 for both requests.
With this lookup, the processor takes the price from the current plan row in its local table. Once a record has been enriched, a later plan change does not by itself cause that lookup to run again. Replaying old usage against the latest plan can consequently produce a different result. If every past request must keep the price it originally selected, the system needs that decision or the history needed to reproduce it.
Keep the versions that events need
Section titled “Keep the versions that events need”An event-time temporal join finds the reference row valid at an event’s time. A versioned table keeps earlier values for a key as well as the times at which they changed. Flink supports this kind of join over a versioned table.1
For acme, the price history is:
| Valid from | Valid until, exclusive | Price per thousand tokens |
|---|---|---|
| Before these requests | 14:20:00 | 0.40 |
| 14:20:00 | Until the next price change | 0.30 |
The request at 14:19:58 selects the first interval even if its record arrives after 14:20. The request at 14:20:02 selects the second. The limit correction at 14:22 produces another version of the plan row, with the same price.
The version time must match the business rule. We chose database commit time as the effective time. A scheduled change taking effect at midnight needs an explicit effective time; its CDC arrival time would tell us something else. A rule that fixes the price when a request starts needs the start time, or the selected price version, in the input. The join cannot infer this policy from timestamps alone.
History also has to survive long enough. A compacted plans topic can restore the latest row but may have discarded the 0.40 version. A new processor cannot reconstruct February’s prices from the current rows in March. Historical replay needs retained versions covering that period, or usage records that already carry the selected version and price.
Wait for both inputs
Section titled “Wait for both inputs”Keeping history solves a delayed usage record only if the needed plan changes are available too. Suppose the usage at 14:20:02 reaches the processor before the plan change. The only price it has seen is 0.40. An immediate lookup would select it, even though the request should use 0.30.
The processor needs to know how far both inputs have progressed. The usage input’s watermark says nothing about a plan change waiting in the CDC path. Flink’s event-time temporal join waits on watermarks from both sides to select the relevant reference version.2 Waiting allows an out-of-order plan change to reach the join before it resolves the usage record. A stalled plan input can delay the answer even while usage keeps arriving.
A plan change that arrives behind the progress already declared can violate the join’s assumptions. Handling it requires an explicit correction path or recalculation; we cannot assume that an event-time join revisits every old result. Once version history has been removed from state, keeping the latest row cannot recover it.
Which changes revise the result?
Section titled “Which changes revise the result?”There is a third useful question: keep a report of all represented usage at the current price. When the price changes to 0.30, even the earlier 1,000 tokens should be revalued. A maintained join between changing tables does this: changes on either side affect the joined result.3 The processor must retain enough information about usage to update the affected rows.
| Calculation | Price selected | Effect of a later plan change |
|---|---|---|
| Enrich each record using a current lookup | The row available at lookup time | Future lookups see it; past outputs stay as they were |
| Event-time temporal join | The version valid at the request’s completion time | A later effective price does not reprice earlier requests |
| Maintained join at current prices | The current plan row | Already represented usage is revalued too |
Our live cost estimate uses the second rule. It can store the chosen version and amount with each request so a correction is a deliberate change. This still leaves the external billing effect to be handled safely; choosing the right price does not make a write idempotent.
The completed request gave the processor both its token count and completion time. To measure how long it ran, the processor also needs its start. Starts and completions travel on separate paths, and either can arrive first. There is no current plan row to look up in that problem: the matching event may still be on its way.
What to remember
Section titled “What to remember”- A matching key finds related data. The calculation must also specify which version it needs.
- Current lookup, event-time temporal join, and a maintained join at current prices answer different questions.
- Historical selection needs meaningful effective times, retained versions, and progress from both inputs.
- A late reference change needs a correction policy. The latest row alone cannot reproduce a historical price decision.
References
Section titled “References”Further reading
Section titled “Further reading”- Tyler Akidau, Slava Chernyak, and Reuven Lax, Streaming Systems (O’Reilly Media, 2018), chap. 9, “Streaming Joins.” The section on temporal validity explains how reference values define intervals and how an out-of-order update changes those intervals.
© 2026 Grig Alex · CC BY-NC-SA 4.0
