Streams, tables, and joins
Joining two streams in time
How a key and a time interval match events from two streams, why either event may need to wait, and when an unmatched record can be reported.
Learn with an AI assistant
Copy the prompt and paste it into a chat with an AI assistant.
You are a tutor for the course "Stream Processing".Topic: "Joining two streams in time"In short: How a key and a time interval match events from two streams, why either event may need to wait, and when an unmatched record can be reported.Topic text in Markdown: https://stream-processing-systems.github.io/tables/joining-two-streams-in-time.mdBase your explanations on the topic text. First ask how I want to study: a short summary, a step-by-step walkthrough, or a deep dive with questions that check my understanding. Use the text's example and terms. If you add anything the text does not say, point it out.The team running our AI assistant wants to measure how long requests take. The gateway records when each request starts. The service that runs the model records when it completes, along with its token usage. The two kinds of events reach the processor through separate Kafka topics.
For each request completed in less than two minutes, the processor should produce one duration. Unlike looking up a plan, this means finding another event that may not have arrived yet. What connects the two events, and how long should the processor wait for a match?
Choose the join key
Section titled “Choose the join key”The account ID is enough to find a plan row, but an account can have several
requests running at once. On February 2, acme has these two:
| Event | Request ID | Event time | Arrival order at the processor |
|---|---|---|---|
| Start | r-201 | 14:19:50 | Third |
| Start | r-202 | 14:19:55 | First |
| Completion | r-202 | 14:20:00 | Fourth |
| Completion | r-201 | 14:20:10 | Second |
Joining on account alone would pair each completion with both starts. The
result would contain four durations, although there were only two requests.
The matching condition must use request_id. That gives 20 seconds for
r-201 and 5 seconds for r-202.
For this report, a request ID identifies one execution attempt, with one start and one successful completion. A retry of the model execution gets its own request ID. A repeated delivery of an existing event keeps its event ID and is recognized before the join. Without that rule, two copies of a completion could produce two joined rows. The matching condition finds every eligible pair; it does not enforce one pair per request.
Bound the time between events
Section titled “Bound the time between events”The report’s second condition is that completion happens at or after the start and strictly before two minutes have elapsed. Joining by a key together with a bounded time difference is called an interval join. Flink SQL supports this kind of condition over two streams of inserted rows.1
For a start at 14:19:50, the eligible completion times are in
[14:19:50, 14:21:50). Completion at 14:21:49 qualifies. Completion at exactly
14:21:50 does not. A completion timestamp earlier than the start also fails
the condition and needs investigation if it claims to describe the same
request.
The interval belongs to the pair. It does not require the events to fall in
the same fixed window.
r-201 crosses the boundary at 14:20, yet its duration is only 20 seconds.
Joining only events in the same minute would lose this match.
Two minutes is the scope of this report. A request completed after three minutes can still be a successful request; it is simply outside this result. The report cannot be used to count all successes or diagnose every failure.
Keep both sides while waiting
Section titled “Keep both sides while waiting”The processor receives the completion of r-201 before its start. It saves
the completion under r-201. When the start arrives, it checks the saved
completion, finds that the time condition holds, and emits a duration of
20 seconds. For r-202, the start arrives first and waits for the completion.
The join therefore keeps state on both sides: starts waiting for completions and completions waiting for starts. When a record arrives, the processor looks for matches on the other side. The two streams must bring events with the same request ID to the same joining processor, just as account keys keep usage state together. The account-based routing used for totals may need to change for this calculation.
The stored event times determine the duration. Arrival order only changes which event waits. If a delayed start makes the calculated duration longer, the calculation has accidentally used processing time for part of the measurement.
Use input progress to stop waiting
Section titled “Use input progress to stop waiting”Without a bound, a start could wait forever for its completion. The two-minute condition gives the join a last eligible event time. It still needs to know when the other input has progressed past that time.
For the start at 14:19:50, no completion at or after 14:21:50 can match. A watermark on the completion input reaching that boundary says that an eligible completion is no longer expected under that input’s progress rule. A saved completion has the reverse problem: it must wait for the start input to pass the latest start time that could match it. An engine may use a combined watermark from both inputs for cleanup, so a slow input can hold the join’s state back.
Waiting for two minutes on the processor’s clock would answer a different question. A start might have waited in Kafka for ten minutes before being read, while its completion is just behind it. Event-time progress lets the join find that pair during replay. It also lets a bounded join remove old records as the inputs advance.1
Progress is still an assumption about later arrivals. A matching event that arrives after the engine has discarded its counterpart needs the chosen late data policy. Unlike a window result, an interval join does not automatically inherit a window’s allowed-lateness setting or its correction behavior. The system must specify how it handles those arrivals and retain history if a separate repair calculation is needed.
Report an unmatched record carefully
Section titled “Report an unmatched record carefully”An inner join emits only matched pairs. That is enough for our duration report. A left outer join, with starts on the left, can also emit a start that has no eligible completion, leaving the completion fields empty.
An unmatched row needs a timing rule. Emitting it as soon as the start arrives would label nearly every request unmatched for a while. One implementation can emit a provisional unmatched row and replace it when a completion comes; another can wait until input progress closes the matching interval. A receiver must understand whichever changes the join emits.
Even after waiting, the row says “no eligible completion was observed under this report’s policy.” The request may have completed after two minutes, its completion record may be missing, or the model service may still be running it. To classify failure, the team needs an explicit failure or cancellation event, or a check of the service’s authoritative request state.
The processor now maintains several useful results: token totals, usage with a historical price, and matched request durations. Each starts with input records and changes some result rows. Describing every state update by hand is one way to build them. A query can instead describe the result that should be maintained as the inputs change.
What to remember
Section titled “What to remember”- Join on the identity of the relationship. An account ID is too broad for matching a request’s start and completion.
- An interval join bounds the time difference between events. The events can cross a fixed window’s boundary and arrive in either order.
- Both sides may need state. Progress and late data rules determine when it can be removed; a matching key alone does not prevent duplicate pairs.
- An unmatched row describes the join’s observations. It does not establish that the request failed.
References
Section titled “References”Further reading
Section titled “Further reading”- Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd ed. (O’Reilly Media, 2026), chap. 12, “Stream Processing.” The section on stream joins compares retaining events on both sides with enriching a stream from a changing table, and explains why the order of the two inputs can affect the result.
© 2026 Grig Alex · CC BY-NC-SA 4.0
