Skip to content

Streams, tables, and joins

Joining two streams in time

How a key and a time interval match events from two streams, why either event may need to wait, and when an unmatched record can be reported.

Learn with an AI assistant

Copy the prompt and paste it into a chat with an AI assistant.

You are a tutor for the course "Stream Processing".
Topic: "Joining two streams in time"
In short: How a key and a time interval match events from two streams, why either event may need to wait, and when an unmatched record can be reported.
Topic text in Markdown: https://stream-processing-systems.github.io/tables/joining-two-streams-in-time.md
Base your explanations on the topic text. First ask how I want to study: a short summary, a step-by-step walkthrough, or a deep dive with questions that check my understanding. Use the text's example and terms. If you add anything the text does not say, point it out.

The team running our AI assistant wants to measure how long requests take. The gateway records when each request starts. The service that runs the model records when it completes, along with its token usage. The two kinds of events reach the processor through separate Kafka topics.

For each request completed in less than two minutes, the processor should produce one duration. Unlike looking up a plan, this means finding another event that may not have arrived yet. What connects the two events, and how long should the processor wait for a match?

The account ID is enough to find a plan row, but an account can have several requests running at once. On February 2, acme has these two:

EventRequest IDEvent timeArrival order at the processor
Startr-20114:19:50Third
Startr-20214:19:55First
Completionr-20214:20:00Fourth
Completionr-20114:20:10Second

Joining on account alone would pair each completion with both starts. The result would contain four durations, although there were only two requests. The matching condition must use request_id. That gives 20 seconds for r-201 and 5 seconds for r-202.

For this report, a request ID identifies one execution attempt, with one start and one successful completion. A retry of the model execution gets its own request ID. A repeated delivery of an existing event keeps its event ID and is recognized before the join. Without that rule, two copies of a completion could produce two joined rows. The matching condition finds every eligible pair; it does not enforce one pair per request.

The report’s second condition is that completion happens at or after the start and strictly before two minutes have elapsed. Joining by a key together with a bounded time difference is called an interval join. Flink SQL supports this kind of condition over two streams of inserted rows.1

For a start at 14:19:50, the eligible completion times are in [14:19:50, 14:21:50). Completion at 14:21:49 qualifies. Completion at exactly 14:21:50 does not. A completion timestamp earlier than the start also fails the condition and needs investigation if it claims to describe the same request.

The interval belongs to the pair. It does not require the events to fall in the same fixed window. r-201 crosses the boundary at 14:20, yet its duration is only 20 seconds. Joining only events in the same minute would lose this match.

Bound the time between events Request r-201 crosses the minute boundary at 14:20 and completes inside the interval from 14:19:50 inclusive to 14:21:50 exclusive. 14:20 Start r-201 · 14:19:50 Matching interval [14:19:50, 14:21:50) Completion r-201 · 14:20:10 14:19:50 14:21:50 EVENT TIME

Two minutes is the scope of this report. A request completed after three minutes can still be a successful request; it is simply outside this result. The report cannot be used to count all successes or diagnose every failure.

The processor receives the completion of r-201 before its start. It saves the completion under r-201. When the start arrives, it checks the saved completion, finds that the time condition holds, and emits a duration of 20 seconds. For r-202, the start arrives first and waits for the completion.

Keep both sides while waiting The completion of r-201 waits for its start between arrivals two and three, while the start of r-202 waits for its completion between arrivals one and four. r-201 waiting for start r-202 waiting for completion completion start start completion 1 2 3 4 ARRIVAL ORDER

The join therefore keeps state on both sides: starts waiting for completions and completions waiting for starts. When a record arrives, the processor looks for matches on the other side. The two streams must bring events with the same request ID to the same joining processor, just as account keys keep usage state together. The account-based routing used for totals may need to change for this calculation.

The stored event times determine the duration. Arrival order only changes which event waits. If a delayed start makes the calculated duration longer, the calculation has accidentally used processing time for part of the measurement.

Without a bound, a start could wait forever for its completion. The two-minute condition gives the join a last eligible event time. It still needs to know when the other input has progressed past that time.

For the start at 14:19:50, no completion at or after 14:21:50 can match. A watermark on the completion input reaching that boundary says that an eligible completion is no longer expected under that input’s progress rule. A saved completion has the reverse problem: it must wait for the start input to pass the latest start time that could match it. An engine may use a combined watermark from both inputs for cleanup, so a slow input can hold the join’s state back.

Waiting for two minutes on the processor’s clock would answer a different question. A start might have waited in Kafka for ten minutes before being read, while its completion is just behind it. Event-time progress lets the join find that pair during replay. It also lets a bounded join remove old records as the inputs advance.1

Progress is still an assumption about later arrivals. A matching event that arrives after the engine has discarded its counterpart needs the chosen late data policy. Unlike a window result, an interval join does not automatically inherit a window’s allowed-lateness setting or its correction behavior. The system must specify how it handles those arrivals and retain history if a separate repair calculation is needed.

An inner join emits only matched pairs. That is enough for our duration report. A left outer join, with starts on the left, can also emit a start that has no eligible completion, leaving the completion fields empty.

An unmatched row needs a timing rule. Emitting it as soon as the start arrives would label nearly every request unmatched for a while. One implementation can emit a provisional unmatched row and replace it when a completion comes; another can wait until input progress closes the matching interval. A receiver must understand whichever changes the join emits.

Even after waiting, the row says “no eligible completion was observed under this report’s policy.” The request may have completed after two minutes, its completion record may be missing, or the model service may still be running it. To classify failure, the team needs an explicit failure or cancellation event, or a check of the service’s authoritative request state.

The processor now maintains several useful results: token totals, usage with a historical price, and matched request durations. Each starts with input records and changes some result rows. Describing every state update by hand is one way to build them. A query can instead describe the result that should be maintained as the inputs change.

  • Join on the identity of the relationship. An account ID is too broad for matching a request’s start and completion.
  • An interval join bounds the time difference between events. The events can cross a fixed window’s boundary and arrive in either order.
  • Both sides may need state. Progress and late data rules determine when it can be removed; a matching key alone does not prevent duplicate pairs.
  • An unmatched row describes the join’s observations. It does not establish that the request failed.
  1. Apache Flink 2.3: Interval Joins.

  • Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd ed. (O’Reilly Media, 2026), chap. 12, “Stream Processing.” The section on stream joins compares retaining events on both sides with enriching a stream from a changing table, and explains why the order of the two inputs can affect the result.

© 2026 Grig Alex · CC BY-NC-SA 4.0