Streams, tables, and joins
From changes to a current table
How event contributions and row replacements build different tables, how a changelog maintains a materialized view, and what is needed to rebuild it.
Learn with an AI assistant
Copy the prompt and paste it into a chat with an AI assistant.
You are a tutor for the course "Stream Processing".Topic: "From changes to a current table"In short: How event contributions and row replacements build different tables, how a changelog maintains a materialized view, and what is needed to rebuild it.Topic text in Markdown: https://stream-processing-systems.github.io/tables/from-changes-to-a-current-table.mdBase your explanations on the topic text. First ask how I want to study: a short summary, a step-by-step walkthrough, or a deep dive with questions that check my understanding. Use the text's example and terms. If you add anything the text does not say, point it out.The usage processor of our AI assistant sends the usage page a new total
whenever it counts a request. For acme, the first four totals are 1,000,
1,400, 2,000, and 2,500 tokens. The page should show 2,500. Adding the four
numbers would give 6,900, far more than the account used.
The processor also receives changes from the plans database.
When acme moves to a new plan, the new row replaces the old one. Both paths
carry records with an account and some numbers. How does a reader know what
to add, what to replace, and what to remove?
Contributions and replacements
Section titled “Contributions and replacements”The gateway’s usage records describe separate requests. After recognizing repeated deliveries, the processor counts each request once:
| Usage record | Tokens in this request | Total for acme |
|---|---|---|
u-101 | 1,000 | 1,000 |
u-102 | 400 | 1,400 |
u-103 | 600 | 2,000 |
u-104 | 500 | 2,500 |
Each number in the middle column contributes to the total. Each number in the last column replaces the previous total. A reader that adds the last column counts the first request four times, the second three times, and so on.
The usage page keeps a table with one row per account. Its key is
account_id, and the value is the latest total received for that account.
When the processor sends acme, 2,500, the page replaces acme, 2,000.
beta has its own row, which stays unchanged.
These two interpretations are made explicit in Kafka Streams, the processing
library for Kafka: a KStream represents independent records, while a
KTable represents changes to current rows. An aggregation over a stream
produces a table because each new aggregate replaces an earlier value for
its key.1
A changelog changes one row at a time
Section titled “A changelog changes one row at a time”A stream of records describing inserts, updates, and deletes in a table is
called a changelog. Our CDC records
form a changelog of plans. The receiving processor uses the key and the
kind of change to maintain its own copy of the table.
An upsert inserts a row if its key is absent and replaces the row if the key is already present. It can represent both a new row and an updated one. A delete removes the row for its key. Flink supports this encoding when a changing table has a unique key.2
Suppose a reader starts with no rows and receives these changes in order:
| Change | Current row for acme: plan, price, limit | Current row for gamma |
|---|---|---|
Upsert acme, team, 0.40, 50,000 | team, 0.40, 50,000 | Absent |
Upsert gamma, starter, 0.60, 5,000 | team, 0.40, 50,000 | starter, 0.60, 5,000 |
Upsert acme, business, 0.30, 500,000 | business, 0.30, 500,000 | starter, 0.60, 5,000 |
Upsert acme, business, 0.30, 400,000 | business, 0.30, 400,000 | starter, 0.60, 5,000 |
Delete gamma | business, 0.30, 400,000 | Absent |
The table holds one current row for acme and no row for gamma. Deleting
gamma’s plan does not erase its past usage or set its total to zero. The
change concerns the plans table, and the usage history has its own meaning.
The totals sent to the usage page form another changelog. Their upserts carry the current total for an account. A changelog can therefore come from changes in a source database or from a calculation’s changing result.
Store the result for readers
Section titled “Store the result for readers”The usage page could read the entire usage history and add up every account’s records on each visit. Instead, the processor maintains the totals and stores them where the page can read one row. A stored query result is called a materialized view. Maintaining it as inputs change lets readers use the prepared answer.2
One million individual request records from acme can be represented by
one row containing their total. That row cannot answer which
requests contributed, what time each happened, or what a different counting
rule would produce. Those questions still need the input history.
Storage and freshness are separate choices. PostgreSQL, for example, stores materialized views but updates them when a refresh is requested.3 Our usage processor updates its stored result as it counts records. Even then, records can wait in Kafka or results can wait on their way to the page. The row is current with respect to the inputs already reflected in it. A fresh database read cannot make an old calculation fresh.
Rebuild from the right history
Section titled “Rebuild from the right history”If the page loses its totals table, it has two ways to recover it. It can
recalculate totals from all accepted usage records, or apply the processor’s
output changelog from its beginning. One route adds contributions; the other
applies replacements. Both reach 2,500 for acme if their histories are
complete and processed correctly.
A compacted topic of replacement totals can also rebuild an empty table without retaining every old total. A topic of individual request contributions cannot lose the older requests and still reproduce their sum. And a table that already contains rows must see the deletes it missed; if their tombstones have expired, it may need a fresh rebuild.
The same order and version rules apply during recovery. Repeating an upsert of 2,500 leaves the row at 2,500. Applying an old upsert of 2,000 afterwards moves it backwards. The reader needs preserved order for each key or a version that lets it reject an older change. If recovery starts from a snapshot, the following changelog must begin at its matching position.
The processor can now keep both current totals and current plans. Combining
them would show what an account’s usage costs at its current price. But
acme changed plans at 14:20. A request completed just before that change
arrived just after it. The current row has already lost the old price, which
that request may still need.
What to remember
Section titled “What to remember”- A usage contribution is added to a total. A replacement total overwrites a row. The fields alone do not tell a reader which rule to use.
- A changelog describes table changes. Upserts need a unique key; deletes remove rows for that key.
- A materialized view stores a calculated answer. Its freshness depends on how far the path maintaining it has progressed.
- Rebuilding requires suitable history, the right starting point, and order or versions that prevent old changes from replacing newer ones.
References
Section titled “References”Further reading
Section titled “Further reading”- Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd ed. (O’Reilly Media, 2026), chap. 12, “Stream Processing.” The sections on state, streams, and immutability explain how one event history can support several stored views and why a view can lag behind the writes that feed it.
© 2026 Grig Alex · CC BY-NC-SA 4.0
