Skip to content

Distributed processing and the log

A queue or a log?

How storage between a producer and a consumer lets each work at its own pace, how a queue hands records to workers and removes them, and how a log keeps them for every reader.

Learn with an AI assistant

Copy the prompt and paste it into a chat with an AI assistant.

You are a tutor for the course "Stream Processing".
Topic: "A queue or a log?"
In short: How storage between a producer and a consumer lets each work at its own pace, how a queue hands records to workers and removes them, and how a log keeps them for every reader.
Topic text in Markdown: https://stream-processing-systems.github.io/distribution/a-queue-or-a-log.md
Base your explanations on the topic text. First ask how I want to study: a short summary, a step-by-step walkthrough, or a deep dive with questions that check my understanding. Use the text's example and terms. If you add anything the text does not say, point it out.

At 09:00 on February 2, the working day begins for most customers of our AI assistant, and requests pour in. The gateway writes 3,000 usage records a second to the input history. The usage processor counts 2,000 a second. Every second, another 1,000 records are left waiting, and the totals on the usage page fall further behind.

One processor is not enough, so the company starts two more. Three processors can share the work only if the results stay right: acme is billed for each request once, its usage page shows its whole total, and the email at 1,500 tokens is sent once, when the account actually crosses 1,500. How should the records be handed out to the processors?

A writer that does not wait for the reader

Section titled “A writer that does not wait for the reader”

The gateway, at least, is not slowed down. It does not hand records to the processor directly. It writes them to the history and waits only until the history has stored each one. How fast the processor reads is not the gateway’s concern: it keeps accepting requests at 3,000 a second, and no record is lost while the processor lags.

A program that writes records to such storage is called a producer, and a program that reads them is a consumer. The gateway is a producer, and the usage processor is a consumer. Storage between them lets each work at its own pace, and either can stop for a while without stopping the other.

Storage does not make the consumer faster, though. After an hour at these rates, the gateway has written 10.8 million records, and the processor has read 7.2 million. The record it is reading now was written 20 minutes ago, so the usage page shows totals as of 20 minutes ago, and the gap keeps growing. The processor can catch up in the evening, when requests slow down, but only if it keeps up with the input over the whole day and the storage has room for everything that waits.

A writer that does not wait for the reader The gateway writes 3,000 records a second, the processor reads 2,000, and the history holds another 1,000 each second. 3,000 records/s 2,000 records/s Gateway 3,000 records/s Input history +1,000 waiting each second Usage processor 2,000 records/s

A queue: whoever is free takes the next record

Section titled “A queue: whoever is free takes the next record”

A common way to share work among several workers is a queue. Records wait in it in the order they arrive, and each one goes to one of the workers: to the next in turn or to the next one that is free.1 When a worker has finished with a record, it acknowledges it, and the queue removes the record. If a worker fails before acknowledging, the queue gives the record to another worker.2

Here is how a queue might hand out acme’s records among processors A, B, and C:

RecordTokensProcessorThat processor’s total for acme
u-1011,000A1,000
u-102400B400
u-103600C600
u-104500A1,500

The work is shared, and so is acme’s state. The account has used 2,500 tokens, but no processor knows that. To show the total, the usage page would have to add up three partial totals. The email goes wrong: the account crossed 1,500 tokens with u-103, when no processor had more than 1,000. Processor A reaches 1,500 at u-104, but that is only its own share, and an email based on it would come late and by coincidence.

Deduplication breaks too. Suppose the gateway sends u-102 again because the acknowledgement was lost. This time the queue gives it to processor C, while B handled the original. Each keeps its own list of counted IDs, so C does not recognize u-102 and adds another 400 tokens to acme’s usage.

The queue also removes each record once it is acknowledged. A processor that restarts from a checkpoint cannot replay the records it acknowledged after that checkpoint, because the queue no longer has them. And other readers miss them too. The finance team reads each month’s records to compare the totals with the billing system. To read them from a queue, the team needs its own queue with its own copy of every record, set up before the records arrive. A reader added in the middle of the month cannot get the records from its beginning.

None of this makes a queue a bad tool. It suits tasks that do not depend on each other. The emails from our outbox are such tasks: any sender can send any email, and once it is sent, nobody needs it again. A usage record is different. It changes an account’s state, which other records of the same account change too.

A log: records stay, readers keep their place

Section titled “A log: records stay, readers keep their place”

Our input history works differently. Storage that appends each record to the end, numbers the positions, and keeps the records after they are read is called a log. Reading does not change a log. Each reader keeps only its own reading progress, the position of the next record to read.

Apache Kafka, an open-source system that stores streams of records as logs and is widely used for this job, works this way. Records are not deleted when they are read; how long they are kept is set separately, and producers and consumers know nothing of each other.3 Many messaging systems track which records each reader has finished and delete them. A Kafka reader reads in order, so its progress is a single number, the position it has reached.4

The finance team now needs no copy. Its reader starts at the first record of February and reads at its own pace, while the usage processor reads the newest records. Neither affects the other: the processor’s progress does not remove anything that the finance reader has not yet reached. A reader added next week can start from the beginning of the month as well, as long as the log still holds those records.

Readers keep their own positions The usage reader and the finance reader have reached different positions in the same log, whose records remain available to both. Log records stay after reading Usage reader near the newest records Finance reader reads the same history 1 2 3 4 POSITION

Here is how the two compare:

QueueLog
Reading a recordThe worker takes it; once acknowledged, it is removedThe reader moves its position; the record stays
Who gets a recordOne of the workersEvery reader of the log
Another purposeNeeds its own copy of every recordReads the same log from its own position
After a failureUnacknowledged records go to another worker; acknowledged ones are goneThe reader returns to a saved position and reads again
SuitsIndependent tasks, such as sending emailsA history that several readers read in order, such as usage records

Some products offer both ways of reading, so the choice is between the two contracts more than between products. What decides it is whether records depend on each other and whether anyone will need them again.

With a log, how do processors A, B, and C share the work? If each of them reads the whole log, each counts every account, and three processors do the work of one three times over. If they take positions in turn, A taking 1, 4, 7, and so on, acme’s records are split among them exactly as in the queue. The partial totals, the missing email, and the copy of u-102 counted twice all come back.

All of acme’s records must go to one processor, which keeps the account’s whole state, while beta’s records may go to another. The account is the key of these records, so the log has to be divided by key. But a log is one numbered sequence, and a processor’s progress is one position in it. How can it be divided so that each part keeps its own order, and what does a checkpoint save when a processor reads only one part?

  • Storage between a producer and a consumer lets each work at its own pace. It gives a slow consumer time to catch up but does not make it faster.
  • A queue hands each record to one worker and removes it once acknowledged. It suits independent tasks; records that change shared state get scattered, and nothing is left to read again.
  • A log keeps records after they are read, and each reader keeps its own position. Several readers read the same history independently, and a reader can go back.
  • Several processors on one log share the work correctly only if all records of one key go to the same processor.
  1. RabbitMQ: Work Queues, Round-robin dispatching and Fair dispatch.

  2. RabbitMQ: Consumer Acknowledgements and Publisher Confirms, Automatic Requeueing.

  3. Apache Kafka 4.4: Introduction, Main Concepts and Terminology.

  4. Apache Kafka 4.4: Design, Consumer Position.

  • Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd ed. (O’Reilly Media, 2026), chap. 12, “Stream Processing.” Compares message brokers that delete a message once it is acknowledged with brokers built on a log, and explains why a log lets readers come back to old messages and new readers start from the past.

© 2026 Grig Alex · CC BY-NC-SA 4.0