Queues and workers

A report takes 15 seconds to generate. If your API generates it inside the request handler, the caller waits for those 15 seconds before getting a response.

You can separate those two jobs. The API puts a task on a queue and returns a task ID. A worker picks up the task and generates the report in another process.

Three running processes

Redis does not run Python. Celery is the task system, and the worker is a separate process you start with its CLI. Both your API and the worker connect to the same Redis broker.

Call or queue

# Run here. The caller waits for the function to return.
generate_report()

# Send a message so a worker can run the function.
task = generate_report.delay()

The second call returns after submitting the task to the broker. It does not wait for the report to finish. If Redis is unavailable, submission can fail; the queue is a dependency, not a way around every failure.

The HTTP response uses 202 Accepted. That tells the caller that the request was accepted for processing. It does not say the work succeeded.

When the worker stops

If Redis stays running, you can submit a task while the worker is stopped. When you start the worker again, it picks up the waiting task. You will demonstrate that on the next page.

This experiment concerns work that has not started yet. Recovery after a worker crashes during execution needs additional decisions about acknowledgments, retries, and duplicate effects.

When this is useful

A separate worker is useful for reports, data imports, and other work that should execute independently of an HTTP request. FastAPI also has a smaller BackgroundTasks feature for lightweight tasks in the API process. Use a queue when you need the separation shown here.

The example follows Celery's first steps. It leaves result storage and the platform's event ledger for another chapter.

Checkpoint

You should be able to point to the process that runs the report and explain why the HTTP response can arrive before the report finishes.