Event-driven architecture explained: queues, streams and APIs

Event-driven architecture is a way of connecting systems in which producers publish events and consumers react to them, without calling each other directly or waiting for an answer. Organizations use it to decouple core systems, keep a replayable record of what happened, and feed analytics, fraud models, operational workflows and AI agents with current data. This guide covers the design decisions that matter in production, with examples from banking and insurance, where the audit bar is highest. It is part of our API Management & Integration series.

What event-driven architecture is

Event-driven architecture has four parts. A producer is the system where something happens. It publishes an event, a small record that says what happened, to a broker. The broker stores the event and delivers it. Consumers subscribe and act on it in their own time.

The producer does not know who the consumers are. A core banking system that publishes "payment cleared" does not change when the fraud team, the data platform and a customer notification service all start reading the event. That decoupling is the main reason firms adopt the pattern. The second reason is time. A consumer that is down or slow catches up later instead of failing the producer's transaction.

Event-driven architecture does not replace APIs. Most estates run three styles side by side, and the design work is deciding which flow belongs where.

Events, commands and queries

Systems exchange three kinds of message, and mixing them up causes most design problems.

An event is a fact about the past. "Payment cleared." "Claim filed." "Address changed." It is named in the past tense, it cannot be refused, and any number of systems may care about it.

A command is a request to do something once. "Release this payment." "Issue this endorsement." It has one intended handler, it can fail or be rejected, and doing it twice is a defect.

A query asks for an answer now. "What is the balance on this account?" "What would this policy cost?" The caller waits for the response.

A simple rule follows. Commands go on queues. Facts go on streams. Questions go to APIs.

Message queue vs event stream vs API call

The line between queues and streams is less sharp than it was. RabbitMQ offers a stream queue type that keeps messages for replay. Recent Apache Kafka releases add share groups (KIP-932), which let several consumers take individual messages from one topic the way a queue does. Check what your distribution or managed service supports before you rely on either. The pattern still decides the design. Choose the product after you know whether a flow carries commands or facts.

Replay and the audit trail

A message queue deletes a message once a consumer acknowledges it. An event stream keeps events for a retention period, measured in time or size, and every consumer group tracks its own position in the log. That retained log is what makes replay possible.

Replay serves four jobs in a regulated firm:

• Reconstructing what happened. During an incident or an exam, you can read the exact sequence of events a system saw.
• Adding a consumer without touching producers. A new fraud model or reporting service reads history from the start and then stays current.
• Rebuilding a read model. If a downstream database is corrupted or its logic changes, you rebuild it from the events instead of from backups.
• Giving AI the same history. A model or agent evaluated on replayed events sees what production saw.

Retention needs a deliberate setting per topic. Compacted topics keep only the latest event for each key. That suits reference data such as customer addresses, but it removes history by design. Tiered storage moves older log segments to cheaper object storage, which makes longer retention affordable.

A topic is not a records system. Retention settings can change, compaction removes events on purpose, and an administrator can delete a topic. Where a rule requires you to keep a record unaltered for a set period, copy the events to a store built for that, with immutability, access logging and legal hold. Treat the topic as the operational copy and the archive as the record.

Ordering

Kafka guarantees order within a partition, not across a topic. The producer chooses the partition from the event key, so the key decides which events stay in order relative to each other.

Choose the key from the business entity whose history must be read in sequence. For payments, that is usually the account. For claims, it is the claim or the policy. Every event for one account then lands in one partition and is read in the order it was written. Events for different accounts are processed in parallel.

Four things break ordering in practice:

• A key that is too coarse, such as the branch. It sends a large share of traffic to one partition and creates a hot spot.
• A key that is too fine, such as the event ID. It spreads one account's events across partitions.
• Changing the partition count on a live topic. It remaps keys to new partitions.
• Producer retries without idempotence. Configure producers as idempotent so a retried send does not land out of order or twice.

Queues have their own version of the problem. A single queue with one consumer processes messages in order. Add competing consumers or requeue failed messages and that order is gone. If a command flow needs order, split the work by key across queues, or use one consumer per key.

Idempotency: handling duplicate messages

Most brokers deliver at least once. A consumer can process a message, crash before it acknowledges, and receive the same message again. In a bank, the duplicate that matters is a second payment. In an insurer, it is a second claim payout or a second letter to a customer.

The fix sits in the consumer. Every event carries a unique ID, and the consumer records the IDs it has processed in the same database transaction as the business change. A repeat ID is acknowledged and skipped.

-- PostgreSQL. Run in the same transaction as the business update.
INSERT INTO processed_messages (message_id, consumer, processed_at)
VALUES (:message_id, 'payments-ledger', now())
ON CONFLICT (message_id, consumer) DO NOTHING;

-- If no row was inserted, the message was already handled.
-- Skip the business update, commit, and acknowledge the message.

Producers have a matching problem. A service that updates its database and then publishes an event can fail between the two steps, leaving a change with no event or an event with no change. The transactional outbox pattern solves this. The service writes the event to an outbox table in the same transaction as the change, and a separate process publishes from the outbox. Change data capture with Debezium can read the outbox table and publish to Kafka without new application code.

Exactly-once is narrower than it sounds. Kafka's exactly-once semantics combine an idempotent producer with transactions, so a read, process and write cycle inside Kafka commits once. When a consumer writes to an outside database, calls an API or sends an email, that guarantee no longer covers the side effect. Design for at-least-once delivery and make every consumer idempotent.

Schema governance

An event is a contract between teams that may never meet. If the claims team renames a field, every consumer that reads it breaks, often silently. Schema governance prevents that.

Register a schema for every event type in a schema registry, and have producers and consumers validate against it. Options include Cloudera Schema Registry, Confluent Schema Registry, AWS Glue Schema Registry and Apicurio Registry. Schemas are usually written in Avro, Protobuf or JSON Schema.

Set a compatibility rule per event type and let the registry enforce it:

• Backward compatible: consumers upgraded to the new schema can still read events written with the old one.
• Forward compatible: consumers still on the old schema can read events written with the new one.
• Full: both.

Adding an optional field is usually safe. Removing a field, renaming it or changing its type is a breaking change. It needs a new event version, with both versions published until every consumer has moved.

Every event should also carry a standard envelope, so any consumer, auditor or support engineer can trace it without knowing the payload. The CNCF CloudEvents specification defines a common set of these attributes. A claims event might look like this:

{
  "event_id": "7f3c9a2e-4b1d-4e8a-9c55-0d2f6b7a1e90",
  "event_type": "ClaimFiled",
  "schema_version": 3,
  "occurred_at": "2026-10-27T14:05:12Z",
  "producer": "claims-intake",
  "correlation_id": "fnol-20481",
  "key": "policy-88412",
  "data": {
    "claim_id": "C-55120",
    "loss_date": "2026-10-26",
    "loss_type": "auto"
  }
}

Three rules matter for regulated data. Each event type has a named owner who approves changes. Personal data stays out of event keys and headers, because those are copied into logs and tools that payload controls do not cover. Where a field must be protected, encrypt or tokenize it before it is published, so the broker never holds it in the clear.

Where APIs and the integration layer fit

APIs handle anything that needs an answer now: a balance, a quote, an eligibility check. An API gateway applies authentication, rate limits and logging to those calls. Behind the API, work that does not need to finish during the call moves to a queue or a stream. A quote API can answer in milliseconds and publish "quote issued" for underwriting analytics to read later.

Event interfaces need documentation as much as REST APIs do. The AsyncAPI specification does for channels and events what OpenAPI does for REST endpoints. An API platform that publishes both gives developers one catalog for every interface.

The integration layer connects the systems that cannot publish clean events themselves. Core banking platforms, policy administration systems and mainframes often speak fixed formats, files or proprietary queues. An integration engine transforms those into events and back: ISO 20022 messages for payments, ACORD standards for insurance, and the legacy formats already flowing through IBM MQ. It also bridges queues into streams, copying messages from an existing queue into Kafka so new consumers can read them without changing the flow that already works. Apache Camel, MuleSoft and WSO2 Micro Integrator are common choices for this layer.

Events for AI agents

AI agents are a new kind of consumer. An agent that handles a claim or reviews a payment alert needs current context, and the event stream is often the best source of it: what happened to this account in the last hour, in order, from the systems of record.

Agents also act. They call APIs and use tools through the Model Context Protocol (MCP), and some of those actions publish new events. That makes the integration layer the natural place to govern them. The controls that already apply to applications apply to agents: an identity per agent, access limited to the topics and APIs it needs, and a log of what it read and what it did.

Two layers do this work. An AI gateway is the enforcement point on the traffic path. It governs every call an agent makes to a model, an MCP server or an API. An agent control plane sits above it and manages the agents themselves: inventory, identity, lifecycle, evaluation and suspension. It uses the gateway to enforce. On WSO2, that control plane is WSO2 Agent Manager.

Two event-specific rules help. Give each agent its own consumer group, so its position in the log, its lag and its reads are visible separately from other consumers. Treat any event an agent publishes the way you would treat one from a new application: a registered schema, a named owner, and a field that records which agent produced it.

One flow, end to end: an insurance claim

Here is how the pieces fit for an insurer's first notice of loss.

1. A customer reports a car accident in the mobile app. The app calls a claims API through the API gateway, which authenticates the user and returns a claim reference within the call.
2. The API places a "register claim" command on a queue. The claims system takes it, creates the claim and acknowledges the message. A duplicate submission is caught by its message ID.
3. The claims system writes the claim and a "claim filed" event to its outbox in one transaction. A publisher moves the event to a Kafka topic keyed by policy number, so every event for that policy stays in order.
4. Fraud scoring, reserving, the data platform and a claims triage agent each read "claim filed" through their own consumer group. None of them slows down the claims system, and each can be added or replayed independently.
5. The triage agent reads the policy's recent events, calls a coverage API through the AI gateway, and publishes "claim triaged" with its own agent identity in the envelope.
6. Every event carries the same correlation ID. An auditor can follow the claim from the first API call to the triage decision without asking the developers.

What auditors check

Auditors and examiners rarely ask about brokers by name. They ask whether you can show control over the data that moves through them. Expect requests for evidence like this:

• An inventory of topics and queues, each with an owner, a data classification and a purpose.
• Schemas under change control, with compatibility enforced by the registry and approvals recorded.
• Access control per topic and queue, granted to service and agent identities rather than shared credentials, and reviewed on a schedule.
• Encryption in transit between clients and brokers, and at rest on broker storage.
• Retention set per topic and matched to the record schedule, with record-bearing events copied to a store that supports immutability and legal hold.
• A correlation ID that lets one transaction be traced across every system it touched.
• Dead-letter queues with an owner, alerts, and a reviewed process for resolving or discarding each message.
• Evidence that duplicates are handled, such as idempotency checks in payment and payout consumers.
• Monitoring for consumer lag, failed deliveries and broker health, with alert thresholds and response records.
• Broker configuration under change management, and disaster recovery and replication tested, with the results kept.

If you can produce each of these from system records alone, the event platform is ready for review.

When to bring in a partner

The hard part of event-driven architecture is not installing a broker. It is deciding which flows belong on a queue, a stream or an API, then running the new design next to what already works without another outage. Duczer East provides event-driven architecture consulting for banks, insurers and other enterprises across message queues, Kafka, APIs and the integration layer, and supports what it builds.