Real-Time Intelligence and Event-Driven Decisions

Software versions and features in this article are as of 2 August 2026. INVISOM has no commercial relationship with any vendor named.

Real time is a property of the decision

Streaming platforms are now mature, open and widely available. Apache Kafka 4.0, released in March 2025, was the first major version to run entirely without ZooKeeper. Apache Flink reached 2.0 the same month and has shipped three feature releases since. The technical barrier to "real-time everything" has never been lower.

That makes the strategic question sharper, not easier. Real-time pipelines cost more to build, run and debug than batch ones, and they introduce failure modes that batch systems never had: out-of-order events, duplicates after a retry, state that must survive a crash. Those costs are worth paying only where acting sooner changes the outcome.

Our argument has two parts. First, latency should be specified per decision, from how fast the decision's value decays, not per dataset or per platform. Most enterprise decisions are well served by minutes or hours. Second, for the decisions that do need real time, the hard engineering problem is not throughput. It is correctness at the moment of action, because the "exactly-once" guarantees of streaming systems end where a decision touches the outside world: a declined card, a dispatched driver, a blocked SIM.

Four latency tiers, and what each one costs

A useful way to start is to place each decision in one of four tiers. The boundaries are approximate; what matters is the question each tier answers.

In-transaction (milliseconds). The decision sits in the request path, so the customer is waiting. Stripe states that its Radar fraud system scores a payment in less than 100 milliseconds, assessing more than 1,000 characteristics of the transaction. Uber's DeepETA model, which feeds fares, pickup estimates, rider-driver matching and delivery planning, must return a prediction within a few milliseconds and is Uber's highest-volume model by queries per second. At this tier, features must be precomputed and served from fast state; there is no time to query a warehouse.

Near real time (seconds). The decision follows the event closely but is not blocking a user. Operational alerts, fraud signals on usage patterns and dynamic inventory updates live here.

Micro-batch (minutes). Freshness matters, but a few minutes of delay costs little. Uber's exactly-once ad events pipeline, described below, deliberately chose two-minute checkpoint intervals and its engineers described the result as not exactly "real-time".

Batch (hours or daily). Planning, credit limit reviews, most reporting, and most model retraining. Our guide to selecting AI use cases applies here too: choose the tier from the decision, not the technology roadmap.

Three questions decide the tier:

The second question is the one most often skipped. If a human analyst works the alert queue twice a day, a sub-second pipeline into that queue buys nothing.

A vendor-neutral reference architecture

Most real-time decision systems converge on the same shape, whatever products implement it.

Capture. Applications publish business events directly, and existing databases emit changes through change data capture (CDC). Open-source Debezium, for example, reads the MySQL binlog or the PostgreSQL logical replication stream and publishes each table's changes to its own topic, running inside Kafka Connect, as a standalone server, or embedded in an application. The outbox pattern, where an application writes an event to an outbox table in the same database transaction as its state change and CDC publishes it, avoids the classic bug of updating the database but failing to publish the event.

Log. A durable, replayable, partitioned event log, most commonly Kafka or a Kafka-compatible service, is the system of record for events in motion. Kafka 4.2, released in February 2026, made share groups ("queues for Kafka") production-ready, so the same platform can now serve both ordered streams and work-queue consumption.

Process. A stateful stream processor such as Flink maintains aggregates, joins and features over time windows, with state that survives failure through periodic checkpoints.

Serve. Decisions need fresh state at low latency: an online feature store, a key-value store, or a streaming database. Materialize and RisingWave, for example, maintain SQL materialized views incrementally as data arrives instead of recomputing them, which lets teams express real-time logic in SQL.

Decide and act. A decision service combines served features, models and rules, applies guardrails, executes the action and writes a decision event back to the log.

Land and replay. The same events land in open tables in the lakehouse for analytics, training and audit. Our data architecture guide covers that layer. Jay Kreps' 2014 argument for the "Kappa" architecture still holds: rather than maintaining separate batch and streaming code paths that must agree, keep the log and reprocess it with the same code when logic changes.

Correctness: ordering, duplicates and late data

Every streaming system forces three trade-offs. Making them explicitly is most of the design work.

Ordering is per partition, so the key is a design decision. Kafka guarantees order only within a partition. Choosing the partition key, such as card number, account, vehicle or subscriber, defines which events are guaranteed to be seen in order. Global ordering is possible only by giving up parallelism. Pick the key that matches the entity the decision is about.

Late data trades completeness against latency. Events arrive out of order because devices go offline and networks retry. Flink handles this with event time and watermarks, which declare that no earlier events are expected. Hold watermarks back and windows wait longer for stragglers; push them forward and late events are dropped or handled separately. There is no setting that delivers both minimum latency and complete data.

Exactly-once is real, but bounded. Since 2017 Kafka has offered idempotent producers and transactions, so a read-process-write cycle inside Kafka can commit its outputs and consumer offsets atomically. Confluent's own benchmarks at the time reported a 3% producer throughput cost for transactions, and 15% to 30% for Kafka Streams with a 100-millisecond commit interval. Flink achieves exactly-once state through checkpoints based on the Chandy-Lamport snapshot algorithm, provided the source can rewind, as Kafka can.

The boundary is the part architects most often miss. Confluent's description is explicit that if a streaming application calls an external service, those side effects are not covered by the guarantee. A text message sent, a payment instruction issued or a card blocked through an API sits outside the transaction.

Uber's ad events system is a good public example of designing for that boundary. Because ad events represent money paid to Uber, double counting would overcharge advertisers and losses would under-report. The team combined Flink with Kafka transactions, generated a unique identifier for each record so downstream writes were idempotent, and relied on upserts in Apache Pinot to remove duplicates. They then set checkpoints every two minutes, accepting latency in exchange for correctness. That is the right trade for billing. It would be the wrong one for fraud in the payment path.

The general rule: make every action idempotent with a decision ID, and let the downstream system reject duplicates. Exactly-once in the pipeline then becomes a performance optimization rather than the only thing protecting the customer.

Event-driven decisions in practice

Fraud in payments. Card and account-to-account fraud is the clearest case for in-transaction decisions, because funds are hard to recover once they move. Stripe's account of Radar shows how the architecture evolves: it moved from an ensemble of gradient-boosted trees and a neural network to a single deep neural network in mid-2022, which Stripe says cut training time by more than 85%, and it added "risk insights" in 2020 so users can see which features contributed to a decline. Stripe reports that Radar incorrectly blocks 0.1% of legitimate payments. These are vendor figures, but the pattern is general: speed, feature breadth and explainability have to be designed together.

Logistics and mobility. Uber's DeepETA is instructive for a different reason. Rather than replacing its routing engine, Uber trained the model to predict the residual between the routing engine's estimate and observed arrival times. The real-time decision combines a deterministic system with a learned correction, which is easier to evolve and to explain than an end-to-end model.

Telecommunications. The Communications Fraud Control Association's 2023 survey estimated global telecom fraud losses at $38.95 billion, up 12% on 2021, with subscription and identity fraud among the leading types. In our analysis this argues for in-transaction checks at activation, SIM swap and number porting, where an irreversible step happens, while many other telecom analytics, such as churn propensity and capacity planning, belong in the micro-batch or batch tiers.

The Gulf. Saudi Arabia's sarie instant payment system, owned by the Saudi Central Bank and launched in 2021, moves transfers of up to SAR 20,000 instantly, 24 hours a day, using identifiers such as mobile numbers. When money settles in seconds around the clock, fraud screening has to sit in the payment path and run without a human on shift. That moves banks' fraud decisions from the seconds tier into the in-transaction tier.

Guardrails for automated action

Speed amplifies errors as well as value. On 1 August 2012, Knight Capital deployed new trading code incorrectly, reactivating dormant logic in its order router. According to the SEC, in the first 45 minutes after the market opened it sent more than 4 million orders while trying to fill 212 customer orders, and lost more than $460 million. Automated warning emails had been generated before the open and were not acted on. The SEC later fined the firm $12 million for inadequate market access controls.

The lesson for any event-driven decision system is that controls must operate at machine speed too:

• Pre-action limits on rate, value and volume per entity and in total, enforced by the decision service rather than by people watching dashboards.
• A kill switch that a named person is authorized and able to use, tested regularly.
• Shadow mode and canary releases for new models and rules, with decision events compared before full rollout.
• Alerts routed to someone who must acknowledge them, not to a mailbox.
• A decision log on the event stream recording inputs, versions and actions, so incidents can be replayed. The same record supports the trust and audit requirements discussed in our companion piece on trusted decision intelligence.

Worked example: which telecom decisions need real time?

Consider a hypothetical mobile operator whose backlog contains four requests labelled "real time". Applying the three questions sorts them quickly.

Only the SIM swap and port-out decision justifies the in-transaction tier, because the harmful step is hard to reverse and the event is available at the moment it happens. The traffic anomaly needs seconds, not milliseconds. The other two gain little from streaming, and building them as streaming jobs would add operating cost without changing a single outcome. The requests that survive this test are the ones that deserve the most careful engineering.

What to do in the next 90 days

1. Inventory the decisions, not the data feeds. List the 15 to 20 decisions that matter most and record their current latency and their real decision window.
2. Assign each a latency tier using the three questions above, and challenge any real-time request where a person will act on the output later.
3. Fix capture first. Introduce CDC and the outbox pattern for the systems that feed in-transaction decisions, before buying new processing technology.
4. Choose partition keys deliberately for each event stream, aligned to the entity each decision is about.
5. Make actions idempotent with a decision ID passed to every downstream system, and test duplicate delivery explicitly.
6. Install guardrails before go-live: limits, a kill switch with a named owner, shadow mode and a replayable decision log.
7. Land every event in open tables so the streaming and batch views of the business are computed from the same log. Our platform placement analysis covers where those tables should live.

Frequently asked questions

When is real-time analytics worth the cost?

When the value of a decision decays faster than your batch cycle and the action can be automated safely. Fraud in the payment path, dispatch and pricing usually qualify. Decisions reviewed by people on a daily rhythm, or reporting, rarely do.

What is the difference between Kafka and Flink?

Kafka is a durable, partitioned event log that stores and distributes events. Flink is a stateful stream processor that computes over those events, such as joins, windows and aggregates, with checkpointed state. Most real-time architectures use a log and a processor together.

Does exactly-once processing guarantee my action happens once?

No. Kafka transactions and Flink checkpoints guarantee exactly-once results inside the streaming system. Calls to external systems, such as sending a message or blocking a card, are outside that guarantee. Make those actions idempotent with a unique decision ID.

What is change data capture?

Change data capture reads a database's transaction log and publishes each insert, update and delete as an event, so other systems can react without polling. Debezium is a widely used open-source example.

Sources and further reading

• How we built it: Stripe Radar, Stripe, 29 March 2023 (vendor material).
• DeepETA: How Uber predicts arrival times, Uber Engineering, 10 February 2022.
• Real-time exactly-once ad event processing with Flink, Kafka and Pinot, Uber Engineering, 23 September 2021.
• Exactly-once semantics are possible, Confluent, June 2017.
• Apache Kafka 4.0.0 and 4.2.0 release announcements; Kafka documentation.
• Apache Flink: stateful stream processing, event time and watermarks, release announcements.
• Debezium architecture; Materialize; RisingWave.
• Questioning the Lambda Architecture, Jay Kreps, 2 July 2014.
• SEC charges Knight Capital, US SEC, October 2013.
• sarie instant payments, Saudi Central Bank.
• Telecommunications fraud in 2023, CFCA, 13 November 2023.