Run the Order Pipeline

Run a complete order flow, inspect it with the CLI, and then review the code end-to-end.

example payment_gateway_resilience

series Durable Execution for Transactions, part 1 of 5

Info · Complete the getting started tutorial first

This tutorial assumes you have already worked through Getting Started, which explains FlowApplication, flow!, stage types, topology, journals, and replay. That example never calls an outside service. This one calls a payment gateway, and the tutorial is about that call.

Let’s walk through the payment gateway example that ships in the repository. The real complexity is streaming side-effecting transactions to a payment gateway, and doing so without ever risking double-charging a credit card, even when resuming after a crash or replaying the flow for audit purposes.

The flow takes orders from two sources, merges them into one stream and validates each order.

  • Validation sends any invalid order, such as one for a zero amount, to intake review and cancels it.
  • Every valid order goes to the payment gateway to be authorized.
  • We fan out all outcomes to multiple sinks. Authorized orders go to shipping, declined orders are recorded, and orders left without a decision go to manual review.

The payment gateway is modelled as a port, the boundary between your code and a third party. A system answers the port. Here a simulated system answers it, and a Stripe system could answer the same port without the flow changing.

We’ll show the simulated system in action by scripting a gateway outage partway through the run. Calls hang and fail, a circuit breaker stops sending them until the gateway recovers, and the orders behind it wait. One order runs out of attempts and goes to manual review. The rest are paid once the gateway is back.

The challenge of transactional streaming

Streaming has always been something of a glue between other systems that make changes to the world. Some of the early streaming systems of the 2000s were originally known as complex event processing systems, like Esper. They matched patterns across streams of events, such as a run of suspicious card swipes, and raised an alert for another system to act on. In the 2010s, Kafka made the log the centre of the architecture, a durable, replayable mover of bytes between the systems that did the work. Its exactly-once guarantee, added later, covers data read and written within Kafka itself, and delivering exactly once to any other system still takes that system’s cooperation.

Today, we have far more tools at our disposal to build reliable streaming systems that perform work directly. ObzenFlow is one of them. At its core, ObzenFlow was designed from the start to safely perform side-effecting work, like charging a credit card, and to keep an entire flow’s worth of streaming data safe to replay without repeating side effects. Replay lets you audit any run, and resume continues a run that crashed or was stopped.

Why payments? Payment authorization is the textbook non-idempotent call. If your process crashes after the gateway authorized but before you recorded it, a naive retry charges the customer twice. If you refuse to retry, you lose orders. Every durable system has to answer the same question: how do you run an external call exactly once when the machine running it can die at any moment?

ObzenFlow’s answer is to make the call itself data. Your code never calls the gateway. It hands a typed request to the runtime through fx.call, the runtime sends it under a stable idempotency key and journals whichever outcome came back, and from then on that answer is just another fact in the flow. When the machine dies, resume rebuilds the flow from those facts and reads the recorded outcome instead of calling the gateway again. The only call that can go out twice is the one the crash interrupted, and it goes out under the same key. That is what you test at the end of this tutorial, when you kill a run, resume it, and check the simulated system’s call log for any payment decided twice.

Now let’s begin. The fastest way to see what this tutorial builds is to run it before reading the code.

Run it first

With one Cargo command you’re going to run a complete order flow and watch it take a hit. Partway through, the payment gateway goes down. Calls hang, a circuit breaker opens, and orders pile up behind it. Then the gateway comes back, the waiting orders are paid, and not one order is lost.

This is the flow you’re about to run, with every stage under its real name.

Then you’ll get to know the ObzenFlow CLI. This is a purpose-built tool to inspect the system’s journals, which are a record of execution, not just logs. They represent every outcome of every decision that the framework makes, including orders and payments, but also runtime decisions within the framework, like retry logic, circuit breakers opening, and so forth. This is the power of event-sourced finite state machines, and the CLI helps you see it in action.

Run the example with Cargo

If you have already cloned the framework repo from Getting Started, stay in that working directory. Otherwise, clone it now.

git clone https://github.com/ObzenFlow/obzenflow
cd obzenflow

Run the payment gateway example. The gateway and the orders are simulated, so it needs no external service and no extra Cargo feature.

cargo run -p obzenflow --example payment_gateway_resilience

The run takes about a minute. Part of that is a scripted gateway outage, so expect a quiet stretch in the middle. The console lines are each sink delivering an order’s outcome, which is fine for a quick glance. When the run finishes, it prints where it was recorded.

payment_gateway_resilience_demo completed. Journal: target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1

Inspect the manifest with the CLI

The path after Journal: is the run’s manifest, and it contains a collection of journals. You can inspect the folder if you’re curious, although journals use a custom codec, so you’ll need the ObzenFlow CLI to print their contents. Its show command takes the path your run printed. Your run has its own id, so use your path wherever this page uses this one.

cargo run -p obzenflow --features cli --bin obzenflow -- \
  show target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1

show prints the whole run, about 1,900 lines, so don’t try to read it all now. It comes in three parts.

  • It opens with a short legend and a numbered list of the run’s journals.
  • Then it prints every record the run wrote, in causal order, each with the stage that wrote it and its payload.
  • It ends with an inventory of the run, with one block for each stage that counts every event the stage wrote.

Let’s demonstrate the effectiveness of show by starting with a specific question. What happened to the orders that reached the payment gateway, including outages? We can start by looking at the count of every payment that was authorized, declined, or left without a decision by the authorize_payment stage. Start at the end and scroll up from the last line to the block headed EFFECTFUL TRANSFORM: authorize_payment, the stage that calls the gateway.

 1EFFECTFUL TRANSFORM: authorize_payment
 2  Reads from: validate_order
 3  Data subscribers: declined_orders, manual_review, paid_orders
 4  Owns:
 5    Transform_authorize_payment_stage_01M7H7NPK6HM6T7H9MK2GF4W97.log
 6      Purpose: data, control and stage lifecycle
 7      Runtime readers: —
 8      Count  Event                                     Author
 9          1  execution/effect.execution_failed@1       authorize_payment
10          1  fact/payment.authorization_unavailable@1  authorize_payment
11         18  fact/payment.authorized@1                 authorize_payment
12          4  fact/payment.declined@1                   authorize_payment

Lines 2 and 3 place the stage in the flow. It reads from validate_order, and three sinks subscribe to it. Line 5 is the journal file it owns. Lines 9 to 12 count what it wrote. Line 11 shows eighteen payments authorized, and line 12 shows four declined. Lines 9 and 10 are the one order left without a decision, a failed request and the fact the stage wrote in its place. That makes twenty-three orders that reached the gateway. The other two never got this far, because validation cancelled them.

The rest of this tutorial builds the flow that wrote this record, then comes back to show to read it one record at a time.

Define the flow

Now you’ll learn how to sketch out the flow that you just ran and build it from the ground up. Sketching with placeholders is an important concept because it lets you work out the flow and shape of an ObzenFlow program before committing to very detailed business logic. For anyone familiar with domain-driven design and event storming, this mechanism in ObzenFlow allows early first drafts of a program to actually compile and even show up in the upcoming ObzenFlow Studio.

The outer shape is the one from Getting Started, a flow! handed to FlowApplication. What’s new here is that a stage can produce several kinds of fact, and the topology fans in and out. You’ll finish by filling in the first real handler, validation, which shows how one stage writes more than one fact.

Sketch the flow with placeholders

Here is the example’s stages block with every handler swapped for placeholder!(), the way you would sketch it before writing any logic. The example that you see below is more or less a fully formed stage definition, but with the placeholder syntax standing in for a handler.

The important piece here is to start thinking in flow and types and how different types move through the program. A stage declaration is a typed signature, In -> { A, B, C } => handler, and every event you name becomes a type the compiler checks.

stages: {
    web_orders = source!(CustomerOrderPlaced => placeholder!());
    store_orders = source!(CustomerOrderPlaced => placeholder!());

    validate_order = transform!(
        CustomerOrderPlaced -> {
            ValidatedOrder,
            InvalidOrder,
            CancelledOrder
        } => placeholder!()
    );

    // The gateway call. It becomes an effectful stage in the next act.
    authorize_payment = transform!(
        ValidatedOrder -> {
            PaymentAuthorized,
            PaymentDeclined,
            PaymentAuthorizationUnavailable
        } => placeholder!()
    );

    paid_orders = sink!(PaymentAuthorized => placeholder!());
    declined_orders = sink!(PaymentDeclined => placeholder!());
    invalid_orders = sink!(InvalidOrder => placeholder!());
    cancelled_orders = sink!(CancelledOrder => placeholder!());
    manual_review = sink!(
        PaymentAuthorizationUnavailable => placeholder!()
    );
}

The topology connects the graph. Both order feeds converge on validation, valid orders cross one explicit external boundary, and typed outcomes fan out to their deliveries.

flow.rs
topology: {
    web_orders |> validate_order;
    store_orders |> validate_order;
    validate_order |> authorize_payment;
    validate_order |> invalid_orders;
    validate_order |> cancelled_orders;
    authorize_payment |> paid_orders;
    authorize_payment |> declined_orders;
    authorize_payment |> manual_review;
}

One detail is easy to miss and worth knowing early. Because the effectful authorize_payment sits below a fan-in, the runtime merges the two order channels deterministically, so delivery order is a pure function of the recorded streams rather than of arrival timing. Live runs with different timing and replays all consume orders in the same merged order.

Implement deterministic validation

Validation is an ordinary transform. Its handler is a pure function of its input, so it unit-tests without runtime scaffolding, and during replay it simply runs again.

A stage that can write more than one kind of fact lists its outcomes in outcome!, one for each way an order can turn out, and returns exactly one of them. The handler is a plain match on whether the order has a reason to fail.

validation.rs
impl TypedTransformHandler for ValidateOrder {
    type Input = CustomerOrderPlaced;
    type Output = outcome![ValidatedOrder | (InvalidOrder, CancelledOrder)];

    fn process(&self, order: CustomerOrderPlaced) -> Result<Self::Output, HandlerError> {
        match invalid_reason_for(&order) {
            None => Ok(outcome(validated_order(order))),
            Some(reason) => Ok(outcome((
                invalid_order(&order, reason.clone()),
                cancelled_order(&order, reason),
            ))),
        }
    }
}

A valid order produces one fact, the ValidatedOrder that goes on to payment. An invalid order produces two. InvalidOrder says why validation failed and goes to invalid_orders for intake review, and CancelledOrder goes to cancelled_orders. Because the list pairs them in one outcome, the handler can’t return one without the other.

outcome(...) builds the value process returns. Its facts are journaled in the order the list writes them, and the compiler checks that the list and the stage’s declaration name the same three facts.

Validation owns exactly one question, whether an order is well formed enough to attempt payment. A zero amount or an unusable payment method fails here. Insufficient funds and address mismatches pass through untouched, because those decisions belong to the gateway, and answering them locally would put a guess in the journal.

Next steps

You studied an entire flow end to end and learned how to sketch your own flows before committing to business logic by using placeholders. You ran it, inspected the manifest with the CLI, and then studied a custom handler that implements validation. This is a perfect structure to build on in the next parts.

The next part covers side effects and then the nitty-gritty world of integrating with third-party systems. It’s certainly the riskiest code that you’ll ever implement, not only in ObzenFlow but in really any framework, because third-party systems are outside of your control. But ObzenFlow gives you all of the tools you’ll need to mitigate this risk and fully leverage its algebraic effect system. You’ll turn effects into data and watch the runtime turn that data into journaled interactions with external systems.