Run and Investigate

Learn how a run is recorded in event-sourced journals, then read what it left behind with the ObzenFlow CLI.

example payment_gateway_resilience

series Durable Execution for Transactions, part 4 of 5

It’s overnight, and you’re the operator on rotation. The payments process is running, and nothing looks wrong. Then an alert fires. One order has landed in manual review. It’s an $18.00 store purchase, and the payment stage never got a decision on it. Every other order that night was authorized or declined. What happened to this one? Now what?

This tutorial will teach the exact steps it takes to understand what happened during any flow’s execution. You’ll find that order in the journals and trace it back to its cause. The next part covers what to do when the machine itself goes down.

Architectural philosophy

Before diving into root cause analysis, it helps to know ObzenFlow’s storage and execution architecture.

Info

ObzenFlow was built from the ground up with site reliability engineers in mind. I’ve managed several SRE teams myself, so I know the whole journey, from the first anomaly, through the incident and how it resolves, to root cause analysis and the postmortem. ObzenFlow is architected to make the journey from anomaly to recovery to root cause analysis and postmortem as seamless as possible.

ObzenFlow’s philosophy begins with durable execution, which means caring about storage and committing to never lose data barring force majeure. Whenever there’s a choice between performance and reliability, ObzenFlow trades off performance. Nothing is 100%, but you can be fairly sure ObzenFlow itself won’t drop data to chase performance vanity metrics that you probably don’t even need or want anyway. Drives, machines and other hardware still fail, though, so you can’t assume there will be no data loss.

Does this mean ObzenFlow doesn’t care about performance? Not at all. But a lot of the benchmarks you see advertise millions of events per second, and two things are worth knowing about that number.

  1. Reaching a million events per second takes an astounding number of trade-offs.
  2. ObzenFlow isn’t a simple byte shipper, and it isn’t built for that volume. At a million events per second you’re most likely moving telemetry, such as clicks and page views, and you’re much better off choosing a different platform.

ObzenFlow is architected for hundreds of events per second, with a smooth route to 10,000, and never for millions. In return you get durable execution with no data loss, full observability through wide events, and the safety contracts of replay and resume.

ObzenFlow is dealing with the complete inverse problem, which is how to handle long-running tasks. Think live inference in a streaming context. ObzenFlow isn’t trying to shove as many bytes into a stage as humanly possible. It’s trying to cope with the fact that one of its AI map-reduce inference stages might take 30 seconds. Say you’re processing millions of events. There’s no way to send a million of them through a stage like that in one second.

That’s why ObzenFlow has the sophisticated middleware you’ve already learned about, and built-in backpressure. All of its trade-offs come from the same reality. Durable execution is typically high consequence, and it’s also slower and more methodical.

With this in mind, we can start to look at ObzenFlow’s storage architecture, specifically how it leverages journals. This will always be your first step when dealing with failure: finding the manifest that ObzenFlow creates during every execution. Then you’ll look at the contents of that manifest, which is a collection of journals. Then we’ll understand what’s in each journal, and we’ll do that through ObzenFlow’s CLI and the show command.

Journal architecture

Understand what ObzenFlow guarantees and what it doesn’t, how it stores data, and where you can find it.

Event-sourced journals

An ObzenFlow flow runs as one process, and everything the process decides is written to journals.

ObzenFlow is event-sourced at its root. Every stage is a Mealy machine, a state machine that produces its output on each transition. When an event arrives from the journals before it, the stage works out its next state and what to do, and appends the result to its own journal as new events. Those events are the stage’s history, and its state is rebuilt from them. That’s why replay and resume can reconstruct a run, and why a charge whose outcome is already in the journal never goes to the gateway again.

A supervised Mealy state machine reading events from one journal, calculating the next state and actions, then writing new events to its own journal.

By default the journals are files on disk. Every append is synced to storage before the stage moves on, so a fact a stage has written survives a killed process and a power cut. The disk can outlive the machine too.

You choose where the journals go. The flow names a location in its journals: line, and disk_journals writes every run under it.

flow.rs
flow! {
    name: "payment_gateway_resilience_demo",
    journals: disk_journals(PathBuf::from("target/payment-gateway-logs")),

    stages: {
        // …
    },

    topology: {
        // …
    }
}

An operator can move them without touching the code. Set root under [journal] in the flow’s config file, and every run writes under that directory instead.

obzenflow.toml
[journal]
root = "/mnt/journals"

Point root at a volume that can be attached to another machine, and that machine can pick the run up from its journals.

The location isn’t the only thing you can swap. ObzenFlow also ships memory-backed journals. They aren’t used for real workloads, because nothing in them outlives the process, but they make a good double during testing, and they show that the journals behind a flow are swappable. Swapping them is one line.

flow.rs
flow! {
    name: "payment_gateway_resilience_demo",
    journals: memory_journals(),
    // …
}

Memory journals have no location and never sync, so a flow that records to them refuses to start when its config file sets root or sync under [journal].

Another use of memory journals is performance testing. They eliminate a lot of the safety-based performance penalty, like fsync on append. If you’re tracking down a performance optimization in your own application, this is also a legitimate use for the memory journals.

The journals are also the boundary between stages. A stage never calls the next one. It appends to its own journal, and the stages after it read that journal at their own pace. That’s the asynchronous boundary. Whatever one stage handed to the next is in a journal on disk, where it outlives the process.

Inside the manifest

Every run of an ObzenFlow application creates all of its stage journals inside a self-contained manifest. This enables easy archival storage long term and quick retrieval at any time in the future to inspect or play back a previous run. Let’s explore what’s inside a run’s manifest. Here’s an example of the payment gateway resilience example.

If you run ls or tree on the manifest directory of the payment gateway resilience example, you should see 39 files. If we group them by what they contain, they follow one pattern that we’ll cover below.

1flow_01M7H7NPKCNKY87R8P9N65E8J1/
2├── run_manifest.json
3├── system.log
4├── system.identity.json
5├── Transform_authorize_payment_stage_01M7H7NPK6HM6T7H9MK2GF4W97.log
6├── Transform_authorize_payment_stage_01M7H7NPK6HM6T7H9MK2GF4W97.identity.json
7├── Transform_authorize_payment_error_stage_01M7H7NPK6HM6T7H9MK2GF4W97.log
8├── Transform_authorize_payment_error_stage_01M7H7NPK6HM6T7H9MK2GF4W97.identity.json
9└── … the same four files for each of the other eight stages
  • Line 2 is the manifest file, which ties every journal back to the flow.
  • Lines 3 and 4 are the pipeline’s journal, which records the run itself, and its identity file.
  • Lines 5 to 8 are one stage’s files. A journal’s name is the stage’s kind, the name you gave it in flow! and its id in this run, so Transform_authorize_payment_stage_… is the authorize_payment stage’s journal. _error_ marks its error journal, and each journal has an .identity.json beside it naming the run and the journal it belongs to.

Nine stages with two journals each, plus the pipeline’s, make nineteen journals.

Info · What is the system journal?

The system journal is actually like a stage journal. The system itself runs an event-sourced finite state machine that is basically a supervisor of all other stages, so system.log follows the exact same format as all of the stage journals. If you’re familiar with actor-based systems, the concept is the same, essentially: every single stage is essentially an event loop, but you need an outer loop to drive all of the stages. That’s what the system log represents: the main system event loop. All of the decisions that come out of that loop land in the system journal, which you can inspect like any other journal.

Let’s now dive a little bit into the run manifest. run_manifest.json is plain JSON, and it ties each journal back to the flow itself. This is an important piece of metadata, because replay and resume both start from it. Here’s the payment stage’s entry.

run_manifest.json
 1"authorize_payment": {
 2  "dsl_var": "authorize_payment",
 3  "stage_type": "Transform",
 4  "is_effectful": true,
 5  "stage_id": "stage_01M7H7NPK6HM6T7H9MK2GF4W97",
 6  "stage_logic_version": "payment-gateway-v2",
 7  "data_journal_file": "Transform_authorize_payment_stage_01M7H7NPK6HM6T7H9MK2GF4W97.log",
 8  "error_journal_file": "Transform_authorize_payment_error_stage_01M7H7NPK6HM6T7H9MK2GF4W97.log",
 9  "inbound": [
10    "validate_order"
11  ],
12  "delivery_order": "single_input"
13}

Here’s an overview of above:

  • Lines 1 and 2 name the stage as you declared it in flow!.
  • Lines 3 and 4 say it’s a transform that performs effects.
  • Lines 7 and 8 name its two journal files.
  • Lines 9 to 11 list validate_order, the stage it reads from.

Every stage has an entry like this, so the manifest holds the whole topology. That’s the map. Somewhere in those nineteen journals is what happened to the order in manual review.

To recap, you’ve learned about the journals themselves and what they represent. You’ve learned about the manifest and the files that it writes. You can easily see plain JSON files yourselves, but what you’ll notice if you try to inspect the log files is that they are binary data. That’s because ObzenFlow uses a custom codec. We’ll explain the rationale for that and how to use our CLI, the ObzenFlow CLI tool, to inspect the binary logs next.

Investigate journals with the CLI

The disk journal that ships with ObzenFlow serializes its records with a custom codec, so tools like cat and jq can’t read the journal files. You read them through the ObzenFlow CLI.

Info · Why a custom codec?

The codec is a trade-off. ObzenFlow journals nearly everything, and every record is a wide event, so without a custom codec the storage would be prohibitively expensive. There are other optimizations that reduce storage, which we’ll share later on. Other formats and journal backends are planned too, and the disk journal is the first step towards them.

The CLI ships in the main repository behind the cli feature, and it never changes the journal files it reads. show reads them in one of two modes.

  • Snapshot, the default, views a run as recorded, whether it ended a minute ago or a year ago.
  • Follow, with --follow, tails the journals while the run is live, and keeps reading until the run settles or you detach with Ctrl-C.

Its commands cover what you’ll need after an incident.

  • show prints every stage’s records as one causal timeline, with what caused each one.
  • inspect reads the manifest and tells you how the run ended, which journal file holds each stage, and where every record sits.
  • verify compares a replay with its original run, and checks a simulated system’s call log against the runs that called it.

obzenflow show is the primary tool that you’ll use to view exactly what happened in any given journal. It’s the main way that you’ll work with show. You can run it against an entire journal, and you’ll see the full data about what happened. Let’s do that. We’ll strip out the main contents of the journal to begin with and just show you the header and the footer below.

$ cargo run -p obzenflow --features cli --bin obzenflow -- \
    show target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1
Observing flow_01M7H7NPKCNKY87R8P9N65E8J1 · snapshot

Journal numbers:
  1 pipeline · 2 store_orders · 3 web_orders · 4 authorize_payment · 5 validate_order
  6 cancelled_orders · 7 declined_orders · 8 invalid_orders · 9 manual_review
  10 paid_orders · 11 store_orders/error · 12 web_orders/error
  13 authorize_payment/error · 14 validate_order/error · 15 cancelled_orders/error
  16 declined_orders/error · 17 invalid_orders/error · 18 manual_review/error
  19 paid_orders/error

┄┄┄┄┄┄┄┄  1,699 lines of records cut  ┄┄┄┄┄┄┄┄

103 displayed; 313 runtime entries hidden (--include-runtime).

MANIFEST     run_manifest.json
  Flow       payment_gateway_resilience_demo
  Run        flow_01M7H7NPKCNKY87R8P9N65E8J1
  Created    2026-11-16 03:51:04 UTC
  ObzenFlow  0.4.0
  Schema     23.0

JOURNALS
  System journal       Entries
  system.log                 6

  Stage                 Data records  Error records
  store_orders                    12              0
  web_orders                      13              0
  authorize_payment               23              0
  validate_order                  27              0
  cancelled_orders                 2              0
  declined_orders                  4              0
  invalid_orders                   2              0
  manual_review                    1              0
  paid_orders                     18              0

Records in an error journal were delivered but not processed.
Each stage has separate data and error journal files.
The manifest identifies those files and describes the run.
Use --full for the complete manifest.

┄┄┄┄┄┄┄┄  128 lines cut, the clock matrix and each stage's inventory  ┄┄┄┄┄┄┄┄

CLI observed 416 journal entries across 19 journals.
Run completed. CLI reached the end of its snapshot.

Whenever you run show, you’ll see the above. Plus, you’ll see thousands of lines of events, and each one of those is event-sourcing-formatted. Let’s see what one of those events looks like.

Imagine that we have a suspicion that one of the effectful transforms for an authorized payment command had made it to the payment backend. We wanted to see if we received a response and if the payment was authorized. We would likely pipe the output of obzenflow show through less and then type /payment.authorized to search for, let’s say, the first authorization and see if things had at least started successfully.

1$ cargo run -p obzenflow --features cli --bin obzenflow -- \
2    show target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1 \
3    --color always | less -R
4/payment.authorized
5EFFECTFUL TRANSFORM (stage: authorize_payment, journal: 4)
6payment.authorized ← authorize_payment(payment.order_validated)
7⟨1:3,2:4,3:1,4:5,5:3,6:2,7:2,8:2,9:2,10:2⟩
8idempotency key: payment-authorize:store-4471
9payload schema version: 1
10{
11  "amount_cents": 8640,
12  "authorization_id": "AUTH-DEMO-1234",
13  "customer_id": "cust-1187",
14  "order_id": "store-4471",
15  "phase": "Warmup"
16}

This is the first authorization of the run. It contains a ton of useful detail.

  • Line 5 names the stage that wrote the record.
  • Line 7 is its causal clock.
  • Line 8 is the idempotency key its call went out under.
  • Lines 10 to 16 are the payload.

What you’ll also see is that your journals don’t only contain business events, but they also contain runtime events too. The state machine is constantly making decisions on your behalf. If you rerun obzenflow show with the --include-runtime flag, you’ll also see many events such as below.

$ cargo run -p obzenflow --features cli --bin obzenflow -- \
    show target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1 \
    --include-runtime
RUNTIME (stage: authorize_payment, journal: 4)
runtime.circuit_breaker.call_classified ← authorize_payment
⟨1:3,2:4,3:1,4:3,5:3,6:2,7:2,8:2,9:2,10:2⟩
payload schema version: 1
{
  "action": "attempt_settled",
  "unit": "authorize_payment",
  "cursor": {
    "recorded_flow_id": "flow_01M7H7NPKCNKY87R8P9N65E8J1",
    "stage_key": "authorize_payment",
    "input_seq": 1,
    "effect_ordinal": 0
  },
  "attempt": 1,
  "health_classification": "success",
  "slow": false,
  "dependency_elapsed_ms": 116,
  "admission_wait_ms": 0
}

As you saw in the previous unit, when you learned about middleware, that middleware is constantly making decisions. What’s interesting is that, because of event sourcing and wide event philosophy, the record of execution for every decision is captured in the log.

You can see that in this case, the circuit breaker settled store-4471’s call and classified it as healthy. The call succeeded on its first attempt and took 116 milliseconds at the gateway, under the 250 milliseconds the breaker counts as slow, and it waited no time for admission. The cursor names the exact call, the first input authorize_payment read and the first effect it performed for it, which is how this record lines up with the authorization two entries later.

Tip

Add --full to see even more of each record, its complete envelope, including its causality, correlation, processing status, and effect provenance.

Understanding vector clocks

To find out why that order ended up in manual review, you look at everything that happened before it. In a distributed system, wall-clock time can’t tell you reliably what that is. Every machine’s clock drifts, and NTP corrects it by nudging it or stepping it, sometimes backwards. Two events stamped a millisecond apart on different clocks may have happened in either order, or may have no connection to each other at all.

Inside one ObzenFlow process, the stages are in the same position as the machines of a distributed system. Each one runs on its own and learns about the others only through their journals, and a replay or a resume runs at a different time, sometimes on a different machine. So ObzenFlow never orders events by wall-clock time. Every record carries a vector clock instead, its stage’s view of order at the moment the record was written. Each journal keeps its own counter, and no clock is shared across the flow, which gives every journal a total order and the flow a causal order across journals. A stage’s view of another journal only moves when it reads from it, so two stages that never talk to each other have no opinion on which went first, and they don’t need one.

Journal numbers and clocks

Line 7 of the authorization panel is that record’s vector clock. Each entry is a journal number and a counter, and show lists the numbers at the top of its output.

Journal numbers:
  1 pipeline · 2 store_orders · 3 web_orders · 4 authorize_payment · 5 validate_order
  6 cancelled_orders · 7 declined_orders · 8 invalid_orders · 9 manual_review
  10 paid_orders · 11 store_orders/error · 12 web_orders/error
  13 authorize_payment/error · 14 validate_order/error · 15 cancelled_orders/error
  16 declined_orders/error · 17 invalid_orders/error · 18 manual_review/error
  19 paid_orders/error

Numbers 11 to 19 are each stage’s error journal, and in this run they stay empty.

In the authorization above, 4:5 is the record’s own position, the fifth entry in the authorize_payment journal. The other entries say how far this record’s history reaches into every other journal. 5:3 is the third entry of validate_order, and 2:4 is the fourth entry of store_orders.

Counters count every entry in a journal, including the runtime records show hides by default, which is why the first authorization is already entry 5 of its journal.

The happened-before relationship

Every stage maintains its own clock. There’s not a single global clock anywhere in the system, but what’s interesting is that, while every stage maintains its own clock, it actually also tracks what it knows about other stages. This is how clocks can become vector clocks, which are useful in distributed systems, especially in troubleshooting and event-sourced distributed systems.

Let’s take a simple view of two stages in the example that we care about.

Info

This is a space-time diagram, the standard way to picture a distributed system since the late 1970s. Each line is a stage. A filled square is a record the stage wrote to its journal, and a hollow square is the stage reading a record from another stage’s journal. It’s here to help you picture a run. The tool you’ll use is obzenflow show, which prints the same records and clocks as text.

What you’re looking at is two orders going through the payment stage during an ordinary stretch of the run.

validate_order writes payment.order_validated for store-4471. The payment stage reads it, calls the gateway, and three records later writes payment.authorized. The two records in between are runtime events. The circuit breaker classifies the call as healthy, and the resilience policy reports that it took one attempt.

The second order, web-10481, goes the same way with one difference. The rate limiter holds its call first. You limited the payment stage to one call a second in the last part, and this call came less than a second after the first. The stage waits, makes the call, and ends on payment.authorized again.

That’s a healthy stretch. The framework stepped in once, and both orders ended the way you’d want.

Business events and runtime events

Most of your time in the journals will go to reading stretches like this one. An investigation usually starts from an incident, an outage or an anomaly, and the useful question is what led up to it. The answer is nearly always in how two kinds of events sit together. Business events are the facts your code wrote, like an order validated or a payment authorized. Runtime events are everything the framework did around them, like a rate limiter holding a call, a retry being scheduled or a circuit breaker opening.

Here’s the rate limiter’s record from the second order, as show --include-runtime prints it.

$ cargo run -p obzenflow --features cli --bin obzenflow -- \
    show target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1 \
    --include-runtime
RUNTIME (stage: authorize_payment, journal: 4)
runtime.rate_limiter.wait_started ← authorize_payment
⟨1:3,2:4,3:4,4:6,5:4,6:2,7:2,8:2,9:2,10:2⟩
payload schema version: 1
{
  "action": "delayed",
  "unit": "authorize_payment",
  "delay_ms": 869,
  "current_rate": 4.013965936184541,
  "limit_rate": 1.0
}

The payload says what the rate limiter did. It delayed a call by 869 milliseconds to keep the stage at one call a second. To see which order was waiting, read the clock. 5:4 is web-10481’s validation, so that validation happened before the limiter held the call. 4:6 is the record’s own position, three entries before web-10481’s authorization at 4:9.

Tip

When a runtime event catches your eye, read its clock to find the business events that happened before it. When a business event looks wrong, read back through that stage’s journal for the runtime events that led up to it. Trust the clock over the order on your screen, because show has to print records from different stages in some order, even when neither came first.

In the next section you’ll use this on the order in manual review. You’ll start from its record, read its clock, and work back through the runtime events that led up to it.

Root cause analysis

Go back to the alert. One order is sitting in manual review, and the payment stage never got a decision on it. Start with the fact that put it there. Pipe show through less and search for payment.authorization_unavailable.

1$ cargo run -p obzenflow --features cli --bin obzenflow -- \
2    show target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1 \
3    --color always | less -R
4/payment.authorization_unavailable
5EFFECTFUL TRANSFORM (stage: authorize_payment, journal: 4)
6payment.authorization_unavailable ← authorize_payment(payment.order_validated)
7⟨1:3,2:11,3:9,4:58,5:11,6:2,7:2,8:2,9:2,10:2⟩
8payload schema version: 1
9{
10  "amount_cents": 1800,
11  "customer_id": "cust-3741",
12  "order_id": "store-4475",
13  "phase": "Outage",
14  "reason": "processor_timeout_simulated"
15}
  • Lines 5 and 6 say the payment stage wrote this fact from a validated order.
  • Lines 10 to 12 are the order from the alert, store-4475, for $18.00.
  • Line 14 is the reason. The processor timed out.
  • Line 7 is its clock. 4:58 is the record’s own position, and 5:11 is this order’s validation.

That clock narrows the search before you read anything else. 5:11 is store-4475’s own validation, so the payment stage had read nothing past it, and the fourteen orders validated after it are ruled out. 4:58 says the cause is somewhere in the 57 entries the payment stage wrote before this one.

A timeout is the gateway’s side of the story. The resilience middleware is part of the payment stage’s state machine, and every transition it makes is an entry in the stage’s journal. Add --include-runtime and read back from 4:58.

Warning
Just a reminder that the column on the left is the logical clock, not wall clock.
ClockRecordWhat it says
4:50runtime.retry.scheduledattempt 2, after 250 ms
4:52runtime.retry.scheduledattempt 3, after 250 ms
4:53runtime.circuit_breaker.openedthree of five calls failed, open for 5 seconds
4:55runtime.retry.exhaustedthree attempts used
4:57effect.execution_failedthe call for order store-4475 timed out
4:58payment.authorization_unavailablethe fact the command wrote
4:61runtime.retry.scheduledthe next order waits on the open breaker
4:62runtime.circuit_breaker.half_open_enteredthe probe goes out
4:63runtime.circuit_breaker.openedthe probe failed, open for 10 seconds
4:66runtime.circuit_breaker.half_open_enteredthe second probe goes out
4:67runtime.circuit_breaker.closedthe probe succeeded
4:69runtime.retry.succeededthe waiting order is paid

The entry that stands out is 4:53. The circuit breaker opened five entries before the order failed. Here’s that record.

1$ cargo run -p obzenflow --features cli --bin obzenflow -- \
2    show target/payment-gateway-logs/flows/flow_01M7H7NPKCNKY87R8P9N65E8J1 \
3    --include-runtime --color always | less -R
4/circuit_breaker.opened
5RUNTIME (stage: authorize_payment, journal: 4)
6runtime.circuit_breaker.opened ← authorize_payment
7⟨1:3,2:11,3:9,4:53,5:11,6:2,7:2,8:2,9:2,10:2⟩
8payload schema version: 1
9{
10  "action": "opened",
11  "unit": "authorize_payment",
12  "cooldown_ms": 5000,
13  "error_rate": 0.6,
14  "failure_count": 3,
15  "trigger": "failure_and_slow_call_rate",
16  "observed_calls": 5,
17  "slow_call_rate": 0.6,
18  "slow_call_count": 3
19}

This is the breaker you configured in the last part, at the moment it opened.

  • Lines 5 and 6 say what you’re looking at, a runtime record from the payment stage, and the event is the circuit breaker opening.
  • Line 7 is its clock. 4:53 is the record’s own position, and 5:11 is store-4475’s validation, the order the stage was working on.
  • Lines 13 to 18 are why it opened. Of the last five calls, three failed and three were slow, and both rates reached the thresholds you set.
  • Line 12 is what happens next. The breaker stays open for 5,000 milliseconds, the open_for you configured.

Postmortem

Hypothetically, at this point, you would likely have enough data to determine the root cause of what happened. Not only that, but you have the manifest, and in the next unit, you’ll learn how to resume an interrupted flow from it.

Here’s what I might consider writing up for this failure.

Summary. During the overnight payments run, the payment gateway stopped answering for about 27 seconds.

Impact. One order, an $18.00 store purchase (store-4475), could not be authorized and was sent to manual review. The fourteen orders queued behind it were authorized or declined about half a minute late. No order was lost, and every order has exactly one payment decision.

Why. The payment gateway began to time out during this transaction. Four calls to the gateway in a row failed, but that should not have triggered our order, store-4475, to fail completely. Here’s what we learned.

  • This was a misconfiguration on our end. The retry gives each order three attempts, and the circuit breaker needs three failed calls before it opens.
  • So the breaker opened on store-4475’s last attempt. The order had no attempt left to wait with, and the retry gave up.
  • The reason we didn’t lose more orders is that the breaker probed the gateway twice and closed when the second probe succeeded. That’s why only one order went to manual review.
  • This is one outage, and it lasted 27 seconds. One outage can’t tell us how long this gateway’s outages usually run.

What changes.

  • Someone needs to decide store-4475 in manual review.
  • We’ll give the retry a fourth attempt. This doesn’t depend on how long an outage lasts. The order that finds an outage will wait for the gateway like the orders behind it.
  • We won’t change how long the retry waits yet. First we’ll pull every breaker opening and closing from the last three months of runs, to see how long this gateway’s outages really last.

Fixing the resilience policy

The breaker opens on the third failed call, and the retry gives each order three attempts. So an order that finds an outage runs out of attempts on the same call that opens the breaker. There are really two options to fix this, from the retry’s side or from the breaker’s side. You could give the retry a fourth attempt, or you could let the breaker open after two failed calls.

Here’s the fix from the retry side. We need to give the retry a fourth attempt.

flow.rs
let authorize_retry = retry()
    .fixed_delay(Duration::from_millis(250))
    .max_attempts(4)
    .attempt_start_window(Duration::from_secs(30));

With four attempts, store-4475’s third call still opens the breaker. The order now has an attempt left, so the retry waits for the breaker instead of giving up, and it starts over with a fresh four attempts when the breaker lets calls through again. The cost is one more call for an order that keeps failing while the breaker stays closed.

You could fix it from the breaker’s side instead, by letting it open after two failed calls. Two failed calls out of five is a rate of 0.4, so the breaker opens while the order still has its third attempt.

flow.rs
let authorize_breaker = circuit_breaker()
    .count_window(5)
    .minimum_calls(5)
    .failure_rate_threshold(0.4)
    .slow_call_duration(Duration::from_millis(250))
    .slow_call_rate_threshold(0.5)
    .open_for(AUTHORIZE_BREAKER_OPEN_FOR)
    .open_for_multiplier(AUTHORIZE_BREAKER_OPEN_FOR_MULTIPLIER)
    .max_open_for(AUTHORIZE_BREAKER_MAX_OPEN_FOR)
    .probes(1);

The cost is a more sensitive breaker. Any two failures among the last five calls now stop every order for five seconds, including blips a retry would have ridden out. That’s why the postmortem changes the retry and leaves the breaker alone.

After you change the setting, check it against the next outage.

  • Find where the circuit breaker opened, the way you did above.
  • Read the payment stage’s next few records.
  • If you see runtime.retry.exhausted right behind it, an order ran out of attempts on the call that opened the breaker, and the numbers need another look.

Size the wait from your gateway’s history.

  • One outage is one data point. Every run keeps its journals, so each past run’s breaker opening and closing gives you one more outage and its length.
  • If the longest of them comes close to two minutes, raise max_outage_wait above it.

Next steps

You started out as the operator on rota, looking at an alert for one order in manual review. From that point, you learned a lot about ObzenFlow.

  • How a run is recorded in event-sourced journals, and how the manifest ties them to the flow.
  • How to read those journals with obzenflow show, business events and runtime events together.
  • How vector clocks tell you what happened before what.
  • How to run a root cause analysis on an outage, from the circuit breaker opening to the postmortem and the fix to the resilience policy.

In Resume an Interrupted Flow, the machine itself goes down. You’ll find the run it left behind, resume it, and check the simulated system’s call log to confirm that no payment was decided twice.