Simulating the kill
I assume you don’t have a dead machine or a dying machine that’s dying predictably, so we’re going to have to make do with the healthy machine, and we’re just going to kill it.
Well, we’re not actually going to kill your machine. We’re going to simulate its untimely demise at the very worst possible moment, in the middle of a high-consequence run processing overnight payments. We’re going to make it die exactly at the moment that a payment is in flight, where the request has left our system. It’s somewhere out in the universe at the third-party payment processing system, and then ObzenFlow dies before we hear back. This is exactly what happens during a power cut. No big deal, right?
The way systems die
Systems die in all sorts of profound ways. A power supply catching fire. A burst water pipe overtop your rack. But there’s also system death by a thousand paper cuts.
We have to start thinking about what a dead system is. An alive system might be incorrect, and that’s not what we’re necessarily talking about in this unit. What we’re talking about is a correct system that dies, but the process of its death triggers potential non-deterministic behaviour and “bad things” happening, like data loss, data corruption, duplicated requests, duplicated events, and so forth.
Before we can think about how systems self-heal, we have to think about how systems are destroyed and destroy themselves.
Think about what “a payment in flight” means at the moment the machine dies. We sent a request and never heard back. Any of these could be true.
- The request never made it to the processor.
- It made it, and the processor declined the payment.
- It made it, the processor charged the card, and the answer was on its way back to us when we died.
- The answer made it back, and we died before we wrote it down.
So a lost order might not be lost. It might have been successful. The result might even have been returned to us, but we’re dead. From where we’re standing, all four look exactly the same.
And that list assumes only one thing went wrong. Maybe the network split at the exact same moment. Maybe the power cut took out us and our internet provider. Maybe it took out us and them.
On August 14, 2003, 55 million people across Ontario and eight American states lost power on the same afternoon, and it began with alarm software that had quietly stopped working. So the people watching the grid saw nothing wrong.
Liveness is one of the trickiest problems in systems development. As we move through this tutorial, just realize one thing. Assumptions are dangerous. But you might have to make assumptions, so always assume the worst-case scenario and then prove otherwise.
Let’s move forward by staging the scenario we’re going to run through, which is a response that’s lost on the way from the payment processor to our system because of a power failure.
Something’s f*$#y
There’s always that one shift that you knew you shouldn’t have agreed to, but the rota had a hole in it, and you’re the hero that filled it. So now you’re in the shit. It’s like the movie Platoon, but not.
It’s time for you to be super brave, and we’re going to simulate a power cut, kind of. What we’re actually going to do is force a slow answer. Essentially, we’ll send out a request via a port to the payment gateway backend, and we’re going to purposefully slow down a response. At the right moment in time, wouldn’t it be terrible if you typed pkill or something like that? Just saying.
There are two things to do.
First, remove the simulated system’s call log.
rm -f target/gateway-calls.jsonlThe simulated system keeps its own call log, target/gateway-calls.jsonl, essentially a double of the record Stripe would keep on its side. It’s the evidence you’ll use at the end to prove nobody was charged twice. Remove it before a fresh run and never before a resume, because the final check reads it across both runs.
Second, start the run with the slow-answer config.
cargo run -p obzenflow --features web-host --example payment_gateway_resilience -- \
--config examples/payment_gateway_resilience/obzenflow.slow-answer.toml --server--features web-host builds in an HTTP server, and --server starts it on port 9090. You’ll need it in a minute.
That config file is the example’s own with one line changed, processor = "slow_answer". It swaps the system behind the payment port for one that takes five minutes to answer the sixth payment. Five decisions go by, and then the run prints this.
⏳ Oh no, this is taking a long time. The processor has decided order web-10483 (customer cust-2958, amount: $64.00) and its answer is on the way. Wouldn't it be terrible if the power went out right now, or if you typed pkill -9 payment_gateway?This is the moment we set up. The processor has charged a customer $64.00, and its answer is on its way back. Left alone, it arrives in five minutes and the night carries on. You have that long to cut the power.
The sh*t clock’s ticking
You’ve got five minutes before that answer lands, which is plenty. Before the lights go out, let’s have a look at what this thing is actually doing. Open a second terminal and find the run’s directory.
$ ls -lao target/payment-gateway-logs/flows/
total 16
drwxr-xr-x 4 kw 4096 Nov 15 22:55 .
drwxr-xr-x 3 kw 4096 Nov 15 22:51 ..
drwxr-xr-x 2 kw 4096 Nov 15 22:51 flow_01M7H7NPKCNKY87R8P9N65E8J1
drwxr-xr-x 2 kw 4096 Nov 15 22:55 flow_01M7H7YJEE4ZP02B1QTHZG2VW1
The names sort oldest to newest. The older one is your first run, and the newest is the one that’s stuck waiting right now. Your run has its own id, so use your path wherever this page uses this one.
Now watch it. --follow keeps reading as the run writes, and --compact prints one line for each record.
$ cargo run -p obzenflow --features cli --bin obzenflow -- \
show target/payment-gateway-logs/flows/flow_01M7H7YJEE4ZP02B1QTHZG2VW1 \
--follow --compact
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM order.invalid ← validate_order(commerce.customer_order_placed) {"amo… [--full]
TRANSFORM order.cancelled ← validate_order(commerce.customer_order_placed) {"a… [--full]
DELIVERY delivery.succeeded ← invalid_orders(order.invalid) {"subject":{"input… [--full]
DELIVERY delivery.succeeded ← cancelled_orders(order.cancelled) {"subject":{"i… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
Follow dumps everything the run has written so far, and then it just sits there. Look at the last few lines. Those are validated orders with no payment decision after them. That’s seventeen customers standing in line behind one slow answer, and nothing more is coming while that answer is in flight.
show prints the same records at different levels of detail. Here’s what each flag does to one payment authorization.
--compactprints it on one line.- With no flag it’s 12 lines, with its clock, its idempotency key and its payload.
--explainadds a note under it that says why the record is there.--fullprints the complete envelope, which is 340 lines.--jsonlprints it as one line of JSON, for another program to read.
--include-runtime works with any of them and adds the runtime’s own records.
Before you pull the plug, check that it’s alive. The run serves a health check on port 9090, so ask it from a third terminal.
$ curl -i http://127.0.0.1:9090/health
HTTP/1.1 200 OK
content-type: text/plain
content-length: 2
date: Mon, 16 Nov 2026 03:56:09 GMT
OK
200 OK means the process is up and answering, slow answer and all. There’s also /ready, which answers READY only while the sources are still reading. Every order was read long before the slow answer, so this run answers it 503 NOT_READY.
Now kill the flow from the same terminal.
pkill -9 payment_gatewaykill -9 gives the process no chance to shut down, which is what a crash looks like. The processor’s answer is still on its way, and now there’s nothing for it to arrive at.
Go back to the second terminal. This is what it shows.
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM order.invalid ← validate_order(commerce.customer_order_placed) {"amo… [--full]
TRANSFORM order.cancelled ← validate_order(commerce.customer_order_placed) {"a… [--full]
DELIVERY delivery.succeeded ← invalid_orders(order.invalid) {"subject":{"input… [--full]
DELIVERY delivery.succeeded ← cancelled_orders(order.cancelled) {"subject":{"i… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
TRANSFORM payment.order_validated ← validate_order(commerce.customer_order_plac… [--full]
Those are the same three validated orders it ended on before the kill. Nothing new has arrived, and nothing has told you the run died. Follow is still waiting for the next record, the way it would for a run that’s only slow. That’s what a dead machine looks like from the outside.
Kill the CLI
Remember that the CLI is its own process, and it’s still running, so let’s kill it. It’ll basically just sit there forever now, waiting for a record that’s never coming. Press Ctrl-C to stop it. It prints a summary of what it read as it lets go of this cold, cruel world.
^C
CLI observed 210 journal entries across 21 journals.
Run outcome not recorded in the observed entries. CLI stopped observing; execution is
independent.
The first line is a count. Follow read 210 entries across 21 journals, which is one journal for the run itself, two for the metrics the server exposes, and a journal and an error journal for each of its nine stages.
Your count will likely differ, and that’s observability. Beside the facts, the journals hold the runtime’s own records. While the server is on, that includes a metrics snapshot, checked four times a second and written only when a stage has recorded something new, so the count depends on your machine’s timing. Nothing new is recorded while the run waits, so the count stays the same however long you wait before you kill it.
The facts are what matter. If you ever want to know whether two runs really differ, replay one with --verify, and ObzenFlow compares its facts with the original’s, stage by stage. Observable by Design covers this and the CLI’s other commands.
The second line is the one to read closely. A run that finishes records its outcome, and follow stops by itself and prints Run completed. This run died before it could record one, so follow can only say Run outcome not recorded and that you stopped it. Stopping it never touches the run, because follow only ever reads.
Notice what it didn’t say. It doesn’t say that the run is dead.
Liveness and determining the time of death
From the outside, a dead process and one you can’t reach look the same. You just watched that happen, when the follow stream looked the same before the kill and after it. Resuming a run that’s still alive would put two processes to work on the same orders, so settle this before you touch anything.
Exploring the health check endpoint
ObzenFlow does have a health check endpoint, so that’s how you would determine the state of a current run. The CLI purposefully works with journals, but not the live process, for a separation of concerns. If you want to verify whether or not the process is still running and what state it is in, you would emit an HTTP GET to the health check endpoint.
So ask it again, from the third terminal.
$ curl -i http://127.0.0.1:9090/health
curl: (7) Failed to connect to 127.0.0.1 port 9090 after 1 ms: Couldn't connect to server
Nothing answers this time, because nothing is left to answer.
On this machine you can also ask the operating system.
pgrep -l payment_gatewayIt prints nothing, so no process by that name is running.
How different parts of the system help to determine liveness
This is a good moment to be clear about which piece does what.
- Your application is the payment example you started in the first terminal. ObzenFlow is a library, so the framework runs inside it. It runs the flow, writes the journals and serves the health check.
- The CLI is a separate program, the
obzenflowbinary you ranshowwith. It only reads journal files and takes no locks, which is how it followed your run without getting in its way.
Understanding locks and requirements before resuming
The journal is ObzenFlow’s asynchronous boundary. Stages hand work to each other through journals, and resume picks a run back up from them. That only works if every journal has exactly one owner. In this tutorial the journals are disk journals, each one a file on your machine, and file locks are how ObzenFlow makes sure of it. They stop two processes from writing the same journal, and they stop a resume from taking over a run another process still owns. Without them, resuming a run that’s still alive would put two processes to work on the same orders.
While your application runs, it holds a lock on each of its journal files. The instant the process exits, the operating system drops those locks. It doesn’t matter how the process exits. A clean shutdown, a crash and a kill -9 all release them the same way.
Resume is your application again, started with --resume-from, and the first thing it does is look at those locks. If any are held, the run is alive, and resume refuses. If none are, the run is dead, and resume goes ahead.
There’s no timing involved. No timeout, no heartbeat, no grace period. The operating system knows whether the process exists, so on one machine the check can’t be wrong, and there’s nothing to wait for. Your run is dead, so when you resume it, the check passes.
Its limit is that locks don’t travel between machines. If you resume from a copy of the journals on another machine, there’s nothing on the original to check, and deciding the original is dead is your job. If you can’t tell, make it dead before you resume from the copy.
Not yet. ObzenFlow is pre-production, and one of the reasons is that its self-healing features aren’t built. Liveness is a very challenging, very complex undertaking. Deciding when a process is really dead brings split brain and all sorts of related problems. Right now ObzenFlow runs on a single node, and it’s intended for a single node, but in the future it will be able to heal itself. Until then, resume is a command an operator runs, and this part is about understanding the failure modes deeply enough to run it safely.
Performing the autopsy
The process is gone, but everything it wrote is still on disk. Now it’s time to throw on our site reliability engineering hats and get down to business. When things go wrong, the journal is the source of truth, and ObzenFlow ships with multiple tools for working with journals. The CLI has an obzenflow show command and an obzenflow inspect command. You’ll learn how to use both for different types of troubleshooting.
The sync setting in a config’s [journal] table decides when a disk journal acknowledges a write. The default, before_ack, acknowledges nothing until it’s synced to the disk itself, past the operating system’s memory, so every acknowledged record survives a crash, a kernel panic or a power cut. off acknowledges as soon as the operating system holds the bytes. It’s faster, and a power cut can lose records it already acknowledged.
Plenty of systems make the faster choice their default and chase throughput. ObzenFlow defaults to correctness. That matters for this page, too. A kill -9 can’t touch the operating system’s memory, but a real power cut can, and with before_ack it wouldn’t matter. Everything your run acknowledged was already on the disk.
Understanding the CLI: show and inspect
The ObzenFlow CLI was created with site reliability engineers in mind, and so it comes with two powerful modes, show and inspect, which are complementary to each other but used for different purposes.
show reads the journals as your application’s history, with facts in causal order and their payloads, clocks and causes. It’s the tool for debugging your own logic, and you’ve been using it all along.
inspect reads them as storage. If you know a database’s WAL dump tool, it’s that, with one difference. A database’s WAL sits behind its tables and is recycled once a checkpoint catches them up. ObzenFlow has no tables behind its journals. The journals are the system of record, so inspect dumps the real thing. It shows which file holds each stage, every committed frame by offset and type, and how the run ended, checked the way replay and resume check it. It’s the tool for debugging the journal layer.
Debugging your logic? Use show.
- You added a validation rule, and want to see that your stage emitted the facts you expected, with the right payloads.
- An order ended up in manual review, and you want to know what produced that record and why.
--explainadds the causal evidence. - You want to watch a run as it happens, one line per record. That’s
--follow --compact. - You want to see how two sources really interleaved. The clocks and the causal order answer it.
- You want to see how your middleware behaved around a call. Add
--include-runtime. - You resumed a run, and want to know which decisions came from the journal and which were made live. You’ll do exactly that further down this page.
- You want the history in
jqor your own tooling. That’s--jsonl.
Debugging the journal layer? Use inspect.
- A machine died, and you want to know exactly where each stage stopped writing. That’s this autopsy.
- You’re about to resume or replay an archive, and want to know its files are intact.
inspectreads them with the checks replay uses, and fails loudly at the file and offset if something’s damaged. - You want to know how a run ended, straight from its own journal. That’s the
statusline. - You need the file that holds one stage, to attach to a bug report or confirm it was backed up. Add
--stage. - You’re upgrading ObzenFlow and want to know which journal schema wrote an archive. That’s the
schemaline. - You want to know whether a particular kind of record was committed at all. Add
--event-type.
An autopsy is a storage question. You want to know exactly where the payment stage stopped writing, so this one’s a job for inspect.
Where the payment stage stopped
Treat this like a production alert, and work backwards from it. Here’s what you know. The machine died, and its health checks failed, which is how you’d have found out. You’ve already confirmed it’s dead. What you don’t know is whether any payments were dropped.
Here’s what’s interesting. When a payment was in flight and no response came back, it’s still safe to resume, whether or not the processor charged it.
That’s ObzenFlow’s effect system and its idempotency keys at work. The processor charged the payment, the machine died before the answer came back, and resuming won’t charge the customer twice. Now it’s time to prove it. Let’s start by inspecting the authorize_payment stage. This list might look overwhelming, but it’s fairly simple. You’ll see a series of events, both your own business events plus the system’s runtime events, prefixed with a byte offset.
1$ cargo run -p obzenflow --features cli --bin obzenflow -- \
2 inspect target/payment-gateway-logs/flows/flow_01M7H7YJEE4ZP02B1QTHZG2VW1 \
3 --stage authorize_payment
4flow_id: flow_01M7H7YJEE4ZP02B1QTHZG2VW1
5flow_name: payment_gateway_resilience_demo
6status: Unknown
7schema: 23.0
8stages: 9
9
10[authorize_payment] Transform_authorize_payment_stage_01M7H7YJE9Y7E5GTG7X5K5BFMH.log (Transform)
11 0 supervisor.stage.authorize_payment.registered
12 510 supervisor.stage.authorize_payment.ready
13 1013 runtime.circuit_breaker.call_classified
14 1013 runtime.resilience.attempts_reported
15 1013 payment.authorized
16 3575 runtime.rate_limiter.wait_started
17 3575 runtime.circuit_breaker.call_classified
18 3575 runtime.resilience.attempts_reported
19 3575 payment.authorized
20 6472 runtime.contract.verification_pending
21 7333 runtime.subscription.progress_reported
22 8781 runtime.rate_limiter.wait_started
23 8781 runtime.circuit_breaker.call_classified
24 8781 runtime.resilience.attempts_reported
25 8781 payment.authorized
26 11646 runtime.contract.verification_pending
27 12507 runtime.subscription.progress_reported
28 13955 runtime.rate_limiter.wait_started
29 13955 runtime.circuit_breaker.call_classified
30 13955 runtime.resilience.attempts_reported
31 13955 payment.declined
32 16878 runtime.contract.verification_pending
33 17739 runtime.subscription.progress_reported
34 19187 runtime.rate_limiter.wait_started
35 19187 runtime.circuit_breaker.call_classified
36 19187 runtime.resilience.attempts_reported
37 19187 payment.declined
38 22076 runtime.contract.verification_pending
39 22937 runtime.subscription.progress_reported
Lines 4 to 8 describe the run. status: Unknown means it never recorded how it ended, the same thing follow told you when you stopped it. Everything from line 10 down is the payment stage’s journal, one record per line.
- Lines 34 to 37 are the fifth payment. They share an offset, 19187, because the stage appended them together, as one frame.
- Lines 15, 19, 25, 31 and 37 are the five decisions, three authorized and two declined.
- Lines 38 and 39 are the runtime’s bookkeeping after the fifth decision, and they’re the last thing the stage ever wrote.
Every append writes one frame. A frame holds a single record, or a group of records that belong together, like a decision and its evidence. It carries a checksum and ends with a fixed trailer, and the trailer is what commits it, so every record in the frame becomes visible at once. A reader counts a frame only once it checks out, so when a frame is cut off part-way through, by a crash or a power cut, none of its records ever reach a reader. If an append fails while the process is still alive, the journal is cut back to where it was before the append began.
Either way, a decision never lands without its evidence. It’s in the journal with the limiter’s wait, the breaker’s verdict and the attempt count, or none of them are there.
There’s no sixth decision. The request for web-10483 left your system, and the process died before the stage could write anything about it.
This is an important distinction. ObzenFlow journals events, not commands. An event is a fact, something that already happened, like an order being validated or a payment being declined. A command is the work a stage does in response, like authorize_payment sending a request to the processor. Commands are ephemeral, and ObzenFlow’s machinery doesn’t need them written down. The order’s validation is in the journal, so resume can always rebuild the same request from it, under the same idempotency key, and replaying an event can’t charge anybody. The cost is that, for a keyed call like this one, the journal has nothing to say about the request until its answer comes back and becomes an event. From here, you can’t tell whether the request never left, or whether the processor charged the card and the answer died on the way back.
The journal doesn’t bend around what you’d like to see in a log. It records the facts the run is built from and recovers from. If you’d like a record of every request that goes out, your application’s logs are a good home for it. Look back at the run’s output and you’ll see exactly that. The only line that mentions web-10483 is the one the example printed itself, and nothing ObzenFlow logged mentions the call at all.
Reviewing the simulated call log
Because this is a simulation, we have an advantage. We have a log of what the payment system itself saw. That’s how we’ll verify that the processor really did charge the missing payment, and later prove that it was safe to send it again.
The log isn’t part of ObzenFlow’s journals. It belongs to the simulated payment processor, the stand-in for Stripe behind the payment port.
Like a real processor, the simulated processor built with ObzenFlow’s simulator behaves like a real processing backend in the sense that it maintains a record of keys (idempotency keys) and just does a very lightweight comparison (just like any third-party backend system that supports idempotency keys, like Stripe does). Here’s basically how the simulator works.
- It receives a request and checks the idempotency key against the keys it has already decided.
- If the key is new, it processes the request, which charges or declines the payment. It logs the outcome as
decided, then returns it. - If it has decided the key before, it doesn’t process the request again. It logs
replayedand returns the answer it gave the first time, so a repeated request never charges twice. - If processing fails, as it does during the scripted outage, it logs
failedand leaves the key undecided, so the next attempt is processed as new. - Each of those is one line in
target/gateway-calls.jsonl, with the key and the run that sent it, written the moment it happens, so the log survives a process that gets killed.
Let’s tail the log and then inspect the last two lines.
1$ tail -n 2 target/gateway-calls.jsonl
2{"result":"decided","key":"payment-authorize:store-4473","flow_id":"flow_01M7H7YJEE4ZP02B1QTHZG2VW1","response":{"authorization":"declined","reason":"AddressMismatch"}}
3{"result":"decided","key":"payment-authorize:web-10483","flow_id":"flow_01M7H7YJEE4ZP02B1QTHZG2VW1","response":{"authorization":"charged","authorization_id":"AUTH-DEMO-1234"}}
- Line 2 is store-4473, the fifth payment. The processor declined it, and that’s the last decision in your journal.
- Line 3 is web-10483, the sixth. The processor charged it $64.00. This is the response we never received, which the backend would have sent after the
pkill.
Your journal holds five decisions and the processor holds six. A customer has been charged, and your side has no record of it. That’s the call that could charge them twice, but now we’ll prove that even after we resume the flow, it won’t. We will finally receive a verdict on the payment, plus it won’t charge them twice.
Resume the run
Resume finishes a run that ended early. You point it at the dead run’s directory, and that run’s journals become its input. It never writes to them. It starts a new run, with journals of its own, that picks up the history exactly where the dead run left off.
It works in two phases.
- Every stage catches up on what the dead run recorded. The facts come back from the journals, and a call with a recorded outcome isn’t made again, so the five payments that already have a decision are read, not re-sent.
- Once a stage reaches the end of what the dead run recorded, its high-water mark, it switches to live work. web-10483 is the first payment past that mark. It has no recorded outcome, so it goes to the processor again under the same idempotency key, and the processor’s memory does the rest.
So now that we have a handle on the incident, the symptoms, and the cause (which was a power outage), it’s time to recover. This is the final phase of any incident, and ObzenFlow has a single command to make this happen, --resume-from. Here’s what we’re going to do.
- Resume the dead run with
--resume-from, and watch it catch up and finish the night. - Check the simulated call log against both runs, to prove no one was charged twice.
- Read the resumed run’s journals, to see which decisions came back from the journal and confirm the night ended with the same counts as your first run.
Resume the dead run
The run is dead, and you know where it stopped. Resume it with the same config, and pass its directory to --resume-from, from the first terminal. The processor answers the repeated call at once.
1$ cargo run -p obzenflow --example payment_gateway_resilience -- \
2 --config examples/payment_gateway_resilience/obzenflow.slow-answer.toml \
3 --resume-from target/payment-gateway-logs/flows/flow_01M7H7YJEE4ZP02B1QTHZG2VW1
4Resuming from flow_01M7H7YJEE4ZP02B1QTHZG2VW1 at target/payment-gateway-logs/flows/flow_01M7H7YJEE4ZP02B1QTHZG2VW1. Recorded facts catch up first, then the run continues live at the recorded high-water mark.
5…
6[replay] 📦 Paid order store-4471 is ready for shipping (customer cust-1187, amount: $86.40, auth AUTH-DEMO-1234)
7[replay] 📦 Paid order web-10481 is ready for shipping (customer cust-2041, amount: $48.20, auth AUTH-DEMO-1234)
8[replay] 📦 Paid order store-4472 is ready for shipping (customer cust-7160, amount: $23.75, auth AUTH-DEMO-1234)
9[replay] ❌ Payment declined for order web-10482: insufficient funds (customer cust-3302, amount: $94.99)
10[replay] ❌ Payment declined for order store-4473: billing address mismatch (customer cust-1453, amount: $57.30)
11📦 Paid order web-10483 is ready for shipping (customer cust-2958, amount: $64.00, auth AUTH-DEMO-1234)
- Lines 6 to 10 are the five decisions the dead run had already recorded. They’re marked
[replay], and no gateway call goes out for them. The sinks receive them a second time, which is the repeat each sink declared it could take. - Line 11 is web-10483, the payment whose answer was in flight when the power went. It has no mark, because it’s a live call. The dead run recorded no decision for it, so the resumed run sends it to the processor again, under the same idempotency key.
Your terminal also prints [replay] lines for the invalid and cancelled orders. They can land before or after web-10483.
Flipping from replay to live
What you’re seeing here is the actual console output from your program. --resume-from is the act of running your application again, so what you will see is your own console logging. This is not the ObzenFlow show tool. This is not a tale of the journal. This is your program’s console output.
The order lines are what the example’s console sinks print as the resumed run delivers each decision, from the shipping handoff and the declined-orders sink. The [replay] prefix is added by ObzenFlow’s console sink to any delivery that came back from the dead run’s journal, which is why web-10483 has none.
If you’re new to event sourcing, this might seem like magic, or too good to be true. It’s really automata theory, which predates even relational databases. Mealy and Moore described the machines behind it in the 1950s.
Behind the scenes, ObzenFlow replays the dead run’s journals. Every stage runs its own catch-up state machine. While it’s rebuilding, the stage folds the recorded facts back in, in the same causal order the live run used, and takes each recorded outcome from the journal instead of making the call again.
When a stage reaches its high-water mark, it writes its own catch-up watermark to its journal and flips to live. For the payment stage, that’s the moment it runs out of recorded decisions, just before its first live call, and you can watch it happen. Your terminal prints stage crossed its catch-up boundary and authored its own watermark stage_name=authorize_payment shortly before the web-10483 line. Once the last stage is live, the pipeline records reconstruction_completed in the run’s system journal, and from then on your flow runs exactly as if it had just started.
From there the rest of the night runs live, scripted outage included, and about 45 seconds later the run finishes.
1payment_gateway_resilience_demo completed (resumed from flow_01M7H7YJEE4ZP02B1QTHZG2VW1 at target/payment-gateway-logs/flows/flow_01M7H7YJEE4ZP02B1QTHZG2VW1).
2Recorded facts were caught up with effects suppressed, then the run continued live from the recorded high-water mark.
3Journal: target/payment-gateway-logs/flows/flow_01M7H7Z5CAWVQYK5M0E8TBHABHLine 3 names the resumed run’s own directory. Resume never writes to the dead run’s journals. It reads them and writes its own.
No checkpoint database, broker or orchestration service decided where to continue. The journals that made the run durable are also the record of how far it got.
A run can be resumed once. Run the same command again and it’s refused, and pointed at the run that took over.
cannot resume …/flows/flow_01M7H7YJEE4ZP02B1QTHZG2VW1: it was already resumed by …/flows/flow_01M7H7Z5CAWVQYK5M0E8TBHABH, the head of this history.
Resume …/flows/flow_01M7H7Z5CAWVQYK5M0E8TBHABH instead.That keeps one history. Two resumes of the same run would be two processes deciding the same orders.
Check the simulated call log
Your journals show what came back. Only the processor’s own record shows what it actually did, so the last step checks the simulated call log you reviewed earlier against both runs, the one that died and the one that resumed it.
cargo run -p obzenflow --features cli --bin obzenflow -- \
verify calls \
--run target/payment-gateway-logs/flows/flow_01M7H7YJEE4ZP02B1QTHZG2VW1 \
--run target/payment-gateway-logs/flows/flow_01M7H7Z5CAWVQYK5M0E8TBHABH \
target/gateway-calls.jsonlok: target/gateway-calls.jsonl holds 27 calls from this lineage, 1 repeated by a resume.Twenty-seven is the 22 payments that got a decision, the four scripted timeouts, and one repeat. The check passes only when every call belongs to that lineage, no payment was decided twice, and each resume repeated at most the one call its kill interrupted.
The repeat is the call you’ve been following. Ask the call log for it.
grep web-10483 target/gateway-calls.jsonl1{"result":"decided","key":"payment-authorize:web-10483","flow_id":"flow_01M7H7YJEE4ZP02B1QTHZG2VW1","response":{"authorization":"charged","authorization_id":"AUTH-DEMO-1234"}}
2{"result":"replayed","key":"payment-authorize:web-10483","flow_id":"flow_01M7H7Z5CAWVQYK5M0E8TBHABH"}- Line 1 is the dead run’s call. The processor decided it and charged the customer.
- Line 2 is the resumed run’s call for the same key. The processor recognized the key and answered from its own record.
replayedis its word for that, and it decided nothing.
The customer was charged once. The resumed run got the answer the dead run never received, and went on with the night.
Your kill landed at the one moment that needs the processor’s help. There are four places it can land.
| Where the kill lands | What the processor has | What resume does | What the check prints |
|---|---|---|---|
| Before the call goes out | Nothing | Sends it for the first time | 0 repeated |
| Call in flight, not yet decided | Nothing | Sends it, and it’s decided once | 0 repeated |
| Decided, answer not yet journaled | One decision | Sends it again under the same key and gets the recorded answer | 1 repeated |
| Answer journaled | One decision | Reads the journal and makes no call | 0 repeated |
Yours was the third row. In the other three, your journal and the processor’s record already agree.
Read the resumed run’s journals
The resumed run’s journals show the last row. Run show on it, and the five decisions the dead run made are tagged [read from journal].
$ cargo run -p obzenflow --features cli --bin obzenflow -- \
show target/payment-gateway-logs/flows/flow_01M7H7Z5CAWVQYK5M0E8TBHABH
EFFECTFUL TRANSFORM (stage: authorize_payment, journal: 4)
payment.authorized ← authorize_payment(payment.order_validated) [read from journal]
⟨1:3,2:4,3:1,4:5,5:3,6:2,7:2,8:2,9:2,10:2,20:3,21:2,22:1,23:2,24:2,25:2,26:5,27:2,28:3,
29:4,30:2⟩
idempotency key: payment-authorize:store-4471
payload schema version: 1
{
"amount_cents": 8640,
"authorization_id": "AUTH-DEMO-1234",
"customer_id": "cust-1187",
"order_id": "store-4471",
"phase": "Warmup"
}
This is store-4471’s authorization. The dead run made that call, and the resumed run read its outcome from the journal. Its clock carries both runs’ journals, 1 to 10 for the resumed run and 20 to 30 for the dead one.
The decisions made after the resume carry no tag, and that includes web-10483. In your journal it looks like any other live decision. Only the processor’s record shows that it had seen that key before.
Scroll to the end, and the night is complete.
EFFECTFUL TRANSFORM: authorize_payment
…
Count Event Author
1 execution/effect.execution_failed@1 authorize_payment
1 fact/payment.authorization_unavailable@1 authorize_payment
18 fact/payment.authorized@1 authorize_payment
4 fact/payment.declined@1 authorize_payment
…
CLI observed 368 journal entries across 19 journals.
Run completed. CLI reached the end of its snapshot.
All 23 validated orders have exactly one decision. Eighteen were authorized, four were declined, and one, store-4475, went to manual review during the scripted outage you investigated in the last part. Those are the same counts as your first run.
Idempotency and exactly-once processing
Resuming isn’t completely free, though. In order to safely resume a flow that’s died or somehow ended abruptly or unexpectedly, one of two things needs to be true.
- The system you’re integrating with supports idempotency (for instance, with idempotency or deduplication keys).
- You manually ensure idempotency.
Obviously, it’s fantastic when a system you’re integrating with has the concept of idempotency support. For instance, Stripe accepts an idempotency key natively. No muss, no fuss. Integrate with it at your leisure. But what if you’re integrating with Joe Dirt’s Payments, and Joe has no concept of idempotency?
Well, to make sure resumes are safe, you need to ask some questions about all of your effects. As long as you can answer the questions, ObzenFlow has the correct settings available at your disposal to accommodate virtually any scenario.
Each effect declares one of two guarantees, exactly-once processing or at-least-once processing. Neither is exactly-once delivery, which no distributed system can promise. web-10483’s request was delivered to the processor twice, and the payment stage ran it twice, once in the dead run and once after the resume. Exactly-once processing means the result is applied once even when the request runs again, so the card was charged once. It rests on at-least-once delivery plus a receiver that recognizes the repeat or isn’t changed by it, which is why some systems call it effectively once. At-least-once processing makes no such promise, and after a crash the call may take effect more than once.
At-least-once processing is explicit. The stage has to acknowledge it with .at_least_once::<Request>(), ObzenFlow journals each attempt before it goes out, and after a crash, resume makes the call again, so it may take effect twice.
Which one to use comes down to what the other end does when the same request arrives twice.
| If the other end, given the same request twice… | Declare | What resume does with a call that was in flight | Watch out for |
|---|---|---|---|
| Changes nothing, like setting a status or an upsert by ID | exactly_once, no key | Sends it again, harmlessly | Operations that only look idempotent, like increments and appends |
| Recognizes the repeat by its idempotency key, like Stripe | exactly_once, with a key | Sends it again under the same key, and the other end answers from memory | The key has to come from a business ID, and the other end has to keep it longer than your recovery takes. Stripe keeps keys for at least 24 hours. |
| Does it twice, and you can live with that, like a notification email | at_least_once, acknowledged on the stage | Makes the call again as a new attempt, so it may take effect twice | Retries are off for that call, and any cleanup is yours |
| Does it twice, and you can’t live with that | Make it idempotent first, then exactly_once | Depends on how you made it idempotent | Check whether it already happened before acting, keyed by a business ID |
In ObzenFlow, that check belongs in the system behind the port, not in extra stages. A stage ahead of the effect can drop a duplicate order, but it can’t know whether a call that was in flight took effect, because only the other end knows that. So the lookup goes inside the call itself. Here’s a sketch of a system for a processor with no idempotency keys. It isn’t part of the example.
1#[obzenflow::port]
2#[async_trait]
3pub trait PaymentGatewayPort {
4 #[effect(type = "payment.authorize", exactly_once)]
5 async fn authorize(
6 &self,
7 request: &AuthorizeRequest,
8 ctx: &EffectContext,
9 ) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError>;
10}
11
12#[obzenflow::system("ledger-processor")]
13#[async_trait]
14impl PaymentGatewayPort for LedgerProcessor {
15 async fn authorize(
16 &self,
17 request: &AuthorizeRequest,
18 _ctx: &EffectContext,
19 ) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError> {
20 if let Some(payment) = self.client.get_payment(&request.order_id).await? {
21 return Ok(payment.outcome_for(request));
22 }
23 let payment = self
24 .client
25 .authorize_payment(request, &request.order_id)
26 .await?;
27 Ok(payment.outcome_for(request))
28 }
29}- Lines 4 to 9 declare the effect
exactly_oncewith no key. The call is idempotent by construction, and the declaration says so. - Lines 20 to 22 ask the processor whether it already has a payment for this order, and return that payment if it does.
- Lines 23 to 27 charge only when there’s none, passing the order ID as the reference, so the next lookup finds it.
Both requests happen inside one effect, and that’s the point. After a crash, resume runs the whole call again, lookup first. Splitting the lookup into an effect of its own wouldn’t work. Its answer would be journaled like any other outcome, and on resume ObzenFlow would replay the old “not found” instead of asking again. The processor also has to support looking a payment up by your reference, and return a charge as soon as it’s made.
There are always edge cases, but that’s what your runbooks are for. If you’re trying to resume a long crashed run, that can get dangerous. Resume is intended for fairly timely resumption of failed workflows, but if you’re trying to resume a batch that failed two days ago, that might pose a problem with backend systems, so your mileage may vary. Stripe, for instance, keeps an idempotency key for at least 24 hours, and may drop it after that.
A planned feature will let an effect declare how long its receiver keeps keys, so that resume won’t send an old in-doubt call again under a key that may have expired.
Recap
You started with a payments run that died partway through. From there, you brought the run back.
- You found the run the machine left behind and saw where each stage had stopped.
- You learned what resume checks before it takes over a run, and the one question it leaves to you.
- You resumed the run, and watched recorded decisions come back from the journal while the rest went out live.
- You proved from the processor’s own call log that the one payment in flight was sent twice and charged once.
- You saw which guarantee fits which kind of backend, and how to make a call idempotent when the backend can’t deduplicate.
ObzenFlow never repeats a journaled outcome. On resume, the only call it sends again is one a crash interrupted before its outcome was recorded. That call goes out under the same idempotency key, so the system behind the port can deduplicate it, while everything deterministic around it simply recomputes.
What you built is the authorization leg of a payment system. AuthorizeRequest producing PaymentAuthorized is an authorization hold, the first half of authorize-then-capture. Capture after fulfilment, and customer-initiated cancellation checked against order state, belong to a later capstone where the order is a state machine. Until then, these pages go deeper into what you used here.
- The example source includes the deterministic fan-in details and complete resilience wiring.
- ObzenFlow’s Run Modes covers live runs, replay with
--verify, resume, and the CLI that reads every run. - ObzenFlow’s Stage Vocabulary covers every stage, effect, and composite form.