Run, replay, and resume.
Durable execution in ObzenFlow means a fact committed to its journal survives the process that wrote it. Work can be run live, replayed for audit, and resumed from its last committed fact after a crash. All three run modes work with inline streaming API calls and inference. Learn how ObzenFlow enables all three of these run modes while bringing these guarantees to streaming workloads.
Live
Run your flow. The framework handles all of the details.
Your first run starts a single binary that answers health checks, serves Prometheus metrics, takes HTTP traffic, and journals every stage’s facts.
Replay
Replay a recorded run and check the result matches.
Recorded inputs pass through your current handlers while effects return their recorded outcomes. Add --verify and ObzenFlow compares the replay with the original, stage by stage.
Resume
Finish an interrupted run without repeating committed effects.
Recorded facts catch up first, with effects suppressed. Live work then reopens at the recorded high-water mark.
Live
What happens on a live run?
Ship your flow as one Rust binary.
You define the flow, and FlowApplication runs it as a supervised service with startup, shutdown, configuration, and a command line built in. Deploy the binary as an ordinary process or container, with no control plane or cluster to install.
One binary
Its config file switches on the endpoints it serves, from health and readiness checks to Prometheus metrics.
Traffic in
Each ingress call opens a typed endpoint that HTTP clients post to, feeding a source in your flow.
The rest of the world
Your stages pull from upstream APIs, deliver to PostgreSQL, and call models, each compiled in with its Cargo feature:
http,postgres, orai.
let mut app = FlowApplication::builder()
.with_config_file(CONFIG_FILE);
let new_accounts_source = app.http_ingress(
AccountIngress,
ingress_config(ACCOUNTS_BASE_PATH),
);
let tx_source = app.http_ingress(
LedgerIngress,
ingress_config(TX_BASE_PATH),
);
app.run_blocking(flow::build_flow(
new_accounts_source,
tx_source,
))?;Prometheus metrics with nothing extra to run.
Compile in the prometheus feature, switch metrics on in config, and the same binary serves /metrics, with no exporter or sidecar to deploy. The numbers here are from a real scrape, six seconds into a run of the prometheus_demo example.
Every stage counts its work
Each stage reports the events it processed and the errors it hit, so you can see where work slows down or fails.
Backpressure on every edge
Each edge reports how many events are in flight between its two stages, so a growing backlog shows up before it stalls the flow.
Resilience in the open
The source’s circuit breaker and rate limiter report every call they let through, refuse, or delay.
The whole flow
Flow totals count everything that entered from sources and every error along the way.
obzenflow_events_total{stage="high_volume_source",…} 6878
obzenflow_events_total{stage="error_processor",…} 6832
obzenflow_events_total{stage="event_counter",…} 6000
obzenflow_errors_total{stage="error_processor",error_kind="unknown",…} 69
obzenflow_backpressure_in_flight{downstream="error_processor",…} 44
obzenflow_backpressure_in_flight{downstream="event_counter",…} 10
obzenflow_backpressure_in_flight{downstream="summary_sink",…} 0
obzenflow_circuit_breaker_requests_total{stage="high_volume_source",…} 6902
obzenflow_circuit_breaker_rejections_total{stage="high_volume_source",…} 0
obzenflow_rate_limiter_delayed_total{stage="high_volume_source",…} 133
obzenflow_flow_events_in_total{flow="prometheus_demo",…} 6878
obzenflow_flow_errors_total{flow="prometheus_demo",…} 69Serve your own endpoints, streaming included.
Host your own HTTP endpoints beside the built-in ones, including endpoints that stream responses as server-sent events, so one binary can be the whole service.
Hold what you’ll stream
The endpoint is a plain struct. This one holds a channel that a sink in your flow publishes balance updates to.
Name the route
Give it a path and the methods it answers.
Stream the response
Each request subscribes to the channel and receives every new balance as a server-sent event, for as long as it stays connected.
Host it
One call serves it from the same binary as the built-in endpoints.
struct BalanceStream {
balances: broadcast::Sender<String>,
}
#[async_trait]
impl HttpEndpoint for BalanceStream {
fn path(&self) -> &str {
"/api/bank/balances/stream"
}
fn methods(&self) -> &[HttpMethod] {
&[HttpMethod::Get]
}
async fn handle(&self, _: Request) -> Result<ManagedResponse, EndpointError> {
let frames = BroadcastStream::new(self.balances.subscribe())
.filter_map(|balance| async move { balance.ok() })
.map(|balance| SseFrame::event("balance", balance));
Ok(ManagedResponse::Sse(SseBody::new(frames)))
}
}
let app = app.with_web_endpoint(Box::new(BalanceStream { balances }));Replay
Replay a recorded run without repeating its effects.
Replay runs your code over the recorded history.
Here the payment gateway example’s authorize_payment stage replays the orders a live run recorded.
The recorded journal or the gateway
On a live run,
fx.callsends the request to the payment gateway and the outcome is journalled. On replay, the same call reads that outcome from the recorded journal, and the gateway is never called.The same outcome, the same journal
Read or called, the outcome is the same, so your handler takes the same branch and the result is written to the same output journal.
async fn process(
&self,
order: ValidatedOrder,
fx: &mut Effects<Self::Output, Self::AllowedEffects>,
) -> Result<StageCompletion<Self::Output>, HandlerError> {
// …
let request = AuthorizeRequest::for_order(&order);
if let Err(error) = fx.call(request).await {
fx.emit(order.authorization_unavailable(&error)).await?;
}
Ok(fx.complete()?)
}Replay makes determinism a checkable claim.
Replay is one of ObzenFlow’s most powerful capabilities because it lets you test a durable flow against its own history. Point ObzenFlow at a completed run and it reruns the flow without polling live sources or repeating declared effects.
Recorded facts, current handlers
Recorded source facts are re-admitted from the archived source journals as an ordered input tape, and your current handlers execute again.
Effects answered from the record
When a handler reaches
fx.call, ObzenFlow resolves the outcome recorded at that exact durable position and returns it without calling the port. When non-deterministic values are isolated behind first-class effects,fx.capture, or the replay-safe context, the same archive reconstructs deterministically.The same state, verified
The reconstruction writes a new set of journals. Add
--verifyand ObzenFlow compares the new fact history with the original run, stage by stage, and reports the first divergence.
Resume
Pick up an interrupted run where it left off.
Safely resume an interrupted flow from its own journals.
Resume completes a run that ended early, for instance after a crash. ObzenFlow knows the exact point where the run terminated unnaturally, so resume replays its journals up to that point, then continues live exactly where it left off.
Resume from any previous manifest
A manifest of a previous run, even a crashed run, contains all of the journals needed to resume. Those journals become the input tape, and ObzenFlow recovers from them.
Exactly-once processing on resume
A call with a recorded outcome is not made again. A call that was in flight when the process died is made again, and the guarantee declared on the call says how it still takes effect once.
Work continues after the frontier
Once the state is rehydrated, the framework flips to live execution and the run continues from its last committed fact.
Next, see what every run recorded.
Observable by Design shows how to read any run from its journals with the ObzenFlow CLI, and the metrics the runtime reports while a flow runs.