Run, replay, and resume.

Durable execution in ObzenFlow means a fact committed to its journal survives the process that wrote it. Work can be run live, replayed for audit, and resumed from its last committed fact after a crash. All three run modes work with inline streaming API calls and inference. Learn how ObzenFlow enables all three of these run modes while bringing these guarantees to streaming workloads.

Live

What happens on a live run?

Start the binary and your flow runs as an ordinary service. Health checks answer, Prometheus can scrape its metrics, HTTP traffic feeds your sources, and every stage starts journalling.

Ship your flow as one Rust binary.

You define the flow, and FlowApplication runs it as a supervised service with startup, shutdown, configuration, and a command line built in. Deploy the binary as an ordinary process or container, with no control plane or cluster to install.

  1. One binary

    Its config file switches on the endpoints it serves, from health and readiness checks to Prometheus metrics.

  2. Traffic in

    Each ingress call opens a typed endpoint that HTTP clients post to, feeding a source in your flow.

  3. The rest of the world

    Your stages pull from upstream APIs, deliver to PostgreSQL, and call models, each compiled in with its Cargo feature: http, postgres, or ai.

let mut app = FlowApplication::builder()
    .with_config_file(CONFIG_FILE);

let new_accounts_source = app.http_ingress(
    AccountIngress,
    ingress_config(ACCOUNTS_BASE_PATH),
);
let tx_source = app.http_ingress(
    LedgerIngress,
    ingress_config(TX_BASE_PATH),
);

app.run_blocking(flow::build_flow(
    new_accounts_source,
    tx_source,
))?;

Prometheus metrics with nothing extra to run.

Compile in the prometheus feature, switch metrics on in config, and the same binary serves /metrics, with no exporter or sidecar to deploy. The numbers here are from a real scrape, six seconds into a run of the prometheus_demo example.

  1. Every stage counts its work

    Each stage reports the events it processed and the errors it hit, so you can see where work slows down or fails.

  2. Backpressure on every edge

    Each edge reports how many events are in flight between its two stages, so a growing backlog shows up before it stalls the flow.

  3. Resilience in the open

    The source’s circuit breaker and rate limiter report every call they let through, refuse, or delay.

  4. The whole flow

    Flow totals count everything that entered from sources and every error along the way.

obzenflow_events_total{stage="high_volume_source",…} 6878
obzenflow_events_total{stage="error_processor",…} 6832
obzenflow_events_total{stage="event_counter",…} 6000
obzenflow_errors_total{stage="error_processor",error_kind="unknown",…} 69

obzenflow_backpressure_in_flight{downstream="error_processor",…} 44
obzenflow_backpressure_in_flight{downstream="event_counter",…} 10
obzenflow_backpressure_in_flight{downstream="summary_sink",…} 0

obzenflow_circuit_breaker_requests_total{stage="high_volume_source",…} 6902
obzenflow_circuit_breaker_rejections_total{stage="high_volume_source",…} 0
obzenflow_rate_limiter_delayed_total{stage="high_volume_source",…} 133

obzenflow_flow_events_in_total{flow="prometheus_demo",…} 6878
obzenflow_flow_errors_total{flow="prometheus_demo",…} 69

Serve your own endpoints, streaming included.

Host your own HTTP endpoints beside the built-in ones, including endpoints that stream responses as server-sent events, so one binary can be the whole service.

  1. Hold what you’ll stream

    The endpoint is a plain struct. This one holds a channel that a sink in your flow publishes balance updates to.

  2. Name the route

    Give it a path and the methods it answers.

  3. Stream the response

    Each request subscribes to the channel and receives every new balance as a server-sent event, for as long as it stays connected.

  4. Host it

    One call serves it from the same binary as the built-in endpoints.

balance_stream.rs
struct BalanceStream {
    balances: broadcast::Sender<String>,
}

#[async_trait]
impl HttpEndpoint for BalanceStream {
    fn path(&self) -> &str {
        "/api/bank/balances/stream"
    }

    fn methods(&self) -> &[HttpMethod] {
        &[HttpMethod::Get]
    }

    async fn handle(&self, _: Request) -> Result<ManagedResponse, EndpointError> {
        let frames = BroadcastStream::new(self.balances.subscribe())
            .filter_map(|balance| async move { balance.ok() })
            .map(|balance| SseFrame::event("balance", balance));
        Ok(ManagedResponse::Sse(SseBody::new(frames)))
    }
}

let app = app.with_web_endpoint(Box::new(BalanceStream { balances }));

Replay

Replay a recorded run without repeating its effects.

Pass a run directory to --replay-from and the same binary reconstructs that run. Live sources stay closed, your current handlers execute again over the recorded inputs, and every declared effect returns the outcome its journal recorded. The replay writes journals of its own, so you can compare it with the original or replay it in turn.

Replay runs your code over the recorded history.

Here the payment gateway example’s authorize_payment stage replays the orders a live run recorded.

  1. The recorded journal or the gateway

    On a live run, fx.call sends the request to the payment gateway and the outcome is journalled. On replay, the same call reads that outcome from the recorded journal, and the gateway is never called.

  2. The same outcome, the same journal

    Read or called, the outcome is the same, so your handler takes the same branch and the result is written to the same output journal.

async fn process(
    &self,
    order: ValidatedOrder,
    fx: &mut Effects<Self::Output, Self::AllowedEffects>,
) -> Result<StageCompletion<Self::Output>, HandlerError> {
    // …
    let request = AuthorizeRequest::for_order(&order);
    if let Err(error) = fx.call(request).await {
        fx.emit(order.authorization_unavailable(&error)).await?;
    }
    Ok(fx.complete()?)
}

Replay makes determinism a checkable claim.

Replay is one of ObzenFlow’s most powerful capabilities because it lets you test a durable flow against its own history. Point ObzenFlow at a completed run and it reruns the flow without polling live sources or repeating declared effects.

  1. Recorded facts, current handlers

    Recorded source facts are re-admitted from the archived source journals as an ordered input tape, and your current handlers execute again.

  2. Effects answered from the record

    When a handler reaches fx.call, ObzenFlow resolves the outcome recorded at that exact durable position and returns it without calling the port. When non-deterministic values are isolated behind first-class effects, fx.capture, or the replay-safe context, the same archive reconstructs deterministically.

  3. The same state, verified

    The reconstruction writes a new set of journals. Add --verify and ObzenFlow compares the new fact history with the original run, stage by stage, and reports the first divergence.

Resume

Pick up an interrupted run where it left off.

A crash ends the process and leaves the journals. Pass the interrupted run to --resume-from and ObzenFlow rebuilds every stage from them, returns each committed effect’s recorded outcome, and continues live from the recorded high-water mark.

Safely resume an interrupted flow from its own journals.

Resume completes a run that ended early, for instance after a crash. ObzenFlow knows the exact point where the run terminated unnaturally, so resume replays its journals up to that point, then continues live exactly where it left off.

  1. Resume from any previous manifest

    A manifest of a previous run, even a crashed run, contains all of the journals needed to resume. Those journals become the input tape, and ObzenFlow recovers from them.

  2. Exactly-once processing on resume

    A call with a recorded outcome is not made again. A call that was in flight when the process died is made again, and the guarantee declared on the call says how it still takes effect once.

  3. Work continues after the frontier

    Once the state is rehydrated, the framework flips to live execution and the run continues from its last committed fact.

Next, see what every run recorded.

Observable by Design shows how to read any run from its journals with the ObzenFlow CLI, and the metrics the runtime reports while a flow runs.