Building durable workflows.

At the heart of ObzenFlow are stages, effects, and composites: a small vocabulary for transforming events, joining streams with reference data, accumulating state, and making external effects explicit. You’ll learn how to compose them into sophisticated durable programs that bring the directness of classical application development to real-time stream processing.

Stages

Compose a flow from a graph of typed stages.

Declare your whole workflow as one typed flow, from the sources that bring data in to the sinks that deliver it. Every decision a stage makes is recorded in its own journal.

Four building blocks of every durable workflow.

Durable execution that streams comes down to four parts. The application handles the operational complexity, your handlers hold the business logic, the flow composes them into stages, and the topology wires them together.

  1. The runner

    FlowApplication is the host boundary that runs the flow. Its lifecycle and endpoints live on Run, Replay, and Resume.

  2. The handlers

    FlowDefinition::materialize builds the handlers and middleware once the run’s configuration is loaded, then returns the flow! that wires them by name.

  3. The flow

    The flow! macro declares everything inside. journals sets where each stage writes its append-only journal, and stages lists every stage in the flow.

  4. The topology

    topology is the graph of stage edges. Forward edges use |>. Backflow edges use <|.

stateful_counter.rs
#[tokio::main]
async fn main() -> Result<()> {
    // The runner: a plain async main that starts the runtime.
    FlowApplication::run(FlowDefinition::materialize(|_| {
        // The handlers: ordinary Rust values, built first.
        let number_feed = number_source(10);
        let even_numbers = even_filter();
        let count_evens = CounterHandler::new();
        let print_result = sinks::ConsoleSink::new(format_result);

        // The flow: types and shape, not plumbing.
        Ok(flow! {
            name: "stateful_counter",
            journals: disk_journals(PathBuf::from("target/counter-logs")),

            stages: {
                numbers = source!(Number with rate_limit(5.0) => number_feed);
                evens = transform!(Number -> EvenNumber => even_numbers);
                counter = stateful!(EvenNumber -> Count => count_evens);
                output = sink!(Count => print_result);
            },

            topology: {
                numbers |> evens;
                evens   |> counter;
                counter |> output;
            }
        })
    }))
    .await?;

    Ok(())
}

Express any problem with seven stage types.

Every flow, no matter how large or complex, is composed of these seven stage types. The DSL makes composition intuitive. Build anything from a plain character transform to live streaming, durable inference that can replay, resume, or survive even the deepest audit.

Sources
FiniteSource is for batch data and ends when its input runs out. InfiniteSource is for live streaming data and runs for the life of the process. One flow can compose both.
Merge
Combines producers of one type into one stream, in the order its strategy sets.
Transform
Processes one input at a time without framework-managed persistent state.
Join
Maintains keyed reference facts beside a forward-moving stream.
Stateful
Folds events into state that survives across inputs.
Sink
Terminates a branch by delivering committed facts downstream.
stage_vocabulary.rs
stages: {
    // StageType::FiniteSource
    products = source!(Product => product_catalog);
    backlog = source!(Order => order_backlog);

    // StageType::InfiniteSource
    live = async_infinite_source!(Order => live_orders);

    // StageType::Merge
    orders = merge!(Order => merges::then_live);

    // StageType::Transform
    validated = transform!(Order -> ValidatedOrder => validate);

    // StageType::Join
    enriched = join!(
        catalog products: Product,
        ValidatedOrder -> EnrichedOrder => enrich
    );

    // StageType::Stateful
    totals = stateful!(EnrichedOrder -> OrderTotals => order_totals);

    // StageType::Sink
    output = sink!(OrderTotals => write_totals);
}

topology: {
    backlog |> orders;
    live |> orders;
    orders |> validated;
    (products, validated) |> enriched;
    enriched |> totals;
    totals |> output;
}

Implement most stages with a single helper call.

You rarely write a handler from scratch. Every handler in this flow is a single helper call, and the merge needs no handler at all, because you just pick a strategy such as merges::then_live.

Sources
ValuesSource, CsvSource, ChannelSource, and the HTTP sources read data in without a hand-written reader.
Transforms
map, filter, filter_map, and try_map reshape, drop, or validate each event with a closure.
Joins
inner, left, and strict, each with live variants, match a stream against reference data by key.
Stateful
reduce, group_by, top_n, and conflate accumulate totals, groups, and rankings, and emit when you choose.
Sinks
ConsoleSink, CsvSink, TracingSink, and the PostgreSQL sink deliver results without a hand-written writer.

You don’t need to memorize these helpers. The Stage Vocabulary tutorial walks through every stage type.

stage_vocabulary.rs
FlowDefinition::materialize(|_| {
    let product_catalog = ValuesSource::new(fixtures::products());

    let order_backlog = CsvSource::builder(OrderCsv)
        .path("data/orders.csv")
        .build()
        .map_err(|e| FlowBuildError::StageResourcesFailed(e.to_string()))?;

    let live_orders = ChannelSource::new(order_rx);

    let validate = transforms::filter_map(|order: Order| order.validate());

    let enrich = joins::inner(
        |product: &Product| product.sku.clone(),
        |order: &ValidatedOrder| order.sku.clone(),
        EnrichedOrder::new,
    );

    let order_totals = stateful::reduce(
        OrderTotals::default(),
        OrderTotals::add,
    )
    .emit(emit::always);

    let write_totals = sinks::ConsoleSink::new(totals_table());

    Ok(flow! {
        // the stages and topology from the flow above
    })
})

Create custom handlers with plain Rust.

When no helper fits, drop down to your own logic. A helper is a function that returns a handler, and a custom handler is a plain Rust type that implements the same trait. Either way, you bind it with let and name it in the stage.

  1. The stage

    Write the stage like any other, with your own handler plugged in at the end. Its input and output must match the types the stage lists.

  2. The struct

    Your handler is an ordinary Rust struct. This one is empty, but it can hold rules or thresholds you pass in when you create it.

  3. The output

    outcome! names each fact the stage lists. This is the output side of the match.

  4. The logic

    Implement the handler trait, and process runs once for each alert. This one suppresses a duplicate and sends every other alert on to be investigated, returning one of those facts either way.

triage.rs
// in flow.rs
let triage_handler = Triage;

Ok(flow! {
    stages: {
        triage = transform!(AlertRaised -> {
            IncidentInvestigationRequested,
            AlertSuppressed,
        } => triage_handler);
        // ...
    },
})

pub struct Triage;

impl TypedTransformHandler for Triage {
    type Input = AlertRaised;
    type Output = outcome![IncidentInvestigationRequested | AlertSuppressed];

    fn process(&self, alert: AlertRaised)
        -> Result<Self::Output, HandlerError>
    {
        if alert.is_duplicate() {
            let suppressed = AlertSuppressed::duplicate(alert);
            return Ok(outcome(suppressed));
        }
        Ok(outcome(IncidentInvestigationRequested::from(alert)))
    }
}

Effects

Isolate I/O with effects.

Congratulations, your SaaS business, the Jelly of the Month Club, is a smash hit. Tens of thousands of paying customers with a monthly subscription. Your system charges a customer when the jelly leaves the warehouse. Learn how to handle this batch safely without ever risking double charging a customer if the jelly billing system crashes mid-charge. We'll even prove it by simulating chaos with Stripe.

Perform every side effect through a port.

A port is the boundary between your own deterministic code and the non-deterministic world outside it. You write a side effect in three pieces. A command performs it, a port declares it, and a system implements it. Here the side effect is a card charge, and the system calls Stripe. On a replay or a resume, the archive returns every recorded outcome without calling the system.

  1. The command

    One line in your flow turns each invoice into a ChargeRequest and performs it through the port. It never knows which system answers.

  2. The port

    It is an ordinary Rust trait with a method for each request. Each method says what it sends and every outcome that can come back.

  3. The system

    Stripe answers the port, and a scripted processor can answer it the same way. Your config picks which one a run uses.

// flow.rs
let charge = perform(ChargeRequest::for_invoice).through(port);

// gateway.rs
#[obzenflow::port]
#[async_trait]
pub trait PaymentGatewayPort {
    #[effect(type = "payment.charge", exactly_once)]
    async fn charge(
        &self,
        request: &ChargeRequest,
        key: &IdempotencyKey,
    ) -> Result<
        outcome![InvoicePaid | InvoiceDeclined | InvoiceNeedsCustomer],
        EffectError,
    >;
}

// stripe.rs
#[obzenflow::system("stripe")]
#[async_trait]
impl PaymentGatewayPort for StripeProcessor { /* … */ }

Put Stripe or a simulated system behind the port.

A system answers a port’s requests. The Jelly Club’s payment port has a system that calls Stripe and a simulated system that stands in for it.

  1. The Stripe system

    You declare StripeProcessor as a system for the payment port. A system is a physical backend, and a port is the logical abstraction over its systems, so one port can have several systems.

  2. A simulated system

    ScriptedProcessor is a second system for the same port. You declare one system for each backend a port might use, such as Stripe in production, Stripe in test, a simulated gateway or even a chaos system that fails on purpose.

  3. Your config picks one

    You plug a system into the port when the flow starts, from a setting in your config. Each environment can then run the same flow against its own backend, such as the simulated gateway in tests and Stripe in production.

// stripe.rs
#[obzenflow::system("stripe")]
#[async_trait]
impl PaymentGatewayPort for StripeProcessor {
    async fn charge(
        &self,
        request: &ChargeRequest,
        key: &IdempotencyKey,
    ) -> Result<
        outcome![InvoicePaid | InvoiceDeclined | InvoiceNeedsCustomer],
        EffectError,
    > {
        let confirm = HttpRequest::post(payment_intents())
            .with_idempotency_key(&key.0)
            // …
    }
}

// gateway.rs
#[obzenflow::system("scripted-processor")]
#[async_trait]
impl PaymentGatewayPort for ScriptedProcessor { /* … */ }

// flow.rs
let port = match settings.processor {
    Processor::Stripe => Port::deferred(StripeProcessor::from_env),
    Processor::Scripted => Port::deferred(/* … */),
};

Replay a run without repeating an effect.

Nothing in your code changes for a replay. That’s the power of the effect system. Your command performs every effect through a port, and ObzenFlow’s runtime handles the rest. It decides at the port whether to call the system or return the recorded outcome, so your command never has to know. On a replay, the tape plays back through your original code. Soon you’ll be able to play the same tape back through different business logic and see what happens.

  1. Archived journals become input tapes

    On a live run, the call sends the request through the port and journals the outcome. On a replay, the same call finds the outcome journaled at the input’s cursor, its position on the tape, and returns it to your command without reaching the system.

  2. Play the tape back without calling Stripe

    Every outcome comes off the tape, so the port never builds its system. You can replay last night’s Stripe run on any machine. That becomes an audit superpower, for instance replaying a previous run in an air-gapped environment.

Resume a run without charging anyone twice.

A crash can happen at any time. A standby can take over, but failover by itself can’t tell which effects already happened. ObzenFlow resumes an incomplete run from its own journal. It replays every journaled event without re-executing side effects, finds the exact point where the run stopped, then continues live where it left off. The edge case is an effect that was in progress during the crash and whose outcome only the third-party system knows. ObzenFlow handles that too.

  1. Kill it with one charge in flight

    Put a simulated system that logs every call behind the port, start a billing run, and kill it with SIGKILL a few seconds in. Some outcomes are journaled, and one charge has reached the system before its outcome could be journaled.

  2. Resume from the journal

    Resume the run from its journal. The journaled outcomes come back without calling the system, and the charge in flight is sent again under the same idempotency key. The totals should match a run that was never killed.

  3. Count every call

    obzenflow verify calls reads the simulated system’s call log across both runs. Every charge was sent once except the one in flight, which was sent twice under the same key, and the system answered the second from its own record.

$ ./nightly_renewals & sleep 3; kill -KILL $!
Receipt to C-32961 for INV-2026-11-004128: $29.00 paid (pi_scripted_000003)
$ ./nightly_renewals --resume-from "$RUN"
Night totals: 169 invoices paid, $9161.00
$ obzenflow show "$RESUMED"
EFFECTFUL TRANSFORM (stage: charge_invoice, journal: 4)
billing.invoice_paid ← charge_invoice(billing.invoice_validated) [read from journal]
idempotency key: invoice:INV-2026-11-004128:1

EFFECTFUL TRANSFORM (stage: charge_invoice, journal: 4)
billing.invoice_declined ← charge_invoice(billing.invoice_validated)
idempotency key: invoice:INV-2026-10-003711:2
$ obzenflow verify calls --run "$RUN" --run "$RESUMED" calls.jsonl
ok: calls.jsonl holds 200 calls from this lineage, 1 repeated by a resume.
$ grep replayed calls.jsonl | jq
{
  "result": "replayed",
  "key": "invoice:INV-2026-10-003711:2",
  "flow_id": "flow_01M6MH4PFHKYZBNZZVH3CTNCJ7"
}

Composites

Automate common patterns with a single composite.

A composite is written like one stage and runs as several, like a flow within a flow. ObzenFlow ships AI map and map-reduce today, and sagas are coming soon.

Run AI map-reduce from a single stage.

Summarizing hundreds of incidents with a model means splitting them into groups, calling the model for each one, collecting the answers, and writing one report. With a composite, that’s one stage in your flow.

ObzenFlow builds and journals the four stages underneath, so a crash or a replay never asks the model again for a group it already summarized.

  1. Map

    Split the incidents into groups of twenty, and the model summarizes each group.

  2. Reduce

    When the input ends, the summaries are collected and the model writes one report.

  3. One stage

    The whole pattern sits in your flow as one stage, with one fact in and one fact out.

let report_builder = ai::map(IncidentSummarizer)
    .chunks(20)
    .reduce(ReportWriter)
    .on_eof();

// in stages: { … }
build_incident_report = ai!(
    IncidentEvidenceLoaded -> IncidentReport
    with ai_breaker
    => report_builder
);

Next, run the flow you built.

Run, Replay, and Resume shows the same application running as a service, replayed from its journals, and resumed after a crash.