Building durable workflows.
At the heart of ObzenFlow are stages, effects, and composites: a small vocabulary for transforming events, joining streams with reference data, accumulating state, and making external effects explicit. You’ll learn how to compose them into sophisticated durable programs that bring the directness of classical application development to real-time stream processing.

Stages
Compose batch and live data in one typed durable workflow.
Sources bring data in; transforms, joins, and merges shape it; stateful stages accumulate it; and sinks deliver it. Every decision is journalled, so a run can be replayed.
Effects
Isolate I/O from deterministic code with algebraic effects.
Declare all non-deterministic code as an effect, which is what enables safe replay and resume. A live run performs each effect. Replay and resume rebuild state from the original effect outcomes.
Composites
Automate common patterns with a single composite.
A composite is written like one stage and runs as several, like a flow within a flow. ai! map-reduce runs as chunk, map, collect, and reduce stages. Sagas are coming soon.
Stages
Compose a flow from a graph of typed stages.
Four building blocks of every durable workflow.
Durable execution that streams comes down to four parts. The application handles the operational complexity, your handlers hold the business logic, the flow composes them into stages, and the topology wires them together.
The runner
FlowApplicationis the host boundary that runs the flow. Its lifecycle and endpoints live on Run, Replay, and Resume.The handlers
FlowDefinition::materializebuilds the handlers and middleware once the run’s configuration is loaded, then returns theflow!that wires them by name.The flow
The
flow!macro declares everything inside.journalssets where each stage writes its append-only journal, andstageslists every stage in the flow.The topology
topologyis the graph of stage edges. Forward edges use|>. Backflow edges use<|.
#[tokio::main]
async fn main() -> Result<()> {
// The runner: a plain async main that starts the runtime.
FlowApplication::run(FlowDefinition::materialize(|_| {
// The handlers: ordinary Rust values, built first.
let number_feed = number_source(10);
let even_numbers = even_filter();
let count_evens = CounterHandler::new();
let print_result = sinks::ConsoleSink::new(format_result);
// The flow: types and shape, not plumbing.
Ok(flow! {
name: "stateful_counter",
journals: disk_journals(PathBuf::from("target/counter-logs")),
stages: {
numbers = source!(Number with rate_limit(5.0) => number_feed);
evens = transform!(Number -> EvenNumber => even_numbers);
counter = stateful!(EvenNumber -> Count => count_evens);
output = sink!(Count => print_result);
},
topology: {
numbers |> evens;
evens |> counter;
counter |> output;
}
})
}))
.await?;
Ok(())
}Express any problem with seven stage types.
Every flow, no matter how large or complex, is composed of these seven stage types. The DSL makes composition intuitive. Build anything from a plain character transform to live streaming, durable inference that can replay, resume, or survive even the deepest audit.
- Sources
- FiniteSource is for batch data and ends when its input runs out. InfiniteSource is for live streaming data and runs for the life of the process. One flow can compose both.
- Merge
- Combines producers of one type into one stream, in the order its strategy sets.
- Transform
- Processes one input at a time without framework-managed persistent state.
- Join
- Maintains keyed reference facts beside a forward-moving stream.
- Stateful
- Folds events into state that survives across inputs.
- Sink
- Terminates a branch by delivering committed facts downstream.
stages: {
// StageType::FiniteSource
products = source!(Product => product_catalog);
backlog = source!(Order => order_backlog);
// StageType::InfiniteSource
live = async_infinite_source!(Order => live_orders);
// StageType::Merge
orders = merge!(Order => merges::then_live);
// StageType::Transform
validated = transform!(Order -> ValidatedOrder => validate);
// StageType::Join
enriched = join!(
catalog products: Product,
ValidatedOrder -> EnrichedOrder => enrich
);
// StageType::Stateful
totals = stateful!(EnrichedOrder -> OrderTotals => order_totals);
// StageType::Sink
output = sink!(OrderTotals => write_totals);
}
topology: {
backlog |> orders;
live |> orders;
orders |> validated;
(products, validated) |> enriched;
enriched |> totals;
totals |> output;
}Implement most stages with a single helper call.
You rarely write a handler from scratch. Every handler in this flow is a single helper call, and the merge needs no handler at all, because you just pick a strategy such as merges::then_live.
- Sources
ValuesSource,CsvSource,ChannelSource, and the HTTP sources read data in without a hand-written reader.- Transforms
map,filter,filter_map, andtry_mapreshape, drop, or validate each event with a closure.- Joins
inner,left, andstrict, each with live variants, match a stream against reference data by key.- Stateful
reduce,group_by,top_n, andconflateaccumulate totals, groups, and rankings, and emit when you choose.- Sinks
ConsoleSink,CsvSink,TracingSink, and the PostgreSQL sink deliver results without a hand-written writer.
You don’t need to memorize these helpers. The Stage Vocabulary tutorial walks through every stage type.
FlowDefinition::materialize(|_| {
let product_catalog = ValuesSource::new(fixtures::products());
let order_backlog = CsvSource::builder(OrderCsv)
.path("data/orders.csv")
.build()
.map_err(|e| FlowBuildError::StageResourcesFailed(e.to_string()))?;
let live_orders = ChannelSource::new(order_rx);
let validate = transforms::filter_map(|order: Order| order.validate());
let enrich = joins::inner(
|product: &Product| product.sku.clone(),
|order: &ValidatedOrder| order.sku.clone(),
EnrichedOrder::new,
);
let order_totals = stateful::reduce(
OrderTotals::default(),
OrderTotals::add,
)
.emit(emit::always);
let write_totals = sinks::ConsoleSink::new(totals_table());
Ok(flow! {
// the stages and topology from the flow above
})
})Create custom handlers with plain Rust.
When no helper fits, drop down to your own logic. A helper is a function that returns a handler, and a custom handler is a plain Rust type that implements the same trait. Either way, you bind it with let and name it in the stage.
The stage
Write the stage like any other, with your own handler plugged in at the end. Its input and output must match the types the stage lists.
The struct
Your handler is an ordinary Rust struct. This one is empty, but it can hold rules or thresholds you pass in when you create it.
The output
outcome!names each fact the stage lists. This is the output side of the match.The logic
Implement the handler trait, and
processruns once for each alert. This one suppresses a duplicate and sends every other alert on to be investigated, returning one of those facts either way.
// in flow.rs
let triage_handler = Triage;
Ok(flow! {
stages: {
triage = transform!(AlertRaised -> {
IncidentInvestigationRequested,
AlertSuppressed,
} => triage_handler);
// ...
},
})
pub struct Triage;
impl TypedTransformHandler for Triage {
type Input = AlertRaised;
type Output = outcome![IncidentInvestigationRequested | AlertSuppressed];
fn process(&self, alert: AlertRaised)
-> Result<Self::Output, HandlerError>
{
if alert.is_duplicate() {
let suppressed = AlertSuppressed::duplicate(alert);
return Ok(outcome(suppressed));
}
Ok(outcome(IncidentInvestigationRequested::from(alert)))
}
}Effects
Isolate I/O with effects.
Perform every side effect through a port.
A port is the boundary between your own deterministic code and the non-deterministic world outside it. You write a side effect in three pieces. A command performs it, a port declares it, and a system implements it. Here the side effect is a card charge, and the system calls Stripe. On a replay or a resume, the archive returns every recorded outcome without calling the system.
The command
One line in your flow turns each invoice into a
ChargeRequestand performs it through the port. It never knows which system answers.The port
It is an ordinary Rust trait with a method for each request. Each method says what it sends and every outcome that can come back.
The system
Stripe answers the port, and a scripted processor can answer it the same way. Your config picks which one a run uses.
// flow.rs
let charge = perform(ChargeRequest::for_invoice).through(port);
// gateway.rs
#[obzenflow::port]
#[async_trait]
pub trait PaymentGatewayPort {
#[effect(type = "payment.charge", exactly_once)]
async fn charge(
&self,
request: &ChargeRequest,
key: &IdempotencyKey,
) -> Result<
outcome![InvoicePaid | InvoiceDeclined | InvoiceNeedsCustomer],
EffectError,
>;
}
// stripe.rs
#[obzenflow::system("stripe")]
#[async_trait]
impl PaymentGatewayPort for StripeProcessor { /* … */ }Put Stripe or a simulated system behind the port.
A system answers a port’s requests. The Jelly Club’s payment port has a system that calls Stripe and a simulated system that stands in for it.
The Stripe system
You declare
StripeProcessoras a system for the payment port. A system is a physical backend, and a port is the logical abstraction over its systems, so one port can have several systems.A simulated system
ScriptedProcessoris a second system for the same port. You declare one system for each backend a port might use, such as Stripe in production, Stripe in test, a simulated gateway or even a chaos system that fails on purpose.Your config picks one
You plug a system into the port when the flow starts, from a setting in your config. Each environment can then run the same flow against its own backend, such as the simulated gateway in tests and Stripe in production.
// stripe.rs
#[obzenflow::system("stripe")]
#[async_trait]
impl PaymentGatewayPort for StripeProcessor {
async fn charge(
&self,
request: &ChargeRequest,
key: &IdempotencyKey,
) -> Result<
outcome![InvoicePaid | InvoiceDeclined | InvoiceNeedsCustomer],
EffectError,
> {
let confirm = HttpRequest::post(payment_intents())
.with_idempotency_key(&key.0)
// …
}
}
// gateway.rs
#[obzenflow::system("scripted-processor")]
#[async_trait]
impl PaymentGatewayPort for ScriptedProcessor { /* … */ }
// flow.rs
let port = match settings.processor {
Processor::Stripe => Port::deferred(StripeProcessor::from_env),
Processor::Scripted => Port::deferred(/* … */),
};Replay a run without repeating an effect.
Nothing in your code changes for a replay. That’s the power of the effect system. Your command performs every effect through a port, and ObzenFlow’s runtime handles the rest. It decides at the port whether to call the system or return the recorded outcome, so your command never has to know. On a replay, the tape plays back through your original code. Soon you’ll be able to play the same tape back through different business logic and see what happens.
Archived journals become input tapes
On a live run, the call sends the request through the port and journals the outcome. On a replay, the same call finds the outcome journaled at the input’s cursor, its position on the tape, and returns it to your command without reaching the system.
Play the tape back without calling Stripe
Every outcome comes off the tape, so the port never builds its system. You can replay last night’s Stripe run on any machine. That becomes an audit superpower, for instance replaying a previous run in an air-gapped environment.
Resume a run without charging anyone twice.
A crash can happen at any time. A standby can take over, but failover by itself can’t tell which effects already happened. ObzenFlow resumes an incomplete run from its own journal. It replays every journaled event without re-executing side effects, finds the exact point where the run stopped, then continues live where it left off. The edge case is an effect that was in progress during the crash and whose outcome only the third-party system knows. ObzenFlow handles that too.
Kill it with one charge in flight
Put a simulated system that logs every call behind the port, start a billing run, and kill it with SIGKILL a few seconds in. Some outcomes are journaled, and one charge has reached the system before its outcome could be journaled.
Resume from the journal
Resume the run from its journal. The journaled outcomes come back without calling the system, and the charge in flight is sent again under the same idempotency key. The totals should match a run that was never killed.
Count every call
obzenflow verify callsreads the simulated system’s call log across both runs. Every charge was sent once except the one in flight, which was sent twice under the same key, and the system answered the second from its own record.
$ ./nightly_renewals & sleep 3; kill -KILL $!
Receipt to C-32961 for INV-2026-11-004128: $29.00 paid (pi_scripted_000003)
$ ./nightly_renewals --resume-from "$RUN"
Night totals: 169 invoices paid, $9161.00
$ obzenflow show "$RESUMED"
EFFECTFUL TRANSFORM (stage: charge_invoice, journal: 4)
billing.invoice_paid ← charge_invoice(billing.invoice_validated) [read from journal]
idempotency key: invoice:INV-2026-11-004128:1
EFFECTFUL TRANSFORM (stage: charge_invoice, journal: 4)
billing.invoice_declined ← charge_invoice(billing.invoice_validated)
idempotency key: invoice:INV-2026-10-003711:2
$ obzenflow verify calls --run "$RUN" --run "$RESUMED" calls.jsonl
ok: calls.jsonl holds 200 calls from this lineage, 1 repeated by a resume.
$ grep replayed calls.jsonl | jq
{
"result": "replayed",
"key": "invoice:INV-2026-10-003711:2",
"flow_id": "flow_01M6MH4PFHKYZBNZZVH3CTNCJ7"
}
Composites
Automate common patterns with a single composite.
Run AI map-reduce from a single stage.
Summarizing hundreds of incidents with a model means splitting them into groups, calling the model for each one, collecting the answers, and writing one report. With a composite, that’s one stage in your flow.
ObzenFlow builds and journals the four stages underneath, so a crash or a replay never asks the model again for a group it already summarized.
Map
Split the incidents into groups of twenty, and the model summarizes each group.
Reduce
When the input ends, the summaries are collected and the model writes one report.
One stage
The whole pattern sits in your flow as one stage, with one fact in and one fact out.
let report_builder = ai::map(IncidentSummarizer)
.chunks(20)
.reduce(ReportWriter)
.on_eof();
// in stages: { … }
build_incident_report = ai!(
IncidentEvidenceLoaded -> IncidentReport
with ai_breaker
=> report_builder
);Next, run the flow you built.
Run, Replay, and Resume shows the same application running as a service, replayed from its journals, and resumed after a crash.