Declare the effectful stage
When a stage needs code that is known to cause side effects, such as a request to a third-party system, or is non-idempotent, such as generating a random number, you declare that request as an effect and use an effectful stage. An effectful stage is declared like any other transform.
authorize_payment = effectful_transform!(
ValidatedOrder -> {
PaymentAuthorized,
PaymentDeclined,
PaymentAuthorizationUnavailable,
}
=> authorize,
);There’s really no difference, except the effectful_transform! here rather than a plain transform. While everything looks the same, it’s important to pay attention to authorize, because that is an effectful command, which you can think of as a special kind of handler for effectful stages.
Commands and ports
A command performs requests through a port, which is an abstraction for any side-effecting code, like a third-party API. Think of it as an interface but with some special features. This one sends an authorization request to the port for each order.
A command is written one of two ways.
One-line commands
If the stage sends one request per input and does nothing else, the command is one line in the flow builder (2), the code above flow! that builds each stage’s handler. perform takes the function that builds the request from the input, and through gives it the port (1).
1let port = simulated_processor(&processor_calls);
2let authorize = perform(AuthorizeRequest::for_order).through(port);
3
4authorize_payment = effectful_transform!(
5 ValidatedOrder -> {
6 PaymentAuthorized,
7 PaymentDeclined,
8 }
9 => authorize,
10);A stage can only journal the facts its declaration lists, here PaymentAuthorized and PaymentDeclined. It must write one of the two for every order. The only exception is an error, and then the order goes to the stage’s error journal.
That’s the first way to write a command.
Custom command structs
But if the command has logic of its own, you declare a struct and mark it #[obzenflow::command]. In this example we handle outages ourselves. When no answer comes back, the command writes a new fact, PaymentAuthorizationUnavailable, which routes to the manual_review stage instead of the error journal.
#[obzenflow::command]
#[derive(Debug, Clone)]
pub struct AuthorizePayment {
#[port]
payment_gateway_port: Port<dyn PaymentGatewayPort>,
// …
}#[port] marks the field that holds the command’s port. The command can perform only the requests that port declares.
The command implements EffectfulTransformHandler. Its type Output names in outcome! the facts this stage can write, the three types its declaration lists.
#[async_trait]
impl EffectfulTransformHandler for AuthorizePayment {
type Input = ValidatedOrder;
type Output = outcome![
PaymentAuthorized
| PaymentDeclined
| PaymentAuthorizationUnavailable
];
async fn process(
&self,
order: ValidatedOrder,
fx: &mut Effects<Self::Output, Self::AllowedEffects>,
) -> Result<StageCompletion<Self::Output>, HandlerError> {
let request = AuthorizeRequest::for_order(&order);
if let Err(error) = fx.call(request).await {
fx.emit(order.authorization_unavailable(&error)).await?;
}
Ok(fx.complete()?)
}
}The flow builder constructs the command with its port, and the stage takes the command as its handler.
let port = simulated_processor(&processor_calls);
let authorize = AuthorizePayment::new(port);
authorize_payment = effectful_transform!(
ValidatedOrder -> {
PaymentAuthorized,
PaymentDeclined,
PaymentAuthorizationUnavailable,
}
=> authorize,
);The command’s fx.call(request) names no method and no gateway. The request type selects the effect, and the port declares that effect.
Declare the effect on the port
A port is the boundary between your own deterministic code and the non-deterministic world outside it. It is an ordinary trait with a method for each side-effecting operation you need to perform. This one has a single method, authorize.
An effect is a side-effecting operation. A port can declare multiple effects. Each effect declares a type (which is what will be written to the journal) and an execution guarantee, which is either at least once or exactly once. This tells the runtime whether or not it’s safe to execute again without an idempotency key.
#[obzenflow::port]
#[async_trait]
pub trait PaymentGatewayPort {
#[effect(type = "payment.authorize", exactly_once)]
async fn authorize(
&self,
request: &AuthorizeRequest,
key: &IdempotencyKey,
ctx: &EffectContext,
) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError>;
}Here’s an overview of above:
type = "payment.authorize"names the effect in the journal.exactly_oncestates the guarantee, that the call takes effect once even if it is sent again. The other guarantee isat_least_once.ctxcarries the run’s context, such as the flow id.authorizereturns exactly one of the outcomesoutcome!lists,PaymentAuthorizedorPaymentDeclined, or anEffectErrorwhen no outcome exists.
The request is a plain struct. Its type selects the effect, and its Keyed impl builds the idempotency key.
#[derive(Debug, Clone, Serialize)]
pub struct AuthorizeRequest {
order_id: String,
customer_id: String,
amount_cents: u64,
payment_method_state: PaymentMethodState,
phase: TrafficPhase,
}
impl Keyed for AuthorizeRequest {
fn idempotency_key(&self) -> IdempotencyKey {
IdempotencyKey(format!("payment-authorize:{}", self.order_id))
}
}Here’s an overview of above:
- A request type belongs to one effect method, so
fx.callwith anAuthorizeRequestresolves toauthorize. authorizetakes akey, so the request implementsKeyed. The key is built from the order id before any I/O happens, so every attempt for one order carries the same key, and the system deduplicates on it.
The port declares authorize without a body. A system implements it.
Implement the port with a system
A system implements the port’s trait. Its authorize holds the code that calls the gateway. SimulatedProcessor is the system in this example. A StripeProcessor would implement the same trait, with a call to Stripe in the body.
System is the framework’s word for whatever implements a port. A system is a physical backend, and a port is the logical abstraction over its systems, so one port can have several systems. In payments, the backend behind a gateway is a payment processor. That’s why the systems here are named SimulatedProcessor and StripeProcessor.
One of the first advantages of the ports and systems method is that we can start off by simulating all of our third-party dependencies. In this case, rather than directly hooking up to Stripe, we can introduce some non-deterministic approximation of what calls to Stripe look like. This helps us work out all sorts of edge cases and non-functional requirements before working on the canonical integration.
1#[obzenflow::system("simulated-processor")]
2#[async_trait]
3impl PaymentGatewayPort for SimulatedProcessor {
4 async fn authorize(
5 &self,
6 request: &AuthorizeRequest,
7 key: &IdempotencyKey,
8 ctx: &EffectContext,
9 ) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError> {
10 let latency_ms: u64 = ctx.rng("processor_latency").u64(50..=200);
11 ctx.sleep(Duration::from_millis(latency_ms)).await;
12 let times_out = self.outage_times_out(request);
13 if times_out {
14 let hang_ms: u64 = ctx.rng("processor_outage_hang").u64(2_000..=3_000);
15 ctx.sleep(Duration::from_millis(hang_ms)).await;
16 }
17 let response = self
18 .calls
19 .try_answer(key, ctx.flow_id(), |_| {
20 if times_out {
21 Err(ProcessorFailure::TimedOut)
22 } else {
23 decide(request)
24 }
25 })
26 .map_err(|error| {
27 EffectError::unavailable("payment_processor", "ledger_write_failed", error)
28 })??;
29 Ok(match response {
30 ProcessorResponse::Charged { authorization_id } => {
31 outcome(request.authorized(authorization_id))
32 }
33 ProcessorResponse::Declined { reason } => outcome(request.declined(reason)),
34 })
35 }
36}Here’s an overview of above:
Line 1 names the system with
#[obzenflow::system]. Every call the system answers records a digest of that name, so a replay against a different system is refused.Lines 10 to 16 simulate the processor’s latency. During the scripted outage, the first four calls also hang for two to three seconds and then time out.
Lines 17 to 28 get the processor’s response.
self.callsis the simulated processor’s own call log, essentially a double of the record Stripe would keep on its side. Like Stripe, it answers a key it has already charged or declined with that same response, so a retry never charges twice. The log lives outside your journal, intarget/gateway-calls.jsonl, and at the end you’ll check it against your runs.Lines 29 to 34 turn the processor’s response into the port’s outcome with
outcome(...), aPaymentAuthorizedor aPaymentDeclined. A timeout comes back as anEffectErrorinstead.
There are two options for how to choose the system behind each port at runtime. You can either hardcode that choice, or you can pull it in from configuration. In the example below, we’ll show you how to wire all of this together via configuration.
Choose the system from config
The example’s obzenflow.toml holds the application’s own settings in an [app] table. processor picks the system behind the payment gateway port, and processor_calls is where the simulated system keeps its call log.
[app]
processor = "simulated"
processor_calls = "target/gateway-calls.jsonl"When the flow is built, FlowDefinition::materialize_with reads that table into a typed struct and hands it to the code that builds the flow. This example has two arms, and both are simulated. One is the ordinary processor, and the other is slow to answer one payment, for a night you’ll run in the last part. In a production build, you might have one match arm for the simulated processor, one for Stripe, and one for a different kind of double. How many you need depends on how complex your operational environment is, and you might even keep separate stand-ins for separate environments, such as QA.
1/// The system behind the payment gateway port, chosen by `processor` in the config's `[app]`.
2#[derive(Clone, Copy, Debug, Deserialize, Serialize)]
3#[serde(rename_all = "snake_case")]
4pub enum Processor {
5 Simulated,
6 SlowAnswer,
7}
8
9/// The `[app]` table of this example's obzenflow.toml.
10#[derive(Debug, Deserialize, Serialize)]
11#[serde(deny_unknown_fields)]
12pub struct PaymentSettings {
13 processor: Processor,
14 processor_calls: PathBuf,
15}
16
17pub fn build_flow() -> FlowDefinition {
18 FlowDefinition::materialize_with(move |_runtime_config, settings: PaymentSettings| {
19 let port = match settings.processor {
20 Processor::Simulated => simulated_processor(&settings.processor_calls),
21 Processor::SlowAnswer => slow_answer_processor(&settings.processor_calls),
22 };
23 Ok(assemble_flow(
24 fixtures::scripted_web_orders(),
25 fixtures::scripted_store_orders(),
26 Some(FIRST_RECOVERY_PAUSE),
27 AUTHORIZE_CALLS_PER_SECOND,
28 PathBuf::from(DEMO_JOURNAL_ROOT),
29 port,
30 ))
31 })
32}Here’s an overview of above:
Lines 1 to 7 name the systems the setting can choose.
processor = "simulated"reads asProcessor::Simulated, and"slow_answer"asProcessor::SlowAnswer.Lines 9 to 15 are the
[app]table as a Rust type.deny_unknown_fieldsmakes a misspelled key fail startup instead of being ignored.Lines 19 to 22 choose the system behind the port.
Line 29 hands the port on, and
assemble_flowgives it to the command withAuthorizePayment::new(port).
Whichever system the setting picks, it returns the port’s types, so from here on you’re thinking in types. The stage takes a ValidatedOrder, and the effect’s outcome is a PaymentAuthorized or a PaymentDeclined fact. Next, you’ll perform the effect and handle each one.
Perform the effect and handle all possible outcomes
A system returns one of two outcomes, and each outcome is a fact. Three words are easy to blur here.
- An effect is the side-effecting operation itself, the request the command performs.
- An outcome is one way the request can turn out. The port lists them in
outcome!, and the system returns exactly one. - A fact is the record of what happened,
PaymentAuthorizedorPaymentDeclined, which the journal keeps.
All possible outcomes from an effect are already expressed in the port’s signature, outcome![PaymentAuthorized | PaymentDeclined], so there’s nothing more to declare. As long as each outcome is a fact the stage declares, it will compile. If there’s anything wrong, you’ll get a compile error.
Now that the port lists every answer the gateway can give, you need the code that performs the request. That’s the command’s process method. The runtime calls it once for each order that reaches the stage, and it does three things.
- Builds the request from the order.
- Performs the effect.
- Deals with whatever comes back.
Most of the time, there’s nothing special to deal with unless you have very unique requirements. For instance, let’s say you wanted to emit multiple events on an error. That’s really what process is for. It’s essentially the equivalent of your regular deterministic handler.
async fn process(
&self,
order: ValidatedOrder,
fx: &mut Effects<Self::Output, Self::AllowedEffects>,
) -> Result<StageCompletion<Self::Output>, HandlerError> {
// …
let request = AuthorizeRequest::for_order(&order);
if let Err(error) = fx.call(request).await {
fx.emit(order.authorization_unavailable(&error)).await?;
}
Ok(fx.complete()?)
}The only reason we needed to implement process ourselves is that we needed a special event, “authorization unavailable”. These types of decisions are typically determined during planning, such as during an event storming session. That’s why you might reach for a custom process implementation, but if you didn’t have that type of requirement, you could write the one-line version of the above, shown below.
let authorize = perform(AuthorizeRequest::for_order).through(port);Here’s a recap of the concepts in this chapter.
- An effect is the request the command performs,
AuthorizeRequest, sent throughfx.call. - An outcome is what the system returns, one of the answers the port lists in
outcome![PaymentAuthorized | PaymentDeclined]. - A fact is what the journal records,
PaymentAuthorizedorPaymentDeclinedfrom the outcome, orPaymentAuthorizationUnavailablefrom the command. processruns once for each order. You only write one when the stage needs a fact the port doesn’t list.fxis the handle to the runtime. Its two type parameters are the facts this stage may write and the effects it may perform, so a call or a fact the stage never declared doesn’t compile.fx.call(request)performs the request. The runtime builds the key, applies the protection you’ll add in the next part, has the system answer and journals the outcome’s fact. During replay it returns the recorded outcome and no system is called.fx.emit(...)writes a fact the command authors itself, here thePaymentAuthorizationUnavailablefor an order nobody decided.fx.complete()finishes the order. It fails if the handler recorded no fact, so no order leaves this stage without one.
Every order now leaves this stage with a fact recorded in its journal. Next, those facts have to leave the flow, and that’s the job of the sinks.
Deliver to sinks with declared safety
While effects are used primarily for side effects within a flow, sinks are used as terminal nodes at the end of a flow. In other words, conceptually, an effect and a sink are similar in that they perform work outside of the flow (for instance, delivering via async to PostgreSQL), but sinks always are terminal, meaning nothing can subscribe to them. They do not produce journaled output in the same way that other stages do.
Sinks can come in a number of forms, and each type of sink represents a different type of backend system.
ConsoleSinkprints facts to the terminal, formatted as text, JSON or a table.CsvSinkwrites facts to a CSV file, through aCsvProjectionyou define.PostgresSinkwrites facts to PostgreSQL. It needs thepostgresfeature.TracingSinkemits each fact as a structuredtracingevent.DiscardSinkconsumes facts on purpose and delivers them nowhere.- For any other destination, you implement
InlineSinkyourself.
In the case below, for demonstration purposes, we’re going to deliver to our built-in console sink, which pretty prints to the console. Again, you’re seeing a little bit of the philosophical underpinnings of ObzenFlow. Even writing to the console is considered external work.
let record_declined = ConsoleSink::new(console::format_declined);
let record_invalid = ConsoleSink::new(console::format_invalid);
let record_cancelled = ConsoleSink::new(console::format_cancelled);
// Inside flow! { stages: { ... } }
declined_orders = sink!(PaymentDeclined => record_declined.idempotent());
invalid_orders = sink!(InvalidOrder => record_invalid.idempotent());
cancelled_orders = sink!(CancelledOrder => record_cancelled.idempotent());Pay attention to the code above. There are two core concepts to pay attention to:
- The concept of a sink
- The concept of redelivery safety
A delivery guarantee is a promise the runtime makes about delivery itself, such as delivering every fact at least once. Redelivery safety is a promise the sink makes to the runtime, that getting the same fact a second time does no harm. The runtime only checks that promise before a replay or resume, because both send recorded facts to the sinks again, and it refuses to start when a sink can’t make it. Every sink is in one of three states.
.idempotent()means a repeat is harmless, like a console line or an upsert keyed by id. Replay and resume deliver to it again..non_idempotent()means a repeat would show up as a duplicate, like a second email. Replay and resume refuse to run unless the operator passes--allow-duplicate-sink-delivery.- A sink that declares nothing is treated as
.non_idempotent().
In this case, we’re selecting idempotency on all of our sinks, because printing the same line to the console twice is harmless.
Next steps
Now you’ve covered ObzenFlow’s algebraic effects system end to end. You’ve learned:
- how a command performs work through a port
- how to connect a port through a system
- how to create multiple systems for a single port
- how to configure them
- eventually, how to deliver outcomes to a sink
We’ll change gears in the next tutorial and actually stress out our application with the simulated outage. You’ll learn how to wire up circuit breakers, retries, and rate limiters around an effect, and then simulate how they interact. This will be an end-to-end lesson on ObzenFlow’s resilience patterns.