Make Side Effects Safe

Declare the payment call as an effect and handle every outcome it can have.

example payment_gateway_resilience

series Durable Execution for Transactions, part 2 of 5

Side effects are at the heart of transactional code. You can think of a side effect as anything that changes another part of the system and is not fully within your control. Often the impact isn’t even returned as part of the function signature.

The best way to think of a side effect is by first thinking about what a function is. A deterministic function takes input and returns output, and can be repeated over and over again. 1 + 1 is deterministic. The + is the operation, the function, and it is a deterministic one.

Now imagine we create a function, and while adding 1 + 1, we also log and update a row in a database. That’s both non-deterministic and side-effecting. The database write can fail, time out or land twice, so the same input no longer gives the same result.

ObzenFlow handles side effects by giving you effectful versions of stages. An effectful transform allows you to hook into ObzenFlow’s algebraic effect system, which abstracts away third-party systems to make all of the requests you perform against them as safe and repeatable as possible (without double charging credit cards, for instance).

In this section you’ll take the riskiest line in the flow, the call that authorizes a payment, and turn it into data. You’ll build it in four small pieces, a stage that declares what can come out, a command that performs the authorization request, a port that describes the call, and a system that answers it. When you’re done, your code never touches the gateway. It hands the runtime a request, and the runtime sends it under a stable key, records the answer, and returns that same answer on every replay. That is how you charge a card once, even through a crash or a replay.

Declare the effectful stage

When a stage needs code that is known to cause side effects, such as a request to a third-party system, or is non-idempotent, such as generating a random number, you declare that request as an effect and use an effectful stage. An effectful stage is declared like any other transform.

flow.rs
authorize_payment = effectful_transform!(
    ValidatedOrder -> {
        PaymentAuthorized,
        PaymentDeclined,
        PaymentAuthorizationUnavailable,
    }
    => authorize,
);

There’s really no difference, except the effectful_transform! here rather than a plain transform. While everything looks the same, it’s important to pay attention to authorize, because that is an effectful command, which you can think of as a special kind of handler for effectful stages.

Commands and ports

A command performs requests through a port, which is an abstraction for any side-effecting code, like a third-party API. Think of it as an interface but with some special features. This one sends an authorization request to the port for each order.

A command is written one of two ways.

One-line commands

If the stage sends one request per input and does nothing else, the command is one line in the flow builder (2), the code above flow! that builds each stage’s handler. perform takes the function that builds the request from the input, and through gives it the port (1).

flow.rs
 1let port = simulated_processor(&processor_calls);
 2let authorize = perform(AuthorizeRequest::for_order).through(port);
 3
 4authorize_payment = effectful_transform!(
 5    ValidatedOrder -> {
 6        PaymentAuthorized,
 7        PaymentDeclined,
 8    }
 9    => authorize,
10);

A stage can only journal the facts its declaration lists, here PaymentAuthorized and PaymentDeclined. It must write one of the two for every order. The only exception is an error, and then the order goes to the stage’s error journal.

That’s the first way to write a command.

Custom command structs

But if the command has logic of its own, you declare a struct and mark it #[obzenflow::command]. In this example we handle outages ourselves. When no answer comes back, the command writes a new fact, PaymentAuthorizationUnavailable, which routes to the manual_review stage instead of the error journal.

gateway.rs
#[obzenflow::command]
#[derive(Debug, Clone)]
pub struct AuthorizePayment {
    #[port]
    payment_gateway_port: Port<dyn PaymentGatewayPort>,
    // …
}

#[port] marks the field that holds the command’s port. The command can perform only the requests that port declares.

The command implements EffectfulTransformHandler. Its type Output names in outcome! the facts this stage can write, the three types its declaration lists.

gateway.rs
#[async_trait]
impl EffectfulTransformHandler for AuthorizePayment {
    type Input = ValidatedOrder;
    type Output = outcome![
        PaymentAuthorized
        | PaymentDeclined
        | PaymentAuthorizationUnavailable
    ];

    async fn process(
        &self,
        order: ValidatedOrder,
        fx: &mut Effects<Self::Output, Self::AllowedEffects>,
    ) -> Result<StageCompletion<Self::Output>, HandlerError> {
        let request = AuthorizeRequest::for_order(&order);
        if let Err(error) = fx.call(request).await {
            fx.emit(order.authorization_unavailable(&error)).await?;
        }
        Ok(fx.complete()?)
    }
}

The flow builder constructs the command with its port, and the stage takes the command as its handler.

flow.rs
let port = simulated_processor(&processor_calls);
let authorize = AuthorizePayment::new(port);

authorize_payment = effectful_transform!(
    ValidatedOrder -> {
        PaymentAuthorized,
        PaymentDeclined,
        PaymentAuthorizationUnavailable,
    }
    => authorize,
);

The command’s fx.call(request) names no method and no gateway. The request type selects the effect, and the port declares that effect.

Declare the effect on the port

A port is the boundary between your own deterministic code and the non-deterministic world outside it. It is an ordinary trait with a method for each side-effecting operation you need to perform. This one has a single method, authorize.

An effect is a side-effecting operation. A port can declare multiple effects. Each effect declares a type (which is what will be written to the journal) and an execution guarantee, which is either at least once or exactly once. This tells the runtime whether or not it’s safe to execute again without an idempotency key.

gateway.rs
#[obzenflow::port]
#[async_trait]
pub trait PaymentGatewayPort {
    #[effect(type = "payment.authorize", exactly_once)]
    async fn authorize(
        &self,
        request: &AuthorizeRequest,
        key: &IdempotencyKey,
        ctx: &EffectContext,
    ) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError>;
}

Here’s an overview of above:

  • type = "payment.authorize" names the effect in the journal.
  • exactly_once states the guarantee, that the call takes effect once even if it is sent again. The other guarantee is at_least_once.
  • ctx carries the run’s context, such as the flow id.
  • authorize returns exactly one of the outcomes outcome! lists, PaymentAuthorized or PaymentDeclined, or an EffectError when no outcome exists.

The request is a plain struct. Its type selects the effect, and its Keyed impl builds the idempotency key.

gateway.rs
#[derive(Debug, Clone, Serialize)]
pub struct AuthorizeRequest {
    order_id: String,
    customer_id: String,
    amount_cents: u64,
    payment_method_state: PaymentMethodState,
    phase: TrafficPhase,
}

impl Keyed for AuthorizeRequest {
    fn idempotency_key(&self) -> IdempotencyKey {
        IdempotencyKey(format!("payment-authorize:{}", self.order_id))
    }
}

Here’s an overview of above:

  • A request type belongs to one effect method, so fx.call with an AuthorizeRequest resolves to authorize.
  • authorize takes a key, so the request implements Keyed. The key is built from the order id before any I/O happens, so every attempt for one order carries the same key, and the system deduplicates on it.

The port declares authorize without a body. A system implements it.

Implement the port with a system

A system implements the port’s trait. Its authorize holds the code that calls the gateway. SimulatedProcessor is the system in this example. A StripeProcessor would implement the same trait, with a call to Stripe in the body.

Info · Systems and processors

System is the framework’s word for whatever implements a port. A system is a physical backend, and a port is the logical abstraction over its systems, so one port can have several systems. In payments, the backend behind a gateway is a payment processor. That’s why the systems here are named SimulatedProcessor and StripeProcessor.

One of the first advantages of the ports and systems method is that we can start off by simulating all of our third-party dependencies. In this case, rather than directly hooking up to Stripe, we can introduce some non-deterministic approximation of what calls to Stripe look like. This helps us work out all sorts of edge cases and non-functional requirements before working on the canonical integration.

gateway.rs
 1#[obzenflow::system("simulated-processor")]
 2#[async_trait]
 3impl PaymentGatewayPort for SimulatedProcessor {
 4    async fn authorize(
 5        &self,
 6        request: &AuthorizeRequest,
 7        key: &IdempotencyKey,
 8        ctx: &EffectContext,
 9    ) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError> {
10        let latency_ms: u64 = ctx.rng("processor_latency").u64(50..=200);
11        ctx.sleep(Duration::from_millis(latency_ms)).await;
12        let times_out = self.outage_times_out(request);
13        if times_out {
14            let hang_ms: u64 = ctx.rng("processor_outage_hang").u64(2_000..=3_000);
15            ctx.sleep(Duration::from_millis(hang_ms)).await;
16        }
17        let response = self
18            .calls
19            .try_answer(key, ctx.flow_id(), |_| {
20                if times_out {
21                    Err(ProcessorFailure::TimedOut)
22                } else {
23                    decide(request)
24                }
25            })
26            .map_err(|error| {
27                EffectError::unavailable("payment_processor", "ledger_write_failed", error)
28            })??;
29        Ok(match response {
30            ProcessorResponse::Charged { authorization_id } => {
31                outcome(request.authorized(authorization_id))
32            }
33            ProcessorResponse::Declined { reason } => outcome(request.declined(reason)),
34        })
35    }
36}

Here’s an overview of above:

  • Line 1 names the system with #[obzenflow::system]. Every call the system answers records a digest of that name, so a replay against a different system is refused.

  • Lines 10 to 16 simulate the processor’s latency. During the scripted outage, the first four calls also hang for two to three seconds and then time out.

  • Lines 17 to 28 get the processor’s response. self.calls is the simulated processor’s own call log, essentially a double of the record Stripe would keep on its side. Like Stripe, it answers a key it has already charged or declined with that same response, so a retry never charges twice. The log lives outside your journal, in target/gateway-calls.jsonl, and at the end you’ll check it against your runs.

  • Lines 29 to 34 turn the processor’s response into the port’s outcome with outcome(...), a PaymentAuthorized or a PaymentDeclined. A timeout comes back as an EffectError instead.

There are two options for how to choose the system behind each port at runtime. You can either hardcode that choice, or you can pull it in from configuration. In the example below, we’ll show you how to wire all of this together via configuration.

Choose the system from config

The example’s obzenflow.toml holds the application’s own settings in an [app] table. processor picks the system behind the payment gateway port, and processor_calls is where the simulated system keeps its call log.

obzenflow.toml
[app]
processor = "simulated"
processor_calls = "target/gateway-calls.jsonl"

When the flow is built, FlowDefinition::materialize_with reads that table into a typed struct and hands it to the code that builds the flow. This example has two arms, and both are simulated. One is the ordinary processor, and the other is slow to answer one payment, for a night you’ll run in the last part. In a production build, you might have one match arm for the simulated processor, one for Stripe, and one for a different kind of double. How many you need depends on how complex your operational environment is, and you might even keep separate stand-ins for separate environments, such as QA.

flow.rs
 1/// The system behind the payment gateway port, chosen by `processor` in the config's `[app]`.
 2#[derive(Clone, Copy, Debug, Deserialize, Serialize)]
 3#[serde(rename_all = "snake_case")]
 4pub enum Processor {
 5    Simulated,
 6    SlowAnswer,
 7}
 8
 9/// The `[app]` table of this example's obzenflow.toml.
10#[derive(Debug, Deserialize, Serialize)]
11#[serde(deny_unknown_fields)]
12pub struct PaymentSettings {
13    processor: Processor,
14    processor_calls: PathBuf,
15}
16
17pub fn build_flow() -> FlowDefinition {
18    FlowDefinition::materialize_with(move |_runtime_config, settings: PaymentSettings| {
19        let port = match settings.processor {
20            Processor::Simulated => simulated_processor(&settings.processor_calls),
21            Processor::SlowAnswer => slow_answer_processor(&settings.processor_calls),
22        };
23        Ok(assemble_flow(
24            fixtures::scripted_web_orders(),
25            fixtures::scripted_store_orders(),
26            Some(FIRST_RECOVERY_PAUSE),
27            AUTHORIZE_CALLS_PER_SECOND,
28            PathBuf::from(DEMO_JOURNAL_ROOT),
29            port,
30        ))
31    })
32}

Here’s an overview of above:

  • Lines 1 to 7 name the systems the setting can choose. processor = "simulated" reads as Processor::Simulated, and "slow_answer" as Processor::SlowAnswer.

  • Lines 9 to 15 are the [app] table as a Rust type. deny_unknown_fields makes a misspelled key fail startup instead of being ignored.

  • Lines 19 to 22 choose the system behind the port.

  • Line 29 hands the port on, and assemble_flow gives it to the command with AuthorizePayment::new(port).

Whichever system the setting picks, it returns the port’s types, so from here on you’re thinking in types. The stage takes a ValidatedOrder, and the effect’s outcome is a PaymentAuthorized or a PaymentDeclined fact. Next, you’ll perform the effect and handle each one.

Perform the effect and handle all possible outcomes

A system returns one of two outcomes, and each outcome is a fact. Three words are easy to blur here.

  • An effect is the side-effecting operation itself, the request the command performs.
  • An outcome is one way the request can turn out. The port lists them in outcome!, and the system returns exactly one.
  • A fact is the record of what happened, PaymentAuthorized or PaymentDeclined, which the journal keeps.

All possible outcomes from an effect are already expressed in the port’s signature, outcome![PaymentAuthorized | PaymentDeclined], so there’s nothing more to declare. As long as each outcome is a fact the stage declares, it will compile. If there’s anything wrong, you’ll get a compile error.

Now that the port lists every answer the gateway can give, you need the code that performs the request. That’s the command’s process method. The runtime calls it once for each order that reaches the stage, and it does three things.

  1. Builds the request from the order.
  2. Performs the effect.
  3. Deals with whatever comes back.

Most of the time, there’s nothing special to deal with unless you have very unique requirements. For instance, let’s say you wanted to emit multiple events on an error. That’s really what process is for. It’s essentially the equivalent of your regular deterministic handler.

gateway.rs
async fn process(
    &self,
    order: ValidatedOrder,
    fx: &mut Effects<Self::Output, Self::AllowedEffects>,
) -> Result<StageCompletion<Self::Output>, HandlerError> {
    // …
    let request = AuthorizeRequest::for_order(&order);
    if let Err(error) = fx.call(request).await {
        fx.emit(order.authorization_unavailable(&error)).await?;
    }
    Ok(fx.complete()?)
}

The only reason we needed to implement process ourselves is that we needed a special event, “authorization unavailable”. These types of decisions are typically determined during planning, such as during an event storming session. That’s why you might reach for a custom process implementation, but if you didn’t have that type of requirement, you could write the one-line version of the above, shown below.

flow.rs
let authorize = perform(AuthorizeRequest::for_order).through(port);

Here’s a recap of the concepts in this chapter.

  • An effect is the request the command performs, AuthorizeRequest, sent through fx.call.
  • An outcome is what the system returns, one of the answers the port lists in outcome![PaymentAuthorized | PaymentDeclined].
  • A fact is what the journal records, PaymentAuthorized or PaymentDeclined from the outcome, or PaymentAuthorizationUnavailable from the command.
  • process runs once for each order. You only write one when the stage needs a fact the port doesn’t list.
  • fx is the handle to the runtime. Its two type parameters are the facts this stage may write and the effects it may perform, so a call or a fact the stage never declared doesn’t compile.
  • fx.call(request) performs the request. The runtime builds the key, applies the protection you’ll add in the next part, has the system answer and journals the outcome’s fact. During replay it returns the recorded outcome and no system is called.
  • fx.emit(...) writes a fact the command authors itself, here the PaymentAuthorizationUnavailable for an order nobody decided.
  • fx.complete() finishes the order. It fails if the handler recorded no fact, so no order leaves this stage without one.

Every order now leaves this stage with a fact recorded in its journal. Next, those facts have to leave the flow, and that’s the job of the sinks.

Deliver to sinks with declared safety

While effects are used primarily for side effects within a flow, sinks are used as terminal nodes at the end of a flow. In other words, conceptually, an effect and a sink are similar in that they perform work outside of the flow (for instance, delivering via async to PostgreSQL), but sinks always are terminal, meaning nothing can subscribe to them. They do not produce journaled output in the same way that other stages do.

Sinks can come in a number of forms, and each type of sink represents a different type of backend system.

  • ConsoleSink prints facts to the terminal, formatted as text, JSON or a table.
  • CsvSink writes facts to a CSV file, through a CsvProjection you define.
  • PostgresSink writes facts to PostgreSQL. It needs the postgres feature.
  • TracingSink emits each fact as a structured tracing event.
  • DiscardSink consumes facts on purpose and delivers them nowhere.
  • For any other destination, you implement InlineSink yourself.

In the case below, for demonstration purposes, we’re going to deliver to our built-in console sink, which pretty prints to the console. Again, you’re seeing a little bit of the philosophical underpinnings of ObzenFlow. Even writing to the console is considered external work.

flow.rs
let record_declined = ConsoleSink::new(console::format_declined);
let record_invalid = ConsoleSink::new(console::format_invalid);
let record_cancelled = ConsoleSink::new(console::format_cancelled);

// Inside flow! { stages: { ... } }
declined_orders = sink!(PaymentDeclined => record_declined.idempotent());
invalid_orders = sink!(InvalidOrder => record_invalid.idempotent());
cancelled_orders = sink!(CancelledOrder => record_cancelled.idempotent());

Pay attention to the code above. There are two core concepts to pay attention to:

  1. The concept of a sink
  2. The concept of redelivery safety

A delivery guarantee is a promise the runtime makes about delivery itself, such as delivering every fact at least once. Redelivery safety is a promise the sink makes to the runtime, that getting the same fact a second time does no harm. The runtime only checks that promise before a replay or resume, because both send recorded facts to the sinks again, and it refuses to start when a sink can’t make it. Every sink is in one of three states.

  • .idempotent() means a repeat is harmless, like a console line or an upsert keyed by id. Replay and resume deliver to it again.
  • .non_idempotent() means a repeat would show up as a duplicate, like a second email. Replay and resume refuse to run unless the operator passes --allow-duplicate-sink-delivery.
  • A sink that declares nothing is treated as .non_idempotent().

In this case, we’re selecting idempotency on all of our sinks, because printing the same line to the console twice is harmless.

Next steps

Now you’ve covered ObzenFlow’s algebraic effects system end to end. You’ve learned:

  • how a command performs work through a port
  • how to connect a port through a system
  • how to create multiple systems for a single port
  • how to configure them
  • eventually, how to deliver outcomes to a sink

We’ll change gears in the next tutorial and actually stress out our application with the simulated outage. You’ll learn how to wire up circuit breakers, retries, and rate limiters around an effect, and then simulate how they interact. This will be an end-to-end lesson on ObzenFlow’s resilience patterns.