Payments you can trust.

Customers trust you to charge them once. But whether it’s a forgotten cron job, a batch rerun or a retry after a crash, a payment job that runs again will charge its cards again. At scale, that might be every single customer.

Early one morning, a payment flow is working through the store orders that piled up overnight while new web orders keep arriving. Then its process is killed with a $39.95 charge for store-4474 still out at the gateway, and no answer ever comes back.

Keep reading.

Did that card get charged, and if the flow runs again, will every customer in it pay twice? Learn how this happened to even one of the best engineering teams.

See the four ways

Run the demo yourself.

ObzenFlow's own payment example processes orders through a simulated, unreliable payment gateway, with retries, a circuit breaker and a rate limit, and nothing to install but Cargo.

View it on GitHub

Four ways ObzenFlow helps you build payment flows that are safe to resume and replay.

Challenges with payments

On Monday, September 28, 2009, GitHub’s nightly billing run charged every customer in it twice. A server migration had left the job scheduled in two places, and both runs charged the cards. The script that voided the second charges missed the customers whose cards were being retried, and their duplicates surfaced two days later from a user’s report and were refunded by hand.

Where ObzenFlow fits

ObzenFlow can’t stop you from forgetting a cron job, but it can make every charge it sends durable and safe to resume or replay. That starts with a gateway that honours idempotency keys, which only you can choose, and the same care should go into choosing the runtime that sends them. The excerpts below come from ObzenFlow’s payment example, and the drawing at the end resumes the run the hero kills, from its record.

Apply every charge exactly once.

In ObzenFlow, exactly-once processing of an effect, like charging a credit card, begins with its abstractions. Every single non-deterministic operation is behind a port. You can think of a port as the boundary between determinism that ObzenFlow can control and non-determinism that it can’t.

  1. A port abstracts the third party

    A port is an abstraction over a third party, so a payment gateway port can be answered by a Stripe system, a simulated system like SimulatedProcessor and so on.

  2. An effect is a request

    Each effect on a port is a request. In this case it’s a request to charge a credit card, but it could be any request a port declares.

  3. Guarantees depend on the port and system

    An effect supports different processing guarantees depending on the port and system combination. For instance, Stripe supports idempotency guarantees, and ObzenFlow relies on them. This charge is declared exactly once, with a key built from the order.

gateway.rs
#[obzenflow::port]
#[async_trait]
pub trait PaymentGatewayPort {
    #[effect(type = "payment.authorize", exactly_once)]
    async fn authorize(
        &self,
        request: &AuthorizeRequest,
        key: &IdempotencyKey,
        ctx: &EffectContext,
    ) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError>;
}

impl Keyed for AuthorizeRequest {
    fn idempotency_key(&self) -> IdempotencyKey {
        IdempotencyKey(format!("payment-authorize:{}", self.order_id))
    }
}

#[obzenflow::system("simulated-processor")]
#[async_trait]
impl PaymentGatewayPort for SimulatedProcessor {
    async fn authorize(
        // …
    ) -> Result<outcome![PaymentAuthorized | PaymentDeclined], EffectError> {
        // …
    }
}

Make every charge a first-class effect.

In this case, authorize a request for an order using this port.

  1. The command holds the port

    AuthorizePayment is the stage’s command. It holds the payment gateway port and never sees the system that answers, which the flow builder chooses. The product page builds a charge like this from its request type to the system that answers it.

  2. The record survives the crash

    Each charge goes out through fx.call, and ObzenFlow records how it ended. When a run killed partway through resumes, every order it already decided is read back from that record, so none of those cards is charged a second time.

  3. Every ending is a fact

    Paid and declined are the request’s outcomes, recorded by the call itself. The command emits unavailable only when a call ends without an answer, and each one goes downstream to the stage that handles it.

gateway.rs
#[obzenflow::command]
#[derive(Debug, Clone)]
pub struct AuthorizePayment {
    #[port]
    payment_gateway_port: Port<dyn PaymentGatewayPort>,
    // …
}

#[async_trait]
impl EffectfulTransformHandler for AuthorizePayment {
    type Input = ValidatedOrder;
    type Output = outcome![
        PaymentAuthorized
        | PaymentDeclined
        | PaymentAuthorizationUnavailable
    ];

    async fn process(
        &self,
        order: ValidatedOrder,
        fx: &mut Effects<Self::Output, Self::AllowedEffects>,
    ) -> Result<StageCompletion<Self::Output>, HandlerError> {
        // …
        let request = AuthorizeRequest::for_order(&order);
        if let Err(error) = fx.call(request).await {
            fx.emit(order.authorization_unavailable(&error))
                .await?;
        }
        Ok(fx.complete()?)
    }
    // …
}

Wire every charge into the flow.

This is where it all hooks in. The flow puts a system behind the port, hands the port to the command and runs the command as a stage like any other.

  1. The flow picks the system

    simulated_processor puts SimulatedProcessor behind the payment gateway port, and AuthorizePayment::new hands that port to the command. A Stripe system would take its place, and the command and the stage would stay the same.

  2. The command runs as a stage

    In stages, authorize_payment runs the authorize command on each validated order and produces the three facts it can end with. The retry, circuit breaker and rate limit wrap every call the command makes, so a failed attempt is retried by the runtime.

  3. Each outcome goes to its own stage

    In topology, validated orders come in from validate_order, and paid orders go on to shipping, declined ones to their own record and unavailable ones to manual review.

flow.rs
let port = simulated_processor(&processor_calls);
let authorize = AuthorizePayment::new(port);
// …

Ok(flow! {
    // …
    stages: {
        // …
        authorize_payment = effectful_transform!(
            ValidatedOrder -> {
                PaymentAuthorized,
                PaymentDeclined,
                PaymentAuthorizationUnavailable,
            }
            with {
                authorize_breaker,
                authorize_retry,
                authorize_limiter,
            }
            => authorize,
        );
        // …
    },

    topology: {
        web_orders |> validate_order;
        store_orders |> validate_order;
        validate_order |> authorize_payment;
        validate_order |> invalid_orders;
        validate_order |> cancelled_orders;
        authorize_payment |> paid_orders;
        authorize_payment |> declined_orders;
        authorize_payment |> manual_review;
    }
})

Replay and resume any run to audit every charge.

In ObzenFlow, every run is journaled with each charge’s outcome and key. Kill a run partway through and resume it with one command. Orders it already decided replay from the record with effects suppressed, a charge that was in flight goes out again under the same key, and the run continues live from where it stopped. Add --verify to a replay and it checks itself against the original run, so every charge in any past run can be audited from its record.

Build your next payment flow on ObzenFlow.

ObzenFlow is open source. Star the project to follow its progress, or start a tutorial and build something yourself.

Have a question? Ask it in GitHub Discussions.