Creating a resilience policy
A resilience policy is the set of middleware the stage lists after the with keyword. In this case, we use all middleware combined as a single policy: a circuit breaker, a retry, and a rate limiter, each of which you’ll learn how to configure in the next three chapters.
As you continue to read, keep in mind that this is really how you specify non-functional requirements in ObzenFlow and encode those in the program itself (typically through a programmatic default and configuration overrides). But all policies start with specifying middleware after the with keyword.
authorize_payment = effectful_transform!(
ValidatedOrder -> {
PaymentAuthorized,
PaymentDeclined,
PaymentAuthorizationUnavailable,
}
with {
authorize_breaker,
authorize_retry,
authorize_limiter,
}
=> authorize,
);By default, a stage doesn’t have any middleware and, for the most part, doesn’t require middleware. However, effects typically do require, at the very minimum, a rate limiter to control flow to external systems and, ideally, a circuit breaker to encode rules about how to behave during failure and outage. We’ll dive into how each middleware works individually, and then we’ll discuss how they combine and work together seamlessly.
Configure the circuit breaker
A circuit breaker works like the one in your house. When a circuit draws too much current, the breaker trips and cuts the power before the wiring overheats. In software, a circuit breaker sits in front of calls to another system and keeps track of failures. When too many of them fail or hang, it trips and stops sending calls for a configurable amount of time that you specify.
This benefits your flow because, rather than waiting on calls that will fail anyway, it fails fast and releases resources back to the system. It also protects external systems and gives them room to recover instead of hammering them while they’re already struggling.
A breaker is always in one of three states.
- While it’s closed, calls go through and the breaker keeps score.
- While it’s open, it refuses every call straight away.
- After a cool-down it turns half-open and lets a probe call through to test the gateway. A probe that succeeds closes the breaker, and one that fails opens it again.
If you’ve used Pekko, Resilience4j or Polly, it’s the same pattern. Here’s the breaker the payment stage uses.
1const AUTHORIZE_BREAKER_OPEN_FOR: Duration = Duration::from_secs(5);
2const AUTHORIZE_BREAKER_OPEN_FOR_MULTIPLIER: f64 = 2.0;
3const AUTHORIZE_BREAKER_MAX_OPEN_FOR: Duration = Duration::from_secs(20);
4
5let authorize_breaker = circuit_breaker()
6 .count_window(5)
7 .minimum_calls(5)
8 .failure_rate_threshold(0.6)
9 .slow_call_duration(Duration::from_millis(250))
10 .slow_call_rate_threshold(0.5)
11 .open_for(AUTHORIZE_BREAKER_OPEN_FOR)
12 .open_for_multiplier(AUTHORIZE_BREAKER_OPEN_FOR_MULTIPLIER)
13 .max_open_for(AUTHORIZE_BREAKER_MAX_OPEN_FOR)
14 .probes(1);The breaker is built as authorize_breaker (5), the name the payment stage lists after with. Its numbers work together to decide three things.
- When it opens (6-10). The breaker looks back over the last five calls, and it waits until it has seen five before it judges at all. A rate over one or two calls means nothing, and without that wait, a run whose first call fails would show a 100% failure rate and open the breaker on a single failure. Once it has five, it opens when three of them failed, or when three took 250 milliseconds or longer at the gateway. Slow calls count because a gateway that hangs hurts the flow before it ever returns an error. A decline never counts as a failure, because it’s the gateway’s answer.
- How long it stays open (11-13). The first time, it stays open for five seconds. Each time a probe finds the gateway still down, the wait doubles, to ten seconds and then twenty, and it never goes past twenty. Once the breaker closes, the wait goes back to five seconds.
- How it tests recovery (14). When the wait is up, it lets one call through as a probe. If the probe succeeds, the breaker closes and starts counting from scratch. If it fails, the breaker opens again for the longer wait.
When you read the outage as state transitions later on, the journal’s record of the breaker opening shows these numbers at work, with five calls observed, three failed and three slow.
Configure the retry
A retry handles the small failures. Networks drop packets, connections reset, and a busy gateway sometimes takes too long to answer one call and then answers the next one fine. Failures like these pass on their own, so the simplest fix is to wait a moment and try again, the way you’d redial a phone call that dropped.
Retrying has two risks. If the gateway is really down, every retry adds load at the worst possible moment. And if a call went through before it timed out, a second call could charge the card twice.
ObzenFlow only retries a call that’s safe to repeat. The payment call is exactly_once, so it carries an idempotency key the gateway uses to recognize a repeat.
It’s possible to specify a retry on its own or a circuit breaker on its own, but they’re most powerful when composed together. The retry handles one slow or failed call, and the breaker handles an outage. Every call the retry makes counts toward the breaker’s window, so a few failed retries are exactly what tells the breaker the gateway is down. Once the breaker opens, the retry stops calling, and the order waits instead. It goes through when the breaker lets calls through again, as long as the outage ends within the retry’s wait limit, two minutes by default. Here’s the retry the payment stage uses.
1let authorize_retry = retry()
2 .fixed_delay(Duration::from_millis(250))
3 .max_attempts(3)
4 .attempt_start_window(Duration::from_secs(30));The retry is built as authorize_retry (1), the name the payment stage lists after with. Its numbers work together like this.
- How it retries (2-3). When a call times out, the retry waits 250 milliseconds and calls again, up to three calls in all, counting the first. Without
fixed_delay,retry()backs off instead, starting at 250 milliseconds and doubling up to four seconds, with jitter so waiting calls don’t all retry at the same moment. A decline is the gateway’s answer, so it’s never retried. - How long it keeps trying (4). No new attempt starts more than 30 seconds after the first call. The window only stops new attempts, so a call that’s already in flight is never cut off.
- What it does when the breaker opens. The retry doesn’t keep calling, and it doesn’t give up either. It waits for the breaker to let calls through again, for up to two minutes into the outage, which is the default since this code doesn’t change it. After each wait, the order gets a fresh three calls and a fresh 30 seconds.
So in this example, a bad call costs an order at most three calls within 30 seconds, and an outage can hold it for up to two minutes before it ends as a recorded failure.
Configure the rate limiter
A rate limiter caps how often the stage calls the gateway. Most payment processors and APIs publish a rate limit, and a client that goes over it gets throttled or turned away. Even without a published limit, a burst of orders shouldn’t turn into a burst of calls against someone else’s system.
ObzenFlow’s limiter is a token bucket. Tokens drip into the bucket at the rate you set, and each call takes one before it goes out. When the bucket is empty, the next call waits for the next token instead of failing, so no order is dropped. Only this stage slows down, and the orders behind it wait in the journal until a token frees up.
The limiter works alongside the other two. Every attempt needs a token, first or retried, so a blip that sets off retries can’t push the stage over its limit. And while the breaker is open, no calls go out, so no tokens are spent on a gateway that’s down. Here’s the limiter the payment stage uses.
let authorize_limiter = rate_limit(authorize_calls_per_second);The example passes one call a second, and a retried call needs a permit like any other.
The payment stage’s set becomes one policy around its payment effect. Every attempt, first or retried, needs the breaker’s admission and a permit from the limiter. While the breaker is open, the retry waits for it instead of failing the call, and however many calls an order takes, the journal records one outcome for its payment.
With that small amount of code, you’ve added a fully configured resilience policy to a flow. ObzenFlow handles all of the complexity of composing individual middleware into a seamless chain that works together as a unified policy.
Next steps
Now here’s an even more powerful concept. Every single decision ObzenFlow’s middleware makes is actually an event in ObzenFlow’s journal, which sits alongside business events that the stage wrote. In other words, ObzenFlow’s wide events.
Philosophy and event-sourced, finite-state-backed architecture means that an event that comes from middleware is treated by the framework in a very similar way as a business event. It’s part of the durable record. When you audit a run, you’ll see every single circuit breaker that opens in the journal. You’ll see every rate limit that fires, and so forth, which is a postmortem analysis and root-cause superpower.
In Run and Investigate, you’ll learn how to inspect journals with the ObzenFlow CLI and the obzenflow show command, including how to read a single record, follow one order through four journals by its clock, and read the outage as state transitions. Then in Resume an Interrupted Flow, you’ll kill a run partway through, resume it, and check that no payment was decided twice.