Run Live AI Inference from a Real Endpoint

A journaled AI tutorial with chunking, map-reduce, recorded prompts, and replay.

example hn_ai_digest_demo

Info · Start with the earlier tutorials first

This tutorial assumes you have already worked through Getting Started and Model Bank Transactions as a Flow. Getting Started explains FlowApplication, flow!, stage types, topology, journals, and replay. The bank tutorial adds HTTP ingress, joins, and projections. This page carries that same flow model into a model-backed workload without losing determinism, inspectability, or replay.

This example pulls live data from the Hacker News Firebase API and runs AI inference through a local Ollama model to produce a journaled, replayable digest. This is the kind of program you would reach for to build an AI summarization service, a content triage pipeline, or a retrieval-and-enrichment flow, any model-backed workload that still needs deterministic scaffolding around the model call.

It’s worth pointing out that this example demonstrates how ObzenFlow handles some of the trickiest parts of running live AI inference for you. Reliability comes from circuit breakers that contain a failing news feed or an overloaded model instead of letting it stall the whole flow, and chunking splits a large input into groups small enough for the model’s context window before anything is sent to it. Out of the box, it connects to Ollama for the inference step itself.

Let’s get started by briefly outlining the program. We’ll grab a number of posts from Hacker News, then run them through local inference to summarize trending topics. This type of summarization is a bread and butter example for local inference.

Step 0. Designing the digest flow

This example started as a little tool I built for myself to test my ideas behind live durable inference, and it was one of the motivating designs in my head before ObzenFlow existed. It grabs trending Hacker News stories, runs all of them through inference, and produces a summary.

This is exactly the kind of workload you might use to summarize support tickets, incidents, or orders, taking a finite or infinite stream of data and working out how to smooth it and chunk it. For instance, you could follow the same kind of approach to emit summaries of support tickets. You could even take it to the next level by leveraging joins to enrich support tickets with detailed customer information and historical support requests. Really, the sky is the limit. This is the basic shape of such a workflow.

Let’s jump right into domain modeling, which we’ve already covered in the previous two tutorials. (If this is too abrupt, please go and re-review those before continuing.)

The facts in this flow

For a Hacker News digest, the interesting things that happen over time are short to list.

  1. A story is fetched from the Hacker News API.
  2. A story is formatted into a stable, display-ready record.
  3. The formatted stories are grouped into chunks of ten.
  4. A chunk is summarized by the model, once per group.
  5. The chunk summaries are reduced into one final digest.

This might be a strange way to think about ingesting data, but in terms of auditability, it’s actually critical.

One of the challenges teams will face over time is that inference is very good at summaries, but it’s also prone to hallucinating. Journaling how input data is fetched and formatted, chunked, and submitted for inference is the difference between being able to pass an audit and being able to respond intelligently during an incident or a production investigation.

Those facts become the event types in the flow. Here they are in trimmed form, drawn from examples/hn_ai_digest_demo/domain.rs and flow.rs.

// Raw item from the Hacker News Firebase API. Almost every field is
// optional because deleted and malformed items do show up in practice.
struct HnStory {
    id: HnStoryId,
    title: Option<String>,
    url: Option<String>,
    by: Option<String>,
    score: Option<u32>,
    descendants: Option<u32>,
}

// A cleaned, display-ready story.
struct FormattedStory {
    id: HnStoryId,
    title: String,
    url: String,
    author: String,
    points: u32,
    comments: u32,
}

// One chunk's partial summary.
struct HnDigestGroupSummary {
    output_markdown: String,
}

// The final evidence record. It carries the input stories, the exact prompts,
// the partial summaries, and the final output together. Shown in full in Step 3.
struct HnDigestSummary { /* ... */ }

The chunks need no type of your own. The framework’s Chunk<FormattedStory> carries each group of stories to the model.

Sketching the flow

With the facts named, you can wire the entire flow before writing a single handler. This is the same placeholder-first workflow you saw in Getting Started and in examples/char_transform_skeleton.rs. Every handler is placeholder!(), but the event types and the topology are real, so the skeleton compiles and you can inspect the shape in ObzenFlow Studio before any business logic exists.

flow! {
    name: "hn_ai_digest_demo",
    journals: disk_journals(PathBuf::from("target/hn-ai-digest-logs")),

    stages: {
        hn_stories     = async_source!(HnStory => placeholder!());
        formatter      = transform!(HnStory -> FormattedStory => placeholder!());
        digest         = stateful!(FormattedStory -> HnDigestSummary => placeholder!());
        digest_summary = sink!(HnDigestSummary => placeholder!());
    },

    topology: {
        hn_stories |> formatter;
        formatter  |> digest;
        digest     |> digest_summary;
    }
}

The topology is worth pausing on. Stories come in, get formatted, get turned into a digest, and leave through the sink.

Walking the skeleton from left to right, here is the thinking behind each placeholder.

  • Ingestion (hn_stories). The first question is where the data comes from and whether it ever ends. Here, it is the Hacker News API. At design time you do not care whether the bytes arrive from the live endpoint or a local mock, only that the stage produces HnStory events.

  • Formatting (formatter). Raw APIs are messy. Fields are optional, some entries are deleted, and titles or URLs can be missing. Rather than let that mess leak into the rest of the flow, you decide to normalize each item into a clean FormattedStory right away.

  • Digest (digest). This is the heart of the program, and it starts with a design decision that shapes everything after it. Do you want one summary per story, or one digest over all of them? You want the digest, and the striking thing at design time is how little else you have to decide. A stream of stories goes in, one HnDigestSummary comes out. Many inputs and one output is the shape of a stateful stage, so you sketch it as a stateful! placeholder and move on. In Step 3 that row becomes an ai! stage, which takes a real handler and has no placeholder form. The genuinely hard questions, such as how to fit dozens or hundreds of stories through a model with a fixed context window, are implementation details of this one stage.

  • Output (digest_summary). Finally, where does the result leave the system? In this example the sink prints to the console, but the same HnDigestSummary could go to email, Slack, a database, or another flow. The sink is the flow boundary, and choosing it is the last design decision rather than the first.

Tip · Think in types first before implementation

Because the skeleton compiles, you do not have to hold all of this in your head at once. You can wire these four placeholders and see the real shape compile before committing to a single line of behaviour. If a stage sits in the wrong place or a type does not line up, you find out now, while the flow is still cheap to change.

Next let’s run the example end-to-end so you can get a feel for what it does.

Step 1. Run the digest end to end

The fastest way to build a mental model of this flow is to run it before reading the code, exactly as you did with the bank service. The default run needs no network connection for its data, because the example ships with a built-in mock Hacker News server. It does still need a local model for inference, so set up Ollama first.

Set up Ollama

brew install ollama
ollama serve
ollama pull llama3.1:8b

You can use any model you like, but a small one such as llama3.1:8b keeps the demo fast. If you would rather use a hosted provider instead of Ollama, set endpoint = "openai" and an OpenAI model under [ai.models] in examples/hn_ai_digest_demo/obzenflow.toml, then export OPENAI_API_KEY before running. Any other provider is defined once under [ai.endpoints.<name>], with its provider, its URL, and the name of the environment variable that holds its key, and [ai.models] picks it with endpoint = "<name>". When you use a hosted provider, your prompts and the story text are sent to that provider, so be deliberate about it.

Run the mock example

If you have already cloned the framework repo from the earlier tutorials, stay in that working directory. Otherwise clone it now.

git clone https://github.com/ObzenFlow/obzenflow
cd obzenflow

Run the example with its default settings. The data comes from the built-in mock server, and inference runs against your local Ollama model.

cargo run -p obzenflow --example hn_ai_digest_demo --features "http-pull ai postgres" -- \
    --config examples/hn_ai_digest_demo/obzenflow.toml

The build includes the postgres feature because the digest sink can also write to PostgreSQL. The obzenflow.toml file selects the console sink and the Ollama model for this run.

The mock server only removes the network dependency on the data side. The model call is still real, which is the point.

The first thing you see is the startup banner that with_presentation adds, which records the configuration for the run.

HN AI Digest Demo
=================
Fetch top HN stories, then generate a markdown digest via Rig-backed LLM transforms.

  source_breaker: 3 failures, 2s cooldown

Below the banner, the run lists the [app] settings it read and the file they came from, including the source mode, the story limit, the poll timeout, and the number of stories per chunk. Nothing about the setup is left implicit.

When the flow finishes, the sink prints the evidence trail and the final digest. The header carries the run metadata and the exact prompts.

HN AI Digest — Summary
======================
mode: mock
base_url: mock://hacker-news/
ai_provider: ollama
ai_model: llama3.1:8b
token_estimator: Heuristic
stories_fetched: 55
groups: 6

Chat prompt (system)
--------------------
You write concise, skimmable Hacker News digests from a
list of headlines + URLs. Be neutral, avoid hype, and do
not invent facts beyond what the titles imply.

Chat prompt (user)
------------------
Write a concise Markdown digest of the following Hacker
News chunk summaries.

- Do not invent facts that are not implied by the titles.
- Start the response immediately with "## What's topical
  today" (no intro).
- Include: Thesis, Themes, Notable stories, Watch.
- Cite stories only with Markdown footnote markers like
  [^12]; preserve the story numbers from the chunk
  summaries and do not use parenthesized citations.
- Do not add URLs, footnote definitions, or a Links
  section; the application appends verified links.
- Avoid generic wrap-ups.

Chunk summaries:
(the six group summaries the map step produced are appended here)

Everything in that header is recorded as part of the run. The provider, the model, the token estimator, the number of groups, and the exact system and user prompts are all fields on a typed event. You’ll learn a little bit more about this in the final section of the tutorial when we cover operations and dig into topics like the manifest.

After the header, the sink prints the input stories it fed to the model, and then the digest the model produced. The block below is adapted from a run against llama3.1:8b, trimmed and wrapped here so it fits without scrolling. Because the data side is mocked the stories are synthetic, and because the model is non-deterministic your wording will differ.

Input data (stories)
--------------------
1. Rust: tooling update #100 (180 points, 0 comments) by author1
    https://example.com/hn/100
2. AI: benchmark #101 (178 points, 1 comments) by author2
    https://example.com/hn/101
3. Databases: postmortem #102 (176 points, 2 comments) by author3
    https://example.com/hn/102
...

Output (markdown)
-----------------
## What's topical today

### Thesis
The current Hacker News discussions revolve around various
topics in computer science, including programming languages,
AI, databases, web development, operating systems, security,
browsers, Linux, compilers, robotics, and distributed
systems.

### Themes

* Programming Languages: Updates to the Rust toolchain and
  other language-related developments. [^1] [^7]
* AI and Machine Learning: Benchmarking different
  frameworks' performance and in-depth looks at AI
  techniques. [^2] [^48]
* Databases: Postmortem analyses of database failures, new
  database systems, and migration guides for distributed
  systems. [^3] [^12] [^21] [^28] [^49]
* Web Development: Launch of a new browser with improved
  features. [^5]
* Operating Systems: Guides for migrating to Linux,
  benchmark results for distributed systems, and case
  studies on Linux development. [^6] [^10] [^51]
* Security: Deep dives into security vulnerabilities,
  migration guides from one framework to another, and
  benchmarking security performance. [^4] [^13] [^22] [^31]

### Notable stories

* A new version of the Rust toolchain has been released
  with various updates. [^1]
* An AI benchmark has been published to compare different
  frameworks' performance. [^2]
* A postmortem analysis of a database failure has been
  shared. [^3]
* A new browser has been launched with improved features. [^5]
* A guide for migrating to Linux is available online. [^10]

### Watch
Keep an eye on the discussions around Rust's tooling
ecosystem, AI benchmarks, and distributed systems design.

### Links

[^1]: [Rust: tooling update #100](<https://example.com/hn/100>)
  · [HN discussion](<https://news.ycombinator.com/item?id=100>)
[^2]: [AI: benchmark #101](<https://example.com/hn/101>)
  · [HN discussion](<https://news.ycombinator.com/item?id=101>)
[^3]: [Databases: postmortem #102](<https://example.com/hn/102>)
  · [HN discussion](<https://news.ycombinator.com/item?id=102>)
...

The wording of the output will differ on every run, because model output is not deterministic. That is precisely why the flow records the inputs and the prompts alongside the output. The non-deterministic part is captured next to everything that produced it.

The links are the one part of the digest the model never writes. The model cites stories by number, the example checks every number against the stories the run fetched, and it appends the ### Links section itself from their titles and URLs. A citation can never point at a story outside the run’s input.

Tip · ObzenFlow's philosophy on AI

The prevailing narrative around AI is that LLMs will take over all of software development and make developers irrelevant. ObzenFlow’s philosophy is different.

LLMs will become a utility, like a faucet or, better yet, like the electrical grid. They will be available everywhere, cheaply, and the interesting question will not be whether to use them but how to use them responsibly in systems that need to explain what they did.

It is nearly inevitable that a large share of software will wrap around AI, and AI will become an implementation detail. That’s our approach here.

Switch to the live Hacker News feed

When you are ready to point the flow at the real API, set source = "live" under [app] in the config. The decoder first fetches v0/topstories.json, then follows it with one v0/item/{id}.json request per story until it reaches max_stories.

# examples/hn_ai_digest_demo/obzenflow.toml
[app]
source = "live"
cargo run -p obzenflow --example hn_ai_digest_demo --features "http-pull ai postgres" -- \
    --config examples/hn_ai_digest_demo/obzenflow.toml

In live mode the header shows mode: live and a real base URL, and stories_fetched reflects whatever the front page held at the time. Everything downstream of the source is identical to the mock run.

Warning · Be responsible with the live Hacker News feed

Larger runs create meaningfully more upstream traffic, because each story is a separate item request. Use the mock server while you iterate on prompts and models, and start small when you switch to live mode.

This example puts an explicit rate limiter on the Hacker News source so live fetches stay polite. The default is 10.0 events per second, and you can lower it with events_per_second under [middleware.stages.hn_stories.rate_limiter]. That limiter complements the live HttpPullSource, which already treats 429 responses as rate limiting, honours Retry-After when it is present, and applies retry with backoff for transient failures.

This is double entry accounting for AI. The input and the output are captured together, as part of the same journaled flow. If the model changes, you can see what changed. If the input changes, you can see how. If you want to replay this run a year from now, the framework allows you to point ObzenFlow to a previous run and replay from a point in time.

Step 2. Understanding live inference flows

The earlier tutorials read local files and accepted pushed requests. What sets this example apart is that it pulls from an external API, runs inference through a local model, and produces a structured digest from potentially hundreds of stories.

The flow shape is still the same FlowApplication and flow! pattern you already know. Where things really change is that now we’re composing a more complex flow from the building blocks ObzenFlow provides, including:

  • async sources that pull from a live RESTful endpoint
  • a transform stage to convert raw input into a type that the rest of the flow can understand
  • the purpose of the example, which is the ai! stage that handles chunking, mapping, and reducing through a model

Finally, we’ll wire that stage into a sink for output.

Most frameworks put the LLM call at the center. ObzenFlow follows a different philosophy: live-inference LLM calls are not any more special than other stages, except that live inference tends to produce non-deterministic, unstructured output. It also involves operational challenges such as token estimation, context window size, batching, and chunking.

ObzenFlow is not an AI framework. It’s a durable execution runtime. It happens to coordinate durable execution of inference, providing resilience for the most common use case. If you need something more exotic, you’ll likely have to drop down into custom handler code, which is completely supported but beyond this tutorial’s scope.

Let’s get started, and we’ll cover all the stages involved.

Reading the stages

Open examples/hn_ai_digest_demo/flow.rs to follow along. Every handler is built with let just above flow!, so the stages block only names them. It has four entries.

let hn_source = HttpPullSource::new(decoder, http_source_config);
let formatter = transforms::map(|story: HnStory| format_story(story));
let digest = ai::map(roles.clone())
    .chunks(max_stories_per_group)
    .reduce(roles)
    .on_eof();
let console_sink = sinks::ConsoleSink::new(format_digest_summary_for_console);
// The two breakers, the rate limiter, and the PostgreSQL sink are bound the same way.

stages: {
    // Ingest: pull stories from the HN API behind a breaker and a rate limiter.
    hn_stories = async_source!(HnStory with {
        source_breaker,
        source_limiter
    } => hn_source);

    // Format each raw story into a stable, display-ready event.
    formatter = transform!(HnStory -> FormattedStory => formatter);

    // The whole map-reduce: chunk the stories, summarize each chunk
    // through the model, then reduce the chunk summaries into one digest.
    digest = ai!(FormattedStory -> HnDigestSummary with ai_breaker => digest);

    // Deliver the digest through the handler the configuration selects.
    digest_summary = sink!(
        HnDigestSummary => handler_set!(console_sink, postgres_sink)
    )?;
}

Each stage maps to the facts from Step 0.

StageWhat it does
hn_storiesPulls stories from the Hacker News Firebase API using HttpPullSource, behind a circuit breaker and a rate limiter so live fetches stay polite. In live mode it walks topstories.json and then fetches each item; in the default mode it reads from the built-in mock server.
formatterTurns each raw HnStory into a typed FormattedStory with a stable text representation. A missing title, author, or URL gets a fallback value, so every story formats.
digestThe ai! stage. It groups the stories into chunks of ten, summarizes each chunk through the model, and reduces the chunk summaries into the final HnDigestSummary. Steps 3 and 4 open it up.
digest_summaryThe sink delivers the final HnDigestSummary through the handler the configuration selects. The console handler prints the evidence trail you saw in Step 1.

The forward path connects them in order, which is the topology you already sketched in Step 0. The chunking, mapping, and reducing all live inside the one digest row rather than in stages you wire yourself.

handler_set! gives the sink two named handlers, and [sinks] handler in the configuration picks one at startup. The default obzenflow.toml picks console_sink, and obzenflow.postgres.toml picks postgres_sink, so moving the digest into PostgreSQL is a configuration change.

Tuning the scaling knobs

Two settings in the config control how much work enters the flow. The example reads [app] as its own HnSettings type when the flow builds, and the run prints the settings it used at startup.

// examples/hn_ai_digest_demo/flow.rs
FlowDefinition::materialize_with(|_runtime_config, settings: HnSettings| {
    let inputs = settings
        .into_inputs()
        .map_err(|error| FlowBuildError::StageResourcesFailed(format!("{error:#}")))?;
    Ok(build_flow_definition(inputs, HnFlowOptions::default()))
})

max_stories (default 60) is passed to the source decoder (hn_story_decoder(base_url, max_stories) in flow.rs), and changes how many items the source pulls into the flow. The source’s rate limiter (default 10.0 per second) changes how aggressively the source is allowed to pull them. You can fetch 10 stories for a quick run or a few hundred for a much larger digest, and you can keep the live source conservative while you do it. The chunking, mapping, and reduce steps absorb the change without any edit to the topology.

# A larger and gentler live run.
[app]
source = "live"
max_stories = 200

[middleware.stages.hn_stories.rate_limiter]
events_per_second = 5.0

A third setting, group_max_stories under [app.digest], sets how many stories each chunk holds. Its default is ten.

Whether the source fetched 10 stories or 200, the map-reduce structure stays the same. You’ll learn how the framework takes care of the hardest details for you in the next section.

Step 3. Inside the digest stage

Now it is time to open up the one stage that makes this more than a wrapper around a single LLM call. You already saw this stage in Step 2, but it is worth putting back in front of you, together with the handler it names, because the rest of this step is a guided tour of its parts.

let digest = ai::map(roles.clone())
    .chunks(max_stories_per_group)
    .reduce(roles)
    .on_eof();

// Inside flow! { stages: { ... } }
digest = ai!(FormattedStory -> HnDigestSummary with ai_breaker => digest);

The input and output types are the same FormattedStory and HnDigestSummary you wired back in Step 0, and with ai_breaker attaches the circuit breaker we get to in Step 4.

So the digest stage really comes down to three parts you provide:

  • Chunking (.chunks(max_stories_per_group)), which closes a chunk every ten stories, or at the natural end of input.
  • The map role (ai::map(roles)), which summarizes one chunk at a time.
  • The reduce role (.reduce(roles)), which folds those chunk summaries into the final digest. .on_eof() says when, which is once, at the natural end of input.

We’ll start with chunking.

Chunking

.chunks(max_stories_per_group) is the stage’s answer to the one hard limit every model imposes.

Every LLM has a context window, a hard limit on how much text it can process in one call, measured in tokens. A token is roughly three quarters of a word in English, but the exact mapping depends on the model’s tokenizer. If you send more tokens than the window allows, the model either rejects the request or, worse, truncates silently. That silent truncation is the exact failure from Step 0, where sending thousands of stories in one shot gives you a confident summary of a sliver of the input and never tells you the rest was dropped.

The digest stage never sends the whole story list in one call. It closes a chunk every ten stories, or at the natural end of input, and the map role summarizes each chunk on its own. A Hacker News story is a title and a URL, so ten of them sit far inside any model’s context window. If you fed the same stage long documents instead of headlines, you would lower group_max_stories until each chunk fit.

With the mock feed’s 55 stories and ten stories to a chunk, you get six chunks, which is the groups: 6 you saw in the header. Each chunk is journaled before the model sees it, so the record shows exactly which stories went into which call.

The map role

The map role runs once per chunk. It builds a prompt from the chunk’s numbered story lines and turns the model’s reply into a partial summary.

impl AiRole<Chunk<FormattedStory>> for HnDigest {
    type Effect = ChatCompletion;
    type Out = HnDigestGroupSummary;

    fn prepare(&self, chunk: &Chunk<FormattedStory>) -> Result<ChatRequestSpec, HandlerError> {
        let lines = chunk
            .numbered()
            .map(|(n, story)| render_story_line(n, story));
        let user = digest_map_prompt(self.interests.as_deref(), chunk.len(), lines);
        Ok(ChatRequestSpec::new(SYSTEM_PROMPT, user.as_str())
            .temperature(0.2)
            .max_tokens(320))
    }

    fn interpret(
        &self,
        _: Chunk<FormattedStory>,
        outcome: AiOutcome<ChatCompletionReply>,
    ) -> Result<HnDigestGroupSummary, HandlerError> {
        let reply = outcome.replied()?;
        Ok(HnDigestGroupSummary {
            output_markdown: strip_accidental_story_echo(reply.text()),
        })
    }
}

A role is plain domain logic around one model call. prepare builds the request from the input. It is synchronous and does no I/O, so replay can recompute it. interpret turns the input and the call’s outcome into the output. The role never calls the model itself. The ai! stage makes the call and journals its outcome.

chunk.numbered() numbers each story by its position in the whole input rather than in its chunk, so story 12 is story 12 in every prompt and in the final digest. digest_map_prompt uses the Prompt builder to assemble a structured, rule-bearing prompt from those lines.

fn digest_map_prompt(
    interests: Option<&str>,
    story_count: usize,
    lines: impl IntoIterator<Item = String>,
) -> UserPrompt {
    let mut p = Prompt::new();
    p.text_if(interests, |i| format!("My interests: {i}"))
        .text("Summarise these Hacker News stories (titles + URLs are provided as input).")
        .rules(/* do not invent facts, cite with footnote markers, ... */)
        .labeled("Output format (follow exactly)", /* Themes + Notable stories */)
        .fenced_lines("Input stories (numbered; do not repeat)", lines);
    p.finish()
}

Each chunk’s reply becomes a small HnDigestGroupSummary that holds only the markdown the model returned.

struct HnDigestGroupSummary {
    output_markdown: String,
}

The reduce role

The reduce role runs once, at the end of input. It receives every story together with the chunk summaries, builds the final prompt, and packages everything into the evidence record.

impl AiRole<Partials<FormattedStory, HnDigestGroupSummary>> for HnDigest {
    type Effect = ChatCompletion;
    type Out = HnDigestSummary;

    fn prepare(
        &self,
        input: &Partials<FormattedStory, HnDigestGroupSummary>,
    ) -> Result<ChatRequestSpec, HandlerError> {
        let user = digest_reduce_prompt(self.interests.as_deref(), input.partials());
        Ok(ChatRequestSpec::new(SYSTEM_PROMPT, user.as_str())
            .temperature(0.2)
            .max_tokens(800))
    }

    fn interpret(
        &self,
        input: Partials<FormattedStory, HnDigestGroupSummary>,
        outcome: AiOutcome<ChatCompletionReply>,
    ) -> Result<HnDigestSummary, HandlerError> {
        let reply = outcome.replied()?;
        let markdown = render_digest_links(reply.text(), input.items())?;
        Ok(self.summary(input, &reply, markdown))
    }
}

One type, HnDigest, plays both roles, one per input type. Partials carries the stories and their chunk summaries in order. render_digest_links checks every [^n] citation in the reply against the stories, rejects a reply that cites a story outside the input, and appends the ### Links section you saw in Step 1.

The role’s summary method assembles the final HnDigestSummary, which is the complete record for the run. The provider, the model, and the exact user prompt come from the reply itself, so the record names the model that actually answered.

struct HnDigestSummary {
    mode: String,
    base_url: String,
    ai_provider: String,
    ai_model: String,
    token_estimator: EstimateSource,
    stories_fetched: usize,
    groups: usize,
    interests: Option<String>,
    chat_prompt_system: SystemPrompt,
    chat_prompt_user: UserPrompt,
    stories: Vec<FormattedStory>,
    group_summaries: Vec<HnDigestGroupSummary>,
    output_markdown: String,
}

This is the part that is easiest to miss when you read the example quickly. ObzenFlow is not just calling an LLM twice. The map role produces bounded, explainable partial results, the framework collects them in order, and the reduce role composes the final answer from those partials rather than from one giant, unstable prompt. The map-reduce shape is what lets the same stage handle 10 stories or 200 without changing.

Step 4. Stage-level middleware

This example attaches protections to two stages, the source and the digest, and leaves the formatter and the sink without any. Middleware is always optional, and a row without with has none, so each operation gets exactly the policy you give it.

A breaker and a rate limiter on the source

The source stage carries two controls, each built with let above the flow and listed in its with clause.

let source_breaker = circuit_breaker()
    .consecutive_failures(HN_SOURCE_BREAKER_FAILURES)
    .open_for(Duration::from_secs(HN_SOURCE_BREAKER_COOLDOWN_SECS));
let source_limiter = rate_limit(source_rate_limit);

// Inside flow! { stages: { ... } }
hn_stories = async_source!(HnStory with {
    source_breaker,
    source_limiter
} => hn_source);

The limiter uses a token bucket internally, so the stage processes up to source_rate_limit events per second with tokens refilling at a steady rate. The default is 10 per second, lowerable with events_per_second under [middleware.stages.hn_stories.rate_limiter]. It works alongside HttpPullSource, which already handles 429 responses, honours Retry-After, and retries transient failures. The explicit limiter means you stay polite before the upstream ever has to push back.

The breaker opens after three consecutive failed fetches and cools down for two seconds before it tries again, which is the source_breaker: 3 failures, 2s cooldown line in the startup banner. It keeps the flow from hammering a news feed that is down. Replay reads the archived stories instead of fetching them, so both controls stand down during replay.

A circuit breaker on the model calls

The digest row carries one circuit breaker.

let ai_breaker = circuit_breaker().consecutive_failures(HN_AI_BREAKER_FAILURES);

// Inside flow! { stages: { ... } }
digest = ai!(FormattedStory -> HnDigestSummary with ai_breaker => digest);

The breaker opens after five consecutive failed model calls, and the row attaches no retry. The breaker prevents the flow from hammering a model that is down or overloaded. A failed model call remains a recorded outcome, and an open breaker refuses later calls until its probe window. This contains the failure at the model boundary without re-executing the role logic around it.

The digest stage makes model calls from two places, the map role once per chunk and the reduce role once at the end. Both share this one breaker, because the authored digest stage is the unit it protects, and configuration addresses the breaker by that same name.

Applying middleware per stage rather than globally is deliberate. A rate limit makes sense on the source but not on the formatter. A circuit breaker makes sense on the model calls but not on the console sink. ObzenFlow lets you put the protection exactly where it belongs.

Tip · Studying the explicit middleware path

The payment_gateway_resilience example attaches a breaker, a retry, and a rate limiter to one declared effect, using the same with clause on an effectful stage.

Step 5. Evidence and replay

The whole reason to wrap a model call in a flow is so the run leaves evidence behind. This example writes journals under target/hn-ai-digest-logs, the same archive shape the earlier tutorials produced.

What gets journaled

The final HnDigestSummary event is the complete record for the run. It carries the provider, the model, the token estimator, the story count, the group count, the exact system prompt, the exact final user prompt, the input stories, every chunk summary, and the final markdown output. The console sink prints much of that directly, which is what you saw in Step 1, but the durable copy is the journal, not the console.

Because every stage writes its own journal, you can inspect each chunk and its summary, and compare outputs across models or across runs. The run archive lives under target/hn-ai-digest-logs/flows/<run-id>/, and its run_manifest.json records the flow structure, the stage metadata, and the journal file mappings, the same manifest pattern from the earlier tutorials.

This is also where the single digest stage shows its true shape. The ai! row you wrote as one line in the stages block lowers into four journaled stages, one each for chunking, mapping, collecting, and reducing, so every step of the map-reduce is independently inspectable on disk.

target/hn-ai-digest-logs/flows/<run-id>/
├── run_manifest.json
├── FiniteSource_hn_stories_stage_*.log
├── Transform_formatter_stage_*.log
├── Stateful_digest__chunk_stage_*.log
├── Transform_digest__map_stage_*.log
├── Stateful_digest__collect_stage_*.log
├── Transform_digest__reduce_stage_*.log
└── Sink_digest_summary_stage_*.log

A matching per-stage error journal and a system.log are written alongside these. The point is that the convenience of one declarative stage costs you nothing in inspectability. The map step that summarized each chunk and the reduce step that produced the digest each left their own durable record.

You do not have to guess the run id. The footer prints it for you when the flow completes, along with the exact replay command.

hn_ai_digest_demo completed. Journal: target/hn-ai-digest-logs/flows/flow_01KSR664VR8TZXPKDJMW7WNQ7X
To verify with a bounded replay, add: --replay-from target/hn-ai-digest-logs/flows/flow_01KSR664VR8TZXPKDJMW7WNQ7X
(Source settings are ignored during replay and catch-up)

The generated digest was delivered by the configured sink handler.
Set source = "live" under [app] to fetch from the real Hacker News API.

Replay

That archive is what replay reads. You point the framework at a previous run and re-execute the flow against the same recorded inputs.

cargo run -p obzenflow --example hn_ai_digest_demo --features "http-pull ai postgres" -- \
    --config examples/hn_ai_digest_demo/obzenflow.toml \
    --replay-from target/hn-ai-digest-logs/flows/<run-id> --verify

The runtime swaps the live source for the archived source journal and reruns the flow against the same stories. Every model call is answered from the journal. Replay uses the model the run recorded, reads no API key, and builds no client, so it needs neither Ollama nor a hosted provider. --verify then compares the replay with the recorded run once it ends.

The interests setting shapes both prompts and is part of every model call’s identity, so a replay with different interests refuses to run rather than quietly answering a different question. If you stored that archive for years, you would still have the evidence needed to answer what data went in, how it was chunked, which prompts were used, which model ran, and what answer came out. The model call is the one non-deterministic part of the flow, and it is the part the journal records most carefully.

Next steps

You have now worked through the three example tutorials. You have seen ObzenFlow handle a finite batch transform, a service that combines batch and live input with joins and projections, and a model-backed digest with chunking, map-reduce, and replay.

If the bank transactions tutorial feels fuzzy after seeing this one, go back to Model Bank Transactions as a Flow and read the projection section again. The two examples reinforce each other, because both build a stable accumulated value and then derive output from it.

From here, ObzenFlow’s architecture gives the deeper rationale behind why ObzenFlow is shaped this way.