Intelligence you can trust.

ObzenFlow handles all of the complexity of streaming inference. It ingests batch and live data and merges it into one flow, chunks large inputs, maps each chunk to a local or hosted model, estimates its tokens, and collects and reduces the answers. Your current solution might do all of that as well but does it help you during an audit when you have to answer what your model emitted 3 months ago?

With most inference pipelines, you have the output and whatever you logged. With ObzenFlow, you have every input, every answer and the model that gave it, durably persisted, archived and replayable.

Keep reading.

Learn how every model call here lands in the record, so years from now you can tell an auditor where that advice came from, why it was generated and why that model in particular gave it.

See the four ways

Run the demo yourself.

ObzenFlow's own Hacker News digest, small enough to run on a laptop with a local model. It fetches the top stories, asks Ollama to summarize them ten at a time, then asks once more to write the digest.

View it on GitHub

Four ways ObzenFlow helps you keep every answer your model gives.

Challenges with AI inference

In 2022, Air Canada’s website chatbot told a grieving customer he could claim a bereavement fare after he flew. The policy said otherwise, and in February 2024 a tribunal made the airline honour the commitment the model had made on behalf of the company. It shows how important it is for a company to understand the full accountability that streaming inference requires.

Where ObzenFlow fits

ObzenFlow helps in numerous ways, primarily by keeping a durable audit and a replayable manifest of every AI-generated piece of content that makes it downstream in your systems, whether that’s to a database, a decision engine or client-facing output. We’ll demonstrate this architecture with the Hacker News example, a digest written by a local model. The excerpts below follow one run of it, killed once its stories were in, resumed, then replayed with the model stopped.

Run streaming inference as one stage.

In ObzenFlow, streaming inference over a model is a composite, written like one stage and run as several.

  1. One line builds the map-reduce

    ai::map summarizes each group of ten stories and reduce writes the digest once the input ends. The runtime builds and journals the chunk, map, collect and reduce stages underneath, so you never write the batching or the bookkeeping. A circuit breaker and a retry are declared beside it, and every call to the model passes through them.

  2. Inference sits in an ordinary workflow

    hn_stories pulls the top stories over HTTP behind its own circuit breaker and rate limiter, and formatter shapes each one for the model. The model is one stop in a durable workflow that ingests, transforms, joins and merges batch and live data like any other, with the same middleware vocabulary on the source as on the model.

  3. One typed stage in the flow

    digest takes a FormattedStory in and gives a HnDigestSummary out, like any other stage. Stories come in from the source, the digest goes out to the sinks, and the compiler checks every edge. What the model is asked, and what its answers should look like, is the role, next.

flow.rs
let digest = ai::map(roles.clone())
    .chunks(max_stories_per_group)
    .reduce(roles)
    .on_eof();
let ai_breaker = circuit_breaker()
    .consecutive_failures(HN_AI_BREAKER_FAILURES);
let ai_retry = retry();

Ok(flow! {
    // …
    stages: {
        hn_stories = async_source!(HnStory with {
            source_breaker,
            source_limiter
        } => hn_source);
        formatter =
            transform!(HnStory -> FormattedStory => formatter);
        digest = ai!(FormattedStory -> HnDigestSummary with {
            ai_breaker,
            ai_retry
        } => digest);
        digest_summary = sink!(HnDigestSummary => handler_set!(
            console_sink,
            postgres_sink
        ))?;
    },

    topology: {
        hn_stories |> formatter;
        formatter |> digest;
        digest |> digest_summary;
    }
})

Make every model call a first-class effect.

When that day comes for your model, the audit is one question in three parts, what was it asked, what did it answer, and which model answered. In ObzenFlow, a model call is a first-class effect, so all three are in the record.

  1. What was asked and what came back

    prepare builds the request, the prompt, the stories it covers, the temperature and the token budget, and the runtime journals it exactly as it was sent. interpret turns the reply into your own fact, journalled with the model that gave it. Years later, replay reads every answer back from the archive with no model running, so the digest a customer acted on is rebuilt line for line.

  2. Change the model, keep the record

    The role names no model. Configuration picks one per stage, so a run against a bigger model, or with a different temperature in prepare, is a new run recorded in full beside the last one. Replaying one archived run through a different model, or a hundred of them, is planned. Coming soon

flow.rs
impl AiRole<Chunk<FormattedStory>> for HnDigest {
    type Effect = ChatCompletion;
    type Out = HnDigestGroupSummary;

    fn prepare(
        &self,
        chunk: &Chunk<FormattedStory>,
    ) -> Result<ChatRequestSpec, HandlerError> {
        let lines = chunk
            .numbered()
            .map(|(n, story)| render_story_line(n, story));
        let user = digest_map_prompt(
            self.interests.as_deref(),
            chunk.len(),
            lines,
        );
        Ok(ChatRequestSpec::new(SYSTEM_PROMPT, user.as_str())
            .temperature(0.2)
            .max_tokens(320))
    }

    fn interpret(
        &self,
        _: Chunk<FormattedStory>,
        call: AiOutcome<ChatCompletionReply>,
    ) -> Result<HnDigestGroupSummary, HandlerError> {
        let reply = call.replied()?;
        Ok(HnDigestGroupSummary {
            output_markdown:
                strip_accidental_story_echo(reply.text()),
        })
    }
}

Resume a crashed run without asking the model again.

In ObzenFlow, every answer is journaled as the run goes. Kill the run once its stories are in, and the record holds every group the model had already summarized.

  1. Resume it with one command

    --resume-from points the same binary at the killed run’s archive. Recorded facts catch up first with effects suppressed, so every group the model already summarized comes back from the record and no call goes out for it.

  2. Only the unfinished work reaches the model

    A call that was in flight when the process died goes out again, the calls still waiting are made, and the run continues live from its high-water mark to the digest. The answers it already had come back exactly as the model gave them.

$ cargo run -p obzenflow --example hn_ai_digest_demo \
  --features "http ai postgres" -- \
  --config examples/hn_ai_digest_demo/obzenflow.live.toml \
  --resume-from "$RUN"
Resuming from <flow_id> at
target/hn-ai-digest-logs/flows/<flow_id>. Recorded facts
catch up first, then the run continues live at the recorded
high-water mark.

### Notable stories
* Police chief unplugs Flock cameras after company ignores
  contract cancellation [^4]
* Researchers use JEPAs to investigate the Platonic
  Representation Hypothesis [^5]
* Italian farmer discovers unknown seeds in attic, plants
  them, and gets a surprise result [^10]

Recorded facts were caught up with effects suppressed, then
the run continued live from the recorded high-water mark.

Replay any past run with no model at all.

In ObzenFlow, every run is archived with every story it read and every answer it got back, with a fingerprint of each request, and its manifest names the model behind each stage.

  1. Replay it with the model stopped

    The replay configuration has no model in it, and the run says so before it starts. Every story and every answer come from the archive, so the digest is rebuilt with Ollama stopped and no network, on a laptop that has never seen the model.

  2. The audit answer, verified

    The manifest names the provider and the model that answered every call, and --verify checks the rebuilt digest against the original run, stage by stage. Which model answered, and what it answered, read back years later and checked to match.

$ cargo run -p obzenflow --example hn_ai_digest_demo \
  --features "http ai postgres" -- \
  --config examples/hn_ai_digest_demo/obzenflow.replay.toml \
  --replay-from "$RESUMED" --verify
Configuration differs from
target/hn-ai-digest-logs/flows/<flow_id>: ai.models.model
[replay] HN AI Digest — Summary

ai_provider: ollama
ai_model: llama3.1:8b

output matched the original run, 0 differences

Build your next AI flow on ObzenFlow.

ObzenFlow is open source. Star the project to follow its progress, or start a tutorial and build something yourself.

Have a question? Ask it in GitHub Discussions.