Intelligence you can trust.
ObzenFlow handles all of the complexity of streaming inference. It ingests batch and live data and merges it into one flow, chunks large inputs, maps each chunk to a local or hosted model, estimates its tokens, and collects and reduces the answers. Your current solution might do all of that as well but does it help you during an audit when you have to answer what your model emitted 3 months ago?
With most inference pipelines, you have the output and whatever you logged. With ObzenFlow, you have every input, every answer and the model that gave it, durably persisted, archived and replayable.
Keep reading.
Learn how every model call here lands in the record, so years from now you can tell an auditor where that advice came from, why it was generated and why that model in particular gave it.
Run the demo yourself.
ObzenFlow's own Hacker News digest, small enough to run on a laptop with a local model. It fetches the top stories, asks Ollama to summarize them ten at a time, then asks once more to write the digest.
Four ways ObzenFlow helps you keep every answer your model gives.
Challenges with AI inference
Where ObzenFlow fits
Run streaming inference as one stage.
Chunk, map, collect and reduce are built and journalled for you from one line, with a circuit breaker and a retry attached.
Every model call is a first-class effect.
Streaming inference on top of ObzenFlow's effect system means that all calls are durable, auditable and replayable.
Resume a crashed run from its journals.
Streaming inference is expensive, but you can always resume from the exact point of an unexpected termination, saving massive cost to recover expensive workflows.
Replay any historical inference workflow.
ObzenFlow can replay previous streaming inference jobs at any time without the original models to answer root cause or audit questions.
Run streaming inference as one stage.
In ObzenFlow, streaming inference over a model is a composite, written like one stage and run as several.
One line builds the map-reduce
ai::mapsummarizes each group of ten stories andreducewrites the digest once the input ends. The runtime builds and journals the chunk, map, collect and reduce stages underneath, so you never write the batching or the bookkeeping. A circuit breaker and a retry are declared beside it, and every call to the model passes through them.Inference sits in an ordinary workflow
hn_storiespulls the top stories over HTTP behind its own circuit breaker and rate limiter, andformattershapes each one for the model. The model is one stop in a durable workflow that ingests, transforms, joins and merges batch and live data like any other, with the same middleware vocabulary on the source as on the model.One typed stage in the flow
digesttakes aFormattedStoryin and gives aHnDigestSummaryout, like any other stage. Stories come in from the source, the digest goes out to the sinks, and the compiler checks every edge. What the model is asked, and what its answers should look like, is the role, next.
let digest = ai::map(roles.clone())
.chunks(max_stories_per_group)
.reduce(roles)
.on_eof();
let ai_breaker = circuit_breaker()
.consecutive_failures(HN_AI_BREAKER_FAILURES);
let ai_retry = retry();
Ok(flow! {
// …
stages: {
hn_stories = async_source!(HnStory with {
source_breaker,
source_limiter
} => hn_source);
formatter =
transform!(HnStory -> FormattedStory => formatter);
digest = ai!(FormattedStory -> HnDigestSummary with {
ai_breaker,
ai_retry
} => digest);
digest_summary = sink!(HnDigestSummary => handler_set!(
console_sink,
postgres_sink
))?;
},
topology: {
hn_stories |> formatter;
formatter |> digest;
digest |> digest_summary;
}
})Make every model call a first-class effect.
When that day comes for your model, the audit is one question in three parts, what was it asked, what did it answer, and which model answered. In ObzenFlow, a model call is a first-class effect, so all three are in the record.
What was asked and what came back
preparebuilds the request, the prompt, the stories it covers, the temperature and the token budget, and the runtime journals it exactly as it was sent.interpretturns the reply into your own fact, journalled with the model that gave it. Years later, replay reads every answer back from the archive with no model running, so the digest a customer acted on is rebuilt line for line.Change the model, keep the record
The role names no model. Configuration picks one per stage, so a run against a bigger model, or with a different temperature in
prepare, is a new run recorded in full beside the last one. Replaying one archived run through a different model, or a hundred of them, is planned. Coming soon
impl AiRole<Chunk<FormattedStory>> for HnDigest {
type Effect = ChatCompletion;
type Out = HnDigestGroupSummary;
fn prepare(
&self,
chunk: &Chunk<FormattedStory>,
) -> Result<ChatRequestSpec, HandlerError> {
let lines = chunk
.numbered()
.map(|(n, story)| render_story_line(n, story));
let user = digest_map_prompt(
self.interests.as_deref(),
chunk.len(),
lines,
);
Ok(ChatRequestSpec::new(SYSTEM_PROMPT, user.as_str())
.temperature(0.2)
.max_tokens(320))
}
fn interpret(
&self,
_: Chunk<FormattedStory>,
call: AiOutcome<ChatCompletionReply>,
) -> Result<HnDigestGroupSummary, HandlerError> {
let reply = call.replied()?;
Ok(HnDigestGroupSummary {
output_markdown:
strip_accidental_story_echo(reply.text()),
})
}
}Resume a crashed run without asking the model again.
In ObzenFlow, every answer is journaled as the run goes. Kill the run once its stories are in, and the record holds every group the model had already summarized.
Resume it with one command
--resume-frompoints the same binary at the killed run’s archive. Recorded facts catch up first with effects suppressed, so every group the model already summarized comes back from the record and no call goes out for it.Only the unfinished work reaches the model
A call that was in flight when the process died goes out again, the calls still waiting are made, and the run continues live from its high-water mark to the digest. The answers it already had come back exactly as the model gave them.
$ cargo run -p obzenflow --example hn_ai_digest_demo \
--features "http ai postgres" -- \
--config examples/hn_ai_digest_demo/obzenflow.live.toml \
--resume-from "$RUN"
Resuming from <flow_id> at
target/hn-ai-digest-logs/flows/<flow_id>. Recorded facts
catch up first, then the run continues live at the recorded
high-water mark.
### Notable stories
* Police chief unplugs Flock cameras after company ignores
contract cancellation [^4]
* Researchers use JEPAs to investigate the Platonic
Representation Hypothesis [^5]
* Italian farmer discovers unknown seeds in attic, plants
them, and gets a surprise result [^10]
Recorded facts were caught up with effects suppressed, then
the run continued live from the recorded high-water mark.
Replay any past run with no model at all.
In ObzenFlow, every run is archived with every story it read and every answer it got back, with a fingerprint of each request, and its manifest names the model behind each stage.
Replay it with the model stopped
The replay configuration has no model in it, and the run says so before it starts. Every story and every answer come from the archive, so the digest is rebuilt with Ollama stopped and no network, on a laptop that has never seen the model.
The audit answer, verified
The manifest names the provider and the model that answered every call, and
--verifychecks the rebuilt digest against the original run, stage by stage. Which model answered, and what it answered, read back years later and checked to match.
$ cargo run -p obzenflow --example hn_ai_digest_demo \
--features "http ai postgres" -- \
--config examples/hn_ai_digest_demo/obzenflow.replay.toml \
--replay-from "$RESUMED" --verify
Configuration differs from
target/hn-ai-digest-logs/flows/<flow_id>: ai.models.model
[replay] HN AI Digest — Summary
ai_provider: ollama
ai_model: llama3.1:8b
output matched the original run, 0 differences
Build your next AI flow on ObzenFlow.
ObzenFlow is open source. Star the project to follow its progress, or start a tutorial and build something yourself.
Have a question? Ask it in GitHub Discussions.