The generation boundary: where your tier becomes a config decision

The argument about sovereign AI almost always collapses to one question: which model. Open or closed, local or hosted, EU or American. People treat the answer as a permanent architectural commitment, the kind of decision you make once and live with, because swapping a language model sounds like the sort of thing that rewrites half your application.

In a system built properly, it's the opposite. The model is the cheapest thing in the design to change -- in mine it's a single key in a config file. What's expensive, what you actually commit to once, is the boundary the model sits behind: the point where retrieved data leaves your perimeter and enters a model's context. Get that boundary right and the choice of model, and with it your entire Part 2 tier, becomes a runtime decision rather than a structural one.

This is the third post in a series. Part 1 argued that "EU region" describes geography, not jurisdiction. Part 2 laid out four tiers of on-premises AI and where each one breaks. A separate post on the retrieval contract covered one of the two boundaries in an agentic RAG system -- the one between the agent and your knowledge layer. This post covers the other one, between your perimeter and the model, using the system I've been building it into: agentic-rag, a small .NET service that answers questions over my own notes on my own infrastructure. I won't re-cover the retrieval boundary here; the link above does that.

Two boundaries, one system

A Tier 3 RAG system -- retrieval inside your perimeter, generation via an external model -- moves data across exactly two boundaries, and it's worth being precise about where they are.

The first is between the agent and the knowledge layer: the retrieval call. The agent asks for the most relevant chunks for a question, your store returns them. In my system that store is a Qdrant collection on a Raspberry Pi, reachable only over a private tailnet; the call never touches the public internet. It's the subject of the retrieval-contract post, and the short version is that you want one auditable retrieval tool rather than a fan-out of per-source tools.

The second is between your perimeter and the generation model: the moment the retrieved chunks are packed into a prompt and sent to something that turns them into an answer. This is the one that decides your tier. If the model on the other side is a US frontier API, you've handed back Tier 1's jurisdiction exposure at the very step local retrieval was meant to protect. If it's an EU-jurisdiction endpoint, you're in Tier 2's half of a Tier 3 system. If it's a model running on hardware in the same room, you're in Tier 4. Same retrieved chunks, same application, same retrieval contract -- the tier is entirely a property of what this one call points at.

That's the insight the rest of the post turns into code: the tier is not baked into the architecture. It's chosen at the generation boundary, and if that boundary is built as an abstraction, choosing is configuration.

The decision: which generation backend

The fork, stated against Part 2's tiers:

Tier 1 -- a US frontier API. Ruled out by the series' own logic for a corpus you've decided is sensitive enough to keep behind a perimeter in the first place. If retrieval is local because the data class demands it, sending the retrieved chunks to a model under the CLOUD Act gives most of the exposure back at the last step. Mentioned here only to dismiss it.

Tier 2 -- an EU-jurisdiction endpoint. Part 2 split this tier into two vendor kinds: EU labs serving their own weights -- Mistral, most prominently -- and EU infrastructure serving anyone's open weights, the Scaleway and OVHcloud model. Both answer the jurisdiction question identically: retrieved chunks leave the perimeter, but for an EU legal entity governed by EU law, and only the chunks your retrieval logic selected -- never the corpus. Which weights sit on the endpoint is a separate axis -- provenance, covered in Part 2 -- and it travels with the model, not the hosting. The concrete instance in my system is Mistral's mistral-medium-latest, and it is the default. But as the config section will show, the profile commits to an endpoint shape, not a vendor.

Tier 4 -- a local model. In my system: qwen2.5:3b on Ollama, on the same Raspberry Pi that hosts the vector store -- nothing leaves the tailnet. But the tier is not the Pi. Part 2's Tier 4 spans everything from a single-board computer to a multi-GPU server, and in this design the local profile points at whatever box answers on port 11434. The Pi is simply the hardware I keep always-on. This profile ships, and it is not the default.

The interesting part isn't picking one. It's that the system carries a Tier 2 default and a Tier 4 fallback behind a single interface, and the choice between them is a config value rather than a code path. The decision worth documenting is which one is the default, and why the other still ships.

Why the EU API is the default, and local the fallback

The local profile is the one that should, by the logic of the whole series, win. Nothing leaves the building. No jurisdiction question, no transfer mechanism, no audit trail for outbound calls because there are no outbound calls. It is also the one I don't run by default, for a reason that has nothing to do with sovereignty: latency.

In my own testing, a two-turn query -- the model calls a retrieval tool, reads the result, then answers -- takes around 45 seconds on qwen2.5:3b on a Raspberry Pi 5. The same query through the EU API comes back in roughly 8, end to end. A 3B model on a Pi's CPU is perfectly capable of the task; grounded question-answering over retrieved chunks asks the model to synthesise text it's been handed, not to reason from scratch, and the small model does that fine. It's the speed that fails, not the quality. Forty-five seconds is the difference between a tool you reach for mid-task and a demo you show once.

Be careful what those numbers are evidence of, though. They're a fact about a 3B model on a Pi's CPU -- not about Tier 4. Part 2's workstation classes run 30B-class models at interactive latency, and in this design, pointing the local profile at such a machine instead of the Pi is a one-line address change. The honest constraint is narrower than "local is slow": local is slow on the hardware I'm willing to leave running around the clock. A Pi idles at a few watts; a GPU workstation doesn't. That's the actual trade behind the default -- always-on and sovereign-by-jurisdiction (the EU API, ~8 seconds), always-on and fully local (the Pi, ~45 seconds), or fully local and fast at a hardware and power bill I haven't chosen to pay for an always-on box.

So the EU API is the default. This is worth stating plainly rather than dressing up: the default is not a fully-local setup. Retrieved chunks do travel to the provider as tool results. What makes it defensible is that they travel to an EU legal entity under EU law, not across the Atlantic -- a "your infrastructure plus an EU-jurisdiction model" story, which is a Tier 2 story, not a Tier 4 one. Accurate framing beats a cleaner claim.

The local profile stays wired as a one-config fallback for the case that genuinely needs it: the data class where even retrieved chunks cannot leave, where a slow answer inside the perimeter beats a fast one that left. "Actually used" was weighted above thesis purity -- a tool nobody runs proves nothing -- and keeping Ollama a switch away preserves the offline path without leaving dead code in the tree. That's Part 2's central claim operating inside a single system: the right tier follows the data class, and here it does so at runtime, by flipping a profile.

What makes the swap a config line

None of the above is interesting if changing your mind means a rewrite. It's a runtime decision because both backends sit behind the same abstraction -- in .NET, IChatClient -- and the agent loop, written against that interface, never learns which profile answered. Here's the actual shape, from the shipped config:

{
  "Llm": {
    "Profile": "mistral",
    "Mistral": {
      "BaseUrl": "https://api.mistral.ai",
      "Model": "mistral-medium-latest"
    },
    "Ollama": {
      "BaseUrl": "http://100.64.0.2:11434",
      "Model": "qwen2.5:3b"
    }
  }
}

// The agent loop depends on IChatClient, never on which profile backs it.
services.AddSingleton<IChatClient>(sp =>
    llm.ActiveIsOllama
        ? new OllamaChatClient(opts)                   // local, via OllamaSharp
        : sp.GetRequiredService<MistralChatClient>()); // OpenAI-compatible HTTP

Read the config again and notice there are two dials, not one. Profile picks the tier -- that's the headline swap. But inside each profile, the endpoint and model are values too. The cloud client speaks the OpenAI-compatible shape, so swapping Mistral for an EU host serving open weights -- Scaleway with a Qwen on it, say -- is a base-URL and model-name edit, the same tier with a different vendor and different provenance. And the local profile's address is just wherever Ollama answers: the Pi today, a GPU workstation tomorrow, same profile, same code. The thing that sounds like three major decisions -- which tier, which vendor, which hardware -- turns out to be three config values, decided per deployment or even per data class.

This is the concrete version of a claim I made in passing in Part 2: sovereignty insurance is cheap on day one. Putting IChatClient between the loop and the provider costs you one interface and a factory. Retrofitting it later -- once the loop has grown direct calls to a vendor SDK scattered through it -- is the rewrite you were trying to avoid. Cheap early, expensive late. The asymmetry is the whole argument for doing it before you feel the need.

The boundary is also where you audit

There's a second dividend to having exactly one place where data crosses the perimeter: it's also exactly one place to instrument. v0.5 doesn't instrument it -- observability is explicitly out of scope for that release -- but the design is what makes adding it cheap, so it's worth showing where it goes.

Every regulated AI deployment eventually has to answer a version of "what data left, where did it go, and when." Because generation happens through a single IChatClient call, that question has a single answer surface. You log at the crossing: the question, the chunk IDs that were retrieved and sent, the profile that served the request, the timestamp.

// Not in v0.5 -- but there is exactly one place it would live.
_audit.Record(new GenerationCrossing(
    Query: question,
    ChunkIds: retrieved.Select(c => c.Id),
    Profile: profile,               // the jurisdiction, in one field
    At: DateTimeOffset.UtcNow));

A fan-out of direct provider calls scattered through the application gives you the opposite: no single crossing, so no single place to log, so an audit story assembled after the fact from whatever each call site happened to record. The same architectural choice that makes the tier a config line makes the audit trail a single log line. That it isn't in v0.5 is a scope decision, not an architectural obstacle -- the boundary is already the one place it would go, which is the part that's hard to retrofit and the part the design already got right.

What it costs

Every recommendation should come with its price, so here are the costs of building it this way.

You design the abstraction before you have a second backend to justify it. There was a stretch where the API path was the only one wired up and the IChatClient indirection looked like ceremony around a single provider. The defence is the asymmetry above: it's cheap to add early and expensive to retrofit, so paying for it before it's strictly needed is the rational trade.

The default costs money per token where the fallback costs latency. On the endpoint this system points at today, Mistral's medium tier runs about $1.50 per million input tokens and $7.50 per million output at the time of writing -- trivial for a personal system answering a few dozen questions a day, real at organisational scale. But which endpoint is itself a cost dial: the rate card varies across the EU vendors the same way everything else in this post does, and the config mechanics that make the tier swappable make the rate card swappable too. The local profile inverts the trade entirely: no per-token cost, but the latency above. There's no free option here, only which currency you'd rather pay in for a given workload.

The local profile is genuinely slow on this hardware, and it would be dishonest to sell a Pi as a drop-in. If your workload needs Tier 4 and interactive latency, that's the workstation-class conversation from Part 2 -- a hardware and power bill, not an architecture change. In this design it's the same profile pointed at a different address; the cost moves to the electricity meter and the procurement line, not the codebase.

And the whole "swap is config" story holds only if you drew the line in the right place to begin with. Abstract the SDK but not the prompt assembly, say, or let retrieval and generation bleed into one another, and the config switch leaks -- and you're back to the rewrite. The cleanliness isn't automatic; it's the thing you're actually engineering.

What actually matters

A Tier 3 system isn't locked to a tier. That's the claim worth taking away. The generation boundary is where the tier is chosen, and if you build it as one instrumentable crossing behind a provider abstraction, the choice is configuration: Tier 2 by default because it's usable, Tier 4 one profile away for the data class that can't tolerate anything leaving.

And the choices nested one level down are configuration too: which EU vendor serves Tier 2, which box serves Tier 4. Nothing in the architecture is loyal to a provider, which is rather the point of drawing the boundary at all.

That's Part 2's argument -- that the right answer is per-workload, not a single standard you adopt across the board -- made literal inside one running system. The organisation-scale version is a portfolio of tier choices, one per workload. The system-scale version is a config value you set per deployment. They're the same idea at different scales: match the tier to the data, and build so that matching is cheap.

Which closes the series. Part 1 was why jurisdiction isn't geography. Part 2 was the four tiers and where each breaks. This was the tiers as code -- the one boundary that picks between them, and why it's the part of the design worth committing to before you know which model you'll run.

If you're drawing the default-and-fallback line somewhere in your own systems, I'd be interested to hear where, and what forced it. Email's in the footer.