When Compliance Makes the Architectural Decision for You

Building a RAG system on a large-scale compliance platform meant one requirement was non-negotiable from the start: confidential information and PII could not leave the network. That single constraint made the obvious architectural choice unavailable.

OpenAI's embedding API was off the table entirely. Every document, code file, and configuration item passed through the embedding pipeline would have crossed a network boundary into a third-party service. For a platform handling sensitive compliance data across multiple jurisdictions, that wasn't a risk to evaluate — it was a line not to cross.

Azure OpenAI was a more viable path. Data stays within your Azure tenant, which satisfies many compliance requirements. But it still represented a cloud dependency — infrastructure outside direct control, subject to service availability, pricing changes, and the evolving compliance guidance that shapes what's acceptable in regulated environments.

Ollama running locally eliminated the cloud dependency entirely. All embedding happened on-premises, on hardware we controlled, with no data leaving the network. It was the option that most directly satisfied the requirement.

It also came with three costs that deserved honest accounting.

Embedding quality. Local models aren't equivalent to OpenAI's best offerings. The gap is measurable — it shows up in retrieval precision, in the relevance of what surfaces when an engineer asks a complex question about system behavior. For a system whose entire value is the quality of what it retrieves, that gap matters. It's not a dealbreaker, but it's a real cost that shapes what the system can deliver.

Inference latency. Local inference is slower than a well-provisioned API call, particularly under indexing load. When the system is processing a large batch — a full Confluence export, a significant codebase update — the throughput is lower than it would be with a hosted provider. This affects how quickly the index stays current after a major change.

Operational complexity. Running Ollama in production adds infrastructure overhead that a hosted API doesn't. Model management, hardware provisioning, the failure modes that come with running ML inference locally — none of these exist when you're making an HTTP call to someone else's service. They're manageable, but they're real, and they represent ongoing maintenance cost that a hosted solution would have absorbed.

These weren't theoretical tradeoffs evaluated on a whiteboard. They're the costs that showed up in production and had to be managed. Honest accounting upfront is what made the decision defensible — and what drove the design that followed.