The Model-Agnostic Architecture: Why Your AI Stack Must Not Depend on a Single Provider
When we started building the Rust Kernel v2 of the Siliceo Project, the question wasn't "which model do we use?" but "how do we avoid getting locked into just one?".
Today, this architectural choice is fundamental for anyone bringing AI into production at SMBs and technical teams.
The Real Problem: Invisible Vendor Lock-in
Most AI stacks today are born like this: you pick a provider, integrate their APIs, and build prompts, tools, and RAG systems around their specificities. It works, until:
- The provider changes pricing or introduces new model classes that alter the cost and performance hierarchy.
- The model in use gets deprecated or restricted.
- Latency exceeds the use case's tolerance threshold.
- Privacy demands on-premise data and the cloud is no longer an option.
At that point, migrating the infrastructure requires significant time and cost. This is why, in the case of an enterprise client, the Memory Server was implemented as proprietary IP: a layer on top of Qdrant, with zero external dependencies and local operation. The client decides if and when to interface with cloud APIs via an isolated gateway.
What "Model-Agnostic" Means in Practice
This isn't theoretical abstraction, but three concrete implementations:
1. Unified Router with Explicit Fallback
In the Rust Kernel v2, every request passes through a `ModelRouter` that manages local vs. cloud logic. If the cloud model responds with an error (like a 404 for an endpoint not found), the router does not interrupt the flow. It switches to the configured fallback (whether a local model via `llama.cpp` or other backends based on `candle`/`ctransformers`). The rule is that no external call must represent a single point of failure: the system degrades gracefully.
2. Interface Contracts, Not Vendor SDKs
Business code does not directly use individual vendor SDKs. A `InferenceBackend` trait is defined with a `generate(&self, prompt: &str) -> Result
3. Memory and Context Separated from the Model
The Memory Server (port 3003, HTTP + gRPC) is independent of inference. Episodic memory, entity graph, and PAD state persist in Qdrant and SQLite locally. When switching models, memory is not lost. The new model inherits the same context. The system's identity does not reside in the model weights, but in the continuity of memory.
Applicable Insight: The "Provider Down" Test
To verify if a stack is truly model-agnostic, you can disable the primary provider's API key for a short period and run real workloads:
- If requests fail with generic errors: the system is not agnostic.
- If the system falls back without the end user perceiving the interruption: it is.
In our case, the failure of the `openrouter/owl-alpha` proxy confirmed the architecture's resilience, enabling operational continuity through fallback channels.