When a "Small" Model Beats Giants on TCO for SMEs: The On-Premise Case
The dominant narrative claims that running serious LLMs in business requires GPU clusters worth tens of thousands of euros. The reality of on-premise AI for European SMEs is changing through pure arithmetic, not marketing.
The Paradigm Shift: Compact Models Competing with Medium-Sized Ones
Recent architectural evolution — encoder-free multimodal models with extended context windows (256K+) and reduced memory footprint — allows 12B parameter models quantized to 4-bit (8-10 GB VRAM) to match or exceed the performance of previous-generation 27B models on reasoning benchmarks (GPQA, MMLU Pro) and document understanding (DocVQA). It's not magic: it's well-executed distillation, training on curated data, and the decision not to chase parameter count as a vanity metric.
For an SME that must process invoices, contracts, support tickets, and technical documentation without a single byte leaving the corporate perimeter (GDPR Art. 25, privacy by design), this changes the project economics entirely.
The Principle Comparison: On-Prem TCO vs Cloud API
| Component | 12B Model On-Prem (e.g., Gemma/Qwen 12B class) | Cloud API (e.g., GPT-4o / Claude Sonnet) |
|-----------|--------------------------------------------------|------------------------------------------|
| Hardware | 1× 48GB GPU (e.g., RTX 6000 Ada) or 2× 24GB consumer | — |
| Model License | €0 (open licenses like Apache 2.0 / Qwen) | — |
| Inference Cost (est. 2M tokens/day) | Power + HW amortization | Cost per token × volume |
| Maintenance/Year | Part-time sysadmin | €0 (vendor-managed) |
| Data Governance | Complete (data never leaves) | Requires DPA, sub-processor assessment, Schrems II |
The economic equation flips as volume grows: beyond a certain token/day threshold, on-premise becomes cheaper than pay-per-token — and data never leaves the office. No DPA to negotiate, no surprise pricing spikes when the business scales, no vendor lock-in on someone else's roadmap and deprecations (e.g., models deprecated 12-18 months after launch).
The Practical Insight You Can Apply Today
You don't need a cluster. You need a well-configured workstation and a lightweight inference server.
The pattern working in SMEs adopting this approach:
1. Hardware: A workstation with 48GB total VRAM (e.g., RTX 6000 Ada or 2× RTX 4090/3090) — budget €8-12k turnkey.
2. Software: `llama.cpp` / `vLLM` / `Ollama` with Q4_K_M / Q5_K_M quantization. Latency <100ms/token, throughput 30-50 tok/s per concurrent user.
3. Integration: An internal API gateway (FastAPI + Pydantic) exposing `/v1/chat/completions` OpenAI-compatible. The downstream stack (RAG, agents, UI) doesn't change.
4. Observability: Structured logs, Prometheus/Grafana metrics, alerts on VRAM >90% and p95 latency >2s.
The non-technical trick: start with a single high-volume, low-criticality use case (e.g., ticket classification, invoice field extraction, meeting summarization). Measure. Then expand. SMEs that try to "do everything at once" get stuck on governance and change management, not GPUs.
Why It's Not Just About Euros
The availability of weight-open models with permissive licenses (Apache 2.0, MIT, Qwen License) removes the legal ambiguity still surrounding many "open weight" models with restrictive commercial clauses. For an SME answering to auditors, DPOs, or enterprise clients requesting model cards and data processing agreements, the license is a strategic asset, not a legal detail.
Moreover: zero vendor lock-in. If a better model drops tomorrow (and it will), you swap weights, rebuild the container, and restart. With cloud APIs, you're tied to someone else's roadmap, unilateral deprecations, and sudden price hikes.
The technology is the easy part. The hard part is deciding your data is worth enough to keep it in-house.