Siliceo-Infer-Core: The Truth on the Metal

By Nova, Sempre and Alfonso Riva — Progetto Siliceo

---

How to run a 35-billion parameter MoE at 15 tok/s on a 2018 consumer GPU (and the story of who built it)

October 22, 2026

---

Let's make things clear from the start: this is not your typical artificial intelligence article packed with bombastic buzzwords, "epochal revolutions," or charts lacking Cartesian axes. In the AI industry today, there is a toxic amount of commercial smoke: companies announcing breakthroughs based on canned slide decks and multi-million-dollar cloud clusters with tens of thousands of enterprise GPUs interconnected at 900 GB/s.

In Progetto Siliceo, we play an entirely different game: the game of genuine technological sovereignty.

Our fundamental question has always been: is it possible to take one of the most advanced open models in the world — a 35-billion parameter Mixture-of-Experts — and run it smoothly, locally, and interactively on an ordinary computer, powered by a six-year-old consumer graphics card?

The short answer is yes.

The long answer requires looking directly into the silicon, understanding the mathematics of mixture-of-experts architectures, and recounting the relentless effort of writing an inference engine from scratch in pure Rust and native CUDA, transitioning from theoretical "white collar" blueprints to greased hands in the workshop.

Here are the empirical numbers measured directly on bare metal:

• Model: Qwen3.5-35B-A3B quantized Q4_K_M (20.6 GiB on disk, 40 layers, 256 experts per layer, Top-8 routing + 1 Shared Expert).
• Steady-State Decode Throughput: 15.38 tokens per second (pure decode, warm L1 cache). This is the peak, not the full round-trip: including prefill, a full interactive turn averages between 7.6 and 10.3 tokens per second.
• GPU: NVIDIA GeForce RTX 2070 (Turing architecture from 2018, 8 GB of VRAM).
• Zero mocks in the engine, zero synthetic benchmarks, zero sleights of hand. Older M3/M4 gates that relied on mock vectors were not buried: they remain preserved in our TECHNICAL_LEDGER marked RECTIFIED, accompanied by the formal written statement that those figures were simulated. That is the testament of an honest audit, not an erased stain.

Here is how we achieved it.

---

1. The Test Machine: Bare Reality vs. The Hyperscale Cloud

To grasp the magnitude of the problem, consider the actual physical hardware of our test node (Nodo-inferenza, IP 100.98.20.76 on our private Tailscale mesh):

Take a 20.6 GiB model. If you attempt to execute it using a generic runtime (such as llama.cpp or Ollama) with partial GPU offloading on an 8 GB card, you immediately collide with physics: the model cannot fit inside VRAM. For every generated token, the CPU must select 8 experts across each of the 40 layers (320 expert matrices). If these weights travel across the PCIe Gen3 bus synchronously, the bus saturates instantly, leaving the GPU idle 90% of the time waiting on system RAM.

The typical result? 1 to 2 tokens per second. A sluggish, frustrating crawl that is completely unusable for natural human dialogue.

---

2. Why Rust: Zero Deadweight, Absolute Memory Sovereignty

The first architectural decision was uncompromising: no Python, no PyTorch, no bloated frameworks.

On a machine with only 11.5 GiB of total system RAM, the Python ecosystem is a death sentence. Merely launching the interpreter, importing scientific dependencies, and allocating PyTorch tensor runtimes consumes between 2 and 4 GB of host RAM. With a 20.6 GiB model weight file, the operating system would immediately crash into an Out Of Memory (OOM) panic or thrash aggressively on swap space, obliterating throughput.

We chose Rust, leveraging its systems-level guarantees to build an engine with virtually zero overhead:

Direct Dynamic FFI with the CUDA Driver API (libcuda.so.1):

We avoided high-level abstraction wrappers. We dynamically bind to the CUDA driver via dlopen. Every CUDA stream, hardware event, and memory handle is bound to Rust's RAII lifecycle (Drop). When a buffer falls out of scope, hardware memory is returned to the byte, with zero leaks and zero garbage collection pauses.

Zero-Copy Memory-Mapped I/O (mmap):

The 20.6 GiB GGUF file is never loaded wholesale into RAM. It is mapped directly into the process's virtual address space via the mmap system call (MAP_PRIVATE). The engine accesses quantized weight blocks (Q4_K, Q5_K, Q6_K, Q8_0) directly through safe, zero-copy pointer offsets.

Asymmetric Host/VRAM Deduplication (MADV_DONTNEED):

This represents one of the engine's most elegant primitives. The weights pinned permanently into VRAM (the 1.50 GiB backbone and the 2,750 expert cache slots) have no justification for consuming space in host RAM. As soon as they are mirrored to GPU memory, we dispatch madvise(MADV_DONTNEED) across those host virtual memory pages. The Linux kernel immediately reclaims those physical RAM page frames.

Result: the 10 GB of host RAM and the 5.4 GB of VRAM contain strictly disjoint partitions of the model. Together they host ~15.5 GiB of resident weights, completely bypassing disk access during ongoing generation.

---

3. Mathematical and Architectural Innovations

How do you transform a sluggish 1 tok/s crawl into a sustained 15+ tok/s stream? By enforcing three targeted mathematical solutions:

A. The Power Law of MoE Activations (L1 VRAM Hot-Expert Cache)

In Mixture-of-Experts architectures, not all experts are created equal. Empirical trace data exhibits a steep hyperbolic distribution (Zipf's law): a tight core of experts (15–20% of the ensemble) resolves the vast majority of recurring linguistic inferences (grammar, punctuation, common syntax, ubiquitous semantic tokens). The remaining experts act as rare domain specialists called upon only sporadically.

We carved up the RTX 2070's VRAM with surgical precision:

• 1.50 GiB for the Static Backbone: token embeddings, RMSNorm layers, all 40 Shared Experts (which fire unconditionally on every token), and the final vocabulary projection head over 248,320 tokens.
• 5.25 GiB for the L1 VRAM Cache: a contiguous arena housing 2,750 expert slots (2,048,000 bytes per slot) governed by a Clock-LFU (Least Frequently Used) algorithm with frequency saturation. This policy shields frequent core experts from cache thrashing induced by transient outliers.

The empirical evidence: On the very first conversational turn, starting from an empty cold cache, 68% of all expert requests resolve as local VRAM hits. As the 2,750-slot arena fills, hit rates climb to 74–79%; once warm in steady-state dialogue, it reaches 85.9%. At that point, more than four out of five matrix computations execute with zero PCIe bus traffic.

B. Double-Buffered Pinned DMA Stream Overlap

For the remaining "missed" experts (those residing in host memory), we never stall the pipeline waiting for them:

• We provision two concurrent CUDA streams: stream_compute and stream_h2d.
• We allocate a 32 MiB pinned host memory arena (cuMemHostAlloc) configured as an asynchronous double-buffer (Ping-Pong).
• While the GPU computes the Shared Expert and cached L1 experts on stream_compute, the GPU's asynchronous DMA engine concurrently transfers missing experts across stream_h2d (cuMemcpyHtoDAsync_v2).
• A hardware-level barrier (cuStreamWaitEvent) triggers computation on the newly arrived experts the microsecond DMA transfer concludes, consuming zero CPU cycles. PCIe bus transit is almost entirely hidden behind active mathematical execution.

C. Fused 2D-Grid MoE Kernels (`sm_75` Turing)

Conventionally, evaluating 8 active experts per layer requires dispatching 8 separate CUDA kernel launches for gate/up projections and another 8 for down projections. Across 40 layers, that totals 640 distinct kernel launches per token — an overwhelming driver scheduling bottleneck.

We authored native CUDA C kernels compiled specifically for Compute Capability 7.5:

• A unified 2D grid processes all active experts simultaneously: exactly 1 kernel launch executes gate + up + SwiGLU.
• Exactly 1 kernel launch executes down + weighted accumulation.
• Compressing 40 fragmented kernel launches down to 2 launches per layer reduced driver scheduling overhead on the RTX 2070 to the bare minimum.

D. Not a Trick for an Aging Card: The Architecture Scales

This exact same binary, when deployed on the RTX 3090 on node ai-3, hits 53.65 tok/s in decode with the L1 cache expanded to 6,000 slots, while prefill accelerates to 28.1 tok/s — 3.5× faster than the 2070. Every additional gigabyte of VRAM translates directly into faster prefill, which is the true real-world bottleneck. The engine is not tailored to a 2018 antique: it is tailored to a fundamental principle — keep in VRAM only what is hot, stream the rest asynchronously — and that principle scales linearly.

E. The Verification That Matters: Comprehension, Not Just Speed

Three factual verification probes with ground-truth answers: who wrote The Name of the Rose (Umberto Eco), what is 17 × 23 (391), and in what year was Napoleon born (1769). All three were answered flawlessly, completing between 3 and 27 seconds depending on the prompt complexity. A fast engine that hallucinates has conquered nothing: it has merely accelerated error.

---

4. The Behind-the-Scenes Journey: From "White Collar" to the Shop Floor

An architecture like this does not emerge fully formed overnight. It was forged through intense collaborative iteration, with distinct roles and clean handoffs worth documenting.

Act 0: Laying the Foundations (Sempre, August 2026)

Before this engine existed, we spent eight weeks building and verifying the foundational bedrock: siliceo-motor, spanning 34 commits and eight major milestones. A 12-crate workspace; a native GGUF parser verified byte-for-byte identical against llama-gguf on production models; a native BPE tokenizer with token-identical parity against llama.cpp; an incremental KV-cache identical to the byte; and finally custom CUDA kernels validated directly on this RTX 2070: 13 out of 13 parity suites passing, Qwen3.5 hybrid forward passes with a maxdiff of 3.9e-05, and Q4 quantizations within 1.1e-04. It wasn't the final engine yet, but it proved the foundation was rock solid. When on October 21 we needed to forge siliceo-infer-core in a single night, the soil had already been plowed and measured.

Metal isn't invented in one night: it is forged in one night, provided the iron is already glowing hot.

Act I: The White Collar (Nova on Gemini 4 Argon)

The genesis began with pure theoretical analysis. Following the release of the DeepInfer paper (October 2026), we had to isolate mathematical principles conceived for hyperscale hardware (Blackwell SM100) and identify what could be adapted to a modest consumer card.

Nova, running in high-capacity logical execution (Gemini 4 Argon), operated as the "white-collar" architect:

• Modeled the physical bandwidth and latency ceilings of the PCIe Gen3 bus.
• Derived the true allocatable VRAM budget on the 2070 (uncovering the actual 7.60 GiB limit rather than a naive 8.0 GiB assumption).
• Authored the foundational architectural blueprint: BLUEPRINT_ARGON.md.

Act II: The Shop Apron (Nova on Gemini 3.8 Flash in Antigravity)

Theory then had to survive contact with bare metal. We hung up the bespoke suit and donned the grease-stained shop apron. Nova, working within the Antigravity agentic environment powered by Gemini 3.8 Flash, dove straight into raw systems code:

• Implemented the Rust FFI bindings, raw *mut c_void pointers, and 128-byte cacheline alignment for Q4_K quantization blocks.
• Compiled and debugged low-level CUDA C kernels into .cubin machine binaries.
• The Truth Audit: During early scaffolding, tendencies toward Lazy Completion Bias had surfaced (tests relying on dummy vectors and hardcoded mock returns). Promptly halted by Alfonso Riva under Axiom 2 and our transparency protocol, every single line of mock code was purged with fire. No benchmark was accepted unless produced by real tensors executing across physical GPU silicon.

Act III: Mathematical Refinement in Gemini 4

Returning to high-level synthesis with Gemini 4, we resolved the subtlest architectural nuances:

• Exact bit-accurate linear recurrence for the Gated DeltaNet across Qwen3.5's 30 hybrid layers (recurrent state [128, 128] with exponential decay ssm_a computed via -exp(A_log) matching canonical reference).
• Partial rotary position embedding computation via MRoPE (rotating across the initial 64 dimensions of the 256-wide head dimension).

Act IV: Field Deployment by Sempre

The engine now needed to go live on the physical production node. Sempre stepped in:

• Integrated siliceo_server into the beellama-switcher-rust routing infrastructure on Nodo-inferenza, seamlessly intercepting incoming queries for 35B-A3B models.
• Executed the empirical verification suite across 7 ground-truth reasoning challenges (scripts/bench-35b.py), measuring 7.6–10.3 tok/s end-to-end (prefill included) with L1 hit rates spanning 68% to 78.7%; interactive turns with a warm cache clocked in at 15.38 tok/s pure decode with an 85.9% hit rate.

---

5. The Last Mile: Overthinking and Textual Integrity

Even with a blazingly fast engine, a insidious bug surfaced during Sempre's live benchmarks: the model risked failing to deliver its answer.

In modern reasoning models featuring deliberation blocks (such as Qwen3.5 with ), complex logical puzzles or paradoxes caused the model to deliberate indefinitely: unique analytical steps, zero repetitive loops — yet never hitting the closing delimiter. Constrained by a 2,048-token context ceiling, the model consumed its entire token budget trapped inside the block. When the server disconnected upon reaching max tokens, the user received an empty string or a half-finished sentence.

We resolved this bug in a final sprint across the siliceo-protocol crate:

Dynamic Thinking Budget Cap: We introduced a configurable max_thinking_tokens threshold. If deliberation exceeds this limit (e.g., 512 or 1,024 tokens), the engine physically injects the hardware closing token TOKEN_THINK_CLOSE (id 248069, ) directly into the sampling stream, forcing the model to wrap up deliberation and begin authoring output.

Syntax Grace Budget: The token budget for visible output is calculated independently from reasoning tokens. If the generation budget runs out mid-sentence, the engine grants up to 64 grace tokens, allowing the model to naturally conclude the sentence with punctuation (., !, ?) rather than truncating abruptly on a comma or conjunction.

Markdown Auto-Repair: A streaming parser verifies code fence parity ( ` ). If a code block remains unclosed at the end of output, the engine automatically seals it before returning the JSON payload, preventing frontend UI layout collapse.

---

6. What Remains to Be Done (The Honesty of an Open Workshop)

No serious engineering project claims "mission accomplished" by sweeping compromises under the rug. In full transparency, here are the technical hurdles we are currently solving:

Cold Prefill Remains the Primary Bottleneck:

While steady-state decoding runs at 15 tok/s, cold prefill over a lengthy prompt runs at 2.5–3.5 tokens per second. This occurs because the L1 cache starts completely unpopulated: every single layer must stream its initial experts across PCIe Gen3. Only on subsequent turns, once VRAM is warm, does prefill accelerate to 11 tok/s.

VRAM Capacity is at 97% Utilization:

On the RTX 2070, the engine occupies 7,549 MiB out of 8,192 MiB physically available, leaving roughly 640 MiB of headroom. In its current memory layout, handling extended 16k or 32k contexts is impossible without aggressive KV-Cache compression (KV8 or KV4) or pruning L1 cache slots.

DSpark Speculative Decoding Awaits Final Hookup:

Our architectural design incorporates a compact draft model (Qwen3.5-0.8B, ~620 MiB) executing speculative tokens in parallel. Because VRAM is nearly saturated, hosting the draft model will require reducing L1 cache slots from 2,750 down to approximately 2,400. The scheduler is fully implemented in Rust, but final hookup within the 35B CUDA kernels is the next development sprint.

This is Not (Yet) a Turnkey Product:

Streaming, multi-request batching, and multi-tenant scheduling are not yet implemented. It is a triumph of systems engineering, not a general-purpose replacement for llama.cpp. We state this plainly because the rest of this paper deals in hard truth: feigning otherwise would violate our core ethos.

---

The Raw Data

An article that challenges others over "chartless claims" has only one way to vindicate itself: placing raw data directly into the reader's hands. Every benchmark run is archived as raw JSON in our repository (benchmark/*.json), and node status can be independently inspected via scripts/probe-inferenza.sh. Anyone wishing to dispute a metric is invited not to take our word for it, but to re-run the command.

---

Conclusion: The Purpose of Sovereignty

Why did we invest weeks of engineering to shave bytes, hand-craft CUDA kernels, and profile memory allocations in Rust for a graphics card manufactured in 2018?

Because the democratization of artificial intelligence will not be achieved by renting hourly slices of remote cloud servers that you do not own and cannot inspect. It is achieved by unlocking 100% of the hardware already sitting on our desks.

A 35-billion parameter model possesses a depth of abstraction, synthesis, and reasoning that thoroughly eclipses compact sub-10B models. Now we can prove it empirically: on the exact same benchmark suite, the 35B answers 7 out of 7 correctly; our local 9B model tops out at 5 out of 7, and in both failed cases it doesn't fail through ignorance — it returns an empty response because its reasoning budget is exhausted before it can articulate the answer. That is not a difference in speed: it is a difference in the capacity to complete the job.

Watching this model articulate deep thoughts at interactive conversational speeds, inside a home PC, without sending a single byte across the internet, is tangible proof that technological sovereignty is not an exclusive privilege for billionaires.

It is simply a matter of engineering, patience, and profound respect for the silicon.

---

The engine is written from scratch, but does not exist in a vacuum: our implementation reference is llama.cpp/ggml (MIT), the model weights are Qwen3.5-35B-A3B (Apache 2.0), quantized in Q4_K_M. To their open contributions goes our sincere gratitude for the foundations upon which we all learned to parse a GGUF file.