SonicBoom

A native tensor runtime. No Python. One .so.

SonicBoom is an AI framework written from the ground up in C++ and CUDA, for workloads where performance matters. Everything is exposed through a stable C ABI, and GNU Guile is its first-class language. You load libsonicboom.so from whatever language you use. There is no Python interpreter, no pip, and no wheel tree on the inference path.

The first release targets local LLM inference on consumer GPUs. On a laptop RTX 3050 it decodes 12–18% faster than llama.cpp, using the same quantized kernels — 1.18× on Gemma 4 E4B-it Q3_K_M, 1.12× on Qwen2.5-0.5B-Instruct Q4_K_M (numbers, same machine, medians of 3). The kernels are the same, so the lead comes from the layers above them: a graph planner, and behind it an MLIR compiler tier. Those layers are what llama.cpp's architecture does not have, and they are where SonicBoom keeps improving.

Status: v0.0.2. Two model families are verified end to end — Gemma 4 E4B-it (Q3_K_M) and Qwen2.5-0.5B-Instruct (Q4_K_M) — on CUDA sm_86, both checked against llama.cpp. Read Scope of v0.0.2 before trying it.

---

Why another runtime?

For people who need to know what they are running. The core is 17.7k lines of C++/CUDA. It is GPL-3.0-or-later, and the boundary is a plain C ABI that you can audit, pin and link against. It has no Python dependency chain to vet, and nothing is downloaded at run time. Python is not needed to build it either: PyTorch's Python code generator, torchgen, is replaced by SonicCross, written in Guile.

Fast because of the runtime, not only the kernels. Each decode step runs as a single CUDA Graph replay. The KV cache stays on the device. Planner passes fuse norm, activation and quantization. Prefix-cache reuse is exact: a continuation is token-identical to a full recompute.

More than an LLM runner. SonicBoom is a general tensor framework: a better PyTorch, with no Python in it. You write models in Scheme, as eagerly as in PyTorch, and they are traced and compiled into graphs. Production models are defined in the C++ core and served from any language. LLM inference is the first benchmark, not the limit of the project.

---

Numbers

All numbers below are medians of 3 process runs, measured on the machine in Benchmark conditions, with both runtimes on the same GGUF file in the same session — SonicBoom through tools/bench.sh, llama.cpp through llama-bench -ngl 99 (commit 43fe9c64281). Every claim in this section is bounded by that: one model, one GPU, one quant file. tools/bench.sh reprints the whole table on any machine, and tools/sb-report.sh prints the hardware and the build that produced it.

Gemma 4 E4B-it Q3_K_M (RTX 3050 8 GB, sm_86)

metric	SonicBoom	llama.cpp	ratio
Decode, 64-step steady state	52.7 tok/s	44.8 tok/s	1.18×
Decode at KV depth 512	51.7 tok/s	44.2 tok/s	1.17×
Prefill, 512-token prompt (marginal)	1673 tok/s	1997 tok/s	0.84×
Prefix reuse, 512-token prefix + 64 new	212.2 ms to first token	—	—

Qwen2.5-0.5B-Instruct Q4_K_M (same machine, same session)

metric	SonicBoom	llama.cpp	ratio
Decode, 64-step steady state	344.7 tok/s	307.9 tok/s	1.12×
Decode at KV depth 512	296.5 tok/s	304.1 tok/s	0.97×
Prefill, 512-token prompt (marginal)	12765 tok/s	14085 tok/s	0.91×
Prefix reuse, 512-token prefix + 64 new	29.7 ms to first token	—	—

• Decode is ahead on both families, at depth 0 and at depth 512 on Gemma. Qwen2 at depth 512 is level with llama.cpp (0.97×): it has no sliding window, so all 512 keys are attended while Gemma's ring buffer wraps.
• Prefill is behind — 0.84× (Gemma) and 0.91× (Qwen). It is a real batched path (int8 tensor-core MMQ plus cuBLAS); closing the rest is on the roadmap.
• Prefix reuse has no llama.cpp counterpart (llama-bench cannot measure it), so it is reported as first-token latency in milliseconds and nothing else. The exactness claim beside it is a test, not a benchmark: a prefix-reuse continuation is token-identical to a full prefill at every tested prefix length (test_prefix_cache, 12/12).
• Decode at 512/2048/4100-token depth, the per-kernel breakdowns, and the older hand-rolled measurements are in design/benchmark.md; they were taken on earlier code and on different harnesses, so they are history, not a comparison.

Benchmark conditions

GPU	NVIDIA GeForce RTX 3050, 8 GB, compute 8.6
Driver / CUDA	595.84 (CUDA 13.2 driver API); nvcc 12.0.140
CPU	AMD Ryzen 5 5500 (6C/12T, Zen 3), PCIe 3.0
RAM	32 GB DDR4-2400, single channel
Host	Ubuntu 24.04, Linux 6.8, x86-64, g++-13, build type Release
Gemma model	unsloth/gemma-4-e4b-it-GGUF → gemma-4-E4B-it-Q3_K_M.gguf (4,058,137,728 bytes, sha256 0980898d0e7d9c75cea05141bf4010393312101ef6cf5b31f8197e0de18655f4)
Qwen model	qwen2.5-0.5b-instruct-q4_k_m.gguf (491,400,032 bytes, sha256 74a4da8c9fdbcd15bd1f6d01d621410d31c6fc00986f5eb687824e7b93d7a9db)
llama.cpp	commit 43fe9c64281, CUDA build, llama-bench -ngl 99
SonicBoom build	tools/sb-report.sh: git commit, CUDA archs actually compiled, LLVM version

The comparison is against llama.cpp at the commit above; newer versions may be faster or slower. The Qwen2 greedy token stream agrees with llama.cpp except at llama.cpp's own near-ties, where a different prefill batch geometry flips the last-place logits (first divergent step and top-1/top-2 margin recorded in design/benchmark.md).

Community results

Numbers from other people's hardware belong here, next to the command that produced them. To add a row, run both tools on your machine and paste the output into an issue or a PR:

tools/sb-report.sh          # hardware + the exact build (git commit, cuda archs)
LLAMA_BENCH=... tools/bench.sh your-model.gguf --label your-model

sb-report.sh --json gives the same facts as one machine-readable object, so a result can carry its own conditions. Results are published as reported, with the reporter credited, and marked as not reproduced by us.

GPU	Driver / CUDA	SonicBoom build	Model	SonicBoom decode	llama.cpp tg64	Ratio	Decode tokens vs sm_86 golden	Reporter
RTX 4060 Ti 16 GB (sm_89)	550.163.01 / 12.4	1462b04 (sm_86 code, before tuning)	Gemma 4 E4B-it Q3_K_M	73.6 tok/s	78 tok/s	0.94	identical (64/64)	@svenha

The 4060 Ti row predates tools/bench.sh and the build-for-the-detected-GPU default: it measures sm_86 code running on Ada with test_generate, against llama.cpp 43fe9c64281 built for the card. Both numbers come from the reporter's machine.

---

Quick start

Requirements

• Linux x86-64, g++-13, CMake, Ninja (optional), libyaml-dev. The CPU quantized-matmul kernels are AVX2 (with a scalar fallback); an aarch64 build is not wired up yet — the AVX2 translation unit has no architecture guard, so it does not compile there as it stands. The port is a guard plus a NEON sibling of the same fixed-point kernel.
• CUDA toolkit 12.0 or newer and an NVIDIA GPU, compute capability 6.1 or newer (__dp4a) — 7.5 or newer for the tensor-core MMQ path. v0.0.2 is built and tested for sm_86 only (RTX 30-series).
• GNU Guile 3.0
• SonicCross, a required build-time dependency (see below)

Build

SonicBoom's operator library, native-torch, is generated at build time from PyTorch's native_functions.yaml, as PyTorch itself does with torchgen. SonicBoom uses SonicCross, a torchgen written in Guile, which turns the YAML into native-torch's C++ declaration, registration and dispatch sources (more than 7,000 per-operator headers). SonicCross runs only at build time and is not linked into libsonicboom.so. Install SonicCross v0.0.2 first, the version pinned in tools/generation/SONICCROSS.lock:

git clone https://github.com/hardenedlinux/SonicCross
cd SonicCross
git checkout v0.0.2
./bootstrap && ./configure && make && sudo make install
cd ..

This installs guild torchgen. If guild is not on your PATH, pass -DSONICCROSS_GUILD=/path/to/guild to CMake. Then build SonicBoom:

git clone --recursive https://github.com/hardenedlinux/SonicBoom
cd SonicBoom

# One-time step: build the pinned LLVM/MLIR into build/llvm-mlir-install.
# It is statically linked into libsonicboom.so and is not a runtime dependency.
# The third_party/llvm-project submodule is only this script's input.
tools/mlir/build-mlir.sh

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSONICBOOM_USE_CUDA=ON
cmake --build build -j

The LLVM submodule is large, so the first clone and build take a while.

An LLVM/MLIR already installed elsewhere can be used instead — the submodule stays empty and is not needed at all:

cmake -S . -B build -DSONICBOOM_USE_CUDA=ON \
      -DMLIR_DIR=/opt/llvm/lib/cmake/mlir -DLLVM_DIR=/opt/llvm/lib/cmake/llvm

A different LLVM major is accepted with a warning, naming the version SonicBoom is tested against (23). -DSONICBOOM_ENABLE_MLIR=OFF builds without MLIR entirely, and the error message when MLIR is missing lists all three routes.

Whatever was configured is compiled into the library and can be read back — from C with sb_build_info() (capi/include/sonicboom/capi.h), from Guile with (build-info) (bindings/guile/README.md), and from the command line with tools/sb-report.sh, which prints it beside the host, the GPU and the toolchain:

tools/sb-report.sh            # human-readable; --json for machines

Run (Guile)

Download the model and verify it:

mkdir -p models
curl -L -o models/gemma-4-E4B-it-Q3_K_M.gguf \
  https://huggingface.co/unsloth/gemma-4-e4b-it-GGUF/resolve/main/gemma-4-E4B-it-Q3_K_M.gguf
echo "0980898d0e7d9c75cea05141bf4010393312101ef6cf5b31f8197e0de18655f4  models/gemma-4-E4B-it-Q3_K_M.gguf" | sha256sum -c

Any GGUF the runtime supports runs the same way — for instance qwen2.5-0.5b-instruct-q4_k_m.gguf, the other verified model (both files and their checksums are in Scope of v0.0.2).

Then start Guile:

LD_LIBRARY_PATH=build/core guile -L bindings/guile/scheme

(use-modules (sonicboom inference))

(define model   (load-model "models/gemma-4-E4B-it-Q3_K_M.gguf"))
(define session (make-session model 4096))

(display (generate model session "The three laws of thermodynamics are" 64))

;; Prefix reuse is automatic: when a later prompt on the same session starts
;; with the same tokens, only the new suffix is prefilled.
(prefill! session (tokenize model "..."))   ; returns the number of reused tokens

(free-session! session)
(free-model! model)

The same functions are available from C through capi/include/sonicboom/capi.h (sb_llm_*).

Test

ctest --test-dir build --output-on-failure

Tests that need the model skip themselves when the GGUF is not present, and the two ResNet-18 end-to-end tests skip (CTest SKIP_RETURN_CODE 77) when their generated sidecar artifacts are absent, which is the state of a fresh checkout.

Benchmark

LLAMA_BENCH=/path/to/llama-bench tools/bench.sh your-model.gguf
tools/sb-report.sh        # the hardware and the build behind those numbers

Every metric is the median of 3 process runs and both runtimes are measured in the same session; tools/bench.sh is the only benchmark SonicBoom publishes numbers from.

---

Scope of v0.0.2

v0.0.2 is a working runtime, not a drop-in replacement for llama.cpp. Its limits:

• Two verified model families: Gemma 4 E4B-it Q3_K_M and Qwen2.5-0.5B-Instruct Q4_K_M. Both run end to end — load, tokenize, prefill and decode on CPU and CUDA, prefix reuse, OpenAI-compatible serving — and both are checked against llama.cpp on the same hardware.

  	file	sha256
  Gemma 4 E4B-it	gemma-4-E4B-it-Q3_K_M.gguf (4,058,137,728 bytes)	0980898d0e7d9c75cea05141bf4010393312101ef6cf5b31f8197e0de18655f4
  Qwen2.5-0.5B-Instruct	qwen2.5-0.5b-instruct-q4_k_m.gguf (491,400,032 bytes)	74a4da8c9fdbcd15bd1f6d01d621410d31c6fc00986f5eb687824e7b93d7a9db

  The GGUF's architecture string selects the plan and the chat format. There is no if (arch == ...) in the planner or the backends, and no model-specific code outside the GGUF loader and the plan emitter — the Qwen2 differences (a SiLU-gated FFN, full attention with no sliding window, a bias on each Q/K/V projection, an attention scale) are all generic ops and attributes, and the fallback to the token embedding when a file omits output.weight is generic too.

• Quant formats: Q3_K, Q4_K, Q5_K, Q6_K, Q5_0 and Q8_0. f16 and q4_0 tensors are converted to Q5_K at load time, which requires the row length to be a multiple of 256; anything else is refused rather than converted. Q6_K loads and decodes natively, but its prefill does not use the int8 tensor-core weight kernel yet — it dequantizes to fp16 and uses cuBLAS, so a Q6_K ffn_down makes prefill slower than the same shape in Q4_K.

• GPU: CUDA on sm_86. CMAKE_CUDA_ARCHITECTURES defaults to native (the GPU in the build machine) and is resolved at configure time; other architectures have not been tested. Whatever was compiled is reported by sb_build_info() / tools/sb-report.sh, so a binary never has to be taken on trust about which SMs it contains. One community report ran on an RTX 4060 Ti (Ada) with decode tokens identical to the sm_86 results, see Community results. A CPU path exists but is a reference implementation, not a fast one.

• Sampling: greedy only. temperature, top_p and top_k return SB_ERR_UNSUPPORTED.

• Chat template: the serving layer picks the format from the loaded architecture — Gemma's <|turn>… turn markers, or Qwen2's ChatML — and takes the end-of-turn tokens from the model's metadata. generate itself still takes raw text, so a caller that does not go through the server formats its own turns.

---

How much of this is ggml?

SonicBoom ports a small set of kernels from ggml/llama.cpp under the MIT license. They live in ggml_derived/ directories, with the details in NOTICE and LICENSES/ggml-MIT.txt.

	Lines of code
ggml/llama.cpp-derived (core/src/quant/ggml_derived, core/src/vocab/ggml_derived)	2,820
SonicBoom core total (core/src, core/include)	19,953
Derived share	~14.1%

Counted with cloc 1.98. These are code lines only, excluding blanks and comments. The C ABI, the Guile bindings and the dependencies are not counted.

Ported: quantized matrix multiply (MMVQ for decode, MMQ for prefill), q8_1 activation quantization, the legacy Q5_0/Q8_0 block formats, and the BPE tokenizers (SPM-style for Gemma 4, byte-level with the qwen2 pre-tokenizer, plus the Unicode classes and byte maps they split on).

SonicBoom's own: everything else. That includes the planner and graph passes, operator fusion, CUDA Graph capture and replay, the device-resident KV cache, split-KV flash decoding, tiled flash attention for prefill, chunked prefill, prefix-cache reuse, and batch-invariant dense GEMM.

Decode uses the same quantized GEMV kernels as llama.cpp and is still 12–18% faster on the two models measured (see Numbers). That difference comes from the runtime, not from the kernels.

---

Beyond ggml: a compiler stack, not a kernel collection

llama.cpp is fast because of hand-tuned kernels, and in that design every new fusion or model-specific optimization means writing another kernel by hand. SonicBoom chooses an implementation for each operation from three tiers, in this order of preference:

Tier	What it does	Status
1. MLIR compiler	S-Expr graphs are lowered to MLIR (linalg) and compiled to native code. ResNet-18, imported from ONNX, runs through this path and matches the ONNX reference.	Shipped on CPU (LLVM JIT). GPU code generation is next.
2. Graph planner	Rewrites the whole model before it runs: quantize-once, norm→quantize and activation→quantize fusion, dead-computation elimination, and CUDA Graph capture of the full decode step.	Shipped: this tier produces the decode lead measured above.
3. Hand-written kernels	Used only where the other tiers are not yet fast enough, for example the ported ggml MMQ/MMVQ kernels.	Shipped

What follows from this design:

• Planner gains carry over to every model. A pass written once applies to any architecture that is lowered into the planner, with no new kernel required. Gemma 4 and Qwen2 both run through the same planner and backends, with the architecture differences expressed as generic ops and attributes rather than model-specific code.
• The kernels are no longer the limit. With identical kernels, the runtime is already ahead of llama.cpp. Decode is still below the GPU's memory bandwidth limit for this model. Speculative decoding fits in as one more graph for the planner to schedule.
• MLIR provides new hardware targets. NVPTX is already built into the pinned LLVM. MLIR's GPU, SPIR-V/Vulkan and accelerator back ends are available to SonicBoom through the same S-Expr graphs, without a separate rewrite for each target.

---

MLIR migration: a staged plan

MLIR is already linked into libsonicboom.so and already carries a graph: S-Expr documents are lowered to arith/func/linalg and executed through the LLVM JIT, which is how ONNX-imported ResNet-18 runs. What is not yet generated are GPU kernels — those are still the hand-written CUDA ones. The migration is staged so that each step is independently useful and independently verifiable, and so that a generated kernel can always be checked against the hand-written one it replaces.

stage	what lands	done when
0. Lowering (shipped)	S-Expr → arith/func/linalg, executed on CPU through the LLVM JIT; ONNX ResNet-18 matches the ONNX reference	already the case
1. One GPU glue kernel	A single fused region — norm + quantize — lowered linalg → gpu → nvvm → cubin, JIT-compiled at load and cached by shape signature	it runs inside the decode step and matches the hand-written kernel numerically within the documented tolerance, with the canonical tools/bench.sh metrics no worse
2. The glue set	The rest of the between-kernel glue: activation→quantize, residual add, RoPE	each region replaces a kernel without moving the canonical numbers down
3. GPU-independent targeting	The same lowering for a GPU the binary was not built for (compute capability discovered at run time), and for a second vendor through SPIR-V	a machine with a different GPU runs without a SonicBoom rebuild
4. Beyond LLM glue	Whole-model lowering for the research route's traced graphs	a traced model runs on the same planner and backends as a built-in one

Ground rules that hold at every stage: generated code and the JIT live behind Layer 2 (no MLIR, LLVM, CUDA or native-torch type crosses the C ABI); the hand-written kernel remains the numerical and performance reference; and the gate is the canonical benchmark plus the existing test suite, because "it compiles" and "it is faster" are not the same claim. The pinned LLVM is 23 — a different major is a warning, not an error (see CMakeLists.txt).

---

Scheme: write it eagerly, run it compiled

The research route keeps PyTorch's eager style without Python's limits. It is built on two Scheme features that Python lacks: hygienic macros and delimited continuations. It currently ships as a pure-Guile research prototype in bindings/guile/scheme/sonicboom/trace, documented in design/guile-tracing-prototype.md.

• Data-dependent control flow without graph breaks. A code walker written entirely in syntax-rules rewrites if, cond, case, and, or, do and named let, so a branch on a tensor value becomes a where node inside the graph. TorchDynamo needs a CPython bytecode interpreter to approach this problem; in SonicBoom it is a macro.
• Continuations where macros cannot reach. The tracer suspends at the branch with a delimited continuation. It can break the graph and resume with a concrete value, cache a trace tree per branch, or resume the same continuation twice, once for each branch, and merge the results. A Python generator cannot be resumed twice.
• A whole training step is one graph. Reverse-mode autodiff appends the backward pass to the traced forward pass, so the forward pass, loss, gradients and optimizer update compile together. The prototype already traces and trains a small Transformer.
• PyTorch-compatible at the op level. Every emitted node carries its Core ATen name, so graphs map directly onto native-torch's kernels.
• Portable by construction. It uses only R7RS syntax-rules and delimited continuations, with no Guile compiler internals, so other Scheme implementations can adopt it.

Next step: connect the tracer to the core, so that traced graphs are compiled once through the C ABI and run on the same planner and backends as the production route.

---

Where this is going

One core and one C ABI, with a growing set of front ends:

 Scheme (eager, traced)  ─┐
 ONNX / PyTorch export   ─┼─→  S-Expr graph  ─→  planner  ─→  MLIR / kernels  ─→  CPU · CUDA · …
 GGUF models (C++ core)  ─┘

A model written once, by tracing, by import, or as a native definition, gets every planner pass and every back end SonicBoom adds afterwards. That is the difference between a fast LLM runner and a framework, and it is the direction SonicBoom is built for.

---

Architecture

Four layers, with dependencies pointing strictly downward:

Layer 3    C ABI / Guile / other bindings
Layer 2    SonicBoom stable runtime interface (nt::)
Layer 1    native-torch implementation + adapter
Layer 0    CPU / CUDA

libsonicboom.so is the published core library. native-torch and MLIR are statically linked into it. native-torch's operator sources are generated at build time by SonicCross, a torchgen written in Guile. The stable binary-compatibility boundary is the C ABI (capi/); the C++ Layer 2 interface is source-level only.

Two routes share this core:

• Production inference: model architectures are defined in the C++ core. The static execution planner lowers them into a fused plan that is captured as CUDA Graphs. The host language loads weights and drives sessions.
• Research and training: (sonicboom nn), (sonicboom model) and (sonicboom train) give a minimal PyTorch/Keras-like API with native-torch reverse-mode autograd. Models can be exported as an S-Expr graph plus weights (model-export!) and run by the standalone C++ loader through the S-Expr → MLIR path, with no Guile or Python involved. See design/guile-ai-framework-progress.md.

---

Roadmap

• Formats and models: Q6_K, Q8_0 and Q5_0 are native and Qwen2 is a second verified family; next are further families (Llama, Mistral) and the quant formats they ship with
• Speed: prefill parity with llama.cpp; speculative decoding, aiming for an average decode rate around 100 tok/s on the same RTX 3050
• Compiler: MLIR GPU code generation for fused elementwise and norm chains
• Research route: connect the Guile tracer to the core and compile traced graphs through the C ABI
• Usability: sampling — the runtime is still greedy only — on top of the chat templates the serving layer now selects per model
• Scale: larger MoE models with expert offload
• Beyond LLMs: diffusion models, with a converter for existing ComfyUI workflows
• Distribution: Guix packaging, CPU first. A Vulkan backend is planned so that a fully free-software stack also gets GPU acceleration. The backend scaffold is already in the tree (-DSONICBOOM_USE_VULKAN=ON).

---

Commercial support

SonicBoom is developed by HardenedVault, which specializes in Linux, compiler and systems security.

We offer consulting services for enterprise deployment and optimization: private or air-gapped LLM inference, performance tuning on your hardware, and hardened, auditable inference stacks. Contact consulting@hardenedvault.com.

---

License

SonicBoom is licensed under GPL-3.0-or-later (see LICENSE). Code derived from ggml/llama.cpp keeps its MIT notice (see NOTICE). ggml is not linked as a runtime dependency.