Luxand LLM SDK runs local LLMs embedded in your desktop, mobile, and edge apps — fully offline, no daemon, no cloud, no telemetry. Link one stable C library and get chat, streaming, vision, tool-calling agents, and schema-constrained JSON from one API. Free for non-commercial use; transparent pricing when you ship. Built by Luxand — on-device recognition SDKs since 2007, shipping inside thousands of applications.
Ship on-device AI without maintaining the cross-platform inference stack yourself.
| Download | Releases — SDK packages for every platform · the Luxand LLM Studio demo app |
| Docs | Developer Manual — getting started · concepts · full API reference |
| Platforms | Android · iOS · Windows · Linux · macOS · ARM / Raspberry Pi |
| Models | any GGUF model — a curated, tested list (Qwen · Gemma · Granite · LFM · …); download or bring your own |
| Backends | CPU · Metal · CUDA |
| Wrappers | C++ · Kotlin/Java · Swift · .NET · Python |
| Engine | pinned llama.cpp, behind a stable C ABI |
From a fresh checkout to a running local model — three commands:
./download-sdk.sh && ./fetch-model.sh # stage the SDK and the Qwen3.5-2B starter pair (model + vision projector)
cmake -S samples/c -B build/c && cmake --build build/c
./build/c/luxllm-c-basic models/Qwen3.5-2B-Q4_K_M.ggufEvery download is a complete pair — text, tools, and image input work out of
the box. ./fetch-model.sh --quality selects the stronger Qwen3.5 4B pair;
./fetch-model.sh --gemma selects the pinned Gemma 4 E2B Q4_0 conversion plus
Luxand's vision-only Q8_0 build of its projector (227 MB instead of 987 MB).
The projectors are Luxand's Q8_0 requantizations of the originals; the license
and modification notice download beside them. All three models are provided
under the Apache License 2.0.
(On Windows: run download-sdk.bat and fetch-model.bat — double-click works — then build
with CMake from a Developer PowerShell; the Visual Studio generator is multi-config, so add
--config Release to the build and run from build\c\Release\. Prefer manual? Grab the SDK
package from Releases and unzip it into
./sdk/.)
Windows tip: if you downloaded this repo as a ZIP, right-click the ZIP → Properties → Unblock → OK before extracting — otherwise Windows marks the extracted scripts as internet-downloaded and shows a "publisher unverified" prompt when you run them. Already extracted? The same right-click → Unblock works on
download-sdk.batandfetch-model.bat— or simply click Run on the prompt: it is Windows' standard notice for files from the internet, not an error.
Just want to try it, no code? Grab Luxand LLM Studio — the free desktop chat app for Windows, macOS, or Linux — from Releases, pick a GGUF model, and chat. It bundles the app runtime and SDK; the platform requirements below still apply. Its source lives in
demos/.
Point CMake at the staged SDK:
find_package(LuxandLLMSDK CONFIG REQUIRED) # cmake -DCMAKE_PREFIX_PATH=./sdk
target_link_libraries(my_app PRIVATE LuxandLLMSDK::LuxandLLMSDKCpp) # C++ wrapper targetA complete chat — load, converse, print — is the whole program:
#include "LuxandLLMSDK.hpp"
#include <cstdio>
int main() {
luxllm::Model model("models/Qwen3.5-2B-Q4_K_M.gguf");
luxllm::Engine engine = model.createEngine("ContextTokens=2048");
luxllm::Session session = engine.createSession();
session.add(luxllm::Role::System, "Answer in one concise sentence.");
session.add(luxllm::Role::User, "Name one practical use for local LLM inference.");
luxllm::Result reply = session.generate("MaxTokens=512");
std::printf("%s\n", reply.text.c_str()); // RAII frees everything, in order
return 0;
}Streaming is the same call with a pull-friendly callback (no background threads, no ABI callbacks):
session.generate("MaxTokens=512", [](std::string_view chunk) {
std::printf("%.*s", (int) chunk.size(), chunk.data());
return true; // return false to stop early
});That's the core. The next section shows the pieces real apps need — all on the same
Model → Engine → Session object model.
Tool-calling agents — OpenAI-style tools in, parsed calls out, with typed argument accessors (no hand-parsing model JSON):
luxllm::Result phase = session.generateWithTools("ToolChoice=Required", toolsJson);
for (int i = 0; i < (int) phase.toolCalls.size(); ++i) {
double a = session.toolArgDouble(i, "a"); // typed, validated
double b = session.toolArgDouble(i, "b");
session.addToolResult(i, std::to_string(a + b)); // feed the result back
}
luxllm::Result answer = session.generate("MaxTokens=512");Schema-constrained JSON — pass a JSON schema; generation is grammar-constrained to it:
luxllm::Result r = session.generate("MaxTokens=512", jsonSchema); // schema-shaped outputMany conversations, one engine — independent sessions batch together in a single decode.
You create sessions and step the engine; there is no llama_batch, no seq_id, no logits
bookkeeping:
luxllm::Engine engine = model.createEngine("MaxParallelSessions=4");
for (auto& s : sessions) { s.add(luxllm::Role::User, prompt(s)); s.start("MaxTokens=512"); }
for (bool more = true; more; ) { more = engine.step(); /* drain each: s.readText(chunk) */ }Also included: vision (image message parts — a vision model loads its matching projector
file via MultimodalProjectorFile=…), reasoning-trace extraction for models that support
it, and a drop-in OpenAI-compatible local server with a chat web UI (next section).
The same six example programs, written both ways — with the SDK and directly on llama.cpp —
took ~3× less code with the SDK. Runnable versions of every snippet above, both ways, are in
the downloadable comparison package
(self-contained; builds against the free SDK — run measure.py to reproduce the numbers
yourself).
The same object model ships as native wrappers over one stable C ABI:
| Language | Package | Language | Package | |
|---|---|---|---|---|
| C / C++ | find_package(LuxandLLMSDK) |
Kotlin / Java | AAR (./sdk/luxllm-android-release.aar) |
|
| Swift | XCFramework (./sdk/LuxandLLMSDK.xcframework) |
.NET / C# | source wrapper + native lib (LUXLLM_NATIVE_PATH) |
|
| Python | pip install luxand-llm==1.0.1 |
Every language has a runnable sample against the same staged ./sdk/:
cmake -S samples/cpp -B build/cpp && cmake --build build/cpp
python -m pip install -r samples/python/requirements.txt
python samples/python/basic.py models/Qwen3.5-2B-Q4_K_M.gguf
dotnet run --project samples/dotnet/Basic -- models/Qwen3.5-2B-Q4_K_M.gguf
swift run --package-path samples/swift/cli luxllm-swift-basic models/Qwen3.5-2B-Q4_K_M.gguf # after ./download-sdk.sh --artifact apple
ADB=/path/to/adb samples/kotlin/chat-gui/run-on-device.sh models/Qwen3.5-2B-Q4_K_M.ggufSee samples/README.md for the complete sample overview.
Beyond the one-lesson-per-file samples, three full GUI chat apps implement one
"advanced chat" spec — streaming with the transcript contract, a thinking toggle
(gated by the model's SupportsThinking readback), a current_date tool through a
streaming tool loop, and image input gated by SupportsImageInput:
| App | Platform | Stack |
|---|---|---|
samples/kotlin/chat-gui |
Android | Jetpack Compose; 📷 camera + 🖼 gallery input |
samples/swift/chat-gui |
macOS + iOS from one Xcode project | SwiftUI; 📷 camera + 🖼 gallery on both |
samples/dotnet/Chat.Gui |
Windows / macOS / Linux | Avalonia (.NET) |
Each is small enough to read in one sitting and structured to be lifted straight into
your own product. For these chat apps, the recommended model is the Gemma-4-E2B vision
pair (./fetch-model.sh --gemma) — the phone-proven pair. The Qwen3.5 pairs work too,
images included: their hybrid architecture reuses earlier turns through the engine's
context checkpoints (on by default).
Every sample is written to be ejected — a single source file with zero references to this
repo. The per-language recipes (what to copy, how to reference the wrapper and the native
SDK) are in samples/README.md. In every
case, keep the activation call (ActivateLibrary("") = keyless Community mode) and replace
the empty string with your license key when you ship.
One call starts a local HTTP server on the loaded model — in-process, fully offline, embedded
in the SDK library itself (LuxLLM_StartServer in C; exposed by every wrapper). It runs on
every platform — including Android and iOS — so an app already written against an
OpenAI-compatible API can go fully local without rewriting its client code: start the server,
change the base URL.
No code required to try it: every desktop SDK package ships the server prebuilt as
bin/luxllm-server — its complete source is the samples/cpp server
sample (luxllm-cpp-server), so the binary you run is also the integration example you can
read and fork:
./sdk/bin/luxllm-server --model models/gemma-4-E2B-it-Q4_0.gguf # ./fetch-model.sh --gemma; Gemma keeps long chats fastPoint any existing OpenAI client at http://127.0.0.1:8080/v1, or open
http://127.0.0.1:8080/ for the built-in chat web UI. /v1/chat/completions covers
streaming, tool calls, JSON-schema response_format, and image input on multimodal models.
The scripted equivalent is samples/python/server.py; the full
endpoint and parameter reference is the embedded-server section of the
Developer Manual.
| Target | Minimum platform | CPU / ABI | Additional runtime |
|---|---|---|---|
| Linux x86-64 | glibc 2.28+ | AVX2 | libgomp.so.1 |
Linux ARM64 baseline (linux-aarch64-armv8-a, linux-aarch64-armv8-a-norepack) |
glibc 2.28+ | ARMv8-A; Raspberry Pi 4 compatible | None beyond the base OS |
Linux ARM64 optimized (linux-aarch64-armv8.2-dotprod-fp16) |
glibc 2.28+ | ARMv8.2 dotprod/FP16; Raspberry Pi 5 / Cortex-A76+ | None beyond the base OS |
| Windows x64 | Windows 10 22H2 or Windows 11 | AVX2 | No VC++ Redistributable required for SDK/server |
| macOS | macOS 13.3+ | Apple silicon | None beyond the OS |
| iOS | iOS 16.4+ | arm64 device and simulator | None beyond the OS |
| Android | Android 7.0 / API 24+ | arm64-v8a (ARMv8.2 dotprod/FP16), x86_64; 16 KB-page compatible |
None beyond Android |
The exact package names, CUDA driver floors, and redistribution requirements are in the Developer Manual.
- C — the plain-C ABI header (
LuxandLLMSDK.h); C++ wrapper — C++17 or newer - Python 3.9+ (pure-Python wrapper) · .NET 8+ (wrapper multi-targets
netstandard2.0/net8.0/net10.0; samples use net8.0 LTS) · Swift 6 toolchain (SwiftPM) - Android development — JDK 17+ and the Android SDK/NDK
The engine library itself has no GPU dependency. On Linux/Windows, use the -cuda
package variant — the same engine plus a loadable CUDA plugin library beside it (needs a
Turing-or-newer NVIDIA GPU and a current driver; no CUDA toolkit install — the runtime is
bundled with the plugin, see the
manual's Distribution chapter).
download-sdk.sh / download-sdk.bat pick it automatically on machines with an NVIDIA
driver (--cpu opts out). Device=Auto
uses the GPU when present and falls back to CPU; a CPU-only machine runs the same binaries
untouched. On Apple platforms, Metal is built in — no separate package.
Two rules keep CUDA deployments out of trouble:
- Never link the
-cudalibrary. Applications always link the baseLuxandLLMSDKlibrary only; the-cudafile is a runtime plugin with no linkableLuxLLM_*symbols. - The plugin is found next to the base library. At startup the SDK loads the plugin
from the exact directory the base library itself was loaded from (in the shipped layout:
sdk/bin/) — keep the two files together wherever your deployment puts them. Next to the executable or on the system path does nothing.
To see which backend is active at runtime, read engine.parameter("Backend")
(C: LuxLLM_GetEngineParameter(..., "Backend", ...)) — it returns CPU, CUDA, or Metal.
A session is used by one thread at a time; different sessions of one engine on different
threads batch together automatically, and LuxLLM_StopGeneration is the any-thread
cancellation path. The full contract is the Threading & Concurrency section of the
Developer Manual.
Runs any GGUF model — Qwen, Gemma, Granite, LFM, and more. We publish a curated, tested list — each model run through a per-model behavioral test suite (chat, streaming, tools, JSON-schema, vision, thinking on/off) against the exact engine build we ship — so you know what works before you download. Get the weights from Hugging Face (or bring your own); check each model's license before redistribution.
The quickstart uses the Qwen3.5 2B pair to keep the first download small —
1.3 GB of model plus a 362 MB vision projector. For stronger chat and tool
behavior, ./fetch-model.sh --quality downloads the Qwen3.5 4B pair
(2.7 GB + 367 MB) highlighted below.
We benchmark open models on this SDK with the agentic subsets of the Berkeley Function-Calling Leaderboard (BFCL V4) — ~40,000 tasks scored, including the complete 600-case multi-turn suite for every model. Every candidate has ~4B or fewer active parameters: the class that decodes at usable speed on mass-market devices. Which model for which job:
| Use case | Model | Agentic Score | Download |
|---|---|---|---|
| The most capable Apple-Silicon Macs, PCs with GPUs, 16 GB+ RAM |
Qwen3.6 35B A3B (2-bit) 2-bit GGUF · UD-Q2_K_XL · reasoning |
72.3% | 12.3 GB |
| Vision at flagship quality image input on the same hardware class |
Gemma 4 26B A4B (2-bit) 2-bit GGUF · UD-Q2_K_XL · reasoning · vision |
71.8% | 10.5 GB +1.2 GB mmproj |
| Mass-market phones & laptops the best score per gigabyte in our table |
Qwen3.5 4B 4-bit GGUF · Q4_K_M · reasoning · vision |
67.0% | 2.7 GB +0.4 GB mmproj |
| Image chat on-device smallest vision-capable Gemma — current 2026-07-16 conversion |
Gemma 4 E2B 4-bit GGUF · Q4_0 · reasoning · vision |
51.1% | 2.8 GB +0.2 GB mmproj |
| The quickstart starter the ./fetch-model.sh starter pair — when every megabyte counts |
Qwen3.5 2B 4-bit GGUF · Q4_K_M · reasoning · vision |
50.5% | 1.3 GB +0.4 GB mmproj |
| Under a gigabyte Raspberry Pi-class boards, older phones |
Granite 4.0 H 1B 4-bit GGUF · Q4_K_M · non-reasoning |
38.1% | 901 MB |
Internal run of a BFCL V4 subset (July 2026) on the Luxand LLM SDK release runtime — subset scores differ from full-suite numbers, so compare models within this table rather than against the official leaderboard. Every score is text-only tool calling; vision quality is a separate model capability. Full table, sub-gigabyte models, charts, and scoring notes: luxand.com/llm-sdk/benchmarks.
include/ Public C ABI header
wrappers/ Source-form wrappers (C++, Kotlin/Java, Swift, .NET, Python)
samples/ Per-language lesson samples + the GUI chat apps
(kotlin/chat-gui, swift/chat-gui, dotnet/Chat.Gui), all against ./sdk/
demos/ Luxand LLM Studio, the desktop chat app (.NET/Avalonia) — prebuilt on Releases, source here
sdk/ Your downloaded SDK artifacts (gitignored)
models/ Your GGUF model weights (gitignored)
The repo is source only: SDK binaries come from Releases, model weights from
Hugging Face (or bring your own). The ./sdk/ convention is shared by every sample: desktop zip
(include/, bin/, cmake/), Apple XCFramework, or Android AAR.
- Community — free. Key-less, all features, all platforms. For evaluation, development, and non-commercial use on your own devices — distributing an app, even a free one, requires a commercial license. No signup, no phone-home. See LICENSE.txt. Building something open-source or non-commercial you want to distribute? Contact us — we issue free keys at our discretion.
- Commercial — a per-product license to ship in a commercial app, with support and updates. Transparent, published pricing — no sales call to start; buy online and your license key is issued instantly. Perpetual/OEM options for larger deployments. See pricing & buy a license →
Applying a commercial key is one call: LuxLLM_ActivateLibrary("<key>") in C — the same call
in every wrapper. Verification is offline (a signed key, checked locally — no callback, no
metering); without a key the SDK simply runs in Community mode.
The engine is llama.cpp. Shipping it is our job. Luxand LLM SDK is built on a pinned llama.cpp build, wrapped behind a stable C ABI. llama.cpp, ggml, and other bundled components are covered by THIRD-PARTY-NOTICES.txt.
What models can I run? Any GGUF model — see Models above for the curated, tested list; you download the weights (Hugging Face, etc.) or bring your own.
Isn't on-device inference slow? On commodity x86 CPU it's for background, short-output work — extraction, classification, summarization, structured JSON, tool routing; long interactive chat wants a GPU. On Apple Silicon (Metal) and CUDA GPUs it's interactive. Capability benchmarks are published at luxand.com/llm-sdk/benchmarks.
We already use an OpenAI-compatible API — can we switch?
Yes — start the embedded server in-process (it
runs on every platform, mobile included) and point your existing client at 127.0.0.1.
Chat completions with streaming, tool calls, JSON-schema response_format, and image input
are covered. No client rewrite.
Isn't this just a llama.cpp wrapper?
The engine is llama.cpp — the proven, de-facto standard engine for GGUF models — pinned to an
exact, validated build. Any GGUF model runs; there's no proprietary runtime. What we build is
everything shipping requires around the engine — and if you've built on llama.cpp directly, you
know that work: cross-platform builds, the ABI, five language bindings, tool-calling /
JSON-schema / reasoning that hold up across models, per-model validation, packaging, support.
That's the SDK. For C/C++, it's a header and one .so to link — no engine to compile, no
build graph to maintain. The same six example programs, written both ways:
~3× less code with the SDK (C++ 276 vs 908 lines; C 533 vs 1306) —
reproduce it yourself.
If I embed llama.cpp directly, what am I signing up to build?
Mostly integration code — but some of it is real machinery. One example: prompt caching.
Each chat turn resends the whole conversation. Without caching, the engine re-processes all
of it before answering — every earlier message, and every attached image again through the
vision encoder. The fix is to keep the KV cache across turns and evaluate only what changed.
Doing that correctly is real work: match the cached prefix token by token, match images by
content hash, find the exact point where re-evaluation must start. llama-server ships this
logic inside the server application. The llama.cpp library itself does not provide it (as
of llama.cpp b9967), so an app that embeds the library writes it from scratch. The SDK
provides it behind the C ABI on every platform — including iOS and Android, where a server
sidecar isn't an option. The payoff is biggest on CPU and phones: without
reuse, a long vision chat can stall for seconds before every reply.
Why not just run Ollama or LM Studio? They're a desktop daemon/GUI for running models on your own machine — not embeddable in a shipped app (no in-process library, no mobile). We're for shipping inference inside your product.
Why not vLLM or TGI? Server-side GPU serving at scale — a different deployment. We're on-device / embedded.
Can I ship a free app without a license? No — any distributed app needs a license key. Keyless Community mode covers development, evaluation, and non-commercial use on your own devices. Building something open-source or non-commercial you want to distribute? Contact us — we issue free keys at our discretion.
Does it phone home? No. No telemetry, no license callback, no metering — ever.
- Website — luxand.com/llm-sdk
- Pricing & buy a license — luxand.com/llm-sdk/pricing
- Documentation — the Developer Manual: getting started in all six languages, core concepts, threading, the embedded server, and the per-function API reference
- llama.cpp vs. Luxand comparison (downloadable, self-contained, reproducible) — comparison package
.zip - Bugs & questions — GitHub Issues on this repo
- Support / commercial inquiries — luxand.com/support