Skip to content

About

Cross-platform SDK for private, offline LLM inference in desktop, mobile, and edge apps.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

Luxand LLM SDK

Embed a local LLM in your app — desktop, mobile, and edge, fully offline

Platforms Wrappers: C++ · Kotlin · Swift · .NET · Python Backends: CPU · Metal · CUDA License: free for evaluation & non-commercial Engine: pinned llama.cpp

Luxand LLM SDK runs local LLMs embedded in your desktop, mobile, and edge apps — fully offline, no daemon, no cloud, no telemetry. Link one stable C library and get chat, streaming, vision, tool-calling agents, and schema-constrained JSON from one API. Free for non-commercial use; transparent pricing when you ship. Built by Luxand — on-device recognition SDKs since 2007, shipping inside thousands of applications.

Ship on-device AI without maintaining the cross-platform inference stack yourself.

Download Releases — SDK packages for every platform · the Luxand LLM Studio demo app
Docs Developer Manual — getting started · concepts · full API reference
Platforms Android · iOS · Windows · Linux · macOS · ARM / Raspberry Pi
Models any GGUF model — a curated, tested list (Qwen · Gemma · Granite · LFM · …); download or bring your own
Backends CPU · Metal · CUDA
Wrappers C++ · Kotlin/Java · Swift · .NET · Python
Engine pinned llama.cpp, behind a stable C ABI

Quickstart

From a fresh checkout to a running local model — three commands:

./download-sdk.sh && ./fetch-model.sh          # stage the SDK and the Qwen3.5-2B starter pair (model + vision projector)
cmake -S samples/c -B build/c && cmake --build build/c
./build/c/luxllm-c-basic models/Qwen3.5-2B-Q4_K_M.gguf

Every download is a complete pair — text, tools, and image input work out of the box. ./fetch-model.sh --quality selects the stronger Qwen3.5 4B pair; ./fetch-model.sh --gemma selects the pinned Gemma 4 E2B Q4_0 conversion plus Luxand's vision-only Q8_0 build of its projector (227 MB instead of 987 MB). The projectors are Luxand's Q8_0 requantizations of the originals; the license and modification notice download beside them. All three models are provided under the Apache License 2.0.

(On Windows: run download-sdk.bat and fetch-model.bat — double-click works — then build with CMake from a Developer PowerShell; the Visual Studio generator is multi-config, so add --config Release to the build and run from build\c\Release\. Prefer manual? Grab the SDK package from Releases and unzip it into ./sdk/.)

Windows tip: if you downloaded this repo as a ZIP, right-click the ZIP → Properties → Unblock → OK before extracting — otherwise Windows marks the extracted scripts as internet-downloaded and shows a "publisher unverified" prompt when you run them. Already extracted? The same right-click → Unblock works on download-sdk.bat and fetch-model.bat — or simply click Run on the prompt: it is Windows' standard notice for files from the internet, not an error.

Just want to try it, no code? Grab Luxand LLM Studio — the free desktop chat app for Windows, macOS, or Linux — from Releases, pick a GGUF model, and chat. It bundles the app runtime and SDK; the platform requirements below still apply. Its source lives in demos/.

Add it to your app (C++)

Point CMake at the staged SDK:

find_package(LuxandLLMSDK CONFIG REQUIRED)                        # cmake -DCMAKE_PREFIX_PATH=./sdk
target_link_libraries(my_app PRIVATE LuxandLLMSDK::LuxandLLMSDKCpp)   # C++ wrapper target

A complete chat — load, converse, print — is the whole program:

#include "LuxandLLMSDK.hpp"
#include <cstdio>

int main() {
    luxllm::Model   model("models/Qwen3.5-2B-Q4_K_M.gguf");
    luxllm::Engine  engine  = model.createEngine("ContextTokens=2048");
    luxllm::Session session = engine.createSession();

    session.add(luxllm::Role::System, "Answer in one concise sentence.");
    session.add(luxllm::Role::User,   "Name one practical use for local LLM inference.");

    luxllm::Result reply = session.generate("MaxTokens=512");
    std::printf("%s\n", reply.text.c_str());   // RAII frees everything, in order
    return 0;
}

Streaming is the same call with a pull-friendly callback (no background threads, no ABI callbacks):

session.generate("MaxTokens=512", [](std::string_view chunk) {
    std::printf("%.*s", (int) chunk.size(), chunk.data());
    return true;                               // return false to stop early
});

That's the core. The next section shows the pieces real apps need — all on the same Model → Engine → Session object model.

What you can build

Tool-calling agents — OpenAI-style tools in, parsed calls out, with typed argument accessors (no hand-parsing model JSON):

luxllm::Result phase = session.generateWithTools("ToolChoice=Required", toolsJson);
for (int i = 0; i < (int) phase.toolCalls.size(); ++i) {
    double a = session.toolArgDouble(i, "a");          // typed, validated
    double b = session.toolArgDouble(i, "b");
    session.addToolResult(i, std::to_string(a + b));   // feed the result back
}
luxllm::Result answer = session.generate("MaxTokens=512");

Schema-constrained JSON — pass a JSON schema; generation is grammar-constrained to it:

luxllm::Result r = session.generate("MaxTokens=512", jsonSchema);   // schema-shaped output

Many conversations, one engine — independent sessions batch together in a single decode. You create sessions and step the engine; there is no llama_batch, no seq_id, no logits bookkeeping:

luxllm::Engine engine = model.createEngine("MaxParallelSessions=4");
for (auto& s : sessions) { s.add(luxllm::Role::User, prompt(s)); s.start("MaxTokens=512"); }
for (bool more = true; more; ) { more = engine.step(); /* drain each: s.readText(chunk) */ }

Also included: vision (image message parts — a vision model loads its matching projector file via MultimodalProjectorFile=…), reasoning-trace extraction for models that support it, and a drop-in OpenAI-compatible local server with a chat web UI (next section).

The same six example programs, written both ways — with the SDK and directly on llama.cpp — took ~3× less code with the SDK. Runnable versions of every snippet above, both ways, are in the downloadable comparison package (self-contained; builds against the free SDK — run measure.py to reproduce the numbers yourself).

One API, five languages

The same object model ships as native wrappers over one stable C ABI:

Language Package Language Package
C / C++ find_package(LuxandLLMSDK) Kotlin / Java AAR (./sdk/luxllm-android-release.aar)
Swift XCFramework (./sdk/LuxandLLMSDK.xcframework) .NET / C# source wrapper + native lib (LUXLLM_NATIVE_PATH)
Python pip install luxand-llm==1.0.1

Every language has a runnable sample against the same staged ./sdk/:

cmake -S samples/cpp -B build/cpp && cmake --build build/cpp

python -m pip install -r samples/python/requirements.txt
python samples/python/basic.py models/Qwen3.5-2B-Q4_K_M.gguf

dotnet run --project samples/dotnet/Basic -- models/Qwen3.5-2B-Q4_K_M.gguf

swift run --package-path samples/swift/cli luxllm-swift-basic models/Qwen3.5-2B-Q4_K_M.gguf  # after ./download-sdk.sh --artifact apple

ADB=/path/to/adb samples/kotlin/chat-gui/run-on-device.sh models/Qwen3.5-2B-Q4_K_M.gguf

See samples/README.md for the complete sample overview.

Complete chat apps included

Beyond the one-lesson-per-file samples, three full GUI chat apps implement one "advanced chat" spec — streaming with the transcript contract, a thinking toggle (gated by the model's SupportsThinking readback), a current_date tool through a streaming tool loop, and image input gated by SupportsImageInput:

App Platform Stack
samples/kotlin/chat-gui Android Jetpack Compose; 📷 camera + 🖼 gallery input
samples/swift/chat-gui macOS + iOS from one Xcode project SwiftUI; 📷 camera + 🖼 gallery on both
samples/dotnet/Chat.Gui Windows / macOS / Linux Avalonia (.NET)

Each is small enough to read in one sitting and structured to be lifted straight into your own product. For these chat apps, the recommended model is the Gemma-4-E2B vision pair (./fetch-model.sh --gemma) — the phone-proven pair. The Qwen3.5 pairs work too, images included: their hybrid architecture reuses earlier turns through the engine's context checkpoints (on by default).

Take a sample into your own project

Every sample is written to be ejected — a single source file with zero references to this repo. The per-language recipes (what to copy, how to reference the wrapper and the native SDK) are in samples/README.md. In every case, keep the activation call (ActivateLibrary("") = keyless Community mode) and replace the empty string with your license key when you ship.

Local OpenAI-compatible server + chat UI

One call starts a local HTTP server on the loaded model — in-process, fully offline, embedded in the SDK library itself (LuxLLM_StartServer in C; exposed by every wrapper). It runs on every platform — including Android and iOS — so an app already written against an OpenAI-compatible API can go fully local without rewriting its client code: start the server, change the base URL.

No code required to try it: every desktop SDK package ships the server prebuilt as bin/luxllm-server — its complete source is the samples/cpp server sample (luxllm-cpp-server), so the binary you run is also the integration example you can read and fork:

./sdk/bin/luxllm-server --model models/gemma-4-E2B-it-Q4_0.gguf   # ./fetch-model.sh --gemma; Gemma keeps long chats fast

Point any existing OpenAI client at http://127.0.0.1:8080/v1, or open http://127.0.0.1:8080/ for the built-in chat web UI. /v1/chat/completions covers streaming, tool calls, JSON-schema response_format, and image input on multimodal models. The scripted equivalent is samples/python/server.py; the full endpoint and parameter reference is the embedded-server section of the Developer Manual.

Requirements

Target Minimum platform CPU / ABI Additional runtime
Linux x86-64 glibc 2.28+ AVX2 libgomp.so.1
Linux ARM64 baseline (linux-aarch64-armv8-a, linux-aarch64-armv8-a-norepack) glibc 2.28+ ARMv8-A; Raspberry Pi 4 compatible None beyond the base OS
Linux ARM64 optimized (linux-aarch64-armv8.2-dotprod-fp16) glibc 2.28+ ARMv8.2 dotprod/FP16; Raspberry Pi 5 / Cortex-A76+ None beyond the base OS
Windows x64 Windows 10 22H2 or Windows 11 AVX2 No VC++ Redistributable required for SDK/server
macOS macOS 13.3+ Apple silicon None beyond the OS
iOS iOS 16.4+ arm64 device and simulator None beyond the OS
Android Android 7.0 / API 24+ arm64-v8a (ARMv8.2 dotprod/FP16), x86_64; 16 KB-page compatible None beyond Android

The exact package names, CUDA driver floors, and redistribution requirements are in the Developer Manual.

  • C — the plain-C ABI header (LuxandLLMSDK.h); C++ wrapper — C++17 or newer
  • Python 3.9+ (pure-Python wrapper) · .NET 8+ (wrapper multi-targets netstandard2.0 / net8.0 / net10.0; samples use net8.0 LTS) · Swift 6 toolchain (SwiftPM)
  • Android development — JDK 17+ and the Android SDK/NDK

GPU or CPU

The engine library itself has no GPU dependency. On Linux/Windows, use the -cuda package variant — the same engine plus a loadable CUDA plugin library beside it (needs a Turing-or-newer NVIDIA GPU and a current driver; no CUDA toolkit install — the runtime is bundled with the plugin, see the manual's Distribution chapter). download-sdk.sh / download-sdk.bat pick it automatically on machines with an NVIDIA driver (--cpu opts out). Device=Auto uses the GPU when present and falls back to CPU; a CPU-only machine runs the same binaries untouched. On Apple platforms, Metal is built in — no separate package.

Two rules keep CUDA deployments out of trouble:

  • Never link the -cuda library. Applications always link the base LuxandLLMSDK library only; the -cuda file is a runtime plugin with no linkable LuxLLM_* symbols.
  • The plugin is found next to the base library. At startup the SDK loads the plugin from the exact directory the base library itself was loaded from (in the shipped layout: sdk/bin/) — keep the two files together wherever your deployment puts them. Next to the executable or on the system path does nothing.

To see which backend is active at runtime, read engine.parameter("Backend") (C: LuxLLM_GetEngineParameter(..., "Backend", ...)) — it returns CPU, CUDA, or Metal.

Threading model

A session is used by one thread at a time; different sessions of one engine on different threads batch together automatically, and LuxLLM_StopGeneration is the any-thread cancellation path. The full contract is the Threading & Concurrency section of the Developer Manual.

Models

Runs any GGUF model — Qwen, Gemma, Granite, LFM, and more. We publish a curated, tested list — each model run through a per-model behavioral test suite (chat, streaming, tools, JSON-schema, vision, thinking on/off) against the exact engine build we ship — so you know what works before you download. Get the weights from Hugging Face (or bring your own); check each model's license before redistribution.

Which model? We measured.

The quickstart uses the Qwen3.5 2B pair to keep the first download small — 1.3 GB of model plus a 362 MB vision projector. For stronger chat and tool behavior, ./fetch-model.sh --quality downloads the Qwen3.5 4B pair (2.7 GB + 367 MB) highlighted below.

We benchmark open models on this SDK with the agentic subsets of the Berkeley Function-Calling Leaderboard (BFCL V4) — ~40,000 tasks scored, including the complete 600-case multi-turn suite for every model. Every candidate has ~4B or fewer active parameters: the class that decodes at usable speed on mass-market devices. Which model for which job:

Use case Model Agentic Score Download
The most capable
Apple-Silicon Macs, PCs with GPUs, 16 GB+ RAM
Qwen3.6 35B A3B (2-bit)
2-bit GGUF · UD-Q2_K_XL · reasoning
72.3% 12.3 GB
Vision at flagship quality
image input on the same hardware class
Gemma 4 26B A4B (2-bit)
2-bit GGUF · UD-Q2_K_XL · reasoning · vision
71.8% 10.5 GB
+1.2 GB mmproj
Mass-market phones & laptops
the best score per gigabyte in our table
Qwen3.5 4B
4-bit GGUF · Q4_K_M · reasoning · vision
67.0% 2.7 GB
+0.4 GB mmproj
Image chat on-device
smallest vision-capable Gemma — current 2026-07-16 conversion
Gemma 4 E2B
4-bit GGUF · Q4_0 · reasoning · vision
51.1% 2.8 GB
+0.2 GB mmproj
The quickstart starter
the ./fetch-model.sh starter pair — when every megabyte counts
Qwen3.5 2B
4-bit GGUF · Q4_K_M · reasoning · vision
50.5% 1.3 GB
+0.4 GB mmproj
Under a gigabyte
Raspberry Pi-class boards, older phones
Granite 4.0 H 1B
4-bit GGUF · Q4_K_M · non-reasoning
38.1% 901 MB

Internal run of a BFCL V4 subset (July 2026) on the Luxand LLM SDK release runtime — subset scores differ from full-suite numbers, so compare models within this table rather than against the official leaderboard. Every score is text-only tool calling; vision quality is a separate model capability. Full table, sub-gigabyte models, charts, and scoring notes: luxand.com/llm-sdk/benchmarks.

Repository layout

include/        Public C ABI header
wrappers/       Source-form wrappers (C++, Kotlin/Java, Swift, .NET, Python)
samples/        Per-language lesson samples + the GUI chat apps
                (kotlin/chat-gui, swift/chat-gui, dotnet/Chat.Gui), all against ./sdk/
demos/          Luxand LLM Studio, the desktop chat app (.NET/Avalonia) — prebuilt on Releases, source here
sdk/            Your downloaded SDK artifacts (gitignored)
models/         Your GGUF model weights (gitignored)

The repo is source only: SDK binaries come from Releases, model weights from Hugging Face (or bring your own). The ./sdk/ convention is shared by every sample: desktop zip (include/, bin/, cmake/), Apple XCFramework, or Android AAR.

Licensing

  • Community — free. Key-less, all features, all platforms. For evaluation, development, and non-commercial use on your own devices — distributing an app, even a free one, requires a commercial license. No signup, no phone-home. See LICENSE.txt. Building something open-source or non-commercial you want to distribute? Contact us — we issue free keys at our discretion.
  • Commercial — a per-product license to ship in a commercial app, with support and updates. Transparent, published pricing — no sales call to start; buy online and your license key is issued instantly. Perpetual/OEM options for larger deployments. See pricing & buy a license →

Applying a commercial key is one call: LuxLLM_ActivateLibrary("<key>") in C — the same call in every wrapper. Verification is offline (a signed key, checked locally — no callback, no metering); without a key the SDK simply runs in Community mode.

Engine & third-party

The engine is llama.cpp. Shipping it is our job. Luxand LLM SDK is built on a pinned llama.cpp build, wrapped behind a stable C ABI. llama.cpp, ggml, and other bundled components are covered by THIRD-PARTY-NOTICES.txt.

FAQ

What models can I run? Any GGUF model — see Models above for the curated, tested list; you download the weights (Hugging Face, etc.) or bring your own.

Isn't on-device inference slow? On commodity x86 CPU it's for background, short-output work — extraction, classification, summarization, structured JSON, tool routing; long interactive chat wants a GPU. On Apple Silicon (Metal) and CUDA GPUs it's interactive. Capability benchmarks are published at luxand.com/llm-sdk/benchmarks.

We already use an OpenAI-compatible API — can we switch? Yes — start the embedded server in-process (it runs on every platform, mobile included) and point your existing client at 127.0.0.1. Chat completions with streaming, tool calls, JSON-schema response_format, and image input are covered. No client rewrite.

Isn't this just a llama.cpp wrapper? The engine is llama.cpp — the proven, de-facto standard engine for GGUF models — pinned to an exact, validated build. Any GGUF model runs; there's no proprietary runtime. What we build is everything shipping requires around the engine — and if you've built on llama.cpp directly, you know that work: cross-platform builds, the ABI, five language bindings, tool-calling / JSON-schema / reasoning that hold up across models, per-model validation, packaging, support. That's the SDK. For C/C++, it's a header and one .so to link — no engine to compile, no build graph to maintain. The same six example programs, written both ways: ~3× less code with the SDK (C++ 276 vs 908 lines; C 533 vs 1306) — reproduce it yourself.

If I embed llama.cpp directly, what am I signing up to build? Mostly integration code — but some of it is real machinery. One example: prompt caching. Each chat turn resends the whole conversation. Without caching, the engine re-processes all of it before answering — every earlier message, and every attached image again through the vision encoder. The fix is to keep the KV cache across turns and evaluate only what changed. Doing that correctly is real work: match the cached prefix token by token, match images by content hash, find the exact point where re-evaluation must start. llama-server ships this logic inside the server application. The llama.cpp library itself does not provide it (as of llama.cpp b9967), so an app that embeds the library writes it from scratch. The SDK provides it behind the C ABI on every platform — including iOS and Android, where a server sidecar isn't an option. The payoff is biggest on CPU and phones: without reuse, a long vision chat can stall for seconds before every reply.

Why not just run Ollama or LM Studio? They're a desktop daemon/GUI for running models on your own machine — not embeddable in a shipped app (no in-process library, no mobile). We're for shipping inference inside your product.

Why not vLLM or TGI? Server-side GPU serving at scale — a different deployment. We're on-device / embedded.

Can I ship a free app without a license? No — any distributed app needs a license key. Keyless Community mode covers development, evaluation, and non-commercial use on your own devices. Building something open-source or non-commercial you want to distribute? Contact us — we issue free keys at our discretion.

Does it phone home? No. No telemetry, no license callback, no metering — ever.

Links

About

Cross-platform SDK for private, offline LLM inference in desktop, mobile, and edge apps.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages