Skip to content
taweiliPublic

About

The agent runtime is a living Lisp image. The harness is not scaffolding around intelligence — it is the operating environment in which intelligence is a first-class, inspectable, hot-patchable process.

Topics

Resources

Stars

13 stars

Watchers

0 watching

Forks

Repository files navigation

sexpr — an Agent OS in Common Lisp

sexpr

The agent runtime is a living Lisp image. The harness is not scaffolding around intelligence — it is the operating environment in which intelligence is a first-class, inspectable, hot-patchable process.

Contents

Quickstart

Get a session running in three steps. For the design thesis, skip to § The design in five sentences; for building from source, see § Development setup.

1. Get the binary

Pre-built release? Drop sexpr on your PATH. Otherwise build once:

git clone https://github.com/quasi/sexpr && cd sexpr
make build          # runs the test suite first; ~64 MB, self-contained
./sexpr --help      # verify

2. Configure a model

Local OpenAI-compatible server (Ollama, llama-server, vLLM, LM Studio, an internal gateway):

export SEXPR_BASE_URL="http://localhost:6969/v1"
export SEXPR_MODEL="Qwythos-9B-v2"

Hosted provider:

export SEXPR_PROVIDER="openai"   # :anthropic | :openai | :gemini | :openrouter
                                  # :ollama | :openai-compatible
export SEXPR_MODEL="gpt-4o-mini" # optional; provider default otherwise
export OPENAI_API_KEY="sk-..."   # or ANTHROPIC_API_KEY, GEMINI_API_KEY, ...

The wrapper ./sexpr.sh sets local-dev defaults (OpenAI-compatible server on localhost:6969, model Qwythos-9B-v2, all four capabilities) and execs ./sexpr under rlwrap:

./sexpr.sh                                        # default goal + all caps
./sexpr.sh --goal "summarize this repo"           # explicit goal
SEXPR_CAPABILITIES=fs-read ./sexpr.sh             # read-only session
SEXPR_MODEL=OtherModel ./sexpr.sh                 # different model
./sexpr.sh --help                                 # wrapper help + env vars

3. Chat

./sexpr --goal "You are a terse pair programmer."
./sexpr --provider anthropic --model claude-3-5-sonnet-latest --goal "..."
./sexpr --capability fs-read --goal "read-only session"

Slash commands and the full flag list are in § Chat interface. To resume a session:

./sexpr --goal "pair programmer" --load my-session.sexp

The Listener reads stdin, so piping works for CI or batch runs:

printf '/help\n/exit\n' | ./sexpr --goal "verify"

The big idea

In 1985, Symbolics shipped the Genera operating system on Lisp Machines — single-user computers whose entire operating environment (OS kernel, GUI, development tools, application runtime) was written in a dialect of Common Lisp. The user interacted with the system through a listener (an interactive REPL), tools were functions, the runtime state was the image (the heap), and code could be redefined live without restarting. By 2025, the AI industry has reinvented most of these ideas under new names: agent harnesses (the listener loop), tool calling (the functions), context engineering (the image as sole state), subagents (lightweight processes in one heap), and MCP servers (the services of Chaosnet).

But the AI industry built all of this in Python and TypeScript, bolted around a stateless model called over HTTP. sexpr inverts this. Instead of wrapping a model in Python, the agent runtime is a living Lisp image on SBCL — the modern descendant of the Symbolics Lisp Machine's runtime. The Lisp machine already solved most of what agent engineering is rediscovering.

Agent-harness concept (2025) Lisp machine concept (1985)
"The transcript is the only state" The image is the only state
Context engineering / compaction GC + ephemeral memory + presentations with history
Tool use / function calling Calling functions. Literally.
Skills with progressive disclosure Systems, packages, autoloading, Document Examiner
MCP servers Processes / services in the same address space
Approvals & permissions Condition system + restarts
Subagents Lightweight processes sharing one heap
Agent OS Genera

sexpr's job is to close the loop: rebuild these ideas around an LLM as a resident process of the image, not its master.

The design in five sentences

  1. The agent loop is eval, made explicit. model-step returns an s-expression, not JSON; tools are Lisp functions wearing metadata; a tool call is a form.
  2. The transcript is a first-class object, not a list of strings. It holds live object references (presentations) and round-trips through print/read.
  3. Compaction is garbage collection. Old transcript ranges reify into summary nodes; the recent window is the nursery.
  4. Approvals are conditions, permissions are restarts. Non-unwinding semantics mean a denied effect leaves the agent's stack intact — the human inspects the exact state and resumes precisely there.
  5. Skills are systems. A skill is an ASDF-loadable directory with a SKILL.md, a SKILL.sexp, and optional scripts. Progressive disclosure becomes load.

Architecture sketch

+-------------------------------------------------------------+
|  sexpr image (one address space, SBCL)                      |
|                                                             |
|  kernel process: scheduler, GC, capability table           |
|  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         |
|  │ agent proc  │  │ agent proc  │  │ human proc  |         |
|  │ (goal loop) │  │ (subagent)  │  │ (listener)  |         |
|  │ transcript  │  │ transcript  │  │             |         |
|  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘         |
|         └────────────────┴────────────────┘                |
|              shared heap: tools, skills, world state        |
|  model endpoint processes: sockets to LLM providers        |
|  MCP bridge process (for legacy / foreign tools)           |
+-------------------------------------------------------------+

Agents are threads with an owner, a goal, a transcript, a capability set, and a budget. The model is a socket. Budgets are dynamic variables. Interrupts come for free because everything is a thread in one image — a human can drop into any agent's debugger, patch a function, and let it continue.

Operation manuals

docs/ holds the operation manuals — practical references for running the binary and the TUI, distinct from the design documents in notes/.

File What it covers
docs/sexpr.md Operating the sexpr binary and the ./sexpr.sh wrapper: model configuration, chat flags and slash commands, the REPL, examples, and Make targets. The how-to-run reference.
docs/sexpr-tui.md Operating ./sexpr-tui.sh: the two channels, flags, environment, layout, keybindings, lifecycle, and troubleshooting. The how-to-run-the-TUI reference.

Repository

sexpr.asd              ASDF system definition (:sexpr + :sexpr/tests)
sexpr-tui.asd          the TUI client system (:sexpr-tui, cl-tuition)
src/
  package.lisp         :sexpr package
  sexpr.lisp           identity + `hello' entry point; test-op method
  provider/            the "Ivory" layer — LLM providers as sockets
    package.lisp       :sexpr.provider package
    provider.lisp      provider-call GF + configuration (cl-llm-provider transport)
  transcript/          events, print/read round-trip, the GC substrate
  kernel/              the agent: budget, transcript, model-step, run-until-finished
  agents/              the multi-agent registry (*agent-table*, subagent registry)
  cli/                 the Listener: chat loop, slash commands, SIGINT, argv/main/build (R015: never :use :sexpr.provider)
  tools/               the tool registry (register-tool!, derive-schema, dispatch)
  builtins/            the registered tools (read-file, write-file, edit-file, shell, lisp)
  sandbox/             the restricted-read eval sandbox (eclector, locked package, eval timeout)
  repl/                the developer REPL (escapes chat via /repl; standalone via `sexpr.sh repl`)
  tui/                 the image-side TUI package (sexpr.tui: swank + push server, snapshot/delta)
  tui-client/          the client process (cl-tuition; projections, never live objects)
sexpr.sh               local-dev wrapper (rlwrap + provider/model/capability defaults)
sexpr-tui.sh           TUI launcher (starts the image if needed, runs the client)
qlfile                 Qlot deps (cl-llm-provider, eclector, rove, cl-tuition)
Makefile               `make hello|test|build|chat|verify|sexpr-tui|verify-wrapper|clean`
tests/                 the rove test suite (cli, kernel, provider, transcript, tools, sandbox, repl, tui)
tools/                 diagnostic probes + wrapper checks
examples/              standalone .lisp examples (run via `sexpr.sh example`)
docs/                  operation manuals (see § Operation manuals)
notes/                 design notes + research base (see below)
  sexpr.md             — thesis, layer-by-layer mapping, open questions
  agent.md             — 2025 harness research (context engineering, skills, MCP)
  symbolics-lisp.md    — Symbolics / Genera / Dynamic Windows lineage
  sbcl.md              — SBCL, Quicklisp, hot-patching, condition system, save-lisp-and-die
  sbcl-libs.md         — CL library survey mapped to the sexpr design
  tui.md               — the remote TUI: cl-tuition client over Swank

Design documents

notes/ holds the research and design documents that motivated the project. notes/sexpr.md is the entry point; the others are the research base it's synthesized from.

File What it covers
notes/sexpr.md The main design doc. Thesis, layer-by-layer mapping of 2025 agent-harness concepts to 1985 Lisp-machine primitives, module-by-module plan, open questions. Start here.
notes/symbolics-lisp.md History and architecture of Symbolics Lisp Machines (LM-2, 3600 family, XL, MacIvory, Open Genera), the Genera OS, CLIM, Dynamic Windows, Document Examiner, and the patch system. The "what we're modernizing" reference.
notes/agent.md Survey of 2025–2026 AI agent harness architecture: the agent loop, context engineering (write/select/compress/isolate), tool dispatch, MCP, subagents, skills with progressive disclosure, and sandboxing. The "what we're translating" reference.
notes/sbcl.md SBCL and Quicklisp deep-dive: the live image, hot redefinition, the condition/restart system, save-lisp-and-die for whole-image snapshots, and Quicklisp as a dependency/skill distribution mechanism. The "our runtime substrate" reference.
notes/sbcl-libs.md Library survey: maps the sexpr design onto existing Common Lisp libraries (ASDF, UIOP, cl-llm-provider, etc.), noting availability and fit. The "what already exists" reference.
notes/tui.md The remote TUI design: a cl-tuition client over a Swank wire, with a chat pane (Converse), an agent-management console (Peek), and a REPL pane (Listener) sharing a focused agent. Where sexpr.tui lives, what :view :tui projects, and the open tensions. Forward-looking — not the research base the others are.

How to read them

  1. sexpr.md first — it synthesizes the others and states the thesis. Read its §0 ("Thesis") for the core argument and §11 ("Sources / lineage") for how it draws on the other three docs.
  2. symbolics-lisp.md if you want the historical/architectural background on what we're modernizing.
  3. agent.md if you want the 2025 state of the art that we're translating into Lisp.
  4. sbcl.md and sbcl-libs.md if you want the implementation substrate and what's already available in the CL ecosystem.
  5. tui.md for the remote interface design — where it lands in the image (sexpr.tui), what the client holds (projections, never live objects), and how the three panes map onto Genera's Converse / Peek / Listener.

Development setup

Prerequisites: SBCL and Quicklisp. One-time bootstrap:

curl -O https://beta.quicklisp.org/quicklisp.lisp
sbcl --load quicklisp.lisp --eval '(quicklisp-quickstart:install)' \
     --eval '(ql:add-to-init-file)'

Then register this project with Quicklisp (one-time, from a checkout):

ln -s "$PWD" ~/quicklisp/local-projects/sexpr

sexpr depends on cl-llm-provider (the unified LLM transport — Anthropic, OpenAI, Gemini, Ollama, OpenRouter). It is not (yet) in the Quicklisp dist, so vendor it into local-projects too:

git clone https://github.com/quasi/cl-llm-provider \
  ~/quicklisp/local-projects/cl-llm-provider

Run the smoke test:

make hello
# or, directly:
sbcl --load ~/.sbclinit \
     --eval '(ql:quickload "sexpr")' \
     --eval '(sexpr:hello)'

Load into an interactive session for development:

make load
# type: (sexpr:hello)

Run the test suite (rove; lives in tests/, declared as the separate :sexpr/tests system):

make test

The TUI client (:sexpr-tui) depends on cl-tuition, which is in the Quicklisp dist — no vendoring needed. ./sexpr-tui.sh quickloads it at runtime, so there is no separate TUI binary to build.

Model provider

The model is a socket (notes/sexpr.md §2). src/provider/ is the "Ivory" layer: a single generic function, sexpr.provider:provider-call, that returns s-expression transcript nodes (:content, :tool-calls, :model, :usage, :finish) — never raw provider objects. cl-llm-provider is the concrete transport; tool execution stays in the sexpr kernel (transport only, by design).

Configure via environment variables (API keys are read directly by cl-llm-provider):

export OPENAI_API_KEY="sk-..."        # or ANTHROPIC_API_KEY, OPENROUTER_API_KEY, ...
export SEXPR_PROVIDER="openai"        # :anthropic | :openai | :gemini | :ollama
                                       # :openrouter | :openai-compatible
export SEXPR_MODEL="gpt-4o-mini"      # optional; defaults to the provider's default
export SEXPR_BASE_URL="http://..."    # OpenAI-compatible endpoint (default:
                                       # https://api.openai.com/v1)
export SEXPR_CAPABILITIES="fs-read"   # comma-separated; grants additive with
                                       # the built-in default :fs-read

Then from Lisp:

(ql:quickload :sexpr)
(in-package :sexpr.provider)
(configure-provider)                    ; picks up the env vars above
;; or, explicitly:
(configure-provider :provider :anthropic :model "claude-3-5-sonnet-latest")

*model-endpoint* is realized lazily on the first provider-call (so configuration never fails just because a key isn't set yet). Swap providers by rebinding — (let ((*model-endpoint* (make-provider :ollama))) ...) — the central design invariant.

Chat interface

Full flag list, slash commands, the REPL, examples, and Make targets: docs/sexpr.md (operation manual).

sexpr.cli is the Listener — the terminal front end. The image is the state; the transcript is its conversation slice; the model is a socket reached only through the kernel, never imported into the Listener (R015). Each line you type becomes a transcript event; run-until-finished folds one bounded model turn; new events render back to the screen.

./sexpr --goal "summarize the design"

Ctrl-C aborts a running turn and returns to the prompt — the partial transcript stays readable, no rollback (R013). A second Ctrl-C at an idle prompt exits.

Slash commands

Command Effect
/exit quit the session
/quit quit the session
/help list the commands
/transcript print the whole transcript
/system TEXT set the system prompt (persona)
/save FILE write the session transcript to FILE
/load FILE load a transcript from FILE, replacing the session
/retry pop the last model reply and re-run one turn
/repl escape into a CL dev REPL (,exit to return)

/save and /load round-trip through the transcript's own print/read (with-standard-io-syntax, *read-eval* nil) — no separate serializer (R012).

Flags

./sexpr --goal "TEXT" [--load FILE] [--provider NAME] [--model NAME]
        [--capability NAME] [--help | -h]
Flag Meaning
--goal TEXT the agent's goal (required to enter the loop)
--load FILE start from a saved transcript
--provider NAME configure the provider transport (qualified call)
--model NAME configure the model name
--capability NAME grant a capability to the agent (repeatable; a
comma-separated list is also accepted). Grants are
additive with the built-in default :fs-read. The
known set is fs-read, fs-write, process,
lisp-eval.
--help, -h print usage and exit (never enters chat)

--provider/--model reach the transport by the single qualified sexpr.provider:configure-provider reference; the Listener never imports the transport package (R015).

Deliberately out (R016)

History, streaming, multiline input, and tool dispatch are not in the Listener. Line history and editing come from an external tool like rlwrap — the wrapper ./sexpr.sh does this automatically, or wrap the binary manually:

./sexpr.sh --goal "..."         # wrapper: rlwrap + local-dev defaults
rlwrap ./sexpr --goal "..."     # manual wrap

No readline, no threading, no streaming — the transcript is the state, and each turn is one bounded fold over it.

REPL

sexpr.repl is an unrestricted Common Lisp REPL that shares the live image — the same agent, transcript, tools, and provider state the chat loop is mid-conversation with. It is a developer surface, not a sandboxed one: everything in :sexpr is in scope.

Escape from chat: /repl

From inside a chat session, /repl drops into the REPL with *agent* bound to the live agent. ,exit returns to the chat loop; the conversation resumes with any redefinitions you made now in effect:

you: make me a todo app
model: <calls edit-file, error in the result>
you: /repl
> (agent-transcript *agent*)           ; inspect the live state
> (trace sexpr.tools:perform-tool)
> (defun my-edit-file ...)             ; hot-redefine the buggy tool
> ,exit
you: /retry                            ; re-run the last turn with the new def
model: <now succeeds>

*agent* is bound by the REPL around its loop body — reach the transcript via (agent-transcript *agent*), the persona via (agent-system *agent*). The REPL shares the chat loop's input/output streams, so ,exit (not EOF) is the escape — EOF would also end the chat.

Standalone: ./sexpr.sh repl

Outside a chat session, ./sexpr.sh repl enters the REPL with no agent (*agent* is NIL). Useful for loading examples, defining tools from scratch, or driving the provider directly:

./sexpr.sh repl
> (load "examples/02-tools.lisp")
> (sexpr.provider:provider-call nil '((:role "user" :content "hi")))
> ,exit

The REPL never calls the model on its own — no provider endpoint or API key is needed. Reach it by evaluating a provider-call form yourself.

REPL conventions

  • One form per prompt. ,exit collides with the CL unquote reader macro, so the line is read whole and string-checked for a leading comma before read-from-string. Multiple forms on one line all eval and print; for multi-line forms use (progn ...) or (load "file").
  • , meta-commands. Only ,exit is defined; an unknown ,foo prints a notice and continues (never reaches the reader).
  • Errors never crash the loop — reader errors, eval errors, and a Ctrl-C mid-form are caught, printed, and re-prompted. Ctrl-C aborts the current form only (no two-press idle-exit dance — ,exit is the explicit escape).
  • Package. The REPL starts in :sexpr.repl, an aggregating package that :uses :sexpr.transcript, :sexpr.kernel, :sexpr.tools, and :sexpr.sandbox — so make-agent, agent-transcript, render-events, eval-in-sandbox, define-tool are unqualified. Switch with (in-package :sexpr.kernel).
  • Nil is suppressed. A form returning a single NIL prints nothing (SBCL convention), so side-effecting (format t ...) calls don't echo a trailing NIL.

sexpr-tui

Remote TUI

Full flags, environment, layout, keybindings, and troubleshooting: docs/sexpr-tui.md (operation manual). Design: notes/tui.md.

The TUI is a separate process (:sexpr-tui, cl-tuition) that renders a projection of a running image. It does not carry the kernel — the two sides stay apart so the client never becomes a second agent society; notes/tui.md is the design.

./sexpr-tui.sh brings up both halves:

./sexpr-tui.sh                 # start the image if needed, then run the TUI
./sexpr-tui.sh --agent Alpha   # same, starting on Alpha
./sexpr-tui.sh --no-server     # attach only; fail if nothing is listening
./sexpr-tui.sh --server-only   # bring the image up and leave it running
./sexpr-tui.sh --dry-run       # show the boot file and the client command

The TUI needs two channels, both owned by the image: swank on 4004 carries request/response (typed sexpr.tui RPCs over a real Swank server), and the push channel on 4005 carries the one-way delta stream. "An image is running" means "port 4004 accepts a connection": the wrapper adopts what is already there and starts nothing if the image is up, and it tears down only an image it started itself — an image you started by hand is left exactly as you left it. The push channel is optional at the client; the wrapper asks for it only when it is actually listening. The image comes up in its own SBCL, fed by a named pipe it holds open, and exits on EOF, so a crash never looks like a hang.

A fresh image is not an empty society. The boot file seeds one default agent whose goal is hello world — it exists to greet whoever opens the TUI. The client picks its default agent from the first snapshot row, so the greeter is what you land on; it runs like any subagent (worker thread, handle, transcript), streams its welcome over the push channel when the provider is configured, and finishes as :failed without one — the image comes up either way. The worker announces its status on the push channel as it starts and finishes, so the agent pane's row tracks it live — RUNNING while a turn is in flight, IDLE once the turn is done — and a chat note typed into a finished agent starts a fresh worker, so the note gets a turn of its own instead of sitting unanswered in the transcript. Its transcript is seeded with the same hello world user line, because the kernel sends the whole transcript to the model as messages and a goal-only call carries no user message — providers whose chat template requires one (llama-server's --jinja) refuse such a request. The image also honors the SEXPR_PROVIDER / SEXPR_MODEL / SEXPR_BASE_URL environment variables (see Model provider), exactly like the CLI does, so the local-dev flow greets out of the box.

Every knob is overridable, with no build step and no binary:

env var default
SEXPR_TUI_HOST 127.0.0.1 where the image listens
SEXPR_TUI_SWANK_PORT 4004 request/response channel
SEXPR_TUI_PUSH_PORT 4005 one-way delta channel
SEXPR_TUI_STATE_DIR /tmp/sexpr-tui boot file, log, pipe
SEXPR_TUI_READY_TIMEOUT 120 seconds to wait for the image
SEXPR_TUI_SBCL sbcl the SBCL binary
SEXPR_TUI_SBCLINIT ~/.sbclinit loaded first, image and client

Flags (--host, --swank-port, --push-port) win over the environment; every other flag is forwarded to sexpr-tui:main verbatim. The boot file is regenerated on every run at $SEXPR_TUI_STATE_DIR/server.lisp, so it can never point the image at a stale checkout, and the image log lands at $SEXPR_TUI_STATE_DIR/server.log — that is where a failed boot explains itself. make sexpr-tui is the same wrapper; make verify-wrapper runs the launcher checks without starting a single process.

Layout

The frame is laid out from the terminal size, not from fixed pane dimensions. layout-dimensions (in src/tui-client/view.lisp) derives every pane's width and height from the window, so the frame comes out exactly as wide and as tall as the screen at any size — a 200x45 terminal is used completely, and a 60x16 one fits without overflow.

size chat viewport REPL viewport agent pane
80x24 54 x 11 54 x 3 24 x 22
120x32 82 x 17 82 x 5 36 x 30
200x45 150 x 27 150 x 8 48 x 43
60x16 38 x 5 38 x 1 20 x 14

A resize event reflows every pane, growing or shrinking. The layout is total: a 0x0 or negative window degrades to the smallest usable panes rather than signalling.

Input focus

Tab moves key input between the chat and REPL inputs, and exactly one of them owns the keys at a time — the other is explicitly blurred. A textarea holding no text renders its placeholder without a cursor, so the focused input's border is drawn in cyan (*focus-color*) as the visible cue. Enter submits the focused input; Ctrl+C and Escape quit; while an approval is pending, Ctrl+G grants once and Ctrl+D denies.

Building the binary

make build produces a self-contained ./sexpr executable via sb-ext:save-lisp-and-die — no Quicklisp or SBCL needed at runtime. make build always runs make test first, so a broken suite never ships a binary:

make build       # ./sexpr at the repo root, ~64 MB
make chat        # sugar for: ./sexpr --goal "pair programmer"
make verify      # binary-level health gate: --help, scripted session,
                 # cross-process --load round-trip, R015 seam
make clean       # removes ./sexpr alongside the fasl/ql artifacts

make chat accepts CHAT_ARGS to override the default goal:

make chat CHAT_ARGS='--goal "summarize the design" --provider ollama --model llama3'

Sessions persist through the transcript's own print/read path — save from inside the loop, load at startup:

./sexpr --goal "pair programmer"
; inside the loop:
; /save my-session.sexp
; /exit

./sexpr --goal "pair programmer" --load my-session.sexp

A scripted session runs from a clean shell with no terminal:

printf '/help\n/exit\n' | ./sexpr --goal "verify"

Status

Design phase, with the model/transport layer wired. The cl-llm-provider dependency is installed and src/provider/ exposes the provider-call boundary (verified end-to-end: configuration → provider realization → HTTP → the provider's condition/restart error recovery). The design has been cross-checked against a concrete library survey (see notes/sbcl-libs.md) — most of the substrate exists in Quicklisp; the novel pieces still ahead are the transcript compactor, the token-budgeted presentation renderer, and a capability-restricted eval sandbox.

Open questions

  • Sandbox granularity: one heap = one fault domain. Genera accepted this; can sexpr afford it when the code-writer is an LLM?
  • Token cost of sexpr I/O: presentations need a :view :model rendering path.
  • Determinism & replay: model calls are nondeterministic; transcript + canned responses is the likely answer.
  • Multi-agent coherence: shared heap invites races; the world model needs transactional accessors, not naked globals.
  • How much LLM, how much symbolics? Keep proposal generation in the model; keep control flow symbolic.

License

GPL-3.0-or-later. See LICENSE for the full text.

New files should carry the SPDX header:

;;; SPDX-License-Identifier: GPL-3.0-or-later

(;;; for Lisp source; use <!-- or # as appropriate for other file types.)

About

The agent runtime is a living Lisp image. The harness is not scaffolding around intelligence — it is the operating environment in which intelligence is a first-class, inspectable, hot-patchable process.

Topics

Resources

Stars

13 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages