Skip to content
JacobHayesPublic

About

Weather your code into a fine protective patina, before production does.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Patina

Weather your code into a fine protective patina — before production does.

Patina runs a Linux x86_64 program many times, each run under a seed that decides thread order, the clock, randomness, and which disk, network and crash faults the program meets. The same seed gives the same run, so a bug that shows up once in a thousand runs becomes a seed you can replay as often as you like. Your program runs unmodified: no rebuild, no special linking, any language.

Patina is experimental. Linux x86_64 only. It needs bwrap (bubblewrap) and unprivileged user namespaces.

Changes in 0.8: UDP sockets between the run's processes, with seeded datagram loss, duplication and reordering; opt-in path swaps that expose check-then-use races (the guide); and replay requires the Patina build that made the run.

Install

curl -fsSL https://github.com/JacobHayes/patina/releases/latest/download/install.sh | sh

This installs one static binary, ~/.local/bin/patina (--prefix DIR for another place), and runs patina doctor, which checks the host and prints the fix for anything missing. The last line should read this host can run patina. On Ubuntu 24.04 the two fixes are:

sudo apt-get install -y bubblewrap
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

Root works too (the guest sees what it would for the user sudo was run by); in a container, see the guide for the options it needs. On a Mac or an arm64 machine, use a Linux x86_64 VM or container.

From a checkout instead, with Rust (rustup reads the pinned toolchain from rust-toolchain.toml): cargo install --locked --path crates/patina-cli.

First run

Give Patina any program and its arguments after --. examples/quickstart is a small Rust program that runs three jobs on threads and checks their results come back in job order.

git clone https://github.com/JacobHayes/patina /tmp/patina
cd /tmp/patina/examples/quickstart && cargo build
patina run -- target/debug/jobs fixed
ok (exit): the program exited 0
guest: exit 0
run directory: /tmp/patina/examples/quickstart/patina-out/run-0
replay: patina replay /tmp/patina/examples/quickstart/patina-out/run-0

The program's own output is in the run directory: patina-out/run-0/inc-0/stdout and stderr. Now run 20 seeds of the version with the bug:

patina sweep --seeds 1..20 -- target/debug/jobs
patina sweep: 20 runs, 14 findings in 1 group, 6 ok   (0.2s, 8 jobs)
outcomes: failure 14, ok 6

failure exit: panicked at src/main.rs:N:N: assertion `left == right` failed: results out of job order; exit 101
  14 of 20 runs   first: seed 1   replayed: identical
  rerun:  patina run --seed 1 -- /tmp/patina/examples/quickstart/target/debug/jobs
  replay: patina replay /tmp/patina/examples/quickstart/patina-out/sweep/seed-1

Each group is one bug: its signature, how many runs hit it, and the lowest seed among them. Patina already replayed that seed once and got the same run. patina run --seed 1 -- target/debug/jobs prints the outcome with the tail of the program's stderr; the run directory holds all of it:

cat patina-out/sweep/seed-1/inc-0/stderr
thread 'main' (2) panicked at src/main.rs:28:5:
assertion `left == right` failed: results out of job order
  left: [2, 1, 3]
 right: [1, 2, 3]

patina replay DIR runs the seed again, into DIR/replay, and compares every system call with the recorded one. Fix the code, rebuild, and run the same sweep command: a sweep directory (--out, default patina-out/sweep) holds one sweep, so a changed program or setting replaces it, and the same sweep with more seeds runs only the new ones.

What works

Any Linux x86_64 executable: static or dynamic, Rust, C, C++, Go, Python, shell scripts and the programs they start. Threads, fork, exec, pipes, files, timers, signals and TCP and UDP on localhost all run, decided by the seed. The program sees a fixed machine: 4 CPUs, one clock that starts at the same time every run, an empty writable /work as its working directory.

What it does not do: network sockets other than TCP and UDP, multicast and broadcast (the call fails with an error, and patina run lists it under refused:), DNS beyond /etc/hosts, connections to anything outside the run, and macOS or arm64 hosts. A call Patina does not model ends the run unsupported and names it. ARCHITECTURE.md lists the limits.

Deeper search: a program rebuilt with coverage flags and Patina's hook file also switches threads and moves the clock inside its own computation, so races with no system call in them are found. The guide has the details.

C, C++  clang -fsanitize-coverage=trace-pc,trace-loads,trace-stores -fno-sanitize-link-runtime, plus sdk/c/patina_compute.c
Rust    patina-dst with features = ["compute"]; the guide's RUSTFLAGS; cargo build --target x86_64-unknown-linux-gnu
Go      import _ "github.com/JacobHayes/patina/go/patina/compute"; CGO_ENABLED=0 go build -gcflags=all=-d=libfuzzer

Outcomes

Every run ends in one outcome; the exit status of patina follows from it.

Outcome Exit Meaning
ok 0 the program exited 0 and broke no invariant
violation 1 an SDK invariant (always) failed
liveness 1 every thread blocked, or the virtual time limit passed
failure 1 the program exited nonzero
crash 1 the program died on a signal
unsupported 2 the program did something Patina does not model, or never finished and filled the log to --log-limit (stop log-limit); the message names it
timeout 2 the wall-time limit passed
nondeterministic 2 a replay differed from its run: a Patina bug, please report it
internal 2 Patina failed, or the host refused a resource
vacuous 2 a sweep's fault or SDK check never had anything to act on
error 2 the command could not be carried out

Where next

  • examples/README.md: one bug each found by scheduling, by an SDK fault point, by disk faults, by crashes and by network faults, in Rust, Python, C and Go.
  • docs/GUIDE.md: sweeps, faults, crashes, the SDK, reading a result, replay, CI.
  • ARCHITECTURE.md: how it works and its limits.
  • patina --help, patina COMMAND --help: every flag of your build.

About

Weather your code into a fine protective patina, before production does.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages