kv_bench (Mace vs RocksDB vs Fjall vs SurrealKV)

Reproducible benchmark comparison of four embedded KV engines: mace, rocksdb, fjall and surrealkv.

Quickstart

1. Clone the repo:

git clone https://github.com/abbycin/kv_bench
cd kv_bench

2. Initialize the Python env (used by the report script):

./scripts/init.sh

3. Run the benchmark for each engine — pass any fast storage path (e.g. an NVMe mount) as the storage root; results are appended to ./scripts/benchmark_results.csv:

./scripts/mace.sh /path/to/nvme
./scripts/rocksdb.sh /path/to/nvme
./scripts/fjall.sh /path/to/nvme
./scripts/surrealkv.sh /path/to/nvme

4. Generate the comparison report:

./scripts/bin/python ./scripts/csv_to_html.py ./scripts/benchmark_results.csv

Output: ./scripts/benchmark_results.html.

Engines and Harnesses

Each engine has its own self-contained benchmark binary with an identical CLI (--path, --workload W1..W6, --threads, --key-size, --value-size, --prefill-keys, --warmup-secs, --measure-secs, --read-path, --durability, --result-file):

• mace: src/bin/mace_bench.rs → target/release/mace_bench (cargo build --release)
• RocksDB: rocksdb/main.cpp → rocksdb/build/release/rocksdb_bench (cmake)
• fjall: src/bin/fjall_bench.rs → target/release/fjall_bench (cargo build --release); W1-W6 only, merge workloads excluded (fjall has no merge operator upstream, see fjall-rs/lsm-tree#104)
• surrealkv: src/bin/surrealkv_bench.rs → target/release/surrealkv_bench (cargo build --release, crates.io surrealkv = "1"); W1-W6 only, merge workloads excluded (surrealkv has no merge operator)

The four harnesses share the same workload definitions, latency accounting and CSV schema, but each is written directly against its engine's API.

What Is Compared

• Comparison unit: rows with identical workload_id, threads, key_size, value_size, durability_mode, read_path
• Fairness rule: every workload (W1-W6) runs one GC/compaction pass after prefill and before warmup/measurement (mace enable_gc(), RocksDB compaction, fjall major_compact()), so engines are not compared with GC artificially disabled while reads may have to touch stale data
  • surrealkv is the exception, deliberately: version 1.0.0 exposes no manual compaction API (CompactionOperations is implemented on a crate-private type and is not re-exported), so its harness flushes and closes the tree after prefill, reopens it — the startup level check runs there — and then waits a fixed settle window (--gc-settle-secs, default 30 s, override with SURREALKV_GC_SETTLE_SECS). A run prints gc_settle_secs=; if that is 0, or shorter than the prefill's compaction needs, the row includes prefill compaction and is not comparable with the other engines on that axis. Nothing else about surrealkv's rows is affected: prefill, warmup, measurement and latency accounting are identical to the other harnesses.
• Throughput metric: workload-level ops (higher is better)
• Tail latency metric: workload-level p99_us (lower is better)
  • This is the workload-level p99 of all operations executed in that row, not per-op-type p99
• Operation semantics: every measured op is one transaction (write = begin + put + commit; read = snapshot or rw transaction), identical across engines, so ops is transactions per second
• Memory / backpressure strategy differs per engine: mace runs with enable_backpressure=true and a 4 GiB bucket cache, RocksDB with bounded write buffers (write_buffer_size=64MB × max_write_buffer_number=16, writes block when full) and a 5 GiB block cache, fjall with default background maintenance and a 4 GiB block cache, surrealkv with a 4 GiB block cache, its default (100 MB) memtable, write stalls on immutable memtables/L0, and value separation at the shared --blob-size threshold. All are on the same multi-GB memory scale.
• Isolation levels differ: mace runs SI, RocksDB OptimisticTransactionDB runs SI-style OCC, fjall OptimisticTxDatabase runs SSI (serializable snapshot isolation, strictly stronger than SI), surrealkv runs MVCC snapshot isolation with an async commit pipeline and OCC. Concretely, every fjall write txn tracks its read set (get marks reads) plus write set and validates on commit; a Conflict is counted as err_op with no retry, same accounting as RocksDB OCC conflicts
  • surrealkv validates a write txn's writes and its locked reads only: get_for_update registers a read for commit-time validation, a plain get does not. The harness uses get_for_update + set + commit on the update path (the same shape as RocksDB's GetForUpdate + Put + Commit), and a plain get on both read paths (begin_with_mode(ReadOnly) + get for snapshot, begin() + get + commit for rw_txn, which registers no read set and matches mace's begin + get + commit). Its conflict accounting therefore belongs next to mace/RocksDB, not next to fjall's SSI
  • Its durable mode is Durability::Immediate (per-commit fsync) and relaxed is Durability::Eventual; like RocksDB's write group and mace's sync registration, concurrent commits are batched into one group sync, so the durable rows measure amortized fsync latency, not one fsync per op at high thread counts
  • Measured conflict rates do not explain the throughput gaps: on W2 (zipf hotspot) max err_op is mace 2.74% / RocksDB 2.78% / fjall 2.28% (fjall aborts the least), and on W3/W4 all three sit at ~0.001%. So fjall's W3/W4 multi-thread collapse is write-path contention (journal/memtable/background compaction under SSI txns), not abort storms — but SSI bookkeeping/validation is still part of its per-txn cost
  • Reading guidance: treat W1/W2/small-value W5/W6 as engine-vs-engine; treat W3/W4 fjall numbers as SSI-transaction-path-vs-SI, not a pure storage-engine delta; surrealkv's W3/W4 rows carry the same get_for_update read validation as RocksDB's, which mace's Update path does not do

Why redb and sled Are Not Compared

Both were evaluated and rejected because their concurrency model cannot be compared fairly with mace/RocksDB on the W1-W6 workloads, all of which include multi-threaded concurrent writes.

• redb (v4.1.0) is a single-writer design (its design doc: "supports a single writer and multiple concurrent readers"; begin_write() blocks on a global write lock). Measured on this harness, its write-heavy throughput does not scale with threads at all (W4: ~3.2k ops/s at 1 thread, ~2.4k at 4), because every write serializes. Per-transaction commit cost is ~0.3-1ms, and relaxed mode additionally degrades quadratically over time: process_freed_pages_nondurable rescans the whole freed-page backlog on every commit. redb's own official benchmark confirms the same individual-write throughput (~1k txn/s), so these are native engine characteristics, not harness artifacts — they just are not comparable with concurrent-writer engines on the same workloads.
• sled (v0.34.7)'s transaction API (Tree::transaction) takes a process-global write lock too (concurrency_control::write()), so using transactions for the per-op write path would reproduce the same single-writer problem. Its plain Tree::insert is concurrent, but then the harness's per-op transaction accounting no longer matches the engine semantics; sled transactions also cannot scan (no rw_txn path), there is no manual compaction API (the shared GC/compaction fairness rule cannot be applied), and durability has no per-op sync switch (the default 500ms background fsync conflicts with the relaxed semantics). The comparison would not be apples-to-apples on any of the three axes the harness controls (transactions, durability, compaction).

Workloads

• W1: 95% read + 5% update, uniform distribution
• W2: 95% read + 5% update, Zipf distribution
• W3: 50% read + 50% update, uniform distribution
• W4: 5% read + 95% update, uniform distribution
• W5: 70% read + 25% update + 5% scan, uniform distribution
• W6: 100% scan, uniform distribution; throughput is counted by scan requests, not scanned key count

Raw CSV path: ./scripts/benchmark_results.csv

Scripts

• mace.sh / rocksdb.sh / fjall.sh / surrealkv.sh — run the W1-W6 matrix for one engine; first argument is the storage root, second optional argument is the result CSV (defaults to ./scripts/benchmark_results.csv)
• csv_to_html.py — single-page interactive HTML report (ops and p99 charts per workload, per key/value profile); colors are per key/value pair across engines, fill styles distinguish engines (mace solid, RocksDB hatch, fjall dot, surrealkv grid)
• compare_baseline.py — mace-vs-rocksdb ratio table from the CSV (pandas, use the scripts/bin venv)
• thread_points.sh — shared power-of-two thread points helper sourced by the run scripts
• init.sh — create the scripts/bin venv (pandas) used by the report scripts