Skip to content

feat(storage): S3-compatible storage engine (aiobotocore) behind make_storage_engine - #1292

Merged
odesenfans merged 3 commits into
mainfrom
feat/s3-storage-engine
Nov 10, 2026
Merged

odesenfans merged 3 commits into
mainfrom
feat/s3-storage-engine

Conversation

@odesenfans

@odesenfans odesenfans commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Rebased on main after #1291 merged; this PR now targets main directly.

Problem

A node's files (the content behind STORE messages) can only live in a local folder (storage.folder). That ties the node to one disk and one machine. Operators who run the node on ephemeral or replicated infrastructure want the files in an object store such as Garage, MinIO or S3.

Changes

  • New storage.engine setting: filesystem (default, no behaviour change) or s3.
  • New storage.s3 block: endpoint_url, region (default garage), bucket, prefix (default ""), access_key_id (STORAGE_S3_ACCESS_KEY_ID), secret_access_key (STORAGE_S3_SECRET_ACCESS_KEY), secret_access_key_file (STORAGE_S3_SECRET_ACCESS_KEY_FILE, which wins and is read once through read_credential), and ca_bundle (an optional PEM file; otherwise the system store or AWS_CA_BUNDLE applies).
  • S3StorageEngine (aiobotocore):
    • one lazily created client per engine, closed on shutdown;
    • path-style addressing;
    • standard retries (5 attempts), 5 s connect timeout, 60 s read timeout;
    • objects are stored at <prefix><hash>;
    • the bucket is checked once on first use;
    • only "no such key" maps to None or False, and any other error (403, missing bucket, 5xx) is raised;
    • streaming reads come in chunks of the requested size.
  • make_storage_engine(config) replaces the five FileSystemStorageEngine(...) constructions (main process, API, fetch/process/txs jobs). StorageEngine.close() (a no-op by default) is called wherever the other resources are torn down.
  • Dependencies: aiobotocore==3.10.0; for tests, moto[server]==5.2.5.
  • Docs: an "Object storage" section in the install guide, and the sample config.

How to test

hatch run testing:test tests/services/storage -v   # in-process moto S3 server, no DB needed
hatch run linting:all

To try it by hand, run a node with:

storage:
  engine: s3
  s3:
    endpoint_url: http://localhost:3900
    bucket: aleph-files
    access_key_id: GK...
    secret_access_key_file: /run/secrets/s3-secret

Point it at an existing bucket (Garage, or MinIO with region: us-east-1), upload a file through /api/v0/storage/add_file, and check that the object <prefix><hash> appears in the bucket.

Notes

  • A HEAD request on a missing bucket returns a bare 404 on S3, the same as a missing key. The first-use HeadBucket check is what keeps a wrong bucket name from making every file look absent.
  • Switching engines does not migrate files: copy storage.folder into the bucket first (rclone copy / aws s3 sync).
  • The secret is read at startup; rotating the S3 key needs a restart.
  • Uses read_credential from feat(config): read PostgreSQL and RabbitMQ passwords from files at connect time #1291 for the secret key file.

foxpatch-aleph
foxpatch-aleph previously approved these changes Nov 9, 2026

@foxpatch-aleph foxpatch-aleph left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, well-tested implementation. I ran all 21 new tests against the exact pinned versions (all pass) and verified edge cases beyond the suite: zero-byte objects and odd-size chunking (exact chunk_size chunks, totals preserved). The 404/NoSuchBucket discrimination, lazy client creation with double-checked locking, connection release on early aclose (proven by the pool-of-1 test), and the controller's finally/aclose fix are all correct. Lifecycle wiring is complete across all five processes, dependency pins are mutually consistent, and credentials are handled safely (never logged, file > env > inline via read_credential). Documentation is unusually thorough on real S3 quirks. Only minor, non-blocking observations remain.

src/aleph/services/storage/factory.py (line 29): The or secret_access_key is None clause is dead code at runtime: a None secret is already falsy and therefore already listed in missing, so the condition can never be true when missing is empty. It exists purely to narrow the Optional type for mypy after the check — a one-line comment saying so would prevent a future cleanup from removing it and breaking the type check.

src/aleph/web/controllers/storage.py (line 706): await response.prepare(request) sits outside the new try/finally. If prepare() raises (client gone during prepare), content.value is never aclosed — and with the S3 engine that is an already-open GetObject response holding a pooled connection until GC, rather than the filesystem engine's unstarted file-reading generator. Moving the prepare() call inside the try would close that gap for one line of change. Pre-existing structure, so non-blocking.

src/aleph/services/storage/s3_engine.py (line 151): Design observation, not a defect: read_iterator opens the GetObject response eagerly (needed to decide None vs iterator) but returns an unstarted generator, so a caller that drops the returned iterator without iterating or aclosing leaks the response until GC. The only current consumer (get_raw_hash) always iterates and now closes explicitly, so there is no leak path today — just worth keeping in mind when adding new consumers of get_hash_content_iterator.

tests/services/storage/test_s3_engine.py (line 129): Note that flipping moto.settings.INITIAL_NO_AUTH_ACTION_COUNT is a process-global toggle while the moto server is module-scoped; it works because the monkeypatch is function-scoped and moto reads the setting per request (verified passing), but the test would race if other tests in this module ever ran concurrently against the same server.

@odesenfans
odesenfans force-pushed the feat/credential-files branch from 1a2bfba to d2cf079 Compare November 10, 2026 10:37
@odesenfans
odesenfans force-pushed the feat/s3-storage-engine branch from 400e765 to 8317150 Compare November 10, 2026 10:37
Base automatically changed from feat/credential-files to main November 10, 2026 10:59
@odesenfans
odesenfans dismissed foxpatch-aleph’s stale review November 10, 2026 10:59

The base branch was changed.

…_storage_engine

A node can now keep its files in an S3-compatible bucket (Garage, MinIO,
AWS S3) instead of a local folder. `storage.engine` selects `filesystem`
(default, unchanged behaviour) or `s3`; the `storage.s3` block holds
endpoint_url, region, bucket, prefix, access_key_id, secret_access_key,
secret_access_key_file and ca_bundle.

- S3StorageEngine: one lazily created aiobotocore client per engine,
  path-style addressing, standard retries (5 attempts), 5 s connect / 60 s
  read timeouts, keys `<prefix><filename>`. The bucket is checked once when
  the client is created; `exists`/`read` map only "no such key" to
  False/None and raise on anything else. read_iterator streams in chunks of
  the requested size and releases the response when the consumer stops.
- make_storage_engine(config) replaces the five FileSystemStorageEngine
  construction sites; the secret is resolved once with read_credential
  (file > env > inline). StorageEngine gains a no-op close(), called on
  shutdown by the API, the main process and the three jobs.
- Tests run against an in-process moto server (moto[server] in the testing
  feature); no database needed.
…tors

- A failed first-use HeadBucket now raises StorageEngineConfigError
  "S3 bucket '<bucket>' at <endpoint> is not usable: ..." (logged at
  ERROR, chained to the ClientError) instead of a bare ClientError.
- GET /storage/raw closes the content iterator in a finally block, so a
  client disconnect releases the pooled S3 connection (or IPFS response)
  at once instead of at garbage collection.
- test_read_iterator_stops_early runs with a one-connection pool and
  fails if read_iterator does not release the response on aclose().
- Tests: StorageEngineConfigError message, verify passed only with
  ca_bundle, iterator closed on client disconnect.
- boto3 added to the testing feature (the tests import it).
- Docs: prefix has no separator, key permissions (Garage, AWS), the
  bucket check runs once per process, a bucket deleted later; sample
  config comments no longer nested (yamlfix mangled them).
read_credential returns Optional[str] since the credential-files branch;
mypy could not see that the missing-settings check excludes None.
@odesenfans
odesenfans force-pushed the feat/s3-storage-engine branch from 8317150 to 2b7477b Compare November 10, 2026 11:02
@odesenfans
odesenfans merged commit ee25732 into main Nov 10, 2026
3 of 4 checks passed
@odesenfans
odesenfans deleted the feat/s3-storage-engine branch November 10, 2026 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants