Repository navigation
feat(storage): S3-compatible storage engine (aiobotocore) behind make_storage_engine - #1292
Conversation
foxpatch-aleph
left a comment
There was a problem hiding this comment.
Solid, well-tested implementation. I ran all 21 new tests against the exact pinned versions (all pass) and verified edge cases beyond the suite: zero-byte objects and odd-size chunking (exact chunk_size chunks, totals preserved). The 404/NoSuchBucket discrimination, lazy client creation with double-checked locking, connection release on early aclose (proven by the pool-of-1 test), and the controller's finally/aclose fix are all correct. Lifecycle wiring is complete across all five processes, dependency pins are mutually consistent, and credentials are handled safely (never logged, file > env > inline via read_credential). Documentation is unusually thorough on real S3 quirks. Only minor, non-blocking observations remain.
src/aleph/services/storage/factory.py (line 29): The or secret_access_key is None clause is dead code at runtime: a None secret is already falsy and therefore already listed in missing, so the condition can never be true when missing is empty. It exists purely to narrow the Optional type for mypy after the check — a one-line comment saying so would prevent a future cleanup from removing it and breaking the type check.
src/aleph/web/controllers/storage.py (line 706): await response.prepare(request) sits outside the new try/finally. If prepare() raises (client gone during prepare), content.value is never aclosed — and with the S3 engine that is an already-open GetObject response holding a pooled connection until GC, rather than the filesystem engine's unstarted file-reading generator. Moving the prepare() call inside the try would close that gap for one line of change. Pre-existing structure, so non-blocking.
src/aleph/services/storage/s3_engine.py (line 151): Design observation, not a defect: read_iterator opens the GetObject response eagerly (needed to decide None vs iterator) but returns an unstarted generator, so a caller that drops the returned iterator without iterating or aclosing leaks the response until GC. The only current consumer (get_raw_hash) always iterates and now closes explicitly, so there is no leak path today — just worth keeping in mind when adding new consumers of get_hash_content_iterator.
tests/services/storage/test_s3_engine.py (line 129): Note that flipping moto.settings.INITIAL_NO_AUTH_ACTION_COUNT is a process-global toggle while the moto server is module-scoped; it works because the monkeypatch is function-scoped and moto reads the setting per request (verified passing), but the test would race if other tests in this module ever ran concurrently against the same server.
1a2bfba to
d2cf079
Compare
400e765 to
8317150
Compare
The base branch was changed.
…_storage_engine A node can now keep its files in an S3-compatible bucket (Garage, MinIO, AWS S3) instead of a local folder. `storage.engine` selects `filesystem` (default, unchanged behaviour) or `s3`; the `storage.s3` block holds endpoint_url, region, bucket, prefix, access_key_id, secret_access_key, secret_access_key_file and ca_bundle. - S3StorageEngine: one lazily created aiobotocore client per engine, path-style addressing, standard retries (5 attempts), 5 s connect / 60 s read timeouts, keys `<prefix><filename>`. The bucket is checked once when the client is created; `exists`/`read` map only "no such key" to False/None and raise on anything else. read_iterator streams in chunks of the requested size and releases the response when the consumer stops. - make_storage_engine(config) replaces the five FileSystemStorageEngine construction sites; the secret is resolved once with read_credential (file > env > inline). StorageEngine gains a no-op close(), called on shutdown by the API, the main process and the three jobs. - Tests run against an in-process moto server (moto[server] in the testing feature); no database needed.
…tors - A failed first-use HeadBucket now raises StorageEngineConfigError "S3 bucket '<bucket>' at <endpoint> is not usable: ..." (logged at ERROR, chained to the ClientError) instead of a bare ClientError. - GET /storage/raw closes the content iterator in a finally block, so a client disconnect releases the pooled S3 connection (or IPFS response) at once instead of at garbage collection. - test_read_iterator_stops_early runs with a one-connection pool and fails if read_iterator does not release the response on aclose(). - Tests: StorageEngineConfigError message, verify passed only with ca_bundle, iterator closed on client disconnect. - boto3 added to the testing feature (the tests import it). - Docs: prefix has no separator, key permissions (Garage, AWS), the bucket check runs once per process, a bucket deleted later; sample config comments no longer nested (yamlfix mangled them).
read_credential returns Optional[str] since the credential-files branch; mypy could not see that the missing-settings check excludes None.
8317150 to
2b7477b
Compare
Rebased on
mainafter #1291 merged; this PR now targetsmaindirectly.Problem
A node's files (the content behind STORE messages) can only live in a local folder (
storage.folder). That ties the node to one disk and one machine. Operators who run the node on ephemeral or replicated infrastructure want the files in an object store such as Garage, MinIO or S3.Changes
storage.enginesetting:filesystem(default, no behaviour change) ors3.storage.s3block:endpoint_url,region(defaultgarage),bucket,prefix(default""),access_key_id(STORAGE_S3_ACCESS_KEY_ID),secret_access_key(STORAGE_S3_SECRET_ACCESS_KEY),secret_access_key_file(STORAGE_S3_SECRET_ACCESS_KEY_FILE, which wins and is read once throughread_credential), andca_bundle(an optional PEM file; otherwise the system store orAWS_CA_BUNDLEapplies).S3StorageEngine(aiobotocore):<prefix><hash>;make_storage_engine(config)replaces the fiveFileSystemStorageEngine(...)constructions (main process, API, fetch/process/txs jobs).StorageEngine.close()(a no-op by default) is called wherever the other resources are torn down.aiobotocore==3.10.0; for tests,moto[server]==5.2.5.How to test
hatch run testing:test tests/services/storage -v # in-process moto S3 server, no DB needed hatch run linting:allTo try it by hand, run a node with:
Point it at an existing bucket (Garage, or MinIO with
region: us-east-1), upload a file through/api/v0/storage/add_file, and check that the object<prefix><hash>appears in the bucket.Notes
storage.folderinto the bucket first (rclone copy/aws s3 sync).read_credentialfrom feat(config): read PostgreSQL and RabbitMQ passwords from files at connect time #1291 for the secret key file.