Skip to content

feat(s3): attribute S3 GET and PUT metrics to WAL sources - #3746

Open
ashsamantaray-stripe wants to merge 1 commit into
AutoMQ:mainfrom
ashsamantaray-stripe:ashsamantaray/wal-s3-request-source-upstream
Open

ashsamantaray-stripe wants to merge 1 commit into
AutoMQ:mainfrom
ashsamantaray-stripe:ashsamantaray/wal-s3-request-source-upstream

Conversation

@ashsamantaray-stripe

Copy link
Copy Markdown
Contributor

Why

S3 GET/PUT metrics currently combine requests from all callers, which makes it difficult to identify how much request traffic comes from WAL persistence and routing. This is a WAL-focused first increment toward #2991.

What

Add a bounded source label to the existing GET/PUT observations in kafka_stream_operation_latency, starting with the WAL paths involved in producer persistence and acknowledgement:

operation_name source WAL path
put_object ROUTER_WAL Persist routed producer payload in RouterWAL.
get_object ROUTER_WAL Read the routed payload on the remote target before appending it.
put_object DELTA_WAL Persist the link record for linked writes, or the full record for local DIRECT writes.
get_object DELTA_WAL Verify the reservation after WAL upload and before append completion.

Local DIRECT writes skip RouterWAL. Local linked writes retain the RouterWAL PUT and avoid the remote payload GET. Existing callers without attribution receive UNCLASSIFIED.

These labels identify WAL ownership. Shared WAL readers and writers also retain their source during replay, close flushes, and synthetic trim writes, so the label alone cannot isolate requests for a single producer ACK. Broader classification of compaction, catchup reads, data-object uploads, startup reservation acquisition, multipart, LIST, and DELETE remains follow-up work.

How

WAL factories set the source explicitly, independently of the object-key layout. Readers and writers carry it in the existing request options, including both preferred-layout and fallback GETs. Options are copied at asynchronous submission so later caller mutation cannot relabel a pending request; the copy retains existing timeout and retry settings.

The source is attached at the existing success/failure observation points, including normal retries and fast retries. Existing counting boundaries, SDK retry behavior, read-merging decisions, routing, and append completion ordering are preserved. The change adds no S3 calls, object metadata, dependencies, configuration switches, or export timers.

GET/PUT histogram registrations are cached lazily by source and the existing status/size dimensions. The four enum values bound cardinality; unused values allocate no series. The label appears on count, sum, p50, p99, and max exports. Count and sum remain cumulative, and aggregation across sources recovers their previous totals. Percentiles describe each source's own latency distribution and cannot be added together.

Why MIXED exists

The storage layer already coalesces eligible range reads of the same object through the same client. If every child read has the same source, the merged GET keeps that source. If sources differ, it uses MIXED.

For example, overlapping ROUTER_WAL and UNCLASSIFIED reads can become one GET. Assigning it to the first caller would misattribute shared work; recording it once for each caller would inflate the count; splitting the reads would change S3 traffic. MIXED preserves one observation for the merged GET, with the same retry accounting as before.

Interpret MIXED as a shared GET with conflicting caller attribution. Include it in totals; its label cannot apportion the request among callers. It is not expected in the normal producer flow: RouterWAL and DeltaWAL use distinct object paths, and this merge rule applies only to reads. Its series is created only when observed.

Reviewer notes

  • DefaultRouterChannelProvider, DefaultWalFactory, and ObjectWALConfig establish WAL ownership; DefaultReader, DefaultWriter, and ObjectReservationService carry it into GET/PUT options.
  • AbstractObjectStorage preserves the existing I/O paths and resolves attribution after eligible reads have been merged.
  • ObjectStorageMetrics and OperationLatencyMetrics partition the existing histogram observations. Focused coverage protects source propagation, fallback reads, reservation-gated completion, retry/hedge accounting, buffer ownership, merged-read attribution, and cumulative exports.

Committed-By-Agent: codex
Co-authored-by: codex <noreply@openai.com>
@ashsamantaray-stripe ashsamantaray-stripe changed the title feat: attribute S3 GET and PUT metrics to WAL sources feat(s3): attribute S3 GET and PUT metrics to WAL sources Nov 8, 2026
@ashsamantaray-stripe
ashsamantaray-stripe marked this pull request as ready for review November 8, 2026 15:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant