Repository navigation
feat(metrics): option to disable default Go/process/promhttp collectors - #2116
Open
Tyagiquamar wants to merge 1 commit into
Open
Tyagiquamar wants to merge 1 commit into
Tyagiquamar wants to merge 1 commit into
Conversation
The operator and metrics exporter always served the Prometheus client library's Go runtime, process and promhttp collectors because both scrapes read the global registry. Register the collector into a private registry and serve that instead when OPERATOR_METRICS_DISABLE_DEFAULT_COLLECTORS is true, which removes go_*, process_* and promhttp_* while keeping each component's own series. Default behaviour is unchanged. Fixes Altinity#2098 Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #2098. Both Go metrics endpoints always expose the Prometheus client
library's default collectors:
go_*andprocess_*on the operator (9999)and on the metrics exporter (8888), plus
promhttp_*on the exporter. On theoperator endpoint these account for 39 of 52 metric families, and deployments
that do not consume them have no way to drop them at the source.
Root cause
Both scrapes gather from the client library's global registry:
pkg/metrics/operator/machinery.goservedprom.DefaultGatherer.pkg/metrics/clickhouse/rest_server.goregistered the exporter onprometheus.DefaultRegistererand servedpromhttp.Handler(), which readsthe same global registry and additionally instruments the handler with
promhttp_*.client_golang'sinit()registers the Go and process collectors on thedefault registry, so they are included regardless of what the components
register.
Fix
New env var
OPERATOR_METRICS_DISABLE_DEFAULT_COLLECTORS(parsed withstrconv.ParseBool; unset or malformed values keep today's behaviour):from, a private
prometheus.NewRegistry()instead of the globals. Theoperator's own series and
target_infoare unaffected;go_*andprocess_*disappear.served with
promhttp.HandlerFor, sogo_*,process_*andpromhttp_*disappear while the
chi_clickhouse_*series are unaffected.docs/prometheus_setup.md, including the Helmoperator.env/metrics.enventries, since the variable is read per container.How tested
The tests scrape the real handlers: with the option unset the payload contains
go_goroutines,process_start_time_secondsand (exporter)promhttp_*; withit set to
trueno family with those prefixes is present, while a probe seriesstanding in for the component's own collector still is. I also verified the
operator opt-out test fails when the previous registry wiring is restored
(
go_memstats_buck_hash_sys_bytes is exposed with the default collectors disabled).Not run: full test suite, e2e, Docker builds; gofmt was checked on
LF-normalized copies because this checkout uses CRLF (plain
gofmt -lflagsevery pre-existing file).
Base is
0.28.0perCONTRIBUTING.md(release branch, notmaster).