Skip to content

feat(metrics): option to disable default Go/process/promhttp collectors - #2116

Open
Tyagiquamar wants to merge 1 commit into
Altinity:0.28.0from
Tyagiquamar:feat/metrics-disable-default-collectors
Open

Tyagiquamar wants to merge 1 commit into
Altinity:0.28.0from
Tyagiquamar:feat/metrics-disable-default-collectors

Conversation

@Tyagiquamar

Copy link
Copy Markdown

Problem

Fixes #2098. Both Go metrics endpoints always expose the Prometheus client
library's default collectors: go_* and process_* on the operator (9999)
and on the metrics exporter (8888), plus promhttp_* on the exporter. On the
operator endpoint these account for 39 of 52 metric families, and deployments
that do not consume them have no way to drop them at the source.

Root cause

Both scrapes gather from the client library's global registry:

  • pkg/metrics/operator/machinery.go served prom.DefaultGatherer.
  • pkg/metrics/clickhouse/rest_server.go registered the exporter on
    prometheus.DefaultRegisterer and served promhttp.Handler(), which reads
    the same global registry and additionally instruments the handler with
    promhttp_*.

client_golang's init() registers the Go and process collectors on the
default registry, so they are included regardless of what the components
register.

Fix

New env var OPERATOR_METRICS_DISABLE_DEFAULT_COLLECTORS (parsed with
strconv.ParseBool; unset or malformed values keep today's behaviour):

  • Operator: the OTel exporter is registered into, and the scrape gathered
    from, a private prometheus.NewRegistry() instead of the globals. The
    operator's own series and target_info are unaffected; go_* and
    process_* disappear.
  • Metrics exporter: the exporter is registered into a private registry and
    served with promhttp.HandlerFor, so go_*, process_* and promhttp_*
    disappear while the chi_clickhouse_* series are unaffected.
  • Default behaviour is byte-for-byte the previous wiring.
  • Documented in docs/prometheus_setup.md, including the Helm operator.env /
    metrics.env entries, since the variable is read per container.

How tested

go test -p 1 -count=1 -run TestDefaultCollectorsDisabled ./pkg/apis/deployment/
# ok github.com/altinity/clickhouse-operator/pkg/apis/deployment 1.526s

go test -p 1 -count=1 -run TestMetricsCollectorsToggle ./pkg/metrics/operator/
# ok github.com/altinity/clickhouse-operator/pkg/metrics/operator 4.467s

go test -p 1 -count=1 -run TestMetricsHandlerCollectorsToggle ./pkg/metrics/clickhouse/
# ok github.com/altinity/clickhouse-operator/pkg/metrics/clickhouse 4.661s

go vet ./pkg/apis/deployment/ ./pkg/metrics/operator/ ./pkg/metrics/clickhouse/
# no output, exit 0

The tests scrape the real handlers: with the option unset the payload contains
go_goroutines, process_start_time_seconds and (exporter) promhttp_*; with
it set to true no family with those prefixes is present, while a probe series
standing in for the component's own collector still is. I also verified the
operator opt-out test fails when the previous registry wiring is restored
(go_memstats_buck_hash_sys_bytes is exposed with the default collectors disabled).

Not run: full test suite, e2e, Docker builds; gofmt was checked on
LF-normalized copies because this checkout uses CRLF (plain gofmt -l flags
every pre-existing file).

Base is 0.28.0 per CONTRIBUTING.md (release branch, not master).

The operator and metrics exporter always served the Prometheus client
library's Go runtime, process and promhttp collectors because both
scrapes read the global registry. Register the collector into a private
registry and serve that instead when
OPERATOR_METRICS_DISABLE_DEFAULT_COLLECTORS is true, which removes
go_*, process_* and promhttp_* while keeping each component's own
series. Default behaviour is unchanged.

Fixes Altinity#2098

Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant