Metrics and telemetry

goethite keeps its metrics with OpenTelemetry. It offers them two ways, and both can be on at once:

• Pull: Prometheus, or an OpenTelemetry Collector, scrapes /metrics on the API. This is on whenever the API is.
• Push: goethite sends them to an OpenTelemetry collector over OTLP/HTTP. This is off until [telemetry] endpoint is set.

With an endpoint set, goethite also sends its log records to the collector, while still writing them to standard error, and, with traces = true, traces of its slow paths.

Both ways carry the same metrics. Pushed, they keep OpenTelemetry's names, such as goethite.queries and goethite.query.duration; at /metrics they have their Prometheus names, such as goethite_queries_total and goethite_query_duration_seconds.

• Queries: answered queries by outcome and protocol (udp, tcp, dot, doh, doq, odoh), and a latency histogram from 100 µs to 2.5 s.
• Turned away: rate-limited and refused queries, refused TCP connections, failed TLS handshakes, and rejected DNS over HTTPS requests.
• Resolution: the cache's size, hits and misses; each upstream's health; recursion's queries, timeouts, failures and DNSSEC results.
• Filtering: rules, enabled lists, whether protection is on or paused, and filtering failures.
• The node: the query log's dropped entries, whether the node is degraded, its version and start time.

They hold counts and states only, never a name that was looked up or a client's address. Those stay in the query log.

What is sent. Every log line at INFO and above, the same lines journald gets. Each record carries its severity, its target (such as goethite::filters), its message and its fields. RUST_LOG changes what standard error shows, not what is sent. Lines about the export itself stay out, so a collector that is down cannot fill its own queue.

What the lines hold. They are about goethite: starting, lists, upstreams, the cluster, problems. A few warnings name what they are about:

• A name that was looked up: when no upstream could answer it, and when filtering it failed (a bug).
• A client's address or ID: when an answer to it could not be encoded, or its configured address is invalid.

If those must not leave the node, keep logs = false.

Turning it off. logs = false stops sending them.

What is traced. With traces = true, goethite opens a span wherever it waits for a network or does slow work:

• resolve: a query that missed the cache. Inside it, one upstream span per attempt at an upstream (its address, transport and result), or with recursion one authoritative span per query sent, and dnssec for validation.
• api: an API request, by method, route and status.
• The rest: lists.refresh and list.download, filter.build, and raft.call with raft.append, raft.vote and raft.snapshot between cluster members.

What is never traced. Cache hits, blocked names and local answers, which are most queries, open no span at all.

Sampling. trace_sample_ratio keeps a share of traces, 5% by default. The choice is made per trace, so a kept trace is whole.

What spans hold. Record types, response codes, upstream addresses, API routes and timings. The name looked up, and with recursion the servers asked (which tell the name too), are added only with query_details = true. Client addresses never are.

Cost. With traces off, a span costs an atomic load. With them on, each query that misses the cache builds its spans whether or not its trace is kept.

See REST API for the scrape configuration. /metrics needs the admin token once one is configured.

[telemetry]
endpoint = "https://collector.example:4318"
headers_file = "/etc/goethite/otlp-headers"
interval = 60

Sending.

• Metrics: posted to <endpoint>/v1/metrics every interval seconds, and once more when goethite stops.
• Log records: posted to <endpoint>/v1/logs in batches, within a second or so of being written.
• Traces: posted to <endpoint>/v1/traces in batches, within a few seconds.

What identifies the node. Each export carries service.name = goethite and service.version. On a cluster member it also carries service.instance.id = the node's name.

The headers file. It holds the headers a collector or vendor asks for, one per line:

# An API key, for example
Authorization: Bearer <key>

Make it readable by root only: goethite reads it, and ca_file, before it drops its privileges. A private CA goes in ca_file.

A minimal OpenTelemetry Collector configuration to receive them:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
exporters:
  debug: {}
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [debug]
    logs:
      receivers: [otlp]
      exporters: [debug]
    traces:
      receivers: [otlp]
      exporters: [debug]

The collector's name. goethite resolves it through its own upstreams and local records, not /etc/resolv.conf: the system's resolver may be goethite itself, and goethite's sandbox does not read that file. Use an address, or a name your upstreams or a local record answer. A container or Kubernetes service name answers only through those.

When the collector is down.

• DNS is never affected. Exports run on threads of their own and give up after 10 seconds, retries included.
• The next metrics export sends the counts so far. Log records and spans wait in bounded queues (2,048 each) and are dropped when they are full; the log lines are still in the journal.
• goethite logs one warning when exports start failing, and one line when they work again.
• goethite_telemetry_export_failures_total counts the failed requests, retries included.

Which processes export. Only goethite run sends telemetry. goethite witness and goethite vrrp log to standard error as before.