Tracing Reference

This reference describes the OpenTelemetry tracing capabilities of Kong Gateway.

Important: OpenTelemetry tracing replaces the deprecated Granular Tracing feature. Granular Tracing has been removed from Kong Gateway starting in 3.7.0.0, and configurations like tracing = on are no longer available. Instead, use the OpenTelemetry Tracing described on this page.

Core instrumentations

The following instrumentations can be used by any plugins built on top of Kong’s tracing API, for example, the OpenTelemetry plugin.

Kong provides a set of core instrumentations for tracing, these can be configured in the tracing_instrumentations configuration.

off: do not enable instrumentations.
request: only enable request-level instrumentations.
all: enable all the following instrumentations.
db_query: trace database query.
dns_query: trace DNS query.
router: trace router execution, including router rebuilding.
http_client: trace OpenResty HTTP client requests.
balancer: trace balancer retries.
plugin_rewrite: trace plugins iterator execution with rewrite phase.
plugin_access: trace plugins iterator execution with access phase.
plugin_header_filter: trace plugins iterator execution with header_filter phase.

Propagation

The tracing API supports propagating the following headers:

w3c - W3C trace context
b3, b3-single - Zipkin headers
jaeger - Jaeger headers
ot - OpenTracing headers

The tracing API detects the propagation format from the headers, and uses the appropriate format to propagate the span context. If no appropriate format is found, it falls back to the default format, which can be user-specified.

The propagation API works for both the OpenTelemetry plugin and the Zipkin plugin.

Headers

See the kong.conf reference for all supported tracing headers.

X-Kong-Request-Id header

The X-Kong-Request-Id header contains a unique identifier for each client request. This is enabled by default both upstream and downstream. This unique ID helps in matching specific requests to their corresponding error logs, which is useful for debugging. If Kong Gateway returns an error by calling the PDK kong.response.error, the request ID will also be included in the response body generated by Kong Gateway. In addition, any Kong Gateway error log generated will contain the same request ID, with the format: request_id: xxx.

The log line produced by the debug header as well as the debug response header contains the request ID. You can use this to search for the debug header output using a log viewer UI. This is especially useful when the debug output is too long to fit in the response header.