Zipkin - Plugin | Kong Docs

Looking for the plugin's configuration parameters? You can find them in the Zipkin configuration reference doc.

Propagate Zipkin distributed tracing spans and report spans to a Zipkin server.

Queueing

The zipkin plugin uses a queue to decouple the production and consumption of data. This reduces the number of concurrent requests made to the upstream server under high load situations and provides buffering during temporary network or upstream outages.

You can set several parameters to configure the behavior and capacity of the queues used by the plugin. For more information about how to use these parameters, see Plugin Queuing Reference in the Kong Gateway documentation.

The queue parameters all reside in a record under the key queue in the config parameter section of the plugin.

Queues are not shared between workers and queueing parameters are scoped to one worker. For whole-system capacity planning, the number of workers need to be considered when setting queue parameters.

Trace IDs in serialized logs

When the Zipkin plugin is configured along with a plugin that uses the Log Serializer, the trace ID of each request is added to the key trace_id in the serialized log output.

The value of this field is an object that can contain different formats of the current request’s trace ID. In case of multiple tracing headers in the same request, the trace_id field includes one trace ID format for each different header format, as in the following example:

"trace_id": {
  "b3": "4bf92f3577b34da6a3ce929d0e0e4736",
  "datadog": "11803532876627986230"
},

How it Works

When enabled, this plugin traces requests in a way compatible with zipkin.

The code is structured around an OpenTracing core using the opentracing-lua library to collect timing data of a request in each of Kong’s phases. The plugin uses opentracing-lua compatible extractor, injector, and reporters to implement Zipkin’s protocols.

Reporter

An OpenTracing “reporter” is how tracing data is reported to another system. This plugin records tracing data for a given request, and sends it as a batch to a Zipkin server using the Zipkin v2 API. Note that zipkin version 1.31 or higher is required.

The http_endpoint configuration variable must contain the full uri including scheme, host, port and path sections (i.e. your uri likely ends in /api/v2/spans).

Spans

The plugin does request sampling. For each request which triggers the plugin, a random number between 0 and 1 is chosen.

If the number is smaller than the configured sample_ratio, then a trace with several spans will be generated. If sample_ratio is set to 1, then all requests will generate a trace (this might be very noisy).

For each request that gets traced, the following spans are produced:

Request span: 1 per request. Encompasses the whole request in kong (kind: SERVER). The Proxy and Balancer spans are children of this span. It contains the following logs/annotations for the rewrite phase:
- krs - kong.rewrite.start
- krf - kong.rewrite.finish
The Request span has the following tags:
- lc: Hardcoded to kong.
- kong.service: The uuid of the service matched when processing the request, if any.
- kong.service_name: The name of the service matched when processing the request, if service exists and has a name attribute.
- kong.route: The uuid of the route matched when processing the request, if any (it can be nil on non-matched requests).
- kong.route_name: The name of the route matched when processing the request, if route exists and has a name attribute.
- http.method: The HTTP method used on the original request (only for HTTP requests).
- http.path: The path of the request (only for HTTP requests).
- If the plugin tags_header config option is set, and the request contains headers with the appropriate name and correct encoding tags, then the trace will include the tags.
- If the plugin static_tags config option is set, then the tags in the config option will be included in the trace.
Proxy span: 1 per request, encompassing most of Kong’s internal processing of a request (kind: CLIENT). Contains the following logs/annotations for the start/finish of the of the Kong plugin phases:
- kas - kong.access.start
- kaf - kong.access.finish
- kbs - kong.body_filter.start
- kbf - kong.body_filter.finish
- khs - kong.header_filter.start
- khf - kong.header_filter.finish
- kps - kong.preread.start (only for stream requests)
- kpf - kong.preread.finish (only for stream requests)
Balancer span(s): 0 or more per request, each encompassing one balancer attempt (kind: CLIENT). Contains the following tags specific to load balancing:
- kong.balancer.try: A number indicating the attempt (1 for the first load-balancing attempt, 2 for the second, and so on).
- peer.ipv4 or peer.ipv6 for the balancer IP.
- peer.port for the balanced port.
- error: Set to true if the balancing attempt was unsuccessful, otherwise unset.
- http.status_code: The HTTP status code received, in case of error.
- kong.balancer.state: An NGINX-specific description of the error, next/failed for HTTP failures, or 0 for stream failures. Equivalent to state_name in OpenResty’s balancer’s get_last_failure function.

Propagation

The Zipkin plugin supports propagation of the following header formats:

This plugin offers extensive options for configuring tracing header propagation, providing a high degree of flexibility. Users can freely customize which headers are used to extract and inject tracing context. Additionally, they have the ability to configure headers to be cleared after the tracing context extraction process, enabling a high level of customization.

 
flowchart LR
   id1(Original Request) --> Extract
   id1(Original Request) -->|"headers (original)"| Extract
   id1(Original Request) --> Extract
   subgraph ide1 [Headers Propagation]
   Extract --> Clear
   Extract -->|"headers (original)"| Clear
   Extract --> Clear
   Clear -->|"headers (filtered)"| Inject
   end
   Extract -.->|extracted ctx| id2((tracing logic))
   id2((tracing logic)) -.->|updated ctx| Inject
   Inject -->|"headers (updated ctx)"| id3(Updated request)

The following examples demonstrate how the propagation configuration options can be used to achieve various use cases.

Extract, clear, and inject

Extract the tracing context using order of precedence: w3c > b3 > jaeger > ot > aws > datadog
Clear b3 and uber-trace-id headers after extraction, if present in the request
Inject the tracing context using the format: w3c

- config:
    propagation:	
      extract: [ w3c, b3, jaeger, ot, aws, datadog ]
      clear: [ b3, uber-trace-id ]
      inject: [ w3c ]

Multiple injection

Extract the tracing context from: b3
Inject the tracing context using the formats: w3c, b3, jaeger, ot, aws, datadog, gcp

- config:
    propagation:	
      extract: [ b3 ]
      inject: [ w3c, b3, jaeger, ot, aws, datadog, gcp ]

Preserve incoming format

Extract the tracing context using order of precedence: w3c > b3 > jaeger > ot > aws > datadog
Inject the tracing context in the extracted header type
Default to w3c for context injection if none of the extract header types were found in the request

- config:
    propagation:	
      extract: [ w3c, b3, jaeger, ot, aws, datadog ]
      inject: [ preserve ]
      default_format: "w3c"

preserve can also be used with other formats, to specify that the incoming format should be preserved in addition to the others:

- config:
    propagation:	
      extract: [ w3c, b3, jaeger, ot, datadog ]
      inject: [ aws, preserve, datadog ]
      default_format: "w3c"

Ignore incoming headers

No tracing context extraction
Inject the tracing context using the formats: b3, datadog

- config:
    propagation:	
      extract: [ ]
      inject: [ b3, datadog ]

Note: Some header formats specify different trace and span ID sizes. When the tracing context is extracted and injected from/to headers with different ID sizes, the IDs are truncated or left-padded to align with the target format.

Refer to the plugin’s configuration reference for a complete overview of the available options and values.

Note: If any of the propagation.* configuration options (extract, clear, or inject) are configured, the propagation configuration takes precedence over the deprecated header_type and default_header_type parameters. If none of the propagation.* configuration options are set, the header_type and default_header_type parameters are still used to determine the propagation behavior.