このページは、まだ日本語ではご利用いただけません。翻訳中です。
Failure Modes and Processing in KIC
This reference describes the different types of Kong Ingress Controller failure modes and how it processes them.
When you run Kong Ingress Controller, you can encounter the following failures:
Error Example | Failure Mode |
---|---|
Reconciler error in logs |
Errors in reconciling Kubernetes resources |
Non-existent service referenced by an Ingress . Example: Ingress with a non-existent backend service |
Failures in translating configuration |
Kong Gateway rejected configuration Example: Ingress with invalid regex in the path |
Failures in applying configuration to Kong Gateway |
Errors when sending configuration to Kong Konnect Example: Failed request logs |
Failures in uploading configuration to Konnect |
Kong Ingress Controller uses different methods to process each failure type, and creates error logs or other evidence, like Prometheus metrics and Kubernetes events, so you can observe and track the failures.
Errors in reconciling Kubernetes resources
When the controllers reconciling a specific kind of Kubernetes resource run into errors in reconciling the resource, a Reconciler error
log line is recorded and the resource is re-queued for another round of reconciliation.
Thee Prometheus metric controller_runtime_reconcile_errors_total
stores the total number of reconcile errors per controller from the start of Kong Ingress Controller. Search for the Reconciler error
keyword in the Kong Ingress Controller container logs to see detailed errors.
Failures in translating configuration
When Kong Ingress Controller finds Kubernetes resources that can’t be correctly translated to Kong Gateway configuration (for example, an Ingress
is using a non-existent Service
as its backend), a translation failure is generated with the namespace and name of the objects causing the failure.
The Kubernetes objects causing translation failures are not translated to Kong Gateway configuration in the translation process. You can use Kubernetes events and Prometheus metrics to observe the translation failures. If Kong Ingress Controller is integrated with Kong Konnect, it will report that a translation error happened in the uploading node status.
Kong Ingress Controller collects all translation failures and generates a Kubernetes Event
with the Warning
type and the KongConfigurationTranslationFailed
reason for each causing object in a translation failure. Prometheus metrics could also reflect the statistics of translation failures:
-
ingress_controller_translation_broken_resource_count
is the number of translation failures that happened in the latest translation -
ingress_controller_translation_count
with thesuccess=false
label is the total number of translation procedures where translation failures happened
You can use kubectl get events -n <namespace> --field-selector reason="KongConfigurationTranslationFailed"
to fetch events generated for translation failures. For example, if an Ingress
named ing-1
in the namespace test
used a non-existent Service
as its backend, you could get the event with the following command:
kubectl get events -n test --field-selector reason="KongConfigurationTranslationFailed"
LAST SEEN TYPE REASON OBJECT MESSAGE
18m Warning KongConfigurationTranslationFailed ingress/ing-1 failed to resolve Kubernetes Service for backend: failed to fetch Service test/httpbin-deployment-1: Service test/httpbin-deployment-1 not found
Failures in applying configuration to Kong Gateway
When Kong Ingress Controller fails to apply translated Kong Gateway configuration to Kong Gateway, Kong Ingress Controller will try to recover from the failure and record the failure into logs, Kubernetes events, and Prometheus metrics. Recovery usually fails because the translated configuration is rejected by Kong Gateway.
If Kong Ingress Controller fails to apply the translated configuration, it then tries to apply the last successful Kong Gateway configuration to new instances of Kong Gateway to attempt a best effort at making them available.
If the FallbackConfiguration
feature gate is enabled, Kong Ingress Controller discovers the Kubernetes objects that caused the invalid configuration, and tries to build a fallback configuration from valid objects and parts of the last valid configuration that are built from the broken objects. See fallback configuration for more information.
Debugging configuration failures
You can observe failures in applying configuration from Kubernetes events and Prometheus metrics:
- Kong Ingress Controller generates an event with the
Warning
type and theKongConfigurationApplyFailed
reason attached to the pod itself when it fails to apply the configuration. - For each object that causes the invalid configuration, Kong Ingress Controller generates a
Warning
event type and theKongConfigurationApplyFailed
reason attached to the object. - The Prometheus metric
ingress_controller_configuration_push_count
with thesuccess=false
label shows the total number of failures from applying the configuration by reason and URL of Kong Gateway admin API. - The Prometheus metric
ingress_controller_configuration_push_broken_resource_count
reflects the number of Kubernetes resources that caused the error in the last configuration push.
If the CLI flag --dump-config
is enabled, then the /debug/config/raw-error
endpoint is enabled on the debug server port of Kong Ingress Controller, which will fetch the raw error returned from Kong Gateway if applying the configuration fails.
For example, let’s say you create an Ingress
with the ImplementationSpecific
path type and an invalid regex in Path
(which can only be only be done when the validating webhook is disabled, otherwise it will be rejected by the webhook):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
konghq.com/strip-path: "true"
name: ingress-invalid-regex
namespace: default
spec:
ingressClassName: kong
rules:
- http:
paths:
- backend:
service:
name: httpbin-deployment
port:
number: 80
path: /~^^/a$
pathType: ImplementationSpecific
You can get the Kubernetes events:
kubectl get events --all-namespaces --field-selector reason=KongConfigurationApplyFailed
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
default 2m9s Warning KongConfigurationApplyFailed ingress/ingress-invalid-regex invalid paths.1: should start with: / (fixed path) or ~/ (regex path)
kong 15s Warning KongConfigurationApplyFailed pod/kong-controller-779cb796f4-7q7c2 failed to apply Kong configuration to https://10.244.1.43:8444: HTTP status 400 (message: "failed posting new config to /config")
Both the events attached to the invalid ingress and attached to the Kong Ingress Controller pod are recorded.
Failures in uploading configuration to Konnect
When Kong Ingress Controller is integrated with Konnect and it fails to send configuration to Konnect, it generates error logs for failed requests, records the failures to Prometheus metrics, and updates the node status of itself in Konnect:
- Kong Ingress Controller parses errors returned from Konnect when uploading the configuration fails. It logs a line at the error level for each Kong Gateway entity that failed to create/update/delete, with the message
Failed to send request to Konnect
. - The Prometheus metrics
ingress_controller_configuration_push_count
andingress_controller_configuration_push_duration_milliseconds_bucket
can also reflect configuration upload failures to Konnect, where thedataplane
label is the URL of Konnect andsuccess=false
APIs.