Trying to figure out what's happening with your request traffic? Not sure why your Envoy configuration isn't working? If you're using Istio as your gateway and need to troubleshoot your ingress traffic requests, here are a few tips for debugging Envoy proxy.
Enable Envoy Debug Logging
By default Envoy system logs are sent to /dev/stderr. This location be overridden using --log-path. Logging to /dev/stderr for system logs and to /dev/stdout for access logs can be useful when running Envoy inside a container. In this way, these two individual logstreams can be separated, and using this approach, logging requires no additional files or directories to be mounted.
We recommend setting the Envoy proxy’s log level to debug in a pre-production environment. Debug logs can help you identify issues before you graduate the associated configuration to your production environment.
Using envoy CLI
The envoy command has a --log-level flag that can be useful for debugging. By default, it’s set to info. To change it to debug, edit the envoy DaemonSet in the istio-system namespace and replace the --log-level info flag with --log-level debug. Setting the Envoy log level to debug can be particilarly useful for debugging TLS connection failures.
Using container image
If you’re using the Envoy image, you can set the log level to debug through the ENVOY_LOG_LEVEL environment variable. The log level for Envoy system logs can be set using the -l or --log-level option.
The available log levels are:
- trace
- debug
- info
- warning/warn
- error
- critical
- off
The default is info.
Setting Envoy logs in the Helm configuration
The Consul helm chart uses envoyExtraArgs: to leverage Envoy command line options. One of the helpful options is --component-log-level. This provides granular control over setting log levels for Envoy components. In the example below, the components upstream, http, router and config are set to the debug log level. These four components are vital when debugging issues with requests between your services(sidecar proxies).
1connectInject: enabled: true envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"
If you haven't set envoyExtraArgs: in consul-values.yaml just yet, you can set the log levels on the fly by using the following kubectl command:
1$ kubectl exec pod/pod-name -c container-name -- curl -X POST http://localhost:19000/logging?config=debug
Example:
1$ kubectl exec pod/static-client-5bf4575d9c-zr2b -c static-client -- curl -X POST http://localhost:19000/logging?config=debug
You will execute the kubectl command for each component. Make sure to append the correct component at the end of the curl command, i.e. logging? component = debug.
If curl is not able to be used in your pod, you can alternatively use kubectl port-forward pod-name 19000 to make the Envoy admin accessible. From another terminal window, you can then curl to change the log levels. The output you receive in the terminal will show the modified component log levels.
1$ curl -X POST http://localhost:19000/logging? component = debug
Access Envoy logs in Kubernetes
Accessing Envoy logs via pods can be done with the following command:
1$ kubectl logs --follow pod/ pod-name -c envoy-sidecar
The --follow flag provides a real time observation into Envoy logs.
Setting and Accessing Envoy logs when not using Helm.
The following command will start an envoy side car proxy, set the log level to debug with -l debug and capture Envoy logs in envoy_logs.txt. The .txt file will need to be created before executing this command.
1$ consul connect envoy -sidecar-for counting-1 -- -l debug --log-path envoy_logs.txt
To have granular control over the Envoy components that is needed to be debugged, use the following command:
1$ consul connect envoy -sidecar-for counting-1 -- --log-path envoy_logs.txt --component-log-level upstream:debug,http:debug,router:debug,config:debug
Find your Istio Ingress Gateway
With Istio as your gateway, you should first look at VirtualService objects. These can show if the hosts are registered to the gateway correctly.
1$ kubectl get virtualservice -o=yaml
However, sometimes, the Envoy inside the gateway container is not properly configured (likely due to a bug). You can dump Envoy configuration to debug this further.
1# find istio ingress gateway pod \ $ kubectl get pods -n istio-system -l app=istio-ingressgateway
Let's use istio-ingressgateway-a93019f9dfw-l39xd as an example pod name.
1# enable debugging on envoy \ $ kubectl exec --namespace=istio-system \ istio-ingressgateway-a93019f9dfw-l39xd \ -c istio-proxy -- curl -X POST \ http://localhost:15000/logging?level=debug
Then, use istioctl tool to dump route configuration (this will show the output from the /config_dump admin endpoint on Envoy):
1$ istioctl proxy-config routes -n istio-system -o=json \ istio-ingressgateway-a93019f9dfw-l39xd
We hope these steps are useful to you. If you're still having trouble configuring Envoy proxy, open up a new thread on the community discussion forum or subscribe to the Layer5 newletter for tips and tricks.
Team
