Trying to figure out what's happening with your request traffic? Not sure why your Envoy configuration isn't working? If you're using Istio as your gateway and need to troubleshoot your ingress traffic requests, here are a few tips for debugging Envoy proxy.

Enable Envoy Debug Logging

By default Envoy system logs are sent to /dev/stderr. This location be overridden using --log-path. Logging to /dev/stderr for system logs and to /dev/stdout for access logs can be useful when running Envoy inside a container. In this way, these two individual logstreams can be separated, and using this approach, logging requires no additional files or directories to be mounted.

We recommend setting the Envoy proxy’s log level to debug in a pre-production environment. Debug logs can help you identify issues before you graduate the associated configuration to your production environment.

Using envoy CLI

The envoy command has a --log-level flag that can be useful for debugging. By default, it’s set to info. To change it to debug, edit the envoy DaemonSet in the istio-system namespace and replace the --log-level info flag with --log-level debug. Setting the Envoy log level to debug can be particilarly useful for debugging TLS connection failures.

Using container image

If you’re using the Envoy image, you can set the log level to debug through the ENVOY_LOG_LEVEL environment variable. The log level for Envoy system logs can be set using the -l or --log-level option.

The available log levels are:

  • trace
  • debug
  • info
  • warning/warn
  • error
  • critical
  • off

The default is info.

Setting Envoy logs in the Helm configuration

The Consul helm chart uses envoyExtraArgs: to leverage Envoy command line options. One of the helpful options is --component-log-level. This provides granular control over setting log levels for Envoy components. In the example below, the components upstream, http, router and config are set to the debug log level. These four components are vital when debugging issues with requests between your services(sidecar proxies).

1connectInject: enabled: true envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"

If you haven't set envoyExtraArgs: in consul-values.yaml just yet, you can set the log levels on the fly by using the following kubectl command:

1$ kubectl exec pod/pod-name -c container-name -- curl -X POST http://localhost:19000/logging?config=debug

Example:

1$ kubectl exec pod/static-client-5bf4575d9c-zr2b -c static-client -- curl -X POST http://localhost:19000/logging?config=debug

You will execute the kubectl command for each component. Make sure to append the correct component at the end of the curl command, i.e. logging? component = debug.

If curl is not able to be used in your pod, you can alternatively use kubectl port-forward pod-name 19000 to make the Envoy admin accessible. From another terminal window, you can then curl to change the log levels. The output you receive in the terminal will show the modified component log levels.

1$ curl -X POST http://localhost:19000/logging? component = debug

Access Envoy logs in Kubernetes

Accessing Envoy logs via pods can be done with the following command:

1$ kubectl logs --follow pod/ pod-name -c envoy-sidecar

The --follow flag provides a real time observation into Envoy logs.

Setting and Accessing Envoy logs when not using Helm.

The following command will start an envoy side car proxy, set the log level to debug with -l debug and capture Envoy logs in envoy_logs.txt. The .txt file will need to be created before executing this command.

1$ consul connect envoy -sidecar-for counting-1 -- -l debug --log-path envoy_logs.txt

To have granular control over the Envoy components that is needed to be debugged, use the following command:

1$ consul connect envoy -sidecar-for counting-1 -- --log-path envoy_logs.txt --component-log-level upstream:debug,http:debug,router:debug,config:debug

Find your Istio Ingress Gateway

With Istio as your gateway, you should first look at VirtualService objects. These can show if the hosts are registered to the gateway correctly.

1$ kubectl get virtualservice -o=yaml

However, sometimes, the Envoy inside the gateway container is not properly configured (likely due to a bug). You can dump Envoy configuration to debug this further.

1# find istio ingress gateway pod \ $ kubectl get pods -n istio-system -l app=istio-ingressgateway

Let's use istio-ingressgateway-a93019f9dfw-l39xd as an example pod name.

1# enable debugging on envoy \ $ kubectl exec --namespace=istio-system \ istio-ingressgateway-a93019f9dfw-l39xd \ -c istio-proxy -- curl -X POST \ http://localhost:15000/logging?level=debug

Then, use istioctl tool to dump route configuration (this will show the output from the /config_dump admin endpoint on Envoy):

1$ istioctl proxy-config routes -n istio-system -o=json \ istio-ingressgateway-a93019f9dfw-l39xd

We hope these steps are useful to you. If you're still having trouble configuring Envoy proxy, open up a new thread on the community discussion forum or subscribe to the Layer5 newletter for tips and tricks.

-

Layer5 Team

Related Blogs

Layer5, the cloud native management company

An empowerer of engineers, Layer5 helps you extract more value from your infrastructure. Creator and maintainer of service mesh standards. Maker of Meshery, the cloud native management plane.