Mutual TLS & Identity Verification

Istio provides transparent mutual TLS to services inside the service mesh where both the client and the server authenticate each others' certificates as part of the TLS handshake. As part of this course, we have deployed Istio with mTLS.

By default istio sets mTLS in PERMISSIVE mode which allows plain text traffic to be sent and accepted by a mesh. We first disallow plain text traffic using PeerAuthentication and setting mTLS mode to STRICT.

Confirm mTLS is being enforced

This can be easily done by executing a simple command:-

Copy
1kubectl get peerauthentication --all-namespaces

Verify mTLS

Citadel is Istio’s key management service. As a first step, confirm that Citadel is up and running:

Copy
1kubectl get deploy -l istio=citadel -n istio-system

Output will be similar to:

Copy
1NAME            READY   UP-TO-DATE   AVAILABLE   AGE
2istio-citadel   1/1     1            1           3m23s

To experiment with mTLS, let's do so by logging into the sidecar proxy of the productpage pod by executing this command:

Copy
1kubectl exec -it $(kubectl get pod | grep productpage | awk '{ print $1 }') -c istio-proxy -- /bin/bash

We are now in the proxy of the productpage pod. Check that all the ceritificates are loaded in this proxy:

Copy
1ls /etc/certs/

You should see 3 entries:

Copy
1cert-chain.pem  key.pem  root-cert.pem

Now, try to make a curl call to the details service over HTTP:

Copy
1curl http://details:9080/details/0

Since, we have TLS between the sidecar's, an HTTP call will not work. The request will timeout. You will see an error like the one below:

Copy
1curl: (7) Failed to connect to details port 9080: Connection timed out

Let us try to make a curl call to the details service over HTTPS but WITHOUT certs:

Copy
1curl https://details:9080/details/0 -k

The request will be denied and you will see an error like the one below:

Copy
1curl: (16) SSL_write() returned SYSCALL, errno = 104

Now, let us use curl over HTTPS with certificates to the details service:

Copy
1curl https://details:9080/details/0 -v --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k

Output will be similar to this:

Copy
1*   Trying 10.107.35.26...
2* Connected to details (10.107.35.26) port 9080 (#0)
3* found 1 certificates in /etc/certs/root-cert.pem
4* found 0 certificates in /etc/ssl/certs
5* ALPN, offering http/1.1
6* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
7*        server certificate verification SKIPPED
8*        server certificate status verification SKIPPED
9* error fetching CN from cert:The requested data were not available.
10*        common name:  (does not match 'details')
11*        server certificate expiration date OK
12*        server certificate activation date OK
13*        certificate public key: RSA
14*        certificate version: #3
15*        subject: O=#1300
16*        start date: Thu, 26 Oct 2018 14:36:56 GMT
17*        expire date: Wed, 05 Jan 2019 14:36:56 GMT
18*        issuer: O=k8s.cluster.local
19*        compression: NULL
20* ALPN, server accepted to use http/1.1
21> GET /details/0 HTTP/1.1
22> Host: details:9080
23> User-Agent: curl/7.47.0
24> Accept: */*
25>
26< HTTP/1.1 200 OK
27< content-type: application/json
28< server: envoy
29< date: Thu, 07 Jun 2018 15:19:46 GMT
30< content-length: 178
31< x-envoy-upstream-service-time: 1
32< x-envoy-decorator-operation: default-route
33<
34* Connection #0 to host details left intact
35{"id":0,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"}

This proves the existence of mTLS between the services on the Istio mesh.

Now lets come out of the container before we go to the next section:

Copy
1exit

Secure Production Identity Framework for Everyone (SPIFFE)

Istio uses [SPIFFE](https://spiffe.io/) to assert the identify of workloads on the cluster. SPIFFE consists of a notion of identity and a method of proving it. A SPIFFE identity consists of an authority part and a path. The meaning of the path in spiffe land is implementation defined. In k8s it takes the form `/ns/$namespace/sa/$service-account` with the expected meaning. A SPIFFE identify is embedded in a document. This document in principle can take many forms but currently the only defined format is x509.

To start our investigation, let us check if the certs are in place in the productpage sidecar:

Copy
1kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- ls /etc/certs

Output will be similar to:

Copy
1cert-chain.pem
2key.pem
3root-cert.pem

Mac users, MacOS should have openssl available. If your machine does not have openssl install, install it using your preferred method.

Here is one way to install it on RHEL or CentOS or its derivatives:

Copy
1sudo yum install -y openssl-devel

Here is one way to install it on Ubuntu or Debian or its derivatives:

Copy
1sudo apt install -y libssl-dev

Now that we have found the certs, let us verify the certificate of productpage sidecar by running this command:

Copy
1kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout  | grep Validity -A 2

Output will be similar to:

Copy
1Not Before: Sep 23 17:32:28 2019 GMT
2    Not After : Dec 22 17:32:28 2019 GMT

Lets also verify the URI SAN:

Copy
1kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout  | grep 'Subject Alternative Name' -A 1

Output will be similar to:

Copy
1X509v3 Subject Alternative Name: critical
2    URI:spiffe://cluster.local/ns/default/sa/bookinfo-productpage