Layer5 and Istio

Recently, I started learning on Service Mesh and it was a very interesting journey as I explored the Service Mesh landscape starting with Layer5 tutorials and by exploring the blogs at Istio.io.

Our organization decided to use the features of Istio for securing, managing and automating microservices. However, we have to support a multi-tenant environment and that became a challenge due to lack of sufficient documentation and clarity. To give a little bit of background on the problem, we have a single cluster with multiple tenants in different namespaces sharing the same cluster . We do not want to provide each tenant their own separate Kubernetes Cluster because that would be additional provisioning overhead, management overhead and also additional resource overhead on the platform.

In this blog, I will go over the concepts and visual representation as well as the steps to learn and build your setup. Now there are different combinations possible depending on the requirements. These could be :

  1. Single Istio Control Plane with its own Ingress-Gateway to ensure traffic for the control plane is coming via the same control plane Ingress-Gateway.
  2. One or more Workspace namespaces for the workload applications and each with its own instance of the ingress gateway
  3. More advanced scenario would be multiple Control Plane and its associated Data Plane or essentially Multiple Service Mesh within a single Kubernetes Cluster. This is presently feasible using the Maistra Istio Operator, but its not fully supported using the upstream Istio at present. I intend to go over this use case in a later article.

Here by Ingress Gateway we mean the Istio Ingress Gateway and not the Kubernetes Ingress Gateway. Now, before we get into the steps for building an Istio Multi-tenant setup, lets quickly review the prerequisite :

  • Kubernetes setup (in the steps below I am using v1.18 on ubuntu 18.04)

    1$ sudo apt-get install -y docker.io
    2$ sudo sh -c “echo ‘deb http://apt.kubernetes.io/ kubernetes-xenial main’ >> /etc/apt/sources.list.d/kubernetes.list”
    3$ sudo sh -c “curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -”
    4$ sudo apt-get update
    5$ sudo apt-get install -y kubeadm=1.18.1–00 kubelet=1.18.1–00 kubectl=1.18.1–00
    6$ sudo kubeadm init — kubernetes-version 1.18.1 — pod-network-cidr 192.168.0.0/16
    7$ mkdir -p $HOME/.kube
    8$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    9$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
    10$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
  • Get the latest istioctl binary

    1$ curl -L https://git.io/getLatestIstio | sh -
    2cd istio-*
    3export PATH=$PWD/bin:$PATH
    4istioctl version
  • Initialize the Istio Operator

    1$ istioctl operator init
    2Using operator Deployment image: docker.io/istio/operator:1.6.3
    3✔ Istio operator installed
    4✔ Installation complete
    5$ kubectl get all -n istio-operator
    6NAME READY STATUS RESTARTS AGE
    7pod/istio-operator-5998f6c744-vrkbk 1/1 Running 1 78m
    8NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    9service/istio-operator ClusterIP 10.107.183.94 <none> 8383/TCP 78m
    10NAME READY UP-TO-DATE AVAILABLE AGE
    11deployment.apps/istio-operator 1/1 1 1 78m
    12NAME DESIRED CURRENT READY AGE
    13replicaset.apps/istio-operator-5998f6c744 1 1 1 78m

Next we will look into the details of the multitenancy scenarios; but before that a few word on the Istio operator would be nice. The Istio Operator follows the Kubernetes Controller and Custom Resource Definition mechanism and basically the above steps create an Istio operator controller in the istio-operator namespace and also registers the CRDs in the Kubernetes Registry. Basically, it extends the API for Kubernetes and allows management of a Custom Resource(CR). A word of caution, the existing upstream Istio Operator does not follow any Operator Framework like kubebuilder or Operator SDK. Hence, many of the expected behavior fail like trying to create two Istio Operator Custom Resource(the community mentions it as iop instances) in different namespaces or multiple iop instances and are under heavy development.

Ok, so now lets jump into action. We will build our Istio Setup for the scenario 1 &2 above shown in the diagram below. There are three different instances of Ingress Gateway(and it could be egress gateway as well) :

a. Ingress Gateway in Control Plane for traffic entry point to the Control plane Observability Components like Kiali(for the Service Mesh Graph), Prometheus(for metrics),Grafana(for visual charts of the metrics)and Jaeger (for distributed tracing between the services) b. Ingress Gateway in the Worskpace 1 namespace for traffic entry point to the workspace 1 microservices. c. Ingress Gateway in the Worskpace 2 namespace for traffic entry point to the workspace 2 microservices.

kubernetes-cluster

Single Control Plane with Multiple Data Plane each with separate Ingress Gateway

  • Creation of Control Plane CR and Data Plane CR A sample CR for creation of the Control plane and Data Plane CR can be referred below :
    1$ cat iop-platform.yaml
    2apiVersion: install.istio.io/v1alpha1
    3kind: IstioOperator
    4metadata:
    5namespace: istio-system
    6name: platform-istiocontrolplane
    7spec:
    8profile: demo
    9$ cat iop-wk1-gw.yaml
    10apiVersion: install.istio.io/v1alpha1
    11kind: IstioOperator
    12metadata:
    13namespace: istio-system
    14name: wk1-gwconfig
    15spec:
    16profile: empty
    17components:
    18ingressGateways:
    19— name: wk1-ingressgw
    20namespace: workspace-1
    21enabled: true
    22label:
    23istio: ingressgateway-1
    24app: istio-ingressgateway-1
    25values:
    26gateways:
    27istio-ingressgateway:
    28debug: error
    29$ cat iop-wk2-gw.yaml
    30apiVersion: install.istio.io/v1alpha1
    31kind: IstioOperator
    32metadata:
    33namespace: istio-system
    34name: wk2-gwconfig
    35spec:
    36profile: empty
    37components:
    38ingressGateways:
    39— name: wk2-ingressgw
    40namespace: workspace-2
    41enabled: true
    42label:
    43istio: ingressgateway-2
    44app: istio-ingressgateway-2
    45values:
    46gateways:
    47istio-ingressgateway:
    48debug: error

It is important to make sure the indentation is correct ,otherwise it would lead to errors. Reference: Istio Website

Deploy the CRs using kubectl:

1$ kubectl create ns istio-system
2$ kubectl create ns workspace-1
3$ kubectl create ns workspace-2
4$ kubectl apply -f ../iop-platform.yaml
5istiooperator.install.istio.io/platform-istiocontrolplane created
6$ kubectl apply -f ../iop-wk1-gw.yaml
7istiooperator.install.istio.io/wk1-gwconfig configured
8$ kubectl apply -f ../iop-wk2-gw.yaml
9istiooperator.install.istio.io/wk2-gwconfig configured
  • Troubleshooting the Istio Operator Controller. Its a good idea to run another window and execute the below command to observe the running logs on the controller pod.
    1$ kubectl logs -f -n istio-operator
    2$(kubectl get pods -n istio-operator -lname=istio-operator -o jsonpath=’{.items[0].metadata.name}’)
image
  • Observe the created objects in istio-system and the workspace namespaces for all pods to be in running state.

    1$ kubectl get all -n istio-system
    2NAME READY STATUS RESTARTS AGE
    3pod/grafana-b54bb57b9-k84xf 1/1 Running 0 79s
    4pod/istio-egressgateway-77c7d594c5-fr79j 1/1 Running 0 83s
    5pod/istio-ingressgateway-766c84dfdc-p6g6t 1/1 Running 0 83s
    6pod/istio-tracing-9dd6c4f7c-ndjvn 1/1 Running 0 79s
    7pod/istiod-7b69ff6f8c-9gwp8 1/1 Running 0 94s
    8pod/kiali-d45468dc4–4sww2 1/1 Running 0 79s
    9pod/prometheus-5fdfc44fb7–67khx 2/2 Running 0 78s
    10<truncated>
    11esudbat@istio:~/istio$ kubectl get all -n workspace-1
    12NAME READY STATUS RESTARTS AGE
    13pod/wk1-ingressgw-5674488c8b-fg2w5 1/1 Running 0 21s
    14<truncated>
    15esudbat@istio:~/istio$ kubectl get all -n workspace-2
  • Now we will label the namespaces for istio-injection

    1$kubectl label namespace workspace-1 istio-injection=enabled
    2$kubectl label namespace workspace-2 istio-injection=enabled
  • Now we will deploy the bookinfo sample application in both namespaces but before that we need to make sure that the Gateway resource is updated with the correct label selector as below :

    1$ cat istio/bookinfo-gateway-1.yaml
    2apiVersion: networking.istio.io/v1alpha3
    3kind: Gateway
    4metadata:
    5name: bookinfo-gateway
    6spec:
    7selector:
    8istio: ingressgateway-1 # use istio gateway-1 controller in workspace-1
    9servers:
    10— port:
    11number: 80
    12name: http
    13protocol: HTTP
    14hosts:
    15— “*”
    16$ cat istio/bookinfo-gateway-2.yaml
    17apiVersion: networking.istio.io/v1alpha3
    18kind: Gateway
    19metadata:
    20name: bookinfo-gateway
    21spec:
    22selector:
    23istio: ingressgateway-2 # use istio gateway-2 controller in workspace-1
    24servers:
    25— port:
    26number: 80
    27name: http
    28protocol: HTTP
    29hosts:
    30— “*”

Deploy the applications in workspace-1 and workspace-2 and deploy the Gateway,VirtualService and DestinationRules.

1$kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -n workspace-1
2$kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml -n workspace-2
3$kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml -n workspace-1
4$kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml -n workspace-2
5$kubectl apply -f samples/bookinfo/networking/destination-rule-all-mtls.yaml -n workspace-1
6$kubectl apply -f samples/bookinfo/networking/destination-rule-all-mtls.yaml -n workspace-2

Essentially what is happening is the Gateway CR is configuring the traffic to come via the ingress-gateway, it specifies the label selector based on which the ingress-gateway is selected and the host from where the traffic is allowed and port on which the traffic is allowed. It is important to ensure the label maps the label on the ingress-gateway. Once this is done the traffic will start flowing the intended gateway and it could be observed on the Kiali dashboard. Please note that in this scenario the Kiali is also accessed via the default control plane ingress-gateway running in the istio-system control plane namespace.

image

Kiali view for Workspace-1

image

Kiali view for Workspace-2

Shortly, we will go over the 3rd scenario which is to run multiple Istio Control Planes (Multiple Service Meshes) within the same Kubernetes Cluster. For that, we will need to build an open-shift setup and deploy the Maistra Istio Operator. Special thanks to the Istio Community for helping me understand the concepts and also answering my queries and of course to Lee Calcote, who helped me embark on my Istio journey.

- Sudeep Batra

Connect with Sudeep Batra on LinkedIn, GitHub, or Twitter.

Related Blogs

Layer5, the service mesh company

Representing the largest collection of service meshes and their maintainers in the world, Layer5 is the service mesh company. Creator and maintainer of service mesh standards. Maker of Meshery, the service mesh management plane.