Fault Injection

In this chapter we will learn how to test the resiliency of an application by injecting systematic faults. Before we start, we will need to reset the virtual services.

Using Meshery, navigate to the Istio management page:

1. Enter default in the Namespace field.
2. Click the (+) icon on the Apply Custom Configuration card and paste the configuration below.

Config:

Copy
1apiVersion: networking.istio.io/v1alpha3
2kind: VirtualService
3metadata:
4  name: reviews
5spec:
6  hosts:
7    - reviews
8  http:
9    - route:
10        - destination:
11            host: reviews
12            subset: v2

Inject a route rule to create a fault using HTTP delay

To start, we will inject a 7s delay for accessing the ratings service for a user jason. reviews v2 service has a 10s hard-coded connection timeout for its calls to the ratings service configured globally.

Using Meshery, navigate to the Istio management page:

1. Enter default in the Namespace field.
2. Click the (+) icon on the Apply Custom Configuration card and paste the configuration below.

This will update the existing virtual service definition for ratings to inject a delay for user jason to access the ratings V1.

In a few, we should be able to verify the virtual service by using the command below:

Copy
1kubectl get virtualservice ratings -o yaml

Config:

Copy
1apiVersion: networking.istio.io/v1alpha3
2kind: VirtualService
3metadata:
4  name: ratings
5spec:
6  hosts:
7    - ratings
8  http:
9    - fault:
10        delay:
11          fixedDelay: 7s
12          percentage:
13            value: 100
14      match:
15        - headers:
16            end-user:
17              exact: jason
18      route:
19        - destination:
20            host: ratings
21            subset: v1
22    - route:
23        - destination:
24            host: ratings
25            subset: v1

Now we login to /productpage as user jason and observe that the page loads but because of the induced delay between services the reviews section will show :

Error fetching product reviews!Sorry, product reviews are currently unavailable for this book.

If you logout or login as a different user, the page should load normally without any errors.

Inject a route rule to create a fault using HTTP abort

In this section, , we will introduce an HTTP abort to the ratings microservices for user jason.

Using Meshery, navigate to the Istio management page:

1. Enter default in the Namespace field.
2. Click the (+) icon on the Apply Custom Configuration card and paste the configuration below.

This will update the existing virtual service definition for ratings to inject a HTTP abort for user jason to access the ratings V1.

In a few, we should be able to verify the virtual service by using the command below:

Copy
1kubectl get virtualservice ratings -o yaml

Config:

Copy
1apiVersion: networking.istio.io/v1alpha3
2kind: VirtualService
3metadata:
4  name: ratings
5spec:
6  hosts:
7    - ratings
8  http:
9    - fault:
10        abort:
11          httpStatus: 500
12          percentage:
13            value: 100
14      match:
15        - headers:
16            end-user:
17              exact: jason
18      route:
19        - destination:
20            host: ratings
21            subset: v1
22    - route:
23        - destination:
24            host: ratings
25            subset: v1

Now we login to /productpage as user jason and observe that the page loads without any new delays but because of the induced fault between services the reviews section will show:

Ratings service is currently unavailable.

Verify fault injection

Verify the fault injection by logging out (or logging in as a different user), the page should load normally without any errors.

Alternative: Manual installation

Follow these steps if the above steps did not work

Route all traffic to version V1 of all services

Copy
1kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

Route all traffic to version V2 of reviews for user Jason

Copy
1kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml

Inject 7s delay for ratings service

Copy
1kubectl apply -f samples/bookinfo/networking/virtual-service-ratings-test-delay.yaml

Inject HTTP abort for ratings service

Copy
1kubectl apply -f samples/bookinfo/networking/virtual-service-ratings-test-abort.yaml