Circuit Breaking

In this chapter we will configure circuit breaking using Istio. Circuit breaking allows developers to write applications that limit the impact of failures, latency spikes, and other undesirable effects of network peculiarities. This task will show how to configure circuit breaking for connections, requests, and outlier detection.

Preparing for circuit breaking

Before we can configure circuit breaking, please try to access the product page app from within Meshery to ensure all the calls are making it through without errors as we did in Observability chapter (see screenshot).

Configure circuit breaking

Now that we have the needed services in place, it is time to configure circuit breaking using a destination rule.

Using Meshery, navigate to the Istio management page:

1. Enter default in the Namespace field.
2. Click the (+) icon on the Apply Custom Configuration card and paste the configuration below.

Manual step for can be found here

This will update the existing destination rule definition for product page service to break the circuit if there are more than one connection and more than one pending request.

In a few, we should be able to verify the destination rule by using the command below:

kubectl get destinationrule productpage -o yaml

Config:

1apiVersion: networking.istio.io/v1alpha3
2kind: DestinationRule
3metadata:
4  name: productpage
5spec:
6  host: productpage
7  subsets:
8    - /obserlabels:
9        version: v1
10      name: v1
11  trafficPolicy:
12    connectionPool:
13      http:
14        http1MaxPendingRequests: 1
15        maxRequestsPerConnection: 1
16      tcp:
17        maxConnections: 1
18    outlierDetection:
19      baseEjectionTime: 3m
20      consecutiveErrors: 1
21      interval: 1s
22      maxEjectionPercent: 100
23    tls:
24      mode: ISTIO_MUTUAL

Time to trip the circuit

In the circuit-breaker settings, we specified maxRequestsPerConnection: 1 and http1MaxPendingRequests:

1. This should mean that if we exceed more than one request per connection and more than one pending request, we should see the istio-proxy sidecar open the circuit for further requests/connections.

Let us now use Meshery to make several calls to product page app by changing the number of concurrent connections to 5 from within Meshery's Performance page.

Once you have updated the fields, you now click on Run Test.

This will run the load test and show the results in a chart. ( see screenshot ).

You should only see a percentage of the requests succeed and the rest trapped by the configured circuit breaker.

Manual Steps

Configure circuit breaking

1kubectl apply -f - <<EOF
2apiVersion: networking.istio.io/v1alpha3
3kind: DestinationRule
4metadata:
5  name: productpage
6spec:
7  host: productpage
8  subsets:
9  - labels:
10      version: v1
11    name: v1
12  trafficPolicy:
13    tls:
14      mode: ISTIO_MUTUAL
15    connectionPool:
16      tcp:
17        maxConnections: 1
18      http:
19        http1MaxPendingRequests: 1
20        maxRequestsPerConnection: 1
21    outlierDetection:
22      consecutiveErrors: 1
23      interval: 1s
24      baseEjectionTime: 3m
25      maxEjectionPercent: 100
26EOF