Lab Exercise #c - Orchestration Cluster Operations and Resilience

Let's solve some more things to better understand the Camunda 8 orchestration cluster operations.

Q. The current orchestration cluster topology has 3 broker nodes with 3 partitions.
Cluster topology is one of the key decisions made by an Architect taking into account the non-functional requirements.
Visit Landing Page -> Topology to see the current topology and partition distribution.

Let's plan to increase the partition count to 5.

Note: Partition count can be increased upwards. However, it cannot be reduced to a lower number.

Show Answer

Answer:


# Start in GCP Cloud Shell
export UNAMESPACE={{UNAMESPACE}}


gcloud container clusters get-credentials {{CLUSTER_NAME}} --region {{REGION}}  --project c8-labs


# run this in background
kubectl port-forward "svc/{{UNAMESPACE}}-zeebe-gateway" 9600:9600 -n {{UNAMESPACE}} &
# You may dry run first to preview the partition distribution by appending ?dryRun=true to the request
curl -X 'PATCH' \
   'http://localhost:9600/orchestration/actuator/cluster' \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -d '{
        "partitions": {
          "count": 5,
          "replicationFactor": 3
        }
      }'

# Wait for a minute or so. Check the status using below. 
# When complete, the lastChange status will show "COMPLETED" (this appears in the last two lines of the output json)
curl --request GET 'http://localhost:9600/orchestration/actuator/cluster'


# Lastly, lets stop the port forward
pkill -f "kubectl port-forward"

Post Command-Set Execution: Click to view screen shot - How to check status COMPLETED?

Click on image to enlarge
Cloud Shell

Visit again Landing Page -> Topology to see the new topology and partition distribution.
Tip: Use Chrome browser for better visualisation of JSON object.

Q. Resilient platform operations is one of the key ingredients of the Camunda 8 platform architecture.

                    General IT operations view
                    
                        In the tech operations world, a single node failing is a normal Tuesday, but two nodes failing at the very same time is incredibly rare.
                        
                        Statistically speaking, this is because independent failures follow a rule of multiplication. 

                        If the chance of one node crashing today is 1 in 1,000, the chance of two specific nodes crashing on the exact same day is 1 in 1,000,000 (one in a million).

Let's do some work to disrupt the broker nodes and see the resiliency in action.

Show Answer

Answer:

Option #1. Evict zeebe-0 pod (or any other zeebe pod) and check the outcome.
Option #2. Simulate hardware failure. Evict a cluster node (VM node) from the Kubernetes cluster. This can be done during live sessions only. (not applicable for self-paced subscription)


# Start in GCP Cloud Shell
gcloud config set project c8-labs
export UNAMESPACE={{UNAMESPACE}}


gcloud container clusters get-credentials {{CLUSTER_NAME}} --region {{REGION}} --project c8-labs


# Evict the pod from Project Lens or via kubectl command
export POD_NAME=$(kubectl get pods -n {{UNAMESPACE}} | grep zeebe-0 | cut -f 1 -d " ")


# Let it be re-added by the system. It may take a while.
kubectl delete pod -n {{UNAMESPACE}} $POD_NAME

Pick an earlier failed process instance via Operator and retry for another attempt.
Are you able to continue with transaction(s) while the rejig is happening in the background?


## Option 2 - Run by the Session Administrator
# Start in GCP Cloud Shell
gcloud config set project c8-labs
export UNAMESPACE={{UNAMESPACE}}


gcloud compute instances delete vm-node-name  --zone=zone-id

Pick an earlier failed process instance via Operator and retry for another attempt.
Are you able to continue with transaction(s) while the rejig is happening in the background?

Next: Lab Exercise #d Orchestration Engine