Chaos Engineering in Minikube

What Nobody Tells You

My key points

I was trying this template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: oom-stress
  namespace: testserver
spec:
  mode: fixed
  value: "3"
  selector:
    namespaces:
      - testserver
    labelSelectors:
      app: testserver
  stressors:
    memory:
      workers: 30
      size: "800MB"
    cpu:
      # Number of worker threads burning CPU
      workers: 12
      # Target CPU load percentage (0-100)
      # load: 100       # this is what pushes HPA over 50%
  duration: "5m"

Do you see a simple stress cpu experiment.

Running Chaos Mesh on minikube taught me 4 lessons.

  1. Containerd vs Docker

Chaos Mesh requires containerd. Docker runtime thorws expected containerd:// but got docker:// and blocks the injection of the cpu stress silently.

  1. cgroup attribution

XYZ-ng workers inject succesfully but CPU never appears in metrics. Minikb’s cgroup accounting does not attribute consuptiom to the pod correctly.

  1. Always describe your chaos

kubectl describe stresschaos mychaos is your truth. AllInjected: True means it worked. Failed or Warning events mean your runtime is wrong.

  1. Metrics server is everything

without it your HPA is blind. It means no scaling ever happens, regardless of how much chaos is inject.

Conclusion

I searched solution for the error in github and then i got many issues opened so I will move to another tool to prove the autoscaling and cpu stress experiment and then see the result.

Offtopic

Lastly, if you have a dog, please remember to take good care of them.🐶

comments powered by Disqus
Built with Hugo
Theme Stack designed by Jimmy