Got this error yesterday when deploying an activemq cluster. The behavior is, the activemq POD keeps in WAITING status.
kubectl describe pod <POD NAME> got the following errors:
0/3 nodes are available: 1 Insuffcient PODS, node affinity conflict.
After a few hours of investigation, finally found out that this issue is caused by that:
- The POD is dependent on PVC. By checking PVC / PV config (`kubectl describe pvc / kubectl describe pv`), the PV is created at AWS AZ us-east-1b;
- The k8s cluster, has 3 nodes. And one node is on us-east-1b. This means that the POD MUST be scheduled to this node (POD cannot access the PVC from different AZ).
- However, there are already 58 PODS running on this node. For the AWS limit, this node cannot accept more PODs. If it is a normal POD (Without PVC) then it can be scheduled to other nodes. However, due to (2), this POD is dependent on the PVC, thus it MUST run on this node (Node affinity).
A good doc tells you the limit of PODs for each EC2 Instance : https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt
To check the which type of EC2 is used for nodes,
kubectl describe node <node> can be used.
SOLVE: After removing unused PODs from this node, the WAITNING pod can start. And we are waiting for IT guys to create another node under same AZ.