Kubernetes Liveness Probe Failed: Connection Refused

Observed Problem

In Kubernetes, health checks are essential to ensure that your pods are running properly and are ready to serve traffic. However, network policies can sometimes unintentionally block these health checks, resulting in issues like:

Warning  Unhealthy  1m (x3 over 3m)  kubelet  Liveness probe failed: Get "http://<pod-ip>:8080/health": dial tcp <pod-ip>:8080: connect: connection refused

This type of error typically indicates that a network policy is preventing the kubelet from accessing the pod's health endpoint.

Analysis

Kubernetes uses health checks, such as readiness and liveness probes, to determine the state of a pod. These probes often run on specific HTTP endpoints or use TCP sockets. If network policies are too restrictive, they might block these critical health checks, causing the kubelet to consider the pod as unhealthy, leading to restarts or preventing traffic from being sent to the pod.

Common Issues

1. Ingress Policy Blocking Health Checks
   
   Health checks are often performed by the Kubernetes node, and if an ingress policy is overly restrictive, it can block access from the kubelet to the pod. This results in health check failures and the pod being marked as unhealthy.
   
   

2. Egress Policy Not Allowing Health Responses
   
   

   If egress policies are applied to a pod that needs to respond to a health check, it’s easy to forget to allow outgoing responses. This can lead to readiness and liveness probe failures.
   
   

3. Both Ingress and Egress Policies Enabled
   
   Having both ingress and egress policies enabled but not properly aligned can cause mismatches, leading to blocked communication. For example, an ingress rule might allow the health check request, but a restrictive egress rule might prevent the pod from responding, causing a timeout.
   
   

Resolution Workflow

1. Identify the Issue
   
   

   Check the pod events for messages like:

   Warning  Unhealthy  1m (x3 over 3m)  kubelet  Liveness probe failed: Get "http://<pod-ip>:8080/health": dial tcp <pod-ip>:8080: connect: connection refused

   
   

   This message indicates that the kubelet cannot reach the health endpoint, most likely due to a network policy restriction.
   
   

2. Test Connectivity to Health Endpoint
   
   

   Use kubectl exec to test connectivity from within the cluster:

   kubectl exec -it another-pod -- curl http://<pod-ip>:8080/health

      
   If the connection fails, it’s likely that a network policy is blocking the request.
   
   

3. Review and Update Network Policies
   
   

   • Ingress Policy Example: Update the ingress policy to allow traffic from the Kubernetes nodes (usually identified by IP blocks or node selectors) to the pod’s health check endpoint.

      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-healthcheck-ingress
      spec:
        podSelector:
          matchLabels:
            app: my-app
        policyTypes:
        - Ingress
        ingress:
        - from:
            - ipBlock:
                cidr: 10.0.0.0/8  # Adjust based on your cluster's node IP range
          ports:
          - protocol: TCP
            port: 8080  # Health check port

   
   

   • Egress Policy Example: Ensure that the pod can send responses back to the kubelet.

      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-healthcheck-egress
      spec:
        podSelector:
          matchLabels:
            app: my-app
        policyTypes:
        - Egress
        egress:
        - to:
            - ipBlock:
                cidr: 10.0.0.0/8  # Adjust based on your cluster's node IP range
          ports:
          - protocol: TCP
            port: 8080  # Health check port

   
   

4. Verify the Fix

• After applying the updated network policies, check the pod events again:

  kubectl describe pod <pod-name>

  
  

• Ensure that the health check messages are no longer showing errors.
  
  

• You can also verify by watching the pod status to see if it transitions to Running without any restarts.

How Otterize Can Help

Manually managing network policies for health checks can be challenging, especially in dynamic environments. Otterize’s Intents Operator and Network Mapper automate the creation and maintenance of these policies, ensuring that:

• Automatic Detection of Health Check Traffic: Otterize detects health check patterns and automatically adjusts network policies to allow kubelet traffic, preventing accidental blocks.
  
  

• Synchronization of Ingress and Egress: Otterize ensures ingress and egress policies are consistent, minimizing the risk of mismatched rules that lead to health check failures.
  
  

Ready to make network policies a breeze?

Stop stressing over network policy details. Let Otterize handle the heavy lifting with ClientIntents, and get back to focusing on your app’s real business.

Stay connected and learn more

If you found this article helpful, we'd love to hear from you! Here are some ways to stay connected and dive deeper into Otterize

🏠 Join our community

Stay ahead of the curve by joining our growing Slack community. Get the latest Otterize updates and discuss real-world use cases with peers.

🌐 Explore our resources

Take a sneak peek at our datasheets:

• Zero Trust and Microsegmentation

• PCI Compliance

• Cloud IAM automation

Or continue your exploration journey with our blogs and tutorials, or self-paced labs.

Don't be a stranger – your insights and questions are valuable to our community!