Connection time out in Kubernetes: it's not what you think!

The “Connection timed out” error might not mean what you think it does. In Kubernetes, it's usually not that your service is broken—it's more likely that network policies (NetPols) are playing hide and seek with your traffic.

If the service didn’t exist, you’d see something like a "DNS NXDOMAIN" error (the DNS name doesn’t exist), or a "connection refused" message (the service exists, but the port isn’t listening). But if you're getting "connection timed out," then something’s blocking your connection—and that something is probably a misconfigured network policy.

Why is this happening?

Network Policies in Kubernetes are the traffic lights that determine what traffic can go through your services. They’re great for securing your cluster but can be a pain when they’re too strict or out of date. Most likely, when you see a timeout, it’s because they’re blocking legitimate traffic that should be allowed.

Network Policies aren’t magical. They don’t automatically adjust when you add new services or change how your apps work. This can result in:

• Legit communication being blocked because policies haven’t caught up with changes

• Security policies that go overboard, leaving your services unable to talk

• Overly lax policies that let everyone in, making your cluster vulnerable

Trying to manually keep network policies updated at scale with all the changes happening in your cluster can be exhausting and easy to get wrong. But the good news is that you can do something about it!

What does the error look like?

Here’s how a connection timeout might appear in different programming languages:

Node.js

Error: connect ETIMEDOUT at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1125:14)

Java

java.net.SocketTimeoutException: connect timed out

Python

socket.timeout: timed out

Go

dial tcp: i/o timeout

How to get past the timeout block

If you’re staring at a connection timeout error, chances are your network policies need a little TLC (tender loving care - if you don’t know the acronym 😉). Here’s what to do:

1. Check Existing Network Policies: Make sure the server pod isn’t covered by a policy that’s too restrictive or missing rules that allow incoming traffic from your client.
   
   

2. Add or Adjust Network Policies: Update your network policies to explicitly allow ingress traffic to the server and egress from the client.
   
   

3. Test Connectivity Using kubectl exec and netcat

To verify if the connection is being blocked by a network policy, you can use kubectl exec to test the connection directly from the source pod:

kubectl exec -it my_source_pod -- nc -zv my_svc 8080

• Replace my_source_pod with the name of the pod acting as the source.

• Replace my_svc with the name of the destination service.

• Replace 8080 with the destination service port.

If no shell is available in the container (distroless image) or netcat is not installed, you can use kubectl debug to achieve the same:

kubectl debug my_source_pod -it --image=ubuntu --target=my_source_container apt-get update && apt-get install -y netcat 

And run the command:

$ nc -zv my_svc 8080

Monitor traffic using tcpdump

To monitor the network traffic and see if packets are being blocked, you can use tcpdump on the host node. This is useful for catching traffic at the kube-proxy level and diagnosing issues:

1. Run an ephemeral container that shares the host network and has access to the host filesystem:

   kubectl debug node/<node-name> -it --image=ubuntu

   
   

2. Update the package sources and install tcpdump and iproute2:

   apt-get update && apt-get install tcpdump iproute2 -y

   
   

3. Use tcpdump to monitor traffic from the source pod IP to the destination service:

   tcpdump -i any src <source-pod-ip> and dst <destination-ip> and port 8080

• Replace <source-pod-ip> with the IP address of the source pod.

• Replace <destination-ip> with the IP address of the destination service.

• Replace 8080 with the relevant port.
  
  

Problematic network policy example

This policy protects the server pod but doesn’t have a rule allowing traffic from the client—hence the timeout:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-server
spec:
  podSelector:
    matchLabels:
      app: server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}

Once you have created a network policy for a target service, each source client that needs to access that destination service also needs a matching network policy, that is the trick here!

Fixed network policy example

In this corrected version, the policy allows traffic from the client pod with the right label:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-client-to-server
spec:
  podSelector:
    matchLabels:
      app: server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: client

In real life, network policies get a lot more complicated because they are not the right abstraction when developing microservices for Kubernetes, as explained in this article.

Making life easier with Otterize

Let’s be real—keeping network policies up to date can be a huge headache. Luckily, Otterize’s Intents Operator and Network Mapper are here to save the day.

Otterize can automatically create and manage network policies that evolve as your app does, so you don’t have to worry about unexpected connection issues or spend hours troubleshooting them.

Here’s a quick tutorial on how to deploy Otterize and start creating network policies, and a longer and detailed guide on mastering cloud-native packets flows in your cluster.

Ready to make network policies a breeze?

Stop stressing over network policy details. Let Otterize handle the heavy lifting with ClientIntents, and get back to focusing on your app’s real business.

Stay connected and learn more

If you found this article helpful, we'd love to hear from you! Here are some ways to stay connected and dive deeper into Otterize

🏠 Join our community

Stay ahead of the curve by joining our growing Slack community. Get the latest Otterize updates and discuss real-world use cases with peers.

🌐 Explore our resources

Take a sneak peek at our datasheets:

• Zero Trust and Microsegmentation

• PCI Compliance

• Cloud IAM automation

Or continue your exploration journey with our blogs and tutorials, or self-paced labs.

Don't be a stranger – your insights and questions are valuable to our community!