DNS resolution failures in Kubernetes are often caused by network policies that block DNS requests. A common oversight is forgetting to allow egress traffic for DNS in the network policy, which prevents pods from reaching the Kubernetes DNS service (usually CoreDNS) on port 53. Without DNS, service discovery fails, leading to connection issues.

Error Messages You Might See

When DNS resolution is blocked, you might see errors like: "Could not resolve host: example.com"

Or in application logs:

Error: Get "http://<service-name>": dial tcp: lookup <service-name> on 10.96.0.10:53: no such host

These errors indicate that the DNS service is unreachable, preventing the application from resolving service names.

Why Is This Happening?

In Kubernetes, DNS resolution is handled by CoreDNS, which listens on port 53 for both UDP and TCP traffic. If your network policy doesn’t explicitly allow egress traffic to port 53, pods will be unable to resolve any domain names, resulting in connection failures when trying to reach internal or external services.

How to Fix DNS Resolution Failures Due to Network Policies

To resolve the issue, you need to adjust your network policies to allow DNS traffic. 

Here’s a step-by-step guide:

1. Check Existing Network Policies: Review the existing network policies to determine if egress traffic to DNS is allowed. If it’s not, this is likely the root cause of the DNS failures.
   
   

2. Test DNS Connectivity: Run a DNS lookup from the affected pod to confirm the issue:
   

   kubectl exec -it my-app-pod -- nslookup example.com

   
   If the command fails, it indicates that DNS traffic is being blocked.

   If the pod is running a distroless image without a shell, use kubectl debug to spin up a debug container and test again:

   kubectl debug my-app-pod -it --image=ubuntu --target=my-app-containerapt-get update && apt-get install -y dnsutils

   
   

   And run the command:

   nslookup example.com

   
   

3. Review and Update Network Policies: Check the current network policy. Here’s an example of a problematic policy that blocks DNS traffic:
   
   

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: restrict-egress
   spec:
     podSelector:
       matchLabels:
         app: my-app
     policyTypes:
     - Egress
     egress:
     - to:
         - podSelector:
             matchLabels:
               app: another-app
       ports:
         - protocol: TCP
           port: 8080

   
   This policy is too restrictive and does not allow egress to CoreDNS.
   
   

4. Fix the Network Policy: Update the network policy to allow DNS traffic.
   
   Here’s a corrected version:
   
   

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: allow-dns-egress
   spec:
     podSelector:
       matchLabels:
         app: my-app
     policyTypes:
     - Egress
     egress:
     - to:
         - namespaceSelector:
             matchLabels:
               kubernetes.io/metadata.name: kube-system
           ports:
           - protocol: UDP
             port: 53
           - protocol: TCP
             port: 53
     - to:
         - podSelector:
             matchLabels:
               app: another-app
       ports:
         - protocol: TCP
           port: 8080

   
   This policy allows egress DNS requests to CoreDNS running in the kube-system namespace on port 53, ensuring that DNS resolution works as expected.
   
   

5. Verify the Fix: Test DNS resolution again:

kubectl exec -it my-app-pod -- nslookup example.com

The command should succeed, confirming that DNS traffic is now allowed.

Monitor DNS Traffic with tcpdump

To further ensure that DNS traffic is flowing correctly, you can use tcpdump to monitor the traffic:

1. Run an ephemeral container that shares the host network:

   kubectl debug node/<node-name> -it --image=ubuntu

   
   

2. Update and install tcpdump:

   apt-get update && apt-get install tcpdump -y

   
   

3. Use tcpdump to monitor DNS traffic:

   tcpdump -i any src <my-app-pod-ip> and port 53

Replace <my-app-pod-ip> with the IP address of the affected pod.

Automate Network Policies with Otterize

Manually managing network policies can be challenging, especially in dynamic environments. 

Otterize's Intent Operator and Network Mapper can help automate the creation and management of these policies, ensuring your pods have the connectivity they need without manual tweaking.

Here’s a quick tutorial on how to deploy Otterize and start creating network policies, and a longer and detailed guide on mastering cloud-native packets flows in your cluster.


Ready to make network policies a breeze?

Stop stressing over network policy details. Let Otterize handle the heavy lifting with ClientIntents, and get back to focusing on your app’s real business.

Stay connected and learn more

If you found this article helpful, we'd love to hear from you! Here are some ways to stay connected and dive deeper into Otterize.

🏠 Join our community

Stay ahead of the curve by joining our growing Slack community. Get the latest Otterize updates and discuss real-world use cases with peers.

🌐 Explore our resources

Take a sneak peek at our datasheets:

• Zero Trust and Microsegmentation

• PCI Compliance

• Cloud IAM automation

Or continue your exploration journey with our blogs and tutorials, or self-paced labs.

Don't be a stranger – your insights and questions are valuable to our community!