Breaking bad policies: Crafting perfect Istio authorization policies and ingress authentication with Otterize

In this article, we're going to learn about Istio AuthorizationPolicies. If you're new to Kubernetes but already somewhat familiar with NetworkPolicy resources, you'll feel right at home. Perhaps not necessarily in your own home, but maybe as comfortable as you can be in a neighbor’s home! However, don't be too intimidated by acronyms like mTLS, JWT, OAuth2, or JWKS. You'll need to understand what they mean sooner rather than later, but the dual purpose of this article is to introduce you to the basics of Istio authorization policies and how they function, as well as to propose an innovative approach using Otterize Intent-Based Access Control (IBAC) and client intents to configure them. In the context of Istio, Otterize can suggest and generate client intents by directly pulling connection metrics from Envoy proxies. What more could we do? Yes, I know! I’ll walk you through that in the second part of the article.

But first, let’s revisit some of the fundamentals.

Istio Fundamentals

The Kubernetes networking model requires that pods can directly communicate with each other using their actual IP addresses. In other words, Kubernetes acts as an open bar once traffic enters the cluster, whether it’s directly routed via IP or encapsulated in the physical network. For security aficionados, that effectively means that Kubernetes is not secured by default and this is NOT the way, dear padawan! 



Luckily, Istio is coming to help, but the learning curve may be intimidating. Let’s break down the most common pitfalls and navigate through concrete use cases and explore how we can automate policies and leverage Envoy information for better visibility.

How can you best navigate Istio's complexity to maximize its security benefits?

Platform admins can secure clusters by creating Kubernetes network policies or using a service mesh like Istio. This introduces powerful new capabilities, but does Istio's complexity outweigh the benefits of its abstractions?

Kubernetes is well-known for its ability to abstract low-level concepts and expose more user-friendly interfaces. These principles also apply to Istio constructs, ensuring that direct management of IP addresses, dealing with authentication, and other application-specific libraries are not necessary unless desired. But this flexibility comes with the other side of the coin. While Istio offers a dynamic and idiomatic way to infuse security and application services into Kubernetes workloads, it also extends the Kubernetes object models with new paradigms that may initially be intimidating. The nuanced nature of authorization policies often leads to common mistakes, which can compromise security and cause unexpected behavior in your microservices ecosystem. 

Understanding Istio Authorization Policies

Authentication and authorization, or simply put, 'who is accessing resources' and 'what resources can be accessed,' play a pivotal role in Istio. Istio Authorization policies are custom resources that encapsulate both concepts into a single object, referencing the identity of a user or workload along with the intent of communication. This allows for more fine-grained control over access to resources, accommodating complex scenarios that traditional access control models, such as Kubernetes network policies, limit to L3-L4.

How does Istio manage identities, authentication, and authorization?

When it comes to establishing service identity, Istio leverages various methods:

- End-user (Ingress or User-to-Mesh): Istio typically checks JWTs (JSON Web Tokens) in HTTP headers to verify the credentials attached to the request using RequestAuthentication.

- Internal Mesh Workload: Kubernetes Service Account (SA) is commonly utilized to identify a calling service. This identity is represented by the source principal in the format: <cluster-domain>/ns/<namespace>/sa/<service-account>.

- PeerAuthentication enforces mTLS or a more permissive approach within the mesh.

In addition to these mechanisms, Istio maintains workload identity by injecting certificates and private keys in the Envoy proxies running as sidecar containers in every Kubernetes Pod, also allowing for mutual TLS (mTLS) bidirectional communication. These certificates, along with Kubernetes service accounts and integration with OAuth2 and JWT, serve as a strong authentication framework.

Authorizations can then be crafted by creating rules that define the source and destination of the request (identity), specify authorized operations on named resources, and optionally include extra conditions.

In fact, the simplest way to understand the capabilities of authorization policies is to use kubectl explain to check the specification schema of the custom resource:

$ kubectl explain authorizationpolicies.security.istio.io.spec --recursive
FIELDS:
  action        <string>
  provider      <Object>
    name        <string>
  rules <[]Object>
    from        <[]Object>
      source    <Object>
        ipBlocks        <[]string>
        namespaces      <[]string>
        notIpBlocks     <[]string>
        notNamespaces   <[]string>
        notPrincipals   <[]string>
        notRemoteIpBlocks       <[]string>
        notRequestPrincipals    <[]string>
        principals      <[]string>
        remoteIpBlocks  <[]string>
        requestPrincipals       <[]string>
    to  <[]Object>
      operation <Object>
        hosts   <[]string>
        methods <[]string>
        notHosts        <[]string>
        notMethods      <[]string>
        notPaths        <[]string>
        notPorts        <[]string>
        paths   <[]string>
        ports   <[]string>
    when        <[]Object>
      key       <string> -required-
      notValues <[]string>
      values    <[]string>
  selector      <Object>
    matchLabels <map[string]string>
  targetRef     <Object>
    group       <string>
    kind        <string>
    name        <string>
    namespace   <string>

Key Configuration Components for Authorization Policies

AuthorizationPolicies allow administrators to clearly define, in a simple and declarative manner, who holds the authorization to execute specific actions within the mesh environment. It’s worth noting that in the absence of any authorization policy, the Kubernetes networking  model remains open to all incoming traffic if no network policy has been defined.

To better understand how authorization policies work, let's examine the critical components that allow them to accept or deny traffic.

The target namespace plays a critical role. When it’s set to the Istio system namespace, typically istio-system, the policies affect the entire service mesh. 

The selector field identifies the destination of the traffic and specifies the resources to which the policies will be applied. If the spec.selector field is empty, the policies apply to all workloads in the namespace. 

The other key components live in the .rule’s  structure:

- .from.source is a list that defines the potential sources of the incoming request. IP blocks can be used, as well as namespaces and the service identities (principals) we mentioned in the previous section. This field requires mTLS enabled.

- .to.operation.hosts is a list of hosts as specified in the HTTP request. It is typically used on Istio gateway for external traffic entering the mesh and not on sidecars for traffic within the mesh. Also, prefix matches should be used rather than exact domain DNS names. For example, use [“my-domain.com”, “my-domain.com:*”] rather than just [“my-domain.com”]. This way, the policy matches the host and all corresponding ports. 

- .to.operation.methods defines a list of HTTP methods specified in the request. If not set, any method is allowed.

- .to.operation.paths defines a list of HTTP paths as specified in the request. If not set, any path is allowed.

- .to.operation.ports defines a list of ports as specified in the connection. If not set, any port is allowed.

- .when defines a list of conditional attributes. The supported attributes include:

• - The HTTP headers names and values

• - The  client source IP address as determined by the X-Forwarded-For header

• - The principal or issuer of the authenticated JWT token

• - The server name indication (SNI)

• - Many more…

Policy evaluation

When evaluating authorization policies for a particular workload, Istio applies them additively and proceeds in a hierarchical order, considering actions first.

- Istio first considers if any deny policy matches the request. If there’s a match, the request is denied.

- If there is at least one allow policy matching the request, and no matching deny policies, the request is allowed.

- If an allow policy is applied but the request doesn't match any of its specifications, the request is denied. This also means that creating an allow policy for a target service automatically implies a default deny for requests that don’t match any of the rules.

- If there is no allow policy applied, the request is allowed.

External Authentication

Istio acts as a security gatekeeper by integrating with external authentication providers that utilize OAuth2 or OIDC protocols. The client receives a JSON Web Token after following an authentication workflow at the edge of the mesh, typically via the Istio ingress gateway routing the request to an internal authentication service. This service implements the integration with the external identity provider (IdP) and returns the JWT.

If an HTTP request matches an Istio RequestAuthentication, Istio verifies that the request matches the JWT rules present in the policy. An additional authorization policy must be attached to the ingress gateway, which explicitly restricts the request principals authenticated in accordance to these rules.

Here’s an example of an RequestAuthentication defined on an Istio ingress gateway:

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: order-validate-jwt
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "https://accounts.google.com"
    jwksUri: "https://www.googleapis.com/oauth2/v3/certs"

In this example, the jwtRules define the expected issuer and the location of the JWT Web Key Sets (JWKS). These JWKS structures contain the public keys needed to verify the JWT issued by the identity provider.

The following AuthorizationPolicy must also be attached to the Istio ingress gateway:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: authenticated-to-gw
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["*"] # Matches any authenticated principal outside the cluster
    when:
    - key: request.auth.claims[iss]
      values: ["https://accounts.google.com"]

This configuration guarantees that only the clients authenticated by the Google OIDC provider will be able to reach the destination service within the mesh.

IBAC and client intents

Client intents are simply a list of calls to services that a client intends to make. This list of client intents can be used to configure different authorization mechanisms such as network policies, Istio authorization policies, cloud IAM, database credentials, and their corresponding permissions. In other words, developers declare what their service intends to access, and in the case of Istio, this declaration can then be converted into an authorization policy along with the associated set of pod labels. 

But here’s the trick: creating client intents doesn't automatically make the resulting security rule an egress rule. The two concepts are not directly correlated. In fact, it’s the opposite—they typically generate ingress rules, which are service-centric. Client intents are a practical way for developers to declaratively express the intended connections between a client and a server, to reduce friction between developers, platform teams, and security teams. It’s like having an Application Dependency Mapping (ADM) that outlines the connections your application needs to function properly—except developers provide the information. Alternatively, you can let Otterize discover these dependencies for you (more on this later).

Here's an example of a client intents file (as a Kubernetes custom resource YAML manifest) for a service named "client" that has network access to another service named "auth-server" and to the metrics database of "production-db":

apiVersion: k8s.otterize.com/v1alpha2
kind: ClientIntents
metadata:
  name: client-intents

spec:
  service:
    name: client
  calls:
    - name: auth-server
    - name: production-db
      type: database
      databaseResources:
        - databaseName: metrics

Otterize utilizes a combination of IBAC and contextual mechanisms to automate policy generation. Client intents, expressed in high-level, human-readable language, define the desired security posture. This abstraction layer separates security requirements from the complexities of network and authorization policies. By focusing on the 'what' (desired security outcomes) rather than the 'how' (specific IAM and security configurations), Otterize simplifies the process, allowing security to scale effectively alongside the development of complex distributed systems.

How Otterize helps developers while keeping SecOps happy

Authorization policies can be difficult to figure out, and honestly, developers don't really care about Istio—they care about their code and getting it shipped to production quickly. To speed things up, it's easier to add security rules directly alongside the code. What developers really need to improve their delivery process is a way to describe the connection patterns between their application components. Then, the platform team can polish them up as needed, use another admission controller like Kyverno for more fine-grained control over what is deployed in production, or just deploy them as is, ensuring security is baked in without slowing anyone down.

Let’s dig deeper into this by taking a look at the following example.

Putting it all together

We’ll configure the following environment:

Application Description

The application code can be found on this repository, under “istio-authorization-policies”.

We consider an online-shop microservices app simulation, where every service is returning a basic message, like { “message”: “I’m the X service” }. 

The order service provides the following:

- An authentication service at /login using a Google OIDC identity provider. This service returns a valid JWT identifying the caller.
- A callback function used by the IdP as a redirection URL to return the token at /callback.
- An /order endpoint.

Other services in the application:

- The frontend service runs the /frontend endpoint.
- The cart service runs the /cart endpoint.
- The product service runs the /product endpoint.

Rules for allowed operations:

- The frontend service is allowed to perform HTTP GET operations on all other services.
- External users are allowed to access /login and /callback on the ingress gateway.
- Once identified with a valid JWT, external users can call /order.

Objectives

This example has 2 main objectives:

- To walk you through the request authentication workflow for authenticating and authorizing external clients (a task beyond the scope of Otterize at the time of writing).

- To demonstrate how the Otterize Network Mapper pulls metrics from Envoy to automatically generate client intents, and teams up with the Intents Operator to manage the lifecycle of Istio authorization policies.

Prerequisites

- Istio deployed in demo mode (an egress and ingress gateway deployments are provisioned by default).

- The online-shop namespace must be labeled with istio-injection=enabled to enable Istio in the namespace and allow for Envoy sidecar deployment.

- A Gateway must be configured in the istio-system namespace.

- A Virtual Service must be created for the ingress gateway to route the traffic to the order service.

- OAuth 2.0 Client Credentials for a Google OIDC Application.

- The /order service is accessible from outside the mesh (via a node port or a load-balancer).

Note: You can find the manifest for the Gateway and the Virtual Service as well as all other configuration elements in the application repository, under istio-authorization-policies/online-shop/kubernetes-manifests.yaml

Request authentication workflow with the Istio ingress gateway

Let’s describe the different steps involved in the diagram above.

1. A user initiates the process by connecting to the public Fully Qualified Domain Name (FQDN) of the order service at the HTTP /login endpoint: 

$ curl http://order-service/login

This request is authorized by the following AuthorizationPolicy applied on the ingress gateway:

external-to-login-callback.yaml

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: external-to-login-callback
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: ALLOW
  rules:
  - to:
    - operation:
        paths: [/login, /callback]

The ingress gateway routes the request to the order service, which initiates the authentication workflow. This involves opening a browser window and directing the user to the sign-in URL for the Google OIDC identity provider:

$ curl http://order-service/login
Go to URL: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=xxxxxx.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%2Fcallback&response_type=code&scope=openid+email&state=uIUQ7clLouAMl0RsnAZn_Q%3D%3D

2. The user navigates to the provided link and signs in using their Google OIDC credentials.

3. The Google OIDC process then redirects the user to the callback function routed by the ingress gateway, which is permitted by the same authorization policy described in step 1.

4. The callback function extracts the user JWT from the response and sends it to the browser.

The user can now make authenticated requests to the order service by attaching the valid JWT to the requests using the authorization header:

$ export TOKEN=<JWT_returned_by_browser> 
$ curl -H "Authorization: Bearer $TOKEN" http://order-service/order
{"message":"This is the order service"}

This request is allowed by the following AuthorizationPolicy:

authenticated-to-gw.yaml

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: authenticated-to-gw
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        requestPrincipals: ["*"] # Matches any authenticated principal outside the cluster
    when:
    - key: request.auth.claims[iss]
      values: ["https://accounts.google.com"]

This policy permits any request authenticated by Google OIDC to pass through the Istio ingress gateway and be routed to the /order endpoint. Any other request destined for the order service will be blocked. For example, a request without an authorization header would return the following output:

$ curl http://order-service/order
RBAC: access denied

Crafting Client intents for Istio authorization policies

When dealing with network security mechanisms, such as Istio authorization policies or native Kubernetes network policies, Otterize provides an architecture based on 2 open-source projects:

• The Intents Operator simplifies identity and access management between services by enabling developers to declare the necessary calls for each service using client intents files. Intents are represented as ClientIntents custom resources in Kubernetes and the Intents Operator is capable of directly configuring network policies, Kafka ACLs, and acting on other enforcement points like Database credentials.
  
  

• The Network Mapper creates a map of in-cluster traffic by capturing DNS traffic and inspecting active connections in the same manner as netstat. It then resolves the IP addresses involved in these connections to the corresponding pods and traces the ownership of each pod up to the root controller. Additionally, in the case of Istio, the Network Mapper leverages the Istio watcher to query Envoy sidecars for HTTP traffic statistics, which helps identify HTTP traffic by paths. The Network Mapper continues to build and update the network map as long as it is deployed.

In our sample application, the frontend service includes a Go routine designed to continuously generate traffic destined to other services by querying all endpoints every 2 seconds.

// Start ticker to check the status of cart, product and order services every 2 seconds
	ticker := time.NewTicker(2 * time.Second)
	go func() {
		for range ticker.C {
			checkService("http://online-shop-cart.online-shop.svc.cluster.local:8083/cart", log)
			checkService("http://online-shop-product.online-shop.svc.cluster.local:8081/product", log)
			checkService("http://online-shop-order.online-shop.svc.cluster.local:8082/order", log)
		}
	}()

Note: request-method and request-path must explicitly be enabled in the Envoy metrics endpoint by creating a Telemetry resource. The configuration is detailed here, and you can directly deploy the manifest by running the following command:

$ kubectl apply -f https://docs.otterize.com/code-examples/network-mapper/istio-telemetry-enablement.yaml

The Otterize Network Mapper is now ready to generate the appropriate ClientIntents.

$ otterize network-mapper export -n online-shop

apiVersion: k8s.otterize.com/v1alpha3
kind: ClientIntents
metadata:
  name: online-shop-frontend
  namespace: online-shop
spec:
  service:
    name: online-shop-frontend
  calls:
    - name: online-shop-cart
      type: http
      HTTPResources:
        - path: /cart
          methods:
            - GET
    - name: online-shop-order
      type: http
      HTTPResources:
        - path: /order
          methods:
            - GET
    - name: online-shop-product
      type: http
      HTTPResources:
        - path: /product
          methods:
            - GET

The computed ClientIntents defines the application patterns for the entire namespace. We can see that the frontend service needs to access the /cart, /product, and /order endpoints from the actual traffic.

We can also directly deploy the ClientIntents manifest in the cluster by running the following command:

$ otterize network-mapper export -n online-shop | kubectl apply -n online-shop -f -
clientintents.k8s.otterize.com/online-shop-frontend created

Next, the Intents Operator reconciles the desired state and provisions 3 Istio AuthorizationPolicies:

$ kubectl get authorizationpolicies -n online-shop
NAME                                                                                AGE
authorization-policy-to-online-shop-cart-from-online-shop-frontend.online-shop      18m
authorization-policy-to-online-shop-order-from-online-shop-frontend.online-shop     18m
authorization-policy-to-online-shop-product-from-online-shop-frontend.online-shop   18m

An AuthorizationPolicy is created for every target service - cart, order, and product. Let’s take a look at the policy securing the product service.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  labels:
    intents.otterize.com/istio-client: online-shop-frontend-online-shop-1dbdf3
    intents.otterize.com/service: online-shop-product-online-shop-1aec17
  name: authorization-policy-to-online-shop-product-from-online-shop-frontend.online-shop
  namespace: online-shop
spec:
  selector:
    matchLabels:
      intents.otterize.com/service: online-shop-product-online-shop-1aec17
  rules:
  - from:
    - source:
        principals:
        - cluster.local/ns/online-shop/sa/frontend
    to:
    - operation:
        methods:
        - GET
        paths:
        - /product

The other AuthorizationPolicies show a similar configuration. The automation highlights the attribution of labels to the pods for various purposes, including the identification of the source and destination of the traffic (who), and the desired communication patterns (what). For example, the frontend pod receives the following labels:

- intents.otterize.com/access-online-shop-cart-online-shop-e012a9=true,

- intents.otterize.com/access-online-shop-order-online-shop-b32cb5=true,

- intents.otterize.com/access-online-shop-product-online-shop-1aec17=true,

- intents.otterize.com/client=online-shop-frontend-online-shop-1dbdf3,

- intents.otterize.com/service=online-shop-frontend-online-shop-1dbdf3

This naming convention covers scenarios where the frontend pod can be the source (client) or destination (server) of the request. In our case, it’s only the source of the traffic.

Also, remember that generating ClientIntents, particularly for the order service, will create an implicit deny-all rule for traffic that doesn't match the internal communication patterns of the application. To allow external users to reach the /order endpoint and ensure request authentication functions correctly, we need to explicitly authorize these requests. Let's create an AuthorizationPolicy for this purpose. It will resemble the AuthorizationPolicy we built previously for the Istio ingress gateway, but with the addition of the /order endpoint.

any-to-order.yaml

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: any-to-order
  namespace: online-shop
spec:
  selector:
    matchLabels:
      run: online-shop-order
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["*"]
    to:
    - operation:
        paths: [/login, /callback, /order]

Deploy this policy by running the command:

$ kubectl apply -f any-to-order.yaml

This systematic approach simplifies the creation and enforcement of security policies across services. By automating the generation and application of ClientIntents and AuthorizationPolicies, you can ensure that service interactions are secure and compliant with predefined security standards. This method reduces the potential for human error and enhances scalability with repeatable, predictable processes. As a result, developers can focus on building features instead of managing complex security details. 

Additionally, this framework dynamically manages access controls in response to code updates, and you can even store ClientIntents alongside your code or have Otterize automatically create pull requests when it detects new traffic flows. It's like setting up a smart home security system—once it's in place, it keeps your house safe while you kick back and relax, maybe even forgetting the codes and keys because the system's got you covered.

TL;DR

Tired of manually configuring security policies? As your Kubernetes deployments grow, managing secure access across services becomes increasingly complex. This scaling challenge often leads to errors and vulnerabilities that slip through the cracks of manual processes. That's why finding a more efficient solution is crucial. With Otterize Intent-Based Access Control (IBAC), you can automate the creation and maintenance of security policies, transforming a cumbersome task into a streamlined, error-free process.

In this article, we've explored key concepts of end-to-end application authentication and authorization, demonstrating how to leverage Istio's authorization policies paired with an automated workflow for secure, scalable service interactions. This not only saves time but also enhances security compliance, allowing you to focus on innovation rather than configuration. Otterize simplifies your workflow and keeps your services secure, no matter how complex your environment gets.

Conclusion

In this example, we dived into Istio configuration within the context of a microservices application, addressing both external user authentication and internal deployment of security policies. We've shown that due to the inherent complexity of application flows, it’s tough for platform engineers to track communication patterns effectively. Placing security-as-code directly in the hands of developers finds the perfect balance, empowering them to manage security proactively. However, for this approach to be scalable and less error-prone, developers need a straightforward, declarative method supported by robust tooling to verify and maintain security requirements close to the code.

Otterize facilitates this by offering a comprehensive architecture that includes multiple operators, custom resources, and integrations with version control systems. This setup ensures that security configurations keep pace with the applications they protect, allowing developers to deploy secure and compliant solutions effortlessly. 

Looking ahead, the adoption of a scalable abstraction model for workload IAM not only bolsters organizational security but also drives innovation in cloud-native security implementations. With numerous integrations yet to explore, Otterize is eager to adapt to your specific use cases. Don’t stop here, and discover how we can tailor our solutions to meet your security needs!

Let’s talk about it! 

Take control: Our Istio tutorials provide a hands-on opportunity to explore these concepts further.

Shape the future: We value your input! Share your thoughts on which tutorials would be most beneficial. Also, what other networking and IAM security automation scenarios are you interested in? Let us know.

Let's talk: Our community Slack channel is a great place to share your feedback, challenges, and any hiccups you encountered during your experience with Otterize—it’s also where you’ll be able to message me directly with any questions you may have. Let's learn from each other and discover solutions together!