Least-privilege for Kafka data

Kafka is a very popular event-streaming platform, used for many use cases: brokering an event-driven architecture, messaging and queuing, aggregating data streams,and so on. What do all of these have in common? Data. And often, this is important, sensitive data: not everyone and everything should be able to read or write it.

Data access management is a big topic, often accompanied by the big name “data governance”, and regrettably often implying a lot of investment in management systems, governance processes, metadata management, policies, reviews and approvals and committees… There are certainly valid reasons to pursue data access management holistically, but could you also achieve really good access control with literally none of that overhead?

Reversing the paradigm

Enter IBAC: intent-based access control.

With IBAC, you don’t need to decide on the Kafka side anything about data access. You could, of course, but you don’t need to. Instead, access is granted to whoever asks for it.

What? Isn’t that the same as no access control? Won’t our data be totally exposed?

No, surprisingly, it’s quite the opposite. You see, with IBAC, a client of Kafka declares its intent to do something with the data, e.g.:

name: orderservice
calls:
  - name: ecomm-kafka
    type: kafka
    topics:
      - name: orders
        operations: [ produce ]

This is just a file: a client intents file, so it doesn’t do anything, other than one critical thing: it expresses what the order service is supposed to do, according to the order service developer.

The intents file for the order service lives with the rest of the order service code, in the same repo. When the code is approved, through whatever mechanisms make sense for the team and the organization, the intents file also gets approved. That’s critical: if access should not be granted, that’s the moment to make that decision, and if it should be granted, this is the time when there’s the most context to make that decision. After all, why would you approve code that produces orders events if you don’t approve access to produce orders events?

Granting the appropriate access automatically

Now, once it’s approved and established that orderservice should produce to the orders topic in the ecomm-kafka broker, the client intents file is handed over to Otterize to “make it so”.

In this example, the orderservice runs as a pod in a Kubernetes cluster. So we can either convert the above client intents file to a Kubernetes YAML with the otterize intents convert CLI command, or just use the Kubernetes custom resource format to begin with:

apiVersion: k8s.otterize.com/v1alpha2
kind: ClientIntents
metadata:
  name: orderservice
spec:
  calls:
    - name: ecomm-kafka
      type: kafka
      topics:
        - name: orders
          operations: [ produce ]

Just upload this to the Kubernetes cluster via kubectl apply. The Otterize OSS intents operator picks it up, and automatically updates the Kafka ACLs to grant access for the orderservice to produce to the orders topic. The Otterize OSS credentials operator even creates and distributes to the client the right certificate, trusted by the Kafka broker, so the client can easily authenticate with mTLS. So for the order service developer, everything just works – in every environment!

You can try it yourself now with this quick, visual tutorial.

Wait… what?

Let’s take a quick deeper look at everything that just happened, just by applying the intent – i.e., just by declaring that orderservice intends to produce to the orders topic on ecomm-kafka. That’s the key bit of information that the IBAC automation needs in order to make everything happen, and the only information the developer needed to supply.

1. 1. An ACL was created in ecomm-kafka, specifying that a specific client (orderservice) is allowed to produce to the orders topic.

   1. 1. The client is identified by giving it a Common Name (CN), based on its name and namespace, that’ll then also be used in the certificate: CN=orderservice.ecomm-services-ns.

2. 2. A certificate was issued with this CN, signed by a CA which Kafka is configured to trust.

   1. 1. For certificates, you can use either SPIRE in your Kubernetes cluster, or Otterize Cloud’s credentials service, or an issuer managed through cert-manager (coming soon). This is a one-time setup, e.g. using Helm.

3. 3. The certificate was made available as a specific Kubernetes secret to orderservice, so it could mount it as a volume and access it easily when calling ecomm-kafka.

In short, this is all the work someone (the developer, the platform engineer, a custom script) would have had to do once the need for the orderservice to access the orders topic arose. And again every time a new need arose. And, ideally, again every time such a need went away – so there are no dangling unnecessary permissions left over, waiting for someone to exploit them.

Principle of least privilege

Let’s look again at what access was granted. The order service only has access to what it needs to get its job done: it can only produce to the orders topic in this particular Kafka. No other topics are exposed, in fact it can’t even read orders events unless the intents file was changed to use operations: [ produce, consume ] and re-approved. So it was granted the least privilege needed to perform its job.

And when these ACLs are applied to Kafka, any clients not having declared their intents will lose all access to that topic. You can also easily configure Kafka to require client intents on all topics, so anything undeclared – and unapproved – is not allowed. All your data would be protected by default, unless a client has declared it needs to access it and that declaration was approved. And that ensures that the only access granted is the least privilege needed for all the systems to do their job.

Moreover, as long as client intents files are properly maintained along with the client code, the problem of dangling permissions is also addressed. Let’s say the order service no longer creates order events; instead, order events are now produced by a checkout service, while the order service updates its state by consuming order events. The order service code is changed, and the intents file should become:

apiVersion: k8s.otterize.com/v1alpha2
kind: ClientIntents
metadata:
  name: orderservice
spec:
  calls:
    - name: ecomm-kafka
      type: kafka
      topics:
        - name: orders
          operations: [ consume ]

The Otterize intents operator would change the ACL for the orderservice so it can no longer produce orders events, only consume orders events. This preserves the principle of least privilege: as code evolves, you don’t need to worry that somebody will forget to remove some server-side permissions because they’re not aware the clients no longer need them.

There’s no free lunch

So does data protection via IBAC come for free?

Of course not, but the unnecessary burden is removed, while the remaining work is moved to where it’s most effective. As anyone who ever tried data governance in a fast-moving organization can attest, the hardest aspect is knowing what to protect vs what to expose, and to whom. Intent-based access control moves the burden of defining the needed access to the party most motivated to do it, and most knowledgeable about what’s needed, at just the time it’s needed: the client of the data.

That burden on the client developer is then a very light one: just declare what you access you need, and that’s exactly what you get. And it promotes good hygiene: be intentional about accessing data. And even as clients are only declaring their individual needs, you can still pull together a uniform view of who has access to what, and what data is protected: simply attach the Otterize OSS components running in your Kubernetes clusters to the free Otterize Cloud service, and see that unified view overlaid with real-time insights about access controls:

The ask from client developers will soon be minimized further: we’re working on an OSS Kafka “sniffer”, akin to our already-released OSS network mapper, that discovers which clients are accessing which topics in the environment where you install it. You can export that as a set of fine-grained intents files that the client developers can just review, so they don’t even need to author them. You could even go for full automation, where the intents files for all clients are automatically updated based on discovered access in some trusted environment, so whatever access occurs in that environment is the only access allowed in production.

Instead of attempting to define global policies for data operations across all possible scenarios, which can be powerful and precise but often carry a tremendous burden and tend to get stale over time, IBAC goes for simplicity: in the case of Kafka, for every client you simply ask whether a given client should be able to produce or consume events of a given topic. In practice, that may give you as much, if not more, effective protection of data than global policies, especially when considered over time when the best of policies may drift from the actual needs.

Plus, you could find out that your existing code review policies can serve as good data protection reviews too, without needing the overhead of new processes, or costly coordination between different teams. You do need to make sure that data access considerations enter the code review process, and if additional reviews need to be triggered – say, a secops review in some cases, or perhaps a review by the team that owns the originating data – this is the place to plug those in. After all, this is the moment when there is the most context to make the right decisions.

To get started with IBAC for Kafka in Kubernetes, check out the hands-on tutorial or the visual tutorial. And for more on IBAC, see the IBAC docs.