Feature: Access Kafka on behalf of end user

[erikgb]

Background

We are managing several multi-tenant Kafka clusters configured with zero-trust principles in mind - meaning that:

• Any interaction requires authentication.
• Kafka ACLs are enabled and pretty fine-grained for both end-users and applications. Especially for access to data. We are a bit more relaxed around metadata.

Our users can use various CLI tools to access Kafka according to their ACLs. But CLI isn't for everyone, so also have a centralized installation of AKHQ providing access to anything the synthetic User:* principal has access to, which is exclusively metadata. With this configuration, AKHQ is only provided with simple access to the cluster (authentication). No ACLs currently address the AKHQ principal.

Not surprisingly, our users are not satisfied with this and want access to data according to ACLs addressing their principal. As they can do using any Kafka CLI tool. And now we are trying to improve the situation. The solution draft, after reviewing the features currently provided by AKHQ is to:

• Use dex to enable OIDC authentication to AKHQ. ✅
• Set up AKHQ external roles and attributes mapping to determine the roles and attributes granted to an authenticated end-user. ✅
• Implement a microservice providing the role mapping endpoint to AKHQ - based on ACLs obtained from the cluster. 🚧

This will require us to add an ACL granting the AKHQ application access to all data in the cluster, which is far from ideal. I've been trying to see how this can be improved, and I have some ideas. I am very interested in getting feedback from AKHQ maintainers and the community in general.

Proposed solution

I want AKHQ to access the Kafka cluster on behalf of the end-user using OIDC/OAuth2. This may sound simple, and many of the required features are already available:

• AKHQ can authenticate with Kafka using OAuth2.
• Dex (as an example) supports the OAuth 2 token exchange specification (RFC 8693).
• AKHQ already has a plugin-based mapping system to map users and groups to roles with attributes. And I imagine it should be possible to provide some of this mapping from Kafka cluster ACLs.

What I am proposing is a new feature that allows AKHQ to use the OAuth2 token exchange API to exchange the end-user OAuth2 token (obtained when the user is authenticated by AKHQ) with a token (still with the end-user as subject) valid to access Kafka configured with OAUTHBEARER authentication.

I have borrowed an image from this excellent blog around OAuth2 token exchange to illustrate what I would like to achieve:

In our case, AKHQ is the Client and the Resource Server is Kafka.

WDYT? We would like to contribute to this if you think it is possible to implement. But would need some guidance on the AKHQ internals.