Enable authentication and authorization with the Spark History Server charm

Charmed Apache Spark includes the Spark History Server charm, which lets users monitor application workflows and logs. By default, Spark History Server does not provide authentication or authorization, both of which are essential in production environments. To address this limitation, you can integrate the Spark History Server charm with the Canonical Identity Platform bundle.

Deploy the Identity Bundle and integrate it with the Spark History Server

To enable authentication and authorization for Spark History Server, complete the following steps. This guide assumes you already deployed Charmed Apache Spark as described in the Charmed Apache Spark deployment guide, including a Spark History Server charm configured with an object storage backend.

Deploy the Identity bundle

Authentication is provided by the Canonical Identity Bundle. Deploy it by following this tutorial, which installs all required Identity Platform components.

The deployment includes these charms:

You must also configure the identity provider you want to use. Configure kratos-external-idp-integrator with the parameters for your provider.

Example configuration for Microsoft Entra ID (Azure AD):

juju config kratos-external-idp-integrator microsoft_tenant_id=<YOUR_TENANT_ID> provider=microsoft client_id=<YOUR_CLIENT_ID> client_secret=<YOUR_CLIENT_SECRET>

For supported identity providers and additional details, see the How to manage external identity providers guide.

The connection between Spark History Server and the Identity Platform is handled by the Charmed OAuth2 Proxy charm. OAuth2 Proxy protects endpoints exposed through ingress (Traefik).

Enable authentication with Charmed OAuth2 Proxy

To set up OAuth2 Proxy, first enable the feature in Traefik, expose the forward-auth offer, and integrate it with Spark History Server through the ingress relation.

juju config traefik-public enable_experimental_forward_auth=True -m <IDENTITY_MODEL>
juju offer traefik-public:experimental-forward-auth forward-auth -m <IDENTITY_MODEL>
juju integrate spark-history-server-k8s admin/<IDENTITY_MODEL>.ingress

Next, deploy OAuth2 Proxy and integrate it with Traefik using the exposed offer:

juju deploy oauth2-proxy-k8s --channel latest/stable --trust
juju integrate oauth2-proxy-k8s:forward-auth admin/<IDENTITY_MODEL>.forward-auth

Then integrate Spark History Server with OAuth2 Proxy:

juju integrate oauth2-proxy-k8s spark-history-server-k8s:oauth2-proxy

Finally, integrate OAuth2 Proxy with the Identity Platform OIDC provider (Charmed Hydra):

juju offer hydra:oauth oauth -m <IDENTITY_MODEL>
juju integrate oauth2-proxy-k8s:oauth admin/<IDENTITY_MODEL>.oauth

After integration completes, get the endpoint by running:

juju run traefik-public/leader show-proxied-endpoints -m <IDENTITY_MODEL>

When you open the URL exposed by Traefik, you are redirected to your configured identity provider for authentication. After successful login, you can access the Spark History Server endpoint.

Authorization Management

By default, all authenticated users can access Spark History Server. To restrict access, configure an allow-list of authorized users. Provide email addresses as a comma-separated list:

juju config spark-history-server-k8s authorized-users="user1@canonical.com,user3@canonical.com"

Oathkeeper integration (deprecated)

Previously, authentication in this guide used the oathkeeper charm. oathkeeper is now deprecated in favor of Charmed OAuth2 Proxy.

If your deployment still uses oathkeeper, migrate by removing integrations with oathkeeper, updating Identity Platform components as described above, and then integrating oauth2-proxy-k8s.