Working with Charmed Kubernetes from within a pod¶

Setup¶

After installing Juju and Charmed Kubernetes (together with applying set up for the latter), now we can look into how to launch Spark jobs from within a pod in Charmed Kubernetes.

First, we create a pod using Charmed Apache Spark container image.

Edit the pod manifest file (we’ll refer to it as shell-demo.yaml) by adding the following content:

apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
  namespace: default
spec:
  containers:
  - name: spark-client
    image: ghcr.io/canonical/charmed-spark:3.5-22.04_stable
    command: ["/bin/pebble", "run", "--hold"]
  serviceAccountName: spark
  hostNetwork: true
  dnsPolicy: Default

Note

Please make sure to use the correct tag for the version of Apache Spark that you’d like to use. For instance, if you want to use Apache Spark 4.0, you should use the image ghcr.io/canonical/charmed-spark:4.0-22.04_stable instead.

The pod can be created with the following command:

kubectl apply -f shell-demo.yaml

We can log into the pod as below:

kubectl exec --stdin --tty shell-demo -- /bin/bash 

Now let’s create the Kubernetes configuration on the pod, with contents from the original server’s .kube/config:

$ mkdir ~/.kube
$ cat > ~/.kube/config << EOF
<KUBECONFIG_CONTENTS_FROM_CHARMED_KUBERNETES>
EOF

Then we need to set up a service account for Spark job submission. Let’s create a user called spark in default namespace:

python3 -m spark8t.cli.service_account_registry create --username spark

Spark Job Submission To Kubernetes Cluster¶

There is a script called spark-submit packaged within the Charmed Apache Spark container image for Spark job submission. We can use the Spark Pi job example again, such as:

python3 -m spak8t.cli.spark_submit --username spark --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples_2.12-3.5.8.jar 100

Or using the snap command (referring practically to the same thing):

spark-client.spark-submit --username spark --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples_2.12-3.5.8.jar 100

Note

Please make sure to use the correct JAR for the version of Apache Spark and Scala that you’d like to use. For instance, if you want to use Apache Spark 4.0.2 (which natively supports Scala 2.13), you should use the JAR spark-examples_2.13-4.0.2.jar instead.

Spark Shell¶

To invoke the Apache Spark shell, you can run the following command within the pod:

python3 -m spark8t.cli.spark_shell --username spark

spark-client.spark-shell --username spark

PySpark Shell¶

To launch a pyspark shell, run the following command within the pod:

python3 -m spark8t.cli.pyspark --username spark

spark-client.pyspark --username spark