Deploying Dagster on Kubernetes

You can find the code for this example on Github.

First, install Dagster using Helm:

helm repo add dagster https://dagster-io.github.io/helm
helm install dagster dagster/dagster

Helm will launch several pods including PostgreSQL and RabbitMQ; you can check the status of the installation with kubectl. If everything worked correctly, you should see output like the following:

$ kubectl get pods
NAME                                                     READY   STATUS      RESTARTS   AGE
dagster-celery-workers-76f9df6b76-8zgz7                  1/1     Running     0          25s
dagster-celery-workers-76f9df6b76-995wx                  1/1     Running     0          25s
dagster-celery-workers-76f9df6b76-c7g7c                  1/1     Running     0          25s
dagster-celery-workers-76f9df6b76-j7kbv                  1/1     Running     0          25s
dagster-celery-workers-76f9df6b76-wsv4p                  1/1     Running     0          25s
dagster-dagit-9f4c7f786-8kpxr                            1/1     Running     0          25s
dagster-flower-b84bcd6dc-ttqkr                           1/1     Running     0          25s
dagster-postgresql-0                                     1/1     Running     0          14s
dagster-rabbitmq-0                                       1/1     Running     0          25s

It will also print instructions to open port forwarding to the Dagit pod; you can run these commands locally to load up the Dagit UI:

export DAGIT_POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=dagster,app.kubernetes.io/instance=dagster,component=dagit" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to open Dagit"
kubectl --namespace default port-forward $DAGIT_POD_NAME 8080:80

You can visit the playground in Dagit to kick off execution. If you used the port fowarding above, this should be at http://127.0.0.1:8080/pipeline/example_pipe/playground.

To execute the pipeline, click the "Preset" button at the top left of the editor and select "example". Finally, click "Launch Execution" and you should see the pipeline execute on Kubernetes!

You can introspect the jobs that were launched with kubectl:

kubectl get jobs
NAME                                               COMPLETIONS   DURATION   AGE
dagster-job-a0cf0a53fef76b3a1d1cd558750f7bb0       1/1           9s         35s
dagster-job-bbc47774c005362518d30e575074ab11       1/1           9s         25s
dagster-run-2fd98c73-0c15-492a-9128-4b98f3b14a80   1/1           29s        41s