This guide is applicable to Dagster Cloud.
In this guide, you'll configure and run a Kubernetes agent. Kubernetes agents are used to launch your code in Kubernetes Jobs and Services.
To complete the steps in this guide, you'll need:
Permissions in Dagster Cloud that allow you to manage agent tokens. Refer to the User permissions documentation for more info.
Access to a container registry to which you can push images and from which pods in the Kubernetes cluster can pull images. This can be:
To have Helm installed. Refer to the Helm installation documentation for more info.
In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.
Keep the token somewhere handy - you'll need it to complete the setup.
In this step, you'll create a Kubernetes namespace for your Dagster Cloud resources. You'll also add the agent token to the Kubernetes cluster.
Create a Kubernetes namespace for your Dagster Cloud resources:
kubectl create namespace <namespace>
For example, if the namespace is
kubectl create namespace dagster-cloud
Add the agent token you created in Step 1 as a secret in the Kubernetes cluster:
kubectl create secret generic dagster-cloud-agent-token \ --from-literal=DAGSTER_CLOUD_AGENT_TOKEN=<your_agent_token> \ --namespace dagster-cloud
In this step, you'll spin up the agent with Helm.
Add the agent chart repository:
helm repo add dagster-cloud https://dagster-io.github.io/helm-user-cloud helm repo update
Install the agent chart, specifying:
helm upgrade \ --install user-cloud dagster-cloud/dagster-cloud-agent \ --namespace dagster-cloud \ --set dagsterCloud.deployment=prod
After the Helm chart is installed into the Kubernetes cluster, you can view the agent in Dagster Cloud.
In Dagster Cloud, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:
Now that you've got your agent running, what's next?
If you're getting Dagster Cloud set up, the next step is to add a code location using the agent.
If you're ready to load your Dagster code, refer to the Adding Code to Dagster Cloud guide for more info.
You can also further configure the jobs the agent runs, including adding secrets, mounting volumes, and setting resource limits.