Ask AI

Configuring and running a Kubernetes agent#

This guide is applicable to Dagster+.

In this guide, you'll configure and run a Kubernetes agent. Kubernetes agents are used to launch your code in Kubernetes Jobs and Services.


Prerequisites#

To complete the steps in this guide, you'll need:


Step 1: Generate a Dagster+ agent token#

In this step, you'll generate a token for the Dagster+ agent. The Dagster+ agent will use this to authenticate to the agent API.

  1. Sign in to your Dagster+ instance.
  2. Click the user menu (your icon) > Organization Settings.
  3. In the Organization Settings page, click the Tokens tab.
  4. Click the + Create agent token button.
  5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.


Step 2: Configure the agent#

In this step, you'll create a Kubernetes namespace for your Dagster+ resources. You'll also add the agent token to the Kubernetes cluster.

  1. Create a Kubernetes namespace for your Dagster+ resources:

    kubectl create namespace <namespace>
    

    For example, if the namespace is dagster-plus:

    kubectl create namespace dagster-plus
    
  2. Add the agent token you created in Step 1 as a secret in the Kubernetes cluster:

    kubectl create secret generic dagster-cloud-agent-token \
        --from-literal=DAGSTER_CLOUD_AGENT_TOKEN=<your_agent_token> \
        --namespace dagster-plus
    

Step 3: Start the agent#

In this step, you'll spin up the agent with Helm.

  1. Add the agent chart repository:

    helm repo add dagster-cloud https://dagster-io.github.io/helm-user-cloud
    helm repo update
    
  2. Create a values.yaml file, indicating the Dagster+ deployment the agent will be responsible for. Note: When your Dagster+ organization is first created, the default deployment is prod:

    dagsterCloud:
      deployment: prod
    
  3. Install the agent chart, specifying:

    • The namespace into which to install the chart
    • The values.yaml file created in the previous step
    helm upgrade \
       --install user-cloud dagster-cloud/dagster-cloud-agent \
       --namespace dagster-plus \
       --values ./values.yaml
    

After the Helm chart is installed into the Kubernetes cluster, you can view the agent in Dagster+.

In Dagster+, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:

Instance Status

Next steps#

Now that you've got your agent running, what's next?

You can also further configure the jobs the agent runs, including adding secrets, mounting volumes, and setting resource limits.