This guide is applicable to Dagster Cloud.
In this guide, you'll configure and run a local agent. Local agents are used to launch user code in operating system subprocesses.
Considering running the agent in production? Running the local agent in production can be a good choice if:
Keep in mind that the local agent’s ability to run jobs is limited by the capacity of the single node on which it’s running.
If you’re running the local agent in production, make sure you’ve set up a supervisor to automatically restart the agent process if it crashes. You’ll also want a system in place to alert you if the VM or container dies, or to automatically restart it.
To complete the steps in this guide, you'll need:
Permissions in Dagster Cloud that allow you to manage agent tokens. Refer to the User permissions documentation for more info.
To install the
dagster-cloud CLI. This should be in the same environment where the agent will run.
Note: Your Dagster application code and its Python and system requirements must also be installed in this environment. We recommend using Python virtual environments to accomplish this.
In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.
Keep the token somewhere handy - you'll need it to complete the setup.
Create a directory on disk to act as your Dagster home. We'll use
~/dagster_home in our examples, but this directory can be located wherever you want.
In the directory you created, create a
dagster.yaml file like the following:
# ~/dagster_home/dagster.yaml instance_class: module: dagster_cloud.instance class: DagsterCloudAgentInstance dagster_cloud_api: agent_token: <YOUR_AGENT_TOKEN> deployment: prod user_code_launcher: module: dagster_cloud.workspace.user_code_launcher class: ProcessUserCodeLauncher
In the file, fill in the following:
agent_token - Add the agent token you created in Step 1. This specifies that the agent will launch work in local processes, reading the API URL and authentication token from environment variables.
deployment - Enter the deployment associated with this instance of the agent.
In the above example, we specified
prod as the deployment. This is present when Dagster Cloud organizations are first created.
Save the file.
Next, run the process agent by pointing at the home directory you created:
dagster-cloud agent run ~/dagster_home/
To view the agent in Dagster Cloud, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:
Now that you've got your agent running, what's next?