This guide is applicable to Dagster Cloud.
In this guide, you'll set up and deploy an Amazon Elastic Container Service (ECS) agent in a new VPC using CloudFormation. Amazon ECS agents are used to launch user code in ECS tasks.
Our CloudFormation template allows you to quickly spin up the ECS agent stack. This template sets up an ECS agent from scratch, creating a new VPC and ECS cluster for the agent to run in. The template code can be found here. Refer to the CloudFormation docs for more info about CloudFormation.
For info about deploying an ECS agent in an existing VPC, check out the ECS agents in existing VPCs guide.
To complete the steps in this guide, you'll need:
In Dagster Cloud:
In Amazon Web Services (AWS), you'll need an account:
Under its VPC quota limit in the region where you're spinning up the agent. By default, AWS allows five VPCs per region. If you're already at your limit, refer to the AWS VPC quotas documentation for info on requesting a quota increase.
With an ECS service-linked IAM role. This role is required to complete the setup in ECS. AWS will automatically create the role in your account the first time you create an ECS cluster in the console. However, the IAM role isn't automatically created when ECS clusters are created via CloudFormation.
If your account doesn't have this IAM role, running the CloudFormation template may fail.
If you haven't created an ECS cluster before, complete one of the following before proceeding:
In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.
Keep the token somewhere handy - you'll need it to complete the setup.
Press the "Launch Stack" button below to install the CloudFormation stack in your AWS account.
Note: Creating the CloudFormation stack may take a few minutes. Refresh the AWS console Stacks page to check the status.
If the installation fails, verify that your AWS account meets the requirements listed above.
After the stack is installed, you'll be prompted to configure it. In the ECS wizard, fill in the following fields:
The page should look similar to the following image. In this example, our organization name is
hooli and our deployment is
After you've finished configuring the stack in AWS, you can view the agent in Dagster Cloud. To do so, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:
Now that you've got your agent running, what's next?
If you're getting Dagster Cloud set up, the next step is to add a code location using the agent.
If you're ready to load your Dagster code, refer to the Adding Code to Dagster Cloud guide for more info.
If you need to upgrade your ECS agent's CloudFormation template, refer to the upgrade guide for more info.