Skip to main content

Configuration reference

note

This guide is applicable to Dagster+.

This reference describes the various configuration options Dagster+ currently supports for Amazon ECS agents.

Per-agent configuration​

We expose AgentMemory, and AgentCpu fields in the Cloud Formation templates to configure the resources that the Dagster+ agent can use.

Per-location configuration​

When adding a code location to Dagster+ with an Amazon ECS agent, you can use the container_context key on the location configuration to add additional ECS-specific configuration that will be applied to any ECS tasks associated with that code location.

note

If you're using the Dagster+ Github action, the container_context key can also be set for each location in your build.yaml file.

The following example build.yaml file illustrates the available fields:

# build.yaml
locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      ecs:
        env_vars:
          - DATABASE_NAME=staging
          - DATABASE_PASSWORD
        secrets:
          - name: 'MY_API_TOKEN'
            valueFrom: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::'
          - name: 'MY_PASSWORD'
            valueFrom: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::'
        secrets_tags:
          - 'my_tag_name'
        server_resources: # Resources for code servers launched by the agent for this location
          cpu: '256'
          memory: '512'
          replica_count: 1
        run_resources: # Resources for runs launched by the agent for this location
          cpu: '4096'
          memory: '16384'
        execution_role_arn: arn:aws:iam::123456789012:role/MyECSExecutionRole
        task_role_arn: arn:aws:iam::123456789012:role/MyECSTaskRole
        mount_points:
          - sourceVolume: myEfsVolume
            containerPath: '/mount/efs'
            readOnly: True
        volumes:
          - name: myEfsVolume
            efsVolumeConfiguration:
              fileSystemId: fs-1234
              rootDirectory: /path/to/my/data
        server_sidecar_containers:
          - name: DatadogAgent
            image: public.ecr.aws/datadog/agent:latest
            environment:
              - name: ECS_FARGATE
                value: true
        run_sidecar_containers:
          - name: DatadogAgent
            image: public.ecr.aws/datadog/agent:latest
            environment:
              - name: ECS_FARGATE
                value: true
        server_ecs_tags:
          - key: MyEcsTagKey
            value: MyEcsTagValue
        run_ecs_tags:
          - key: MyEcsTagKeyWithoutValue
        repository_credentials: MyRepositoryCredentialsSecretArn
note

If you have an older Dagster+ deployment, you may have a dagster_cloud.yaml file instead of a build.yaml file.

Environment variables and secrets​

Using the container_context.ecs.env_vars and container_context.ecs.secrets properties, you can configure environment variables and secrets for a specific code location.

# build.yaml
locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      ecs:
        env_vars:
          - DATABASE_NAME=testing
          - DATABASE_PASSWORD
        secrets:
          - name: 'MY_API_TOKEN'
            valueFrom: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::'
          - name: 'MY_PASSWORD'
            valueFrom: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::'
        secrets_tags:
          - 'my_tag_name'
PropertyDescription
container_context.ecs.env_varsA list of keys or key-value pairs for task inclusion. If value unspecified, pulls from agent task. Example: FOO_ENV_VAR set to foo_value, BAR_ENV_VAR set to agent task value.
container_context.ecs.secretsIndividual secrets specified using the same structure as the ECS API.
container_context.ecs.secrets_tagsA list of tag names. Each secret tagged with any of those tag names in AWS Secrets Manager will be included in the launched tasks as environment variables. The name of the environment variable will be the name of the secret, and the value of the environment variable will be the value of the secret.

Refer to the following guides for more info about environment variables:

Op isolation​

By default, each Dagster job will run in its own ECS task, with each op running in its own subprocess within the task.

You can also configure your Dagster deployment with the ecs.ecs_executor to run each op in its own ECS task. This allows you to allocate different CPU, memory, and ephemeral storage for each step. For example:

import dagster as dg
from dagster_aws.ecs import ecs_executor


@dg.asset(op_tags={"ecs/cpu": "256", "ecs/memory": "512"})
def light_asset():
    ...


@dg.asset(
    deps=[light_asset],
    op_tags={"ecs/cpu": "4096", "ecs/memory": "16384"},
)
def heavy_asset():
    ...


@dg.definitions
def defs() -> dg.Definitions:
    return dg.Definitions(
        assets=[light_asset, heavy_asset],
        executor=ecs_executor,
    )

For more details on configuration options and supported tags, see Launching steps as ECS tasks.

Per-job configuration: Resource limits​

You can use job tags to customize the CPU and memory of every run for that job:

import dagster as dg

@dg.asset
def my_asset(context):
  context.log.info('running')

@dg.job(
  tags = {
    "ecs/cpu": "256",
    "ecs/memory": "512",
  }
)
def my_job():
  my_asset()

Fargate tasks only support certain combinations of CPU and memory.

If the ecs/cpu or ecs/memory tags are set, they will override any defaults set on the code location or the deployment.

Per-deployment configuration​

This section describes the properties of the dagster.yaml configuration file used by Amazon ECS agents. Typically, this file is created by the CloudFormation template that deploys the agent and can be found within the agent task definition's command.

To change these properties, edit the CloudFormation template and redeploy the CloudFormation stack.

# dagster.yaml
instance_class:
  module: dagster_cloud
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token: <Agent Token String>
  deployments:
    - <Deployment Name>
    - <Optional Additional Deployment Name>
  branch_deployments: <true|false>

user_code_launcher:
  module: dagster_cloud.workspace.ecs
  class: EcsUserCodeLauncher
  config:
    cluster: <Cluster Name>
    subnets:
      - <Subnet Id 1>
      - <Subnet Id 2>
    security_group_ids:
      - <Security Group ID>
    service_discovery_namespace_id: <Service Discovery Namespace Id>
    service_discovery_role_arn: <Optional IAM Role ARN for cross-account Cloud Map>
    execution_role_arn: <Task Execution Role Arn>
    task_role_arn: <Task Role Arn>
    log_group: <Log Group Name>
    launch_type: <"FARGATE"|"EC2">
    server_process_startup_timeout: <Timeout in seconds>
    ecs_timeout: <Timeout in seconds>
    server_resources:
      cpu: '<CPU value>'
      memory: '<Memory value>'
    server_sidecar_containers:
      - name: SidecarName
        image: SidecarImage
        <Additional container fields>
    run_resources:
      cpu: '<CPU value>'
      memory: '<Memory value>'
    run_sidecar_containers:
      - name: SidecarName
        image: SidecarImage
        <Additional container fields>
    mount_points:
      - <List of mountPoints to pass into register_task_definition>
    volumes:
      - <List of volumes to pass into register_task_definition>
    server_ecs_tags:
      - key: MyEcsTagKey
        value: MyEcsTagValue
    run_ecs_tags:
      - key: MyEcsTagKeyWithoutValue
    repository_credentials: MyRepositoryCredentialsSecretArn

isolated_agents:
  enabled: <true|false>
agent_queues:
  include_default_queue: <true|false>
  additional_queues:
    - <queue name>
    - <additional queue name>

dagster_cloud_api properties​

PropertyDescription
dagster_cloud_api.agent_tokenAn agent token for the agent to use for authentication.
dagster_cloud_api.deploymentsThe names of full deployments for the agent to serve.
dagster_cloud_api.branch_deploymentsWhether the agent should serve all branch deployments.

user_code_launcher properties​

PropertyDescription
config.clusterName of ECS cluster with Fargate or EC2 capacity provider
config.launch_typeECS launch type:
• FARGATE
• EC2 Note: Using this launch type requires you to have an EC2 capacity provider installed and additional operational overhead to run the agent.
config.subnetsAt least one subnet is required. Dagster+ tasks require a route to the internet so they can access our API server. How this requirement is satisfied depends on the type of subnet provided:
• Public subnets - The ECS agent will assign each task a public IP address. Note that ECS tasks on EC2 launched within public subnets do not have access to the internet, so a public subnet will only work for Fargate tasks.
•Private subnets - The ECS agent assumes you've configured a NAT gateway with an attached NAT gateway. Tasks will not be assigned a public IP address.
config.security_group_idsA list of security groups to use for tasks launched by the agent.
config.service_discovery_namespace_idThe name of a private DNS namespace.
The ECS agent launches each code location as its own ECS service. The agent communicates with these services via AWS CloudMap service discovery.
config.service_discovery_role_arnAn optional IAM role ARN to assume when making AWS Cloud Map (service discovery) API calls. Use this when the Cloud Map namespace lives in a different AWS account than the ECS agent, for example in AWS Organizations setups where networking is centralized in a dedicated account. The agent assumes this role via STS to obtain temporary credentials (1-hour sessions, auto-refreshed).
Requirements: The agent's task role must have sts:AssumeRole permission for this role, and the target role must include a trust policy that allows the agent's task role to assume it. The target role needs the same servicediscovery:* permissions listed in the cross-account service discovery setup.
config.execution_role_arnThe ARN of the Amazon ECS task execution IAM role. This role allows ECS to interact with AWS resources on your behalf, such as getting an image from ECR or pushing logs to CloudWatch.
Note:This role must include a trust relationship that allows ECS to use it.
config.task_role_arnThe ARN of the Amazon ECS task IAM role. This role allows the containers running in the ECS task to interact with AWS.
Note: This role must include a trust relationship that allows ECS to use it.
config.log_groupThe name of a CloudWatch log group.
config.server_process_startup_timeoutThe amount of time, in seconds, to wait for code to import when launching a new service for a code location. If your code takes an unusually long time to load after your ECS task starts up and results in timeouts in the Deployment tab, you can increase this setting above the default. Note This setting isn't applicable to the time it takes for a job to execute.
• Default - 180 (seconds)
config.ecs_timeoutHow long (in seconds) to wait for ECS to spin up a new service and task for a code server. If your ECS tasks take an unusually long time to start and result in timeouts, you can increase this setting above the default.
• Default - 600 (seconds)
config.ecs_grace_periodHow long (in seconds) to continue polling if an ECS API endpoint fails during creation of a new code server (because the ECS API is eventually consistent).
• Default - 30 (seconds)
config.server_resourcesThe resources that the agent should allocate to the ECS service for each code location that it creates. If set, must be a dictionary with a cpu and/or memory key and string values. Note: Fargate tasks only support certain combinations of CPU and memory.
config.server_sidecar_containersAdditional sidecar containers to include along with the Dagster container. If set, must be a list of dictionaries with valid ECS container definitions.
config.run_resourcesThe resources that the agent should allocate to the ECS task that it creates for each run. If set, must be a dictionary with a cpu and/or memory key and string values. Note: Fargate tasks only support certain combinations of CPU and memory.
config.run_sidecar_containersAdditional sidecar containers to include along with the Dagster container. If set, must be a list of dictionaries with valid ECS container definitions.
config.mount_pointsMount points to include in the Dagster container. If set, should be a list of dictionaries matching the mountPoints field when specifying a container definition to boto3.
config.volumesAdditional volumes to include in the task definition. If set, should be a list of dictionaries matching the volumes argument to register_task_definition in boto3.
config.server_ecs_tagsAdditional ECS tags to include in the service for each code location. If set, must be a list of dictionaries, each with a key key and optional value key.
config.run_ecs_tagsAdditional ECS tags to include in the task for each run. If set, must be a list of dictionaries, each with a key key and optional value key.
config.repository_credentialsOptional arn of the secret to authenticate into your private container registry. This does not apply if you are leveraging ECR for your images, see the AWS private auth guide.
config.enable_ecs_execBoolean that determines whether tasks created by the agent should be configured with the needed linuxParameters and permissions to use ECS Exec to shell into the task. Also grants the SYS_PTRACE linux capability to enable running tools like py-spy to debug slow or hanging tasks. Defaults to false. Note: For ECS Exec to work, the task IAM role must be granted certain permissions.

Cross-account service discovery​

If your AWS Cloud Map namespace lives in a different AWS account than the ECS agent (for example, centralized networking in an AWS Organizations setup), you can configure the agent to assume a cross-account IAM role for service discovery API calls using the service_discovery_role_arn configuration option.

To set this up:

  1. In the account that owns the Cloud Map namespace (e.g., the Networks account), create an IAM role with the required servicediscovery:* permissions:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "CloudMapAccess",
          "Effect": "Allow",
          "Action": [
            "servicediscovery:CreateService",
            "servicediscovery:DeleteService",
            "servicediscovery:DeregisterInstance",
            "servicediscovery:GetNamespace",
            "servicediscovery:GetOperation",
            "servicediscovery:ListInstances",
            "servicediscovery:ListServices",
            "servicediscovery:ListTagsForResource",
            "servicediscovery:TagResource"
          ],
          "Resource": "*"
        }
      ]
    }

  2. Add a trust policy on that role that allows the ECS agent's task role to assume it:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::AGENT_ACCOUNT_ID:role/YOUR_AGENT_TASK_ROLE"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }

  3. Add sts:AssumeRole permission to the agent's task role (in the agent's account):

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "arn:aws:iam::NETWORK_ACCOUNT_ID:role/YOUR_CROSS_ACCOUNT_ROLE"
        }
      ]
    }

  4. Set service_discovery_role_arn in your agent's dagster.yaml to the ARN of the role created in step 1.

Note: Both roles must include a trust relationship that allows ECS to use them:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

isolated_agents properties​

PropertyDescription
isolated_agents.enabledWhen enabled, agents are isolated and will not be able to access each others' resources. See the Running multiple agents guide for more information.

agent_queues properties​

These settings specify the queue(s) the agent will obtain requests from. See Routing requests to specific agents.

PropertyDescription
agent_queues.include_default_queueIf true, agent processes requests from default queue
agent_queues.additional_queuesList of additional queues for agent processing

Using a custom agent image​

By default, the Dagster+ ECS agent uses the official docker.io/dagster/dagster-cloud-agent image. You can build and use your own custom agent image if you need additional dependencies, a different base image, or more control over the agent environment.

Building a custom agent image​

Create a Dockerfile that installs the dagster-cloud package with the ecs extra:

FROM python:3.12-slim

ARG DAGSTER_VERSION=1.12.19
RUN pip install dagster-cloud[ecs]==${DAGSTER_VERSION}

CMD ["dagster-cloud", "agent", "run"]

Build and push the image to your container registry (such as Amazon ECR):

docker build -t 123456789012.dkr.ecr.us-east-1.amazonaws.com/dagster-cloud-agent:custom .
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/dagster-cloud-agent:custom

Configuring CloudFormation to use a custom image​

To use your custom agent image, modify the CloudFormation template that deploys the agent. Locate the agent's ECS task definition and update the Image property to point to your custom image:

AgentTaskDefinition:
  Type: AWS::ECS::TaskDefinition
  Properties:
    ContainerDefinitions:
      - Name: DagsterCloudAgent
        Image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/dagster-cloud-agent:custom
        # ... rest of container definition

After updating the template, redeploy the CloudFormation stack to apply the changes.