Amazon ECS agent configuration reference#

This reference describes the various configuration options Dagster+ currently supports for Amazon ECS agents.

Per-location configuration#

When adding a code location to Dagster+ with an Amazon ECS agent, you can use the container_context key on the location configuration to add additional ECS-specific configuration that will be applied to any ECS tasks associated with that code location.

Note: If you're using the Dagster+ Github action, the container_context key can also be set for each location in your dagster_cloud.yaml file.

The following example dagster_cloud.yaml file illustrates the available fields:

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      ecs:
        env_vars:
          - DATABASE_NAME=staging
          - DATABASE_PASSWORD
        secrets:
          - name: "MY_API_TOKEN"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::"
          - name: "MY_PASSWORD"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::"
        secrets_tags:
          - "my_tag_name"
        server_resources: # Resources for code servers launched by the agent for this location
          cpu: 256
          memory: 512
          replica_count: 1
        run_resources: # Resources for runs launched by the agent for this location
          cpu: 4096
          memory: 16384
        execution_role_arn: arn:aws:iam::123456789012:role/MyECSExecutionRole
        task_role_arn: arn:aws:iam::123456789012:role/MyECSTaskRole
        mount_points:
          - sourceVolume: myEfsVolume
            containerPath: "/mount/efs"
            readOnly: True
        volumes:
          - name: myEfsVolume
            efsVolumeConfiguration:
              fileSystemId: fs-1234
              rootDirectory: /path/to/my/data
        server_sidecar_containers:
          - name: DatadogAgent
            image: public.ecr.aws/datadog/agent:latest
            environment:
              - name: ECS_FARGATE
                value: true
        run_sidecar_containers:
          - name: DatadogAgent
            image: public.ecr.aws/datadog/agent:latest
            environment:
              - name: ECS_FARGATE
                value: true
        server_ecs_tags:
          - key: MyEcsTagKey
            value: MyEcsTagValue
        run_ecs_tags:
          - key: MyEcsTagKeyWithoutValue
        repository_credentials: MyRepositoryCredentialsSecretArn

Environment variables and secrets#

Using the container_context.ecs.env_vars and container_context.ecs.secrets properties, you can configure environment variables and secrets for a specific code location.

# dagster_cloud.yaml

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      ecs:
        env_vars:
          - DATABASE_NAME=testing
          - DATABASE_PASSWORD
        secrets:
          - name: "MY_API_TOKEN"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::"
          - name: "MY_PASSWORD"
            valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::"
        secrets_tags:
          - "my_tag_name"

Refer to the following guides for more info about environment variables:

• Dagster+ environment variables and secrets
• Using environment variables and secrets in Dagster code

Per-job configuration: Resource limits#

You can use job tags to customize the CPU and memory of every run for that job:

from dagster import job, op

@op()
def my_op(context):
  context.log.info('running')

@job(
  tags = {
    "ecs/cpu": "256",
    "ecs/memory": "512",
  }
)
def my_job():
  my_op()

Fargate tasks only support certain combinations of CPU and memory.

If the ecs/cpu or ecs/memory tags are set, they will override any defaults set on the code location or the deployment.

Per-deployment configuration#

This section describes the properties of the dagster.yaml configuration file used by Amazon ECS agents. Typically, this file is created by the CloudFormation template that deploys the agent and can be found within the agent task definition's command.

To change these properties, edit the CloudFormation template and redeploy the CloudFormation stack.

instance_class:
  module: dagster_cloud
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token: <Agent Token String>
  deployments:
    - <Deployment Name>
    - <Optional Additional Deployment Name>
  branch_deployments: <true|false>

user_code_launcher:
  module: dagster_cloud.workspace.ecs
  class: EcsUserCodeLauncher
  config:
    cluster: <Cluster Name>
    subnets:
      - <Subnet Id 1>
      - <Subnet Id 2>
    security_group_ids:
      - <Security Group ID>
    service_discovery_namespace_id: <Service Discovery Namespace Id>
    execution_role_arn: <Task Execution Role Arn>
    task_role_arn: <Task Role Arn>
    log_group: <Log Group Name>
    launch_type: <"FARGATE"|"EC2">
    server_process_startup_timeout: <Timeout in seconds>
    server_resources:
      cpu: <CPU value>
      memory: <Memory value>
    server_sidecar_containers:
      - name: SidecarName
        image: SidecarImage
        <Additional container fields>
    run_resources:
      cpu: <CPU value>
      memory: <Memory value>
    run_sidecar_containers:
      - name: SidecarName
        image: SidecarImage
        <Additional container fields>
    mount_points:
      - <List of mountPoints to pass into register_task_definition>
    volumes:
      - <List of volumes to pass into register_task_definition>
    server_ecs_tags:
      - key: MyEcsTagKey
        value: MyEcsTagValue
    run_ecs_tags:
      - key: MyEcsTagKeyWithoutValue
    repository_credentials: MyRepositoryCredentialsSecretArn

isolated_agents:
  enabled: <true|false>
agent_queues:
  include_default_queue: <true|false>
  additional_queues:
    - <queue name>
    - <additional queue name>

dagster_cloud_api properties#

user_code_launcher properties#

isolated_agents properties#

agent_queues properties#

These settings specify the queue(s) the agent will obtain requests from. See Routing requests to specific agents.