Orchestration on Docker¶

APIs¶

dagster_docker.DockerRunLauncher RunLauncher[source]¶

Config Schema:

image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource):
username (dagster.StringSource):
password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition[source]¶

Config Schema:

image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource):
username (dagster.StringSource):
password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

retries (selector, optional):

Whether retries are enabled or not. By default, retries are enabled.

Default Value:

{
    "enabled": {}
}

Config Schema:

enabled (strict dict, optional):

Default Value:

{}

disabled (strict dict, optional):

Default Value:

{}

max_concurrent (dagster.IntSource, optional):

Limit on the number of containers that will run concurrently within the scope of a Dagster run. Note that this limit is per run, not global.

tag_concurrency_limits (List[strict dict], optional):

A set of limits that are applied to steps with particular tags. If a value is set, the limit is applied to only that key-value pair. If no value is set, the limit is applied across all values of that key. If the value is set to a dict with applyLimitPerUniqueValue: true, the limit will apply to the number of unique values for that key. Note that these limits are per run, not global.

( experimental ) This API may break in future versions, even between dot releases.

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster_docker import docker_executor

from dagster import job

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.

Ops¶

dagster_docker.docker_container_op = <dagster._core.definitions.op_definition.OpDefinition object>[source]¶

Config Schema:

image (dagster.StringSource):

The image in which to run the Docker container.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource):
username (dagster.StringSource):
password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

entrypoint (List[String], optional):

The ENTRYPOINT for the Docker container

command (List[String], optional):

The command to run in the container within the launched Docker container.

( experimental ) This API may break in future versions, even between dot releases.

An op that runs a Docker container using the docker Python API.

Contrast with the docker_executor, which runs each Dagster op in a Dagster job in its own Docker container.

This op may be useful when:

You need to orchestrate a command that isn’t a Dagster op (or isn’t written in Python)
You want to run the rest of a Dagster job using a specific executor, and only a single op in docker.

For example:

from dagster_docker import docker_container_op

from dagster import job

first_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo HELLO"],
    },
    name="first_op",
)
second_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo GOODBYE"],
    },
    name="second_op",
)

@job
def full_job():
    second_op(first_op())

You can create your own op with the same implementation by calling the execute_docker_container function inside your own op.

dagster_docker.execute_docker_container(context, image, entrypoint=None, command=None, networks=None, registry=None, env_vars=None, container_kwargs=None)[source]¶

( experimental ) This API may break in future versions, even between dot releases.

This function is a utility for executing a Docker container from within a Dagster op.

Parameters:

image (str) – The image to use for the launched Docker container.
entrypoint (Optional[Sequence[str]]) – The ENTRYPOINT to run in the launched Docker container. Default: None.
command (Optional[Sequence[str]]) – The CMD to run in the launched Docker container. Default: None.
networks (Optional[Sequence[str]]) – Names of the Docker networks to which to connect the launched container. Default: None.
registry – (Optional[Mapping[str, str]]): Information for using a non local/public Docker registry. Can have “url”, “username”, or “password” keys.
env_vars (Optional[Sequence[str]]) – List of environemnt variables to include in the launched container. ach can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the calling environment.
container_kwargs (Optional[Dict[str[Any]]]) – key-value pairs that can be passed into containers.create in the Docker Python API. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

Pipes¶

dagster_docker.PipesDockerClient¶: alias of _PipesDockerClient[_PipesDockerClient]

class dagster_docker.PipesDockerLogsMessageReader(*args, **kwargs)[source]¶: ( experimental ) This API may break in future versions, even between dot releases.