Skip to main content

Orchestration on Docker

APIs

dagster_docker.DockerRunLauncher RunLauncher

Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition
beta

This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster_docker import docker_executor

from dagster import job

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.

Ops​

dagster_docker.docker_container_op = <dagster._core.definitions.op_definition.OpDefinition object>
beta

This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.

An op that runs a Docker container using the docker Python API.

Contrast with the docker_executor, which runs each Dagster op in a Dagster job in its own Docker container.

This op may be useful when:

  • You need to orchestrate a command that isn’t a Dagster op (or isn’t written in Python)
  • You want to run the rest of a Dagster job using a specific executor, and only a single op in docker.

For example:

from dagster_docker import docker_container_op

from dagster import job

first_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo HELLO"],
    },
    name="first_op",
)
second_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo GOODBYE"],
    },
    name="second_op",
)

@job
def full_job():
    second_op(first_op())

You can create your own op with the same implementation by calling the execute_docker_container function inside your own op.

dagster_docker.execute_docker_container
beta

This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.

This function is a utility for executing a Docker container from within a Dagster op.

Parameters:

  • image (str) – The image to use for the launched Docker container.
  • entrypoint (Optional[Sequence[str]]) – The ENTRYPOINT to run in the launched Docker container. Default: None.
  • command (Optional[Sequence[str]]) – The CMD to run in the launched Docker container. Default: None.
  • networks (Optional[Sequence[str]]) – Names of the Docker networks to which to connect the launched container. Default: None.
  • registry – (Optional[Mapping[str, str]]): Information for using a non local/public Docker registry. Can have “url”, “username”, or “password” keys.
  • env_vars (Optional[Sequence[str]]) – List of environemnt variables to include in the launched container. ach can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the calling environment.
  • container_kwargs (Optional[Dict[str[Any]]]) – key-value pairs that can be passed into containers.create in the Docker Python API. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

Pipes​

class dagster_docker.PipesDockerClient

A pipes client that runs external processes in docker containers.

By default context is injected via environment variables and messages are parsed out of the log stream, with other logs forwarded to stdout of the orchestration process.

Parameters:

  • env (Optional[Mapping[str, str]]) – An optional dict of environment variables to pass to the container.
  • register (Optional[Mapping[str, str]]) – An optional dict of registry credentials to login to the docker client.
  • context_injector (Optional[PipesContextInjector]) – A context injector to use to inject context into the docker container process. Defaults to PipesEnvContextInjector.
  • message_reader (Optional[PipesMessageReader]) – A message reader to use to read messages from the docker container process. Defaults to DockerLogsMessageReader.
class dagster_docker.PipesDockerLogsMessageReader