DagsterDocs

Shell (dagster-shell)

The Dagster shell library provides op factories for executing inline shell scripts or script files.

APIs

dagster_shell.create_shell_command_op(shell_command, name, description=None, required_resource_keys=None, tags=None)[source]

This function is a factory that constructs ops to execute a shell command.

Note that you can only use shell_command_op if you know the command you’d like to execute at pipeline construction time. If you’d like to construct shell commands dynamically during pipeline execution and pass them between ops, you should use shell_op instead.

Examples:

# pylint: disable=no-value-for-parameter
from dagster import graph
from dagster_shell import create_shell_command_op


@graph
def my_graph():
    a = create_shell_command_op('echo "hello, world!"', name="a")
    a()
Parameters
  • shell_command (str) – The shell command that the constructed op will execute.

  • name (str) – The name of the constructed op.

  • description (Optional[str]) – Human-readable description of this op.

  • required_resource_keys (Optional[Set[str]]) – Set of resource handles required by this op. Setting this ensures that resource spin up for the required resources will occur before the shell command is executed.

  • tags (Optional[Dict[str, Any]]) – Arbitrary metadata for the op. Frameworks may expect and require certain metadata to be attached to a op. Users should generally not set metadata directly. Values that are not strings will be json encoded and must meet the criteria that json.loads(json.dumps(value)) == value.

Raises

Failure – Raised when the shell command returns a non-zero exit code.

Returns

Returns the constructed op definition.

Return type

OpDefinition

dagster_shell.create_shell_script_op(shell_script_path, name='create_shell_script_op', input_defs=None, **kwargs)[source]

This function is a factory which constructs an op that will execute a shell command read from a script file.

Any kwargs passed to this function will be passed along to the underlying @op decorator. However, note that overriding config or output_defs is not supported.

You might consider using @graph to wrap this op in the cases where you’d like to configure the shell op with different config fields.

Examples:

# pylint: disable=no-value-for-parameter
from dagster import file_relative_path, graph
from dagster_shell import create_shell_script_op


@graph
def my_graph():
    a = create_shell_script_op(file_relative_path(__file__, "hello_world.sh"), name="a")
    a()
Parameters
  • shell_script_path (str) – The script file to execute.

  • name (str, optional) – The name of this op. Defaults to “create_shell_script_op”.

  • input_defs (List[InputDefinition], optional) – input definitions for the op. Defaults to a single Nothing input.

Raises

Failure – Raised when the shell command returns a non-zero exit code.

Returns

Returns the constructed op definition.

Return type

OpDefinition

dagster_shell.shell_op(context, shell_command)

Legacy APIs

dagster_shell.create_shell_command_solid(shell_command, name, description=None, required_resource_keys=None, tags=None)[source]

This function is a factory that constructs solids to execute a shell command.

Note that you can only use shell_command_solid if you know the command you’d like to execute at pipeline construction time. If you’d like to construct shell commands dynamically during pipeline execution and pass them between solids, you should use shell_solid instead.

Examples:

# pylint: disable=no-value-for-parameter
from dagster import pipeline
from dagster_shell import create_shell_command_solid


@pipeline
def pipe():
    a = create_shell_command_solid('echo "hello, world!"', name="a")
    a()
Parameters
  • shell_command (str) – The shell command that the constructed solid will execute.

  • name (str) – The name of the constructed solid.

  • description (Optional[str]) – Human-readable description of this solid.

  • required_resource_keys (Optional[Set[str]]) – Set of resource handles required by this solid. Setting this ensures that resource spin up for the required resources will occur before the shell command is executed.

  • tags (Optional[Dict[str, Any]]) – Arbitrary metadata for the solid. Frameworks may expect and require certain metadata to be attached to a solid. Users should generally not set metadata directly. Values that are not strings will be json encoded and must meet the criteria that json.loads(json.dumps(value)) == value.

Raises

Failure – Raised when the shell command returns a non-zero exit code.

Returns

Returns the constructed solid definition.

Return type

SolidDefinition

dagster_shell.create_shell_script_solid(shell_script_path, name='create_shell_script_solid', input_defs=None, **kwargs)[source]

This function is a factory which constructs a solid that will execute a shell command read from a script file.

Any kwargs passed to this function will be passed along to the underlying @solid decorator. However, note that overriding config or output_defs is not supported.

You might consider using @composite_solid to wrap this solid in the cases where you’d like to configure the shell solid with different config fields.

Examples:

# pylint: disable=no-value-for-parameter
from dagster import file_relative_path, pipeline
from dagster_shell import create_shell_script_solid


@pipeline
def pipe():
    a = create_shell_script_solid(file_relative_path(__file__, "hello_world.sh"), name="a")
    a()
Parameters
  • shell_script_path (str) – The script file to execute.

  • name (str, optional) – The name of this solid. Defaults to “create_shell_script_solid”.

  • input_defs (List[InputDefinition], optional) – input definitions for the solid. Defaults to a single Nothing input.

Raises

Failure – Raised when the shell command returns a non-zero exit code.

Returns

Returns the constructed solid definition.

Return type

SolidDefinition

dagster_shell.shell_solid(context, shell_command)