Dagstermill¶

dagstermill.define_dagstermill_solid(name, notebook_path, input_defs=None, output_defs=None, config_schema=None, required_resource_keys=None, output_notebook=None)[source]¶

Wrap a Jupyter notebook in a solid.

Parameters

name (str) – The name of the solid.
notebook_path (str) – Path to the backing notebook.
input_defs (Optional[List[InputDefinition]]) – The solid’s inputs.
output_defs (Optional[List[OutputDefinition]]) – The solid’s outputs. Your notebook should call yield_result() to yield each of these outputs.
required_resource_keys (Optional[Set[str]]) – The string names of any required resources.
output_notebook (Optional[str]) – If set, will be used as the name of an injected output of type FileHandle that will point to the executed notebook (in addition to the AssetMaterialization that is always created). This respects the FileManager configured on the pipeline system storage, so, e.g., if s3_system_storage is configured, the output will be a S3FileHandle.

Returns

SolidDefinition

dagstermill.get_context(solid_config=None, mode_def=None, run_config=None)¶

Get a dagstermill execution context for interactive exploration and development.

Parameters

solid_config (Optional[Any]) – If specified, this value will be made available on the context as its solid_config property.
mode_def (Optional[dagster.ModeDefinition]) – If specified, defines the mode to use to construct the context. Specify this if you would like a context constructed with specific resource_defs or logger_defs. By default, an ephemeral mode with a console logger will be constructed.
run_config (Optional[dict]) – The environment config dict with which to construct the context.

Returns

DagstermillExecutionContext

dagstermill.yield_event(dagster_event)¶

Yield a dagster event directly from notebook code.

When called interactively or in development, returns its input.

Parameters: dagster_event (Union[dagster.Materialization, dagster.ExpectationResult, dagster.TypeCheck, dagster.Failure]) – An event to yield back to Dagster.

dagstermill.yield_result(value, output_name='result')¶

Yield a result directly from notebook code.

When called interactively or in development, returns its input.

Parameters

value (Any) – The value to yield.
output_name (Optional[str]) – The name of the result to yield (default: 'result').

class dagstermill.DagstermillExecutionContext(pipeline_context, resource_keys_to_init, solid_name, solid_config=None)[source]¶

Dagstermill-specific execution context.

Do not initialize directly: use dagstermill.get_context().

property environment_config¶

The environment_config for the context

Type: dagster.EnvironmentConfig

get_tag(key)[source]¶

Get a logging tag defined on the context.

Parameters: key (str) – The key to get.
Returns: str

has_tag(key)[source]¶

Check if a logging tag is defined on the context.

Parameters: key (str) – The key to check.
Returns: bool

property log¶

The log manager for the context.

Call, e.g., log.info() to log messages through the Dagster machinery.

Type: dagster.LogManager

property logging_tags¶

The logging tags for the context.

Type: dict

property pipeline_def¶

The pipeline definition for the context.

This will be a dagstermill-specific shim.

Type: dagster.PipelineDefinition

property pipeline_run¶

The pipeline run for the context.

Type: dagster.PipelineRun

property resources¶

A dynamically-created type whose properties allow access to resources.

Type: collections.namedtuple

property run_config¶

The run_config for the context.

Type: dict

property run_id¶

The run_id for the context.

Type: str

property solid¶

The solid for the context.

In interactive contexts, this may be a dagstermill-specific shim, depending whether a solid definition was passed to dagstermill.get_context.

Type: dagster.Solid

property solid_config¶

A dynamically-created type whose properties allow access to solid-specific config.

Type: collections.namedtuple

property solid_def¶

The solid definition for the context.

In interactive contexts, this may be a dagstermill-specific shim, depending whether a solid definition was passed to dagstermill.get_context.

Type: dagster.SolidDefinition

class dagstermill.DagstermillError[source]¶: Base class for errors raised by dagstermill.

class dagstermill.DagstermillExecutionError(*args, **kwargs)[source]¶: Errors raised during the execution of dagstermill solids.