Dagstermill

dagstermill.define_dagstermill_solid(name, notebook_path, input_defs=None, output_defs=None, config_schema=None, required_resource_keys=None, output_notebook=None, asset_key_prefix=None)[source]

Wrap a Jupyter notebook in a solid.

Parameters

  • name (str) – The name of the solid.

  • notebook_path (str) – Path to the backing notebook.

  • input_defs (Optional[List[InputDefinition]]) – The solid’s inputs.

  • output_defs (Optional[List[OutputDefinition]]) – The solid’s outputs. Your notebook should call yield_result() to yield each of these outputs.

  • required_resource_keys (Optional[Set[str]]) – The string names of any required resources.

  • output_notebook (Optional[str]) – If set, will be used as the name of an injected output of type FileHandle that will point to the executed notebook (in addition to the AssetMaterialization that is always created). This respects the FileManager configured on the pipeline resources via the “file_manager” resource key, so, e.g., if s3_file_manager is configured, the output will be a : py:class:~dagster_aws.s3.S3FileHandle.

  • asset_key_prefix (Optional[Union[List[str], str]]) – If set, will be used to prefix the asset keys for materialized notebooks.

Returns

SolidDefinition

dagstermill.get_context(solid_config=None, mode_def=None, run_config=None)

Get a dagstermill execution context for interactive exploration and development.

Parameters

  • solid_config (Optional[Any]) – If specified, this value will be made available on the context as its solid_config property.

  • mode_def (Optional[dagster.ModeDefinition]) – If specified, defines the mode to use to construct the context. Specify this if you would like a context constructed with specific resource_defs or logger_defs. By default, an ephemeral mode with a console logger will be constructed.

  • run_config (Optional[dict]) – The environment config dict with which to construct the context.

Returns

DagstermillExecutionContext

dagstermill.yield_event(dagster_event)

Yield a dagster event directly from notebook code.

When called interactively or in development, returns its input.

Parameters

dagster_event (Union[dagster.Materialization, dagster.ExpectationResult, dagster.TypeCheck, dagster.Failure]) – An event to yield back to Dagster.

dagstermill.yield_result(value, output_name='result')

Yield a result directly from notebook code.

When called interactively or in development, returns its input.

Parameters

  • value (Any) – The value to yield.

  • output_name (Optional[str]) – The name of the result to yield (default: 'result').

class dagstermill.DagstermillExecutionContext(pipeline_context: dagster.core.execution.context.system.SystemPipelineExecutionContext, resource_keys_to_init: Set[str], solid_name: str, solid_config: Any = None)[source]

Dagstermill-specific execution context.

Do not initialize directly: use dagstermill.get_context().

property environment_config

The environment_config for the context

Type

dagster.EnvironmentConfig

get_tag(key: str) → str[source]

Get a logging tag defined on the context.

Parameters

key (str) – The key to get.

Returns

str

has_tag(key: str) → bool[source]

Check if a logging tag is defined on the context.

Parameters

key (str) – The key to check.

Returns

bool

property log

The log manager for the context.

Call, e.g., log.info() to log messages through the Dagster machinery.

Type

dagster.DagsterLogManager

property logging_tags

The logging tags for the context.

Type

dict

property pipeline_def

The pipeline definition for the context.

This will be a dagstermill-specific shim.

Type

dagster.PipelineDefinition

property pipeline_run

The pipeline run for the context.

Type

dagster.PipelineRun

property resources

A dynamically-created type whose properties allow access to resources.

Type

collections.namedtuple

property run_config

The run_config for the context.

Type

dict

property run_id

The run_id for the context.

Type

str

property solid

The solid for the context.

In interactive contexts, this may be a dagstermill-specific shim, depending whether a solid definition was passed to dagstermill.get_context.

Type

dagster.Solid

property solid_config

A dynamically-created type whose properties allow access to solid-specific config.

Type

collections.namedtuple

property solid_def

The solid definition for the context.

In interactive contexts, this may be a dagstermill-specific shim, depending whether a solid definition was passed to dagstermill.get_context.

Type

dagster.SolidDefinition

class dagstermill.DagstermillError[source]

Base class for errors raised by dagstermill.