Dagstermill¶
-
dagstermill.
define_dagstermill_solid
(name, notebook_path, input_defs=None, output_defs=None, config_schema=None, required_resource_keys=None, output_notebook=None, asset_key_prefix=None)[source]¶ Wrap a Jupyter notebook in a solid.
- Parameters
name (str) – The name of the solid.
notebook_path (str) – Path to the backing notebook.
input_defs (Optional[List[InputDefinition]]) – The solid’s inputs.
output_defs (Optional[List[OutputDefinition]]) – The solid’s outputs. Your notebook should call
yield_result()
to yield each of these outputs.required_resource_keys (Optional[Set[str]]) – The string names of any required resources.
output_notebook (Optional[str]) – If set, will be used as the name of an injected output of type
FileHandle
that will point to the executed notebook (in addition to theAssetMaterialization
that is always created). This respects theFileManager
configured on the pipeline resources via the “file_manager” resource key, so, e.g., ifs3_file_manager
is configured, the output will be a : py:class:~dagster_aws.s3.S3FileHandle.asset_key_prefix (Optional[Union[List[str], str]]) – If set, will be used to prefix the asset keys for materialized notebooks.
- Returns
-
dagstermill.
get_context
(solid_config=None, mode_def=None, run_config=None)¶ Get a dagstermill execution context for interactive exploration and development.
- Parameters
solid_config (Optional[Any]) – If specified, this value will be made available on the context as its
solid_config
property.mode_def (Optional[
dagster.ModeDefinition
]) – If specified, defines the mode to use to construct the context. Specify this if you would like a context constructed with specificresource_defs
orlogger_defs
. By default, an ephemeral mode with a console logger will be constructed.run_config (Optional[dict]) – The environment config dict with which to construct the context.
- Returns
-
dagstermill.
yield_event
(dagster_event)¶ Yield a dagster event directly from notebook code.
When called interactively or in development, returns its input.
- Parameters
dagster_event (Union[
dagster.Materialization
,dagster.ExpectationResult
,dagster.TypeCheck
,dagster.Failure
]) – An event to yield back to Dagster.
-
dagstermill.
yield_result
(value, output_name='result')¶ Yield a result directly from notebook code.
When called interactively or in development, returns its input.
- Parameters
value (Any) – The value to yield.
output_name (Optional[str]) – The name of the result to yield (default:
'result'
).
-
class
dagstermill.
DagstermillExecutionContext
(pipeline_context: dagster.core.execution.context.system.SystemPipelineExecutionContext, resource_keys_to_init: Set[str], solid_name: str, solid_config: Any = None)[source]¶ Dagstermill-specific execution context.
Do not initialize directly: use
dagstermill.get_context()
.-
property
environment_config
¶ The environment_config for the context
- Type
dagster.EnvironmentConfig
-
get_tag
(key: str) → str[source]¶ Get a logging tag defined on the context.
- Parameters
key (str) – The key to get.
- Returns
str
-
has_tag
(key: str) → bool[source]¶ Check if a logging tag is defined on the context.
- Parameters
key (str) – The key to check.
- Returns
bool
-
property
log
¶ The log manager for the context.
Call, e.g.,
log.info()
to log messages through the Dagster machinery.
The logging tags for the context.
- Type
-
property
pipeline_def
¶ The pipeline definition for the context.
This will be a dagstermill-specific shim.
-
property
pipeline_run
¶ The pipeline run for the context.
- Type
-
property
resources
¶ A dynamically-created type whose properties allow access to resources.
- Type
collections.namedtuple
-
property
solid
¶ The solid for the context.
In interactive contexts, this may be a dagstermill-specific shim, depending whether a solid definition was passed to
dagstermill.get_context
.- Type
dagster.Solid
-
property
solid_config
¶ A dynamically-created type whose properties allow access to solid-specific config.
- Type
collections.namedtuple
-
property
solid_def
¶ The solid definition for the context.
In interactive contexts, this may be a dagstermill-specific shim, depending whether a solid definition was passed to
dagstermill.get_context
.
-
property