DagsterDocs

Modes & Resources

Modes

class dagster.ModeDefinition(name=None, resource_defs=None, logger_defs=None, executor_defs=None, description=None, intermediate_storage_defs=None, _config_mapping=None, _partitioned_config=None)[source]

Define a mode in which a pipeline can operate.

A mode provides pipelines with a set of resource implementations, loggers, system storages, and executors.

Parameters
  • name (Optional[str]) – The name of the mode. Must be unique within the PipelineDefinition to which the mode is attached. (default: “default”).

  • resource_defs (Optional[Dict[str, ResourceDefinition]]) – A dictionary of string resource keys to their implementations. Individual solids may require resources to be present by these keys.

  • logger_defs (Optional[Dict[str, LoggerDefinition]]) – A dictionary of string logger identifiers to their implementations.

  • executor_defs (Optional[List[ExecutorDefinition]]) – The set of executors available when executing in this mode. By default, this will be the ‘in_process’ and ‘multiprocess’ executors (default_executors).

  • description (Optional[str]) – A human-readable description of the mode.

  • intermediate_storage_defs (Optional[List[IntermediateStorageDefinition]]) – The set of intermediate storage options available when executing in this mode. By default, this will be the ‘in_memory’ and ‘filesystem’ system storages.

  • _config_mapping (Optional[ConfigMapping]) – Experimental

  • _partitions (Optional[PartitionedConfig]) – Experimental


Resources

@dagster.resource(config_schema=None, description=None, required_resource_keys=None, version=None)[source]

Define a resource.

The decorated function should accept an InitResourceContext and return an instance of the resource. This function will become the resource_fn of an underlying ResourceDefinition.

If the decorated function yields once rather than returning (in the manner of functions decorable with @contextlib.contextmanager) then the body of the function after the yield will be run after execution resolves, allowing users to write their own teardown/cleanup logic.

Parameters
  • config_schema (Optional[ConfigSchema]) – The schema for the config. Configuration data available in init_context.resource_config. If not set, Dagster will accept any config provided.

  • description (Optional[str]) – A human-readable description of the resource.

  • version (Optional[str]) – (Experimental) The version of a resource function. Two wrapped resource functions should only have the same version if they produce the same resource definition when provided with the same inputs.

  • required_resource_keys (Optional[Set[str]]) – Keys for the resources required by this resource.

class dagster.ResourceDefinition(resource_fn, config_schema=None, description=None, required_resource_keys=None, version=None)[source]

Core class for defining resources.

Resources are scoped ways to make external resources (like database connections) available to solids during pipeline execution and to clean up after execution resolves.

If resource_fn yields once rather than returning (in the manner of functions decorable with @contextlib.contextmanager) then the body of the function after the yield will be run after execution resolves, allowing users to write their own teardown/cleanup logic.

Depending on your executor, resources may be instantiated and cleaned up more than once in a pipeline execution.

Parameters
  • resource_fn (Callable[[InitResourceContext], Any]) – User-provided function to instantiate the resource, which will be made available to solid executions keyed on the context.resources object.

  • config_schema (Optional[ConfigSchema) – The schema for the config. If set, Dagster will check that config provided for the resource matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the resource.

  • description (Optional[str]) – A human-readable description of the resource.

  • required_resource_keys – (Optional[Set[str]]) Keys for the resources required by this resource. A DagsterInvariantViolationError will be raised during initialization if dependencies are cyclic.

  • version (Optional[str]) – (Experimental) The version of the resource’s definition fn. Two wrapped resource functions should only have the same version if they produce the same resource definition when provided with the same inputs.

configured(config_or_config_fn, config_schema=None, description=None)

Wraps this object in an object of the same type that provides configuration to the inner object.

Parameters
  • config_or_config_fn (Union[Any, Callable[[Any], Any]]) – Either (1) Run configuration that fully satisfies this object’s config schema or (2) A function that accepts run configuration and returns run configuration that fully satisfies this object’s config schema. In the latter case, config_schema must be specified. When passing a function, it’s easiest to use configured().

  • config_schema (ConfigSchema) – If config_or_config_fn is a function, the config schema that its input must satisfy.

  • description (Optional[str]) – Description of the new definition. If not specified, inherits the description of the definition being configured.

Returns (ConfigurableDefinition): A configured version of this object.

static hardcoded_resource(value, description=None)[source]

A helper function that creates a ResourceDefinition with a hardcoded object.

Parameters
  • value (Any) – The value that will be accessible via context.resources.resource_name.

  • description ([Optional[str]]) – The description of the resource. Defaults to None.

Returns

A hardcoded resource.

Return type

[ResourceDefinition]

static mock_resource(description=None)[source]

A helper function that creates a ResourceDefinition which wraps a mock.MagicMock.

Parameters

description ([Optional[str]]) – The description of the resource. Defaults to None.

Returns

A resource that creates the magic methods automatically and helps

you mock existing resources.

Return type

[ResourceDefinition]

static none_resource(description=None)[source]

A helper function that returns a none resource.

Parameters

description ([Optional[str]]) – The description of the resource. Defaults to None.

Returns

A resource that does nothing.

Return type

[ResourceDefinition]

class dagster.InitResourceContext(resource_config, resources, resource_def=None, instance=None, pipeline_run=None, log_manager=None, pipeline_def_for_backwards_compat=None)[source]

Resource-specific initialization context.

resource_config

The configuration data provided by the run config. The schema for this data is defined by the config_field argument to ResourceDefinition.

Type

Any

resource_def

The definition of the resource currently being constructed.

Type

ResourceDefinition

log_manager

The log manager for this run of the pipeline

Type

DagsterLogManager

resources

The resources that are available to the resource that we are initalizing.

Type

ScopedResources

pipeline_run

The pipeline run to use. When initializing resources outside of execution context, this will be None.

Type

Optional[PipelineRun]

run_id

The id for this run of the pipeline. When initializing resources outside of execution context, this will be None.

Type

Optional[str]

dagster.make_values_resource(**kwargs)[source]

A helper function that creates a ResourceDefinition to take in user-defined values.

This is useful for sharing values between solids.

Parameters

**kwargs – Arbitrary keyword arguments that will be passed to the config schema of the returned resource definition. If not set, Dagster will accept any config provided for the resource.

For example:

@solid(required_resource_keys={"globals"})
def my_solid_a(context):
    print(context.resources.globals["my_str_var"])

@pipeline(
    mode_defs=[
        ModeDefinition(
            resource_defs={"globals": make_values_resource(my_str_var=str, my_int_var=int)}
        )
    ]
)
def my_pipeline():
    my_solid()
Returns

A resource that passes in user-defined values.

Return type

ResourceDefinition

dagster.build_init_resource_context(config=None, resources=None, instance=None)[source]

Builds resource initialization context from provided parameters.

build_init_resource_context can be used as either a function or context manager. If there is a provided resource to build_init_resource_context that is a context manager, then it must be used as a context manager. This function can be used to provide the context argument to the invocation of a resource.

Parameters
  • resources (Optional[Dict[str, Any]]) – The resources to provide to the context. These can be either values or resource definitions.

  • config (Optional[Any]) – The resource config to provide to the context.

  • instance (Optional[DagsterInstance]) – The dagster instance configured for the context. Defaults to DagsterInstance.ephemeral().

Examples

context = build_init_resource_context()
resource_to_init(context)

with build_init_resource_context(
    resources={"foo": context_manager_resource}
) as context:
    resource_to_init(context)