Types

Dagster includes facilities for typing the input and output values of ops (“runtime” types).

Built-in types

Use this type only for inputs and outputs, in order to establish an execution dependency without communicating a value. Inputs of this type will not be passed to the op compute function, so it is necessary to use the explicit In API to define them rather than the Python 3 type hint syntax.

All values are considered to be instances of Nothing.

Examples:

@op
def wait(_) -> Nothing:
    time.sleep(1)
    return

@op(
    ins={"ready": In(dagster_type=Nothing)},
)
def done(_) -> str:
    return 'done'

@job
def nothing_job():
    done(wait())

# Any value will pass the type check for Nothing
@op
def wait_int(_) -> Int:
    time.sleep(1)
    return 1

@job
def nothing_int_job():
    done(wait_int())

Making New Types

class dagster.DagsterType

Define a type in dagster. These can be used in the inputs and outputs of ops.

Parameters:

type_check_fn (Callable[[TypeCheckContext, Any], [Union[bool, TypeCheck]]]) – The function that defines the type check. It takes the value flowing through the input or output of the op. If it passes, return either True or a TypeCheck with success set to True. If it fails, return either False or a TypeCheck with success set to False. The first argument must be named context (or, if unused, _, _context, or context_). Use required_resource_keys for access to resources.
key (Optional[str]) –

The unique key to identify types programmatically. The key property always has a value. If you omit key to the argument to the init function, it instead receives the value of name. If neither key nor name is provided, a CheckError is thrown.

In the case of a generic type such as List or Optional, this is generated programmatically based on the type parameters.
name (Optional[str]) – A unique name given by a user. If key is None, key becomes this value. Name is not given in a case where the user does not specify a unique name for this type, such as a generic class.
description (Optional[str]) – A markdown-formatted string, displayed in tooling.
loader (Optional[DagsterTypeLoader]) – An instance of a class that inherits from DagsterTypeLoader and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the @dagster_type_loader decorator to construct these arguments.
required_resource_keys (Optional[Set[str]]) – Resource keys required by the type_check_fn.
is_builtin (bool) – Defaults to False. This is used by tools to display or filter built-in types (such as String, Int) to visually distinguish them from user-defined types. Meant for internal use.
kind (DagsterTypeKind) – Defaults to None. This is used to determine the kind of runtime type for InputDefinition and OutputDefinition type checking.
typing_type – Defaults to None. A valid python typing type (e.g. Optional[List[int]]) for the value contained within the DagsterType. Meant for internal use.

type_check

Type check the value against the type.

Parameters:

context (TypeCheckContext) – The context of the type check.
value (Any) – The value to check.

Returns: The result of the type check.Return type: TypeCheck

property description

Description of the type, or None if not provided.

Type: Optional[str]

property display_name: Either the name or key (if name is None) of the type, overridden in many subclasses.

property has_unique_name

Whether the type has a unique name.

Type: bool

property loader

Loader for this type, if any.

Type: Optional[DagsterTypeLoader]

property required_resource_keys

Set of resource keys required by the type check function.

Type: AbstractSet[str]

property typing_type

The python typing type for this type.

Type: Any

property unique_name: The unique name of this type. Can be None if the type is not unique, such as container types.

dagster.PythonObjectDagsterType

Define a type in dagster whose typecheck is an isinstance check.

Specifically, the type can either be a single python type (e.g. int), or a tuple of types (e.g. (int, float)) which is treated as a union.

Examples:

ntype = PythonObjectDagsterType(python_type=int)
assert ntype.name == 'int'
assert_success(ntype, 1)
assert_failure(ntype, 'a')

ntype = PythonObjectDagsterType(python_type=(int, float))
assert ntype.name == 'Union[int, float]'
assert_success(ntype, 1)
assert_success(ntype, 1.5)
assert_failure(ntype, 'a')

Parameters:

python_type (Union[Type, Tuple[Type, ...]) – The dagster typecheck function calls instanceof on this type.
name (Optional[str]) – Name the type. Defaults to the name of python_type.
key (Optional[str]) – Key of the type. Defaults to name.
description (Optional[str]) – A markdown-formatted string, displayed in tooling.
loader (Optional[DagsterTypeLoader]) – An instance of a class that inherits from DagsterTypeLoader and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the @dagster_type_loader decorator to construct these arguments.

dagster.dagster_type_loader

Create an dagster type loader that maps config data to a runtime value.

The decorated function should take the execution context and parsed config value and return the appropriate runtime value.

Parameters: config_schema (ConfigSchema) – The schema for the config that’s passed to the decorated function. Examples:

@dagster_type_loader(Permissive())
def load_dict(_context, value):
    return value

class dagster.DagsterTypeLoader

Dagster type loaders are used to load unconnected inputs of the dagster type they are attached to.

The recommended way to define a type loader is with the @dagster_type_loader decorator.

class dagster.DagsterTypeLoaderContext

The context object provided to a @dagster_type_loader-decorated function during execution.

Users should not construct this object directly.

property job_def: The underlying job definition being executed.

property op_def: The op for which type loading is occurring.

property resources: The resources available to the type loader, specified by the required_resource_keys argument of the decorator.

dagster.usable_as_dagster_type

Decorate a Python class to make it usable as a Dagster Type.

This is intended to make it straightforward to annotate existing business logic classes to make them dagster types whose typecheck is an isinstance check against that python class.

Parameters:

python_type (cls) – The python type to make usable as python type.
name (Optional[str]) – Name of the new Dagster type. If None, the name (__name__) of the python_type will be used.
description (Optional[str]) – A user-readable description of the type.
loader (Optional[DagsterTypeLoader]) – An instance of a class that inherits from DagsterTypeLoader and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the @dagster_type_loader decorator to construct these arguments.

Examples:

# dagster_aws.s3.file_manager.S3FileHandle
@usable_as_dagster_type
class S3FileHandle(FileHandle):
    def __init__(self, s3_bucket, s3_key):
        self._s3_bucket = check.str_param(s3_bucket, 's3_bucket')
        self._s3_key = check.str_param(s3_key, 's3_key')

    @property
    def s3_bucket(self):
        return self._s3_bucket

    @property
    def s3_key(self):
        return self._s3_key

    @property
    def path_desc(self):
        return self.s3_path

    @property
    def s3_path(self):
        return 's3://{bucket}/{key}'.format(bucket=self.s3_bucket, key=self.s3_key)

dagster.make_python_type_usable_as_dagster_type: Take any existing python type and map it to a dagster type (generally created with DagsterType) This can only be called once on a given python type.

Testing Types

dagster.check_dagster_type

Test a custom Dagster type.

Parameters:

dagster_type (Any) – The Dagster type to test. Should be one of the built-in typesbuilt-in types, a dagster type explicitly constructed with as_dagster_type(), @usable_as_dagster_type, or PythonObjectDagsterType(), or a Python type.
value (Any) – The runtime value to test.

Returns: The result of the type check.Return type: TypeCheck Examples:

assert check_dagster_type(Dict[Any, Any], {'foo': 'bar'}).success

Built-in types​

Making New Types​

Testing Types​

Built-in types

Making New Types

Testing Types