Config

Config Types

The following types are used to describe the schema of configuration data via config. They are used in conjunction with the builtin types above.

class dagster.ConfigSchema[source]

ConfigSchema is a placeholder type. Any time that it appears in documentation, it means that any of the following types are acceptable:

  1. A Python scalar type that resolves to a Dagster config type (int, float, bool, or str

    e.g. @solid(config_schema=int) or @solid(config_schema=str)

  2. Built-in python collections such as list, or dict). list is exactly equivalent to Array[Any] and dict is equivalent to Permissive()

    e.g. @solid(config_schema=list) or @solid(config_schema=dict)

  3. A Dagster config type: Int, IntSource, Float, Bool, String, StringSource, Any, Array, Noneable, Enum, Selector, Shape, or Permissive.

  4. A bare python dictionary, which will be automatically wrapped in Shape. Values of the dictionary are resolved recursively according to the same rules. E.g.

    • {'some_config': str} is equivalent to Shape({'some_config: str}).

    • {'some_config1': {'some_config2': str}} is equivalent to Shape({'some_config1: Shape({'some_config2: str})}).

  5. A bare python list of length one, whose single element will be wrapped in a Array is resolved recursively according to the same rules. E.g.

    • [str] is equivalent to Array[str].

    • [[str]] is equivalent to Array[Array[str]].

    • [{'some_config': str}] is equivalent to Array(Shape({'some_config: str})).

  6. An instance of Field.

class dagster.Field(config, default_value=<class 'dagster.config.field_utils.__FieldValueSentinel'>, is_required=None, description=None)[source]

Defines the schema for a configuration field.

Fields are used in config schema instead of bare types when one wants to add a description, a default value, or to mark it as not required.

Config fields are parsed according to their schemas in order to yield values available at pipeline execution time through the config system. Config fields can be set on solids, on loaders and materializers for custom, and on other pluggable components of the system, such as resources, loggers, and executors.

Parameters

  • config (Any) –

    The schema for the config. This value can be any of:

    1. A Python primitive type that resolves to a Dagster config type (int, float, bool, str, or list).

    2. A Dagster config type: Int, Float, Bool, String, StringSource, Path, Any, Array, Noneable, Enum, Selector, Shape, or Permissive.

    3. A bare python dictionary, which will be automatically wrapped in Shape. Values of the dictionary are resolved recursively according to the same rules.

    4. A bare python list of length one which itself is config type. Becomes Array with list element as an argument.

  • default_value (Any) –

    A default value for this field, conformant to the schema set by the dagster_type argument. If a default value is provided, is_required should be False.

    Note: for config types that do post processing such as Enum, this value must be the pre processed version, ie use ExampleEnum.VALUE.name instead of ExampleEnum.VALUE

  • is_required (bool) – Whether the presence of this field is required. Defaults to true. If is_required is True, no default value should be provided.

  • description (str) – A human-readable description of this config field.

Examples

class dagster.Selector(fields, description=None)[source]

Define a config field requiring the user to select one option.

Selectors are used when you want to be able to present several different options in config but allow only one to be selected. For example, a single input might be read in from either a csv file or a parquet file, but not both at once.

Note that in some other type systems this might be called an ‘input union’.

Functionally, a selector is like a Dict, except that only one key from the dict can be specified in valid config.

Parameters

fields (Dict[str, Field]) – The fields from which the user must select.

Examples:

@solid(
    config_schema=Field(
        Selector(
            {
                'haw': {'whom': Field(String, default_value='honua', is_required=False)},
                'cn': {'whom': Field(String, default_value='世界', is_required=False)},
                'en': {'whom': Field(String, default_value='world', is_required=False)},
            }
        ),
        is_required=False,
        default_value={'en': {'whom': 'world'}},
    )
)
def hello_world_with_default(context):
    if 'haw' in context.solid_config:
        return 'Aloha {whom}!'.format(whom=context.solid_config['haw']['whom'])
    if 'cn' in context.solid_config:
        return '你好，{whom}!'.format(whom=context.solid_config['cn']['whom'])
    if 'en' in context.solid_config:
        return 'Hello, {whom}!'.format(whom=context.solid_config['en']['whom'])
class dagster.Permissive(fields=None, description=None)[source]

Defines a config dict with a partially specified schema.

A permissive dict allows partial specification of the config schema. Any fields with a specified schema will be type checked. Other fields will be allowed, but will be ignored by the type checker.

Parameters

fields (Dict[str, Field]) – The partial specification of the config dict.

Examples

@solid(config_schema=Field(Permissive({'required': Field(String)})))
def partially_specified_config(context) -> List:
    return sorted(list(context.solid_config.items()))
class dagster.Shape(fields, description=None)[source]

Schema for configuration data with string keys and typed values via Field .

Parameters

fields (Dict[str, Field]) –

class dagster.Array(inner_type)[source]
class dagster.Noneable(inner_type)[source]
class dagster.Enum(name, enum_values)[source]

Defines a enum configuration type that allows one of a defined set of possible values.

Parameters

Example

@solid(
    config_schema=Field(
        Enum(
            'CowboyType',
            [
                EnumValue('good'),
                EnumValue('bad'),
                EnumValue('ugly'),
            ]
        )
    )
)
def resolve_standoff(context):
    # ...
class dagster.EnumValue(config_value, python_value=None, description=None)[source]

Define an entry in a Enum().

Parameters

  • config_value (str) – The string representation of the config to accept when passed.

  • python_value (Optional[Any]) – The python value to convert the enum entry in to. Defaults to the config_value.

  • description (Optional[str]) –

dagster.StringSource

Use this type when you want to read a string config value from an environment variable. The value passed to a config field of this type may either be a string literal, or a selector describing how to look up the value from the executing process’s environment variables.

Examples:

@solid(config_schema=StringSource)
def secret_solid(context) -> str:
    return context.solid_config

execute_solid(
    secret_solid,
    run_config={
        'solids': {'secret_solid': {'config': 'test_value'}}
    }
)

execute_solid(
    secret_solid,
    run_config={
        'solids': {'secret_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE'}}}
    }
)
dagster.IntSource

Use this type when you want to read an integer config value from an environment variable. The value passed to a config field of this type may either be a integer literal, or a selector describing how to look up the value from the executing process’s environment variables.

Examples:

@solid(config_schema=IntSource)
def secret_int_solid(context) -> str:
    return context.solid_config

execute_solid(
    secret_int_solid,
    run_config={
        'solids': {'secret_int_solid': {'config': 3}}
    }
)

execute_solid(
    secret_int_solid,
    run_config={
        'solids': {'secret_int_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_INT'}}}
    }
)

Config Utilities

@dagster.configured(configurable, config_schema=None, **kwargs)[source]

A decorator that makes it easy to create a function-configured version of an object. The following definition types can be configured using this function: ResourceDefinition, ExecutorDefinition, CompositeSolidDefinition, SolidDefinition, LoggerDefinition, IntermediateStorageDefinition, and SystemStorageDefinition.

If the config that will be supplied to the object is constant, you may alternatively invoke this and call the result with a dict of config values to be curried. Examples of both strategies below.

Parameters

  • configurable (IConfigMappable) – An object that can be configured.

  • config_schema (ConfigSchema) – The config schema that the inputs to the decorated function must satisfy.

  • **kwargs – Arbitrary keyword arguments that will be passed to the initializer of the returned object.

Returns (Callable[[Union[Any, Callable[[Any], Any]]], IConfigMappable])

Examples

dev_s3 = configured(s3_resource)({'bucket': 'dev'})

@configured(s3_resource):
def dev_s3(_):
    return {'bucket': 'dev'}

@configured(s3_resource, {'bucket_prefix', str}):
def dev_s3(config):
    return {'bucket': config['bucket_prefix'] + 'dev'}