Config¶
Config Types¶
The following types are used to describe the schema of configuration
data via config
. They are used in conjunction with the
builtin types above.
-
class
dagster.
ConfigSchema
[source]¶ This is a placeholder type. Any time that it appears in documentation, it means that any of the following types are acceptable:
A Python scalar type that resolves to a Dagster config type (
int
,float
,bool
, orstr
). For example:@solid(config_schema=int)
@solid(config_schema=str)
A built-in python collection (
list
, ordict
).list
is exactly equivalent toArray
[Any
] anddict
is equivalent toPermissive
. For example:@solid(config_schema=list)
@solid(config_schema=dict)
A Dagster config type:
A bare python dictionary, which will be automatically wrapped in
Shape
. Values of the dictionary are resolved recursively according to the same rules. For example:{'some_config': str}
is equivalent toShape({'some_config: str})
.{'some_config1': {'some_config2': str}}
is equivalent toShape({'some_config1: Shape({'some_config2: str})})
.
A bare python list of length one, whose single element will be wrapped in a
Array
is resolved recursively according to the same rules. For example:[str]
is equivalent toArray[str]
.[[str]]
is equivalent toArray[Array[str]]
.[{'some_config': str}]
is equivalent toArray(Shape({'some_config: str}))
.
An instance of
Field
.
-
class
dagster.
Field
(config, default_value=<class 'dagster.config.field_utils.__FieldValueSentinel'>, is_required=None, description=None)[source]¶ Defines the schema for a configuration field.
Fields are used in config schema instead of bare types when one wants to add a description, a default value, or to mark it as not required.
Config fields are parsed according to their schemas in order to yield values available at pipeline execution time through the config system. Config fields can be set on solids, on loaders and materializers for custom, and on other pluggable components of the system, such as resources, loggers, and executors.
- Parameters
config (Any) –
The schema for the config. This value can be any of:
A Python primitive type that resolves to a Dagster config type (
int
,float
,bool
,str
, orlist
).A Dagster config type:
A bare python dictionary, which will be automatically wrapped in
Shape
. Values of the dictionary are resolved recursively according to the same rules.A bare python list of length one which itself is config type. Becomes
Array
with list element as an argument.
default_value (Any) –
A default value for this field, conformant to the schema set by the
dagster_type
argument. If a default value is provided,is_required
should beFalse
.Note: for config types that do post processing such as Enum, this value must be the pre processed version, ie use
ExampleEnum.VALUE.name
instead ofExampleEnum.VALUE
is_required (bool) – Whether the presence of this field is required. Defaults to true. If
is_required
isTrue
, no default value should be provided.description (str) – A human-readable description of this config field.
Examples:
@solid( config_schema={ 'word': Field(str, description='I am a word.'), 'repeats': Field(Int, default_value=1, is_required=False), } ) def repeat_word(context): return context.solid_config['word'] * context.solid_config['repeats']
-
class
dagster.
Selector
(fields, description=None)[source]¶ Define a config field requiring the user to select one option.
Selectors are used when you want to be able to present several different options in config but allow only one to be selected. For example, a single input might be read in from either a csv file or a parquet file, but not both at once.
Note that in some other type systems this might be called an ‘input union’.
Functionally, a selector is like a
Dict
, except that only one key from the dict can be specified in valid config.Examples:
@solid( config_schema=Field( Selector( { 'haw': {'whom': Field(String, default_value='honua', is_required=False)}, 'cn': {'whom': Field(String, default_value='世界', is_required=False)}, 'en': {'whom': Field(String, default_value='world', is_required=False)}, } ), is_required=False, default_value={'en': {'whom': 'world'}}, ) ) def hello_world_with_default(context): if 'haw' in context.solid_config: return 'Aloha {whom}!'.format(whom=context.solid_config['haw']['whom']) if 'cn' in context.solid_config: return '你好,{whom}!'.format(whom=context.solid_config['cn']['whom']) if 'en' in context.solid_config: return 'Hello, {whom}!'.format(whom=context.solid_config['en']['whom'])
-
class
dagster.
Permissive
(fields=None, description=None)[source]¶ Defines a config dict with a partially specified schema.
A permissive dict allows partial specification of the config schema. Any fields with a specified schema will be type checked. Other fields will be allowed, but will be ignored by the type checker.
Examples:
@solid(config_schema=Field(Permissive({'required': Field(String)}))) def partially_specified_config(context) -> List: return sorted(list(context.solid_config.items()))
-
class
dagster.
Shape
(fields, description=None)[source]¶ Schema for configuration data with string keys and typed values via
Field
.Unlike
Permissive
, unspecified fields are not allowed and will throw aDagsterInvalidConfigError
.
-
class
dagster.
Array
(inner_type)[source]¶ Defines an array (list) configuration type that contains values of type
inner_type
.- Parameters
inner_type (type) – The type of the values that this configuration type can contain.
-
class
dagster.
Noneable
(inner_type)[source]¶ Defines a configuration type that is the union of
NoneType
and the typeinner_type
.- Parameters
inner_type (type) – The type of the values that this configuration type can contain.
Examples:
config_schema={"name": Noneable(str)} config={"name": "Hello"} # Ok config={"name": None} # Ok config={} # Error
-
class
dagster.
Enum
(name, enum_values)[source]¶ Defines a enum configuration type that allows one of a defined set of possible values.
- Parameters
Examples:
@solid( config_schema=Field( Enum( 'CowboyType', [ EnumValue('good'), EnumValue('bad'), EnumValue('ugly'), ] ) ) ) def resolve_standoff(context): # ...
-
classmethod
from_python_enum
(enum, name=None)[source]¶ Create a Dagster enum corresponding to an existing Python enum.
- Parameters
enum (enum.EnumMeta) – The class representing the enum.
name (Optional[str]) – The name for the enum. If not present, enum.__name__ will be used.
Example:
class Color(enum.Enum): RED = enum.auto() GREEN = enum.auto() BLUE = enum.auto() @solid( config_schema={"color": Field(Enum.from_python_enum(Color))} ) def select_color(context): # ...
-
class
dagster.
EnumValue
(config_value, python_value=None, description=None)[source]¶ Define an entry in a
Enum
.
-
class
dagster.
ScalarUnion
(scalar_type, non_scalar_schema, _key=None)[source]¶ Defines a configuration type that accepts a scalar value OR a non-scalar value like a
List
,Dict
, orSelector
.This allows runtime scalars to be configured without a dictionary with the key
value
and instead just use the scalar value directly. However this still leaves the option to load scalars from a json or pickle file.- Parameters
scalar_type (type) – The scalar type of values that this configuration type can hold. For example,
int
,float
,bool
, orstr
.non_scalar_schema (ConfigSchema) – The schema of a non-scalar Dagster configuration type. For example,
List
,Dict
, orSelector
.key (Optional[str]) – The configuation type’s unique key. If not set, then the key will be set to
ScalarUnion.{scalar_type}-{non_scalar_schema}
.
Examples:
solids: transform_word: inputs: word: value: foobar
becomes, optionally,
solids: transform_word: inputs: word: foobar
-
dagster.
StringSource
¶ Use this type when you want to read a string config value from an environment variable. The value passed to a config field of this type may either be a string literal, or a selector describing how to look up the value from the executing process’s environment variables.
Examples:
@solid(config_schema=StringSource) def secret_solid(context) -> str: return context.solid_config execute_solid( secret_solid, run_config={ 'solids': {'secret_solid': {'config': 'test_value'}} } ) execute_solid( secret_solid, run_config={ 'solids': {'secret_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE'}}} } )
-
dagster.
IntSource
¶ Use this type when you want to read an integer config value from an environment variable. The value passed to a config field of this type may either be a integer literal, or a selector describing how to look up the value from the executing process’s environment variables.
Examples:
@solid(config_schema=IntSource) def secret_int_solid(context) -> str: return context.solid_config execute_solid( secret_int_solid, run_config={ 'solids': {'secret_int_solid': {'config': 3}} } ) execute_solid( secret_int_solid, run_config={ 'solids': {'secret_int_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_INT'}}} } )
-
dagster.
BoolSource
¶ Use this type when you want to read an boolean config value from an environment variable. The value passed to a config field of this type may either be a boolean literal, or a selector describing how to look up the value from the executing process’s environment variables. Set the value of the corresponding environment variable to
""
to indicateFalse
.Examples:
@solid(config_schema=BoolSource) def secret_bool_solid(context) -> str: return context.solid_config execute_solid( secret_bool_solid, run_config={ 'solids': {'secret_bool_solid': {'config': True}} } ) execute_solid( secret_bool_solid, run_config={ 'solids': {'secret_bool_solid': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_BOOL'}}} } )
Config Utilities¶
-
@
dagster.
configured
(configurable: dagster.core.definitions.configurable.ConfigurableDefinition, config_schema: Optional[Dict[str, Any]] = None, **kwargs: Any)[source]¶ A decorator that makes it easy to create a function-configured version of an object. The following definition types can be configured using this function:
If the config that will be supplied to the object is constant, you may alternatively invoke this and call the result with a dict of config values to be curried. Examples of both strategies below.
- Parameters
configurable (ConfigurableDefinition) – An object that can be configured.
config_schema (ConfigSchema) – The config schema that the inputs to the decorated function must satisfy.
**kwargs – Arbitrary keyword arguments that will be passed to the initializer of the returned object.
- Returns
(Callable[[Union[Any, Callable[[Any], Any]]], ConfigurableDefinition])
Examples:
dev_s3 = configured(s3_resource, name="dev_s3")({'bucket': 'dev'}) @configured(s3_resource): def dev_s3(_): return {'bucket': 'dev'} @configured(s3_resource, {'bucket_prefix', str}): def dev_s3(config): return {'bucket': config['bucket_prefix'] + 'dev'}