The replacement for
Job binds a
Graph and the resources it needs to be executable.
Jobs are created by calling
GraphDefinition.to_job() on a graph instance, or using the
job(name=None, description=None, resource_defs=None, config=None, tags=None, logger_defs=None, executor_def=None, hooks=None, op_retry_policy=None, version_strategy=None, partitions_def=None)¶
Creates a job with the specified parameters from the decorated graph/op invocation function.
Using this decorator allows you to build an executable job by writing a function that invokes ops (or graphs).
name (Optional[str]) – The name for the Job. Defaults to the name of the this graph.
Describes how the job is parameterized at runtime.
If no value is provided, then the schema for the job’s run config is a standard format based on its ops and resources.
If a dictionary is provided, then it must conform to the standard config schema, and it will be used as the job’s run config for the job whenever the job is executed. The values provided will be viewable and editable in the Dagit playground, so be careful with secrets.
ConfigMapping object is provided, then the schema for the job’s run config is
determined by the config mapping, and the ConfigMapping, which should return
configuration in the standard format to configure the job.
PartitionedConfig object is provided, then it defines a discrete set of config
values that can parameterize the pipeline, as well as a function for mapping those
values to the base config. The values provided will be viewable and editable in the
Dagit playground, so be careful with secrets.
tags (Optional[Dict[str, Any]]) – Arbitrary metadata for any execution of the Job. Values that are not strings will be json encoded and must meet the criteria that json.loads(json.dumps(value)) == value. These tag values may be overwritten by tag values provided at invocation time.
op_retry_policy (Optional[RetryPolicy]) – The default retry policy for all ops in this job. Only used if retry policy is not defined on the op definition or op invocation.
version_strategy (Optional[VersionStrategy]) – Defines how each op (and optionally, resource) in the job can be versioned. If provided, memoizaton will be enabled for this job.
partitions_def (Optional[PartitionsDefinition]) – Defines a discrete set of partition keys that can parameterize the job. If this argument is supplied, the config argument can’t also be supplied.
JobDefinition(mode_def, graph_def, name=None, description=None, preset_defs=None, tags=None, hook_defs=None, op_retry_policy=None, version_strategy=None, _op_selection_data=None)¶
execute_in_process(run_config=None, instance=None, partition_key=None, raise_on_error=True, op_selection=None)¶
Execute the Job in-process, gathering results in-memory.
The executor_def on the Job will be ignored, and replaced with the in-process executor. If using the default io_manager, it will switch from filesystem to in-memory.
(Optional[Dict[str (run_config) – The configuration for the run
Any]] – The configuration for the run
instance (Optional[DagsterInstance]) – The instance to execute against, an ephemeral one will be used if none provided.
partition_key – (Optional[str]) The string partition key that specifies the run config to execute. Can only be used to select run config for jobs with partitioned config.
raise_on_error (Optional[bool]) – Whether or not to raise exceptions when they occur.
op_selection (Optional[List[str]]) – A list of op selection queries (including single op
names) to execute. For example:
some_op and all its ancestors (upstream dependencies).
some_op, all its ancestors, and its descendants
(downstream dependencies) within 3 levels down.
['*some_op', 'other_op_a', 'other_op_b+']: select
some_op and all its
other_op_a itself, and
other_op_b and its direct child ops.
Apply a set of hooks to all op instances within the job.