Partitions

class dagster.Partition[source]

Partition is the representation of a logical slice across an axis of a pipeline’s work

Parameters
  • value (Any) – The object for this partition

  • name (str) – Name for this partition

class dagster.PartitionSetDefinition[source]

Defines a partition set, representing the set of slices making up an axis of a pipeline

Parameters
  • name (str) – Name for this partition set

  • pipeline_name (str) – The name of the pipeline definition

  • partition_fn (Callable[void, List[Partition]]) – User-provided function to define the set of valid partition objects.

  • solid_selection (Optional[List[str]]) – A list of solid subselection (including single solid names) to execute with this partition. e.g. ['*some_solid+', 'other_solid']

  • mode (Optional[str]) – The mode to apply when executing this partition. (default: ‘default’)

  • run_config_fn_for_partition (Callable[[Partition], [Dict]]) – A function that takes a Partition and returns the run configuration that parameterizes the execution for this partition, as a dict

  • tags_fn_for_partition (Callable[[Partition], Optional[dict[str, str]]]) – A function that takes a Partition and returns a list of key value pairs that will be added to the generated run for this partition.

dagster.utils.partitions.date_partition_range(start, end=None, delta=None, delta_range='days', fmt=None, inclusive=False, timezone=None)[source]

Utility function that returns a partition generating function to be used in creating a PartitionSet definition.

Parameters
  • start (datetime) – Datetime capturing the start of the time range.

  • end (Optional(datetime)) – Datetime capturing the end of the partition. By default, the current time is used. The range is not inclusive of the end value.

  • delta (Optional(timedelta)) – Timedelta representing the time duration of each partition. DEPRECATED: use ‘delta_range’ instead, which handles timezone transitions correctly.

  • delta_range (Optional(str)) – string representing the time duration of each partition. Must be a valid argument to pendulum.period.range (“days”, “hours”, “months”, etc.).

  • fmt (Optional(str)) – Format string to represent each partition by its start time

  • inclusive (Optional(bool)) – By default, the partition set only contains date interval partitions for which the end time of the interval is less than current time. In other words, the partition set contains date interval partitions that are completely in the past. If inclusive is set to True, then the partition set will include all date interval partitions for which the start time of the interval is less than the current time.

  • timezone (Optional(str)) – Timezone in which the partition values should be expressed.

Returns

Callable[[], List[Partition]]