dagster code-server startcommand that can be used to launch a code server, much like
dagster api grpc. Unlike
dagster api grpc, however,
dagster code-server startruns the code in a subprocess, so it can reload code from the Dagster UI without needing to restart the command. This can be useful for jobs that load code from some external source and may want to reload job definitions without restarting the process.
dagster.yamlthat can be used to decrease latency when a sensor emits multiple run requests within a single tick. See the docs for more information.
k8s_job_executorcan now be used to launch each step of a job in its own Kubernetes, pod, even if the Dagster deployment is not using the
K8sRunLauncherto launch each run in its own Kubernetes pod.
io_manager_defparam on an asset.
maxResumeRunAttemptsto null in the helm chart would cause it to be set to a default value of 3 instead of disabling run retries.
k8s_job_executorwould sometimes fail with a 409 Conflict error after retrying the creation of a Kubernetes pod for a step, due to the job having already been created during a previous attempt despite raising an error.
op_namewas passed to
load_assets_from_dbt_manifest, and a
selectparameter was specified, a suffix would be appended to the desired op name.
dagster_dbt.cli. This new resource only support
dagster_dbt.asset_decoratorthat allows you to specify a compute function for a selected set of dbt assets that loaded as an
max_materializations_per_minuteparameter (with a default of 1) to
AutoMaterializationPolicy.lazy()allows you to set bounds on the volume of work that may be automatically kicked off for each asset. To restore the previous behavior, you can explicitly set this limit to
MonthlyPartitionsDefinitionnow support and
build_asset_reconciliation_sensor, a single new data version from an observable source asset could trigger multiple runs of the downstream assets. This has been fixed.
IntEnvVarwithin raw run config - although they just returned the name of the env var rather than retrieve its value. This has been fixed to error directly.
*was being interpreted as glob pattern, rather than as a dbt selector argument. We now explicitly set the default selection pattern as
dagster-cloud serverless deploydid not create a unique image tag if the
--imagetag was not specified.
TimeWindowPartitionMappingobjects are now current-time aware. Subsequently, only upstream/downstream partitions existent at the current time are returned.
ExecuteJobResultwas renamed to
ExecuteJobResultremains a deprecated alias)
AssetSelection.key_prefixesmethod allows matching asset keys starting with a provided prefix.
ConfigurablePickledObjectADLS2IOManagerthat uses pythonic config
DataProcResourcefollows the Pythonic resource system. The existing
private_keys that have been
base64encoded to avoid issues with newlines in the private key. Non-base64 encoded keys are still supported. See the
SnowflakeIOManagerdocumentation for more information on
base64encoded private keys.
daemon_hearbeatstable for existing Postgres storage instances that were migrating from before
1.2.2. This should unblock users relying on the existence of a primary key constraint for replication.
SensorResultevaluation where multipartitioned run requests containing a dynamic partition added in a dynamic partitions request object would raise an invalid partition key error.
ExecuteJobResultis now a deprecated alias for the new name,
load_assets_from_connections, you may now provide an instance of the
AirbyteResourceclass, rather than just
dagster-celeryCLI accepted an inconsistent configuration format - it now matches the same format as the
celery_executor. Thanks @boenshao!
ecs_timeoutin your ECS user code launcher config to extend how long the ECS agent polls for new code servers to start. Extending this timeout is useful if your code server takes an unusually long time to start up - for example, because it uses a very large image.
load_assets_from_package_module and the other core
load_assets_from_ methods now accept a
source_key_prefix argument, which allows applying a key prefix to all the source assets that are loaded.
OpExecutionContext now has an
RunFailureSensorContext now has a
The Pythonic resource system now supports a set of lifecycle hooks which can be used to manage setup and teardown:
class MyAPIClientResource(ConfigurableResource): api_key: str _internal_client: MyAPIClient = PrivateAttr() def setup_for_execution(self, context): self._internal_client = MyAPIClient(self.api_key) def get_all_items(self): return self._internal_client.items.get()
Added support for specifying input and output config on
SubmitRunContext are now exposed as public dagster exports.
[ui] Downstream cross-location dependencies of all source assets are now visible on the asset graph. Previously these dependencies were only displayed if the source asset was defined as a regular asset.
[ui] A new filtering experience is available on the Runs page after enabling feature flag “Experimental Runs table view with filtering”.
[dagster-aws] Allow the S3 compute log manager to specify a
show_url_only: true config option, which will display a URL to the S3 file in dagit, instead of the contents of the log file.
PickledObjectS3IOManager now fully supports loading partitioned inputs.
PickedObjectADLS2IOManager now fully supports loading partitioned inputs.
ConfigurablePickledObjectGCSIOManager follow the Pythonic resource system. The existing
gcs_pickle_io_manager remain supported.
BigQueryResource follows the Pythonic resource system. The existing
bigquery_resource remains supported.
PickledObjectGCSIOManager now fully supports loading partitioned inputs.
[dagster-postgres] The event watching implementation has been moved from listen/notify based to the polling watcher used by MySQL and SQLite.
make_slack_on_run_failure_sensor, thanks @danielgafni!
SnowflakeResource follows the Pythonic resource system. The existing
snowflake_resource remains supported.
AssetMetadataValue.valuethat would cause an infinite recursion error.
volumeMountsvalues have been added to the agent helm chart.
load_assets_from_airbyte_projectnow take a
AutoMaterializePolicys on Airbyte assets
NumReplicasparameter on the agent template in CloudFormation, or the
dagsterCloudAgent.replicasfield in Helm.
enableZeroDowntimeDeploysparameter to true in the CloudFormation stack for your agent.
AssetsDefinition.from_graph, as well as the
@graph_multi_assetdecorators now support specifying
async defops/assets no longer prematurely finalize async generators during execution.
build_asset_reconciliation_sensor) could incorrectly launch new runs for partitions that already had an in-progress run. This has been fixed.
run_request_for_partitionnow throws an error. Instead, users should yield directly instantiated run requests via
graph_multi_assetnow support specifying
resource_defsdirectly (thanks @kmontag42)!
node_info_to_auto_materialize_policy_fnparam added to
load_assets_from_dbt_*functions. (thanks @askvinni)!
allPartitionsargument is provided.